Metabase for Test Analytics: Limits & Wins

Most teams treat test results like a checkbox: green is good, red is bad, ship or block. The interesting signal lives between those two states — runtime variance, retry patterns, the same eight tests appearing in every postmortem. When someone finally asks "can pass/fail be metrics?" the answer is yes, but only if you store them properly and query them honestly. Metabase gets proposed early in that conversation because it's already connected to the company Postgres or BigQuery instance, the UI is approachable, and it requires no dedicated data engineer to stand up.

The problem is that Metabase is a general-purpose BI tool, not a test observability platform. It will happily let you build a dashboard that looks authoritative while quietly hiding the nuance that makes test data actionable. Understanding exactly where that line sits saves you from shipping a vanity dashboard to leadership and calling it done.

By the end of this article you'll know which test analytics Metabase handles well, which SQL patterns make it useful, and where you need a purpose-built layer — Allure, ReportPortal, or a time-series store — to get the signal that actually drives decisions.

What Metabase Actually Sees in Your Test Data

Metabase queries whatever is in your relational or columnar store. It doesn't parse JUnit XML, consume OpenTelemetry spans, or talk to your CI API directly. That means your test results have to be structured first — rows in Postgres, BigQuery, or ClickHouse with columns like test_name, suite, status, duration_ms, run_at, branch, and build_id. If you're already shipping parsed JUnit output to a warehouse (via a custom GitHub Actions step, a Buildkite plugin, or a tool like ReportPortal's data export), Metabase can read that table and produce trend charts, failure-rate breakdowns, and duration histograms with no custom code.

Where Metabase fits in a modern test architecture is at the reporting and exploration layer, not the ingestion or correlation layer. It sits downstream of your parser, downstream of your flakiness detector, and downstream of any enrichment (git blame, JIRA ticket, SLO tag). Think of it as the visualization surface for a warehouse you've already built. If that warehouse doesn't exist yet, Metabase won't help you build it — and that's the first thing most teams misunderstand when they reach for it.

Building Useful Test Dashboards in Metabase

Assume you have a test_runs table in Postgres with the schema above. The first query worth building is a rolling 14-day failure rate per suite, grouped by branch. Metabase's GUI query builder can produce this, but the SQL editor gives you control over the window:

SELECT
  suite,
  branch,
  DATE_TRUNC('day', run_at) AS run_date,
  COUNT(*) FILTER (WHERE status = 'failed') AS failures,
  COUNT(*) AS total,
  ROUND(
    100.0 * COUNT(*) FILTER (WHERE status = 'failed') / NULLIF(COUNT(*), 0),
    2
  ) AS failure_rate_pct
FROM test_runs
WHERE run_at >= NOW() - INTERVAL '14 days'
  AND status != 'skipped'
GROUP BY suite, branch, run_date
ORDER BY run_date DESC, failure_rate_pct DESC;

The status != 'skipped' filter matters more than it looks. Skipped tests distort every rate you report — include them in the denominator and your failure rate will drop every time someone marks a test as pending. Metabase won't warn you about this; it just computes what you give it.

For flakiness, you need a test that has both passed and failed across multiple runs of the same commit or build. A simple flake detector query:

SELECT
  test_name,
  suite,
  COUNT(DISTINCT status) AS distinct_statuses,
  COUNT(*) AS total_runs,
  MAX(run_at) AS last_seen
FROM test_runs
WHERE run_at >= NOW() - INTERVAL '30 days'
GROUP BY test_name, suite
HAVING COUNT(DISTINCT status) > 1
   AND COUNT(*) >= 5
ORDER BY total_runs DESC;

This surfaces tests that flip between pass and fail with enough sample size to be meaningful. Wire this as a Metabase table card pinned to your CI channel via the Slack notification feature, and triage time for "is this flaky or a real regression?" dropped from over 20 minutes of log-diving to under 4 in practice — because the on-call engineer sees the flip count before opening a single build log. For deeper per-test instability analysis, tooling that traces the root causes of unstable Python tests gives you the next layer Metabase can't reach on its own.

Duration percentiles are where Metabase's charting genuinely shines. Use a PERCENTILE_CONT query in Postgres or BigQuery and plot P50/P90/P95 per suite over time. A suite whose P95 runtime doubles over two weeks is a signal worth a Slack alert before it starts blocking pipelines. Set a Metabase pulse on that card with a threshold filter — it's not PagerDuty, but it's good enough for a non-production signal.

Where Metabase Dashboards Break Down in Practice

The most common mistake is treating Metabase's time-series charts as flakiness dashboards. Metabase plots aggregate counts well, but it has no concept of test identity across runs, no retry-aware deduplication, and no way to distinguish a test that failed once from one that failed and retried three times before passing. If your CI framework retries on failure (Pytest's --reruns, Playwright's retries config), and you're inserting every attempt into test_runs, your failure rate is inflated and your flake detector above will fire on almost everything. You need to deduplicate by (build_id, test_name, MAX(attempt)) before Metabase ever sees the data. Flake rate averages hide your riskiest tests precisely because this kind of raw aggregation obscures which tests are actually blocking releases.

The second pitfall is organizational: Metabase dashboards get built by whoever has warehouse access, shared to a Confluence page, and then never maintained. Test schemas evolve — a new column, a renamed suite, a status value that changes from "error" to "broken" — and the dashboard silently returns wrong numbers. Assign ownership. Add a last_validated note to each card description. Treat it like production code, not a one-time report.

Can Pass/Fail Be Metrics? Yes — With These Caveats

The question "can pass/fail be metrics?" comes up constantly, and the honest answer is: pass/fail ratios are necessary metrics, not sufficient ones. A dashboard showing 98% pass rate looks healthy until you realize the suite grew from 200 to 2,000 tests and the 2% failure is always the same 40 tests. Failure rate falls when you add tests to a passing suite — Metabase will chart that decline and make things look like they're improving. Without normalizing by test count or tracking per-test failure frequency, pass/fail aggregates mislead more than they inform. Metabase has no built-in guard against this; the SQL you write is the only protection.

The second myth is that a Metabase dashboard replaces purpose-built tools. It doesn't. Allure gives you per-test history, attachments, and step-level breakdowns — use it when developers need to debug individual failures. ReportPortal adds ML-based auto-analysis and launch comparison — use it when you're running hundreds of builds a day and need pattern detection you can't write in SQL. Metabase earns its place for cross-suite trend reporting, executive-facing KPIs, and ad-hoc exploration against a warehouse you already maintain. For the broader CI observability metrics that sit alongside flakiness — queue depth, agent utilization, pipeline duration percentiles — Metabase with a well-structured table is a reasonable starting point before you graduate to Grafana + Prometheus.

Metabase is a solid first layer for test analytics if your data is already structured and your SQL is honest about retries, skips, and sample size. Its limits are real: no retry-aware deduplication, no test-identity tracking, no ingestion pipeline. Start with the failure-rate and flake-detector queries above, assign dashboard ownership, and treat schema changes as breaking changes. When Metabase starts feeling like a workaround rather than a tool, that's the signal to evaluate ReportPortal or a dedicated time-series stack.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.