Building a Test Dashboard in Metabase

Most CI pipelines emit JUnit XML, ship it somewhere, and call it done. The green/red badge satisfies the PR gate, but it tells you nothing about whether your suite is getting slower, which five tests show up in every postmortem, or whether a recent framework upgrade silently inflated your P95 run time by 40 seconds. That signal exists — it just lives in a database nobody queries.

Metabase sits in a useful middle ground for test analytics: it speaks SQL fluently, connects directly to PostgreSQL, BigQuery, or ClickHouse where you're already storing parsed JUnit results, and produces shareable dashboards without a Grafana deployment or a dedicated data-eng team. The trade-off is real — Metabase has no native concept of test flakiness, no retry-aware aggregation, and no streaming — but for teams that want a test analytics layer without standing up a full observability stack, it earns its place.

By the end of this article you'll have a working schema, the SQL queries that power the four most valuable panels — pass rate trend, slowest tests, flake rate by suite, and failure frequency heatmap — and a clear picture of where Metabase will fight you.

What a Test Engineer Dashboard in Metabase Actually Models

A Metabase test dashboard is a SQL-backed reporting layer over a normalized test-results store. The canonical schema has three tables: runs (one row per CI pipeline execution, with branch, commit SHA, trigger type, and wall-clock duration), test_cases (one row per test per run, with status, duration_ms, suite name, and retry count), and failures (one row per failure with message, stack digest, and a foreign key back to test_cases). Parse your JUnit XML in CI — a 30-line Python script using xml.etree.ElementTree is enough — and INSERT on every run. ClickHouse handles the append-only write pattern best at scale; PostgreSQL works fine under 50 million rows.

This fits into a centralized test automation reporting architecture as the presentation tier. Metabase connects over a read replica, so your CI writes never contend with dashboard queries. The dashboard itself becomes the artifact your engineering leaders, SDETs, and on-call engineers share — one URL, always current, no Slack screenshot chains.

Building the Four Core Panels: SQL, Schema, and Measurable Outcomes

Start with the pass rate trend panel. A 7-day rolling view grouped by day and branch surfaces regressions that a single-run badge hides entirely.

-- Pass rate by day, last 30 days
SELECT
  DATE_TRUNC('day', r.started_at)  AS run_date,
  r.branch,
  ROUND(
    100.0 * SUM(CASE WHEN tc.status = 'passed' THEN 1 ELSE 0 END)
    / COUNT(*), 2
  ) AS pass_rate_pct
FROM test_cases tc
JOIN runs r ON r.id = tc.run_id
WHERE r.started_at >= NOW() - INTERVAL '30 days'
GROUP BY 1, 2
ORDER BY 1 DESC, 2;

Wire this to a Metabase Line chart with run_date on X, pass_rate_pct on Y, and branch as the series breakout. Set a Goal Line at 98% — anything below it for more than two consecutive days should trigger a Slack alert via Metabase's built-in pulse or a webhook.

The flake rate by suite panel is where Metabase earns real triage value. Flakiness here is defined as a test that has both a passed and a failed status across runs on the same branch within a 7-day window — not just any failure.

-- Flaky tests: passed AND failed on same branch, last 7 days
SELECT
  tc.suite_name,
  tc.test_name,
  COUNT(DISTINCT CASE WHEN tc.status = 'failed' THEN r.id END) AS fail_runs,
  COUNT(DISTINCT CASE WHEN tc.status = 'passed' THEN r.id END) AS pass_runs,
  COUNT(DISTINCT r.id) AS total_runs,
  ROUND(100.0 * COUNT(DISTINCT CASE WHEN tc.status = 'failed' THEN r.id END)
    / COUNT(DISTINCT r.id), 1) AS flake_rate_pct
FROM test_cases tc
JOIN runs r ON r.id = tc.run_id
WHERE r.started_at >= NOW() - INTERVAL '7 days'
  AND r.branch = 'main'
GROUP BY 1, 2
HAVING fail_runs > 0 AND pass_runs > 0
ORDER BY flake_rate_pct DESC
LIMIT 25;

Display this as a Metabase Table with conditional formatting: red above 20%, yellow 5–20%. One team using this exact query against a PostgreSQL store of Playwright results cut triage time from 22 minutes per failure to under 4 once engineers could see, at a glance, that 80% of their Monday failures came from three tests in the checkout suite with a flake rate above 35%. The slowest tests P95 panel and a failure heatmap (failures by suite × day-of-week) round out the dashboard; both are straightforward PERCENTILE_CONT and COUNT GROUP BY queries respectively. For the heatmap, use Metabase's Pivot Table visualization with suite on rows, day-of-week on columns, and failure count as the value — it renders as a heat grid natively.

Pitfalls That Bite Even Experienced Teams

Storing raw JUnit XML in a text column is the most common mistake. It feels fast to ship, but every dashboard query becomes a regex or XPath extraction at query time, and Metabase's SQL editor won't save you from a 45-second full-table scan. Parse at ingest — in your GitHub Actions step, CircleCI orb post-step, or a Buildkite plugin — and write normalized rows. The 30-line Python parser costs you once; the full-table scan costs you on every refresh. A second mistake is not storing retry counts. If your Pytest or Playwright config retries twice on failure and you only store the final status, your flake query is blind. Add a retry_count INTEGER column and populate it from the JUnit rerun elements or from your framework's JSON reporter. Without it, you're measuring deterministic failures and flakes with the same instrument.

A third org-level failure: dashboard ownership falls to nobody. Metabase questions drift — someone changes the schema, the query silently returns zero rows, and the dashboard looks healthy. Assign a named owner per question, add a -- last verified: YYYY-MM-DD comment in each SQL block, and wire a Metabase alert on "result count = 0 for 2 consecutive days" to a Slack channel. The alert costs five minutes to set up and has caught broken ingestion pipelines more than once.

What Most Teams Get Wrong About Test Dashboards

Pass rate is not a quality signal — it's a lagging indicator of suite health. A suite that retries aggressively can post 99% pass rate while hiding a 30% flake rate underneath. If your dashboard only shows pass/fail aggregates, you're reading the output of your retry policy, not your code quality. The signal that actually drives decisions is failure frequency per test, duration variance over time, and the correlation between specific commits and failure spikes. These are the panels that let an engineering leader act on quality KPIs rather than just report them.

Metabase is not a replacement for a purpose-built test observability tool. It has no native support for distributed tracing correlation, no OpenTelemetry ingestion, and no AI-assisted failure clustering. Use Metabase when your team already has a SQL store of test results and needs a low-friction reporting layer with good sharing UX. Use ReportPortal when you need ML-based failure triage at scale across thousands of daily runs. Use Datadog or Honeycomb when your test failures need to be correlated with service-level traces. Continuous quality feedback loops require the right tool at the right layer — Metabase covers the analytics layer well, but it doesn't own the full loop.

A Metabase test dashboard built on a normalized schema — runs, test_cases, failures — with the four core panels described here gives your team a shared, queryable view of suite health in under a day of setup. The next concrete step: instrument your CI pipeline to parse JUnit XML at the post-step level and write to your database on every run. Once you have 30 days of data, the flake rate and duration variance queries will immediately surface the tests worth fixing first.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.