CI Test Dashboard: Run Results & Failure Triage

Most teams treat test results like a checkbox: green is good, red is bad, ship or block. The interesting signal lives in everything between those two states — runtime variance, retry counts, the same five tests appearing in every postmortem. That signal is where engineering decisions actually get made, and a well-built CI test dashboard is the instrument that surfaces it.

The technical problem is data scatter. JUnit XML lands in an S3 bucket, Allure reports live on a separate host, Grafana watches infrastructure, and nobody has joined those streams into a single failure triage view. When an on-call engineer needs to know whether tonight's red build is a product regression or an infra flake, they're tabbing between four tools and losing ten minutes before they've written a single query.

By the end of this article you'll have a concrete schema for storing run results, a set of triage charts worth building, and the queries that power them — plus the mental model for what your dashboard should answer versus what it should deliberately ignore.

What a CI Test Dashboard Actually Measures

A CI test dashboard is not a pretty wrapper around pass/fail counts. It's a time-series view of test run behavior across dimensions: suite, branch, environment, executor, and retry depth. The primary entities are runs (a pipeline execution), cases (individual test outcomes within a run), and spans (timing and dependency data per case). When those three are stored relationally — or in a columnar store like ClickHouse or BigQuery — you can answer questions that raw JUnit XML cannot: which tests regressed on main in the last 72 hours, which failures co-occur with a specific worker pool, and what the P95 duration looks like for a given test class over rolling 14 days.

In a modern test architecture, this dashboard sits downstream of your CI system (GitHub Actions, Jenkins, Buildkite) and upstream of your alerting layer (PagerDuty, Slack). It consumes structured test result events — either parsed JUnit XML or OpenTelemetry spans emitted by your test runner — and feeds aggregated metrics to Grafana or a purpose-built tool like ReportPortal. The embedded CI failure analysis dashboard pattern goes further, co-locating triage context directly inside the pipeline UI rather than requiring a separate hop to an external tool.

Building the Triage Charts: Schema, Queries, and Panels

Start with a minimal PostgreSQL (or ClickHouse) schema that captures what you need for failure triage without over-engineering it:

CREATE TABLE test_cases (
  run_id        TEXT        NOT NULL,
  suite         TEXT        NOT NULL,
  test_name     TEXT        NOT NULL,
  status        TEXT        NOT NULL,  -- passed | failed | skipped | flaky
  duration_ms   INTEGER     NOT NULL,
  retry_count   SMALLINT    DEFAULT 0,
  branch        TEXT,
  executor_id   TEXT,
  recorded_at   TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX ON test_cases (test_name, recorded_at DESC);
CREATE INDEX ON test_cases (status, branch, recorded_at DESC);

With that in place, the most useful triage chart is failure rate by test over a rolling window — not aggregate suite health, but per-test signal so you can see which case is responsible for the noise:

-- Top 20 failing tests on main, last 7 days
SELECT
  test_name,
  COUNT(*)                                        AS total_runs,
  SUM(CASE WHEN status = 'failed' THEN 1 END)    AS failures,
  ROUND(
    100.0 * SUM(CASE WHEN status = 'failed' THEN 1 END) / COUNT(*), 2
  )                                               AS failure_rate_pct,
  PERCENTILE_CONT(0.95) WITHIN GROUP
    (ORDER BY duration_ms)                        AS p95_ms
FROM test_cases
WHERE branch = 'main'
  AND recorded_at >= NOW() - INTERVAL '7 days'
GROUP BY test_name
HAVING COUNT(*) >= 10          -- ignore low-volume noise
ORDER BY failure_rate_pct DESC
LIMIT 20;

Feed that query into a Grafana table panel with conditional coloring: failure_rate_pct above 5% gets amber, above 15% gets red. The threshold isn't arbitrary — below 5% is usually noise or environment variance; above 15% on a high-frequency test is a defect or a systemic infra problem worth a dedicated ticket. Pairing this table with a time-series panel of retry_count per test exposes flakes that pass on retry and would otherwise be invisible in a pure pass/fail view. For the Grafana + Loki wiring that ties log context to these failure rows, the test failure triage with Grafana and Loki pattern is the cleanest reference implementation — triage time dropped from 22 minutes per failure to under 4 once log correlation was wired directly into the panel.

For GitHub Actions specifically, emit structured JSON from your test runner and parse it in a post-step:

# .github/workflows/ci.yml (post-test step)
- name: Upload test metrics
  if: always()
  run: |
    python scripts/parse_junit.py \
      --input reports/junit.xml \
      --branch "${{ github.ref_name }}" \
      --run-id "${{ github.run_id }}" \
      --pg-dsn "${{ secrets.TEST_DB_DSN }}"

The if: always() is non-negotiable — you need failure data from failed runs, not just successful ones. The GitHub Actions test results visualization guide covers the annotation API and summary markdown rendering if you want triage context surfaced directly in the PR check UI without leaving GitHub.

Pitfalls That Undermine Triage Dashboards in Practice

Aggregating at the suite level instead of the test level is the most common structural mistake. Suite-level pass rates look stable even when three tests are failing on 40% of runs — the passing majority masks the signal. The fix is obvious in retrospect: your primary grouping key must be test_name, not suite. Suite-level rollups belong in an executive summary, not a triage view. This mistake persists because most out-of-the-box CI dashboards (Jenkins Blue Ocean, GitHub's built-in checks) default to suite aggregation and teams never replace them with something more granular.

Ignoring retry depth is the second failure mode. A test that passes on retry 2 of 3 is not a passing test — it's a flake that's hiding behind your retry budget. If your schema doesn't capture retry_count, you're systematically underreporting failure rates and building false confidence into your dashboards. The third mistake is dashboard sprawl without ownership: five Grafana dashboards built by five different engineers, none of them alerting, none of them linked from the postmortem template. A dashboard nobody checks during an incident isn't an observability asset — it's decoration.

What Most Teams Get Wrong About Test Metrics

Pass rate is not a quality metric — it's a lagging indicator that can stay high while real defects accumulate. A suite with a 98% pass rate and 40 tests in permanent skip, 12 tests behind a feature flag, and 6 tests with retry_count ≥ 2 is not a healthy suite. The number that matters is unretried pass rate on first attempt, broken down by test age and change frequency. Similarly, treating flakiness as unfixable background noise is a learned helplessness that compounds over time. Flakes have structure — failure sequencing analysis reliably separates infra-induced flakes (bursty, correlated with executor churn) from logic flakes (random, uncorrelated with environment) and points you at the right fix faster than bisecting individual failures.

The other persistent myth is that a dashboard solves the triage problem by itself. Dashboards surface patterns; they don't explain causation. A spike in failure rate after a deploy could be a regression, a config drift, or — counterintuitively — the result of fixing a flake that was previously masking real failures. That last scenario is documented and worth understanding: failure rate spikes after a flake is fixed because the retry buffer that was suppressing real failures disappears. Without that context, an on-call engineer reads the dashboard spike as a new regression and wastes an hour chasing the wrong commit.

A CI test dashboard earns its keep when on-call engineers open it first, not last. Start with the per-test failure rate query above, wire retry depth into your schema from day one, and set Grafana alert thresholds at 5% and 15% failure rate per test. Once the basics are stable, layer in quality KPIs that engineering leaders actually use to connect test signal to delivery outcomes. The data is already in your CI runs — the only missing piece is the schema and the will to query it.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.