Test Run KPIs: Pass Rate, Failure Signatures

Most teams treat a test run dashboard as a scoreboard: green build ships, red build blocks, repeat. The useful signal is elsewhere — in the gap between your first-attempt pass rate and your final pass rate, in the five test names that appear in every postmortem, in the runtime variance that precedes a flop week by three days. Those are the numbers that drive real engineering decisions.

The problem is that most CI dashboards conflate these signals. A single "pass rate" number that includes retries, skips, and quarantined tests tells you almost nothing about suite health. It tells you even less about where to spend the next engineering hour.

By the end of this article you'll have a concrete schema for tracking first-attempt vs. final pass rate, a SQL-based approach to surfacing failure signatures, and a Grafana panel structure that makes test triage a sub-five-minute task instead of a twenty-minute archaeology dig.

First-Attempt Pass Rate vs. Final Pass Rate: Why Both Numbers Belong on Your Dashboard

First-attempt pass rate is the fraction of test executions that pass on their first run, with no retries. Final pass rate is the fraction that pass after all allowed retries are exhausted. The delta between them is your flakiness tax. A suite running at 97% final pass rate but 84% first-attempt pass rate isn't healthy — it's burning 13% of your CI capacity on retries and masking instability that will eventually become a production incident. As detailed in the analysis of how retry count inflates pass rate without fixing flakes, this gap grows silently until a retry budget change exposes it all at once.

Failure signatures are the complementary signal. A failure signature is the tuple of (test name, failure message fingerprint, stack trace hash, branch, environment) that lets you cluster individual failures into recurring patterns. Without signature grouping, every failure looks unique and triage starts from zero each time. With it, you can see that 60% of your red builds trace to three root causes — and prioritize accordingly. These two KPIs — first-attempt rate and failure signature frequency — are the minimum viable observability layer for any CI test dashboard worth building.

Building the Dashboard: Schema, Queries, and Grafana Panels for Test Triage

Start with a test_runs table in PostgreSQL or ClickHouse. The schema needs at minimum: run_id, test_name, suite, branch, attempt_number, status (passed/failed/skipped), duration_ms, failure_message, stack_hash, and created_at. JUnit XML from Pytest, Playwright, or any standard runner maps cleanly onto this with a small ingest script. Allure and ReportPortal both expose REST APIs that return equivalent fields if you're already using a report aggregator.

-- First-attempt pass rate per day, last 30 days
SELECT
  date_trunc('day', created_at)        AS run_date,
  COUNT(*) FILTER (WHERE attempt_number = 1)                          AS first_attempts,
  COUNT(*) FILTER (WHERE attempt_number = 1 AND status = 'passed')    AS first_attempt_passes,
  ROUND(
    COUNT(*) FILTER (WHERE attempt_number = 1 AND status = 'passed')::numeric
    / NULLIF(COUNT(*) FILTER (WHERE attempt_number = 1), 0) * 100, 2
  )                                                                    AS first_attempt_pass_rate,
  ROUND(
    COUNT(*) FILTER (WHERE status = 'passed' AND attempt_number = (
      SELECT MAX(t2.attempt_number) FROM test_runs t2 WHERE t2.run_id = test_runs.run_id
        AND t2.test_name = test_runs.test_name
    ))::numeric / NULLIF(COUNT(DISTINCT run_id || test_name), 0) * 100, 2
  )                                                                    AS final_pass_rate
FROM test_runs
WHERE created_at >= now() - interval '30 days'
GROUP BY 1
ORDER BY 1;

For failure signatures, hash the first two frames of the stack trace and group by that hash alongside the normalized failure message. A Python ingest step using hashlib.md5 on '\n'.join(tb_lines[:2]).encode() is enough to collapse 80% of duplicate signatures without a full ML pipeline. Store the hash in stack_hash at ingest time, then query it:

-- Top failure signatures, last 7 days
SELECT
  stack_hash,
  failure_message,
  COUNT(DISTINCT run_id)   AS affected_runs,
  COUNT(*)                 AS total_failures,
  MIN(created_at)          AS first_seen,
  MAX(created_at)          AS last_seen
FROM test_runs
WHERE status = 'failed'
  AND created_at >= now() - interval '7 days'
GROUP BY stack_hash, failure_message
ORDER BY affected_runs DESC
LIMIT 20;

Wire both queries into Grafana using the PostgreSQL or ClickHouse data source. A time-series panel for first-attempt pass rate with a threshold annotation at your SLO (say, 92%) gives engineering leaders the quality KPI view they need without requiring them to read raw CI logs. A second panel — a table panel showing the top failure signatures sorted by affected_runs — is your triage queue. Once this dashboard was wired to Loki log links for each stack_hash, triage time at one platform team dropped from 22 minutes per failure to under 4. The table panel Grafana JSON for the signature query uses "transformations": [{"id": "sortBy", "options": {"fields": [{"desc": true, "displayName": "affected_runs"}]}}] — sort descending on impact, not recency.

In GitHub Actions, emit a JUnit XML artifact on every run and add a step that POSTs parsed results to your ingest endpoint. A minimal approach using pytest --junitxml=results.xml followed by a curl -X POST $INGEST_URL -F file=@results.xml step keeps the pipeline coupling to two lines. For Buildkite or CircleCI, the same pattern applies via their artifact upload APIs. The key discipline: always include the attempt number in the payload — most default JUnit parsers drop it, which collapses first-attempt and retry data into a single row and makes the core KPI uncomputable.

Where CI Test Dashboards Break Down in Practice

The most common mistake is treating pass rate as a single-dimensional metric. Teams track final pass rate, hit their target (say, 95%), and declare the suite healthy. But pass rate drift across pipeline stages hides broken stages behind an aggregate that looks fine. A deploy-stage test suite running at 60% first-attempt rate doesn't move the overall number much if the unit suite is at 99% — until the deploy stage starts blocking releases. Segment pass rate by suite, stage, and branch before you set any SLO threshold.

The second failure mode is signature staleness. Teams build the failure signature table, use it for two sprints, then stop maintaining the hash normalization logic as frameworks upgrade and stack trace formats change. Within a month, every failure hashes uniquely and the triage queue is useless again. Treat the ingest normalization code like production code: version it, test it, and alert when the ratio of unique hashes to total failures spikes above a threshold (e.g., >70% unique signatures in a 24-hour window is a signal the normalizer is broken, not that you have 70% novel failures).

Myths That Keep Test Run Dashboards Shallow

Myth 1: A high final pass rate means the suite is reliable. It means the suite eventually passes. Those are different things. A 98% final pass rate with a 79% first-attempt rate means roughly one in five tests requires a retry — that's not reliability, that's a hidden flakiness budget being consumed silently. Myth 2: Failure signatures are only useful for flaky tests. Signatures are equally powerful for deterministic failures. When the same AssertionError: expected 200, got 503 appears across 40 runs on the main branch over 48 hours, the signature table surfaces it as a P1 long before anyone files a ticket. The pattern-recognition value applies to any recurring failure, not just non-deterministic ones.

Myth 3: Dashboards replace triage. A dashboard is a triage accelerator, not a replacement for root-cause analysis. It tells you which signature to investigate first, not why the underlying defect exists. Teams that treat a red signature row as "done" because it's on the board — without linking it to a tracked fix — find the same signature reappearing three sprints later. Close the loop: every high-frequency signature needs an owner and a linked issue, or the dashboard becomes a leaderboard nobody acts on. For a sharper look at why surface-level metrics mislead, the analysis of why a 100% pass rate is often the least informative number in your test report is worth the read.

The foundation of useful test run observability is narrow: track first-attempt pass rate separately from final pass rate, hash failure signatures at ingest time, and build your triage queue around signature frequency rather than raw failure count. Start with the SQL queries above against whatever store your CI already writes to. Once the data model is clean, the Grafana panels follow in an afternoon. The hard part isn't the tooling — it's the discipline of keeping the ingest pipeline honest as your suite evolves.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.