Most teams treat test results like a checkbox: green is good, red is bad, ship or block. The interesting signal lives in everything that happens between those two states — runtime variance, retry counts, the same five tests appearing in every postmortem. A 100% pass rate doesn't tell you whether those tests ran at all, whether they passed on the third retry, or whether the suite silently shed 40 assertions last sprint.

The problem isn't that green builds are bad. It's that a single aggregate percentage compresses a multi-dimensional dataset into one bit of information, and that compression is almost always lossy in the ways that matter most. A suite that passes 100% after three flake-driven retries carries meaningfully different risk than one that passes cleanly on the first run.

By the end of this article you'll have concrete SQL, a Python analyzer, and a Grafana panel query you can drop into your existing pipeline to surface the signals that a pass-rate summary buries — and a clearer mental model for which numbers actually correlate with production incidents.

What a Pass Rate Actually Measures (and What It Doesn't)

A pass rate is a ratio of passed test executions to total executions in a given run. That's it. It says nothing about which tests ran, how many retries were consumed, how long each test took relative to its historical baseline, or whether the suite coverage changed since the last deploy. When your CI config sets retry: 3 in GitHub Actions or --reruns 3 in pytest, a flaky test that fails twice and passes once is recorded as a pass — and the aggregate rate stays green.

In a modern test architecture, pass rate belongs in the same category as CPU utilization: a useful sanity check, but a terrible primary signal. The metrics that correlate with escaped defects are first-attempt pass rate (stripping retries), P95 runtime drift (a leading indicator of environmental decay), and test-level failure frequency over a rolling window. As detailed in what test results actually tell you beyond pass/fail, the aggregate obscures the per-test story that drives triage decisions.

Extracting the Signals That Pass Rate Buries

Start at the storage layer. If you're ingesting JUnit XML into PostgreSQL (or ClickHouse for scale), the schema should capture attempt_number, duration_ms, and run_id alongside the outcome. With that in place, the first query to run is first-attempt pass rate by suite, partitioned by branch:

-- PostgreSQL: first-attempt pass rate per suite, last 14 days
SELECT
  suite_name,
  branch,
  COUNT(*) FILTER (WHERE attempt_number = 1 AND outcome = 'passed') AS first_pass,
  COUNT(*) FILTER (WHERE attempt_number = 1)                        AS first_attempts,
  ROUND(
    100.0 * COUNT(*) FILTER (WHERE attempt_number = 1 AND outcome = 'passed')
    / NULLIF(COUNT(*) FILTER (WHERE attempt_number = 1), 0), 2
  ) AS first_attempt_pass_rate
FROM test_executions
WHERE run_at >= NOW() - INTERVAL '14 days'
GROUP BY suite_name, branch
ORDER BY first_attempt_pass_rate ASC;

This single query routinely surfaces suites sitting at 99.8% overall pass rate but 81% first-attempt pass rate — meaning nearly one in five tests is leaning on retries to stay green. That gap is your flakiness tax, and it's invisible in the headline number. For deeper guidance on schema design for this kind of query, see test result storage options across PostgreSQL, ClickHouse, and BigQuery.

Next, layer in runtime drift. A test that historically runs in 400 ms but now consistently takes 2.1 s hasn't failed — but something changed. The following Python snippet flags tests whose P95 duration has increased more than 2× over a 7-day rolling baseline:

import pandas as pd

df = pd.read_sql("""
  SELECT test_id, run_at, duration_ms
  FROM test_executions
  WHERE run_at >= NOW() - INTERVAL '30 days'
    AND outcome = 'passed'
""", conn)

baseline = df[df.run_at < df.run_at.max() - pd.Timedelta('7d')]
recent   = df[df.run_at >= df.run_at.max() - pd.Timedelta('7d')]

p95_base   = baseline.groupby('test_id')['duration_ms'].quantile(0.95)
p95_recent = recent.groupby('test_id')['duration_ms'].quantile(0.95)

drift = (p95_recent / p95_base).rename('drift_ratio').reset_index()
print(drift[drift.drift_ratio > 2.0].sort_values('drift_ratio', ascending=False))

In one platform team's experience, wiring this output into a Grafana panel backed by a Prometheus push-gateway cut the time to identify environment-related slowdowns from 22 minutes per failure to under 4 — because engineers stopped reading CI logs and started reading the drift panel. The Grafana panel JSON for a time-series of P95 drift by test is straightforward: use a stat or timeseries panel with a PromQL query like test_duration_p95_seconds{suite="checkout"} / test_duration_p95_baseline_seconds{suite="checkout"}, threshold-colored at 1.5× and 2×.

Finally, add a GitHub Actions step that fails the build not on pass rate, but on first-attempt pass rate falling below your agreed threshold:

# .github/workflows/ci.yml (partial)
- name: Assert first-attempt pass rate
  run: |
    python scripts/check_first_attempt_rate.py \
      --threshold 0.95 \
      --junit-dir test-results/
  # Fails the step if first_attempt_pass_rate < 0.95

This gate enforces the metric you actually care about, not the one your retry config inflates.

Where Even Senior Engineers Misconfigure This

The most common mistake is setting retries at the runner level (--reruns 3 in pytest, retries: 2 in Playwright config) without recording the attempt number in the test result artifact. JUnit XML supports a <rerunFailure> element, but most CI integrations — including the default GitHub Actions test-results action — silently drop retry metadata unless you explicitly configure the reporter. The result: your storage layer never sees attempt numbers, so first-attempt pass rate is uncomputable and you're flying blind on flakiness.

The second mistake is scoping dashboards to the last build rather than a rolling window. A single build is a sample of one; it tells you almost nothing about trend. Teams that track flaky test detection in CI consistently find that a 14-day rolling window is the minimum for separating genuine flakiness from one-off environment noise. Org-level pressure to ship often pushes teams toward "is today green?" thinking — which is exactly the mental model a 100% pass rate headline reinforces.

Myths That Keep Pass Rate on the Dashboard

Myth 1: A green build means the suite is healthy. A build is green when no unretried test failure blocks the pipeline. It says nothing about suite coverage stability, assertion depth, or whether someone quietly @pytest.mark.skip-ed the three tests that caught last quarter's regression. Myth 2: Pass rate trends upward as quality improves. Pass rate trends upward when teams add retries, remove unstable tests, or reduce suite scope — none of which improve quality. The metric is gameable by accident, which is worse than being gameable on purpose. As outlined in the broader discussion of why pass/fail metrics mislead, the aggregate flattens exactly the variance that signals systemic problems.

Myth 3: Dashboards solve the problem. A Grafana dashboard showing pass rate over time is still showing pass rate over time — it's just prettier. The fix is metric selection, not visualization tooling. Allure is excellent for per-test history and retry visualization within a project; ReportPortal adds cross-project aggregation and ML-based defect triage, which earns its operational overhead when you're running 50+ microservice test suites. Neither tool makes a bad metric meaningful. Swap the primary KPI first, then choose the visualization layer.

Pass rate isn't useless — it's just a starting point that most teams mistake for a destination. Replace it as your primary gate with first-attempt pass rate, add P95 runtime drift as a leading indicator, and store attempt-level metadata from day one so you can query it later. If you're building out the broader reporting layer, the anatomy of a useful test report covers the full set of fields worth capturing. Start with the SQL query above against your existing JUnit data — the gap between overall and first-attempt pass rate is usually surprising enough to justify the rest of the work.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.