Most teams treat flaky tests as a janitor problem: someone cleans them up when the noise gets unbearable, then the cycle repeats. What's missing isn't effort — it's a structured cadence that forces the signal into a format leadership and ICs can both act on. A quarterly reliability scorecard does that. It turns "we have flaky tests" into a trend line with owners, thresholds, and a paper trail.

The technical problem is aggregation. Flakiness data lives in JUnit XML artifacts, CI run logs, retry counts, and test-framework metadata — none of it in the same place, none of it normalized. Building a scorecard means pulling those sources into a single queryable store, defining the metrics that matter (not just pass/fail rate), and scheduling a review that isn't just a retrospective on feelings.

By the end of this article you'll have a concrete scorecard template, the SQL and Python to populate it, a Grafana dashboard structure that goes beyond flake rate, and the organizational anti-patterns that cause quarterly reviews to die after one cycle.

What a Reliability Scorecard Actually Measures

A reliability scorecard is a time-bounded snapshot of your test suite's health across a fixed window — typically a quarter — expressed as a set of tracked metrics with defined owners and trend direction. It is not a dashboard. A dashboard is always-on and reactive; a scorecard is periodic, deliberate, and tied to a decision: do we invest in reliability this quarter or not? The distinction matters because dashboards get ignored; scorecards get presented to engineering managers who control headcount.

In a modern test architecture the scorecard sits downstream of your CI observability layer. Raw data flows from GitHub Actions, Jenkins, or Buildkite into a store — ClickHouse, BigQuery, or PostgreSQL — where you can query across runs, branches, and time windows. The scorecard then pulls from that store on a schedule (a weekly cron is fine; a quarterly roll-up is what you review). If you're already auto-detecting flaky tests in CI, you already have the event stream; the scorecard is just the aggregation layer on top.

Building the Scorecard: Schema, Queries, and Trend Lines

Start with a normalized test-run table. Every CI run writes one row per test case with at minimum: run_id, test_name, suite, status (pass/fail/skip), retry_count, duration_ms, branch, and created_at. Parsing JUnit XML reports is the fastest path to populating this table without instrumenting every framework individually.

-- Quarterly flake rate per suite (PostgreSQL / BigQuery compatible)
SELECT
  suite,
  COUNT(DISTINCT test_name)                          AS total_tests,
  COUNT(DISTINCT CASE WHEN retry_count > 0
        AND status = 'pass' THEN test_name END)      AS flaky_tests,
  ROUND(
    COUNT(DISTINCT CASE WHEN retry_count > 0
          AND status = 'pass' THEN test_name END)
    * 100.0
    / NULLIF(COUNT(DISTINCT test_name), 0), 2)       AS flake_rate_pct,
  PERCENTILE_CONT(0.95) WITHIN GROUP
    (ORDER BY duration_ms)                           AS p95_duration_ms
FROM test_runs
WHERE created_at >= DATE_TRUNC('quarter', CURRENT_DATE)
GROUP BY suite
ORDER BY flake_rate_pct DESC;

Flake rate per suite is the headline metric, but the scorecard needs trend lines, not point-in-time numbers. Run this query for each of the last four quarters and plot the delta. A suite whose flake rate went from 8% → 6% → 5% → 4% is improving; a suite flat at 7% for three quarters has an owner problem, not a tooling problem. Teams that identify flaky tests with real data consistently find that the top 5% of tests by retry count account for 60–70% of total CI wasted time — that's where the scorecard should focus remediation effort.

# Python: generate quarterly scorecard CSV from BigQuery
from google.cloud import bigquery
import pandas as pd

client = bigquery.Client()
query = """
SELECT
  DATE_TRUNC(created_at, QUARTER)  AS quarter,
  suite,
  COUNTIF(retry_count > 0 AND status = 'pass')
    / COUNT(*)                     AS flake_rate,
  AVG(duration_ms) / 1000.0        AS avg_duration_s,
  SUM(retry_count)                 AS total_retries
FROM `myproject.ci_data.test_runs`
WHERE created_at >= '2024-01-01'
GROUP BY 1, 2
ORDER BY 1 DESC, flake_rate DESC
"""
df = client.query(query).to_dataframe()
df.to_csv("reliability_scorecard_Q.csv", index=False)

The Grafana panel that makes this reviewable in 90 seconds uses a Time Series visualization with one series per suite, a threshold line at your agreed SLO (e.g., flake rate ≤ 3%), and annotations marking deploys or framework upgrades. Wire it to a Loki datasource for log-level failure context per test. One team reduced triage time from 22 minutes per failure to under 4 once they linked the Grafana panel annotations directly to Loki log streams keyed by test_name. Beyond flake rate, the CI observability dashboard should surface: P95 test duration by suite, skip rate trend (skips that grow quietly are technical debt), retry cost in CI minutes, and mean time to green per branch. These metrics expose problems that a raw pass/fail view never shows.

Pitfalls That Kill Quarterly Reviews After One Cycle

The most common failure mode is metric sprawl: teams add every available signal to the scorecard because the data is there. Twelve metrics with no thresholds produce a document everyone nods at and nobody acts on. Pick five metrics maximum, assign a threshold to each, and mark every metric red/yellow/green before the review meeting. If a metric has no threshold it has no owner. A related problem is quarterly cadence without weekly checkpoints — by the time Q3 ends, the Q2 regression is three months old and the engineer who introduced it has moved teams.

The second pitfall is treating the scorecard as a reporting artifact rather than a decision input. Reliability work competes with feature work for sprint capacity. If the scorecard doesn't produce a concrete ask — "we need two engineer-weeks to fix the top-10 flaky tests in the checkout suite" — it will lose that competition every quarter. Tie the scorecard to your SLO-driven testing strategy so reliability targets are contractual, not aspirational. That framing changes the conversation from "nice to have" to "we're breaching an SLO."

Myths That Undermine Flaky-Test Scorecards

Myth 1: flake rate is the only reliability metric worth tracking. Flake rate measures non-determinism, but a suite can have 0% flakiness and still be a reliability liability — if P95 duration is 40 minutes, if skip rate is climbing, or if mean-time-to-green on main is 2 hours. The scorecard template should treat flake rate as one signal in a vector, not the scalar that defines suite health. Myth 2: once a test is fixed, remove it from the scorecard. Fixed tests should stay on the watch list for at least two quarters. Recurrence is common — especially when the root cause was environmental rather than deterministic — and a test that reappears after removal looks like a new problem instead of a known pattern.

Myth 3: a dashboard replaces a scorecard. Dashboards are excellent for on-call triage; they are poor forcing functions for organizational investment. The quarterly scorecard creates a moment where a number is officially bad enough to require a response. Without that moment, flakiness stays in the background noise. The real cost of flaky tests — in CI minutes, in developer trust erosion, in delayed releases — only becomes visible when you aggregate it over a quarter and put a dollar figure next to the retry count. That's what moves budget.

A quarterly reliability scorecard works when it has five or fewer metrics, hard thresholds, named owners, and a direct line to sprint planning. Start with the SQL above against whatever store your CI already writes to, add a Grafana trend panel with a 3% flake-rate threshold line, and schedule the first review before the data is perfect — imperfect data reviewed consistently beats perfect data reviewed never. For a broader organizational framing, the quality scorecard guide covers how to extend this pattern across multiple teams and product areas.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.