A 3% average flake rate sounds manageable. It isn't — not when that number is the mean across 2,000 tests and three of them are failing on 40% of runs against main the week before a release. Averages flatten the distribution exactly where the risk concentrates. The teams that get burned aren't the ones ignoring flakiness; they're the ones watching the wrong statistic.

The core problem is a measurement mismatch. Average flake rate is a population-level metric applied to a release-blocking decision that is fundamentally about individual test behavior over a specific time window. A test that flaked twice in six months and a test that flaked six times in the last two weeks can share the same rate — and only one of them should be on your quarantine list today.

By the end of this article you'll have a SQL-based scoring model, a Grafana panel config, and a mental model for replacing average flake rate with a recency-weighted, percentile-aware signal that actually predicts release risk.

Why Flake Rate as a Single Number Is a Lossy Metric

Flake rate is defined as flaky_runs / total_runs over some lookback window. The problem isn't the formula — it's that a single scalar collapses two dimensions that matter independently: magnitude (how often does it fail non-deterministically?) and recency (is it getting worse right now?). A test with a 5% rate over 90 days that was clean for 75 of those days and failed 10 times in the last 15 is a different animal than one that failed once a week, steadily, for the full period. Same number, opposite triage priority.

This is why pass/fail metrics are misleading at the aggregate level — the signal you need is in the shape of the distribution, not the summary statistic. In a modern test architecture where results land in ClickHouse, BigQuery, or PostgreSQL, you already have the raw run-level data to compute something far more useful. The gap is almost never data availability; it's query design.

Building a Recency-Weighted Flake Risk Score

Start with your raw test results table. The schema below assumes PostgreSQL but maps directly to BigQuery or ClickHouse with minor syntax changes (INTERVAL vs DATE_SUB, quantile() vs PERCENTILE_CONT).

-- Recency-weighted flake score per test, last 30 days
-- Weight: runs in the last 7 days count 3x, last 8-14 days 2x, older 1x
WITH weighted_runs AS (
  SELECT
    test_id,
    test_name,
    suite,
    run_at,
    result,  -- 'pass' | 'fail' | 'flaky'
    CASE
      WHEN run_at >= NOW() - INTERVAL '7 days'  THEN 3
      WHEN run_at >= NOW() - INTERVAL '14 days' THEN 2
      ELSE 1
    END AS weight
  FROM test_runs
  WHERE run_at >= NOW() - INTERVAL '30 days'
    AND result IN ('pass', 'fail', 'flaky')
),
scored AS (
  SELECT
    test_id,
    test_name,
    suite,
    SUM(weight) FILTER (WHERE result = 'flaky') AS weighted_flaky,
    SUM(weight)                                  AS weighted_total,
    COUNT(*) FILTER (WHERE result = 'flaky')     AS raw_flaky_count,
    COUNT(*)                                     AS raw_total,
    MAX(run_at) FILTER (WHERE result = 'flaky')  AS last_flaked_at
  FROM weighted_runs
  GROUP BY test_id, test_name, suite
)
SELECT
  test_id,
  test_name,
  suite,
  ROUND(weighted_flaky::numeric / NULLIF(weighted_total, 0) * 100, 2) AS flake_risk_score,
  raw_flaky_count,
  raw_total,
  last_flaked_at
FROM scored
WHERE raw_flaky_count >= 2          -- filter noise: single-occurrence events
ORDER BY flake_risk_score DESC
LIMIT 50;

The weight tiers mean a test that flaked three times yesterday outscores one that flaked nine times last month — which is exactly the behavior you want when deciding what to quarantine before a Friday deploy. The raw_flaky_count >= 2 guard prevents one-off environment hiccups from dominating the list.

Wire this into Grafana using the PostgreSQL data source and a Time Series panel. The panel JSON below renders the top-10 flake-risk tests as a bar chart refreshed every 5 minutes — suitable for a CI health dashboard displayed on a team TV or embedded in a Slack workflow.

{
  "type": "barchart",
  "title": "Top 10 Flake Risk Score (Recency-Weighted, 30d)",
  "datasource": { "type": "postgres", "uid": "your-pg-uid" },
  "targets": [{
    "rawSql": "SELECT test_name AS metric, flake_risk_score AS value FROM flake_risk_scores ORDER BY flake_risk_score DESC LIMIT 10;",
    "format": "table"
  }],
  "fieldConfig": {
    "defaults": {
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "green", "value": 0 },
          { "color": "yellow", "value": 10 },
          { "color": "red",   "value": 25 }
        ]
      }
    }
  }
}

Pair the score with a P95 runtime alert. Tests that are both high-flake-risk and runtime-volatile (P95 duration > 3× median) are your highest-priority quarantine candidates — they're slow enough that retries compound CI queue time significantly. One team running Playwright E2E on GitHub Actions cut their blocked-pipeline incidents from roughly 11 per sprint to 2 after quarantining the 6 tests that surfaced at the top of this query. Triage time dropped from around 18 minutes per incident to under 5 once the Grafana panel was wired to Loki log links for each failing test ID. For a fuller picture of how to identify flaky tests with real data, the run-level schema design matters as much as the query logic.

Pitfalls That Undermine Flake Risk Scoring in Practice

Using retry-masked results as your input data. Jenkins, GitHub Actions, and Buildkite all support automatic retries, and many teams store only the final result — meaning a test that failed twice and passed on the third attempt is recorded as pass. Your flake score is then computed on sanitized data and will systematically undercount the worst offenders. Fix this at the ingestion layer: store every attempt with an attempt_number column and a final_result flag. JUnit XML doesn't support this natively, but Allure's retries attachment and ReportPortal's retry tracking do. Use Allure when your team already owns the Allure server and needs rich history UI; use ReportPortal when you need multi-project aggregation and REST-driven integrations with CI orchestration like Argo Workflows.

Choosing a lookback window without anchoring it to your release cadence. A 90-day window for a team shipping weekly is almost useless for release-gate decisions — it's dominated by tests that have since been fixed or deleted. A 7-day window for a team on a quarterly release cycle misses slow-burning instability. The window should be 2–3× your release interval, with the recency weighting doing the work of surfacing recent acceleration. Teams that skip this calibration end up treating their flake budget as a fixed number rather than a dynamic signal tied to shipping rhythm.

What Teams Consistently Misread About Flake Distributions

Myth: a low average flake rate means the suite is stable enough to release. A suite with 1,800 tests at 0% flake and 20 tests at 30% flake has an average rate under 0.4%. That 0.4% will not appear on any red dashboard. Those 20 tests will block your pipeline. The average obscures a bimodal distribution where nearly all instability is concentrated in a small tail — which is the normal shape of flakiness in real suites, not an edge case. This is the same reason a 100% pass rate is often the least informative number in your report: aggregate green hides localized rot.

Myth: flakiness is uniformly distributed across test types. In practice, E2E tests (Playwright, Selenium) and integration tests with real network I/O account for the majority of flakiness in most suites, while unit tests contribute a small fraction. Scoring all tests in the same pool without segmenting by type produces a list dominated by E2E tests that may already be under a known quarantine policy — while a quietly degrading integration test layer goes unnoticed. Segment your flake risk scores by suite or test_type and set separate thresholds. A 15% flake risk score on an E2E test and a 15% score on a service-contract test represent very different severities and fix timelines.

Replace your average flake rate widget with the recency-weighted score query above — even a rough version in a ClickHouse or BigQuery view will immediately surface tests your current dashboard is hiding. If you want to close the loop from detection to remediation, the pattern described in connecting production signals back to test improvement gives a concrete workflow for turning that scored list into actionable engineering work rather than a dashboard nobody acts on.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.