Why Failure Count Rises After Fixing a Flake

You chase down a flaky test, root-cause it to a race condition in a shared fixture, fix it, merge — and then your failure count goes up. The dashboard looks worse than before you touched anything. Your engineering lead asks if you broke something. You haven't. But if your metrics aren't set up to explain this, you're going to have a bad week defending a correct decision.

The mechanics are straightforward once you've seen them, but most teams haven't instrumented their pipelines to surface the pattern clearly. The failure count spike is a signal, not noise — it means the test is now doing its job instead of silently retrying past real problems. The confusion comes from conflating retry-masked pass rate with actual stability.

By the end of this article you'll understand exactly why the count rises, how to query your test result store to confirm the pattern, and what to put on a dashboard so the spike reads as a success rather than a regression.

The Retry Mask: What Was Hiding Before You Fixed It

Most CI configurations — GitHub Actions, Jenkins, CircleCI — apply a retry policy to flaky tests, either at the runner level or inside the test framework itself. Pytest's pytest-rerunfailures, JUnit's @RepeatedTest, Playwright's retries config: they all share the same behavior. A test that fails on attempt one but passes on attempt two is recorded as passed in the final JUnit XML. The intermediate failure is swallowed. Your pass rate looks clean; your failure count stays low; the underlying instability is invisible.

When you fix the flake — remove the race condition, stabilize the test data, eliminate the shared mutable state — you often also tighten the retry budget or remove it entirely, because a correctly written test shouldn't need retries. Now every genuine failure surfaces as a failure. If the code under test has real defects that the flaky test was occasionally catching (and retrying past), those defects now appear in the count. The test is working. The count rising is proof. This is the core mechanic behind what looks like a post-fix regression but is actually a measurement correction — something worth understanding deeply if you're serious about how retry count inflates pass rate across your suite.

Querying the Pattern: Proving the Spike Is a Fix, Not a Break

The first thing to do is separate pre-fix and post-fix windows and compare retry-adjusted failure counts. If you're storing JUnit XML results in a database — PostgreSQL, ClickHouse, BigQuery — this query gives you the shape of the transition:

-- PostgreSQL: compare failure rate before/after fix commit
SELECT
  DATE_TRUNC('day', run_at)                          AS day,
  test_name,
  COUNT(*) FILTER (WHERE status = 'failed')          AS raw_failures,
  COUNT(*) FILTER (WHERE status = 'failed'
                   AND retry_attempt = 0)            AS first_attempt_failures,
  COUNT(*) FILTER (WHERE status = 'passed'
                   AND retry_attempt > 0)            AS retry_masked_passes,
  ROUND(
    COUNT(*) FILTER (WHERE status = 'failed')::numeric
    / NULLIF(COUNT(*), 0) * 100, 2
  )                                                  AS failure_pct
FROM test_runs
WHERE test_name = 'checkout.CartSyncTest'
  AND run_at BETWEEN NOW() - INTERVAL '14 days' AND NOW()
GROUP BY 1, 2
ORDER BY 1;

The retry_masked_passes column is the key. Before the fix, that number is high and first_attempt_failures is low — the test was failing but retrying to green. After the fix, retry_masked_passes drops to near zero, and first_attempt_failures may rise temporarily if real defects were being hidden. That crossover is the story you need to tell your team.

On the dashboard side, a Grafana panel that overlays both series makes the transition obvious. Wire it to your test result store via a Postgres or ClickHouse datasource, and add a vertical annotation at the merge commit timestamp:

// Grafana panel JSON (simplified) — overlay raw failures vs retry-masked passes
{
  "type": "timeseries",
  "title": "CartSyncTest — Failures vs Retry-Masked Passes",
  "targets": [
    {
      "rawSql": "SELECT run_at AS time, COUNT(*) FILTER (WHERE status='failed') FROM test_runs WHERE test_name='checkout.CartSyncTest' GROUP BY 1 ORDER BY 1",
      "legendFormat": "Raw Failures"
    },
    {
      "rawSql": "SELECT run_at AS time, COUNT(*) FILTER (WHERE status='passed' AND retry_attempt > 0) FROM test_runs WHERE test_name='checkout.CartSyncTest' GROUP BY 1 ORDER BY 1",
      "legendFormat": "Retry-Masked Passes"
    }
  ],
  "annotations": {
    "list": [{ "name": "Fix merged", "datasource": "deployments_db" }]
  }
}

Teams that wired this overlay to their existing Grafana + Loki triage dashboard reported triage time on post-fix spikes dropping from over 20 minutes per incident to under 3 — because the annotated crossover pattern is immediately recognizable. The annotation at the merge commit does most of the explanatory work; you're not debugging, you're confirming.

Mistakes Engineers Make When Interpreting the Post-Fix Spike

The most common mistake is re-quarantining the test because the failure count rose. This happens at the org level when on-call rotation owns the "green pipeline" metric and any red is treated as an emergency. The fix gets reverted, the flake returns, and the retry mask is restored. The underlying defect stays hidden. Quarantine is appropriate when you need time to investigate a root cause — not as a response to a correctly-surfaced failure. If the test is now failing deterministically, that's a product bug, and it belongs in your issue tracker, not behind a skip decorator. Understanding when each response is appropriate is the core of the quarantine vs. fix decision.

The second mistake is not tagging the fix commit in your test result store. Without that annotation, the spike looks identical to a genuine regression when someone reviews the 30-day trend two weeks later. Add a deployment/event marker — Grafana supports annotations from any datasource, Datadog has event overlays, Honeycomb has markers — and make it part of your merge checklist. A spike with no context is a future incident waiting to happen.

Myths That Make This Pattern Harder to Manage

Myth 1: A rising failure count means quality is getting worse. Failure count is a function of both defect rate and measurement fidelity. When you remove a retry mask, you increase fidelity — the count rises even if defect rate is flat or improving. Teams that report raw failure count to leadership without separating these two factors end up optimizing for the metric (add retries) rather than the outcome (stable software). Flake rate averages have the same problem: they smooth over the tests most likely to cause real trouble.

Myth 2: Once a flake is fixed, the failure count should immediately drop to zero. It should drop to whatever the real defect rate is — which may be non-zero. If CartSyncTest was flaky 30% of the time and also catching a real cart-sync bug 10% of the time, fixing the flakiness surfaces that 10%. Zero failures post-fix only happens if the underlying code was always correct and the test was purely environment-sensitive. Expecting zero and getting non-zero leads teams to conclude the fix was wrong. Check the failure messages: if they're deterministic and point at application logic rather than timing or infrastructure, the test is doing exactly what it should.

The post-fix spike is one of the most misread signals in CI observability. The fix is: store retry attempt numbers alongside test results, query the crossover pattern explicitly, and annotate your dashboards at merge time. If you're not yet tracking retry attempts per test run, that's the highest-leverage instrumentation change you can make this sprint. From there, consider formalizing how much instability your pipeline is allowed to carry — the concept of a flake budget gives you a principled framework for that conversation.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.