Why Quarantine Drops Your Flake Rate

Most teams treat flake rate as a background hum — annoying, but not worth stopping for. Then a release gets blocked by a test that has failed intermittently 47 times in 90 days and nobody can tell you whether the product is broken or the test is. The signal you need was always there; you just never separated it from the noise long enough to act on it.

Flake rate — the percentage of test runs that produce a non-deterministic result across identical code — is a compound metric. A handful of high-frequency offenders can drag the entire suite's rate into territory that erodes engineer trust, inflates retry budgets, and masks genuine regressions. The counterintuitive truth is that removing those tests from the main execution path, even temporarily, causes the suite-wide flake rate to drop sharply — sometimes by 60–80% — without fixing a single underlying bug.

This article explains the mechanics behind that drop, shows you how to wire quarantine logic into a real CI pipeline, and covers the measurement traps that cause teams to misread the improvement signal.

Flake Rate Meaning and Why a Few Tests Dominate It

Flake rate for a test is flaky_runs / total_runs over a rolling window — typically 7 or 30 days. Suite-wide flake rate aggregates that across every test, weighted by execution count. The problem is that distribution is almost never uniform. In a mature CI system, 5–10% of tests routinely account for 70–90% of all flaky runs. This is not an exaggeration; it matches data from large open-source projects and internal platform teams alike. When those outliers are included in the denominator and numerator of every pipeline run, they dominate the aggregate. As noted in the analysis of how flake rate averages hide your riskiest tests, a single test flaking at 40% can suppress the apparent severity of a dozen tests flaking at 5%.

Quarantine is a formal state in your test registry: a test is tagged, excluded from the blocking gate, and routed to a separate non-blocking job that still collects results. It is not deletion and it is not a skip — skips stop producing data entirely, which distorts every rate you report downstream. Quarantine preserves observability while removing the test's ability to block pipelines or inflate suite-wide flake rate. The moment those high-frequency offenders stop contributing to the main execution path, the aggregate rate collapses toward the true baseline of your healthy tests.

Wiring Quarantine Into CI and Measuring the Drop

Start with a query. Pull the last 30 days of JUnit XML results into PostgreSQL or ClickHouse and rank tests by flaky run count. This is the gate for quarantine eligibility — not vibes, not a Slack complaint.

-- PostgreSQL: top flaky tests by flaky run count, last 30 days
SELECT
  test_name,
  COUNT(*) FILTER (WHERE result = 'flaky') AS flaky_runs,
  COUNT(*)                                  AS total_runs,
  ROUND(
    COUNT(*) FILTER (WHERE result = 'flaky')::numeric / COUNT(*) * 100, 2
  ) AS flake_pct
FROM test_run_results
WHERE run_at >= NOW() - INTERVAL '30 days'
GROUP BY test_name
HAVING COUNT(*) >= 20          -- ignore low-volume noise
ORDER BY flaky_runs DESC
LIMIT 20;

Any test above a threshold you own — a reasonable starting point is flake_pct >= 15 and flaky_runs >= 5 — gets tagged quarantine in your test registry. In Pytest, that's a marker; in JUnit-based stacks it's a category or group annotation. The CI job splits on that tag.

# GitHub Actions: split blocking vs. quarantine jobs
jobs:
  test-blocking:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pytest -m "not quarantine" --junitxml=results/blocking.xml

  test-quarantine:
    runs-on: ubuntu-latest
    continue-on-error: true          # never blocks merge
    steps:
      - uses: actions/checkout@v4
      - run: pytest -m "quarantine" --junitxml=results/quarantine.xml
      - uses: actions/upload-artifact@v4
        with:
          name: quarantine-results
          path: results/quarantine.xml

continue-on-error: true is the critical line. The quarantine job still runs, still uploads artifacts, and still feeds your results store — but a failure there never sets the pipeline status to red. Your Grafana CI reliability dashboard will now show two separate flake-rate series: suite_flake_rate_blocking and suite_flake_rate_quarantine. Within one sprint of quarantining the top 10 offenders, most teams see blocking flake rate drop from double digits to under 3%. One platform team reported triage time falling from 22 minutes per failure to under 4 once the dashboard separated quarantine noise from real signal.

For Python-heavy suites, a lightweight analyzer can automate the tagging decision. The script below reads a results directory, computes per-test flake rate across recent runs, and emits a pytest.ini markers block.

import xml.etree.ElementTree as ET
from pathlib import Path
from collections import defaultdict

counts = defaultdict(lambda: {"total": 0, "flaky": 0})

for xml_file in Path("results/history").glob("*.xml"):
    tree = ET.parse(xml_file)
    for tc in tree.iter("testcase"):
        name = f"{tc.attrib['classname']}.{tc.attrib['name']}"
        counts[name]["total"] += 1
        # flaky = has both a failure child and a rerun-passed sibling
        if tc.find("failure") is not None and tc.attrib.get("rerun_status") == "passed":
            counts[name]["flaky"] += 1

candidates = [
    name for name, c in counts.items()
    if c["total"] >= 20 and (c["flaky"] / c["total"]) >= 0.15
]

print("[pytest]")
print("markers =")
for name in candidates:
    print(f"  quarantine: {name}")

This is a practical tool for finding the causes of unstable Python tests — not root cause directly, but the ranked list that tells you where to spend the next engineering hour. Pair it with Loki log correlation on the quarantine job's output to surface environment-specific patterns (port conflicts, shared fixtures, timing dependencies) without manual log archaeology.

Where Quarantine Strategies Break Down in Practice

The most common failure mode is treating quarantine as a one-way door. Teams quarantine aggressively in Q1, declare victory on the dashboard, and never schedule the fix work. Six months later, the quarantine bucket holds 200 tests, coverage of critical paths has silently eroded, and the real cost of deferred fixes shows up as escaped defects rather than CI noise. Quarantine rate — the ratio of quarantined tests to total tests — needs an SLO with teeth: anything above 8% should trigger a mandatory triage sprint, not a Jira backlog item that ages quietly.

The second failure mode is coarse tagging granularity. Engineers mark an entire test class as quarantine because one method is flaky, which removes deterministic coverage from the gate. Tag at the method level, always. The third is ignoring the quarantine job's results entirely because it's non-blocking. Wire the quarantine XML into the same results store as blocking runs; the flake rate trend in that bucket is your leading indicator of whether the underlying environment problems are getting worse, not better.

Myths About What the Flake Rate Drop Actually Means

A sharp drop in suite-wide flake rate after quarantine is real signal, but it is not the same as fixing flakiness. The bugs are still there. Flake rate resets after a suite refactor for the same reason: you've changed the measurement boundary, not the underlying instability. Teams that report quarantine-driven improvements to leadership as "we fixed our flaky tests" are setting up a trust problem when the quarantine bucket eventually overflows back into the suite. Frame the metric correctly: blocking flake rate dropped; quarantine backlog is the liability on the balance sheet.

The second myth is that a CI reliability dashboard showing green flake rate means the suite is healthy. A CI reliability dashboard showing blocking_flake_rate = 1.2% alongside quarantine_count = 180 is not a healthy suite — it's a healthy illusion. Flake rate is one dimension. Also track quarantine age (median days since quarantine tag was applied), quarantine coverage loss (which critical paths have no blocking test), and retry cost in CI minutes. Those three numbers together tell you whether quarantine is a triage tool or a graveyard.

Quarantine is a measurement intervention before it is a fix strategy. Done right — with per-method tagging, a non-blocking CI job that still collects data, a SQL-driven eligibility gate, and an SLO on quarantine rate — it gives your suite back its signal integrity within days. The next step is building the fix loop: prioritize quarantine candidates by business-path criticality, not just flake frequency, and set a team norm that no test stays quarantined longer than two sprints without a resolution decision. The data to drive that conversation is already in your results store.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.