Quarantine Rate and the Cost of Deferred Fixes

Most teams treat quarantine as a triage tool, not a debt register. A test goes flaky, gets tagged @pytest.mark.skip or moved to a non-blocking job, and the team moves on. The problem is that "moves on" never includes a timestamp, an owner, or a cost model. Weeks later, ten percent of your suite is quarantined and your pipeline pass rate looks pristine — because you've excluded every uncomfortable signal from the denominator.

Quarantine rate — the percentage of tests in a quarantined state at any point in time — is one of the few metrics that makes deferred fixes visible at the organizational level. It doesn't fluctuate with a bad deploy or a flapping environment. It accumulates. And accumulation is exactly what makes it useful as a leading indicator of suite rot.

By the end of this article you'll have a SQL-based quarantine rate tracker, a Grafana panel to surface aging quarantines, and a mental model for why the real cost of a quarantined test isn't the flake itself — it's the compounding interest on the fix you keep skipping.

Quarantine Rate as a Debt Metric, Not a Safety Valve

A quarantined test is a test your team has explicitly decided not to trust — but also not to fix right now. That decision has a legitimate short-term use: isolating a genuinely environment-dependent test while an infra issue is resolved, or parking a newly-written test that needs stabilization. The problem is the phrase "right now." Without a defined SLA and a tracking mechanism, "right now" becomes "never," and the quarantine list becomes a graveyard with green CI badges on top of it.

Quarantine rate sits at the intersection of flaky-test management and technical debt accounting. It belongs in the same dashboard as the real cost of flaky tests — not because it measures the same thing, but because it measures what happens when teams stop paying that cost and start deferring it. A quarantine rate above 3–5% is a signal worth escalating; above 10%, you're likely shipping with significant blind spots in your regression coverage.

Tracking Quarantine Rate Over Time with SQL and Grafana

The first step is making quarantine state a first-class field in your test results store. If you're writing JUnit XML into PostgreSQL or ClickHouse, add a quarantined_at timestamp and a quarantine_reason enum. If you're using ReportPortal, the "defect type" tagging system already supports this — map your "Not a Defect" bucket to a quarantine label and query against it. Allure doesn't have native quarantine tracking, so you'll need to carry it in a custom label or a side table.

With PostgreSQL, a rolling quarantine rate query looks like this:

-- Quarantine rate per week, last 12 weeks
SELECT
  date_trunc('week', run_date)            AS week,
  COUNT(*) FILTER (WHERE quarantined)     AS quarantined_count,
  COUNT(*)                                AS total_tests,
  ROUND(
    100.0 * COUNT(*) FILTER (WHERE quarantined) / COUNT(*), 2
  )                                       AS quarantine_rate_pct
FROM test_results
WHERE run_date >= NOW() - INTERVAL '12 weeks'
GROUP BY 1
ORDER BY 1;

Pair this with an aging query — how long has each quarantined test been parked — because a test quarantined for 3 days is noise, but one quarantined for 47 days is a liability:

SELECT
  test_name,
  suite,
  owner_team,
  quarantined_at,
  DATE_PART('day', NOW() - quarantined_at) AS days_quarantined,
  quarantine_reason
FROM test_results
WHERE quarantined = TRUE
ORDER BY days_quarantined DESC
LIMIT 50;

Wire both queries into a Grafana dashboard using the PostgreSQL data source. A time-series panel on quarantine rate over 12 weeks will show you whether the list is growing, stable, or shrinking — the shape of that curve is the metric. A table panel for aging quarantines, sorted by days_quarantined descending, gives team leads a weekly action list without any manual triage. One team running this setup reduced their median quarantine age from 34 days to 9 days within two sprints, not because they fixed tests faster, but because the visibility created accountability. Triage time per aging quarantine dropped from roughly 18 minutes of Slack archaeology to under 3 minutes once owner and reason were queryable fields.

For GitHub Actions pipelines, you can emit quarantine metadata as a structured annotation step and push it to your results store on every run:

# .github/workflows/test.yml (relevant fragment)
- name: Push quarantine metadata
  if: always()
  run: |
    python scripts/push_quarantine_report.py \
      --results junit-results.xml \
      --db-url ${{ secrets.RESULTS_DB_URL }} \
      --branch ${{ github.ref_name }} \
      --run-id ${{ github.run_id }}

The Python script parses JUnit XML, checks each test name against a quarantine_registry.json (a flat file checked into the repo), and upserts the quarantined_at and owner_team fields. Keeping the registry in version control means quarantine additions and removals appear in PRs — which is the cheapest possible review gate for preventing silent accumulation.

Where Quarantine Tracking Breaks Down in Practice

The most common failure mode is treating quarantine as a skip. Teams use @pytest.mark.skip, xit() in Jest, or a non-blocking job segment — and never record when or why. The test disappears from failure counts, pass rate looks fine, and there's no artifact linking the exclusion to a ticket or owner. This is structurally identical to the problem where skipped tests distort every rate you report — the denominator shrinks silently and your metrics become fiction. Fix: require that any quarantine action writes to the registry with a timestamp, a Jira/Linear ticket, and a review-by date.

The second failure mode is setting no SLA. Without an explicit policy — say, "quarantines older than 21 days trigger a team-level review" — the list grows indefinitely. This isn't a tooling problem; it's an org-level one. Engineering managers often don't see quarantine rate in their weekly metrics because it's buried in a test dashboard that only SDETs check. Surfacing it in the same Datadog or Grafana view that shows deployment frequency and MTTR changes the conversation. If it's on the same screen as the metrics leadership already reviews, it gets reviewed.

Myths That Let Quarantine Debt Compound Unnoticed

Myth 1: A stable quarantine count means stable quality. A flat quarantine count while your suite grows means your quarantine rate is actually falling — which looks like improvement but might just be dilution. This is the same denominator trap described in why failure rate drops as your suite grows: adding passing tests to a suite with a fixed number of quarantined tests makes the ratio look better without resolving anything. Track absolute count and rate together. Myth 2: Quarantine is a valid long-term strategy for environment-dependent tests. It isn't. A test that's quarantined because it depends on a staging environment that's unreliable is a test that's telling you something about your infra. Parking it indefinitely defers that signal. The decision framework for when to quarantine versus when to fix should include a hard expiry: if the underlying condition hasn't been resolved in N days, the test either gets fixed or gets deleted.

Myth 3: High quarantine rate is a test-quality problem, not an engineering-culture problem. The tests are symptomatic. The real issue is that fixing a flaky test rarely appears in a sprint as a first-class ticket — it gets done when someone has slack time, which means it often doesn't get done. Quarantine rate makes that prioritization gap visible in a way that a flake count alone doesn't, because it measures not just the existence of the problem but the team's response velocity to it.

Start by querying your existing test results store for any test excluded from pass/fail accounting — skipped, non-blocking, or explicitly tagged. Compute how long each has been in that state. If you can't answer that question in under five minutes, you don't have a quarantine strategy; you have a quarantine accumulation. Build the aging table first, wire it to a Grafana panel your team lead already opens, and set a 21-day review SLA. The metric only drives action if it's visible to the people who can authorize the fix.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.