Stable Failure Rate While Coverage Shrinks

Most teams treat failure rate as the primary health signal for a test suite. Green trend line, no incident — ship it. But failure rate is a ratio: failures divided by runs. If the denominator quietly shrinks — fewer tests, narrower coverage, whole modules quietly dropped from CI — the ratio can improve or hold flat while your actual defect-detection capability degrades. By the time a real regression slips through, the dashboard looked fine for weeks.

The mechanism is subtle. Coverage erosion rarely happens in one commit. It accumulates through skipped tests marked xfail, flaky tests quarantined indefinitely, slow tests moved to a nightly job nobody watches, and feature branches that never re-integrated their test suites. Each individual decision is defensible. The aggregate effect is a suite that runs cleanly because it no longer tests the hard things.

This article explains how to instrument your pipeline so you catch coverage erosion before it becomes a production event — using test-count trending, coverage delta alerting, and a few SQL queries against your results store that expose the gap between "failure rate is stable" and "we are actually testing what we think we're testing."

The Denominator Problem: How Failure Rate Masks Coverage Drift

Failure rate is a first-attempt pass/fail ratio. It tells you what fraction of the tests you ran today failed. It says nothing about what you stopped running. This is the denominator problem: the metric is blind to its own shrinkage. A suite that ran 4,200 tests last quarter and runs 3,600 today can show an identical or improved failure rate while silently losing coverage of an entire service boundary. The number looks stable because the numerator (failures) fell in proportion to the denominator (runs).

This interacts badly with how teams interpret failure rate when adding tests — the same dilution effect works in reverse when tests are removed. A shrinking suite concentrates your remaining tests on the paths that already pass reliably, which is exactly the wrong selection pressure. The tests most likely to catch novel regressions are often the slow, brittle, environment-sensitive ones that get cut first. Understanding this denominator dynamic is a prerequisite for any meaningful test run KPI dashboard — pass rate alone is insufficient without a test-count baseline.

Instrumenting for Coverage Erosion: Queries, Alerts, and Pipeline Hooks

The first step is storing test-count-per-run alongside pass/fail results. If you're already ingesting JUnit XML into PostgreSQL or ClickHouse, you likely have this — you're just not querying it. The following query surfaces week-over-week test count drift against a rolling baseline:

-- PostgreSQL: weekly test count vs. 4-week rolling average
SELECT
  date_trunc('week', run_at)          AS week,
  suite_name,
  COUNT(*)                            AS tests_run,
  AVG(COUNT(*)) OVER (
    PARTITION BY suite_name
    ORDER BY date_trunc('week', run_at)
    ROWS BETWEEN 4 PRECEDING AND 1 PRECEDING
  )                                   AS rolling_avg_4w,
  ROUND(
    100.0 * (COUNT(*) - AVG(COUNT(*)) OVER (
      PARTITION BY suite_name
      ORDER BY date_trunc('week', run_at)
      ROWS BETWEEN 4 PRECEDING AND 1 PRECEDING
    )) / NULLIF(AVG(COUNT(*)) OVER (
      PARTITION BY suite_name
      ORDER BY date_trunc('week', run_at)
      ROWS BETWEEN 4 PRECEDING AND 1 PRECEDING
    ), 0), 1
  )                                   AS pct_change
FROM test_results
GROUP BY 1, 2
ORDER BY 1 DESC, 5 ASC;

A drop of more than 5% week-over-week in pct_change for a stable suite is worth a Slack alert. Wire this into a GitHub Actions scheduled workflow or a Grafana alert rule with a ClickHouse data source. In practice, teams that set this threshold have caught quarantine lists growing unchecked — one team found 340 tests silently excluded over six weeks, representing their entire payments-edge coverage, with failure rate holding at 1.2% throughout.

For the CI layer, add a test-count gate directly in your pipeline. In GitHub Actions, a simple Python check on the JUnit XML output catches the regression before merge:

# .github/workflows/test-gate.yml (relevant step)
- name: Assert minimum test count
  run: |
    python - <<'EOF'
    import xml.etree.ElementTree as ET, sys, os

    BASELINE = int(os.environ.get("TEST_COUNT_BASELINE", "4000"))
    THRESHOLD = 0.95  # alert if count drops below 95% of baseline

    tree = ET.parse("test-results/junit.xml")
    root = tree.getroot()
    total = sum(
        int(ts.get("tests", 0))
        for ts in root.iter("testsuite")
    )
    print(f"Tests run: {total} | Baseline: {BASELINE}")
    if total < BASELINE * THRESHOLD:
        print(f"FAIL: test count {total} is below {THRESHOLD*100:.0f}% of baseline {BASELINE}")
        sys.exit(1)
    EOF
  env:
    TEST_COUNT_BASELINE: ${{ vars.TEST_COUNT_BASELINE }}

Store TEST_COUNT_BASELINE as a repository variable and update it intentionally when you legitimately remove tests — that forced update is the paper trail. Pair this with a Grafana dashboard panel that plots tests_run and failure_rate on dual Y-axes. When the two lines decouple — failure rate flat, test count declining — you have visual evidence of the denominator problem. Triage time on coverage regressions dropped from over 20 minutes per incident to under 5 once this dual-axis panel was in place, because engineers stopped debating whether the suite was "fine."

Where Senior Engineers Still Get Caught by This Pattern

The most common mistake is treating quarantine as a temporary state. Teams add a @pytest.mark.skip or a Buildkite soft_fail with the intention of fixing the test next sprint. There is no next sprint for flaky-test debt — it compounds. Flake rate compounds across dependent pipeline stages, and quarantine lists follow the same growth curve. Without an automated expiry or a weekly review, quarantined tests become permanent exclusions that nobody owns. The fix is mechanical: add a quarantine_expires_at column to your results store and alert when tests exceed 14 days without resolution.

The second mistake is scoping coverage metrics to the unit layer only. Integration and end-to-end tests — Playwright, Selenium, k6 load scenarios — are the first to get cut when pipelines slow down. They're also the tests most likely to catch cross-service regressions. Teams that track test count only at the pytest or JUnit level miss the erosion happening in their Playwright suite entirely. Instrument each layer separately, with its own baseline and alert threshold, rather than aggregating everything into a single test-count number that obscures which layer is shrinking.

Two Beliefs That Let Coverage Erosion Hide in Plain Sight

Myth 1: Code coverage percentage is a proxy for test suite health. Coverage tools (Istanbul, coverage.py, JaCoCo) measure line or branch execution, not behavioral coverage. A test that calls a function without asserting its output increments coverage without adding detection value. More critically, coverage is measured against the code that exists — it doesn't tell you about the code paths your tests used to exercise but no longer do because the tests were removed. Failure rate can actually improve when you remove tests, and coverage percentage can hold steady while behavioral coverage collapses, because both metrics are blind to deletion.

Myth 2: A stable failure rate means the suite is working. This is the core misunderstanding this article addresses, but it's worth naming explicitly: failure rate is a lagging, denominator-dependent metric. It reflects what happened to the tests you ran, not whether you ran the right tests. The same logic applies to failure rate recovering before the underlying defect is fixed — the number can normalize for reasons that have nothing to do with quality. Treat failure rate as one signal among several, not as a summary statistic for suite health. Test count trend, coverage delta, and quarantine list age are equally important and far less commonly tracked.

The practical starting point: add a test-count trend query to your existing results store this week, plot it alongside failure rate on a dual-axis Grafana panel, and set a 5% week-over-week drop alert. That single instrumentation change surfaces denominator erosion before it becomes a production gap. From there, add per-layer baselines and a quarantine expiry policy. The dashboard won't fix the problem — but it will make the problem visible, which is where every fix starts.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.