Why Failure Rate Drops as Your Test Suite Grows

Most teams treat a declining failure rate as a sign of progress. Fewer red builds, happier stakeholders, smoother releases. But if that rate is dropping while your suite is growing, you may be watching a statistical artifact rather than a quality signal. The denominator is doing the work, not your engineers.

This is the dilution effect: as you add hundreds of stable, low-risk tests — happy-path smoke checks, redundant API contracts, trivial unit assertions — the handful of genuinely volatile tests that actually catch regressions get buried in the average. Your failure rate looks better because it's now spread across more tests, not because your code is more reliable.

By the end of this article you'll understand exactly why this happens mathematically, how to restructure your metrics to see through it, and what queries and dashboard configurations will surface the tests that actually matter — before a diluted aggregate fools your team into shipping a defect.

The Dilution Effect: What the Math Is Actually Doing

Failure rate is almost always computed as failed_runs / total_runs over some time window. When a suite has 200 tests and 10 fail consistently, you have a 5% failure rate that's hard to ignore. Add 800 stable tests — integration coverage for well-baked infrastructure, golden-path E2E flows — and those same 10 volatile tests now produce a 1% failure rate. Nothing changed in the code. The signal was diluted by volume.

This isn't a contrived edge case. It's the natural trajectory of a maturing suite. Teams add tests faster than they retire them, stable coverage accumulates, and per-test failure history gets averaged away into a single aggregate that means progressively less. The metric that was a useful proxy at 200 tests becomes actively misleading at 2,000. Treating aggregate failure rate as a health signal without accounting for suite composition is how a 100% pass rate becomes your least useful number — and how a 1% failure rate earns false confidence.

Measuring Failure Concentration, Not Just Failure Rate

The fix is to stop aggregating across the whole suite and start tracking per-test failure frequency over a rolling window. Store your JUnit XML results in a queryable store — ClickHouse or BigQuery both work well at CI scale — and run concentration queries rather than simple averages.

-- ClickHouse: top 10 tests by failure frequency, last 30 days
SELECT
  test_name,
  countIf(status = 'failed') AS fail_count,
  count()                    AS total_runs,
  round(countIf(status = 'failed') / count() * 100, 2) AS fail_pct,
  max(run_at)                AS last_failure
FROM test_results
WHERE run_at >= now() - INTERVAL 30 DAY
GROUP BY test_name
HAVING total_runs >= 10          -- exclude rarely-run tests
ORDER BY fail_count DESC
LIMIT 10;

The HAVING total_runs >= 10 guard matters. Tests that ran twice and failed once look like 50% failure rate; they're noise. What you want is the tests with high absolute failure counts — those are the ones blocking engineers daily regardless of what the aggregate says.

Layer this into a Grafana dashboard with a Loki datasource for log correlation and triage time collapses fast. One team running ~3,200 tests in GitHub Actions cut triage time from 22 minutes per failure to under 4 once they wired the per-test failure panel to Loki log lines filtered by test_name. The Grafana + Loki triage setup that enables this is straightforward once you're emitting structured test logs with a consistent test_name field. For GitHub Actions specifically, tools like BuildPulse, Trunk Flaky Tests, and Datadog CI Visibility all ingest JUnit XML artifacts and expose per-test trend APIs — the best tools for tracking test failure trends in GitHub Actions differ mainly in how they handle flake classification and retention depth.

Beyond raw failure counts, add a failure concentration index — the percentage of all failures attributable to your top-N tests. If 80% of your failures come from 5 tests out of 3,000, your suite health story is really a story about those 5 tests. Tracking this ratio over time tells you whether you're actually fixing the high-signal failures or just adding more stable tests that cosmetically improve the aggregate.

# Python: compute failure concentration from parsed JUnit XML results
from collections import Counter
import xml.etree.ElementTree as ET

def failure_concentration(xml_paths, top_n=10):
    counts = Counter()
    for path in xml_paths:
        tree = ET.parse(path)
        for tc in tree.iter('testcase'):
            if tc.find('failure') is not None or tc.find('error') is not None:
                counts[tc.get('name')] += 1
    total_failures = sum(counts.values())
    top = counts.most_common(top_n)
    top_failures = sum(v for _, v in top)
    print(f"Top {top_n} tests account for {top_failures}/{total_failures} "
          f"({100*top_failures/total_failures:.1f}%) of all failures")
    return top

Run this as a post-pipeline step and emit the concentration ratio to your metrics store. When it climbs above 70%, you have a targeted triage problem, not a broad quality problem — and that distinction drives completely different engineering responses.

Two Mistakes That Make Dilution Invisible

The first mistake is using a single dashboard panel — aggregate pass rate over time — as the primary health signal for an executive or engineering review. When suite size is growing, that panel will almost always trend upward regardless of real quality. Teams see the line go up and stop asking questions. The fix is to always co-locate the suite size trend on the same panel. A rising pass rate alongside rising test count is a yellow flag, not a green one. Pipeline pass rate drift already hides broken stages; dilution makes that problem worse by masking it behind volume growth.

The second mistake is treating flakiness and failure rate as the same dimension. A test that fails 30% of the time due to a race condition is a flaky test; a test that fails 30% of the time because a feature is genuinely broken is a regression detector. Averaging them into one failure rate number destroys the distinction. Segment by failure type — deterministic vs. non-deterministic — before you aggregate anything. Flake rate averages already hide your riskiest tests; don't compound that by mixing them with true failures in a suite-level metric.

What Teams Get Wrong About Suite Growth and Quality

The dominant myth is that more tests equals more coverage equals better quality. Coverage percentage and test count are both supply-side metrics — they measure what you built, not what it catches. A suite of 5,000 tests with 90% line coverage can still miss the three integration paths where defects actually escape to production. Defect escape rate is the demand-side metric that closes this loop, and it's the number most coverage dashboards don't show. If your suite is growing but escape rate is flat or rising, the new tests aren't covering the right surface.

The second myth is that a stable, low failure rate means the suite is healthy. Stability can mean your tests are good. It can also mean your tests aren't exercising anything risky. Test duration variance is often a better proxy for suite health than failure rate — a test whose runtime swings wildly is telling you something about environmental instability even when it passes. Pairing failure concentration analysis with duration variance as a stability signal gives you a two-axis view that's much harder to fool with suite growth alone.

Aggregate failure rate is a lagging, dilution-prone metric that gets less reliable as your suite scales. Replace it — or at least supplement it — with failure concentration, per-test trend queries, and escape rate. The concrete next step: run the ClickHouse query above against your last 30 days of results and check whether your top 10 failing tests account for more than 60% of all failures. If they do, you don't have a suite health problem; you have a targeted triage problem, and that's far cheaper to fix.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.