Failure Rate Rises When You Remove Tests

Most teams treat test deletion as a maintenance win: fewer flaky tests, cleaner reports, faster pipelines. Then the failure rate climbs. Someone files a ticket wondering if the suite regressed. It didn't — the denominator shrank, and the math is doing exactly what it should. The problem is that almost no one models this before pulling the trigger on a bulk delete.

The failure rate metric is a ratio. When you remove tests that were passing reliably, you reduce the denominator without touching the numerator. The same number of failing tests now represents a larger share of a smaller suite. This is not a bug in your reporting — it is an accurate signal that your suite's composition has changed, and it carries real implications for how you interpret trend lines, SLOs, and release gates.

By the end of this article you will know how to model the denominator effect before a deletion, write a pre-flight SQL check against your results store, and set up a Grafana annotation so the rate spike is never mistaken for a regression.

The Denominator Effect: What Actually Moves the Metric

Failure rate is failures / total_executed. Every test you remove that was passing reduces total_executed by one while leaving failures unchanged. If your suite has 1,000 tests, 20 failing, and you delete 200 consistently-green tests, your failure rate jumps from 2% to 2.5% — a 25% relative increase with zero new defects introduced. At scale the effect compounds: delete 500 passing tests and the same 20 failures now read as 4%.

This is the same arithmetic that explains why failure rate drops as your test suite grows — the relationship is symmetric. Growth dilutes failures; shrinkage concentrates them. Where this gets dangerous is in automated release gates keyed to a failure rate threshold. A deletion sprint that crosses a 3% gate threshold blocks a release not because quality degraded but because the denominator moved. Teams that haven't modeled this get burned by their own tooling.

Pre-Flight Modeling and Annotating the Spike

Before any bulk deletion, run a denominator impact query against your results store. This example targets a PostgreSQL schema where each row is a test execution:

-- Estimate post-deletion failure rate
WITH current AS (
  SELECT
    COUNT(*)                                        AS total_executed,
    COUNT(*) FILTER (WHERE result = 'failed')       AS failures
  FROM test_executions
  WHERE run_id = (SELECT MAX(run_id) FROM test_executions)
),
deletions AS (
  -- IDs you plan to remove
  SELECT COUNT(*) AS to_remove
  FROM test_executions
  WHERE run_id = (SELECT MAX(run_id) FROM test_executions)
    AND test_id = ANY(ARRAY[/* your deletion list */]::int[])
    AND result = 'passed'
)
SELECT
  c.failures::float / c.total_executed                            AS current_rate,
  c.failures::float / (c.total_executed - d.to_remove)           AS projected_rate,
  (c.total_executed - d.to_remove)                               AS new_denominator
FROM current c, deletions d;

Run this before merging a test-removal PR. If projected_rate crosses a release-gate threshold, you have two options: adjust the gate temporarily with a documented rationale, or stagger the deletion across multiple runs so the rate moves gradually. The second option is almost always preferable because it keeps your trend line interpretable.

Once the deletion lands, annotate the event in Grafana so every future viewer of the trend chart sees the inflection point in context. The Grafana HTTP API makes this a one-liner you can drop into a GitHub Actions post-step:

# .github/workflows/test-suite-maintenance.yml (post-deletion step)
- name: Annotate Grafana — suite deletion
  run: |
    curl -s -X POST "$GRAFANA_URL/api/annotations" \
      -H "Authorization: Bearer $GRAFANA_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{
        "dashboardUID": "test-suite-health",
        "panelId": 12,
        "time": '"$(date +%s%3N)"',
        "tags": ["suite-change", "deletion"],
        "text": "Removed '"$DELETED_COUNT"' passing tests — denominator reduced"
      }'

Teams that wired this annotation into their deletion workflow reported that triage time on apparent rate spikes dropped from over 20 minutes per incident to under 3, because engineers could immediately see the cause without bisecting the git log. Pair this with a first-attempt vs. final pass rate KPI panel so you can confirm the underlying failure signatures haven't actually changed — only the denominator did.

Where Engineers Get Tripped Up During Suite Pruning

The most common mistake is deleting tests without auditing their result distribution first. Engineers reach for "delete everything that's been skipped or quarantined" without checking whether those tests were passing before quarantine. Skipped tests distort every rate you report precisely because they are already excluded from the denominator — removing them has no rate impact — but tests that were actively passing and are now being deleted absolutely do. Conflating the two categories in a single cleanup PR is the root cause of most surprise rate spikes.

The second mistake is not versioning the gate thresholds alongside the suite size. A 2% failure rate gate that was calibrated against a 2,000-test suite is miscalibrated against a 1,200-test suite after a pruning cycle. Thresholds should be stored as configuration with a suite_size_at_calibration field, and any deletion that changes suite size by more than 10% should trigger a gate recalibration review. Most teams skip this step because it feels like process overhead — until it blocks a deploy on a Friday.

Myths That Survive Every Postmortem on This Topic

Myth 1: a rising failure rate after deletion means something broke. It means the denominator shrank. These are different facts and require different responses. The correct response to a denominator-driven rate increase is an annotation and a threshold review, not a code bisect. Myth 2: removing flaky tests improves suite reliability. It improves the appearance of reliability. The flakiness was a signal about a real instability — in test infrastructure, environment, or application state. Deleting the test removes the signal; it does not fix the instability. Flake rate resets after a suite refactor for exactly this reason: the underlying conditions persist and surface in different tests.

Myth 3: pass rate is the right primary metric for suite health. Pass rate is a lagging, composition-sensitive indicator. Duration variance, retry rates, and failure signature clustering are harder to game and more predictive of real quality signals. A suite where 98% of tests pass but 40% of failures cluster in the same three services every sprint is not a healthy suite — it is a suite with a well-hidden chronic problem. Treat pass rate as a threshold check, not a quality proxy.

Suite pruning is legitimate engineering work — bloated, redundant tests slow pipelines and obscure real signal. The discipline is in modeling the denominator impact before the deletion, annotating the event so trend lines stay interpretable, and recalibrating release gates when suite composition changes materially. If you want to go deeper on how composition changes interact with rate metrics over time, the analysis of how adding tests to a passing suite affects failure rate is the complementary read.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.