Failure Rate Recovers. The Defect Doesn't.

Most teams treat a recovering failure rate as a resolved problem. The dashboard goes green, the on-call alert clears, the retro gets closed. But failure rate is a lagging, aggregated signal — it measures what already happened across a window of runs, not whether the root cause is gone. A defect can still be live in production while your 7-day pass rate climbs back to 98%.

This happens more often than post-mortems admit. A flaky environment gets restarted, a retry policy absorbs the failures, a new test is added that passes cleanly and dilutes the denominator — and suddenly the rate looks healthy again. The defect never shipped a fix; the metric just stopped reflecting it. As suites grow, failure rate drops structurally even without any quality improvement, which makes this masking effect worse at scale.

By the end of this article you'll be able to instrument your pipeline to distinguish metric recovery from defect resolution, write the queries that surface the gap, and set up alerting that doesn't clear until there's evidence the fix actually landed.

The Gap Between Metric Recovery and Defect Resolution

Failure rate recovery is a statistical event: enough recent runs passed that the rolling window average crossed your threshold. Defect resolution is a causal event: the code path that caused failures was changed, verified, and deployed. These two events are not the same, and they don't happen at the same time. The gap between them is where escaped defects live — and it's exactly what defect escape rate is designed to measure.

In a modern test architecture, you need at least three signals to distinguish the two states: (1) the failure rate trend itself, (2) the first-pass rate of the specific tests that were failing — without retries — and (3) a deployment event marker indicating a fix was actually merged and released. Without all three wired together, your observability layer can only tell you that things look better, not that they are better.

Instrumenting the Gap: Queries, Markers, and Alerting

Start by separating retry-adjusted pass rate from first-attempt pass rate in your test results store. If you're writing to PostgreSQL or ClickHouse, the schema distinction matters: store attempt_number alongside each result row. The query below computes both rates per test per day so you can see them diverge:

-- PostgreSQL: first-attempt pass rate vs. overall pass rate per test
SELECT
  test_name,
  run_date,
  COUNT(*) FILTER (WHERE attempt_number = 1 AND status = 'passed')::float
    / NULLIF(COUNT(*) FILTER (WHERE attempt_number = 1), 0) AS first_attempt_pass_rate,
  COUNT(*) FILTER (WHERE status = 'passed')::float
    / NULLIF(COUNT(*), 0) AS overall_pass_rate
FROM test_results
WHERE run_date >= NOW() - INTERVAL '14 days'
GROUP BY test_name, run_date
ORDER BY test_name, run_date;

When overall_pass_rate climbs while first_attempt_pass_rate stays flat, you're looking at retry absorption — not recovery. That divergence is your primary signal that the defect is still live.

Next, inject deployment markers into the same data pipeline. In GitHub Actions, emit a structured event on every merge to main:

# .github/workflows/deploy.yml (excerpt)
- name: Emit deployment marker
  run: |
    curl -X POST "$RESULTS_INGEST_URL/events" \
      -H "Content-Type: application/json" \
      -d '{
        "event_type": "deployment",
        "sha": "${{ github.sha }}",
        "timestamp": "${{ steps.ts.outputs.iso8601 }}",
        "fix_refs": ${{ toJSON(github.event.commits.*.message) }}
      }'

With deployment markers in the same store, you can write a query that flags any test whose first-attempt pass rate recovered before a deployment event landed — the exact signature of metric-only recovery. One team running this on ClickHouse against 40k daily test results found that 11% of "recovered" tests in a given month had no corresponding deployment within 48 hours of their rate improvement. Every one of those was a retry-masked defect. Triage time on those incidents dropped from over 20 minutes per failure to under 5 once the dashboard surfaced the deployment gap directly alongside the rate chart.

For alerting, don't clear a failure-rate alert on rate alone. In Prometheus/Alertmanager, add a condition that requires a deployment event within the resolution window:

# alertmanager rule: only resolve if a deployment was recorded
- alert: TestFailureRateRecovered
  expr: |
    test_first_attempt_pass_rate{job="ci"} > 0.97
    AND ON(test_name)
    (time() - deployment_last_timestamp{job="ci"}) < 172800
  for: 30m
  labels:
    severity: info
  annotations:
    summary: "{{ $labels.test_name }} rate recovered with a recent deployment"

Without the deployment join, the alert resolves on retries. With it, you get a meaningful signal: rate is up and a fix was shipped. Wire this into your CI failure analysis dashboard so the recovery state is visible in context, not just in an alert channel.

Where Engineers Instrument Correctly but Still Get Burned

The most common mistake is trusting aggregate window size to provide resolution. Teams set a 7-day rolling window because it smooths noise, but that same smoothing delays the signal that a defect is still active. A test that fails once every three runs will show a 67% pass rate on day one and a 90%+ rate by day seven — with no fix shipped. Shrink your alerting window to 24–48 hours for tests that were recently failing, and keep the longer window only for baseline health reporting. These are different use cases and should not share the same query.

The second mistake is treating duration variance as unrelated to this problem. A test that starts passing again but whose P95 runtime has spiked 3× is often hitting a slow fallback path — the defect is still there, the test is just no longer failing hard. Correlate pass rate recovery with duration stability before closing an incident. If the rate recovered but the P95 is still elevated, you're not done.

Why Design Matters More Than Dashboards Here

The most widespread myth is that better dashboards solve this problem. They don't — they visualize it. The actual fix is upstream: test design that makes defect presence structurally visible. A test that retries three times internally and reports a single pass/fail result has already destroyed the signal before it reaches any dashboard. Retry logic belongs at the runner level with full attempt logging, not embedded silently in test code. If your JUnit XML doesn't include attempt attributes, your observability layer is working with incomplete data by construction.

A related misunderstanding is that failure rate is the right primary metric for defect tracking. It isn't — it's a proxy for test suite health, not product quality. A defect that only manifests under specific load conditions, or only on certain browser/OS combinations in Playwright or Selenium, may never produce a failure rate signal above your threshold. Failure clustering across environment dimensions is a more reliable defect indicator than aggregate rate, because it preserves the specificity that averaging destroys. Pass rate tells you something is wrong. Cluster analysis tells you what and where.

Failure rate recovering is a necessary condition for closing an incident — not a sufficient one. The minimum bar is: first-attempt pass rate is up, a deployment marker exists within the resolution window, and duration variance hasn't spiked. If any of those three are missing, the defect is still a candidate. Start by adding attempt_number to your test results schema if it isn't there; everything else in this article depends on that single column being populated correctly.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.