Rerun Depth Skews MTTD Across Pipelines

Most teams treat a green pipeline as a clean bill of health. But if that green came after three automatic reruns of the same five tests, you didn't detect a passing suite — you delayed detecting a failing one. The timestamp CI records as "test result" is the moment the final retry resolved, not the moment the first failure fired. That gap is silent, it accumulates across pipelines, and it directly corrupts your Mean Time to Detect calculations.

Rerun depth — the number of retry attempts configured per test or per job — is rarely treated as a reporting variable. It's treated as a reliability knob. That mental-model split is where the problem starts: the same configuration that reduces noise in your pass-rate dashboard is inflating the clock on every real defect that slips through.

By the end of this article you'll be able to measure how much rerun depth is skewing your MTTD, instrument your pipelines to capture first-failure timestamps separately from final-resolution timestamps, and write the queries that expose the distortion per pipeline stage.

What "Rerun Depth" Actually Means for Test Results

A test rerun (sometimes called a retry) is a re-execution of a failed test within the same CI run, triggered automatically by the runner. In pytest it's --reruns N via pytest-rerunfailures; in Go it's go test -count=N or third-party wrappers like gotestsum --rerun-fails; in GitHub Actions it's retry-on-error or a matrix re-trigger. Rerun depth is the maximum number of those attempts — depth 3 means a test can fail twice and still report green on the third pass. The test result recorded in your JUnit XML, Allure report, or ReportPortal launch is almost always the final-attempt outcome, not the first-failure outcome.

In a modern test architecture with parallel stages — unit → integration → e2e → contract — rerun depth compounds. A depth-2 retry at the integration stage adds up to ~4 minutes of wall-clock latency before a real defect surfaces as a pipeline failure. Multiply that across dependent stages and the actual detection event can lag the first failure signal by 10–15 minutes in a typical mid-size suite. That latency is invisible in standard dashboards because the event log only records the resolved state, not the intermediate failures that were silently retried away.

Instrumenting Pipelines to Capture First-Failure Timestamps

The fix starts at the data layer. You need two timestamps per test execution: first_failure_at (when the initial failure fired) and resolved_at (when the final attempt completed). Most runners don't emit both natively, so you wire them in yourself.

For pytest with pytest-rerunfailures, a conftest hook captures attempt-level data before the runner discards it:

# conftest.py
import time, json, os

_first_failures: dict[str, float] = {}

def pytest_runtest_logreport(report):
    if report.when == "call" and report.failed:
        key = report.nodeid
        if key not in _first_failures:
            _first_failures[key] = time.time()

def pytest_sessionfinish(session, exitstatus):
    out = []
    for item in session.items:
        nid = item.nodeid
        out.append({
            "nodeid": nid,
            "first_failure_at": _first_failures.get(nid),
            "resolved_at": time.time(),
        })
    with open(os.environ.get("RERUN_AUDIT_PATH", "rerun_audit.json"), "w") as f:
        json.dump(out, f)

Push that JSON to your warehouse alongside the JUnit XML. In BigQuery or ClickHouse, the MTTD query becomes honest:

-- ClickHouse: true MTTD vs reported MTTD, by pipeline stage
SELECT
    pipeline_stage,
    avg(dateDiff('second', first_failure_at, defect_opened_at)) AS true_mttd_s,
    avg(dateDiff('second', resolved_at,      defect_opened_at)) AS reported_mttd_s,
    avg(dateDiff('second', first_failure_at, resolved_at))      AS avg_rerun_lag_s,
    max(rerun_depth)                                            AS max_depth
FROM test_run_events
WHERE run_date >= today() - 30
  AND final_status = 'failed'   -- only runs where defect eventually surfaced
GROUP BY pipeline_stage
ORDER BY avg_rerun_lag_s DESC;

In one platform team's ClickHouse instance, this query revealed that the e2e stage — configured at depth 3 with a 90-second retry delay — was adding an average of 6.4 minutes of rerun lag to every true defect. Triage time dropped from 22 minutes per failure to under 5 once the Grafana dashboard was wired to first_failure_at instead of resolved_at, and the on-call Slack alert was updated to fire on first failure rather than final resolution. For Jenkins pipelines specifically, silent retry misconfiguration creates similar invisible gaps — the rerun gap patterns in Jenkins follow the same root cause: the recorded timestamp is the retry resolution, not the original fault.

For Go test suites using gotestsum, pass --rerun-fails=3 --rerun-fails-report=rerun.json — that JSON contains per-package attempt counts you can join against your test result store. In GitHub Actions, annotate the step with FIRST_FAILURE_TS as an output before the retry wrapper kicks in; the GitHub pipeline reporting layer can then surface both timestamps in the job summary without custom tooling.

Where Senior Engineers Still Get Burned by Retry Configuration

The most common mistake is setting rerun depth per-suite rather than per-test-class. A depth-3 blanket retry on a 2,000-test suite means deterministic failures — wrong environment variable, broken migration, bad test data — get retried three times before surfacing. The retry was designed for flaky network calls in e2e tests; it silently absorbs real regressions in unit tests that should have zero tolerance for failure. The fix is to scope retries: depth 2–3 for known-flaky e2e tests tagged explicitly, depth 0 for unit and contract tests.

The second mistake is not accounting for rerun depth when computing flake rate. If a test passes on attempt 3 of 3, most flake-detection pipelines mark it as "passed" and never increment the flake counter. Your flake rate looks clean while the underlying instability compounds across stages — exactly the dynamic described in how flake rate compounds across dependent pipeline stages. Track pass-on-first-attempt (POFA) as a separate metric from final-pass-rate; the gap between them is your true flake signal.

Myths About Reruns That Distort How Teams Report Quality

Myth 1: A green pipeline means the suite passed. It means the suite resolved green after up to N attempts. These are not the same statement. When you report pass rate to engineering leadership, you're reporting final-resolution rate — which is a function of both code quality and retry configuration. Increase depth from 1 to 3 and your reported pass rate will rise without a single line of production code changing. This is the same category of distortion covered in how suite composition skews the metrics you report up — the number looks better because the measurement changed, not the system.

Myth 2: Reruns fix flakiness. Reruns mask flakiness at the reporting layer. The test is still non-deterministic; you've just moved the failure signal from "visible in the dashboard" to "hidden in retry logs." Teams that rely on reruns as a long-term flakiness strategy accumulate technical debt in the form of inflated MTTD and degraded pipeline trust. Reruns are a short-term circuit breaker while you fix root cause — not a substitute for auto-detecting and quarantining flaky tests systematically. Treat any test that requires a retry to pass as a defect-in-progress, not a resolved run.

Rerun depth is a first-class reporting variable, not just an infrastructure knob. Start by auditing your current retry configuration per stage and computing the gap between first_failure_at and resolved_at in your test result store. If that gap averages more than two minutes on your slowest stage, your MTTD numbers are telling a story that's at least partially fictional. Fix the timestamp, then fix the retry scope — in that order.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.