Retry Count Inflates Pass Rate Without Fixing Flakes

Most teams treat a green pipeline as permission to ship. When retries are enabled, that green is often a lie — a flaky test failed, the runner tried again, it passed on attempt two, and the suite reported 100%. The failure happened. The root cause is still there. The dashboard just stopped showing it.

The technical problem is a metric-layer one: pass rate is computed over final attempt outcomes, not attempt counts. A test that fails three times and passes on the fourth is indistinguishable, in aggregate, from a test that passed on the first try. Retry configuration turns a reliability signal into noise, and most CI platforms — GitHub Actions, CircleCI, Jenkins, Buildkite — ship with retry support enabled by default or one config line away.

By the end of this article you'll be able to measure retry-adjusted pass rate separately from raw pass rate, write the SQL to expose which tests are masking failures, and decide when retries are a legitimate stability tool versus a metric-laundering habit.

What "Fixing" a Flake With Retries Actually Means (And What It Doesn't)

Fixing, in the context of flaky tests, means removing the non-determinism: isolating the race condition, eliminating the shared-state dependency, correcting the async timing assumption. A retry does none of that. It re-executes the same code against the same environment and hopes the probabilistic failure doesn't fire again. When it doesn't, the test is marked passed. When it does, you get a retry-exhausted failure — which is the only moment the problem becomes visible.

This matters architecturally because retries operate at the runner layer (the retry key in a GitHub Actions step, --reruns in pytest-rerunfailures, retryCount in a JUnit Surefire config), while test analytics operate at the results layer. Unless your results pipeline explicitly captures attempt number alongside outcome, the two layers never reconcile. ReportPortal ingests JUnit XML and can surface retry counts per item; Allure does the same with its @Retries annotation metadata. But neither tool will alert you that your suite's apparent 98% pass rate is actually a 91% first-attempt pass rate — you have to build that view yourself, and most teams never do.

Measuring Retry-Adjusted Pass Rate: SQL, Python, and the Dashboard Query That Exposes the Gap

The first step is capturing attempt-level data, not just final outcomes. In a JUnit XML pipeline feeding PostgreSQL or ClickHouse, add an attempt column when you parse results. With pytest-rerunfailures, each rerun emits a separate <testcase> node in the XML with a rerun tag — parse that explicitly.

-- ClickHouse: first-attempt pass rate vs. final-attempt pass rate
SELECT
    suite_name,
    countIf(attempt = 1 AND status = 'passed')   AS first_pass,
    countIf(attempt = 1)                          AS first_total,
    countIf(attempt = max_attempt AND status = 'passed') AS final_pass,
    countIf(attempt = max_attempt)                AS final_total,
    round(first_pass  / first_total  * 100, 2)   AS first_attempt_pass_rate,
    round(final_pass  / final_total  * 100, 2)   AS reported_pass_rate
FROM (
    SELECT *, max(attempt) OVER (PARTITION BY run_id, test_name) AS max_attempt
    FROM test_results
    WHERE run_date >= today() - 30
)
GROUP BY suite_name
ORDER BY (reported_pass_rate - first_attempt_pass_rate) DESC;

The delta between first_attempt_pass_rate and reported_pass_rate is your retry inflation index. A delta above 5 percentage points on a suite running 1,000 tests daily means roughly 50 failures per day are being silently absorbed. Wire this query into a Grafana panel (use the ClickHouse data source plugin, or BigQuery connector if that's your warehouse) as a time-series so you can see whether the gap is growing — a widening delta means flake density is increasing even while your dashboard stays green.

# Python: flag tests with high retry rates from parsed JUnit XML
import xml.etree.ElementTree as ET
from collections import defaultdict

def retry_rate_by_test(xml_path: str) -> dict:
    tree = ET.parse(xml_path)
    counts = defaultdict(lambda: {"runs": 0, "retries": 0})
    for tc in tree.iter("testcase"):
        name = tc.attrib["name"]
        counts[name]["runs"] += 1
        if tc.find("rerun") is not None:
            counts[name]["retries"] += 1
    return {
        k: round(v["retries"] / v["runs"], 3)
        for k, v in counts.items()
        if v["runs"] > 0
    }

rates = retry_rate_by_test("results.xml")
hot = {k: v for k, v in rates.items() if v > 0.15}
print(f"Tests with >15% retry rate: {sorted(hot.items(), key=lambda x: -x[1])}")

Any test above a 15% retry rate is a triage candidate, not a passing test. In one platform team's experience, wiring this script into a nightly Slack report (via a simple curl to the Slack webhooks API) dropped mean time to flake triage from 22 minutes per failure to under 4 — because engineers stopped hunting through CI logs and started reading a ranked list. For a deeper look at what other signals live in your results beyond the pass/fail surface, test execution metadata like attempt counts and duration variance carries more diagnostic weight than outcome alone.

On the CI side, limit retry scope explicitly. In GitHub Actions, scope retries to infrastructure-class failures only:

# .github/workflows/test.yml
- name: Run pytest
  id: pytest
  run: pytest --tb=short -q
  continue-on-error: true

- name: Retry on infra failure only
  if: steps.pytest.outcome == 'failure' && contains(steps.pytest.outputs.stderr, 'ConnectionRefusedError')
  run: pytest --tb=short -q --last-failed

Blanket retry: 3 on every test step is the configuration equivalent of muting your smoke detector. Conditional retries on known transient infrastructure errors (network timeouts, container startup races) are legitimate; retrying application-logic test failures is not.

Why Senior Engineers Still Ship Retry-Inflated Dashboards

The most common mistake is treating retry configuration as a reliability improvement rather than a measurement decision. Teams add --reruns 3 to stabilize a noisy suite during a crunch, the pipeline goes green, and the configuration never gets revisited. Six months later the retry count is load-bearing — remove it and the suite drops to 85% pass rate, which is the honest number. The org-level cause is that retries are cheap to add and their cost is invisible until you measure first-attempt rates explicitly. Pass rate drift on a dashboard often hides exactly this: a gradually increasing retry dependency that the top-line metric never surfaces.

A second failure mode is aggregating test results at the suite level before storing them. If your results pipeline records one row per test run (final outcome only), you've permanently discarded the attempt-level data needed to compute retry inflation. This is a schema decision that's painful to reverse. Store attempt number, attempt outcome, and attempt duration as first-class columns from day one — adding them later requires re-parsing historical artifacts, if those artifacts are even still retained.

The Myths That Keep Flakes Hidden Behind Green Pipelines

Myth 1: A passing test is a reliable test. A test that passes on attempt three has demonstrated it can fail under normal conditions. That's the opposite of reliable. Myth 2: Retries are a flake-fixing strategy. Retries are a flake-tolerance strategy. There's a meaningful difference — tolerance defers the fix, and deferred fixes compound. The work of actually stopping flakes from returning requires root-cause analysis: identifying whether the failure is environmental, timing-dependent, or a genuine intermittent defect. Retries make that analysis harder by reducing the failure signal frequency.

Myth 3: High pass rate means low defect risk. If your pass rate is computed over retry-exhausted outcomes, it's a measure of retry budget, not code quality. Teams that rely on pass/fail as their primary quality signal miss the fact that a flaky test covering a critical path is more dangerous than a consistently failing test covering a minor edge case — because the flaky one gives false confidence. The corrective is to track first-attempt pass rate, retry rate per test, and flake recurrence rate as separate metrics, each with its own alert threshold.

Retries belong in your toolbox for infrastructure transience, not as a substitute for fixing non-deterministic tests. Start by adding attempt-level columns to your results schema, run the retry inflation query against the last 30 days, and surface the delta in your existing Grafana or Datadog dashboard. If the gap is above 3 percentage points, you have a flake backlog that your current metrics are actively hiding. Prioritize the top-10 retry offenders by test name — that list is your real fix queue.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.