Rerun Gaps in Jenkins: Silent Retry Misconfiguration

Most teams treat a green Jenkins build as a solved problem. What they miss is the quiet machinery underneath: retry plugins firing on every timeout, rerun counts never surfacing in the JUnit XML, and a pass rate that climbs while the underlying instability stays perfectly intact. The build looks stable. The test pipeline is lying.

The specific failure mode here is rerun gap — the delta between how many times a test actually executed and how many executions appear in your aggregated results. Jenkins exposes this gap more than most CI systems because its retry surface area is fragmented across at least three independent layers: the retry step in Declarative Pipeline, the Naginator plugin for job-level reruns, and test-framework-level retries in Pytest or JUnit. When those layers overlap without coordination, you get inflated pass rates and no audit trail.

By the end of this article you'll know how to detect rerun gaps from raw Jenkins data, write a query that surfaces misconfigured retry stacks, and wire a Grafana panel that makes the gap visible before it poisons your reliability scorecard.

What a Rerun Gap Is and Why Jenkins Creates Them

A rerun gap exists when the number of test executions recorded in your result store is lower than the number of actual runs that occurred in CI. In Jenkins, this happens because the junit archiver step captures only the final XML written to the workspace — not intermediate runs. If a Pytest session retries three times via pytest-rerunfailures and passes on the third attempt, the archived XML contains one <testcase> element with status passed and zero evidence of the two failures that preceded it. The rerun gap is two.

This matters architecturally because Jenkins pipelines are not a single execution unit. A test pipeline in a large monorepo might span parallel stages, matrix builds, and post-build rerun jobs — each with its own retry configuration and its own XML output path. Without a unified execution ID threading through all of them, your result aggregator (Allure, ReportPortal, a ClickHouse table, whatever) sees disconnected snapshots. The gap isn't a bug you can file; it's a structural property of how Jenkins composes stages. Understanding it is prerequisite to fixing it.

Finding and Closing the Gap: Queries, Pipeline Config, and Dashboards

Start with the data you already have. If you're storing JUnit XML results in PostgreSQL or ClickHouse, a rerun gap shows up as tests with a suspiciously low failure_count relative to their flakiness_score — or more directly, as tests that appear in build logs (via Loki) more times than they appear in your results table. This query finds the discrepancy:

-- PostgreSQL: tests where log execution count exceeds result record count
-- Assumes test_results and loki_test_executions are populated by your ingestion pipeline
SELECT
  r.test_name,
  r.build_id,
  COUNT(r.id)                        AS result_records,
  MAX(l.execution_count)             AS log_executions,
  MAX(l.execution_count) - COUNT(r.id) AS rerun_gap
FROM test_results r
JOIN loki_test_executions l
  ON l.build_id = r.build_id
  AND l.test_name = r.test_name
WHERE r.build_date >= NOW() - INTERVAL '14 days'
GROUP BY r.test_name, r.build_id
HAVING MAX(l.execution_count) > COUNT(r.id)
ORDER BY rerun_gap DESC
LIMIT 50;

Any row returned here is a test where Jenkins ran it more times than your result store knows about. A rerun gap of 2+ on the same test across multiple builds is the fingerprint of a misconfigured retry stack — typically pytest-rerunfailures set to 3 retries at the framework level and a retry(2) step wrapping the stage. That's up to 9 actual executions per nominal "run," none of them visible in the archived XML.

On the pipeline side, the fix is explicit retry telemetry. Emit a structured log line at each retry attempt and attach a stable execution_id:

// Jenkinsfile (Declarative Pipeline)
stage('Test') {
  steps {
    script {
      def attempt = 0
      retry(3) {
        attempt++
        echo "RETRY_TELEMETRY attempt=${attempt} stage=Test build=${env.BUILD_ID}"
        sh 'pytest tests/ --rerun-failures 0 --junitxml=results/junit.xml'
      }
    }
    junit 'results/junit.xml'
  }
}

Setting --rerun-failures 0 in the Pytest invocation is intentional: delegate retries entirely to the pipeline layer so you have one authoritative retry counter. The RETRY_TELEMETRY log lines get ingested by Loki, where a LogQL query feeds the Grafana panel. One team using this pattern dropped triage time from 22 minutes per unstable build to under 5 minutes once the Grafana panel showed retry counts alongside pass/fail — engineers stopped asking "why is the Jenkins build unstable?" and started reading the gap metric directly. For deeper context on what Jenkins test reporting should actually surface, the structural gaps here are the same ones that make standard build dashboards misleading.

// Grafana panel JSON (partial) — rerun gap time series
{
  "type": "timeseries",
  "title": "Rerun Gap by Test (14d)",
  "targets": [{
    "expr": "sum by (test_name) (jenkins_rerun_gap_total{job=~\"$job\"})",
    "legendFormat": "{{test_name}}"
  }],
  "fieldConfig": {
    "defaults": { "thresholds": {
      "steps": [
        { "color": "green", "value": 0 },
        { "color": "yellow", "value": 1 },
        { "color": "red", "value": 3 }
      ]
    }}
  }
}

Expose the jenkins_rerun_gap_total metric from a small Python exporter that tails Loki for RETRY_TELEMETRY lines and increments a Prometheus counter. The threshold at 3 catches the overlapping-retry-layer scenario before it distorts your quarterly numbers.

Where Engineers Trip Up: Naginator, Matrix Builds, and XML Overwriting

The most common mistake is running Naginator (job-level rerun) on top of an already-retrying pipeline stage. Naginator re-executes the entire job, which means a new workspace, a new XML file written to the same path, and the junit archiver overwriting the previous run's results. You end up with one result record per job execution, not per test execution — the gap is invisible because the evidence was deleted. The fix: write JUnit XML to a path that includes ${BUILD_NUMBER} and archive with a glob pattern. Simple, but routinely skipped because the default template doesn't do it.

The second pitfall is matrix builds where each axis writes to the same results/junit.xml path. In a parallel matrix, the last stage to finish wins the write race. Engineers notice this when a test that failed on python3.10 doesn't appear in the report because python3.12 finished last and its passing result overwrote it. Use results/junit-${MATRIX_AXIS}.xml and archive results/junit-*.xml. This is also why Jenkins is slow to diagnose — the symptom (missing failures) is indistinguishable from a genuine pass unless you're tracking execution counts independently.

Myths That Keep Rerun Gaps Hidden in Production

Myth 1: A stable pass rate means the suite is stable. It means retries are working, which is not the same thing. Retry counts inflate pass rates without addressing root causes — a test that passes on attempt 3 every time is a flaky test with a mask on. Myth 2: The JUnit XML is the ground truth. It's the final-state snapshot. Intermediate failures, retry durations, and attempt counts are not in the spec and most frameworks don't emit them. If your observability stops at the XML, you're missing the execution history. Myth 3: Marking a build "unstable" in Jenkins is informative. It's a status, not a diagnosis. An unstable build could mean one flaky test retried twice, or it could mean 40% of a suite is non-deterministic — the label doesn't distinguish them.

The corrective frame is to treat retry data as a first-class signal in your test insights pipeline, not as an implementation detail of the test runner. Rerun counts, attempt durations, and gap metrics belong in the same store as your pass/fail data, queryable alongside it. Teams that instrument at this level stop asking why a build is unstable and start asking which specific tests are driving retry load — a question that has a tractable answer.

Rerun gaps are a data integrity problem before they're a flakiness problem. Start by auditing your Jenkins pipeline for overlapping retry layers, then add structured RETRY_TELEMETRY log lines and a Loki-backed gap metric. Once the gap is visible in Grafana, prioritize any test with a gap above 2 across more than three builds — those are your highest-leverage flake candidates. For a structured way to track progress over time, the quarterly reliability scorecard gives you the right cadence to measure whether your fixes are holding.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.