Why CI Test Summaries Undercount Failures

Most teams treat the CI test summary like a scoreboard: 312 passed, 4 failed, merge blocked. The number feels authoritative because the pipeline printed it. What it rarely reflects is the actual failure surface — retried tests that passed on the third attempt and vanished from the count, setup failures that aborted an entire class before a single @Test ran, or timeout-killed workers that reported zero results instead of N failures. The summary says four. The real number might be thirty.

The undercounting problem is structural. JUnit XML, the de facto interchange format across Pytest, JUnit 5, Playwright, and every CI adapter in existence, has ambiguous semantics around skipped, aborted, and retried cases. Tooling layers — GitHub Actions test annotations, Jenkins test trend charts, Allure report aggregators — each make different choices about what counts. Those choices compound silently.

By the end of this article you will know exactly where the count drops, how to write a SQL query that recovers the missing failures from raw XML ingestion, and which dashboard patterns give your CI failure triage an honest baseline instead of a flattering one.

Where the JUnit XML Count Actually Breaks Down

JUnit XML defines four terminal states per test case: passed, failure, error, and skipped. The problem is that CI orchestration adds states the schema never anticipated: retried-then-passed, aborted-by-timeout, never-started-due-to-fixture-crash, and quarantined-flaky. Most report parsers map these to skipped or simply drop them. GitHub Actions' built-in test annotation parser, for example, counts only <failure> and <error> elements — a Pytest test marked xfail that actually fails (an xpass) appears as passed. Buildkite's test analytics SDK has similar edge cases around parallel step aggregation.

The second break point is suite-level vs. case-level attribution. When a @BeforeAll method throws, JUnit 5 emits a single <testsuite errors="1"> with no child <testcase> elements. The suite error increments the error counter, but the 40 tests that never ran are invisible. Your test results dashboard shows one error; your engineers just lost coverage on 40 paths. This is the most common source of systematic undercounting in large Java and Kotlin test suites, and it is trivially reproducible yet almost never surfaced by default tooling.

Building an Accurate Embedded CI Failure Analysis Dashboard

The fix starts at ingestion, not at the dashboard layer. Parse raw JUnit XML into a structured store — ClickHouse or BigQuery for scale, PostgreSQL for teams under ~10M test results/month — and preserve every attribute the schema allows, including testsuite/@tests, testsuite/@errors, testsuite/@skipped, and the presence or absence of child <testcase> elements. The derived failure count is not just COUNT(failure); it includes orphaned suite errors and inferred-never-ran cases.

-- PostgreSQL: recover failures hidden by suite-level aborts
SELECT
  run_id,
  suite_name,
  suite_tests_declared,
  COUNT(tc.id)                          AS cases_reported,
  suite_tests_declared - COUNT(tc.id)   AS cases_never_ran,
  SUM(CASE WHEN tc.status = 'failure' THEN 1 ELSE 0 END) AS explicit_failures,
  SUM(CASE WHEN tc.status = 'error'   THEN 1 ELSE 0 END) AS explicit_errors,
  suite_errors                          AS suite_level_errors
FROM test_suites ts
LEFT JOIN test_cases tc ON tc.suite_id = ts.id
GROUP BY run_id, suite_name,
         suite_tests_declared, suite_errors
HAVING suite_tests_declared - COUNT(tc.id) > 0
    OR suite_errors > 0
ORDER BY cases_never_ran DESC;

Run this query against a week of CI data and the cases_never_ran column is rarely zero on any team with more than a few hundred tests. Wire it into a Grafana panel with a run_id variable and you have an embedded CI failure analysis view that shows true blast radius per build. One platform team at a mid-size SaaS org found that their "4 failures per run" average was masking an average of 17 never-ran cases per run once suite aborts were counted — a 4× undercount that had been invisible for eight months.

For retry inflation, the fix is a status-history column. When Pytest-rerunfailures or Gradle's --rerun-tasks re-executes a test, emit both attempts to your store with a attempt_number field. Your summary query then counts a test as failed if any attempt failed, not just the last one. This matters for flaky-test triage: a test that fails on attempt 1 and passes on attempt 3 is a flaky test, not a passing test, and it should feed your latency and dependency failure signals the same way an outright failure does.

# GitHub Actions: preserve attempt metadata in JUnit output
- name: Run tests
  run: pytest --junitxml=results.xml --reruns 2 --reruns-delay 1

- name: Annotate with attempt context
  uses: dorny/test-reporter@v1
  with:
    name: Pytest Results
    path: results.xml
    reporter: java-junit
    # dorny does NOT deduplicate retries — feed raw XML to your store separately
    fail-on-error: false

- name: Upload raw XML for ingestion
  uses: actions/upload-artifact@v4
  with:
    name: junit-raw-${{ github.run_attempt }}
    path: results.xml

Uploading the raw XML per run_attempt means your ingestion pipeline sees every attempt independently. From there, a Grafana CI test dashboard panel that groups by test_id, run_id and uses MIN(status) over attempts gives you the pessimistic (honest) view, while MAX(status) gives you the optimistic (what CI reported) view. The delta between those two panels is your retry-hidden failure rate — a metric worth tracking as a first-class engineering signal alongside P95 duration.

Pitfalls in CI Test Dashboard Engineering Run Results

Aggregating at the wrong layer is the most common mistake. Teams pipe JUnit XML through their CI platform's native aggregator (Jenkins JUnit plugin, GitHub Actions test annotations) and then export those aggregated counts to Grafana or Datadog. By then, the suite-level errors and retry deduplication choices are already baked in and invisible. You cannot recover the raw signal from a pre-aggregated metric. The fix is always to ingest raw XML independently of whatever the CI platform shows in its UI — treat the CI summary as a convenience view, not the source of truth.

Treating quarantine as resolution is the second trap. ReportPortal and some Allure Enterprise configurations let you mark a test as "known issue," which removes it from the failure count in dashboards. That's useful for noise reduction but catastrophic for trend analysis if the quarantine list is never pruned. Teams end up with 60+ quarantined tests, a green dashboard, and a test suite that is structurally broken. Quarantined tests should still feed a separate automated triage pipeline so they don't disappear from engineering awareness entirely.

What Most Teams Get Wrong About Test Failure Counts

Myth: the CI summary is the failure count. It is a rendering of one parser's interpretation of JUnit XML, filtered through the platform's retry and skip semantics. Two CI platforms running the same XML file will often report different failure counts because they disagree on whether error equals failure, whether skipped counts against the total, and whether a suite-level abort propagates to child counts. The only authoritative count comes from parsing the raw XML yourself with explicit, documented rules. If you want an LLM to help interpret patterns in that raw data, feeding it the actual XML rather than the summary text yields dramatically better signal.

Myth: flaky tests that eventually pass aren't failures. They are intermittent failures with a retry budget that happened to be sufficient this run. Treating them as passes removes them from failure trend charts, which makes your suite look more stable than it is and hides the infrastructure or test-isolation problems driving the flakiness. A related myth is that a passing overall rate means the suite is healthy — a suite with 95% pass rate but 40 tests in permanent quarantine and 15 suite-level aborts per run is not healthy. Tracing test execution at the span level makes these hidden failure modes visible in a way that JUnit XML alone never will.

The CI test summary is a starting point, not a measurement. Ingest raw JUnit XML, preserve suite-level error counts and per-attempt status, and build your failure triage charts against that data rather than the platform's pre-aggregated view. Run the suite-abort query above against one week of CI history — the gap between declared tests and reported cases is almost always larger than your team expects, and closing that gap is where real stability work begins.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.