Zero-Failure Runs Can Hide Unexecuted Tests

A zero-failure run feels like a win. The dashboard is green, the PR merges, the deployment ships. But "no failures" and "everything ran" are not the same statement, and conflating them is one of the most persistent blind spots in CI test analytics. Skipped tests, excluded tags, collection errors, and dynamically-generated test IDs can all produce a clean run that covers far less than you think.

The problem compounds silently. A test that never executes doesn't show up in failure trends, doesn't accumulate flake history, and doesn't trigger any alert. Your historical failure trends look stable — because the signal was never written in the first place. The absence of data looks identical to the presence of passing data.

By the end of this article you'll know how to instrument your pipeline to track expected vs. actual test counts, surface unexecuted tests in your CI test dashboard, and write the queries that make the gap visible before it becomes a production incident.

The Difference Between "Passed" and "Executed"

JUnit XML and most test runners report on the tests they ran. They have no obligation to report on the tests they didn't run. A pytest --ignore=tests/payments/ flag, a misconfigured testPathPattern in Jest, or a Playwright project filter that silently drops a browser target all produce structurally valid reports with zero failures — and zero coverage of the omitted surface. The report is truthful; the interpretation is wrong.

In a modern test architecture, "executed" needs to be a first-class metric alongside pass rate and duration. This means maintaining a test inventory — a canonical list of known test IDs — and diffing each run's collected tests against that inventory. The gap is your blind spot. This is architecturally similar to the problem of incomplete span coverage in passing traces: the absence of instrumentation looks like health, not silence.

Building a Test Inventory and Execution Gap Dashboard

Start by separating collection from execution. In Pytest, --collect-only emits every discovered test without running them. Run this as a pre-step in your GitHub Actions workflow and store the output:

# .github/workflows/ci.yml (relevant steps)
- name: Collect test inventory
  run: |
    pytest --collect-only -q 2>/dev/null \
      | grep "::" \
      | sort > artifacts/test_inventory.txt

- name: Run tests
  run: pytest --junitxml=artifacts/results.xml

- name: Upload artifacts
  uses: actions/upload-artifact@v4
  with:
    name: test-artifacts
    path: artifacts/

Now you have two lists per run: what should have run and what did run. Parse the JUnit XML to extract executed test IDs and diff them. The following Python snippet writes the gap to a structured log that Loki or Datadog Logs can ingest:

import xml.etree.ElementTree as ET, json, sys

inventory = set(open("artifacts/test_inventory.txt").read().splitlines())
tree = ET.parse("artifacts/results.xml")
executed = {
    f"{tc.get('classname')}::{tc.get('name')}"
    for tc in tree.iter("testcase")
}

missing = inventory - executed
print(json.dumps({
    "run_id": sys.argv[1],
    "expected": len(inventory),
    "executed": len(executed),
    "missing_count": len(missing),
    "missing_tests": sorted(missing)[:50]   # cap log size
}))

Feed that JSON into ClickHouse or BigQuery alongside your normal JUnit results. A simple Grafana panel query against ClickHouse surfaces the trend:

-- Grafana panel: execution gap over time
SELECT
    toDate(run_at)        AS day,
    avg(expected)         AS avg_expected,
    avg(executed)         AS avg_executed,
    avg(missing_count)    AS avg_gap
FROM ci_test_runs
WHERE run_at >= now() - INTERVAL 30 DAY
GROUP BY day
ORDER BY day

When we wired this dashboard at a mid-size fintech, the gap chart immediately revealed that a tagging refactor three weeks earlier had silently dropped 47 integration tests from every PR run. Triage time for the eventual production incident dropped from over 30 minutes to under 5 once the execution gap was a named, visible metric — because engineers could see exactly which tests hadn't run rather than hunting through logs. The same pattern applies to how failure rate can fall when tests are added to a suite: the denominator changes, and the metric looks better while coverage may not be.

Pitfalls in CI Test Failure Triage When Counts Look Fine

The most common mistake is trusting the test count on the summary line rather than diffing against a baseline. CI dashboards — Allure, ReportPortal, GitHub's built-in test summary — all display the count of tests in the report. If your inventory shrinks because a developer adds @pytest.mark.skip to an entire module before a deadline, the count drops, the pass rate stays at 100%, and no alert fires. Use Allure when you need rich per-test history and attachments for manual triage; use ReportPortal when you need cross-project trend aggregation at org scale — but neither will catch a shrinking inventory unless you build the diff check yourself.

A second failure mode is dynamic test ID instability. Parameterized tests in Pytest or data-driven tests in TestNG generate IDs from input data. If the input dataset changes, test IDs change, and your inventory diff produces false positives — tests that look "missing" but are actually renamed. Stabilize IDs by hashing input parameters into a deterministic suffix, or maintain a separate alias map. Without this, teams disable the gap check because it's too noisy, which removes the only signal that matters.

Myths That Keep Execution Gaps Invisible

Myth 1: A green run means coverage is intact. Pass rate measures the ratio of failures to executed tests — it says nothing about unexecuted tests. A suite that ran 200 of 300 tests with zero failures has a 100% pass rate and a 33% execution gap. These are orthogonal metrics. Myth 2: Skips are intentional and documented. In practice, skips accumulate. A skip added for a flaky environment condition in Q1 is still there in Q3, now covering infrastructure that was fixed months ago. Treat skip count as a metric with an SLO, not a free escape hatch. Flake rate averages have the same problem — aggregate numbers obscure the specific tests that carry the most risk.

Myth 3: The test dashboard will surface this automatically. Dashboards reflect the data they receive. If your pipeline never writes "expected test count" to the results store, no dashboard can show the gap. The instrumentation has to come first — the visualization is downstream. Most embedded CI failure analysis tools (Datadog CI Visibility, BuildPulse, Trunk Flaky Tests) operate on the JUnit XML you give them. Garbage in, blind spot out.

Zero-failure runs are worth celebrating only when you can also show that the expected tests executed. The instrumentation is a two-step addition to any pipeline: collect the inventory before the run, diff it against the JUnit output after. Store the gap as a time-series metric, put it on the same CI test dashboard as pass rate and P95 duration, and set an alert when the gap exceeds your threshold. Start with --collect-only in Pytest or the equivalent dry-run flag in your runner, and treat any unexplained shrinkage in executed count as a severity-2 finding.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.