Passing Traces Hide Incomplete Span Coverage

A green trace feels like confirmation. The request completed, the test passed, the dashboard shows no errors. But a trace that finishes without errors and a trace that captured every meaningful unit of work are two very different things. Span coverage — the proportion of your code paths that actually emit spans during a test run — can degrade silently while your pass rate stays at 100%.

The technical problem is subtle: instrumentation is additive and fragile. A library upgrade drops an auto-instrumentation hook. A new async code path spawns goroutines or threads that never inherit the parent context. A feature flag routes traffic through an uninstrumented branch. In every case, the trace still closes, the test still passes, and the gap becomes invisible until a production incident forces you to reconstruct what actually happened.

By the end of this article you will know how to measure span coverage as a first-class metric, write assertions that fail a build when coverage drops, and avoid the org-level patterns that let these gaps accumulate undetected.

What Span Coverage Actually Measures (and Why It's Not Trace Count)

Span coverage is the ratio of expected instrumentation points to observed spans within a trace for a given code path. It is distinct from trace volume (how many traces you collected) and from error rate (how many spans finished with a non-OK status). A system can emit millions of traces with zero errors and still have 40% of its critical paths dark — never emitting a span — because the instrumentation was never written, was dropped during a refactor, or was swallowed by a context propagation failure.

In a modern test architecture, span coverage sits at the intersection of testing with observability and contract testing. You are asserting not just that the system behaved correctly, but that it remained observable while doing so. This distinction matters most during performance testing with traces: a k6 load test that drives 10k RPS through an uninstrumented checkout path gives you throughput numbers but zero latency breakdown — exactly the data you need when something degrades under load.

Building Span Coverage Assertions Into Your CI Pipeline

The first step is defining a span manifest — a machine-readable list of span names you expect to see for each user journey or test scenario. Store it in version control alongside the tests that drive those journeys. A minimal format is a JSON file keyed by test ID:

{
  "checkout_happy_path": [
    "http.server /api/checkout",
    "db.query insert_order",
    "queue.publish order_created",
    "payment.charge stripe_authorize"
  ],
  "checkout_retry_on_timeout": [
    "http.server /api/checkout",
    "db.query insert_order",
    "payment.charge stripe_authorize",
    "payment.charge stripe_authorize"
  ]
}

After each test run, query your trace backend and compare observed spans against the manifest. The following Python snippet targets the Tempo HTTP API but the pattern is identical for Jaeger or Honeycomb:

import requests, json, sys

MANIFEST = json.load(open("span_manifest.json"))
TEMPO_URL = "http://tempo:3200"

def get_spans(trace_id):
    r = requests.get(f"{TEMPO_URL}/api/traces/{trace_id}")
    r.raise_for_status()
    return [s["operationName"] for s in r.json()["data"][0]["spans"]]

def assert_coverage(test_id, trace_id):
    expected = MANIFEST[test_id]
    observed = get_spans(trace_id)
    missing = [s for s in expected if s not in observed]
    if missing:
        print(f"FAIL [{test_id}] missing spans: {missing}")
        sys.exit(1)

assert_coverage("checkout_happy_path", os.environ["TRACE_ID"])

Wire this into GitHub Actions as a post-test step so a coverage regression blocks the merge, not just the functional assertion:

- name: Assert span coverage
  run: python scripts/assert_span_coverage.py
  env:
    TRACE_ID: ${{ steps.run-tests.outputs.trace_id }}
    TEMPO_URL: ${{ secrets.TEMPO_URL }}

One team running this pattern against a Node.js monorepo found that a Prisma version bump had silently dropped all db.query spans from their ORM layer — every test passed, but three weeks of traces were missing database timing entirely. Once the assertion was in place, the same class of regression was caught at PR time. Triage time for span-gap incidents dropped from roughly 25 minutes of manual trace comparison to under 3 minutes. For parallel test runs, note that context propagation failures compound — trace coverage drops when tests run in parallel due to shared context objects being mutated across goroutines or threads, so your manifest assertions should run per-worker, not against a merged trace.

Where Logs and Traces in Test Automation Break Down

The most common mistake is treating instrumentation as a one-time setup task rather than a living contract. Teams add OpenTelemetry auto-instrumentation at project start, ship it, and never revisit it. When a new gRPC service gets added, or a background worker is extracted into a separate process, the new code paths are dark by default. Nobody notices because the traces that do exist look complete — they just don't cover the new paths. The fix is a coverage gate in CI, not a quarterly audit.

A second failure mode is conflating log presence with span coverage. Logs and traces serve different purposes: a log line confirms something happened; a span captures duration, parent context, and propagation chain. Teams that instrument heavily with structured logging via Loki or Datadog Logs often assume they have equivalent observability. They don't. A missing span breaks distributed context — you lose the ability to correlate a slow downstream call back to the originating request. Failure localization in CI depends on that correlation chain being intact; without spans, you're back to grep-and-guess across log timestamps.

What Testing Traces Teams Get Wrong About "Passing" Observability

The dominant myth is that a passing trace is a complete trace. It isn't. A trace closes when the root span closes — not when every expected child span has been emitted. If a context propagation bug causes a downstream call to start a new root instead of a child span, you get two "passing" traces where you should have one, and the parent-child relationship — the one you need for latency attribution — is gone. This is why span timing gaps can hide latency regressions even when every individual trace shows a clean status. Pass/fail on the trace is necessary but not sufficient.

A related misconception is that high span volume implies good coverage. Volume is a function of traffic; coverage is a function of code path instrumentation. A checkout flow that handles 50k requests per day but never instruments its fraud-check microservice has zero coverage for that path regardless of volume. The same logic applies to synthetic monitoring: a synthetic that hits your API endpoint and gets a 200 back tells you the happy path is up, but it says nothing about whether the three internal services that served that response are emitting spans with the right attributes. Treating synthetic pass rates as a proxy for trace coverage is how teams end up blind to degradation in exactly the services they care about most.

Span coverage is a metric your pipeline should enforce, not a property you infer from a green dashboard. Start by building a span manifest for your two or three most critical user journeys, add a post-test assertion step in CI, and track coverage percentage as a time-series metric in Grafana or Datadog alongside your pass rate. When the two diverge, you have found a real signal. For a broader framework on what to measure beyond pass rate, the piece on coverage as a vanity metric is a useful next read.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.