Span Timing Gaps Hide Latency Regressions

A green test suite is not a performance guarantee. A test that asserts response.status == 200 and len(results) > 0 will pass whether the endpoint answered in 40 ms or 4 seconds — and most assertion libraries don't care. The latency regression ships, production P95 climbs, and the postmortem eventually asks why no test caught it. The answer is almost always the same: the test measured outcome, not duration, and the span data that could have told the story had gaps nobody was watching.

The specific failure mode is span timing gaps — intervals between the end of one traced operation and the start of the next that inflate wall-clock duration without triggering any assertion. They hide in serialization waits, connection pool exhaustion, middleware chains, and async handoffs. Because the business logic still completes correctly, the test passes. Because the gap lives between spans rather than inside one, it rarely shows up in a single trace waterfall unless you're looking for it.

By the end of this article you'll know how to instrument your test runs with OpenTelemetry to surface inter-span gaps, write queries against that data in ClickHouse or Honeycomb, and build alerting thresholds that catch latency regressions before they reach production — without rewriting your existing test assertions.

What Span Timing Gaps Actually Are and Why Tests Miss Them

A span timing gap is the wall-clock duration between span_A.end_time and span_B.start_time for two consecutive spans in the same trace. OpenTelemetry records both timestamps with nanosecond precision, but nothing in the spec requires that gap to be zero — or even small. A 200 ms gap between a DB query span ending and a serialization span starting means the application spent 200 ms doing something untraced: acquiring a lock, waiting on a thread pool, or sitting in a queue. The trace looks complete; the gap is invisible unless you compute it explicitly.

In a test context, this matters because latency, throughput, and stability are three separate signals, and standard test assertions only touch the first one indirectly — usually by checking a timeout that's set far too generously. A test with a 10-second timeout on an endpoint that regressed from 80 ms to 800 ms will pass indefinitely. The gap accounting that would reveal the regression exists in the span data, but only if you're instrumenting test execution as a first-class trace and querying inter-span durations rather than total span durations.

Instrumenting Tests and Querying for Inter-Span Gaps

Start by wrapping your test execution in an OpenTelemetry tracer so each test case becomes a root span, and each major operation (HTTP call, DB query, cache lookup) becomes a child span. With Pytest and the opentelemetry-sdk package, a conftest fixture handles this cleanly:

# conftest.py
import pytest
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317")))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("test.suite")

@pytest.fixture(autouse=True)
def trace_test(request):
    with tracer.start_as_current_span(request.node.nodeid) as span:
        span.set_attribute("test.suite", request.node.module.__name__)
        yield span

This gets every test into your collector. The next step is computing gaps. If you're exporting to ClickHouse (a common choice for high-volume span data), this query finds the top inter-span gaps per trace, ordered by gap size descending:

-- ClickHouse: inter-span gap analysis
SELECT
    trace_id,
    parent_span_id,
    SpanName AS span_name,
    toUnixTimestamp64Nano(Timestamp) AS start_ns,
    toUnixTimestamp64Nano(Timestamp) + (Duration * 1000) AS end_ns,
    (toUnixTimestamp64Nano(Timestamp)
        - lagInFrame(toUnixTimestamp64Nano(Timestamp) + (Duration * 1000))
          OVER (PARTITION BY trace_id ORDER BY Timestamp)) AS gap_ns
FROM otel_traces
WHERE ServiceName = 'api-service'
  AND toDate(Timestamp) = today()
HAVING gap_ns > 50000000  -- 50 ms threshold
ORDER BY gap_ns DESC
LIMIT 100;

The lagInFrame window function computes the gap between each span's start and the previous span's end within the same trace. Any row with gap_ns > 50000000 (50 ms) is a candidate for investigation. In one platform team's integration suite, wiring this query into a Grafana panel backed by a ClickHouse datasource dropped triage time from 22 minutes per latency failure to under 4 — because engineers could immediately see where in the trace the time was going rather than bisecting logs manually.

For teams on Honeycomb, the equivalent is a derived column and a BubbleUp query. Create a derived column inter_span_gap_ms using INTERVAL(span.start_time_unix_nano - LAG(span.end_time_unix_nano)) grouped by trace.trace_id, then set a SLO trigger on P95 of that column exceeding your baseline. The alert fires on the shape of the gap distribution, not just individual outliers — which is where CI observability metrics beyond raw pass/fail start earning their keep. Pair this with a Grafana alert rule that pages only when the 7-day rolling P95 gap exceeds 1.5× the 30-day baseline, and you get signal without noise.

Where Experienced Teams Still Get the Instrumentation Wrong

The most common mistake is sampling away the evidence. Head-based sampling at 10% is standard for production cost control, but it means 90% of your test-execution traces — including the ones with the worst gaps — never reach storage. For test runs, use a tail-based sampler configured to keep 100% of traces where total duration exceeds your P90 baseline. The volume is manageable (test suites are not production traffic), and you need the outliers, not the median. Dropping them is why teams spend hours in postmortems with no trace to show.

The second mistake is attributing gap time to the wrong span. When engineers see a slow trace, they usually blame the longest span. But a 400 ms gap between a cache-miss span and the subsequent DB query span is not the DB's fault — it's the application's scheduling or connection pool. Fixing the DB query won't help. This mental-model error persists because most trace UIs render span duration, not inter-span gaps, as the primary visual. Build a dedicated gap-analysis panel rather than relying on the waterfall view to surface it organically. It won't.

Myths That Let Latency Regressions Ship Behind Green Tests

Myth 1: A passing test with a loose timeout is a latency test. It isn't. A 10-second timeout on a 100 ms endpoint is a liveness check, not a performance assertion. Real latency coverage requires asserting against a percentile budget — assert p95_ms < 200 — derived from the span data of the test run itself, not a hardcoded constant set two years ago. Myth 2: Total span duration is the right metric. It's one metric. A span that takes 600 ms because of three 200 ms gaps between 1 ms child spans is a completely different problem than a span that takes 600 ms because a single DB query is slow. Treating them identically leads to wrong fixes. The gap distribution is a separate signal from span duration and needs its own threshold and alert.

Myth 3: Flaky tests and latency regressions are separate problems. They often share a root cause — resource contention, connection pool limits, or serialization bottlenecks that manifest as timing variance. A test that sometimes fails a timeout assertion and sometimes doesn't is exhibiting the same gap instability discussed here, just past a threshold. If you're already tracking which tests carry the most release risk, cross-reference them with high inter-span gap variance — the overlap is rarely coincidental, and fixing the gap usually stabilizes the flake.

Span timing gaps are a measurement gap first and a code problem second. Before optimizing anything, instrument your test traces with OpenTelemetry, compute inter-span gaps with the ClickHouse or Honeycomb queries above, and set P95 thresholds that reflect your actual SLOs — not arbitrary timeouts. From there, closing the loop from production latency signals back to test coverage is the natural next step: production traces tell you where gaps appear under real load; test traces tell you whether your fixes hold in CI before the next deploy.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.