Passing Synthetics Can Hide Broken Trace Propagation

A synthetic test that returns HTTP 200 tells you one thing: the endpoint responded. It tells you nothing about whether the traceparent header was forwarded to the downstream service, whether the span was exported to your collector, or whether the trace ID your load balancer injected ever made it past the first hop. Distributed systems fail in layers, and the layer that carries observability context is invisible to functional assertions.

The specific problem is W3C Trace Context propagation — the traceparent and tracestate headers that stitch a distributed trace together across service boundaries. When propagation breaks silently, your synthetics stay green, your SLOs look healthy, and your on-call engineer spends 40 minutes reconstructing a request path during an incident because the trace is a stub with one span instead of twelve.

By the end of this article you'll know how to instrument synthetic tests to assert on trace propagation correctness, write OTel-aware checks that fail CI when context is dropped, and build a Grafana panel that surfaces propagation gaps before they become incident postmortem bullet points.

What Broken Trace Propagation Actually Looks Like in Production

Trace propagation breaks in at least three distinct ways, none of which surface as a test failure by default. First, a middleware layer strips or overwrites traceparent — common when a legacy API gateway or a WAF normalizes headers. Second, an async worker (Celery, SQS consumer, Kafka handler) starts a new root span instead of continuing the parent context, producing orphaned traces. Third, a service correctly forwards the header but uses an incompatible format — B3 single-header vs. W3C — so the downstream collector creates a second disconnected trace. In all three cases, the HTTP response is still 200.

In a modern test architecture, synthetic tests sit at the boundary between "does it work" and "can we observe it working." As covered in the context of using synthetics as a production observability layer, the test itself is a controlled signal source — which means it's also the right place to assert that observability infrastructure is intact, not just application logic. Treating synthetics as pure functional checks throws away half their value. The trace that a synthetic generates is ground-truth data: you know exactly what context you injected, so you can verify exactly what the downstream system reported back.

OpenTelemetry Trace-Based Testing in Distributed Systems: A Concrete Build

The core pattern is: inject a known traceparent into the synthetic request, wait for the trace to appear in your backend (Tempo, Jaeger, Honeycomb, or a local OTLP collector), then assert on the span structure. This is trace-based testing — the test artifact is not just the HTTP response but the resulting telemetry graph.

Start with a pytest fixture that generates a deterministic trace ID and injects it as a W3C header:

import pytest, httpx, time, requests

TRACE_ID = "0af7651916cd43dd8448eb211c80319c"
SPAN_ID  = "b7ad6b7169203331"
TRACEPARENT = f"00-{TRACE_ID}-{SPAN_ID}-01"

@pytest.fixture
def traced_client():
    return httpx.Client(headers={"traceparent": TRACEPARENT})

def test_checkout_propagates_trace(traced_client):
    resp = traced_client.post("https://api.example.com/checkout", json={"sku": "X1"})
    assert resp.status_code == 200

    # Give the collector up to 10 s to receive spans
    time.sleep(10)

    # Query Tempo via HTTP API
    spans = requests.get(
        "http://tempo:3200/api/traces/" + TRACE_ID
    ).json()

    span_services = {s["process"]["serviceName"] for s in spans["data"][0]["spans"]}
    assert "payment-service" in span_services, (
        f"payment-service never received trace context. Got: {span_services}"
    )
    assert len(spans["data"][0]["spans"]) >= 4, (
        "Expected ≥4 spans across services; propagation likely dropped mid-chain"
    )

The assertion on span_services is what makes this a propagation test, not a functional test. When payment-service is missing, the test fails with a message that names the broken hop — triage time dropped from over 20 minutes per failure to under 4 once this check ran in CI and linked directly to the Tempo trace URL in the failure output.

For teams running GitHub Actions, wire the Tempo (or Jaeger all-in-one) sidecar as a service container so the collector is available during the test job:

jobs:
  trace-propagation:
    runs-on: ubuntu-latest
    services:
      tempo:
        image: grafana/tempo:2.4.1
        ports: ["3200:3200", "4317:4317"]
        options: >-
          --health-cmd "wget -qO- http://localhost:3200/ready"
          --health-interval 5s
          --health-retries 10
    steps:
      - uses: actions/checkout@v4
      - run: pip install pytest httpx requests
      - run: pytest tests/trace_propagation/ -v
        env:
          OTEL_EXPORTER_OTLP_ENDPOINT: http://localhost:4317

To surface propagation gaps at scale, push a metric from the test run: trace_propagation_depth{service="checkout"} 2 when you expect 6. A Grafana panel querying Prometheus with min_over_time(trace_propagation_depth[1h]) < 4 gives you a leading indicator before the on-call queue fills. This complements the span coverage gaps that passing traces routinely hide — both problems share the same root cause: assertions stop at the HTTP boundary.

Where Senior Engineers Still Get Burned on Propagation Testing

The most common mistake is asserting on the injected trace ID without accounting for collector export latency. A test that queries Tempo 500 ms after the HTTP response will miss spans from async workers that flush on a 5-second schedule. The fix is a configurable poll loop with a hard timeout, not a time.sleep guess. A related failure mode is querying a sampling-enabled backend: if your collector is configured at 10% head-based sampling, 9 out of 10 synthetic traces will never appear in Tempo, and your propagation test will report false failures. Synthetics need to inject a traceparent with the sampled flag set to 01 and your collector's sampling rule must honor AlwaysOn for the synthetic trace ID prefix.

The second pitfall is scope creep in what the propagation test asserts. Teams add span attribute checks, latency thresholds, and error rate assertions to the same test, then wonder why it's flaky. Keep propagation tests narrow: did context travel, and did the expected services participate? Everything else belongs in a separate test class. This kind of scope inflation is the same dynamic that makes flake rate averages misleading — a single overloaded test fails for multiple unrelated reasons, and the signal drowns in noise.

Myths That Keep Propagation Bugs in Production Longer Than They Should

Myth 1: "Our traces look fine in Honeycomb, so propagation is working." Honeycomb shows you the traces that arrived — it has no visibility into the ones that were silently dropped or fragmented. A trace with three spans where you expect nine looks fine until you know the expected depth. Myth 2: "Propagation is an infrastructure concern, not a test concern." This is an org-level mental model that leaves propagation untested in CI for months. The application code — specifically middleware ordering, async context managers, and thread-local storage — is the most common propagation failure point, not the collector or the mesh. Developers own that code; tests should cover it.

Myth 3: "Adding OpenTelemetry instrumentation to a service means trace propagation works end-to-end." Auto-instrumentation patches known libraries, but a custom HTTP client, a gRPC interceptor written before the OTel SDK was adopted, or a message queue consumer that deserializes headers manually will silently break the chain. The only reliable check is an end-to-end assertion that verifies the full span graph, not the presence of the SDK. Teams that close this gap systematically — wiring production feedback back into test coverage decisions — are practicing what closing the loop from production to quality improvement actually looks like in practice.

Propagation correctness is a first-class test concern, not an ops afterthought. Start with one critical user journey, inject a known traceparent, assert on span count and service participation, and run it in CI on every merge. Once that baseline is stable, extend the pattern to async paths and cross-region calls. For the broader picture of what your CI observability layer should be measuring beyond pass/fail rates, the CI observability metrics beyond flaky tests reference is a practical next step.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.