Passing Synthetics Can Hide Degraded Trace Depth

Most teams treat a green synthetic as proof that a service is healthy. The check passed, the SLA timer reset, the on-call engineer went back to sleep. What that green check doesn't capture is whether the trace behind that request contained 14 spans or 3 — and whether the missing 11 represent a caching layer that silently stopped participating, a downstream gRPC call that got swallowed by a timeout, or a database query that was never instrumented after the last refactor. The synthetic passed because the response code was 200. The system was already degraded.

This is the gap between synthetic correctness and observability completeness. A synthetic test validates the surface contract — status code, response body, latency threshold. It says nothing about whether the distributed execution path that produced that response is intact, healthy, or even the same path it was last week. Trace depth — the number of spans, the presence of expected service nodes, the propagation of context headers across process boundaries — is a separate signal that most synthetic frameworks don't assert on at all.

By the end of this article you'll know how to instrument your synthetic layer to assert on trace structure, write span-count queries against a ClickHouse or Tempo backend, and build alerting that fires when trace depth degrades even when your HTTP checks stay green.

What Trace Depth Actually Measures in a Distributed System

Trace depth is the count of spans within a trace, often measured as the number of unique service nodes and the total span count per root transaction. In a well-instrumented system, a checkout request might consistently produce 18–22 spans across 6 services. When that number drops to 9 without a corresponding code change, something structurally changed: a middleware stopped propagating the traceparent header, a sidecar proxy was updated and dropped the B3 headers it used to forward, or an async worker decoupled from the trace context entirely. None of those events necessarily break the HTTP surface contract.

This matters for OpenTelemetry trace-based testing in distributed systems because the trace is the ground truth of what actually executed. Synthetic tests that only assert on HTTP semantics are checking the envelope, not the letter. When you treat synthetic tests as a production observability layer, you need to assert on the trace structure they produce — not just whether the endpoint responded. Trace depth regression is one of the earliest indicators of instrumentation drift, dependency degradation, and silent partial failures, all of which will eventually show up in your P95 latency or error budget before they appear in your pass/fail rate.

Asserting on Span Count and Trace Structure in CI and Production

The first step is capturing the trace ID produced by your synthetic and querying it back from your tracing backend. With OpenTelemetry and Tempo, you can inject a known X-Trace-Id header, wait for the trace to flush (typically 5–15 seconds), and then query the Tempo HTTP API to validate structure:

# Query Tempo for a specific trace and count spans
TRACE_ID="4bf92f3577b34da6a3ce929d0e0e4736"
curl -s "http://tempo:3200/api/traces/${TRACE_ID}" \
  | jq '[.batches[].scopeSpans[].spans[]] | length'
# Expected: >= 18. If < 10, alert.

Wire this into a Pytest fixture that runs post-synthetic and fails the test when span count falls below a configured floor. The threshold isn't arbitrary — baseline it from 30 days of trace data before you set it. In ClickHouse, where many teams store OTEL spans at scale, the baselining query looks like this:

-- ClickHouse: P5 span count per root operation over 30 days
SELECT
  SpanName,
  quantile(0.05)(span_count) AS p5_floor,
  quantile(0.50)(span_count) AS p50,
  quantile(0.95)(span_count) AS p95
FROM (
  SELECT
    TraceId,
    SpanName,
    count() AS span_count
  FROM otel_traces
  WHERE ParentSpanId = ''
    AND Timestamp >= now() - INTERVAL 30 DAY
  GROUP BY TraceId, SpanName
)
GROUP BY SpanName
ORDER BY SpanName;

Set your synthetic assertion floor at the P5 value — you're not chasing the happy path, you're catching structural collapse. Once you have floors per operation, you can add a Grafana alert panel that fires when rolling P50 span count drops more than 20% below the 30-day baseline, even while your HTTP synthetic stays green. One team using this approach on a payment service found that a misconfigured Envoy filter had been silently dropping trace context propagation for 11 days — the synthetic passed every 60 seconds the entire time, but the span count had dropped from 21 to 4. Triage time dropped from 22 minutes per failure to under 4 once the span-count alert was wired to the same Slack channel as the synthetic alert.

For GitHub Actions CI, add a post-test step that queries Tempo or Jaeger after your integration suite runs and fails the workflow if span floors aren't met:

# .github/workflows/trace-depth-check.yml (partial)
- name: Assert trace depth
  run: |
    python scripts/assert_trace_depth.py \
      --tempo-url $TEMPO_URL \
      --operation "POST /checkout" \
      --min-spans 15 \
      --trace-id ${{ env.LAST_TRACE_ID }}
  env:
    TEMPO_URL: ${{ secrets.TEMPO_URL }}

The Python script fetches the trace, counts spans, and exits non-zero if the count is below the floor. This makes trace depth a first-class CI gate, not an afterthought in your CI observability metrics dashboard.

Where Teams Break Trace Depth Assertions in Practice

The most common mistake is asserting on span count without accounting for sampling. If your production tracing runs at 10% head-based sampling and your synthetic runs against the same pipeline, 9 out of 10 traces will have zero spans — not because the system is broken, but because they were dropped at the collector. The fix is to force 100% sampling for synthetic-originated requests using a dedicated traceparent flag or a Collector processor rule that matches a synthetic-specific header and sets the sampling decision to 01. Skipping this step produces noisy, unreliable span-count assertions that engineers quickly learn to ignore.

The second mistake is treating span count as the only structural signal. A trace can have 20 spans and still be missing the db.query spans from your primary datastore — because the ORM was updated and the instrumentation library version wasn't pinned. Assert on span presence by name or service, not just total count. A simple JSONPath check against the Tempo response for $.batches[*].resource.attributes[?(@.key=="service.name")].value.stringValue will tell you which services actually participated. Missing a service node is a more precise signal than a raw count drop, and it points directly at the component that lost instrumentation.

The Myths That Let Trace Depth Regressions Go Undetected

Myth 1: If the synthetic passes, the system is observable. Observability is a property of your instrumentation coverage, not your test results. A passing synthetic tells you the contract held; it says nothing about whether you could reconstruct the execution path if something went wrong. This is the same category of error as confusing a falling failure rate with improving quality — the metric improved, but the underlying signal got diluted. Trace depth is the instrumentation analog: the test passed, but the observability surface shrank.

Myth 2: Trace depth is a DevOps concern, not a testing concern. This org-level mental model is why the problem persists. SDETs own the synthetic layer. Platform engineers own the collector and backend. Neither team asserts on the intersection. The result is that instrumentation drift — a span that disappears after a library upgrade, a service that stops propagating context after a sidecar change — goes undetected for weeks. Closing that gap requires closing the loop between production signals and test strategy, which means SDETs need to own span-count assertions the same way they own response-time assertions. Trace structure is a testable contract.

Passing synthetics and healthy traces are not the same thing — and conflating them is how instrumentation regressions hide for weeks. Start by baselining your P5 span count per root operation in ClickHouse or Tempo, set assertion floors in your synthetic framework, and add a span-presence check for your critical service nodes. If you're already running OpenTelemetry, the data is there; you're just not asserting on it yet. That's the gap worth closing first.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.