Grafana Tempo for Tracing Slow Test Runs

Most teams know a test suite is slow. They don't know why. Wall-clock time on a CI job tells you the symptom; it says nothing about whether the P95 slowdown lives in a flaky DB fixture, a retry storm on an external HTTP dependency, or a test runner that serializes teardown across 40 workers. The difference between a 4-minute suite and a 9-minute suite is often three spans buried in a trace nobody thought to capture.

Grafana Tempo is a distributed tracing backend designed for high-volume, low-cost trace storage — no indexing on every attribute by default, which makes it cheap enough to run against test infrastructure without a separate budget conversation. When you instrument your test runner with OpenTelemetry and ship spans to Tempo, a slow run stops being a mystery and becomes a flame graph you can share in a Slack thread.

By the end of this article you'll have a working pattern for instrumenting Pytest or JUnit-based suites with OTEL spans, routing them into Tempo, and building a Grafana panel that surfaces P95 test duration by span name — so the next time a run blows the SLA, you find the culprit in under two minutes.

What Tempo Actually Adds to a Test Observability Stack

Tempo stores traces as objects in an object store (S3, GCS, or local disk) and resolves them by trace ID. It does not index span attributes by default, which is the key architectural difference from Jaeger or Zipkin. That trade-off means storage costs stay flat even at high trace volume, but it also means you need a correlated signal — typically a Loki log line or a Prometheus metric — to find the trace ID before you can drill in. For test infrastructure this is rarely a problem: your CI job already emits a run ID, and that run ID becomes the trace ID anchor.

In a modern test architecture, Tempo sits between your test runner and Grafana. The runner emits OTEL spans (one root span per test, child spans for fixtures, HTTP calls, DB queries), the OTEL Collector batches and forwards them to Tempo, and Grafana's Tempo data source lets you query by trace ID or use TraceQL to filter by duration, span name, or status. This is a different layer from distributed tracing for application failures — here the test itself is the service being traced, not just the system under test.

Instrumenting Pytest and Querying Tempo with TraceQL

Start with the OTEL SDK for Python and a conftest.py hook. The pattern below creates a root span per test item, attaches the test node ID as a span attribute, and records pass/fail as a span status. The OTEL Collector endpoint is injected via environment variable so the same conftest works locally and in GitHub Actions or Buildkite.

# conftest.py
import pytest
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
import os

provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint=os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"]))
)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("pytest.suite")

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_protocol(item, nextitem):
    with tracer.start_as_current_span(item.nodeid) as span:
        span.set_attribute("test.file", str(item.fspath))
        span.set_attribute("test.class", item.cls.__name__ if item.cls else "")
        outcome = yield
        rep = outcome.get_result() if not outcome.excinfo else None
        if outcome.excinfo:
            span.set_status(trace.StatusCode.ERROR, str(outcome.excinfo[1]))
        else:
            span.set_status(trace.StatusCode.OK)

Wire the OTEL Collector to Tempo in your docker-compose or Kubernetes config. The critical setting is receivers.otlp on the Collector and exporters.otlp pointing at Tempo's gRPC port (default 4317). Keep batch.send_batch_size at 512 or higher — test suites emit bursts, not a steady stream, and small batch sizes cause head-of-line blocking that inflates the spans you're trying to measure.

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"

exporters:
  otlp:
    endpoint: "tempo:4317"
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp]

Once spans are flowing, TraceQL is where Tempo earns its keep for test triage. The query below finds every trace where a test span exceeded 8 seconds in the last hour — the kind of query you'd run after a CI alert fires:

{ span.test.file =~ "tests/integration/.*" && duration > 8s }

Paste that into the Grafana Explore panel with the Tempo data source selected. You'll get a list of trace IDs with durations; click any one and the flame graph shows exactly which child span — fixture setup, an HTTP call, a DB migration — ate the time. One team running Playwright E2E tests against a staging environment cut triage time from 22 minutes per slow-run investigation to under 4 once this panel was wired up alongside their Loki-based log triage board. The Loki log line carries the trace ID as a structured field; clicking it jumps directly to the Tempo flame graph via Grafana's trace-to-logs correlation.

Where Engineers Waste Time Setting This Up

Instrumenting the system under test instead of the test runner. The most common mistake is adding OTEL to the application being tested and assuming you'll see test latency there. You won't — you'll see application latency. The test runner is the root service. If your fixture spins up a container and waits 6 seconds for it to be healthy, that 6 seconds lives in the fixture span, not in any application trace. Instrument both layers if you need full context, but start with the runner. This confusion is especially common when teams follow assertion latency patterns designed for production observability and try to apply them directly to test infrastructure.

Skipping the trace ID correlation between CI logs and Tempo. Tempo without a correlated log signal requires you to already know the trace ID — which you usually don't when you're starting a triage. Emit the root trace ID as a structured log field in your CI job output (a single logger.info("trace_id=%s", span.get_span_context().trace_id) in the conftest teardown is enough). Ship those logs to Loki with the same job labels your Grafana dashboards already use. Without this, Tempo becomes an island you can only reach if you already know where you're going.

Myths About Tracing Test Runs That Slow Teams Down

"Tracing is for microservices, not test suites." This conflates the mechanism with the use case. A Pytest session with 400 tests, parallel workers, shared fixtures, and external service calls is a distributed system in every meaningful sense. Spans capture causality and duration in ways that log lines and timing summaries cannot. Teams that treat test infrastructure as a second-class observability citizen end up with dashboards that show aggregate pass rates but can't answer "which fixture is slow on Tuesdays." If you're already evaluating CI visibility tooling options, Tempo gives you the trace depth that most SaaS CI platforms charge extra for.

"Flaky slow tests are a different problem from slow tests." Duration variance and flakiness are often the same root cause observed at different thresholds. A test that takes 1.2s normally and 9s occasionally isn't randomly flaky — it has a dependency with a long tail. Tempo's flame graphs make the tail visible. AI-based classifiers that look only at pass/fail history will frequently misread these as intermittent regressions; the trace data is what disambiguates. Parallel test execution compounds this: span context can be dropped or misattributed across workers, a problem covered in detail when examining why trace coverage drops in parallel runs. Fix the instrumentation before trusting the data.

The setup cost for Tempo on test infrastructure is low — a Collector sidecar, a conftest hook, and an S3 bucket. The operational payoff is a flame graph for every slow run instead of a wall-clock number and a guess. Start with one test file that's already known to be slow, instrument it, and build the TraceQL query that catches the offender. Once the pattern is proven, rolling it across the suite is a config change, not a rewrite.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.