CI Failure Localization: Logs, Traces & Triage

Most teams treat a red CI run as a binary event: find the failing test, re-run it, ship or block. The expensive part isn't the failure — it's the 20-minute archaeology session that follows. An engineer opens the pipeline, scrolls through 4,000 log lines, opens a second tab for the trace, opens a third for the prior benchmark, and eventually triangulates a root cause that was sitting in plain sight the whole time. That friction compounds across every engineer, every sprint.

The technical problem is failure localization: given a failing test in CI, pinpoint the exact code path, log event, or performance regression responsible — without manual log spelunking. It requires stitching together three data streams that most teams keep siloed: structured CI logs, distributed execution traces, and benchmark baselines.

By the end of this article you'll have a concrete pipeline for correlating those three streams, a SQL schema for storing localization events in ClickHouse or PostgreSQL, and a Grafana panel query that surfaces the signal your on-call engineer actually needs during triage.

What Failure Localization Actually Means in a CI Context

Failure localization is the practice of narrowing a test failure to its proximate cause — not just the assertion that fired, but the log event, span, or metric deviation that preceded it. In a distributed system under test, the assertion is almost always a lagging indicator. The real signal is a 400ms latency spike in the auth service three seconds before the timeout, or a missing log line that should have confirmed a Kafka message was consumed.

In a modern test architecture, localization sits between raw test execution and structured failure analysis. It feeds the triage dashboard, informs failure sequencing logic that separates infra flakes from logic flakes, and provides the evidence layer that AI classifiers need to avoid misattribution. Without it, you're doing triage on symptoms; with it, you're debugging causes.

Building an Embedded CI Failure Analysis Pipeline

The foundation is a trace-aware test runner. Instrument your Pytest or JUnit suite with OpenTelemetry so every test case emits a root span. Set the span name to the test node ID and attach the CI run ID, git SHA, and branch as resource attributes. Ship spans to a collector that fans out to both Honeycomb (for ad-hoc trace queries) and a local ClickHouse table for batch analytics.

# conftest.py — OpenTelemetry span per test (Pytest + OTEL SDK 1.24)
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
import pytest, os

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("ci.test_runner")

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_call(item):
    with tracer.start_as_current_span(item.nodeid) as span:
        span.set_attribute("ci.run_id", os.environ.get("CI_RUN_ID", "local"))
        span.set_attribute("git.sha", os.environ.get("GIT_SHA", "unknown"))
        outcome = yield
        if outcome.excinfo:
            span.set_status(trace.StatusCode.ERROR, str(outcome.excinfo[1]))
            span.record_exception(outcome.excinfo[1])

Next, correlate spans with CI log lines. In GitHub Actions, set ACTIONS_STEP_DEBUG=true and stream logs to Loki with the Grafana Agent. Tag every log line with ci_run_id and test_node_id using a Loki pipeline stage. Now a single LogQL query can pull every log line that overlapped with a failing span's time window — no manual scrolling.

# Loki pipeline stage in grafana-agent.yaml
logs:
  positions_directory: /tmp/loki-positions
  configs:
    - name: ci_runner
      pipeline_stages:
        - json:
            expressions:
              test_node_id: test_node_id
              ci_run_id: ci_run_id
        - labels:
            test_node_id:
            ci_run_id:

For benchmark regression detection, store P50/P95 duration per test node ID in ClickHouse. The query below flags any test whose latest run exceeded its 30-day P95 by more than 20% — a reliable signal for performance-induced flakiness before it becomes an outright failure.

-- ClickHouse: benchmark regression detection
SELECT
    test_node_id,
    run_id,
    duration_ms,
    quantileIf(0.95)(duration_ms, run_date >= today() - 30) OVER (PARTITION BY test_node_id) AS p95_30d,
    duration_ms / p95_30d AS ratio
FROM test_executions
WHERE run_date = today()
  AND ratio > 1.20
ORDER BY ratio DESC
LIMIT 50;

Wire all three streams into a Grafana dashboard using a mixed datasource panel. One row shows the Loki log volume for the failing test's time window; the second row overlays the OTEL trace waterfall from Tempo; the third shows the ClickHouse benchmark ratio trend. Triage time dropped from 22 minutes per failure to under 4 once this dashboard replaced the three-tab archaeology workflow — the engineer sees cause, context, and history in a single scroll. This kind of observability-driven test analysis turns raw pipeline output into actionable engineering signal.

Where CI Triage Pipelines Break Down in Practice

The most common mistake is correlating on wall-clock time instead of trace context. Log lines are timestamped by the agent, not by the test runner, and clock skew between a GitHub Actions runner and your Loki ingestor can be 200–800ms — enough to miss the exact log event that preceded a failure. Always propagate the W3C traceparent header into your log fields and join on trace ID, not timestamp range. This is an org-level problem as much as a tooling one: teams that own the log pipeline and teams that own the test runner rarely talk.

The second pitfall is storing raw JUnit XML as the canonical record. JUnit XML was designed for pass/fail reporting, not localization. It has no concept of spans, no structured error payload, and no benchmark baseline. Teams that treat it as their source of truth end up rebuilding context from scratch on every incident. Parse JUnit XML at ingest, extract failure messages and durations into a structured schema, and discard the XML. The failure rate can recover while the underlying defect persists — a structured schema makes that pattern visible; raw XML buries it.

Myths That Keep Test Triage Slow

Myth 1: The stack trace is the root cause. In integration and E2E suites, the stack trace tells you where the assertion fired, not why the system was in that state. A TimeoutException in a Playwright test is almost never caused by the line that threw it — it's caused by a slow downstream service, a missing fixture teardown, or a saturated CI runner. The stack trace is the symptom; the span and the log line before it are the cause. Treating the stack trace as sufficient leads to "fixed" tickets that reopen two sprints later.

Myth 2: A passing re-run means the failure was noise. Retry-on-failure masks localization entirely. If your CI pipeline auto-retries and the second run passes, the failure event is often not recorded, the trace is orphaned, and the benchmark delta is discarded. Over time you accumulate a class of failures that "never happen twice" but consume 30% of your pipeline runtime in retries. Record every attempt — including the passing retry — with its own span and link them via a parent ci_run_id. That's the data that lets you apply embedding-based deduplication to surface recurring failure signatures before they become incidents.

Failure localization isn't a dashboard feature — it's a data architecture decision made at instrumentation time. Get the OpenTelemetry hooks in place, enforce structured log fields with trace IDs, and store benchmark baselines before you build any UI. Once those three streams are correlated, the triage dashboard almost writes itself. A practical next step: audit one recent high-friction postmortem and map exactly which data was missing at the moment the on-call engineer needed it. That gap is your instrumentation backlog.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.