LLM Triage for a Failing Test Suite

Most CI dashboards tell you that something broke. They rarely tell you why — not in the way an engineer who has context tells you. You're left pivoting between Allure reports, Loki log tails, Slack threads, and a JIRA backlog, manually assembling a picture that should already exist. The signal is all there; it's just not synthesized.

Wiring an LLM into your triage loop is not about replacing engineers — it's about collapsing the time between "red build" and "understood failure." The model reads what you already produce: JUnit XML, stack traces, log excerpts, test history diffs. It outputs a structured hypothesis. You decide whether it's right.

By the end of this article you'll have a working Python pipeline that feeds failure context to an LLM, a prompt structure that produces actionable output rather than generic summaries, and a clear-eyed view of where the approach breaks down in production CI environments.

What LLM Triage Actually Does to Your Signal Pipeline

LLM-assisted triage sits between your test reporter and your alerting layer. It is not a test runner, a flakiness detector, or a root-cause analysis engine — it is a context aggregator with a natural-language interface. You feed it structured failure data (JUnit XML, log lines, recent git diff, historical flake rate), and it produces a ranked hypothesis: infrastructure noise, regression, environment drift, or test-code defect. That classification alone can route the failure to the right team without a human reading three separate tools.

In a modern test architecture this sits downstream of your results store (PostgreSQL, BigQuery, ClickHouse) and upstream of your notification layer (PagerDuty, Slack). It does not replace structured log-based triage in Grafana and Loki — it consumes the same data those dashboards surface and adds a reasoning layer. Think of it as a Slack bot that has actually read the logs before it pages you.

Building the Triage Pipeline: Prompt, Parser, and CI Hook

Start with a Python script that parses JUnit XML, pulls the last 50 log lines for each failing test from Loki, and assembles a context payload. The junit-xml and httpx packages handle the heavy lifting. Keep the context window tight — GPT-4o and Claude 3.5 Sonnet both perform well here, but token cost scales with payload size, so cap log excerpts at 60 lines per failure.

import xml.etree.ElementTree as ET
import httpx, json, os

LOKI_URL = os.environ["LOKI_URL"]
LLM_API_KEY = os.environ["LLM_API_KEY"]   # OpenAI or Anthropic key
LLM_MODEL   = os.environ.get("LLM_MODEL", "gpt-4o")

def fetch_logs(test_name: str, run_id: str, lines: int = 50) -> str:
    query = f'{{job="ci-tests"}} |= "{test_name}" |= "{run_id}"'
    r = httpx.get(
        f"{LOKI_URL}/loki/api/v1/query_range",
        params={"query": query, "limit": lines},
        timeout=10,
    )
    entries = r.json()["data"]["result"]
    return "\n".join(e[1] for stream in entries for e in stream["values"])

def build_prompt(failures: list[dict]) -> str:
    blocks = []
    for f in failures:
        blocks.append(
            f"TEST: {f['name']}\n"
            f"ERROR: {f['message']}\n"
            f"FLAKE_RATE_30D: {f['flake_rate']:.1%}\n"
            f"LOGS:\n{f['logs']}\n"
        )
    return (
        "You are a senior QE engineer. For each failing test below, output JSON with keys: "
        "'test', 'hypothesis' (one of: regression|flake|env_drift|test_defect), "
        "'confidence' (0-1), 'evidence' (≤2 sentences), 'suggested_owner' (team slug).\n\n"
        + "\n---\n".join(blocks)
    )

def call_llm(prompt: str) -> list[dict]:
    r = httpx.post(
        "https://api.openai.com/v1/chat/completions",
        headers={"Authorization": f"Bearer {LLM_API_KEY}"},
        json={"model": LLM_MODEL, "messages": [{"role": "user", "content": prompt}],
              "response_format": {"type": "json_object"}},
        timeout=30,
    )
    return json.loads(r.json()["choices"][0]["message"]["content"])["results"]

The response_format: json_object parameter (available in GPT-4o and GPT-4-turbo) eliminates the fragile regex parsing that breaks on model updates. For Anthropic's Claude, use a system prompt that instructs strict JSON output and validate with Pydantic before you route anything downstream.

Wire this into GitHub Actions as a post-test step. Pass the JUnit XML artifact path and the Loki run ID as environment variables. The step only fires on failure, so it adds zero cost to green builds:

- name: LLM triage on failure
  if: failure()
  env:
    LOKI_URL: ${{ secrets.LOKI_URL }}
    LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
    LLM_MODEL: gpt-4o
    JUNIT_PATH: test-results/results.xml
    RUN_ID: ${{ github.run_id }}
  run: python scripts/llm_triage.py

One team running ~1,400 Pytest tests per PR reduced mean triage time from 22 minutes per failure to under 4 minutes after wiring this pipeline — the bulk of the saving came from eliminating the Loki → Allure → Slack pivot. The LLM's hypothesis was correct roughly 74% of the time across a 30-day sample; the remaining 26% still surfaced the right logs, so engineers started from a better position even when the classification was wrong. For deeper pattern work across test history, embedding-based pattern detection complements this approach by finding structurally similar failures across runs.

Where Senior Engineers Still Get Burned

Over-trusting the hypothesis on high-flake tests. If a test has a 30-day flake rate above 15%, the LLM will almost always classify it as flake — because that's the statistically dominant signal in the context. That's not wrong, but it masks genuine regressions hiding behind a noisy test. Gate the LLM output against flake rate: if flake_rate > 0.10 and the hypothesis is flake, flag it for human review regardless of confidence score. Feeding raw flake rates without percentile context is a related trap — a test that flaked 3 times in 3 runs looks identical to one that flaked 3 times in 300.

Prompt drift in long-running pipelines. Model versions change under you. GPT-4o's JSON output format shifted between the 2024-05-13 and 2024-08-06 snapshots in ways that broke downstream Pydantic parsers silently — the pipeline kept running but routed every failure to a default owner. Pin your model version explicitly (gpt-4o-2024-08-06), add a schema validation step, and alert on parse failures, not just on API errors. This is the same class of silent degradation that causes multi-agent LLM systems to fail quietly in CI.

Myths That Will Waste Your Integration Effort

"A bigger context window means better triage." Feeding the model 500 log lines and the full git diff sounds thorough. In practice, LLMs lose precision on long, noisy inputs — the relevant stack frame gets diluted by framework boilerplate. Structured, curated context (error message, 50 log lines, flake rate, last-pass commit) consistently outperforms raw dumps. The same discipline applies to LLMs reading JUnit XML directly — what you exclude matters as much as what you include.

"LLM triage replaces the need for clean test data." It doesn't. If your suite has order-dependent tests, environment bleed, or a composition that skews your reported metrics, the model inherits those problems. Garbage-in still applies. An LLM classifying a failure caused by test-order pollution will confidently output env_drift — which is plausible and wrong. Fix the underlying data quality first: deduplicate test IDs, normalize suite names across CI workers, and ensure your results store has accurate timestamps. The model is a reasoning layer, not a data-cleaning layer.

LLM triage earns its place when it collapses context-gathering time, not when it replaces engineering judgment. Start with a single failing suite, a tight prompt, and a Pydantic schema that fails loudly. Measure hypothesis accuracy over 30 days before you route anything to PagerDuty automatically. If you want to extend the pattern, look at risk-based test selection — using AI to decide which tests to run is the natural complement to using AI to explain why they failed.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.