LLM That Reads JUnit XML and Explains Failures

Most teams treat JUnit XML like exhaust: it gets archived, maybe parsed by a dashboard, and consulted only when someone needs to prove a test failed. The <failure message> and <system-out> nodes contain the actual signal — stack traces, assertion diffs, environment noise — but reading them at scale means either writing brittle regex or burning an engineer's afternoon. That's the gap an LLM fills well, and it's a narrow, tractable problem that doesn't require a multi-agent orchestration nightmare.

The technical problem is straightforward: JUnit XML is structured but verbose. A single testsuites document from a Playwright or JUnit 5 run can contain dozens of failures, each with a different root cause — a flaky network stub, a real regression, a misconfigured environment. Grouping and explaining those failures in human language, consistently, is exactly the kind of pattern-matching task where a large language model earns its keep.

By the end of this article you'll have a working Python pipeline that parses JUnit XML, extracts failure context, calls an LLM (OpenAI or Claude), and posts structured explanations to Slack or a PR comment — with the prompt design, failure-grouping logic, and CI wiring included.

What JUnit XML Actually Contains (and Why LLMs Can Read It)

JUnit XML — the de facto interchange format for test results across Pytest, JUnit 5, Playwright, and most CI systems — encodes four things an LLM needs: the test name, the failure type, the failure message, and the captured stdout/stderr. The <failure> node carries a type attribute (e.g., AssertionError, NullPointerException) and a text body that is usually a full stack trace. The <system-out> node captures log lines emitted during the test. Together they form a dense, self-contained context window — typically 200–800 tokens per failure — which is well within what GPT-4o or Claude 3.5 Sonnet can process cheaply in a single call.

Where this fits in a modern test architecture: JUnit XML sits at the boundary between raw CI output and your observability layer. You're already parsing JUnit XML reports to extract flaky-test signals for dashboards and trend analysis. Adding an LLM explanation step is a transform in that same pipeline — not a separate product. The output is a structured JSON blob (failure class, plain-language summary, suggested owner, confidence) that can be stored alongside the raw XML and surfaced wherever your team already looks: Slack, GitHub PR comments, Allure attachments, or a Grafana annotation.

Building the Parser-to-LLM Pipeline

Start with extraction. Python's xml.etree.ElementTree is sufficient; no heavy dependency needed. Pull the fields that matter and skip the noise:

import xml.etree.ElementTree as ET
from dataclasses import dataclass
from typing import List

@dataclass
class FailureRecord:
    classname: str
    test_name: str
    failure_type: str
    failure_message: str
    stdout: str

def extract_failures(xml_path: str) -> List[FailureRecord]:
    tree = ET.parse(xml_path)
    root = tree.getroot()
    records = []
    # Handle both <testsuites> and bare <testsuite> roots
    suites = root.findall(".//testsuite") or [root]
    for suite in suites:
        for tc in suite.findall("testcase"):
            failure = tc.find("failure")
            if failure is None:
                continue
            records.append(FailureRecord(
                classname=tc.get("classname", ""),
                test_name=tc.get("name", ""),
                failure_type=failure.get("type", ""),
                failure_message=failure.get("message", "")[:600],
                stdout=(tc.findtext("system-out") or "")[:800],
            ))
    return records

Truncating at 600 and 800 characters keeps each failure under ~400 tokens. You want to batch 3–5 failures per LLM call to amortize latency without blowing the context window — group by classname prefix first so related failures stay together.

import openai, json

SYSTEM_PROMPT = """You are a senior QE engineer. Given a list of JUnit test failures,
return a JSON array. Each element: {
  "test_name": str,
  "root_cause_category": "assertion" | "environment" | "flaky" | "regression" | "unknown",
  "plain_summary": str (1-2 sentences, no jargon),
  "suggested_owner": "frontend" | "backend" | "infra" | "unknown",
  "confidence": "high" | "medium" | "low"
}. Be terse. Do not hallucinate stack frames."""

def explain_failures(records: List[FailureRecord], model="gpt-4o") -> list:
    payload = [
        {"test": r.test_name, "type": r.failure_type,
         "message": r.failure_message, "stdout": r.stdout}
        for r in records
    ]
    resp = openai.chat.completions.create(
        model=model,
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": json.dumps(payload)},
        ],
        temperature=0,
    )
    return json.loads(resp.choices[0].message.content).get("failures", [])

temperature=0 is non-negotiable for diagnostic output — you want determinism, not creativity. The response_format: json_object parameter (available on GPT-4o and GPT-3.5-turbo-1106+) eliminates the most common integration failure: the model wrapping JSON in a markdown fence. For Claude, use a <json> XML tag in the prompt instead.

Wire this into GitHub Actions as a post-test step. The full run — parse, batch, call, post to Slack — takes under 8 seconds for a 40-failure report at GPT-4o pricing (~$0.003 per run). One team reduced triage time from 22 minutes per failure to under 4 once this replaced the "open Allure, read stack trace, grep Loki" loop. For deeper context on what to do after you have the explanation, the auto-triaging failures with LLMs walkthrough covers routing and ticket creation. Be aware of the failure modes described in how AI failure classifiers misread cascading test failures — when one upstream service fails and 30 tests fall, the LLM will confidently explain each one individually rather than recognizing the shared root cause unless you cluster first.

Where This Pipeline Breaks in Practice

The most common mistake is sending raw stack traces without truncation or sanitization. A JUnit 5 failure from a Spring Boot integration test can include 80-line stack traces with framework internals that add zero signal and eat tokens. Engineers copy-paste the full system-out into the prompt, hit rate limits on large suites, and conclude "LLMs don't work for this." They do — you just need to strip frames below your own package prefix before sending. A simple heuristic: keep the first 5 lines of the stack trace and the last 2; discard everything in between that matches java.lang, sun.reflect, or framework packages.

The second failure mode is skipping failure clustering. If 15 tests fail because a shared Testcontainers database didn't start, sending all 15 individually produces 15 nearly identical "database connection refused" explanations and wastes budget. Group by failure_type and the first line of the message before batching — deduplicate to one representative record per cluster, explain it once, then fan the explanation back out. This also prevents the LLM from being confidently wrong about each test's individual cause when the real cause is environmental.

Myths About LLMs and JUnit Failure Analysis

Myth 1: The LLM needs the full codebase to explain a failure. It doesn't — not for the initial triage. The failure message, type, and captured stdout are almost always sufficient to classify root cause category and suggest an owner. Retrieval-augmented approaches that pull source files are useful for generating fix suggestions, but that's a second, optional step. Start with what's in the XML. Myth 2: This only works for simple assertion failures. In practice, the model handles TimeoutException, NullPointerException, and environment-class failures better than assertion failures, because those have more distinctive message patterns. Assertion failures ("expected 200 but was 404") are actually the ambiguous case — the LLM needs the test name and class context to say anything useful beyond the obvious.

Myth 3: Pass/fail rate is the metric to optimize. If you're wiring an LLM to your JUnit output and the only question you're asking is "did it pass?", you're ignoring the most valuable output. The root_cause_category distribution over time — what fraction of failures are environment vs. regression vs. flaky — is a far more actionable signal for platform investment decisions. Track it. A rising environment share is an infra debt indicator; a rising flaky share means your quarantine process isn't working.

The pipeline described here — extract, cluster, prompt, post — is a weekend project, not a platform initiative. Start with a single test suite that generates the most noise in your postmortems, run it for two weeks, and measure triage time before and after. Once the root_cause_category distribution is stable, you have a leading indicator worth tracking in your engineering metrics. From there, the natural next step is connecting failure explanations to production context — which is exactly what connecting test failures to production logs covers.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.