Embeddings for Test-Failure Dedup at Scale

Most teams treat test failure deduplication as a string-matching problem: normalize the message, strip the line numbers, hash it. That works until your suite hits a few thousand failures a week and the same underlying defect produces seventeen slightly different stack traces depending on which worker picked it up, which retry count you're on, or which dependency version got resolved. At that point, exact-match bucketing creates more noise than it removes — you get 40 "unique" failures that are really one bad migration.

The problem is that test failure meaning is semantic, not lexical. Two failures can share zero tokens and still point to the same root cause. Conversely, two failures can share a common exception class and be completely unrelated. This is the gap where vector embeddings live: they encode the meaning of a failure artifact — message, trace, test name, file path — into a dense numeric representation that supports similarity search rather than equality checks.

By the end of this article you'll have a working Python pipeline that embeds JUnit XML failure bodies, stores them in a vector index, and queries for near-duplicate clusters — giving your CI failure analysis dashboard a deduplicated failure stream instead of a raw firehose.

What Embedding-Based Dedup Actually Does to Your Failure Signal

A vector embedding is a fixed-length float array (typically 384–1536 dimensions) produced by a model trained to place semantically similar text close together in that space. For test failures, the input is the concatenation of the test name, failure message, and the top N frames of the stack trace — stripped of memory addresses and timestamps but otherwise raw. The output is a point in high-dimensional space. Two failures caused by the same defect land near each other; two failures that merely share a common library frame land further apart. Cosine similarity gives you a distance metric you can threshold.

In a modern test architecture this sits between your raw JUnit XML ingestion layer and your triage queue. Rather than writing every failure as a distinct record, you query the vector index first: if a failure lands within cosine distance 0.12 of an existing cluster centroid, it's a duplicate and gets appended to that cluster's count. Only genuinely novel failures — those with no near neighbor — open new triage tickets. This is the layer where failure clustering can mask distinct root causes if your threshold is too loose, which is why tuning the similarity cutoff against labeled data matters more than picking the fanciest model.

Building the Embedding Pipeline: From JUnit XML to Clustered Failures

Start with extraction. Parse JUnit XML with Python's xml.etree.ElementTree, pull testcase name, failure message, and the first 20 lines of the stack trace, then concatenate them into a single string. Truncate at ~512 tokens — most embedding models have a context limit and the discriminative signal is front-loaded in the trace anyway.

import xml.etree.ElementTree as ET
from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")  # 384-dim, fast, good enough

def extract_failure_text(junit_path: str) -> list[dict]:
    tree = ET.parse(junit_path)
    failures = []
    for tc in tree.iter("testcase"):
        f = tc.find("failure")
        if f is None:
            continue
        text = f"{tc.attrib.get('classname','')}.{tc.attrib.get('name','')} "
        text += (f.attrib.get("message", "") + " " + (f.text or ""))[:2000]
        failures.append({"id": tc.attrib.get("name"), "text": text.strip()})
    return failures

def embed_failures(failures: list[dict]) -> np.ndarray:
    texts = [f["text"] for f in failures]
    return model.encode(texts, batch_size=64, normalize_embeddings=True)

all-MiniLM-L6-v2 from Sentence Transformers is the right default: 384 dimensions, ~14k tokens/sec on a single CPU core, and it scores within 3–5% of OpenAI text-embedding-3-small on semantic similarity benchmarks while costing nothing per call. Switch to text-embedding-3-small via the OpenAI API if you need better multilingual coverage or have non-English log output.

Next, store and query. Use pgvector if you're already on PostgreSQL (zero new infra), or Qdrant if you want a dedicated vector store with filtering on metadata like branch, pipeline stage, or test suite name. The query pattern is the same either way: for each new failure embedding, find the nearest neighbor; if cosine similarity exceeds your threshold (start at 0.88, tune from there), attach to that cluster.

-- pgvector: find nearest existing cluster within threshold
SELECT cluster_id, failure_text, 1 - (embedding <=> $1::vector) AS similarity
FROM failure_clusters
ORDER BY embedding <=> $1::vector
LIMIT 1;

-- If similarity >= 0.88, UPDATE cluster hit count.
-- Else INSERT as new cluster with a fresh cluster_id.

Wire this into your CI post-step as a GitHub Actions job that runs after the test matrix completes, consumes the uploaded JUnit artifacts, and posts a deduplicated summary to Slack via webhook. One team running ~9,000 test executions per day reduced their Slack failure-alert volume from 340 messages/day to 28 distinct cluster notifications — triage time dropped from roughly 22 minutes per failure to under 4 once engineers stopped re-reading the same trace in different fonts. For distinguishing infrastructure flakes from logic flakes, add a metadata filter on retry count: failures that only appear on retry ≥ 2 get tagged flake_candidate before they even reach the cluster index.

Where Embedding Pipelines Break Down in Practice

Threshold drift is the most common failure mode. Teams set a cosine similarity cutoff during initial calibration on a small labeled sample, then never revisit it as the codebase evolves. New frameworks, new assertion libraries, and new logging formats shift the embedding distribution. What was a clean 0.88 boundary six months ago now over-clusters or under-clusters. Fix this by maintaining a small labeled validation set (200–400 failure pairs, manually tagged same-root/different-root) and running a threshold sweep monthly. Track cluster purity as a metric, not just cluster count.

Embedding the wrong text is an org-level mistake. Engineers often embed only the exception message because it's the most visible field in the test report UI. But the message alone is frequently generic — AssertionError: expected True, got False — and the discriminative signal is in the stack frames or the test name prefix. Conversely, embedding the full 200-line trace buries the signal in framework boilerplate. The right input is: test class + method name + failure message + first 15–20 application-code frames, with third-party frames stripped. This is a data-shaping problem, not a model problem. Test flakiness meaning in your embedding space is only as good as what you feed in — garbage in, collapsed clusters out.

Myths About Semantic Dedup That Slow Teams Down

"We need a bigger model to get good clusters." In practice, all-MiniLM-L6-v2 outperforms GPT-4 embeddings on short, structured technical text like stack traces — larger models are optimized for prose. The real lever is input preprocessing, not model size. Teams that spend a week cleaning their failure text extraction consistently see better cluster quality than teams that spend the same time swapping models. A test insights report built on clean embeddings from a small model beats one built on raw text fed to a frontier model.

"Dedup solves flakiness." Dedup reduces triage noise; it doesn't fix the underlying instability. Failure rate can recover while the defect persists — and the same is true for flaky clusters. A cluster whose hit count stops growing looks resolved in the dashboard, but the root cause may still be live, just suppressed by a retry policy or a timing change. Treat cluster hit-count trends as a leading indicator, not a resolution signal. True test flakiness elimination requires fixing the source; dedup just makes sure you're looking at one ticket instead of forty when you finally go digging. Pair the embedding pipeline with duration-variance tracking — test insights from timing data often surface the same instability before the failure rate moves at all.

The implementation path is straightforward: extract failure text from JUnit XML, embed with all-MiniLM-L6-v2, store in pgvector or Qdrant, query by cosine similarity before opening new triage tickets. Start with a cosine threshold of 0.88, validate it against 200 labeled pairs, and schedule a monthly threshold sweep. Once the pipeline is stable, wire cluster metadata — hit count, first-seen, last-seen, retry rate — into your existing observability stack. The signal was always there; dedup just makes it readable.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.