RAG Pipeline Over Your Own Test History

Most teams treat test results like a checkbox: green is good, red is bad, ship or block. The interesting signal lives in everything between — runtime variance, retry counts, the same five tests surfacing in every postmortem. When an engineer opens a Jenkins failure at 2 AM, they don't need a generic LLM answer about what NullPointerException means. They need the three times this exact test failed in the last 30 days, what changed in the codebase each time, and whether it's a flaky outlier or a real regression.

Retrieval-Augmented Generation (RAG) over your own test history is the architecture that makes that possible. Instead of sending a raw error message to an LLM and hoping for a useful response, you first retrieve semantically similar historical failures from your own data, then pass that grounded context to the model. The answer is anchored to your system's actual behavior, not the model's training data.

By the end of this article you'll have a working mental model — and enough concrete code — to wire a RAG pipeline over a test result store, whether that's PostgreSQL, BigQuery, or ClickHouse. You'll also know where the approach breaks down and what most teams misconfigure on the first attempt.

What a Test-History RAG Pipeline Actually Does

A RAG pipeline has two phases: retrieval and generation. In the test-history context, retrieval means embedding your failure records — test name, error message, stack trace excerpt, pipeline stage, branch, and timestamp — into a vector store, then at query time finding the closest historical matches to the current failure. Generation means passing those matches as context to an LLM (GPT-4o, Claude 3.5 Sonnet, or a self-hosted Mistral) and asking it to synthesize a root-cause hypothesis. The LLM never hallucinates a fix for code it hasn't seen; it reasons over real examples you've retrieved.

This fits into a modern test architecture as a layer above your existing result store. Your CI pipeline (GitHub Actions, Jenkins, Buildkite) already emits JUnit XML or JSON. You parse and persist those results — ideally in a queryable store like BigQuery or ClickHouse, as covered in depth when querying test history in BigQuery vs Snowflake. The RAG pipeline reads from that same store, generates embeddings on ingest, and exposes a query interface that your on-call engineer or Slack bot can hit when a failure lands.

Building the Pipeline: Ingest, Embed, Retrieve, Generate

Start with a structured ingest schema. Every failure record needs enough context to be useful when retrieved. A minimal PostgreSQL table:

CREATE TABLE test_failures (
  id          BIGSERIAL PRIMARY KEY,
  run_id      TEXT NOT NULL,
  test_name   TEXT NOT NULL,
  suite       TEXT,
  stage       TEXT,          -- unit | integration | e2e
  branch      TEXT,
  error_msg   TEXT,
  stack_trace TEXT,
  duration_ms INT,
  flaky       BOOLEAN DEFAULT FALSE,
  created_at  TIMESTAMPTZ DEFAULT NOW(),
  embedding   VECTOR(1536)   -- pgvector extension
);

The embedding column uses pgvector (PostgreSQL ≥ 15 + pgvector 0.5+). On ingest, concatenate the fields that carry diagnostic signal and call the embedding API:

import openai, psycopg2, textwrap

def embed_failure(record: dict) -> list[float]:
    text = textwrap.dedent(f"""
        test: {record['test_name']}
        stage: {record['stage']}
        error: {record['error_msg']}
        trace: {record['stack_trace'][:800]}
    """).strip()
    resp = openai.embeddings.create(model="text-embedding-3-small", input=text)
    return resp.data[0].embedding

def store_failure(conn, record: dict):
    vec = embed_failure(record)
    with conn.cursor() as cur:
        cur.execute("""
            INSERT INTO test_failures
              (run_id, test_name, suite, stage, branch, error_msg, stack_trace, duration_ms, embedding)
            VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s)
        """, (record['run_id'], record['test_name'], record['suite'],
              record['stage'], record['branch'], record['error_msg'],
              record['stack_trace'], record['duration_ms'], vec))
    conn.commit()

Retrieval at query time is a cosine-similarity search. Pass the current failure through the same embedding function, then pull the top-k neighbors:

def retrieve_similar(conn, current_failure: dict, k: int = 5) -> list[dict]:
    vec = embed_failure(current_failure)
    with conn.cursor() as cur:
        cur.execute("""
            SELECT test_name, stage, branch, error_msg, stack_trace, created_at,
                   1 - (embedding <=> %s::vector) AS similarity
            FROM test_failures
            WHERE created_at > NOW() - INTERVAL '60 days'
            ORDER BY embedding <=> %s::vector
            LIMIT %s
        """, (vec, vec, k))
        cols = [d[0] for d in cur.description]
        return [dict(zip(cols, row)) for row in cur.fetchall()]

Now generate. Inject the retrieved records as context and constrain the model to reason only over them:

def diagnose(current_failure: dict, history: list[dict]) -> str:
    context = "\n\n".join(
        f"[{i+1}] {h['created_at'].date()} | stage={h['stage']} | branch={h['branch']}\n"
        f"error: {h['error_msg']}\ntrace: {h['stack_trace'][:400]}"
        for i, h in enumerate(history)
    )
    prompt = f"""You are a test-reliability engineer. Use ONLY the historical failures below.
Identify the most likely root cause for the current failure and note if it appears flaky.

CURRENT FAILURE:
test: {current_failure['test_name']}
error: {current_failure['error_msg']}

HISTORICAL CONTEXT:
{context}

Respond with: root_cause, confidence (low/medium/high), is_flaky (true/false), recommended_action."""
    resp = openai.chat.completions.create(
        model="gpt-4o", messages=[{"role": "user", "content": prompt}], temperature=0
    )
    return resp.choices[0].message.content

In practice, wiring this to a Slack slash command cut triage time from roughly 22 minutes per failure to under 4 — the engineer gets a structured hypothesis before they've even opened the CI log. The is_flaky field feeds directly into a quarantine workflow; anything scoring true with high confidence gets tagged and tracked separately, which matters because flake rate compounds across dependent pipeline stages in ways that aggregate pass-rate dashboards completely obscure.

Where Senior Engineers Still Get This Wrong

The most common mistake is embedding only the error message, ignoring pipeline stage and branch. A TimeoutException in a unit test means something entirely different from the same exception in an end-to-end suite against a staging environment — a distinction worth reading about in detail when considering how the same test failure means different things depending on where in the pipeline it appears. Without stage metadata in the embedding text, your retrieval surface collapses and the model gets misleading context. Always embed the full diagnostic tuple, not just the raw error string.

The second mistake is skipping embedding freshness. Teams embed on first ingest and never re-embed when they upgrade models. text-embedding-ada-002 and text-embedding-3-small live in different vector spaces — mixing them produces nonsense similarity scores. Pin your embedding model version in the schema (embed_model TEXT DEFAULT 'text-embedding-3-small'), and run a backfill migration whenever you upgrade. The third mistake is over-retrieving: k=20 bloats the context window, dilutes signal with distant matches, and inflates token costs. Five to eight neighbors is the practical ceiling for failure triage; tune with precision@k against a labeled sample of known root causes before going to production.

Myths That Slow Teams Down Before They Start

"We need a dedicated vector database." Pinecone and Weaviate are fine choices, but pgvector on an existing PostgreSQL instance handles tens of millions of embeddings with an IVFFlat index before you feel any latency. Most test fleets generate fewer than 500K failure records per year. Start with pgvector; migrate if you hit real scale. Similarly, "RAG requires a huge history." Sixty days of failures — even from a mid-sized suite — is enough to surface meaningful patterns. The signal-to-noise ratio matters more than raw volume; a clean schema with good metadata beats five years of poorly structured logs.

"The LLM will figure out what's important." It won't, not reliably. The quality of retrieval determines the quality of generation — full stop. If your embeddings are noisy or your retrieval window is stale, the model confidently synthesizes garbage. Treat the retrieval layer with the same rigor you'd apply to a pattern detection pipeline built on test history embeddings: validate recall, measure precision, and instrument the pipeline with OpenTelemetry spans so you can debug retrieval misses in Honeycomb or Grafana Tempo when the model's answer doesn't match reality.

A RAG pipeline over test history is not magic — it's a retrieval problem with an LLM on top. Get the ingest schema right, embed the full diagnostic tuple, keep your embedding model version consistent, and constrain generation to retrieved context. Start with pgvector and 60 days of history. Wire the output to wherever your engineers already triage failures — Slack, PagerDuty, a Grafana annotation. Measure triage-time-to-hypothesis as your north-star metric, and iterate from there.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.