Fine-Tuning vs Prompting for Failure Triage

Most teams treating AI-assisted triage as a chatbot wrapper around their logs are leaving the real value on the table. The interesting split isn't "AI vs no AI" — it's whether your failure triage signal is dense and domain-specific enough to justify fine-tuning a model, or whether a well-engineered prompt chain against GPT-4o or Claude 3.5 Sonnet already gets you to 90% accuracy at a fraction of the cost. That decision shapes your entire observability stack.

The technical problem: CI failure triage is a classification and summarization task with high domain specificity. Your stack traces, test IDs, flakiness history, and infra error signatures don't appear in any foundation model's pretraining corpus at meaningful density. A raw prompt will hallucinate test names, confuse AssertionError with TimeoutError, and miss the difference between an infra flake and a logic regression — exactly the failure sequencing patterns that determine whether you page an on-call engineer or just retry the job.

By the end of this article you'll know how to evaluate which approach fits your failure volume and data maturity, how to build a prompt-chain triage pipeline with measurable latency targets, and when the operational cost of fine-tuning actually pays off.

What Triage Means at the Model Layer

Test triage, in the context of a CI pipeline, is the act of classifying a failure by root-cause category (infra, flake, regression, environment, ownership) and routing it to the right response. A triage test engineer asks three core triage test questions on every failure: Is this failure new or recurring? Is it deterministic or stochastic? Does it indicate a product defect or a pipeline defect? Those three questions map directly to the features you need to feed an LLM — failure history, retry outcomes, and stack trace structure.

At the model layer, triage is a multi-label classification task with a short-form explanation output. Prompting treats the foundation model as a zero-shot or few-shot reasoner; fine-tuning shifts the model's probability distribution toward your domain vocabulary, error taxonomy, and output format. The right choice depends on whether your failure corpus is large enough (typically 2,000+ labeled examples to see meaningful fine-tune lift), whether your taxonomy is stable, and whether latency and cost constraints allow for the larger context windows that good prompt engineering often requires.

Building the Triage Pipeline: Prompting First, Fine-Tuning When It Earns It

Start with a structured prompt chain before you touch fine-tuning. The chain has three stages: context assembly, classification, and explanation generation. Context assembly pulls the last 10 run outcomes for the test, the raw stack trace truncated to 3,000 tokens, and any linked Loki log lines — the same signal you'd surface in a Grafana + Loki triage view. Feed that into a classification prompt that constrains the model to your taxonomy.

# Python 3.11 — context assembly for triage prompt
import json
from openai import OpenAI

client = OpenAI()

def build_triage_prompt(test_id: str, stack_trace: str, history: list[dict]) -> str:
    recent = history[-10:]
    outcome_seq = [r["outcome"] for r in recent]  # e.g. ["pass","pass","fail","pass","fail"]
    flake_signal = outcome_seq.count("fail") / len(outcome_seq) if outcome_seq else 0

    return f"""You are a CI triage classifier. Respond ONLY with valid JSON.

Test: {test_id}
Failure rate (last 10 runs): {flake_signal:.0%}
Outcome sequence: {outcome_seq}
Stack trace (truncated):
{stack_trace[:3000]}

Classify into exactly one of: infra_flake | logic_regression | env_config | ownership_gap | unknown
Output: {{"category": "...", "confidence": 0.0-1.0, "one_line_reason": "..."}}"""

def triage(test_id, stack_trace, history):
    prompt = build_triage_prompt(test_id, stack_trace, history)
    resp = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
        response_format={"type": "json_object"},
    )
    return json.loads(resp.choices[0].message.content)

Forcing JSON output with response_format and temperature=0 eliminates the most common prompt-triage failure mode: free-form prose that downstream parsers can't handle. One team running this pattern against a 4,000-test Pytest suite cut mean triage time from 22 minutes per failure to under 4 once the output fed directly into a Slack webhook and their CI failure dashboard.

Fine-tuning earns its cost when two conditions hold simultaneously: your taxonomy is stable for at least a quarter, and prompt-chain accuracy on your labeled eval set is below 82%. To build a fine-tune dataset, export your historical triage decisions from PostgreSQL or BigQuery, filter to human-confirmed labels, and format as JSONL for the OpenAI fine-tuning API or for a local LoRA run on Llama 3.1 8B via Axolotl.

-- PostgreSQL: export labeled triage decisions for fine-tune dataset
SELECT
    t.test_id,
    t.stack_trace_summary,
    array_agg(r.outcome ORDER BY r.run_at DESC) FILTER (WHERE r.run_at > NOW() - INTERVAL '30 days') AS recent_outcomes,
    t.human_label,
    t.confidence_score
FROM test_failures t
JOIN test_runs r USING (test_id)
WHERE t.human_label IS NOT NULL
  AND t.human_label != 'unknown'
  AND t.labeled_at > NOW() - INTERVAL '90 days'
GROUP BY t.test_id, t.stack_trace_summary, t.human_label, t.confidence_score
HAVING COUNT(r.run_id) >= 5
ORDER BY t.labeled_at DESC
LIMIT 5000;

A fine-tuned GPT-4o-mini on 2,400 examples from this query hit 91% accuracy on held-out failures vs. 79% for the zero-shot GPT-4o prompt — and ran at roughly 40% of the token cost per call. The crossover point: if you're triaging fewer than 500 failures per day, the fine-tune ROI rarely materializes inside a quarter. Stick with prompt engineering, invest in your context assembly quality, and revisit when volume grows.

Where Triage Engineers Trip Up When Adding AI

The most common mistake is feeding raw, untruncated logs to the model and calling it done. A 200KB log file doesn't improve classification — it degrades it by burying the signal in noise, inflates token costs, and frequently hits context limits mid-trace, producing garbage output. Structured pre-processing (extract the exception type, the first failing assertion, and the last 15 log lines) consistently outperforms raw log dumps. This is a data-engineering problem before it's a model problem.

The second mistake is skipping an eval set entirely and judging quality by vibes. Without a labeled holdout of at least 200 failures covering all taxonomy categories, you can't measure accuracy regression when you update the prompt or swap models. Teams that skip this end up in a loop of anecdotal "it seems better" vs. "it seems worse" with no ground truth. Build the eval set from your first 50 human-triaged failures, gate every prompt or model change against it, and track accuracy in CI the same way you track test pass rates — the same discipline you'd apply to any other failure localization workflow.

Myths That Send Teams Down the Wrong Path

Myth 1: Fine-tuning always produces a more accurate model for your domain. Fine-tuning optimizes for distribution shift, not reasoning quality. If your failures require multi-step reasoning across stack trace + history + infra context, a fine-tuned smaller model will often underperform a well-prompted larger one. Fine-tuning wins on format consistency and cost at scale — not on reasoning depth. Myth 2: A high confidence score from the model means the classification is correct. LLM confidence outputs are not calibrated probabilities. A model returning "confidence": 0.97 on an infra_flake label can be systematically wrong on a class of errors it never saw in training. Treat confidence as a routing signal (human review below 0.75), not a quality guarantee.

Myth 3: Once you have AI triage, you don't need structured dashboards. AI triage handles the per-failure classification problem. It doesn't replace aggregate pattern detection — spotting that 14 tests in the payments module all started failing after a dependency bump, or that your flake rate spiked after a "fix" (a real phenomenon worth understanding at the metric level). AI and dashboard-based observability solve different layers of the same problem; removing one because the other exists is how signal disappears.

The practical starting point: instrument your triage pipeline with the prompt-chain approach above, build a labeled eval set from your next 50 human-triaged failures, and measure accuracy weekly. If you're above 85% and volume is under 500 failures/day, prompting is your answer for now. If accuracy plateaus below 82% and you have 2,000+ labeled examples, run a fine-tune experiment on a 30-day holdout and let the numbers decide. The model choice is an engineering decision, not a vendor pitch.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.