AI Classifiers Conflating Assertions With Timeouts

Most AI-assisted failure classification pipelines were trained on JUnit XML and Pytest output where the failure message is the ground truth. That assumption holds until you're operating at scale — hundreds of parallel jobs, flaky infrastructure, network partitions — and your classifier starts routing TimeoutError: Waited 30000ms into the same bucket as AssertionError: expected 200 but got 404. Both show up as a FAILED test. The model sees a failure string and picks a label. The signal degrades silently.

The conflation isn't a model quality problem in isolation — it's a feature engineering problem. Assertion errors and timeout errors carry fundamentally different causal structures: one is a logic regression, the other is an environmental condition. When a classifier collapses them into a single "test failure" category, triage queues fill with false positives, on-call engineers chase non-existent bugs, and flakiness rates look artificially stable.

By the end of this article you'll understand exactly where classifiers go wrong at the representation layer, how to build a pre-classification normalizer that separates error taxonomy before the model sees it, and how to validate that the fix is working in production using measurable triage metrics.

The Structural Difference Between Assertion Errors and Timeouts

An assertion error is deterministic and reproducible given the same code path: the system under test returned a value the test did not expect. It carries a diff — expected vs. actual — and its root cause lives in application logic or test logic. A timeout error is probabilistic and environment-sensitive: the system took longer than a threshold, which may reflect a slow dependency, thread starvation, a saturated CI runner, or a race condition. Failure sequencing can distinguish these at the suite level, but a classifier operating on individual test records has to make that distinction from the error string alone — and most don't.

In a well-structured test architecture, these two failure classes map to different remediation owners: assertion errors go to the feature team, timeouts go to platform or infra. Conflating them in a classifier means your routing logic is wrong by construction. The downstream effect is that flaky and regression labels get mixed, priority queues become unreliable, and the feedback loop that should accelerate triage instead introduces noise. Reading a test failure correctly requires knowing which of these two causal trees you're in before you even open a log.

Building a Pre-Classification Normalizer to Separate Error Taxonomy

The fix starts before the model. Insert a deterministic normalizer stage that extracts structured error metadata from raw JUnit XML or JSON test output and emits a typed failure_class field. The classifier then operates on that typed field rather than raw message strings. This is not a workaround — it's correct feature engineering.

import re
from dataclasses import dataclass
from typing import Literal

FailureClass = Literal["assertion", "timeout", "infra", "unknown"]

TIMEOUT_PATTERNS = [
    re.compile(r"(?i)(timeout|timed out|waited \d+ms|deadline exceeded)"),
    re.compile(r"(?i)(connection refused|socket hang up|ECONNRESET)"),
]
ASSERTION_PATTERNS = [
    re.compile(r"(?i)(assertionerror|expected .+ (but got|to equal|to be)|assert .+ ==)"),
]

@dataclass
class NormalizedFailure:
    test_id: str
    raw_message: str
    failure_class: FailureClass
    duration_ms: int
    retry_count: int

def classify_failure(test_id: str, message: str, duration_ms: int, retry_count: int) -> NormalizedFailure:
    for p in TIMEOUT_PATTERNS:
        if p.search(message):
            return NormalizedFailure(test_id, message, "timeout", duration_ms, retry_count)
    for p in ASSERTION_PATTERNS:
        if p.search(message):
            return NormalizedFailure(test_id, message, "assertion", duration_ms, retry_count)
    return NormalizedFailure(test_id, message, "unknown", duration_ms, retry_count)

Duration and retry count are first-class fields here, not afterthoughts. A test that fails on the first attempt in 200ms with an assertion diff is categorically different from one that fails after three retries at 29,800ms. Duration variance is often the strongest signal that a timeout failure is environmental rather than deterministic, and it should be a feature your model actually sees.

Once you have typed failures, feed them into your classifier with explicit label separation. If you're using a fine-tuned model (OpenAI fine-tune, a local Llama variant, or a Claude prompt chain), your training data needs balanced representation of both classes with their correct labels — most open-source test failure datasets are assertion-heavy and will bias your model. A simple ClickHouse query against your historical test results can expose the imbalance:

-- ClickHouse: check label distribution in training data
SELECT
    failure_class,
    count()                          AS total,
    round(count() * 100.0 / sum(count()) OVER (), 1) AS pct
FROM test_failures
WHERE run_date >= today() - 90
GROUP BY failure_class
ORDER BY total DESC;

If assertion is above 70% of your corpus, your classifier will underperform on timeouts in production. Oversample timeout and infra failures, or use class-weighted loss if you're fine-tuning. One team wired this normalizer into their GitHub Actions post-run step and routed structured output to a PostgreSQL failures table; triage time dropped from 22 minutes per failure to under 4 once the classifier stopped mixing causal categories and the on-call dashboard stopped showing false regression alerts.

Where Engineers Wire This Up Wrong

The most common mistake is running the AI classifier directly on raw message fields from JUnit XML without any normalization. JUnit XML is notoriously inconsistent — Playwright, Selenium, and k6 all emit different timeout string formats, and a model trained on Pytest output will misclassify Playwright's Test timeout of 30000ms exceeded as an assertion error because the word "exceeded" appears in assertion failure messages too. The fix is a tool-aware normalization layer, not a smarter model.

The second mistake is treating classifier confidence scores as a reliable signal without calibration. A model that outputs 0.82 confidence for "assertion" on a timeout failure is confidently wrong, and if your routing logic gates on a 0.75 threshold, that failure goes to the wrong queue every time. Calibrate your classifier against a held-out labeled set from your own pipeline — not a benchmark dataset — and set per-class confidence thresholds independently. Timeouts in particular have high surface-form variance and tend to be systematically underconfident or overconfident depending on your training distribution.

Myths That Keep Classifier Quality Low

Myth 1: More training data fixes conflation. It doesn't, if the data is mislabeled or imbalanced. A classifier trained on 50,000 mislabeled examples learns the wrong boundary with high confidence. Label quality beats label quantity — audit a random 200-record sample from your training corpus before you scale data collection. Myth 2: Pass/fail is enough signal for routing. It isn't. A timeout on a critical payment flow in a staging pipeline is a blocking infra issue; the same timeout on a low-priority smoke test in a nightly run is noise. Pipeline stage context changes the severity of a failure independent of its class, and your classifier should receive stage metadata as a feature.

Myth 3: Once deployed, the classifier is stable. Test infrastructure drifts — new frameworks, new runner images, new timeout defaults — and the classifier's input distribution shifts with it. A model that was 91% accurate at deploy time can degrade to 74% within two quarters without any model changes, purely from input drift. Instrument your classifier with a shadow-label pipeline: for every classified failure, also run the deterministic normalizer and log disagreements. When disagreement rate exceeds 15%, retrain. This is standard ML ops discipline applied to test observability, and most teams skip it entirely.

The conflation of assertion errors and timeouts in AI classifiers is a feature engineering failure, not a model capability gap. Fix the normalization layer first, audit your training label distribution, and instrument disagreement rate as an ongoing health metric. If you're building the broader failure analysis infrastructure around this, the embedded CI failure analysis dashboard patterns are a practical next layer — they give classifier output a place to surface actionable signal rather than disappear into a log.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.