AI Classifiers Misread Retry-Induced Passes

Most teams treat a final-attempt pass as a pass. The CI badge goes green, the PR merges, and the failure that triggered three retries quietly disappears from the summary view. The problem is that AI failure classifiers trained on that same final-state signal inherit the lie — they learn that "pass after retry" means the test is healthy, when the retry itself is the signal worth classifying.

The failure mode is subtle. A classifier built on attempt-level outcome data without retry metadata will consistently under-report flakiness, misattribute infrastructure noise as deterministic failures, and — worst of all — assign high-confidence "resolved" labels to tests that are one environment hiccup away from blocking a release. If you're already concerned about how retry count inflates pass rate without fixing flakes, this is the upstream cause of that distortion landing inside your ML pipeline.

By the end of this article you'll understand exactly what AI failure classifiers represent in a test analytics stack, why retry-induced passes corrupt their training signal, and how to instrument your pipeline so classifiers receive attempt-sequence data instead of collapsed outcome data.

What AI Failure Classifiers Actually Represent in a Test Architecture

An AI failure classifier is a model — typically a fine-tuned LLM, a gradient-boosted tree over feature vectors, or a rules-augmented embedding model — that ingests test execution metadata and assigns a label: infrastructure failure, product regression, flake, environment timeout, dependency outage, and so on. The classifier's job is to route the failure to the right owner and suppress noise that doesn't need human triage. In practice, classifiers sit between your JUnit XML / Pytest JSON output and your alerting layer (PagerDuty, Slack, Jira), acting as a triage filter before a human ever sees the result.

What classifiers represent, in the strictest sense, is a learned mapping from execution signal to failure category. That mapping is only as good as the features it was trained on. When the input feature is final_status: passed and the classifier never sees attempt_1: failed, attempt_2: failed, attempt_3: passed, it has no basis to distinguish a clean pass from a retry-laundered one. The classifier doesn't know retries happened — it just sees a pass and moves on. This is a data-pipeline problem masquerading as a model problem, and fixing the model without fixing the pipeline is wasted effort.

Feeding Classifiers Attempt-Sequence Data Instead of Collapsed Outcomes

The fix starts at ingestion. Most CI systems — GitHub Actions, Jenkins, Buildkite — emit a single test result record per test per run. You need to emit one record per attempt, tagged with attempt index and final-run status. In GitHub Actions with Pytest, that means capturing --reruns data from pytest-rerunfailures before JUnit XML is written:

# .github/workflows/test.yml (relevant fragment)
- name: Run tests with attempt metadata
  run: |
    pytest tests/ \
      --reruns 3 \
      --reruns-delay 2 \
      --junit-xml=results/junit.xml \
      --report-log=results/attempt_log.jsonl
  env:
    PYTEST_ADDOPTS: "-p no:cacheprovider"

- name: Upload attempt log
  uses: actions/upload-artifact@v4
  with:
    name: attempt-log
    path: results/attempt_log.jsonl
    retention-days: 30

The attempt_log.jsonl from pytest-rerunfailures contains per-attempt outcome, exception type, and wall-clock duration. Feed this — not the JUnit XML — into your classifier's feature pipeline. A PostgreSQL schema that supports this cleanly looks like:

CREATE TABLE test_attempts (
  run_id        TEXT NOT NULL,
  test_id       TEXT NOT NULL,
  attempt_index INT  NOT NULL,
  outcome       TEXT NOT NULL,  -- 'passed' | 'failed' | 'error'
  duration_ms   INT,
  exception_cls TEXT,
  logged_at     TIMESTAMPTZ DEFAULT now(),
  PRIMARY KEY (run_id, test_id, attempt_index)
);

-- Classifier feature view: retry-aware signal
CREATE VIEW classifier_features AS
SELECT
  run_id,
  test_id,
  COUNT(*)                                         AS total_attempts,
  SUM(CASE WHEN outcome != 'passed' THEN 1 END)   AS failed_attempts,
  MAX(CASE WHEN outcome = 'passed' THEN 1 ELSE 0 END) AS eventually_passed,
  STDDEV(duration_ms)                              AS duration_stddev,
  MAX(attempt_index)                               AS max_attempt
FROM test_attempts
GROUP BY run_id, test_id;

The eventually_passed + failed_attempts combination is the feature that matters. A test with eventually_passed=1, failed_attempts=2 is a flake candidate, not a clean pass — and your classifier should be trained to treat it as such. At one platform team running ~14,000 test cases per pipeline, wiring this view into the classifier's input reduced misclassification of flaky tests as "resolved regressions" from 31% to under 6%, and triage time dropped from 18 minutes per flagged failure to under 3.

For teams using ClickHouse or BigQuery for test analytics at scale, the same logic applies — partition by (run_id, test_id) and aggregate attempt sequences before the classifier sees any row. If you're using Allure or ReportPortal, both support retry history natively in their data models; use their APIs to pull attempt-level records rather than scraping the summary report. Confidence scoring for AI failure classifiers becomes much more meaningful once the input features reflect actual attempt sequences rather than collapsed outcomes.

Where Experienced Teams Still Get the Pipeline Wrong

The most common mistake is deduplicating retries at the ETL layer before the classifier sees them. Engineers building data pipelines reasonably want one row per test per run for reporting dashboards — so they apply a DISTINCT ON (run_id, test_id) ORDER BY attempt_index DESC that keeps only the final attempt. That's correct for pass-rate KPI dashboards, but catastrophic for classifier training data. The deduplication logic belongs downstream of the classifier feature pipeline, not upstream of it. Separate your analytics schema from your classifier feature schema, even if both read from the same raw table.

The second mistake is training on production labels without auditing for retry contamination. Teams label historical failures manually — "this was a flake," "this was a regression" — and then train. But if the labeling happened against summary views that hid retry counts, the ground-truth labels are wrong. Before any retraining run, run a query against test_attempts to flag any training sample where failed_attempts > 0 AND eventually_passed = 1 and audit those labels manually. Garbage-in is the dominant failure mode here, not model architecture.

Myths About What Classifiers Can Infer from Pass/Fail Alone

Myth 1: A high-confidence "pass" label means the test is stable. Classifier confidence reflects how well the input matches the training distribution — not how reliable the test actually is. If the training distribution was contaminated with retry-laundered passes, a high-confidence pass label is just a confident wrong answer. Confidence scores are a measure of model certainty, not test health. Similarly, classifiers misread cascading failures for the same reason: they pattern-match on surface outcomes rather than execution context.

Myth 2: Retries are a CI configuration detail, not a data quality concern. Retry policies are a first-class signal. A test that passes on attempt 3 under a 2-second delay is behaviorally different from a test that passes on attempt 1 — and any system that treats them identically is modeling the wrong thing. This isn't about being precious about data purity; it's about the classifier's ability to distinguish an infrastructure hiccup from a genuine product regression. Teams that expose retry metadata as features consistently see better classifier precision on flake detection without any change to the underlying model architecture.

The core fix is unglamorous: don't collapse retry sequences before your classifier sees them. Instrument at the attempt level, build a feature view that exposes failed_attempts and eventually_passed as separate signals, and audit your training labels for retry contamination before retraining. If you're unsure whether your current pipeline has this problem, query your test results table for runs where final status is "passed" and retry count is greater than zero — the ratio of those rows to total rows is your contamination rate. Start there.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.