AI Flakiness Detection: What Actually Works

Most teams define a flaky test the same way: it failed, you reran it, it passed, you moved on. That workflow buries the signal. Retry counts, failure-rate trends over rolling windows, correlated failures across test classes, time-of-day variance — none of that surfaces from a simple pass/fail log scrape. The teams that actually reduce flakiness aren't running smarter retries; they're running smarter analysis on the history those retries leave behind.

The promise of AI for test flakiness detection is real but narrowly scoped. Classification models and embedding-based clustering genuinely outperform static threshold rules at separating environmental noise from deterministic bugs — but only when the underlying data pipeline is clean. Feed a model noisy JUnit XML with missing timestamps and merged retry results, and you get confident garbage.

This article covers what actually works in production: the data shapes that enable useful models, the queries and Python snippets that operationalize detection, and the failure modes that make most AI flakiness tooling land as shelfware six months after rollout.

What AI Flakiness Detection Actually Measures

AI flakiness detection is the practice of applying statistical models — classifiers, anomaly detectors, or embedding-based similarity search — to test execution history in order to distinguish non-deterministic failures from genuine regressions. It is not a replacement for root-cause analysis; it is a triage layer that routes failures to the right queue faster. A well-tuned model tells you "this test has a 91% historical flake probability on this runner image" before an engineer spends 20 minutes bisecting a red build.

In a modern test architecture it sits between your result store (ClickHouse, BigQuery, PostgreSQL, or a purpose-built layer like ReportPortal) and your alerting surface (PagerDuty, Slack, Grafana). The three structural patterns of flakiness — timing, ordering, and environment — each produce distinct feature signatures in execution history, and that's exactly what a classifier can exploit. Without this layer, every red build carries equal urgency, which means none of them do.

Building a Detection Pipeline That Earns Its Place

Start with the feature set. The most predictive features for a gradient-boosted classifier (XGBoost or LightGBM both work well here) are: failure rate over the last 30 runs, pass-after-retry rate, failure rate variance by runner/agent label, and co-failure count with other tests in the same suite run. Pull these from your result store before you touch any model.

-- ClickHouse: per-test flakiness features over a 30-day window
SELECT
    test_name,
    countIf(status = 'FAILED') / count()            AS failure_rate,
    countIf(retry_passed = 1) / countIf(status = 'FAILED') AS retry_pass_rate,
    uniqExact(runner_label)                          AS runner_variety,
    stddevPop(duration_ms)                           AS duration_stddev
FROM test_runs
WHERE run_at >= now() - INTERVAL 30 DAY
GROUP BY test_name
HAVING count() >= 10   -- ignore low-volume tests
ORDER BY failure_rate DESC;

Feed that result set into a lightweight Python classifier. You don't need a neural network here — a random forest with 50 estimators trained on 90 days of labeled history (label: "engineer marked as flaky in Jira/Linear") will hit F1 > 0.85 on most mid-size suites. The key is labeling quality: if your team retries silently without tagging, your training set is corrupted. Fix the silent retry misconfiguration before you train anything.

# Python: train and score flakiness classifier
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

df = pd.read_csv("test_features_90d.csv")
features = ["failure_rate","retry_pass_rate","runner_variety","duration_stddev"]
X, y = df[features], df["is_flaky"]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
clf = RandomForestClassifier(n_estimators=50, class_weight="balanced", random_state=42)
clf.fit(X_train, y_train)
print(classification_report(y_test, clf.predict(X_test)))

For teams whose suites are too large or too young for supervised labeling, embedding-based clustering is the better entry point. Convert each test's failure-message text to embeddings (OpenAI's text-embedding-3-small or a local sentence-transformer both work), then run DBSCAN or HDBSCAN to surface clusters of semantically similar failures. Tests that cluster tightly but span unrelated features are almost always environment-driven flakes, not regressions. This is the approach described in depth in pattern detection using embeddings. One team using this on a 4,000-test Playwright suite reduced triage time from 22 minutes per failure to under 4 once the cluster labels were wired into their Grafana oncall dashboard.

Wire the scored output back into CI as a GitHub Actions step that annotates the PR with flake probability, or push it to a Slack channel via webhook. The annotation doesn't block the build — it changes the human decision. A score above 0.80 routes to a "likely flake" board; below 0.40 triggers a Sentry issue for immediate investigation.

Where Detection Pipelines Break Down in Practice

The most common failure mode is training on aggregated results instead of individual run records. If your ETL collapses retries into a single "final status" row before the feature query runs, your retry_pass_rate feature is always null and your model learns nothing useful. This happens because most JUnit XML consumers (Allure, the Jenkins JUnit plugin) are designed to show you the final outcome, not the full retry trace. Preserve raw attempt-level records in your result store; aggregate only at query time.

The second pitfall is ignoring test-ordering effects. A classifier trained only on individual test metrics will misclassify order-dependent failures as random flakes because they appear non-deterministic in isolation. Before scoring, check co-failure graphs: if test B always fails within the same run as test A, that's a dependency chain, not independent flakiness. Order dependency poisons suite-wide noise in exactly this way, and no amount of model tuning compensates for a missing feature that captures it. Add a co-failure count column to your feature set.

Myths That Keep Teams Stuck on Manual Triage

Myth 1: A high flake score means you can ignore the failure. A flake probability is a routing signal, not a suppression switch. Tests with a 0.90 flake score still fail for real reasons roughly 10% of the time — and those real failures tend to be the most interesting ones. Teams that auto-quarantine high-score tests without review accumulate silent regressions. Use the score to deprioritize, never to discard. Myth 2: More training data always improves detection. Beyond roughly 90 days of history, most suites show concept drift — the test code, the infra, and the app all change, making old failure patterns misleading. A rolling 60–90 day window with periodic retraining outperforms a two-year dataset trained once.

Myth 3: AI detection replaces the need for structured observability. Models surface which tests are flaky. They don't tell you why. For that you still need correlated traces, Loki log queries keyed to test run IDs, and CI observability metrics that go beyond flaky-test counts. The detection layer and the observability layer are complementary, not substitutes. Teams that ship a classifier and call the problem solved typically see flake rates plateau rather than decline, because they've improved routing without improving the feedback loop that drives fixes.

AI flakiness detection earns its keep when it's a thin, well-fed layer on top of clean execution data — not a black box bolted onto a broken pipeline. Start with the ClickHouse feature query above, validate your retry data hygiene, and train on labeled history before reaching for embeddings or LLMs. Once detection is stable, the next lever is root-cause analysis with a structured decision tree — that's where detection turns into resolution.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.