Most flaky-test triage starts and ends with retry rate. That's a trap. Two tests can have identical retry rates and completely different root causes — one is a race condition in your app code, the other is a DNS timeout on an overloaded CI runner. Treating them the same wastes engineering time and trains your team to ignore the signal entirely.

Failure sequencing is the practice of analyzing when in a run failures appear — their ordinal position, their timing relative to suite start, and how they cluster across concurrent workers — to distinguish infrastructure-induced flakes from logic-induced ones. Infrastructure flakes have a different temporal fingerprint than logic flakes, and that fingerprint is recoverable from data you already collect.

By the end of this article you'll have a SQL-based detection query, a Python classifier sketch, and a mental model for reading failure sequences that will cut misclassification in half without adding a single new instrumentation point.

The Two Failure Classes and Why Sequence Exposes Them

Infrastructure flakes originate outside your application logic: runner OOM kills, shared-resource contention (ports, temp dirs, Docker socket), network timeouts to external services, and clock skew between test nodes. They tend to cluster at the beginning of a run (cold-start resource pressure), at the end (resource exhaustion), or in bursts across workers when a shared dependency degrades. They also correlate with absolute wall-clock time — 3 AM UTC when your cloud provider does maintenance, or Monday mornings when the shared Postgres test cluster gets hammered. Test duration variance is often the first visible symptom before the failures even start.

Logic flakes originate in application code: async timing assumptions, test-order-dependent state, missing teardown, or race conditions in the system under test. They appear at consistent ordinal positions across runs, are reproducible on the same machine with the same seed, and their duration signature is stable — the test runs for roughly the same time before failing. The failure message is deterministic even if the trigger is not. Knowing which class you're dealing with changes everything about remediation: infrastructure flakes need platform fixes, logic flakes need code fixes. Conflating them means neither gets fixed properly.

Building a Sequence-Based Classifier Against Your Results Store

Start with the data shape. Every JUnit XML report carries a timestamp per test case and a suite-level start time. If you're ingesting into PostgreSQL, ClickHouse, or BigQuery, you should have at minimum: run_id, test_name, suite_start_ts, test_start_ts, duration_ms, status, worker_id. The derived column you need is ordinal position within the run and offset from suite start.

-- PostgreSQL: compute failure offset and ordinal per run
WITH ranked AS (
  SELECT
    run_id,
    test_name,
    status,
    worker_id,
    duration_ms,
    EXTRACT(EPOCH FROM (test_start_ts - suite_start_ts)) AS offset_seconds,
    ROW_NUMBER() OVER (PARTITION BY run_id, worker_id ORDER BY test_start_ts) AS ordinal
  FROM test_results
  WHERE status = 'failed'
    AND run_ts >= NOW() - INTERVAL '14 days'
)
SELECT
  test_name,
  COUNT(*)                          AS fail_count,
  AVG(offset_seconds)               AS avg_offset_s,
  STDDEV(offset_seconds)            AS stddev_offset_s,
  AVG(ordinal)                      AS avg_ordinal,
  STDDEV(ordinal)                   AS stddev_ordinal,
  COUNT(DISTINCT worker_id)         AS distinct_workers
FROM ranked
GROUP BY test_name
HAVING COUNT(*) >= 5
ORDER BY stddev_offset_s DESC;

High stddev_offset_s with high distinct_workers is the infrastructure flake signature — the test fails at unpredictable times across many workers, which points to a shared resource. Low stddev_offset_s but consistent avg_ordinal is a logic flake — it always fails at roughly the same point in the same worker's sequence, meaning something upstream is polluting state.

# Python classifier — runs after the SQL pull
import pandas as pd

def classify_flake(df: pd.DataFrame) -> pd.DataFrame:
    df = df.copy()
    # Infrastructure signal: high timing variance, multi-worker spread
    df["infra_score"] = (
        df["stddev_offset_s"] / (df["avg_offset_s"] + 1)
    ) * df["distinct_workers"]
    # Logic signal: low timing variance, consistent ordinal
    df["logic_score"] = 1 / (df["stddev_ordinal"] + 0.1)
    df["classification"] = df.apply(
        lambda r: "infra" if r["infra_score"] > r["logic_score"] else "logic",
        axis=1
    )
    return df[["test_name", "fail_count", "infra_score", "logic_score", "classification"]]

Wire this classifier into your CI post-processing step. In GitHub Actions, run it as a job that depends on your test matrix, reads the uploaded JUnit artifacts, and posts a summary to Slack with counts by class. One team running this against a 4,200-test Pytest suite reduced mean triage time from 22 minutes per failure to under 5 — not because the classifier was perfect, but because engineers stopped debating root cause and started from a defensible hypothesis. For deeper log correlation once you have a candidate, wiring Grafana and Loki to your failure timeline closes the loop between the test signal and the infrastructure signal.

Where Sequence Analysis Breaks Down in Practice

Parallelism without worker tagging is the most common data quality failure. If your CI runner doesn't emit a stable worker_id or shard index into the JUnit XML, all your ordinal math collapses into noise. Pytest-xdist writes worker IDs into the XML by default; Playwright's built-in reporter does not. Fix this at the reporter layer before you invest in the analytics layer — a missing column cannot be imputed later. Similarly, if your suite randomizes test order per run (which you should do for isolation), store the seed so you can reconstruct ordinal position deterministically for comparison across runs.

Treating classification as binary is the second mistake. Some tests are both: a logic race condition that only triggers under resource pressure is neither cleanly infra nor cleanly logic. The scores above are continuous — use them as a triage priority signal, not a hard label. Teams that hard-label and route to separate queues ("infra team owns infra flakes") end up with the hybrid cases falling through the cracks indefinitely. Keep a "mixed" bucket and review it weekly. Preventing flake recurrence requires understanding which class you fixed and whether the fix addressed the actual trigger.

Myths That Keep Teams Stuck on Retry-Rate Dashboards

"Flakiness is a test problem, not an infrastructure problem." This is the most expensive myth in CI reliability. When a runner runs out of ephemeral disk space at test 847 of 900 every Tuesday, the test suite looks flaky, but no amount of test refactoring fixes it. Sequence analysis surfaces this by showing the failure cluster at high ordinals on specific days — a pattern invisible in a simple pass/fail trend. The same logic applies to shared test databases under connection-pool pressure: the failures look random until you plot them against worker count and suite offset simultaneously. Where in the pipeline a failure appears changes what it means — the same principle applies within a single run.

"More retries reduce flake impact." Retries mask infrastructure flakes and make them harder to classify — a test that passes on retry has no failure sequence entry to analyze. Worse, aggressive retry policies on infra flakes burn runner minutes without fixing anything. Use retries sparingly (max 1, never on the first run), log every retry attempt with its ordinal and offset, and treat retry-pass as a weak positive signal for infra flake classification rather than a clean result. The goal is signal fidelity, not green dashboards.

Failure sequencing won't replace deep debugging, but it will stop your team from spending three hours on a DNS timeout because someone assumed it was a test isolation bug. Start by adding worker_id and suite_start_ts to your JUnit ingestion pipeline this week, run the SQL above against two weeks of history, and see whether your highest-retry tests cluster by offset or by ordinal. The answer will tell you where to spend the next sprint. For the broader pattern-recognition framework this fits into, the piece on reading a test failure like an engineer is worth the twenty minutes.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.