Most teams know they have flaky Python tests. What they don't have is a systematic way to answer why — is it thread-unsafe shared state, a leaky database fixture, time-dependent logic, or just a slow external call racing against a hard timeout? Grep-and-guess costs 20+ minutes per incident and produces no institutional memory. The signal you need is already in your CI output; you're just not storing or querying it yet.

The problem is architectural: JUnit XML gets uploaded, Allure renders a pretty report, and the history disappears after 30 days. Without a queryable time-series of test outcomes enriched with metadata — worker node, Python version, test duration, retry count, error class — you can't distinguish a genuinely broken test from one that fails only on the second retry, only on ARM runners, only after test_db_migration runs first.

By the end of this article you'll have a working tool: a pytest plugin that emits structured failure metadata, a ClickHouse schema to store it, SQL queries that surface root-cause clusters, and a quarterly reliability scorecard template you can drop into Grafana or a Google Sheet.

What "Unstable" Actually Means in a Python Test Suite

Instability is not a binary. A test that fails 2% of the time on main but 40% of the time on a specific CI runner has a different root cause than one that fails intermittently regardless of environment. Root-cause taxonomy matters before you write a single line of tooling: ordering dependencies (test A pollutes global state for test B), resource contention (thread pool exhaustion under pytest-xdist parallelism), external coupling (HTTP calls without VCR cassettes or a mock server), and time sensitivity (assertions against datetime.now() or sleep-based waits) are four distinct failure modes that require four distinct fixes.

In a modern test architecture, this tool sits between your CI runner and your observability stack. Pytest emits structured events; a lightweight collector enriches them with runner metadata; a columnar store (ClickHouse, BigQuery, or even PostgreSQL with partitioning) holds the history; and a dashboard layer surfaces trends. The identification step — confirming a test is flaky — is a prerequisite, but root-cause analysis is the harder, higher-value layer on top of it.

Building a Root-Cause Analyzer: Plugin, Schema, and Queries

Start with a conftest.py hook that captures structured failure data at the point of failure — not just the message, but the exception type, the worker ID, the full nodeid, and wall-clock duration. This is the raw signal everything else depends on.

# conftest.py
import json, os, time, pytest

_results = []

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    rep = outcome.get_result()
    if rep.when == "call":
        exc_type = None
        if rep.failed and call.excinfo:
            exc_type = call.excinfo.type.__name__
        _results.append({
            "nodeid":      item.nodeid,
            "outcome":     rep.outcome,          # "passed" | "failed" | "error"
            "exc_type":    exc_type,
            "duration_s":  rep.duration,
            "worker":      os.environ.get("PYTEST_XDIST_WORKER", "main"),
            "runner_os":   os.environ.get("RUNNER_OS", "unknown"),
            "python":      os.environ.get("PYTHON_VERSION", "unknown"),
            "run_id":      os.environ.get("GITHUB_RUN_ID", "local"),
            "timestamp":   int(time.time()),
        })

def pytest_sessionfinish(session, exitstatus):
    with open("test_results.jsonl", "a") as f:
        for r in _results:
            f.write(json.dumps(r) + "\n")

Ship that JSONL to ClickHouse after every run. The schema below partitions by month and keeps a ReplacingMergeTree so re-runs don't double-count:

CREATE TABLE test_runs (
    run_id       String,
    nodeid       String,
    outcome      LowCardinality(String),
    exc_type     LowCardinality(String),
    duration_s   Float32,
    worker       LowCardinality(String),
    runner_os    LowCardinality(String),
    python       LowCardinality(String),
    ts           DateTime
) ENGINE = ReplacingMergeTree(ts)
PARTITION BY toYYYYMM(ts)
ORDER BY (nodeid, run_id);

Now the root-cause queries become straightforward. This one surfaces tests whose failure rate is statistically correlated with a specific worker — the clearest signal of ordering or resource-contention bugs under pytest-xdist:

SELECT
    nodeid,
    worker,
    countIf(outcome = 'failed') AS failures,
    count()                     AS total,
    round(failures / total, 3)  AS flake_rate
FROM test_runs
WHERE ts >= now() - INTERVAL 30 DAY
GROUP BY nodeid, worker
HAVING flake_rate > 0.05 AND total > 20
ORDER BY flake_rate DESC
LIMIT 40;

A second query clusters by exc_type to separate AssertionError (logic flakiness) from TimeoutError / ConnectionRefusedError (infrastructure coupling) from FixtureError (test ordering). Teams that wired this dashboard to Loki for log correlation cut triage time from over 20 minutes per failure to under 4 — because the exception class alone eliminates two of the four root-cause categories before anyone opens a log file. Pair this with a systematic remediation workflow and you're closing loops in days rather than quarters.

Pitfalls: Where Root-Cause Tooling Usually Breaks Down

Storing only the final retry outcome is the most common mistake. If pytest-rerunfailures retries three times and the test eventually passes, the run is marked green and the failure disappears from your dataset. You need to capture every attempt — the hook above does this, but only if pytest-rerunfailures is configured with --report-log or your hook explicitly handles rep.wasxfail and retry markers. Without that, your flake rate is systematically understated, sometimes by 40–60% on a suite with aggressive retry policies.

The second pitfall is treating all workers as equivalent. In GitHub Actions matrix builds, ubuntu-latest images are periodically updated mid-sprint; a test that starts failing on week 3 of a sprint may be reacting to a glibc bump, not a code change. Storing RUNNER_OS and the image digest (available via $ImageVersion on hosted runners) lets you correlate failures with runner image versions — a query that has surfaced real infrastructure bugs that would otherwise have been blamed on test code. The cost of misattributing those failures to developers is not just wasted time; it erodes trust in the test suite itself.

Myths That Keep Python Flakiness Unfixed

Myth 1: A low average flake rate means the suite is healthy. A 1.5% average across 2,000 tests can mask five tests each failing 60% of the time — tests that sit on critical paths and block releases weekly. Averages are the wrong aggregation; you want a per-test flake rate ranked by CI-blocking impact. Myth 2: Quarantining flaky tests is a fix. Quarantine is a circuit breaker, not a resolution. Without root-cause data feeding a remediation backlog with SLO-style targets — "this test returns to the suite within two sprints or is deleted" — quarantine becomes a graveyard. Flake rate averages hide your riskiest tests precisely because the worst offenders are diluted by hundreds of stable ones.

Myth 3: More retries solve the problem. Retries reduce visible red builds; they don't reduce the underlying failure rate, and they inflate CI duration. A suite that relies on three retries to stay green is spending real compute budget hiding a signal. The right response to a retry-dependent test is root-cause analysis, not a higher --reruns count. Aligning your test reliability targets with broader SLO-driven testing practices gives those targets teeth — a flake rate above threshold triggers a ticket, not just a re-run.

The quarterly reliability scorecard closes the loop: pull the top-20 flaky tests by blocking impact, group them by root-cause category (ordering, infrastructure, time-sensitivity, external coupling), assign owners, and track trend lines sprint over sprint. A Grafana time-series panel on flake_rate per category — not per test — shows whether your remediation work is moving the needle structurally. Start with the ClickHouse schema above, run the worker-correlation query against last quarter's data, and you'll have your first scorecard within a day.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.