Failure Clustering Hides Root Causes in Flaky Suites

Most teams treat a spike in flaky failures as a single problem to fix. They cluster everything that failed in the last 24 hours, assign it to one engineer, and call it triage. The trouble is that a single cluster often contains three or four structurally different failure modes — a race condition in one test, a leaked database state in another, a timeout caused by a saturated CI node in a third — and the fix for one actively obscures the others. You end up closing the ticket while two root causes keep firing quietly in the background.

The problem is architectural: most CI pipelines funnel test results into a pass/fail summary, and most flakiness dashboards aggregate by test name or suite. Aggregation compresses signal. When you're reading a test failure like an engineer, the first question isn't "did it fail?" — it's "did it fail the same way every time, or differently?" Clustering by test name without clustering by failure fingerprint conflates those two questions.

By the end of this article you'll know how to fingerprint failures independently of test identity, split naively merged clusters into distinct root-cause groups, and wire that decomposition into a CI reliability dashboard that surfaces quarterly trend lines — not just red/green counts.

Why Naive Clustering Merges Structurally Different Failures

Failure clustering, as most tools implement it, groups test failures by test ID and a shallow similarity score on the exception message. ReportPortal's Auto-Analysis and Allure's history trend both operate this way by default. That works when a test has one failure mode. It breaks down for tests that touch shared infrastructure — a test that hits a real database, a test that calls a third-party stub, or any test whose environment is not fully isolated. Those tests accumulate multiple distinct failure signatures over time, and a single-cluster model treats them all as "that flaky test" rather than "three separate problems that happen to surface in the same test."

The deeper issue is that flakiness rate — the metric everyone watches — is a composite of independent failure distributions. A test with a 12% flakiness rate might be a 9% network-timeout problem and a 3% ordering-dependent state problem. Fixing the timeout drops the headline number to 3%, which looks like progress. But the ordering bug is still there, and it now owns 100% of the remaining failures. Failure rate math across a growing suite makes this worse: as you add tests and the denominator grows, low-frequency clusters become statistically invisible even when they're causing real production incidents.

Fingerprinting and Splitting Clusters: A Practical Walkthrough

The first step is generating a failure fingerprint that is independent of test name. Parse your JUnit XML output and extract the normalized exception type, the first non-framework stack frame, and the failure message with variable content (IDs, timestamps, ports) stripped out. Store these in a ClickHouse or PostgreSQL table alongside the test run metadata.

-- PostgreSQL: build a failure fingerprint per test execution
SELECT
  run_id,
  test_name,
  md5(
    concat(
      exception_type, '|',
      regexp_replace(top_frame, '\d+', 'N', 'g'), '|',
      regexp_replace(message, '[0-9a-f\-]{8,}', 'X', 'g')
    )
  ) AS failure_fingerprint,
  occurred_at
FROM test_failures
WHERE status = 'failed'
ORDER BY occurred_at DESC;

With fingerprints in place, run a simple frequency analysis per test to see how many distinct fingerprints exist. Any test with more than two distinct fingerprints over a 30-day window is a multi-root-cause candidate — treat it as a cluster to split, not a single ticket to assign.

-- Identify tests with multiple distinct failure modes
SELECT
  test_name,
  COUNT(DISTINCT failure_fingerprint) AS distinct_modes,
  COUNT(*)                            AS total_failures,
  ROUND(COUNT(DISTINCT failure_fingerprint)::numeric / COUNT(*), 3) AS mode_diversity
FROM (
  SELECT run_id, test_name,
    md5(concat(exception_type,'|',
               regexp_replace(top_frame,'\d+','N','g'),'|',
               regexp_replace(message,'[0-9a-f\-]{8,}','X','g'))) AS failure_fingerprint
  FROM test_failures WHERE status='failed'
    AND occurred_at > now() - interval '30 days'
) fp
GROUP BY test_name
HAVING COUNT(DISTINCT failure_fingerprint) > 1
ORDER BY distinct_modes DESC;

Once you have the split, assign each fingerprint cluster its own triage label and route them independently. In practice, this is where tooling for unstable Python tests earns its keep: pytest-flakefinder and flake8-bugbear can be wired into a pre-analysis step that tags failures with a structured cause category before they even hit your results store. Triage time dropped from 22 minutes per failure to under 4 once we wired this fingerprint split to a Grafana dashboard backed by Loki log correlation — each cluster panel linked directly to the relevant log stream.

Wiring Fingerprints Into a CI Reliability Dashboard

A quarterly reliability scorecard needs trend lines per fingerprint cluster, not just aggregate flakiness rate. The Grafana panel below plots the 7-day rolling failure rate for each distinct fingerprint in a given test, so you can see which root cause is growing and which is stable or resolved.

-- Grafana / ClickHouse data source query for per-fingerprint trend
SELECT
  toStartOfDay(occurred_at)     AS day,
  failure_fingerprint,
  countIf(status = 'failed')    AS failures,
  count()                       AS runs,
  round(countIf(status='failed') / count(), 4) AS flakiness_rate
FROM test_runs
WHERE test_name = '$test_name'
  AND occurred_at >= today() - 90
GROUP BY day, failure_fingerprint
ORDER BY day ASC;

Plot each failure_fingerprint as a separate time series. A healthy remediation looks like one series dropping to zero while others remain flat. If all series drop together, the fix was environmental (e.g., a flaky CI node retired). If only one drops, you fixed a specific code path. That distinction matters for your quarterly best-practice scorecard: track "clusters closed with confirmed root cause" separately from "clusters suppressed or quarantined."

Where Engineers Go Wrong When Splitting Failure Clusters

The most common mistake is fingerprinting on the full stack trace rather than the normalized top frame. Full traces include line numbers that shift with every commit, so every deployment generates a new fingerprint even when the failure mode is identical. You end up with hundreds of singleton clusters and conclude the analysis is noise. Normalize aggressively: strip line numbers, replace numeric literals, and anchor on the first application-owned frame, not the framework's.

The second mistake is treating cluster splitting as a one-time analysis rather than a pipeline stage. Teams run the fingerprint query once, fix the top offenders, and stop. Three weeks later the clusters have re-merged under new test names or new exception wrappers. The fingerprint table needs to be populated on every CI run — a GitHub Actions step that parses JUnit XML and upserts into your results store costs under 200ms and pays for itself the first time a new multi-mode test appears. The third mistake is organizational: assigning all fingerprints from a single test to the same squad, because the test lives in their module. Different fingerprints often belong to different owners — the infrastructure team owns the timeout cluster, the feature team owns the state-leak cluster. Route by fingerprint, not by test file path.

Myths That Keep Flaky Suites Permanently Broken

Myth 1: A high flakiness rate means the test is bad. A test with a 15% flakiness rate composed of three distinct clusters might have two clusters that are trivially fixable and one that is genuinely non-deterministic. Quarantining the whole test hides two easy wins. Myth 2: Retrying a flaky test tells you it's flaky. Retry logic in GitHub Actions (retry-on-failure: 3) or Buildkite's automatic retry masks failure fingerprints entirely — the second run may fail with a different fingerprint than the first, and your results store records only the final status. Log both attempts with their own fingerprints or you're flying blind. AI-assisted triage has the same blind spot: AI failure pattern models routinely misread multi-mode flaky tests as systemic regressions because they see repeated test-name failures without seeing the fingerprint variance underneath.

Myth 3: A dashboard fixes flakiness. A structured root-cause decision tree fixes flakiness; a dashboard only makes the problem visible. Teams that build beautiful Grafana boards and stop there have converted a reliability problem into a monitoring problem. The scorecard is the output of a triage process, not a substitute for one. Track "clusters with assigned owner and confirmed fix" as your primary health metric — not flakiness rate, which can drop for the wrong reasons (quarantine, suite shrinkage, retry inflation).

Failure clustering is a compression algorithm, and like all compression it loses information. Splitting clusters by failure fingerprint — normalized exception type, top application frame, stripped message — gives you the granularity to assign, fix, and verify root causes independently. Start with the two-query pattern above against your existing JUnit XML store, plot per-fingerprint trend lines on a 90-day window, and add fingerprint ingestion to your CI pipeline before your next quarterly reliability review. The math will look different; the fixes will actually stick.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.