Flake Rate Resets After a Suite Refactor

You refactor the test suite — split a 4,000-test monolith into domain-scoped modules, rename half the specs, consolidate fixtures — and suddenly the flake rate dashboard shows a dramatic improvement. The number drops from 6.2% to 1.8% overnight. Nobody fixed a single flaky test. What happened is a measurement artifact, not a reliability improvement, and acting on it as signal will cost you the next release.

The root cause is deceptively simple: most flake rate calculations are keyed to test identity, and a refactor changes test identity. Historical failure data no longer maps to the new test names, so the denominator resets with zero prior runs and zero recorded flakes. The metric looks healthy because it has no memory.

This article explains exactly why that reset happens at the data layer, how to build a flake tracking system that survives refactors, and what a flake budget actually means as an operational concept — not a vanity target.

How Test Identity Breaks Flake History

Flake rate is a rolling statistic: flaky runs / total runs over some window, grouped by test identifier. In JUnit XML that identifier is typically classname + testname. In Pytest it's the node ID — tests/checkout/test_cart.py::TestCart::test_add_item. When you move that file, rename the class, or flatten the module hierarchy, the node ID changes. Your results store — whether that's a PostgreSQL table, BigQuery dataset, or ReportPortal — now sees a brand-new test with zero history, and the old broken record becomes an orphan with no current runs to inflate the rate.

This is not a bug in your tooling; it's a consequence of using mutable string keys as test identity. The fix requires either a stable content-based identifier (a hash of the test's logical intent, which is hard to define) or an explicit migration step that re-keys historical data when names change. Without one of those two mechanisms, every significant refactor is a flake-rate amnesia event. As a side effect it also distorts how suite composition skews the metrics you report up to leadership — the portfolio looks cleaner than it is.

Tracking Flake Budget Across Refactors

What is a flake budget? A flake budget is an explicit, time-boxed allowance for tolerated non-determinism across the suite — expressed as a count of flaky tests or a rate ceiling, with a defined remediation SLA when the budget is exceeded. It's the difference between "we know we have 14 flaky tests and we're burning down to 8 by end of sprint" and "our dashboard says 1.8% so we're fine." The budget survives refactors because it lives in a separate tracking store that is not reset by test renaming.

The simplest durable implementation is a flake_registry table with a stable flake_id column that you control, decoupled from the test's display name:

-- PostgreSQL: flake registry decoupled from test node ID
CREATE TABLE flake_registry (
  flake_id        UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  canonical_name  TEXT NOT NULL,          -- human-readable, mutable
  node_id_pattern TEXT NOT NULL,          -- regex; update on rename
  first_seen      TIMESTAMPTZ NOT NULL,
  last_flake_run  TIMESTAMPTZ,
  status          TEXT CHECK (status IN ('active','watch','resolved')),
  owner_team      TEXT
);

-- Join to run results by pattern match, not exact key
SELECT
  fr.flake_id,
  fr.canonical_name,
  COUNT(*) FILTER (WHERE tr.outcome = 'flaky') AS flake_count,
  COUNT(*)                                      AS total_runs,
  ROUND(
    COUNT(*) FILTER (WHERE tr.outcome = 'flaky')::numeric / COUNT(*) * 100, 2
  ) AS flake_pct
FROM test_runs tr
JOIN flake_registry fr
  ON tr.node_id ~ fr.node_id_pattern
WHERE tr.run_at > NOW() - INTERVAL '30 days'
GROUP BY fr.flake_id, fr.canonical_name
ORDER BY flake_pct DESC;

When a refactor renames tests/checkout/test_cart.py::TestCart::test_add_item to tests/cart/test_add_item.py::test_add_item, you update node_id_pattern in the registry — one row, one migration — and 90 days of flake history stays intact. The flake_id UUID never changes. Triage time dropped from 22 minutes per failure to under 4 once a team wired this registry to a Grafana dashboard backed by Loki log correlation, because engineers stopped re-investigating tests they had already triaged under a previous name.

For the CI side, emit a structured annotation in GitHub Actions that includes the stable flake_id when a retry succeeds:

# .github/workflows/test.yml (excerpt)
- name: Run Pytest with flake annotation
  run: |
    pytest --reruns 2 --reruns-delay 1 \
      --json-report --json-report-file=results.json
  env:
    FLAKE_REGISTRY_URL: ${{ secrets.FLAKE_REGISTRY_URL }}

- name: Annotate flaky runs
  if: always()
  run: |
    python scripts/annotate_flakes.py \
      --report results.json \
      --registry $FLAKE_REGISTRY_URL \
      --build-id ${{ github.run_id }}

annotate_flakes.py queries the registry by node_id_pattern, resolves the stable flake_id, and writes a structured log line that Loki can ingest. The key insight: retry count inflates pass rate without fixing flakes, so you need this annotation layer to distinguish "passed on retry with known flake ID" from "genuinely passed." Without it, the budget accounting is wrong from the start.

Where Flake Tracking Breaks Down in Practice

The most common mistake is treating the refactor as a cleanup opportunity and archiving old flake data rather than migrating it. The reasoning is usually "those tests are gone anyway" — but the failure modes aren't gone, only the names are. A timing-sensitive database test that was flaky under one name will be flaky under its new name within two weeks. Archiving the history means two weeks of false confidence before the budget alarm fires again. Keep the data; update the key.

The second mistake is setting a flake budget as a rate ceiling without controlling for how failure rate drops as your suite grows. A 2% flake rate on 500 tests is 10 flaky tests. A 2% rate on 2,000 tests is 40 — a meaningfully different CI tax — but both "pass" the budget check. Express the budget as an absolute count with a rate ceiling, not a rate alone. Teams that skip this end up with dashboards that show compliance while the actual flaky-test backlog quietly doubles.

Myths That Survive Every Suite Refactor

Myth 1: A lower flake rate after a refactor means the refactor improved reliability. It means test identity changed and history reset. Verify by checking how many active flake_registry entries have status = 'resolved' versus status = 'active' with a new node_id_pattern. If the resolved count didn't move, no reliability work happened. Relatedly, flake rate averages hide your riskiest tests even when the history is intact — post-refactor, the average is even less trustworthy.

Myth 2: A full suite rewrite resets the flake budget to zero and that's acceptable. It resets the measurement, not the underlying instability. Async timing issues, shared database state, and environment-sensitive assertions travel with the logic they test. A rewrite is a good time to audit and resolve flakes explicitly — not to declare amnesty by default. Teams that treat the rewrite as a clean slate typically see flake rates climb back to pre-rewrite levels within one quarter, with no institutional memory of which tests were already triaged.

The next concrete step: before your next refactor, export every active flake from your results store into a registry with stable IDs, and write the migration script that re-keys node_id_pattern as part of the rename PR. Treat it the same way you'd treat a database column rename — with an explicit migration, not a silent drop. If you want to pressure-test your current flake accounting, query how many tests in your last 30-day window have fewer than 10 recorded runs; that's your reset surface area.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.