Flake Rate Compounds Across Pipeline Stages

Most teams treat flakiness as a per-test problem: find the offender, quarantine it, move on. What they miss is that flake rate is multiplicative across dependent stages. A unit stage with 2% flake probability feeding an integration stage with 3% flake probability feeding a deploy-gate stage with 1.5% flake probability doesn't give you a pipeline with ~2% flakiness — it gives you a pipeline that fails cleanly roughly 93.6% of the time, meaning one in sixteen runs fails for no real reason before a single line of production code is questioned.

The compounding effect is why engineering teams report that "we fixed the flaky tests" three sprints in a row and still have on-call engineers re-running pipelines at midnight. Individual test flake rates look manageable in isolation; the pipeline-level failure rate tells a different story.

By the end of this article you'll have the probability model, a SQL query to surface compounded flake risk per pipeline chain, and the operational patterns that keep the math from eating your release cadence.

The Probability Math Behind Pipeline-Level Flake Budgets

A flake budget is the maximum aggregate flake probability a pipeline chain can tolerate before the false-failure rate becomes operationally unacceptable — typically defined as the point where re-run cost exceeds the cost of the slowdown you're trying to prevent. If your team agrees that more than 5% of pipeline runs should not fail for non-deterministic reasons, your flake budget is 0.05. The budget isn't per test; it's per pipeline path from trigger to deploy gate.

The compounding model is straightforward: if stage i has a clean-pass probability p_i, the end-to-end clean probability is ∏ p_i. Three stages at 98% each yields 94.1% — already over a 5% flake budget. This is why flake rate averages hide your riskiest tests: a suite average of 0.8% flake looks fine until you realize it's spread across eight sequential stages, each contributing independently to the product. Where a stage fits in the dependency graph matters as much as its own flake rate.

Measuring Compounded Flake Risk in Your Actual Pipeline Data

Start by modeling your pipeline as a directed graph in your test results store. Each row in your runs table needs a pipeline_run_id, a stage_name, an upstream_stage_id, and a outcome (pass/fail/retry). With that schema in PostgreSQL or BigQuery, the following query computes per-stage flake rates and the rolling compounded clean probability across a 14-day window:

-- PostgreSQL: compounded clean probability per pipeline chain
WITH stage_stats AS (
  SELECT
    stage_name,
    upstream_stage_id,
    COUNT(*) FILTER (WHERE outcome = 'pass' AND retry_count = 0) AS clean_passes,
    COUNT(*) FILTER (WHERE outcome = 'pass' AND retry_count > 0) AS flake_passes,
    COUNT(*) AS total_runs
  FROM pipeline_stage_runs
  WHERE run_at >= NOW() - INTERVAL '14 days'
  GROUP BY stage_name, upstream_stage_id
),
flake_rates AS (
  SELECT
    stage_name,
    upstream_stage_id,
    1.0 - (flake_passes::float / NULLIF(total_runs, 0)) AS clean_prob
  FROM stage_stats
),
chained AS (
  SELECT
    f1.stage_name AS entry_stage,
    f2.stage_name AS exit_stage,
    f1.clean_prob * f2.clean_prob AS compounded_clean_prob
  FROM flake_rates f1
  JOIN flake_rates f2 ON f2.upstream_stage_id = f1.stage_name
)
SELECT
  entry_stage,
  exit_stage,
  ROUND((1 - compounded_clean_prob) * 100, 2) AS compounded_flake_pct
FROM chained
ORDER BY compounded_flake_pct DESC;

The join on upstream_stage_id is the key — it respects the actual dependency graph rather than treating stages as independent. Extend the CTE chain for three or four hops if your pipeline is deeper; each additional join multiplies the probabilities correctly. Feed the output into a Grafana table panel with a threshold at your flake budget (e.g., red above 5%) and you have a live compounded-risk dashboard without any new instrumentation.

For GitHub Actions specifically, emit stage outcomes as structured logs and ship them to Loki with the pipeline_run_id as a label. A LogQL query can then drive the same compounded-flake panel in real time:

# GitHub Actions step — emit structured outcome for Loki ingestion
- name: Emit stage outcome
  if: always()
  run: |
    echo "{\"pipeline_run_id\":\"${{ github.run_id }}\",\
    \"stage\":\"integration\",\
    \"outcome\":\"${{ job.status }}\",\
    \"retry_count\":${{ env.RETRY_COUNT }}}" \
    | curl -s -X POST "$LOKI_PUSH_URL" \
      -H "Content-Type: application/json" \
      --data-binary @-

Once this is wired up, triage time for "why did the pipeline fail" drops dramatically — one team reduced per-failure investigation from 22 minutes to under 4 by correlating compounded flake signals in Loki directly with the Grafana panel rather than manually diffing re-run logs. The compounded view also surfaces something averages never will: a stage with a modest 2% individual flake rate that sits at the head of six downstream stages is more expensive than a 6% flaky stage at the tail. Prioritize by position in the graph, not by raw rate. This ties directly into how pass rate drift hides broken stages — a head-stage flake inflates downstream failure counts and makes healthy stages look unreliable.

Where Teams Miscalculate Their Flake Budget

The most common mistake is defining the flake budget at the test level rather than the pipeline level. A policy like "no test may have flake rate above 2%" sounds rigorous but is mathematically incoherent — twenty tests each at 1.9% flake in a sequential stage produce a stage-level clean probability of roughly 68%. The budget must be set at the stage level first, then back-calculated to a per-test ceiling based on how many tests run in that stage. A 100-test stage targeting 99% stage-level clean probability needs each test to average below 0.01% flake — a very different target than a blanket 2% policy.

The second mistake is resetting flake history after a suite refactor or test rename without adjusting the budget model. Renaming a test resets its flake counter to zero in most tracking systems, which makes the compounded probability look artificially healthy. If you've ever wondered why flake rate resets after a suite refactor feel like a false win, this is the mechanism — the pipeline's actual reliability hasn't changed, but your metrics say it has. Carry forward flake history by stable test ID, not display name.

Myths That Keep Compounded Flake From Getting Fixed

Myth 1: Retries solve compounded flake. Retries reduce the user-visible failure rate but they don't reduce the compounded probability that a pipeline run will cost extra time. A three-retry policy on a 5%-flaky stage cuts visible failures but adds latency on every flaky run and masks the underlying signal — your compounded flake budget math now needs to account for retry cost, not just pass/fail. Retries are a coping mechanism, not a fix. Myth 2: Parallel stages don't compound. True for independent parallel stages, but most real pipelines have a fan-in gate (e.g., "all parallel jobs must pass before deploy"). Any flaky parallel branch that feeds a required fan-in contributes to the product. The compounded probability for a fan-in is ∏ p_i across all branches, same as sequential.

Myth 3: A low suite-level failure rate means compounding isn't a problem. Suite-level failure rate is a lagging, diluted metric — especially once you factor in that failure rate naturally drops as your suite grows, even if absolute flaky-test count rises. Pipeline-level compounded flake is the right denominator. Track the percentage of pipeline runs that required at least one re-run due to a non-deterministic failure; that number is what engineers and engineering leaders feel, and it's the number worth committing to reduce.

The compounding model isn't theoretical — it's why your "mostly green" suite still generates re-run tickets every week. Start with the SQL above against your existing runs table, set a pipeline-level flake budget, and then prioritize flake fixes by graph position rather than raw rate. If you're surfacing this data for stakeholders, the multi-pipeline visibility patterns for engineering leaders give you a framework for presenting compounded risk without drowning decision-makers in per-test noise.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.