GitLab CI's Flaky Test Widget, Reviewed

Most teams treat test results like a checkbox: green is good, red is bad, ship or block. The interesting signal lives in everything between those two states — runtime variance, retry counts, the same five tests showing up in every postmortem. GitLab CI ships a built-in flaky test widget that surfaces some of that signal without any extra tooling. Whether it's enough depends on how seriously your team is taking reliability as an engineering discipline.

The widget landed in GitLab 13.4 and has been quietly iterated on since. It ingests JUnit XML from your pipeline artifacts, tracks tests that passed on retry within the same pipeline run, and surfaces them in the merge request and project-level test reports. The data model is straightforward; the gaps are where things get interesting.

By the end of this review you'll know exactly what the widget tracks, where its detection logic breaks down, how to extend it with a SQL-backed dashboard and quarterly scorecard, and when you should route around it entirely.

What GitLab's Flaky Test Widget Actually Tracks

GitLab defines a test as flaky when it fails on at least one attempt and passes on a subsequent attempt within the same pipeline run. That definition is narrow on purpose — it keeps false-positive rates low — but it means tests that fail consistently across two consecutive pipelines and then pass on the third are invisible to the widget. The detection window is a single pipeline, not a rolling time series.

Under the hood, GitLab stores flaky test records in the ci_flaky_test_results table (GitLab.com) or your self-managed PostgreSQL instance. Each record carries the test suite name, test case name, branch, pipeline ID, and a flaky_runs counter. The project-level Test Cases page aggregates these into a sortable list. For a deeper look at how the full JUnit XML ingestion pipeline feeds this table, the end-to-end GitLab CI test reports walkthrough covers the artifact upload path and parser behavior in detail.

Building a Reliability Dashboard on Top of the Widget's Data

The widget's UI is fine for a quick triage pass, but it doesn't give you trend lines, P95 retry rates, or a quarterly scorecard — the artifacts that actually drive engineering decisions in a reliability review. The fastest path to those is querying GitLab's API and pushing results into your own store.

# Fetch flaky test history via GitLab REST API (v4)
# Requires a project-level token with read_api scope
import requests, os, datetime

GL_URL = os.environ["GL_URL"]          # e.g. https://gitlab.example.com
PROJECT_ID = os.environ["GL_PROJECT"]
TOKEN = os.environ["GL_TOKEN"]

resp = requests.get(
    f"{GL_URL}/api/v4/projects/{PROJECT_ID}/test_cases",
    headers={"PRIVATE-TOKEN": TOKEN},
    params={"status": "failed", "per_page": 100},
)
cases = resp.json()

rows = [
    {
        "name": c["name"],
        "suite": c["classname"],
        "flaky_runs": c.get("recent_failures", {}).get("count", 0),
        "branch": c.get("recent_failures", {}).get("base_branch", "unknown"),
        "fetched_at": datetime.datetime.utcnow().isoformat(),
    }
    for c in cases
]
# Write to BigQuery / ClickHouse / Postgres from here

Pipe those rows into ClickHouse or BigQuery and you can run a trend query that the widget UI can't produce on its own:

-- ClickHouse: weekly flaky-test rate trend (last 12 weeks)
SELECT
    toStartOfWeek(fetched_at)          AS week,
    count(*)                           AS flaky_cases,
    uniqExact(suite)                   AS affected_suites,
    round(count(*) / max(total_runs) * 100, 2) AS flaky_rate_pct
FROM test_reliability.flaky_snapshots
WHERE fetched_at >= now() - INTERVAL 12 WEEK
GROUP BY week
ORDER BY week;

Wire that query to a Grafana time-series panel with a threshold annotation at 5% and you have a ci reliability dashboard example output that shows flaky rate trending up or down week-over-week — the kind of output you can paste into a quarterly reliability review without further explanation. One team at a mid-size SaaS org reduced triage time from 22 minutes per failure to under 4 once the Grafana panel was linked to Loki log lines via the test case name as a label.

For a trend line scorecard / quarterly best practice, add a materialized view that buckets flaky tests by severity tier (P0 = blocks default branch, P1 = blocks MR merge, P2 = flaky on feature branches only) and tracks resolution SLA compliance. The quarterly reliability scorecard template covers the full schema and the stakeholder-facing summary format. Pair it with CI observability metrics beyond flaky tests — queue depth, artifact size growth, runner saturation — to give leadership a complete picture rather than a single vanity metric.

Where the Widget Trips Up Senior Engineers

The most common mistake is trusting the widget's flaky_runs counter as a severity signal without normalizing it against execution volume. A test that runs 500 times a week and flakes twice is categorically different from one that runs 10 times and flakes twice — same counter, opposite priority. Engineers who sort by raw flaky count end up fixing the wrong tests first. Normalize by dividing flaky runs by total executions per branch, not globally, because a test that only runs on a slow nightly job will always look artificially stable in a cross-branch aggregate.

The second trap is the single-pipeline detection window described above. Tests with intermittent infrastructure failures — a flaky Docker registry pull, a shared database that occasionally times out — will not register as flaky if they fail the entire pipeline rather than recovering within it. These show up as consistent failures, get assigned to a developer, and waste hours before someone notices the pattern spans multiple unrelated test cases. If you're seeing that pattern, the flaky test root cause analysis decision tree is a faster path to the infrastructure vs. test-code split than reading stack traces manually.

Myths the Widget's Simplicity Reinforces

Myth 1: If it's not in the widget, it's not flaky. The widget only catches same-pipeline retries. Tests that alternate pass/fail across pipeline runs — the most expensive kind, because they erode confidence in the default branch — are invisible to it. You need a cross-pipeline detection layer for that signal. Parsing the raw JUnit XML artifacts and extracting flaky signals across runs gives you the rolling-window view the widget omits. Myth 2: Quarantining a flaky test resolves the problem. GitLab lets you mark tests as known failures, which stops them from blocking pipelines. That's a triage tool, not a fix. Teams that quarantine without a resolution SLA accumulate technical debt that compounds — the real numbers on flaky test cost show that unresolved quarantine lists grow at roughly 30% quarter-over-quarter without active ownership.

Myth 3: The widget's data is enough for a reliability scorecard. It isn't. The widget has no concept of MTTR, no SLA tracking, no runner-level attribution, and no way to correlate flaky tests with deployment frequency or incident rate. A genuine reliability scorecard template / quarterly best practice requires joining flaky test data with pipeline duration metrics, deployment events, and on-call alert history. The widget is a starting point, not a reporting layer.

GitLab's flaky test widget earns its place as a zero-config first signal — it's on by default, costs nothing to enable, and catches the obvious same-pipeline flakes that would otherwise get lost in retry noise. Its ceiling is low. Teams serious about reliability will hit that ceiling within a quarter. The concrete next step: export the widget data via the REST API, land it in ClickHouse or BigQuery, and build the trend-line dashboard described above. That's the move from observing flakiness to managing it.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.