Buildkite Test Analytics for Flaky Suites

Most teams treat test results like a checkbox: green is good, red is bad, ship or block. The interesting signal lives in everything between those two states — retry counts, runtime variance, the same twelve tests appearing in every postmortem. Buildkite Test Analytics was built specifically to surface that layer, and if you're already on Buildkite pipelines, you're one collector configuration away from having it.

The hard part isn't ingestion — it's knowing which metrics to act on, how to query the history meaningfully, and where the tool's model breaks down under a genuinely unstable suite. Flaky tests aren't just an annoyance; the compounding cost in engineer time and delayed deploys is measurable and significant.

By the end of this article you'll know how Buildkite Test Analytics models flakiness, how to query its API and export data for deeper analysis, what a useful CI reliability dashboard looks like with real panel examples, and where the tool's defaults will mislead you if you don't correct them.

What Buildkite Test Analytics Actually Tracks

Buildkite Test Analytics ingests JUnit XML (or its own JSON schema) from any CI runner and stores per-test execution history: duration, outcome, branch, commit, and retry count. It computes a flakiness score per test — essentially a rolling pass/fail inconsistency rate over the last N runs on the same branch — and surfaces tests whose scores cross a configurable threshold. The score is not a p-value; it's closer to a weighted flip-rate, which matters when you're interpreting it.

In a modern test architecture, Test Analytics sits between your runner layer (Pytest, Playwright, RSpec) and your observability stack (Datadog, Grafana, Honeycomb). It doesn't replace a proper metrics pipeline — it's a purpose-built lens on test-execution data. Think of it as the first triage layer: it tells you which tests are unreliable and how often, but not why. The "why" still requires correlated logs, traces, or environment telemetry. For teams already exporting JUnit XML, parsing those reports directly can complement what Buildkite surfaces natively.

Wiring the Collector, Querying the API, and Building a CI Reliability Dashboard

The collector setup is a single upload step appended to your pipeline. Add this after your test run in pipeline.yml:

steps:
  - label: ":pytest: Run tests"
    command: |
      pytest --junitxml=junit.xml -q
    plugins:
      - buildkite/test-collector#v1.10.2:
          files: "junit.xml"
          format: "junit"
    env:
      BUILDKITE_ANALYTICS_TOKEN: "${BUILDKITE_ANALYTICS_TOKEN}"

That's the floor. The ceiling is querying the REST API to pull historical data into your own warehouse. Buildkite exposes /analytics/organizations/{org}/test_suites/{suite}/tests — paginate it, filter by scope=flaky, and push to BigQuery or ClickHouse for trend analysis. A minimal Python exporter:

import requests, json, os

BASE = "https://api.buildkite.com/v2/analytics/organizations"
ORG  = os.environ["BK_ORG_SLUG"]
SUITE = os.environ["BK_SUITE_SLUG"]
TOKEN = os.environ["BUILDKITE_API_TOKEN"]

def fetch_flaky_tests(page=1):
    url = f"{BASE}/{ORG}/test_suites/{SUITE}/tests"
    r = requests.get(url, headers={"Authorization": f"Bearer {TOKEN}"},
                     params={"scope": "flaky", "page": page, "per_page": 100})
    r.raise_for_status()
    return r.json()

tests = []
page = 1
while True:
    batch = fetch_flaky_tests(page)
    if not batch:
        break
    tests.extend(batch)
    page += 1

with open("flaky_export.json", "w") as f:
    json.dump(tests, f)

Once the data is in ClickHouse or BigQuery, you can build a CI reliability dashboard that goes well beyond the built-in UI. A useful Grafana panel query against a ClickHouse table (bk_test_runs) for a P95 runtime trend looks like this:

SELECT
  toStartOfDay(finished_at) AS day,
  name,
  quantile(0.95)(duration_ms) AS p95_ms,
  countIf(result = 'failed') / count() AS fail_rate
FROM bk_test_runs
WHERE finished_at >= now() - INTERVAL 30 DAY
  AND scope = 'flaky'
GROUP BY day, name
ORDER BY fail_rate DESC
LIMIT 20

This query is the difference between a vanity dashboard and an actionable one. Teams that wired this panel to a Grafana alert on fail_rate > 0.15 cut mean triage time from over 20 minutes per failure to under 4 — because the on-call engineer already knew which test, which branch, and the runtime percentile before opening a single log. Pair the trend data with a quarterly reliability scorecard to track whether remediation work is actually moving the needle over time.

Where Senior Engineers Still Get Burned by the Default Config

The most common mistake is trusting the built-in flakiness score without understanding its lookback window. By default, Buildkite calculates flakiness over the last 50 runs on the default branch only. If your flaky test only fires on feature branches under load, or only during nightly runs, it will never cross the threshold and the UI will show it as stable. Engineers see "no flaky tests" and ship — then hit the same failure in production-adjacent staging. Fix this by explicitly setting branch filters in your API queries and pulling data from all branches, not just main.

The second pitfall is treating the flakiness score as a ranking for prioritization without accounting for test frequency. A test that runs 500 times a day with a 5% flip rate is far more damaging than one that runs twice a week at 20%. The built-in UI doesn't weight by volume. Failure clustering across high-frequency tests can also obscure the fact that what looks like one flaky test is actually three distinct root causes — a nuance the Buildkite UI collapses into a single score.

Three Things Teams Consistently Misread in Test Analytics Data

Myth 1: A stable flakiness score means the test is fixed. Score stability just means the flip rate hasn't changed — a test flipping at exactly 10% every week has a stable score and is still broken. Myth 2: Retry-on-failure masks flakiness from Analytics. Buildkite Test Analytics does record retry attempts when you use its native retry tracking, but only if the final outcome is reported correctly. If your CI step retries silently at the shell level before JUnit XML is written, those retries are invisible to Analytics entirely — your flakiness score is artificially suppressed. Audit your retry logic before trusting the score.

Myth 3: The dashboard is the fix. Visibility is a prerequisite, not a solution. Teams that build a beautiful CI observability dashboard — runtime trends, fail rates, P95 panels — and then don't assign ownership to the top-ten flaky tests see no improvement in suite reliability. The dashboard surfaces the signal; a systematic remediation process is what converts that signal into fewer incidents. Metrics without process are just expensive wallpaper.

Buildkite Test Analytics is a capable first layer for flaky-test triage, but its defaults are optimized for simplicity, not depth. Export the raw data via the API, query it in ClickHouse or BigQuery, and build branch-aware dashboards that weight flakiness by test frequency. From there, instrument your remediation work against a quarterly scorecard so you can tell the difference between a suite that's improving and one that's just quiet this sprint.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.