Buildkite Analytics: A First Look

Most teams treat test results like a checkbox: green is good, red is bad, ship or block. The interesting signal lives between those two states — runtime variance, retry counts, the same five tests surfacing in every postmortem. Buildkite Test Analytics is Buildkite's native attempt to surface that signal without forcing you to stand up a separate observability stack just for tests.

The product launched in 2022 and has matured steadily. It ingests JUnit XML or uses language-specific collectors to store per-test execution history, compute flake rates, and render trend dashboards inside the Buildkite UI. The architectural bet is tight pipeline integration over maximum flexibility — a deliberate trade-off worth understanding before you commit.

By the end of this article you'll know how Buildkite Analytics stores and queries test data, how to wire it into a real pipeline, where its query model hits a ceiling, and which teams should look elsewhere.

What Buildkite Test Analytics Actually Stores

Buildkite Test Analytics is a time-series store for individual test executions, not a log aggregator or a general-purpose metrics backend. Each test run produces a test execution record containing: test name, suite, result (passed/failed/skipped), wall-clock duration, branch, commit SHA, and a flakiness signal derived from pass/fail alternation within the same commit. The retention window is 90 days on all plans as of mid-2024.

In a modern test architecture it sits between your CI runner and your observability layer — downstream of execution, upstream of alerting. It is not a replacement for Datadog CI Visibility or a full CI observability platform; it is scoped specifically to test health. That narrow scope is both its strength (low setup cost) and its constraint (no spans, no traces, no infra correlation out of the box).

Wiring Buildkite Analytics Into a Real Pipeline

The fastest integration path is the JUnit XML uploader. Add the collector step after your test step, pass it your API token, and every execution record lands in the Analytics UI within seconds. For Python/Pytest, the native collector gives you richer metadata.

# .buildkite/pipeline.yml
steps:
  - label: ":pytest: Run test suite"
    command: "pytest tests/ --junitxml=junit.xml -p no:randomly"
    plugins:
      - test-collector#v1.10.2:
          files: "junit.xml"
          format: "junit"
    env:
      BUILDKITE_ANALYTICS_TOKEN: "${BUILDKITE_ANALYTICS_TOKEN}"

The plugin handles chunked uploads for large suites (tested reliably up to ~50k test records per run). One non-obvious config: set -p no:randomly in Pytest if you use pytest-randomly — random ordering inflates apparent flake rates in Buildkite's algorithm because it correlates ordering changes with result changes.

Once data is flowing, the Analytics UI exposes per-test trend charts and a flakiness score. The score is computed as the ratio of runs where a test changed result without a code change — a useful proxy, but not the same as a statistically rigorous flake rate. For deeper flake triage, the Buildkite-specific flake workflow covers how to combine the score with branch filters to separate infrastructure flakes from test-logic flakes.

For teams that want to query the raw data programmatically, Buildkite exposes a REST API. Here's a Python snippet that pulls the 20 slowest tests on main over the last 7 days — useful for a weekly P95 duration report piped into Slack:

import requests, os

TOKEN = os.environ["BUILDKITE_ANALYTICS_TOKEN"]
ORG   = "your-org"
SUITE = "your-suite-slug"

resp = requests.get(
    f"https://analytics-api.buildkite.com/v1/organizations/{ORG}/suites/{SUITE}/tests",
    headers={"Authorization": f"Token token={TOKEN}"},
    params={"branch": "main", "period": "7d", "order": "slowest", "per_page": 20},
)
for t in resp.json()["tests"]:
    print(f"{t['name']:60s}  p95={t['p95_duration']:.2f}s  runs={t['run_count']}")

One team running a 12,000-test Playwright suite reported triage time dropping from ~20 minutes per failure to under 5 minutes once they replaced manual log-grepping with a Slack bot that called this endpoint and posted the top-5 slowest and top-5 flakiest tests on every failed build. The API rate limit is 100 requests/minute per token — sufficient for CI hooks, tight for bulk backfill.

Pitfalls Engineers Hit After the First Week

Flake scores inflated by suite restructuring. When you rename a test class or move specs between files, Buildkite Analytics treats the new fully-qualified name as a new test with zero history. The old name accumulates a phantom "disappeared" signal. Teams that refactor heavily see flake dashboards spike after a restructure, chasing ghosts. The fix is to stabilize test IDs before enabling the dashboard as a decision gate, or to use the API to tombstone renamed tests explicitly.

Treating the 90-day window as permanent storage. Several teams have built reporting workflows against the Analytics API only to discover that data older than 90 days is silently gone — no export, no archive hook. If you need long-term trend analysis (quarter-over-quarter flake rates, regression correlation with deploy frequency), you need to ETL the data out on a schedule into BigQuery, ClickHouse, or even a simple Postgres table. Metabase over a Postgres mirror is a low-overhead option for teams that don't want to run a full BI stack. Don't discover the 90-day cliff the week before a quarterly engineering review.

What Teams Misread About Buildkite Analytics

Myth: a flakiness score of zero means the test is reliable. Buildkite's score measures result alternation within a commit window. A test that consistently fails on one specific agent pool, or that always passes on retry, can score zero flakiness while being completely unreliable in practice. Systematic flake triage requires correlating agent metadata, retry counts, and environment variables — none of which Buildkite Analytics exposes in its current query model. The score is a filter, not a verdict.

Myth: the built-in dashboard is sufficient for engineering leadership reporting. The Analytics UI is optimized for per-engineer, per-PR triage — it is not a rollup dashboard for tracking suite health trends across teams. There is no native concept of a "team" filter, no SLA threshold alerting, and no way to embed charts in Confluence or a status page. Leadership-grade reporting still requires an external layer. More broadly, test results as a source of truth only holds when the collection and retention model is understood — gaps in either will silently distort the picture you present upward.

Buildkite Test Analytics earns its place when you're already on Buildkite and want per-test history without standing up a separate backend. Its JUnit XML ingestion is reliable, the flake score is a useful first filter, and the REST API gives you enough to build lightweight automation. Its hard limits — 90-day retention, no trace correlation, no team-level rollups — mean it's a starting point, not a complete observability strategy. If you're hitting those limits, the next step is an ETL pipeline into a queryable store and a dashboard layer on top.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.