Datadog CI Visibility vs Homegrown Grafana

Most teams land on one of two extremes: pay for Datadog CI Visibility and accept the opinionated data model, or wire up Grafana and own every inch of the pipeline. Neither is obviously correct. The right answer depends on how much your team values time-to-insight versus control over the schema, and whether you're willing to maintain the plumbing when your JUnit XML format drifts or your Loki label cardinality explodes.

The technical problem is real: CI test results are structurally rich — test name, suite, duration, retry count, exit code, runner labels, git ref — but most teams collapse that into a single pass/fail signal and throw away the rest. Both Datadog and Grafana can surface that signal, but they make radically different assumptions about who owns the data model and where the query logic lives.

By the end of this article you'll have a clear decision framework, working config for both paths, and honest numbers on where each approach breaks down under real pipeline load.

What Datadog CI Visibility and Grafana Are Actually Solving

Datadog CI Visibility is a purpose-built product that ingests pipeline and test span data via the datadog-ci CLI or native integrations for GitHub Actions, Jenkins, CircleCI, and Buildkite. It models tests as traces — each test run is a span with duration, status, error message, and a parent pipeline span — so you get distributed-tracing semantics applied to your test suite. What it actually tracks goes well beyond pass/fail: it records flakiness scores, known-flaky flags, wall-clock vs CPU time, and git blame context. The cost is lock-in: your data lives in Datadog's store, queryable only through their UI or API.

A homegrown Grafana setup means you own the ingestion pipeline — typically parsing JUnit XML out of CI artifacts, writing rows into ClickHouse, PostgreSQL, or BigQuery, and building panels on top. Grafana itself is just the visualization layer; the intelligence is in your schema and your queries. The trade-off is explicit: you get full control over retention, cardinality, and cross-team joins, but you're also on the hook when the ETL breaks at 2 a.m. before a release. For teams already running Grafana for infrastructure metrics, extending it to test results is low marginal cost — the operational model is already there.

Building a Test Dashboard in Grafana from JUnit XML

Ingesting JUnit XML into a queryable store

The most portable ingestion path is a Python parser that runs as a CI post-step, reads JUnit XML artifacts, and upserts rows into PostgreSQL. Keep the schema flat — one row per test case — so Grafana's SQL data source can query it without CTEs on every panel.

# parse_junit.py — runs after pytest, before artifact upload
import xml.etree.ElementTree as ET, psycopg2, os, datetime

conn = psycopg2.connect(os.environ["PG_DSN"])
cur = conn.cursor()

tree = ET.parse("test-results.xml")
for suite in tree.iter("testsuite"):
    for case in suite.iter("testcase"):
        status = "fail" if case.find("failure") is not None else \
                 "skip" if case.find("skipped") is not None else "pass"
        cur.execute("""
            INSERT INTO test_runs
              (run_id, pipeline, git_sha, suite, name, duration_s, status, recorded_at)
            VALUES (%s,%s,%s,%s,%s,%s,%s,%s)
            ON CONFLICT (run_id, name) DO NOTHING
        """, (
            os.environ["CI_RUN_ID"],
            os.environ["CI_PIPELINE_NAME"],
            os.environ["GITHUB_SHA"],
            suite.attrib.get("name"),
            case.attrib.get("name"),
            float(case.attrib.get("time", 0)),
            status,
            datetime.datetime.utcnow(),
        ))
conn.commit()

Wire this into GitHub Actions as a post-step that always runs, even on failure — if: always() in the step definition. For a deeper look at how JUnit XML maps to Grafana panel queries, the JUnit XML as a Grafana data source guide covers the full schema-to-panel path including time-series flakiness panels.

Grafana SQL panel: P95 duration by suite

Once rows are in PostgreSQL, a single query gives you the duration distribution that Datadog surfaces as a built-in widget — except now you own it:

SELECT
  suite,
  PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY duration_s) AS p95_s,
  COUNT(*) FILTER (WHERE status = 'fail') AS failures,
  COUNT(*) AS total_runs
FROM test_runs
WHERE recorded_at >= NOW() - INTERVAL '7 days'
  AND pipeline = '$pipeline'
GROUP BY suite
ORDER BY p95_s DESC
LIMIT 20;

This query, set as a Table panel with a threshold on p95_s, replaced a 22-minute manual triage process with a sub-4-minute scan — the same outcome described in the Grafana + Loki failure triage workflow when you add Loki log correlation on the run_id field.

Datadog CI Visibility equivalent

On the Datadog side, the same view is a saved CI Visibility query with no infrastructure to maintain:

@test.status:fail @git.branch:main
| stats p95(@duration) by @test.suite

Datadog's flakiness detection runs automatically once a test has 3+ runs in a 7-day window — no schema design required. The cost is roughly $0.05–$0.10 per 1,000 test spans depending on contract, which becomes material at 500k+ spans/day. At that volume, a ClickHouse-backed Grafana setup with compressed columnar storage is meaningfully cheaper and faster on aggregate queries.

Where Both Setups Quietly Break Down

The most common mistake with Datadog CI Visibility is treating the flakiness score as a definitive signal without auditing the retry behavior underneath it. If your pipeline retries failed tests automatically — which most do — Datadog marks a test as "flaky" based on mixed pass/fail within a single run, but it doesn't surface the retry count as a first-class dimension. You can end up with a clean flakiness dashboard that masks a suite where 30% of passes are retry-rescued. Retry count inflating pass rate is a separate problem from flakiness, and conflating them leads to misallocated triage effort.

With homegrown Grafana, the recurring failure mode is schema drift. JUnit XML is loosely specified — Pytest, JUnit 5, and Playwright all emit slightly different attribute names and nesting. Teams start with a clean schema, then six months later have three NULL-heavy columns because a new test runner emits classname where the parser expected name. Add a schema validation step in CI that asserts required fields before the upsert, or you'll be backfilling data during an incident.

Myths That Push Teams Toward the Wrong Choice

"Datadog CI Visibility gives you everything out of the box." It gives you test-span data out of the box. Cross-referencing test failures with deployment events, feature flags, or infrastructure metrics still requires manual dashboard construction in Datadog — and those joins are expensive in both query time and cost. Multi-pipeline visibility across org boundaries is a known gap: Datadog's CI Visibility is scoped per pipeline/repo, and stitching a cross-team reliability view requires either a custom dashboard or exporting data to a warehouse anyway. If that's your primary use case, you're building infrastructure regardless of which tool you pick.

"A homegrown Grafana setup is too much maintenance for most teams." This is true at 5 engineers and false at 50. The parser above is ~50 lines of Python and a single table. The real maintenance cost is in the Grafana dashboard JSON — panels drift, variable queries break on schema changes, and nobody owns the dashboard. The fix is treating dashboard JSON as code: store it in git, apply it via Grafana's provisioning API on deploy, and run a smoke test against a staging Grafana instance in CI. That's an afternoon of work, not a platform team.

If your team is under 20 engineers, already paying for Datadog, and doesn't need cross-team joins, CI Visibility is the pragmatic choice — the span model is solid and setup is under an hour. Above that scale, or if you're already running ClickHouse or BigQuery for other analytics, the Grafana path pays for itself in flexibility and cost. Either way, start by defining the three queries that would change a decision — P95 by suite, failure rate by git author, flake rate by week — and work backward to the data model that answers them.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.