CI Observability Metrics Beyond Flaky Tests

Most CI dashboards are a graveyard of pass/fail percentages and a flaky-test count that nobody acts on. The teams that actually improve pipeline reliability treat their test infrastructure the way SREs treat production services: SLOs, error budgets, trend lines, and structured alert routing. The signal that matters — build duration drift, retry amplification, test-suite coupling, stage-level failure concentration — doesn't live in a green/red badge. It lives in the time-series data you're already generating and mostly ignoring.

The specific problem this article addresses: most observability investment in QE stops at identifying flaky tests and never gets wired into a coherent reliability picture. You end up with a flake list, a Slack alert, and no systemic view of whether your pipeline is trending better or worse over a quarter.

By the end you'll have a concrete schema, working queries, a Grafana panel config, and a quarterly scorecard pattern you can deploy against your existing JUnit XML or OpenTelemetry test spans within a sprint.

The Metric Surface Area CI Dashboards Actually Need

A CI observability dashboard is not a test report viewer. It's a reliability telemetry layer that aggregates structured test-result events — duration, retry count, stage, branch, runner, environment — into time-series and histogram signals queryable by engineering leaders and on-call engineers alike. The distinction matters because report viewers answer "what failed?" while observability dashboards answer "is the system getting worse, and where is the pain concentrated?"

In a modern test architecture this layer sits between your CI orchestrator (GitHub Actions, Buildkite, Argo Workflows) and your analytics backend (ClickHouse, BigQuery, or PostgreSQL). Raw JUnit XML or OpenTelemetry test spans land in object storage or a streaming ingest pipeline; a transform layer normalises them into a test_runs fact table; Grafana or a BI tool queries that table. Connecting logs, metrics, and traces to that same backbone means a single slow test can be correlated with a Loki log line and a Tempo trace in one click — which is where triage time actually collapses.

Building the Dashboard: Schema, Queries, and Panel Config

Start with a normalised schema. Every test execution emits one row — not one row per suite. The columns that unlock the metrics below are: run_id, test_id, suite, branch, stage, status (pass/fail/skip/retry), duration_ms, retry_attempt, runner_label, committed_at, recorded_at.

-- ClickHouse: P95 duration trend per suite, last 30 days
SELECT
  toStartOfDay(recorded_at)        AS day,
  suite,
  quantile(0.95)(duration_ms)      AS p95_ms,
  countIf(status = 'fail')         AS failures,
  countIf(retry_attempt > 0)       AS retried
FROM test_runs
WHERE recorded_at >= now() - INTERVAL 30 DAY
  AND branch = 'main'
GROUP BY day, suite
ORDER BY day, suite;

That single query powers three panels: a P95 duration trend line (catch suite slowdowns before they breach SLOs), a failure concentration heatmap by suite, and a retry amplification chart. Retry amplification — retried / total_runs — is the metric most teams miss entirely. A suite with a 2% raw failure rate but a 15% retry rate is burning CI minutes and masking real instability.

# Grafana panel JSON fragment — retry amplification time-series
{
  "type": "timeseries",
  "title": "Retry Amplification Rate by Suite (main branch)",
  "targets": [{
    "rawSql": "SELECT day, suite, retried / count(*) AS retry_rate FROM (...) GROUP BY day, suite",
    "format": "time_series"
  }],
  "fieldConfig": {
    "defaults": {
      "thresholds": {
        "steps": [
          { "color": "green", "value": 0 },
          { "color": "yellow", "value": 0.05 },
          { "color": "red",    "value": 0.12 }
        ]
      }
    }
  }
}

For the quarterly reliability scorecard, roll the same table into a weekly aggregate and track four KPIs: suite pass rate on first attempt, P95 duration delta vs. prior period, unique flaky test count, and retry amplification rate. A structured quarterly scorecard built on these four signals gives engineering leaders a trend line that's honest — it can't be gamed by re-running failures until green.

# Python: emit test span to OTLP collector from pytest
# pytest conftest.py — requires opentelemetry-sdk, opentelemetry-exporter-otlp
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317"))
)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("pytest.suite")

def pytest_runtest_call(item):
    with tracer.start_as_current_span(item.nodeid) as span:
        span.set_attribute("test.suite", item.module.__name__)
        span.set_attribute("test.retry_attempt",
                           getattr(item, "_rerun_count", 0))

Wiring pytest directly to an OpenTelemetry collector means every test execution becomes a queryable trace in Honeycomb or Grafana Tempo — no custom log parsing, no brittle JUnit XML scraping. One team reduced triage time from 22 minutes per failure to under 4 once the dashboard linked directly to the Loki log context for each failing span.

Pitfalls That Kill Dashboard Adoption Before It Starts

Aggregating across branches indiscriminately is the most common schema mistake. A feature branch with 40% failure rate will drown your main-branch signal unless branch is a first-class filter — not an afterthought. The fix is enforcing branch segmentation at ingest, not at query time. Similarly, mixing retry attempts into the primary pass/fail count inflates the apparent pass rate; always record retry_attempt and filter on retry_attempt = 0 for your "first-attempt pass rate" SLO.

Building dashboards nobody owns is an org-level failure that looks like a tooling problem. A Grafana board with 14 panels and no designated reviewer decays within a quarter: thresholds go stale, queries break on schema changes, and engineers stop trusting it. Assign a named dashboard owner per team, version-control the Grafana JSON in the same repo as the CI config, and treat dashboard alerts like production alerts — route them to PagerDuty or a dedicated Slack channel with runbook links, not to a general noise channel.

What Most Teams Get Wrong About CI Reliability Metrics

Pass/fail rate is not a reliability metric. It's an output metric that reflects the current state of a single run, not the trend of your system. Pass/fail numbers are actively misleading when retries are enabled, because a suite that passes on attempt 3 counts as green. The metrics that actually indicate reliability are first-attempt pass rate, retry amplification, and P95 duration variance over rolling windows — none of which appear in a standard CI badge.

Flaky-test dashboards are not CI observability dashboards. Flake detection is one input signal; a full CI observability layer also tracks infrastructure variance (runner cold-start latency, cache hit rates, parallelism efficiency), stage-level failure concentration (are 80% of failures in one stage?), and cross-branch regression velocity. The metrics that drive real decisions in 2026 combine test-result telemetry with runner-level and deployment-level context — treating them as separate concerns is what keeps triage slow and postmortems repetitive.

A CI observability dashboard that only tracks flaky tests is a smoke detector with the battery half-in. The schema, queries, and panel patterns above give you a working foundation in a sprint. The next concrete step: pull 90 days of JUnit XML or OTLP test spans into ClickHouse or BigQuery, run the retry-amplification query, and find the suite whose ratio surprises you most — that's where your first reliability SLO should be set.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.