Developing Dashboards for CI/CD Pipelines

Most CI/CD dashboards are built to reassure, not to inform. A green percentage at the top of a Confluence page tells you nothing about whether your suite is degrading, which test files are responsible for half your retry budget, or why P95 duration jumped 40 seconds after a dependency bump last Tuesday. The interesting signal lives in trend lines, variance, and the correlation between pipeline stage and failure cluster — not in a single pass/fail number.

The gap between a test report and an engineering dashboard is a schema problem as much as a visualization problem. Raw JUnit XML gets ingested, a green badge gets rendered, and the underlying row data is never queried again. Building a dashboard that actually drives decisions means designing the storage layer first, then the queries, then the panels — in that order.

By the end of this article you will have a concrete schema, working SQL, a Grafana panel config, and a clear model for which metrics belong on which audience's screen — from SDET to engineering leader.

What a CI/CD Test Dashboard Actually Measures

A CI/CD test dashboard is not a test report. A report is a snapshot — one run, one commit. A dashboard is a time-series view over many runs, designed to surface patterns that no single run can reveal: rising flake rates, creeping duration variance, failure clusters that correlate with specific authors or file paths. The distinction matters because the storage model, query shape, and refresh cadence are all different.

In a modern test architecture, the dashboard sits downstream of your result ingestion pipeline. JUnit XML or Playwright JSON gets parsed on every CI run, normalized into a structured store (ClickHouse, BigQuery, or PostgreSQL depending on volume), and queried by Grafana, Datadog, or Looker. Pass/fail as a primary metric is where most dashboards stop — and where the useful ones start. The metrics that matter are flake rate per test, P50/P95 duration by suite, retry count per pipeline stage, and failure-to-fix cycle time.

Building Test Automation Dashboards: Schema, Queries, and Panels

Start with a schema that preserves run-level and test-level granularity. A flat table in ClickHouse or PostgreSQL with one row per test execution is the right unit:

-- PostgreSQL schema for test run results
CREATE TABLE test_results (
  run_id        TEXT        NOT NULL,
  pipeline      TEXT        NOT NULL,  -- e.g. 'main', 'pr', 'nightly'
  branch        TEXT,
  commit_sha    TEXT,
  suite         TEXT        NOT NULL,
  test_name     TEXT        NOT NULL,
  status        TEXT        NOT NULL,  -- passed | failed | skipped | flaky
  duration_ms   INTEGER     NOT NULL,
  retry_count   SMALLINT    DEFAULT 0,
  started_at    TIMESTAMPTZ NOT NULL,
  worker_node   TEXT,
  PRIMARY KEY (run_id, suite, test_name)
);

CREATE INDEX idx_test_results_started ON test_results (started_at DESC);
CREATE INDEX idx_test_results_status  ON test_results (status, started_at DESC);

With that schema in place, the queries that power your most useful panels are straightforward. Flake rate over a rolling 7-day window:

-- Flake rate per test, last 7 days
SELECT
  test_name,
  COUNT(*)                                        AS total_runs,
  SUM(CASE WHEN status = 'flaky' THEN 1 ELSE 0 END) AS flaky_runs,
  ROUND(
    100.0 * SUM(CASE WHEN status = 'flaky' THEN 1 ELSE 0 END) / COUNT(*), 2
  )                                               AS flake_pct
FROM test_results
WHERE started_at >= NOW() - INTERVAL '7 days'
GROUP BY test_name
HAVING flake_pct > 2.0
ORDER BY flake_pct DESC
LIMIT 25;

Wire this into a Grafana table panel using the PostgreSQL data source. Set the panel refresh to 5 minutes and add a threshold at 5 % flake rate to trigger a red cell color. Teams that instrument this properly find that flake rate compounds across dependent pipeline stages — a 3 % flake in a unit-test stage becomes an 18 % effective failure rate by the time three downstream stages have run. Surfacing per-stage flake separately on the dashboard (not just aggregate) is what makes that compounding visible.

For duration variance, add a Grafana time-series panel backed by a percentile query. Test duration variance often reveals instability earlier than failure rate does — a suite whose P95 is drifting upward is telling you something before any test turns red:

-- P50 / P95 duration by suite, bucketed by hour
SELECT
  DATE_TRUNC('hour', started_at)          AS bucket,
  suite,
  PERCENTILE_CONT(0.50) WITHIN GROUP (ORDER BY duration_ms) AS p50_ms,
  PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY duration_ms) AS p95_ms
FROM test_results
WHERE started_at >= NOW() - INTERVAL '48 hours'
GROUP BY 1, 2
ORDER BY 1 DESC;

On the GitHub Actions side, emit a structured summary after each run so the ingestion pipeline has clean input. A minimal step that posts JUnit XML to your ingestion endpoint:

- name: Upload test results
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: junit-results
    path: reports/junit.xml

- name: Ingest to results store
  if: always()
  run: |
    curl -X POST "$RESULTS_INGEST_URL" \
      -H "Authorization: Bearer $RESULTS_TOKEN" \
      -F "file=@reports/junit.xml" \
      -F "pipeline=${{ github.workflow }}" \
      -F "branch=${{ github.ref_name }}" \
      -F "commit=${{ github.sha }}"
  env:
    RESULTS_INGEST_URL: ${{ secrets.RESULTS_INGEST_URL }}
    RESULTS_TOKEN: ${{ secrets.RESULTS_TOKEN }}

One team running ~4,000 tests per PR cut triage time from 22 minutes per failure to under 4 once the dashboard was wired to Loki for log correlation — clicking a red cell in the Grafana table panel opened the matching Loki log stream for that test name and run ID directly. That linkage is a one-line Grafana data link config, not a bespoke tool. For teams evaluating tooling: use Grafana when you own the data store; use Datadog when your test telemetry is already flowing through their agent.

Where CI/CD Dashboard Builds Go Wrong

The most common mistake is building the dashboard before defining the audience. An SDET needs test-name granularity and log links. An engineering leader needs suite-level trend lines and cycle time. Putting both on one panel set produces a dashboard nobody trusts. Separate views — same underlying data, different aggregation level — solve this without duplicating the schema.

The second mistake is treating dashboard refresh rate as a proxy for data freshness. A 30-second Grafana refresh against a PostgreSQL table with 2 million rows and no time-partitioning will degrade query performance within weeks of launch. Partition by month, add a materialized view for the flake-rate query, and set the dashboard refresh to 2–5 minutes. The third mistake is ignoring the skipped status. Skipped tests accumulate silently — suites that skip 15 % of their cases on main are hiding coverage gaps that no failure rate metric will surface. Add a skipped-rate panel from day one.

What Most Teams Get Wrong About Test Reports and Dashboards

The most persistent myth is that a good test report and a good dashboard are the same artifact at different zoom levels. They are not. Allure and ReportPortal produce excellent per-run reports — deep failure context, attachment support, history comparison per test. Use Allure when your primary consumer is a developer debugging a single failure. Use ReportPortal when you need multi-project aggregation and role-based access across teams. Neither is a substitute for a time-series dashboard that answers "is this suite getting worse over the last 30 days?"

The second myth is that 100 % pass rate on the dashboard means the suite is healthy. A suite that retries aggressively, skips environment-dependent tests, and quarantines flaky cases can show 100 % green while masking serious instability. The dashboard should expose retry counts, quarantine list size, and skip rate as first-class metrics — not footnotes. If those numbers are trending up while pass rate holds steady, the suite is not getting more reliable; it is getting better at hiding failures.

A CI/CD dashboard earns its place when it changes a decision — which test gets fixed today, whether a flaky suite blocks a release, where infra cost is being burned on retries. Start with the schema, write the three or four queries that answer your most expensive recurring questions, and build the panels around those. From there, layering in AI-assisted failure analysis becomes a natural next step once the structured data is already flowing.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.