Pipeline Test Result Heat Maps Explained

Most teams look at a pipeline run the same way they read a traffic light: red means stop, green means go. The heat map breaks that habit. It renders every test, every stage, and every run as a cell in a two-dimensional grid — time on one axis, test identity on the other — and the color encodes pass, fail, skip, or flaky. Patterns that are invisible in a flat pass-rate dashboard become immediately obvious: a column of red that started Tuesday, a diagonal smear of flakiness that tracks with a specific committer, a row that has never been fully green in six weeks.

The technical problem is data density. A single GitHub Actions workflow with 400 tests run 30 times a day produces 12,000 result cells daily. No alert threshold surfaces the slow-burn degradation in that volume; no single pass-rate metric tells you whether the failures are concentrated in one test or spread evenly across a suite. A heat map is the right data structure for that shape of problem.

By the end of this article you'll know how to model test result data for heat map rendering, build a working Grafana panel backed by ClickHouse or PostgreSQL, and avoid the three instrumentation mistakes that make heat maps misleading rather than actionable.

What a Test Result Heat Map Actually Encodes

A pipeline test result heat map is a matrix visualization where each cell represents one test's outcome in one pipeline run. The X-axis is typically run number or wall-clock time; the Y-axis is test name or test suite. Cell color maps to outcome: green (pass), red (fail), yellow (flaky/retry-resolved), grey (skipped). The result is a raster image of your suite's health history — dense, scannable, and honest about variance in a way that aggregated metrics are not.

In a modern test architecture the heat map sits between raw JUnit XML storage and executive-facing dashboards. It's the triage layer — the view an SDET opens when a postmortem asks "when did this start?" or "is this isolated or systemic?" It complements stage-aware failure analysis because the same red cell means something different in a unit-test stage versus a post-deploy smoke stage. Encoding the pipeline stage as a facet — either a separate panel or a color-hue modifier — is what separates a useful heat map from a pretty one.

Building a Heat Map Panel: Data Model, Query, and Grafana Config

The foundation is a normalized test-result table. Whether you're on PostgreSQL for smaller pipelines or ClickHouse for high-volume CI, the schema is the same shape:

-- PostgreSQL / ClickHouse compatible DDL
CREATE TABLE test_results (
  run_id        TEXT        NOT NULL,
  pipeline      TEXT        NOT NULL,
  stage         TEXT        NOT NULL,
  test_id       TEXT        NOT NULL,  -- fully qualified name
  outcome       TEXT        NOT NULL,  -- 'pass' | 'fail' | 'flaky' | 'skip'
  duration_ms   INTEGER,
  branch        TEXT,
  committed_at  TIMESTAMPTZ NOT NULL,
  inserted_at   TIMESTAMPTZ DEFAULT now()
);
CREATE INDEX ON test_results (pipeline, committed_at, test_id);

The flaky outcome is not a JUnit XML native value — you derive it during ingestion: if a test fails on attempt 1 but passes on attempt 2+ within the same run, write flaky, not pass. This single decision is what makes retry-masking visible on the heat map rather than hidden inside a green cell. For more on how retry depth distorts downstream metrics, see how rerun depth skews MTTD.

The Heat Map Query

For Grafana's native Heat Map panel (or a State Timeline panel, which renders better for sparse matrices), you need one row per test per time bucket:

-- ClickHouse: outcome per test per hour, last 7 days
SELECT
  toStartOfHour(committed_at)          AS time,
  test_id,
  argMax(outcome, inserted_at)         AS last_outcome,
  countIf(outcome = 'fail')            AS fail_count,
  countIf(outcome = 'flaky')           AS flaky_count
FROM test_results
WHERE pipeline = 'api-service'
  AND committed_at >= now() - INTERVAL 7 DAY
  AND branch = 'main'
GROUP BY time, test_id
ORDER BY time, test_id;

Map last_outcome to a Grafana value-to-color override: pass → green (#37872D), fail → red (#C4162A), flaky → amber (#FF9830), skip → grey (#808080). Use the State Timeline panel type rather than Heatmap when your Y-axis cardinality exceeds ~50 tests — Heatmap buckets values numerically and will mangle categorical outcomes.

GitHub Actions Integration

GitHub Actions doesn't expose a native test result overview beyond job-level pass/fail. To feed the heat map, parse JUnit XML from your artifact store during a post-run step and write directly to your results table:

# .github/workflows/ingest-results.yml (runs after test job)
- name: Ingest JUnit XML to ClickHouse
  if: always()
  run: |
    python scripts/ingest_junit.py \
      --xml-glob "test-results/**/*.xml" \
      --pipeline "api-service" \
      --stage "unit" \
      --run-id "${{ github.run_id }}" \
      --branch "${{ github.ref_name }}" \
      --committed-at "${{ github.event.head_commit.timestamp }}"

# scripts/ingest_junit.py (core logic)
import glob, xml.etree.ElementTree as ET, clickhouse_driver, os, sys

client = clickhouse_driver.Client(host=os.environ["CH_HOST"])

for path in glob.glob(sys.argv[sys.argv.index("--xml-glob")+1]):
    tree = ET.parse(path)
    rows = []
    for tc in tree.iter("testcase"):
        outcome = "pass"
        if tc.find("failure") is not None: outcome = "fail"
        elif tc.find("skipped") is not None: outcome = "skip"
        rows.append({
            "run_id": ..., "pipeline": ..., "stage": ...,
            "test_id": f"{tc.get('classname')}.{tc.get('name')}",
            "outcome": outcome, "duration_ms": int(float(tc.get("time",0))*1000),
            "branch": ..., "committed_at": ...
        })
    client.execute("INSERT INTO test_results VALUES", rows)

Teams that wired this pipeline to a Grafana State Timeline reported triage time dropping from 22 minutes per failure to under 4 — the heat map made the "started Tuesday, only on main, only in the auth suite" pattern visible in seconds rather than requiring manual log correlation. For storage trade-offs at scale, the PostgreSQL vs. ClickHouse vs. BigQuery comparison covers when each backend earns its keep.

Three Instrumentation Mistakes That Corrupt the Heat Map

Writing retries as passes. Most JUnit reporters record the final outcome. If your CI retries a failed test three times and it eventually passes, the XML shows green. The heat map cell goes green. The flakiness is invisible. Fix this at ingestion: compare attempt-level logs or use a retry-aware reporter (Pytest's pytest-rerunfailures with --report-log, or Playwright's built-in retry JSON) before writing to the results table. This is the single biggest source of misleading heat maps in the wild.

Using wall-clock commit time instead of run-start time. When a branch cut three days ago is merged today, its committed_at timestamp places failures in the past — the heat map shows a retroactive red column that looks like a historical regression. Use run_start_at as the X-axis anchor. The second mistake is mixing branch results without a filter: PR branch failures pollute the main-branch heat map with noise that has no deployment consequence. Always expose a branch facet and default the panel to main or your trunk. Branch filtering skews are subtle but they make trend lines meaningless.

What Most Teams Misread in a Test Heat Map

A fully green row means the test is healthy. Not necessarily. A test that is consistently skipped shows as grey — not red — but a grey row spanning 30 runs means the test has not executed in production conditions for weeks. A test that always passes in under 5 ms is probably asserting nothing meaningful. Green and grey both deserve scrutiny; the heat map surfaces the pattern, but you still have to ask why. Similarly, a single red column (one bad run) is almost always less important than a diagonal amber smear (flakiness spreading across runs as a dependency degrades).

The heat map is the dashboard. It's a triage entry point, not a complete quality signal. It tells you where and when; it doesn't tell you why or what the business impact is. Wire cells to drill-down links — clicking a red cell should open the Loki log stream or the Allure report for that run, not a dead end. Engineering leaders need the heat map summarized into fleet-level trends; for that layer, multi-pipeline visibility dashboards aggregate what the heat map surfaces per pipeline into a portfolio view.

A pipeline test result heat map earns its place in your observability stack when it's backed by clean data — retry-resolved outcomes, branch-filtered queries, and run-start timestamps. Start with the schema above, wire one pipeline's JUnit XML through the ingestion script, and render a State Timeline in Grafana against your existing ClickHouse or PostgreSQL instance. Once the first pattern surprises you, you'll instrument the rest. For the next layer up, read about tracking quality over time without vanity metrics to make sure what you're measuring drives decisions rather than reports.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.