Test Suite Aging Reports: PAR Visualization

Most teams treat test results like a checkbox: green is good, red is bad, ship or block. The interesting signal lives in the dimension almost nobody visualizes — time. A test that passed reliably for six months and now retries twice per run is not healthy. It is aging. The difference between a stable suite and a liability is often invisible until you plot test-level behavior across hundreds of pipeline runs and watch the decay curves emerge.

A PAR aging report — Pass rate, Age (time since first recorded result), Retry count — gives you a structured view of which tests are drifting toward failure without yet tripping your red threshold. It answers the embedded CI failure analysis question that dashboards built on raw pass/fail counts cannot: "Which tests are quietly getting worse?" This matters most in pipelines with 2,000+ tests where aggregate failure rate looks stable but individual test health is eroding.

By the end of this article you will have the SQL schema, a Grafana panel config, a Python classifier, and the mental model to build a living aging report wired to your test results database — whether that is PostgreSQL, BigQuery, or ClickHouse.

What a PAR Aging Report Actually Measures

A PAR aging report is a per-test time-series aggregation that combines three dimensions: rolling pass rate (7- or 14-day window), test age (days since first execution recorded in the results store), and retry delta (change in average retry count over the same window). Plotting these three together reveals four behavioral archetypes: stable, degrading, volatile, and stale. Each archetype demands a different engineering response — stabilization, triage, quarantine, or deletion.

In a modern test architecture, the aging report sits between your raw JUnit XML ingestion layer and your alerting/triage layer. It is not a replacement for a run-level dashboard; it is the layer that answers "why is this week's failure rate 3 points higher than last week's?" without requiring a human to diff two Allure reports manually. Be aware that failure rate drops as your suite grows, which means aggregate metrics will mask individual test decay — PAR visualization exists precisely to counter that dilution effect.

Building the PAR Pipeline: Schema, Query, and Grafana Panel

Start with a normalized results table. If you are ingesting JUnit XML via a custom parser or a tool like ReportPortal, you need at minimum: test_id, run_id, executed_at, status (pass/fail/skip/retry), and retry_count. The following PostgreSQL schema is the minimum viable foundation:

CREATE TABLE test_runs (
  test_id       TEXT        NOT NULL,
  run_id        TEXT        NOT NULL,
  executed_at   TIMESTAMPTZ NOT NULL,
  status        TEXT        NOT NULL,  -- 'pass','fail','skip','retry'
  retry_count   INT         NOT NULL DEFAULT 0,
  duration_ms   INT,
  PRIMARY KEY (test_id, run_id)
);

CREATE INDEX idx_test_runs_executed_at ON test_runs (executed_at DESC);
CREATE INDEX idx_test_runs_test_id     ON test_runs (test_id, executed_at DESC);

With that in place, the core PAR query computes a rolling 14-day window per test. In ClickHouse this runs in under 200ms on 50M rows; in PostgreSQL with the indexes above, expect 2–4 seconds on 5M rows without materialized views.

-- PostgreSQL: 14-day PAR aging report
WITH windowed AS (
  SELECT
    test_id,
    MIN(executed_at)                                        AS first_seen,
    ROUND(AVG(CASE WHEN status = 'pass' THEN 1.0 ELSE 0.0 END)::NUMERIC, 4)
                                                            AS pass_rate_14d,
    ROUND(AVG(retry_count)::NUMERIC, 2)                    AS avg_retries_14d,
    COUNT(*)                                               AS run_count_14d,
    NOW() - MIN(executed_at)                               AS age
  FROM test_runs
  WHERE executed_at >= NOW() - INTERVAL '14 days'
    AND status != 'skip'
  GROUP BY test_id
)
SELECT
  test_id,
  EXTRACT(DAY FROM age)  AS age_days,
  pass_rate_14d,
  avg_retries_14d,
  run_count_14d,
  CASE
    WHEN pass_rate_14d >= 0.99 AND avg_retries_14d < 0.05 THEN 'stable'
    WHEN pass_rate_14d >= 0.90 AND avg_retries_14d < 0.20 THEN 'degrading'
    WHEN pass_rate_14d <  0.90                            THEN 'volatile'
    ELSE                                                       'stale'
  END AS health_bucket
FROM windowed
ORDER BY pass_rate_14d ASC, avg_retries_14d DESC;

Feed this query into Grafana using the PostgreSQL data source and a Table panel with cell color overrides keyed on health_bucket. Set the color thresholds to: stable → green, degrading → yellow, volatile → red, stale → grey. Add a second panel — a Scatter plot with age_days on X, pass_rate_14d on Y, and point size mapped to avg_retries_14d. This scatter is the PAR visualization: tests drifting toward the bottom-left corner are your highest-risk items regardless of what your aggregate dashboard shows.

For GitHub Actions pipelines, publish results to the database at the end of each workflow using a lightweight Python uploader. One team using this pattern on a 3,400-test Playwright suite dropped triage time from 22 minutes per failure to under 4 once the Grafana panel was wired to Loki for log correlation — clicking a volatile test opened its last five error logs inline. You can also push the health_bucket field to a Slack webhook so that any test transitioning from degrading to volatile pages the owning squad directly, without waiting for a postmortem. If you want to go further, wiring an AI failure analyzer to the volatile bucket gives you root-cause hypotheses on the tests most likely to block a release.

Where PAR Reports Break Down in Practice

The most common mistake is computing PAR against a results table that includes skipped tests in the denominator. A test that is skipped 40% of the time will show an artificially inflated pass rate and low retry count — it looks stable when it is actually absent. Filter status != 'skip' before any rate calculation, and separately track skip rate as its own signal. The deeper issue is that skipped tests distort every rate you report, and the PAR report is not immune.

The second failure mode is window size mismatch. Teams that run nightly regression suites on a 7-day window get only 7 data points per test — statistically meaningless for pass-rate variance. Use a 30-day window for nightly suites and a 7-day window for suites that run on every PR. Mixing window sizes across suite types in the same PAR table produces composition skew that makes the report misleading at the aggregate level. Segment by suite type before you aggregate.

Myths That Keep Teams Staring at Red/Green Dashboards

Myth 1: A passing test is a healthy test. Pass rate is a lagging indicator. A test that passes on the third retry every single run is not passing — it is surviving. Retry count is the leading indicator of imminent failure, and it is invisible on any dashboard that only reports final status. Myth 2: Aging is a flakiness problem. Flakiness is one cause of aging, but not the only one. Tests also age because the system under test drifts (schema changes, API contract drift, environment configuration rot) while the test stays frozen. That is a coverage gap, not a flake — and quarantining it as a flake hides a real product risk.

Myth 3: A PAR dashboard solves the problem. Visualization surfaces the signal; it does not fix anything. The report is an input to an engineering decision — delete, quarantine, rewrite, or escalate. Teams that build the dashboard and then do nothing with the volatile bucket within two sprints end up with a beautiful chart of a degrading suite. The report needs an owner and a triage SLA, not just a Grafana URL in the wiki. Tie it to your sprint planning cycle or it will be ignored by the third week.

A PAR aging report is not a one-time audit — it is a standing signal that your suite is drifting before your release process feels it. Start with the PostgreSQL schema and the 14-day query above, get the Grafana scatter rendering, and set a weekly review cadence for anything in the volatile bucket. Once that habit is in place, explore correlating aging tests with production incident timelines using a production-to-test feedback loop — that is where the report starts generating real engineering value.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.