Artifact Expiry Silently Breaks Failure Trends

Most teams configure artifact retention once, during initial CI setup, and never revisit it. Thirty days feels generous. Then six months later, someone pulls a failure trend chart and the left half of the graph is a flat line — not because the suite was stable, but because the underlying JUnit XML artifacts were purged and the dashboard is silently querying against nulls. The trend looks like improvement. It isn't.

The problem is structural: CI platforms (GitHub Actions, Jenkins, CircleCI, Buildkite) store test artifacts ephemerally by default. When those artifacts expire, any analytics layer that reconstructs trends from raw files — rather than a durable data store — loses its history without warning. No error, no gap marker, just missing data that looks like a clean baseline.

By the end of this article you'll know how to detect whether your trend data has already been silently truncated, how to build an artifact-independent persistence layer that survives expiry policies, and which retention decisions actually matter versus which ones are organizational theater.

How Artifact Expiry Corrupts Trend Integrity

Artifact expiry is a storage-cost policy, not a data-quality policy — but it behaves like one. GitHub Actions defaults to 90 days for artifacts uploaded via actions/upload-artifact; Jenkins' built-in artifact retention discards everything outside the last N builds if you've set a build rotation policy; CircleCI's free tier caps at 30 days. None of these platforms emit a signal to downstream consumers when a referenced artifact no longer exists. A Grafana panel backed by a ClickHouse table populated by a nightly artifact-scraping job simply stops receiving rows. The time-series gap is indistinguishable from "no failures occurred."

This matters most for defect trend analysis, where the entire value proposition is comparing current failure rates against a stable historical baseline. If your baseline window overlaps with an expiry boundary, every regression detection threshold you've calculated is wrong. Flakiness scores, mean-time-to-failure, and P95 test duration baselines are all derived quantities — garbage-in applies here as cleanly as anywhere else in data engineering.

Building a Persistence Layer That Outlives CI Artifacts

The fix is architectural: decouple your analytics data from the artifact lifecycle by writing parsed test results into a durable store at publish time, not at query time. Parse the JUnit XML in CI, extract the fields you care about, and write a row to PostgreSQL or ClickHouse before the job finishes. The artifact can expire on whatever schedule your finance team demands — your trend data survives.

Here's a minimal GitHub Actions step that ships test result rows to a PostgreSQL instance immediately after the test run:

- name: Publish test results to analytics DB
  if: always()
  env:
    DB_URL: ${{ secrets.TEST_ANALYTICS_DB_URL }}
  run: |
    python scripts/ingest_junit.py \
      --junit-glob "reports/**/*.xml" \
      --run-id   "${{ github.run_id }}" \
      --sha      "${{ github.sha }}" \
      --branch   "${{ github.ref_name }}" \
      --db-url   "$DB_URL"

The ingest_junit.py script uses junitparser to walk every XML file and upserts one row per test case into a test_results table. The key columns are run_id, sha, branch, suite, classname, test_name, status, duration_ms, and inserted_at. Once that row exists, the artifact is irrelevant to trend queries.

-- Failure rate per test over rolling 30 days, artifact-independent
SELECT
  test_name,
  COUNT(*)                                          AS total_runs,
  SUM(CASE WHEN status = 'failed' THEN 1 ELSE 0 END) AS failures,
  ROUND(
    100.0 * SUM(CASE WHEN status = 'failed' THEN 1 ELSE 0 END) / COUNT(*), 2
  )                                                 AS failure_pct
FROM test_results
WHERE inserted_at >= NOW() - INTERVAL '30 days'
GROUP BY test_name
HAVING COUNT(*) > 10
ORDER BY failure_pct DESC;

This query runs against your own table, not against CI artifact storage. Pair it with a Grafana time-series panel using inserted_at as the time column and you get a trend that's immune to retention policies. One team running this pattern against ClickHouse reported that triage time dropped from 22 minutes per failure to under 4 once the dashboard stopped returning phantom flat-lines and engineers could trust what they were seeing. For teams already using GitHub Actions, the available tooling for tracking failure trends over time covers several managed options that handle this persistence layer for you, if you'd rather not operate the database yourself.

Retention Mistakes Even Experienced Teams Repeat

The most common mistake is treating artifact retention and analytics retention as the same decision. They aren't. Artifacts (full logs, screenshots, videos, heap dumps) are expensive and legitimately short-lived — 14–30 days is defensible. Parsed test result rows are tiny: a full day's worth of JUnit XML across a large monorepo typically parses to under 5 MB of structured rows. Conflating the two leads teams to either over-retain expensive blobs or under-retain the cheap structured data that actually drives decisions. Separate the policies explicitly; document them in your CI runbook.

A subtler mistake is backfilling trend data from artifacts after the fact — running a scheduled job that scrapes CI artifact APIs nightly. This works until it doesn't: the job fails silently on a Saturday, nobody notices, and three weeks of data vanish. Backfill jobs also race against expiry on short-retention plans. Write-at-publish-time is the only reliable pattern. If you're also tracking duration variance as a stability signal, you need that duration_ms column persisted at the same moment — a backfill job that reconstructs durations from log timestamps introduces measurement error that compounds over months.

Myths That Let the Problem Hide in Plain Sight

Myth 1: "Our dashboard would show a gap if data was missing." Only if you've explicitly configured null-gap detection. Grafana's default behavior connects the last known point to the next known point with a line. A 30-day expiry boundary looks like a smooth trend, not a missing-data warning. Allure's trend charts have the same behavior — they render whatever history the artifact directory contains; if old artifacts are gone, the chart simply starts later. Audit your dashboards by querying COUNT(*) GROUP BY DATE(inserted_at) and looking for unexpected zeroes before trusting any trend.

Myth 2: "Flakiness scores are stable enough to use even with partial history." Flakiness is a rate calculated over a window. Shorten the window by losing history and you get higher variance in the score — a test that failed 3 times in 100 runs looks very different from one that failed 3 times in 20 runs. Teams that rely on flakiness scores to gate quarantine decisions will either under-quarantine (missing the historical failure context) or over-quarantine (inflated rates from short windows). When reading a failure signal correctly, the denominator matters as much as the numerator — and artifact expiry silently shrinks your denominator.

Artifact expiry is a legitimate infrastructure cost control. Letting it eat your analytics history is not a necessary trade-off — it's a missing abstraction. Audit your current retention policies against your oldest reliable trend data point today; if they don't match, you already have a gap. Start with the write-at-publish pattern, get the structured rows into a table you control, and rebuild your baselines from data you can actually trust. From there, an embedded failure analysis dashboard gives you the visualization layer without the artifact dependency.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.