GitLab CI Test Reports: End to End

Most teams treat CI build logs like a fire alarm: red means stop, green means ship. GitLab CI's test reporting layer offers something more useful — a structured, queryable record of every test run, retry, and duration across every pipeline. The signal buried in that data is where real engineering decisions get made: which tests are silently flaking, which suites are aging into dead weight, which failures are correlated with a specific runner or branch pattern.

The problem is that GitLab's test report feature is deceptively easy to misconfigure. Teams wire up artifacts:reports:junit, see the summary tab, and assume they're done. They're not. Without retention tuning, downstream aggregation, and a clear distinction between raw results and actionable insights, the reports become a prettier version of the same checkbox they already had.

By the end of this article you'll know how GitLab CI test reports actually work under the hood, how to build a durable analytics layer on top of them, and where the architecture breaks down in ways that senior engineers still get caught by.

How GitLab CI Test Reports Actually Work

GitLab CI test reports are JUnit XML files collected as a special artifact type during a pipeline job. When a job completes, GitLab parses any XML matched by artifacts:reports:junit, stores the structured data separately from the raw artifact blob, and surfaces it in the merge request (MR) widget and the pipeline's Tests tab. The key distinction: the raw XML blob follows normal artifact retention rules, but the parsed test data is stored in GitLab's database and survives even after the artifact expires — up to the configured artifacts:expire_in window for the parsed records themselves.

In a modern test architecture, this sits at the collection layer: your test runner (Pytest with --junitxml, JUnit 5, Playwright's built-in reporter, k6 with a JUnit output plugin) emits XML, GitLab ingests it, and the MR widget shows new failures, fixed tests, and total counts. That's the out-of-the-box story. Everything beyond that — trend analysis, flakiness scoring, per-suite P95 runtimes — requires you to pull the data out via the API or ship it to an external store. Understanding this boundary is what separates teams that get real insights from test reports versus teams that just have a nicer UI on the same pass/fail signal.

Wiring GitLab CI Test Reports Into a Durable Analytics Pipeline

Start with the job definition. The minimal correct config looks like this — and the glob pattern matters more than most people realize:

test:unit:
  stage: test
  script:
    - pytest tests/ --junitxml=reports/junit.xml -v
  artifacts:
    when: always
    expire_in: 30 days
    reports:
      junit: reports/junit.xml
    paths:
      - reports/junit.xml

when: always is non-negotiable. Without it, a failed job won't upload artifacts, and GitLab never sees the XML — you get a silent gap in the test summary. The paths entry alongside reports:junit ensures the raw file is also available for downstream jobs or external consumers. Set expire_in to at least 30 days; shorter windows break any trend query that spans a sprint boundary.

For multi-job pipelines, GitLab merges all JUnit reports across jobs into a single pipeline-level summary. Use a consistent naming convention like reports/junit-${CI_JOB_NAME}.xml and a glob in the report path:

artifacts:
  reports:
    junit: reports/junit-*.xml

Once the artifacts are flowing, pull the parsed data via the GitLab Pipelines API to build a real analytics layer. The /projects/:id/pipelines/:pipeline_id/test_report endpoint returns structured JSON with suite names, case names, durations, and statuses. A nightly Python job that ingests this into PostgreSQL or ClickHouse gives you the foundation for trend queries:

-- P95 test duration per suite over the last 30 days
SELECT
  suite_name,
  PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY duration_seconds) AS p95_duration,
  COUNT(*) FILTER (WHERE status = 'failed') AS failure_count,
  COUNT(*) AS total_runs
FROM test_case_results
WHERE pipeline_created_at >= NOW() - INTERVAL '30 days'
GROUP BY suite_name
ORDER BY p95_duration DESC;

This query alone surfaces the suites worth investigating first — not the loudest failures, but the ones with high runtime variance and a non-trivial failure count. Teams that have wired this dashboard to Loki for log correlation report triage time dropping from 20+ minutes per failure to under 5, because the slow outlier and its log context appear on the same screen. For the flakiness signal specifically, join on test_case_name across pipeline runs and flag any case that alternates between passed and failed on the same commit SHA — that's your retry-masked flake.

Where GitLab Test Report Pipelines Break Down

Artifact expiry misalignment is the most common silent failure. Teams set expire_in: 1 day for storage cost reasons, then wonder why their trend dashboard has gaps. The parsed test data in GitLab's DB may persist longer than the raw XML, but any external consumer that re-fetches the artifact for re-parsing hits a 404. Fix: keep raw artifacts for at least 14 days and archive the parsed JSON to object storage (S3, GCS) on first ingest — treat the API response as ephemeral, not the source of truth.

Parallel job report collisions are the second trap. When multiple jobs write to the same XML path inside a single job (e.g., parallel matrix builds), the last writer wins and you silently lose test cases. Each parallel slice needs a unique output path, typically keyed on ${CI_NODE_INDEX} or ${CI_JOB_NAME}. The glob pattern in reports:junit handles the merge correctly; the mistake is assuming GitLab merges files written to the same path. It doesn't — it picks one. This is also where continuous quality feedback loops break down organizationally: the data loss is invisible until someone audits total test counts across branches.

What Teams Consistently Get Wrong About GitLab Test Reports

The MR widget is not an analytics layer. GitLab's built-in test summary shows new failures and fixed tests relative to the target branch — it's a diff view, not a trend view. Teams mistake "no new failures" for "suite health is stable." A suite can have 40% of its cases flaking intermittently, all of them passing on this particular run, and the MR widget shows green. The KPI dashboard that engineering leaders actually need requires a time-series store and explicit flakiness scoring, neither of which GitLab provides out of the box.

Retry masking is an underappreciated data corruption vector. GitLab's retry: keyword re-runs failed jobs and, if the retry passes, marks the pipeline green. The test report attached to the final (passing) job run is what gets surfaced — the intermediate failure disappears from the UI. If you're not capturing the test report from every attempt (including failed ones) and storing them independently, your flakiness signal is systematically suppressed. Wire artifact collection to when: always on every retry attempt and tag each record with the attempt number when ingesting into your store. A test that passes on attempt 3 of 3 is not a healthy test — it's a flake that got lucky, and tracking quality over time requires that distinction to be explicit in your data model.

GitLab CI test reports are a solid collection primitive — reliable, well-integrated into the MR workflow, and API-accessible for external consumers. Their ceiling is low if you stop at the UI. The teams getting real value are the ones treating the JUnit XML as an event stream: ingested into ClickHouse or PostgreSQL, queried for P95 runtimes and flakiness rates, and correlated with deployment and log data. Start by auditing your current expire_in settings and confirming artifact upload on job failure — those two fixes alone close the most common data gaps before you build anything else on top.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.