A NullPointerException in CheckoutServiceTest means something very different when it fires on a developer's PR branch versus when it fires in the post-deploy smoke suite on staging at 2 a.m. Both are "test failures" in the JUnit XML sense. Neither is the same problem. Teams that flatten all failures into a single red/green view are throwing away the most actionable part of the signal — the where.

The technical problem is that most result-collection pipelines strip context. By the time a failure lands in Allure, ReportPortal, or a Slack notification, the stage metadata — pre-merge, post-merge, scheduled regression, canary smoke — has been discarded or buried. What remains is a test name and a stack trace. That's enough to know something broke; it's not enough to know what to do about it.

By the end of this article you'll have a concrete model for tagging failures by pipeline stage, a SQL query that surfaces stage-correlated failure patterns, and a clear mental framework for why test latency and test throughput metrics read completely differently depending on where in the pipeline they're measured.

Pipeline Stage as First-Class Signal in Test Failure Analysis

A pipeline stage is not just an execution environment — it's a threat model. Pre-merge checks run against unreviewed code on ephemeral infrastructure with mocked dependencies; a failure there is a hypothesis. Post-merge integration runs against merged code on shared infrastructure with real service dependencies; a failure there is a regression. Production canary smoke runs against live traffic with real data; a failure there is an incident. Same test, three different response protocols.

In a modern test architecture, stage context belongs in the result record itself — not just in the CI log. Whether you're writing to PostgreSQL, BigQuery, or ClickHouse, every test result row should carry a pipeline_stage enum (pr_check, post_merge, scheduled_regression, canary_smoke, load_test) alongside the standard test_name, duration_ms, status, and run_id. Without that column, reading a test failure like an engineer is guesswork — you're pattern-matching without a key dimension of context.

Tagging, Querying, and Routing Failures by Stage

Start at the source: inject stage metadata into the test run environment before any test executes. In GitHub Actions, this is two lines:

# .github/workflows/ci.yml (Actions v4)
- name: Run integration tests
  env:
    TEST_PIPELINE_STAGE: ${{ github.event_name == 'pull_request' && 'pr_check' || 'post_merge' }}
  run: pytest tests/integration --tb=short -q

In Pytest, pick that up in conftest.py and attach it to every result via a custom JUnit XML property — Allure and ReportPortal both surface custom properties in their UIs. Now every result record carries the stage. The same pattern works in Buildkite (BUILDKITE_PIPELINE_SLUG) and Jenkins (JOB_NAME parsing) with minor adaptation.

Once stage is in your results store, the queries get interesting. This ClickHouse query surfaces tests that fail disproportionately in canary_smoke relative to post_merge — the classic sign of an environment-sensitive test that passes on shared CI infra but breaks against live dependencies:

-- ClickHouse: stage-skewed failure rate, last 30 days
SELECT
    test_name,
    countIf(pipeline_stage = 'canary_smoke' AND status = 'FAILED')
        / countIf(pipeline_stage = 'canary_smoke') AS canary_fail_rate,
    countIf(pipeline_stage = 'post_merge' AND status = 'FAILED')
        / countIf(pipeline_stage = 'post_merge') AS post_merge_fail_rate,
    canary_fail_rate - post_merge_fail_rate AS stage_delta
FROM test_results
WHERE run_at >= now() - INTERVAL 30 DAY
GROUP BY test_name
HAVING stage_delta > 0.15
ORDER BY stage_delta DESC
LIMIT 25;

A stage_delta above 0.15 (15 percentage points) is a strong signal: this test is not flaky in the random sense — it's environment-sensitive. Route it to the platform team, not the test author. Teams that wired this query to a Grafana dashboard backed by ClickHouse reported triage time dropping from ~22 minutes per canary failure to under 4, because the on-call engineer immediately knew whether the failure was stage-specific or suite-wide. For the Grafana panel wiring, the Grafana + Loki triage setup covers the log-correlation side of that same workflow.

Test Latency and Test Throughput Read Differently Per Stage

Test latency (P95 duration) and test throughput (tests executed per minute) are not flat metrics — their baselines differ by stage. A P95 of 8 seconds for an API contract test is acceptable in a scheduled nightly regression where you have 90 minutes of runway; it's a pipeline-blocker in a PR check where engineers expect sub-3-minute feedback. Similarly, a throughput drop from 120 to 80 tests/minute matters enormously in a load test suite (k6 or Gatling) but is noise in a unit test run. Grafana panels for these metrics must be faceted by pipeline_stage — a single global P95 panel is decorative, not operational. For multi-pipeline visibility at the leadership layer, stage-faceted latency trends are what separate a useful dashboard from a vanity one.

Where Stage-Aware Triage Still Breaks Down

The most common mistake is collapsing retry logic across stages. Teams configure three retries on flaky tests globally, then wonder why their canary smoke suite masks real production regressions — the test passed on retry 2, so the pipeline went green, and the incident ticket opened 40 minutes later via PagerDuty. Retry policy must be stage-specific: zero retries in canary smoke and post-deploy checks (failures there are signals, not noise), up to two retries in pre-merge PR checks where infra flakiness is expected. Argo Workflows and GitHub Actions both support stage-scoped retry configuration; most teams just don't use it.

The second failure mode is routing all failures to the same Slack channel regardless of stage. A #test-failures channel that receives both a developer's PR flake and a canary smoke alert trains engineers to ignore the channel entirely. Separate routing by stage — PR failures go to the PR author via GitHub check annotations, post-merge failures go to the owning team channel, canary failures page the on-call. This is a Slack webhook routing problem, not a test problem, but it's where the signal actually dies in most orgs.

Myths That Make Stage Context Invisible

Myth 1: A failure is a failure — fix it wherever it appears. This treats test failures as atomic facts rather than contextual signals. A test that fails only in pr_check on one engineer's branch is almost certainly a local environment or test isolation issue. The same test failing in scheduled_regression across five consecutive nightly runs is a product regression. Acting on the first with the urgency of the second wastes sprint capacity; ignoring the second because it "only fails in nightly" delays a real fix. Myth 2: Pass rate is the metric that matters. Pass rate without stage breakdown is a vanity metric — a suite with 98% pass rate in PR checks and 72% pass rate in canary smoke is not a healthy suite, it's a suite that's hiding production risk behind optimistic pre-merge infrastructure.

Myth 3: Flakiness is a property of the test, not the stage. A test that flakes 30% of the time in canary smoke but 2% of the time in post-merge is not a flaky test in the classical sense — it's a test that's sensitive to the canary environment's state (stale cache, partial rollout, race on feature flags). Treating it with standard flaky-test quarantine removes a valid production signal. The AI misclassification problem is especially acute here: embedding-based models trained on test names and stack traces without stage features will confidently mislabel environment-sensitive failures as random flakes.

The fix is not a new tool — it's a schema change and a routing policy. Add pipeline_stage to your results store, facet every metric by it, and split your alert routing accordingly. If you're starting from scratch on the data model, the ClickHouse query above is a reasonable first cut. If you already have the data and need pattern detection across thousands of historical runs, embedding-based pattern detection on test history is the next layer worth building.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.