TeamCity Flaky Test Detection, Explained

Most teams treat test results like a checkbox: green is good, red is bad, ship or block. TeamCity's flaky test detection is one of the few CI-native features that tries to break that binary — automatically flagging tests that alternate between pass and fail across consecutive builds without a code change as the trigger. If you've ever watched a test go red, re-triggered the build, watched it go green, and shipped anyway, you already understand the problem it's solving.

The technical challenge is that "flaky" is a distribution problem, not a binary state. A test can be 95% reliable and still destroy team trust if it fails during a release gate. TeamCity's detection gives you a starting point, but the signal it surfaces is narrow: it catches alternating outcomes in recent history, not slow-burn intermittency, environment-correlated failures, or timing-dependent races that only appear under load.

By the end of this article you'll understand exactly how TeamCity's algorithm works, how to extend it with SQL-level analysis and a quarterly reliability scorecard, and where the built-in detection leaves gaps that cost real engineering time.

How TeamCity's Flaky Detection Algorithm Actually Works

TeamCity marks a test as flaky when it observes status inversions — a test that passed in build N fails in build N+1 (or vice versa) without any changes to the test's source file between those builds. The detection window is configurable but defaults to the last 15 builds. A test that flips status at least once within that window, with no correlated code diff, gets the flaky badge in the UI and is excluded from build-failure attribution by default. That exclusion is the part most teams miss: a flaky test can be silently eating failures without ever blocking a build.

Under the hood, TeamCity stores per-test history in its internal database (HSQLDB for small installs, PostgreSQL or SQL Server for production). The flakiness score is a ratio of inversions to total runs in the window — not a raw count. This matters because a test that runs 200 times and flips once is treated very differently from one that runs 10 times and flips 3 times. Where TeamCity's model is weak: it doesn't weight by recency, it doesn't correlate against agent topology or environment variables, and it resets the window on each new detection cycle, so a test that was flaky three months ago and is flaky again today looks identical to one that's been consistently flaky the whole time.

Extending TeamCity Data Into a Real Reliability Dashboard

TeamCity exposes its test history via REST at /app/rest/testOccurrences?locator=test:(name:<testName>),count:100. Pull that into a PostgreSQL warehouse and you can run the kind of analysis the UI won't give you. The query below computes a 30-day inversion rate per test, weighted toward the last 7 days — the shape of signal that actually predicts whether a test will bite you in the next sprint.

-- Assumes test_runs(build_id, test_name, status, build_ts, agent_name)
WITH ranked AS (
  SELECT
    test_name,
    status,
    build_ts,
    LAG(status) OVER (PARTITION BY test_name ORDER BY build_ts) AS prev_status,
    build_ts > NOW() - INTERVAL '7 days' AS is_recent
  FROM test_runs
  WHERE build_ts > NOW() - INTERVAL '30 days'
),
inversions AS (
  SELECT
    test_name,
    COUNT(*) FILTER (WHERE status != prev_status)                  AS total_flips,
    COUNT(*) FILTER (WHERE status != prev_status AND is_recent)    AS recent_flips,
    COUNT(*)                                                        AS total_runs
  FROM ranked
  WHERE prev_status IS NOT NULL
  GROUP BY test_name
)
SELECT
  test_name,
  ROUND(total_flips::numeric / NULLIF(total_runs, 0), 4)  AS flip_rate_30d,
  ROUND(recent_flips::numeric / NULLIF(total_runs, 0), 4) AS flip_rate_7d,
  total_runs
FROM inversions
WHERE total_flips > 0
ORDER BY flip_rate_7d DESC;

Pair this with agent-level grouping (GROUP BY test_name, agent_name) and you'll surface environment-correlated flakiness that TeamCity's built-in detection misses entirely — tests that only flip on Windows agents, or on agents with less than 4GB free RAM at build time. One platform team at a mid-size SaaS org cut their flaky-test triage time from 22 minutes per failure to under 4 once they wired this query output to a Grafana dashboard backed by Loki log correlation.

For a Grafana panel, use the PostgreSQL data source and plot flip_rate_7d as a time series with a threshold annotation at 0.05 (5% inversion rate). That's a reasonable P95-informed cutoff: tests above it are failing roughly 1 in 20 runs, which is enough to meaningfully erode CI signal. If you're building a CI reliability dashboard, this rate alongside P95 build duration and retry counts gives you a far more honest picture than pass/fail percentage alone.

# Grafana panel JSON fragment (PostgreSQL datasource)
{
  "type": "timeseries",
  "title": "Flaky Test Inversion Rate (7d rolling)",
  "targets": [{
    "rawSql": "SELECT date_trunc('day', build_ts) AS time, test_name, AVG(flip_rate_7d) AS value FROM flaky_summary GROUP BY 1, 2 ORDER BY 1",
    "format": "time_series"
  }],
  "fieldConfig": {
    "defaults": {
      "thresholds": {
        "steps": [
          {"color": "green", "value": 0},
          {"color": "yellow", "value": 0.02},
          {"color": "red", "value": 0.05}
        ]
      }
    }
  }
}

For quarterly planning, roll this up into a reliability scorecard that tracks trend lines per test suite, not just point-in-time counts. A scorecard that shows a suite's inversion rate dropping from 8% to 2% over 12 weeks is a concrete engineering outcome you can defend in a planning retro — far more useful than "we fixed 14 flaky tests."

Where TeamCity's Detection Trips Up Experienced Teams

The most common mistake is treating the flaky badge as a quarantine, not a signal. TeamCity's default behavior excludes flaky tests from build failure attribution, which sounds helpful until you realize it means a test suite can have 30 flaky tests silently accumulating while every build stays green. Teams that don't actively monitor the flaky test list — or don't have a rotation to triage it — end up with a growing shadow failure surface. The real cost of flaky tests isn't just wasted CI minutes; it's the engineering trust erosion that makes teams stop believing their test suite at all.

A second pitfall is the 15-build default window. For teams running 50+ builds a day, 15 builds is less than 20 minutes of history. A test that's intermittently flaky on a weekly cadence — perhaps tied to a cron-driven integration or a third-party API rate limit — will never accumulate enough inversions in that window to get flagged. Widening the window to 100 builds in TeamCity's Administration → Investigations and Muting settings is a one-minute config change that dramatically improves detection recall for slow-burn flakiness.

Myths About Automated Flaky Detection in TeamCity

Myth 1: If TeamCity doesn't flag it, it's not flaky. TeamCity's inversion model is a subset of flakiness patterns. Tests that are consistently slow and occasionally breach a timeout, tests that fail only under parallel execution, and tests with order-dependency failures won't generate status inversions in a predictable enough pattern to trip the detector. You need JUnit XML parsing with runtime variance analysis running alongside the built-in detection to catch the full spectrum. Myth 2: Muting a flaky test fixes the problem. TeamCity's mute feature suppresses the failure notification; it does nothing to the underlying defect. Muted tests are a debt register, not a resolution queue. Teams that mute without a linked ticket and SLA for resolution end up with muted tests that are 18 months old and no one remembers why.

Myth 3: More detection automation means less human judgment. Automated detection narrows the search space; it doesn't replace root cause analysis. A test flagged as flaky by TeamCity still needs a human (or an AI-assisted classifier) to determine whether the root cause is a race condition in the test, a resource leak in the application, or genuine non-determinism in the environment. Skipping that step and bulk-quarantining everything the detector flags is how teams end up hiding real regressions behind a flakiness label.

TeamCity's flaky detection is a solid baseline — better than nothing, worse than a purpose-built observability layer. Wire its REST API into a PostgreSQL warehouse, build a Grafana dashboard with recency-weighted inversion rates, and run a quarterly scorecard review against trend lines rather than raw counts. The next concrete step: pull 30 days of test history from TeamCity's API this week and run the inversion-rate query above. The distribution will tell you more than six months of build dashboards.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.