Resiliency Tests That Enforce Defined SLOs

Most teams treat SLOs as a monitoring concern — something Datadog or Prometheus watches after the fact. The gap shows up in postmortems: a service degraded for 40 minutes before anyone noticed because the SLO existed on a dashboard nobody checked, not in a test that would have caught the absence of that definition before deploy. The signal worth acting on is whether your SLOs are defined, reachable, and actually enforced — not just declared in a YAML file.

Resiliency tests close that gap. They are executable assertions — running in CI and in production-adjacent environments — that verify your error budgets are specified, your latency targets are measurable, and your system behaves within contract under fault injection. They sit at the intersection of SLO-driven testing and chaos engineering, but they are simpler than either term implies.

By the end of this article you will know how to structure resiliency tests that fail a pipeline when SLOs are undefined or breached, wire them into GitHub Actions, and build the scorecard queries that make quarterly reviews meaningful rather than ceremonial.

What a Resiliency Test Actually Asserts About Your SLOs

A resiliency test is a parameterized, repeatable test case that verifies a system's behavior against a declared reliability contract — not just that the system is "up." Concretely, it asserts three things: the SLO exists (the target is defined and version-controlled), the SLO is measurable (the telemetry pipeline can produce the denominator and numerator), and the SLO is satisfied under the fault conditions you care about (latency spike, dependency timeout, partial data loss). Failing any one of those three is a pipeline-blocking event, not a Slack notification.

In a modern test architecture these tests live in two places: a pre-deploy suite in CI that validates definition and measurability, and a continuous synthetic suite in production that validates satisfaction. The CI layer catches "we renamed the metric and forgot to update the SLO spec." The synthetic layer catches "the SLO is defined but the p99 has been drifting for six days." Together they form the closed loop described in production-to-test quality feedback — where production signals drive test coverage, not the other way around.

Building the CI Pipeline That Fails When SLOs Are Missing or Breached

Start with a machine-readable SLO manifest. A simple JSON schema works; the important constraint is that every service in your registry must have a corresponding entry or the test fails. Store the manifest in the repo alongside the service code so drift is caught at PR time.

# slo_manifest.json (abbreviated)
{
  "services": [
    {
      "name": "checkout-api",
      "availability_target": 0.999,
      "latency_p99_ms": 250,
      "error_budget_window_days": 30,
      "telemetry_metric": "http_request_duration_seconds"
    }
  ]
}

A Pytest suite then loads the manifest and queries your metrics backend — Prometheus, Datadog, or a ClickHouse table of OTEL spans — to assert measurability. The following fixture uses the Prometheus HTTP API; swap the base URL for Datadog's metrics query endpoint if needed.

import pytest, requests, json

MANIFEST = json.load(open("slo_manifest.json"))
PROM = "http://prometheus.internal:9090"

@pytest.mark.parametrize("svc", MANIFEST["services"])
def test_slo_metric_exists(svc):
    metric = svc["telemetry_metric"]
    r = requests.get(f"{PROM}/api/v1/query",
                     params={"query": f'count({metric}{{service="{svc["name"]}"}})'},
                     timeout=5)
    data = r.json()["data"]["result"]
    assert data, f"No data for {metric} on {svc['name']} — SLO unmeasurable"

@pytest.mark.parametrize("svc", MANIFEST["services"])
def test_latency_p99_within_slo(svc):
    q = (f'histogram_quantile(0.99, rate({svc["telemetry_metric"]}_bucket'
         f'{{service="{svc["name"]}"}}[5m]))')
    r = requests.get(f"{PROM}/api/v1/query", params={"query": q}, timeout=5)
    result = r.json()["data"]["result"]
    assert result, f"No histogram data for {svc['name']}"
    p99_s = float(result[0]["value"][1])
    assert p99_s * 1000 <= svc["latency_p99_ms"], (
        f"{svc['name']} p99={p99_s*1000:.0f}ms exceeds SLO of {svc['latency_p99_ms']}ms")

Wire this into GitHub Actions so it runs on every PR that touches the manifest or a service directory. The fail-fast: false flag is intentional — you want the full picture of which services are out of contract, not just the first failure.

name: SLO Resiliency Gate
on:
  pull_request:
    paths: ["slo_manifest.json", "services/**"]
jobs:
  slo-check:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install pytest requests
      - run: pytest tests/resiliency/ -v --tb=short
        env:
          PROMETHEUS_URL: ${{ secrets.PROMETHEUS_URL }}

For the fault-injection layer, use a tool like Chaos Toolkit or k6 with fault scenarios. A k6 script that introduces 200ms artificial latency via a proxy and then asserts the SLO threshold is not breached is more honest than a unit test mocking a timeout. One team running this pattern against their payment service reduced triage time from 22 minutes per failure to under 4 once failures linked directly to the SLO spec line rather than a generic 500 error in Loki. Pair these results with CI observability metrics so you can distinguish a resiliency regression from a flaky network call in the test environment itself.

Where Resiliency Test Suites Break Down in Practice

The most common failure mode is testing the SLO definition file rather than the SLO. Teams write a JSON schema validator, it passes, and they ship. The manifest is syntactically valid but the metric name changed three sprints ago. The fix is the measurability assertion shown above — the test must hit the live telemetry backend, not just lint a file. A related mistake is running resiliency tests only in staging, where traffic is synthetic and metric cardinality is low. A p99 computed from 40 requests per minute is not the same signal as one computed from 40,000. Gate on staging but also run the synthetic suite in production on a read-only probe path.

The second pitfall is treating skipped resiliency tests as passing. If a service is excluded from the manifest "temporarily," the skip silently inflates your pass rate. This is the same distortion problem that makes skipped tests dangerous in any rate calculation — the denominator shrinks and your reliability picture looks better than it is. Enforce a policy: every service in the registry must have a manifest entry, and any test marked skip must carry a linked issue with an expiry date.

Myths That Undermine SLO Enforcement Through Testing

Myth 1: An SLO dashboard means the SLO is enforced. Grafana showing a burn rate panel is not the same as a test that blocks a deploy when the burn rate exceeds threshold. Dashboards are observational; tests are contractual. Until a resiliency test fails a pipeline, the SLO is a suggestion. Myth 2: High availability test coverage implies the SLO is achievable. Coverage tells you what code paths were exercised; it says nothing about whether the system can sustain its error budget under realistic load or partial failure. A service can have 90% branch coverage and a completely unmeasurable SLO because no one wired up the histogram bucket.

Myth 3: Flaky resiliency tests mean the system is flaky. Sometimes they do — but more often a flaky resiliency test means the test is querying a metric with insufficient retention, hitting a Prometheus scrape gap, or running against a staging environment with no baseline traffic. Before quarantining a resiliency test, check the query window and the scrape interval. A quarterly reliability scorecard that tracks resiliency test flake rate separately from functional test flake rate gives you the data to make that call with evidence rather than intuition.

Resiliency tests are not a monitoring replacement — they are the enforcement layer that makes your SLOs load-bearing. Start with the manifest-plus-measurability pattern, gate it in CI, and add fault injection once the baseline is stable. The quarterly scorecard query is your feedback loop: if resiliency test failures are trending up while functional failures hold steady, you have a reliability definition problem, not a code quality problem. That distinction is worth having before the next incident review.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.