Passing Synthetics Can Report Healthy SLOs

Your synthetic monitors are green. Your SLO dashboard shows 99.94% availability. Your on-call engineer is getting paged anyway because real users are hitting 503s. This contradiction is not a fluke — it is a structural property of how most teams wire synthetics to SLO calculations, and it bites hardest during the incidents that matter most.

The root cause is a conflation of two distinct things: probe success (did the HTTP request return 200?) and service health (did the system do the correct thing for real users at real load?). Synthetics measure the former and teams report it as the latter. The gap between those two measurements is where incidents hide.

By the end of this article you will be able to identify the specific architectural gaps that allow passing synthetics to mask degraded SLOs, instrument your stack to surface the discrepancy in real time, and avoid the organizational patterns that let the problem persist.

The Structural Gap Between Probe Health and SLO Truth

A synthetic test is a controlled probe: a known request, from a known origin, against a known endpoint, on a fixed schedule. It is designed to be deterministic. That determinism is its value for regression detection — and its liability during incidents. Real traffic is not a fixed probe. It is a distribution of request shapes, session states, payload sizes, and concurrency levels that your synthetic almost certainly does not replicate.

SLO error budgets are supposed to reflect user-facing reliability. When you feed them synthetic probe results instead of real-user request telemetry, you are measuring a narrow corridor of your system — often the healthiest one. Load balancers frequently route synthetic probes to the same healthy node. CDN edge caches serve synthetic GETs from cache while origin is on fire. A checkout flow synthetic that hits a single SKU will pass while the product catalog is partially unavailable for every other SKU. If your resiliency tests don't enforce defined SLOs against realistic traffic shapes, the SLO number is a confidence metric for your probe infrastructure, not your service.

Wiring Synthetics and Real-Traffic SLOs to Surface Divergence

The fix is not to abandon synthetics — it is to stop using them as the sole SLO signal and instead correlate them with real-traffic error rates. In Prometheus + Grafana, maintain two separate recording rules: one for synthetic probe success rate and one for real-user 5xx rate derived from your service mesh or application metrics.

# recording rules — prometheus
- record: job:synthetic_probe_success:rate5m
  expr: |
    rate(probe_success_total{job="blackbox"}[5m])

- record: job:real_user_error_rate:rate5m
  expr: |
    sum(rate(http_requests_total{status=~"5..",source="user"}[5m]))
    / sum(rate(http_requests_total{source="user"}[5m]))

With both series in place, a Grafana alert rule can fire when they diverge — synthetic healthy, real-user errors elevated — rather than waiting for a human to notice the contradiction.

# grafana alert — divergence detector
expr: |
  (job:synthetic_probe_success:rate5m > 0.99)
  and
  (job:real_user_error_rate:rate5m > 0.005)
for: 3m
labels:
  severity: page
annotations:
  summary: "Synthetic passing but real-user error rate {{ $value | humanizePercentage }}"

On the tracing side, OpenTelemetry gives you the instrumentation layer to compare synthetic and organic request paths directly. Tag synthetic spans with synthetic=true at the SDK level, then query Honeycomb or Tempo to compare P95 latency and error rate by that attribute. In practice, teams that have done this find that synthetic P95 is often 40–60% lower than organic P95 during degraded states — not because the system is lying, but because the probe bypasses the session-management, auth-token refresh, and cart-hydration paths that real users hit. Once we wired this divergence dashboard to Loki log-based alerting at one platform team, triage time dropped from roughly 22 minutes per incident to under 4 because on-call could immediately see which path was degraded and which was not.

# otel sdk — python, tag synthetic spans
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def run_synthetic_probe():
    with tracer.start_as_current_span("synthetic.checkout") as span:
        span.set_attribute("synthetic", True)
        span.set_attribute("probe.origin", "datadog-synthetics")
        # ... probe logic

For teams using Datadog Synthetics or Checkly, export probe results via webhook into the same time-series store as your APM error rates. Do not let the synthetic vendor's dashboard be the authoritative SLO view — that siloes the signal. A ClickHouse table that receives both streams lets you write a single query joining probe outcomes to real-user request outcomes by timestamp window, which is far more actionable than toggling between two vendor UIs during an incident.

Pitfalls That Keep the Gap Invisible Until It Hurts

Routing asymmetry is the most common culprit. Synthetic probes originate from fixed IPs and often get routed through a different load-balancer pool, bypass WAF rules, or hit a canary exclusion list. Engineers set this up intentionally to prevent probe traffic from polluting real-user analytics — and then forget that the probe is no longer seeing the same infrastructure path. The fix is to periodically run a chaos experiment that deliberately degrades the path real users take and verify that your synthetic probe actually detects it. If it does not, you have confirmed the routing asymmetry and need to address it before the next incident. The same logic applies to broken trace propagation, where synthetics pass but distributed context is silently dropped.

SLO burn-rate alerts fed exclusively by synthetic data produce false confidence at exactly the wrong moment. Because synthetics run on a schedule (often every 60 seconds), a 5-minute burn-rate window may contain only 5 data points. Real-user traffic produces thousands. The statistical weight of those 5 passing probes can dominate a blended SLO calculation and suppress the alert threshold. Separate your SLO burn-rate alerts by signal source — never blend probe success with request-level error rates into a single ratio without weighting by volume.

Myths That Survive Because the Dashboard Looks Fine

"If synthetics pass, the SLO is being met." This is the central myth, and it survives because most incidents eventually resolve before anyone audits the SLO calculation methodology. Synthetic pass rate is a proxy metric — useful for regression detection and smoke testing, not sufficient as the sole SLO input. The same cognitive trap appears elsewhere in test reporting: skipped tests distort every rate you report in the same way that excluded traffic distorts your availability number. The SLO is only as honest as the completeness of its input signal.

"More synthetic checks means better coverage." Adding probes against more endpoints increases the surface area of what you can detect — but it does not fix the structural issue that each probe is a single, narrow, low-concurrency request. A synthetic that checks your search endpoint once per minute tells you nothing about what happens when 800 concurrent users hit it with faceted filters during a flash sale. Complementing synthetics with production observability instrumentation — real-user monitoring, service mesh telemetry, structured logs — is the only way to close the coverage gap. Probe count is not a substitute for signal diversity.

The next concrete step is a 30-minute audit: pull your current SLO data source, identify what percentage of its input is synthetic probe data versus real-user telemetry, and check whether your probe traffic is routed identically to organic traffic. If either answer is uncomfortable, you have found the gap. From there, the divergence alerting pattern above gives you a working starting point — no new vendors required, just honest signal separation.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.