Thundering herd of circuit breakers

A client side circuit breaker stays open for hours, blocking 95% of traffic, and won't close. Our service shows no sign of being unhealthy. To understand the anomaly, we build a custom span visualizer on top of OpenTelemetry trace data and find a circuit breaker trap that our existing dashboards don't reveal.

Valeria PonyaevaOctober 7, 2026 9 min readOriginal

The Case Opens¶

Our service has several clients. One of them has the shape of a pipeline operating on a schedule that produces very high load in the mornings and almost none otherwise. Another one produces a more organic traffic pattern, driven by customer traffic. Our service has a known problem of struggling to cope with the combined morning traffic. While this is not ideal, it is also not a big problem as the clients are robust enough to deal with it.

Let's focus on the client serving organic customer traffic (from now on the client) that's calling our service (from now on the service). During the morning period, it deals with some timeouts, potentially opening and closing a circuit breaker, and recovers without major issues.