Thundering Herd: What Breaks When Everyone Looks at Once
A motion alert fires. In the next few seconds, every person watching that camera's feed, plus everyone who just got a push notification about it, opens the stream at once. That's a thundering herd, and it is not the same load pattern as the slow, steady climb of background viewership you'd see on a normal evening. It's a step function. Zero to a crowd, in the time it takes a notification to land on a few hundred phones.
I closed the last post in this series arguing that load tests need to treat viewers-per-stream and streams-per-instance as separate axes, because a test that only varies total connection count misses the failure mode each axis produces on its own. Thundering herd applies that same lesson to time. A test plan that ramps load gradually and calls it done has verified the system survives one shape of demand. It has said nothing about whether the system survives the shape that actually matters most in a security or alert-driven product: everyone arriving in the same six seconds, on the one stream that just became interesting.
That's worth its own dedicated scenario, not a bigger number plugged into the steady-load test. A gradual ramp gives your autoscaler, your connection pool, your SFU's session setup path room to breathe between each new joiner. A thundering herd compresses all of that into a single narrow window: ICE negotiation, keyframe requests, and TURN allocation checks for every new viewer, all landing at once. Whatever margin your system carries under gradual load gets consumed in seconds, all at once, with none of the natural spacing a steady ramp provides.
Think about where those viewers are actually coming from. A push notification fan-out isn't a smooth curve either. Notification delivery clusters tightly around the moment it's sent, so the client-side request pattern arrives at your gateway already shaped like a spike, not a ramp. Your load test needs to generate that same clustering on purpose. A script that opens a batch of connections in a tight burst is closer to the real failure pattern than a load generator that ramps smoothly over thirty seconds, because the real event doesn't ramp. It arrives.
When something breaks, work the pipeline stage by stage
Load tests, thundering-herd or otherwise, exist to find the place where the system gives first. When they succeed at that job, you get a real incident, or at least a real near-miss, and the job shifts from testing to diagnosis. Here's the order I actually work through, because guessing at the bottleneck wastes more time than checking each stage in sequence.
CPU. Transcoding is almost always the cost center. Packetization and encryption barely register next to it on any modern instance. If CPU is climbing, the transcode pipeline is where I look first, not the connection-handling code around it. The catch is that the metric you need doesn't live at the host level. A host-level average across a dozen streams will read comfortably mid-range right up until the incident, because it's hiding the one stream on that host that's actually saturated. Profile per-stream. The hot stream is the one holding up the whole instance, and it's invisible in a fleet-wide average.
Network. Don't look at "network" as one number. Ingress, from source to gateway, inter-instance and TURN relay traffic, and egress, from gateway to viewers, are three separate cost centers with three separate failure signatures. A blended network metric will tell you utilization is elevated. It won't tell you whether that's a source struggling to push RTSP upstream, a relay tax from viewers who can't get a direct ICE path, or an egress ceiling from a stream that just picked up a few hundred new viewers. Split the metric before you start chasing the cause, or you'll spend the incident chasing the wrong one.
Transcoding. Before spending a single hour optimizing the transcode pipeline, confirm it needs to run at all. I covered the codec and profile-compatibility logic for this earlier in the series: plenty of viewer requests can be served by forwarding an already-compatible stream without touching the encoder. Eliminating a transcode step beats speeding one up, every time, because the eliminated step costs nothing at any scale and the sped-up step still costs something at every scale. Check for unnecessary work before you optimize necessary work.
Latency. Break the full path down before you blame any one segment: capture and encode at the source, source-to-gateway over RTSP, gateway processing, gateway-to-viewer over ICE and WebRTC, and client-side decode and render. In most setups I've worked, GOP length and jitter buffer settings eat more of the latency budget than the network transit itself does. Teams reach for network-level fixes, shorter routes, better peering, when the bigger win was sitting in an encoder GOP setting or a jitter buffer tuned for a network condition that no longer applies. Check the configuration knobs before you go looking for a network problem to solve.
Browser playback. Sometimes every server-side metric is clean and playback is still choppy. When that happens, the usual suspects are jitter buffer tuning, packet loss concealment behavior, and decoder performance on lower-end client devices. This is a client-side problem. Chasing it on the server wastes the time you don't have during an incident, and it's exactly the failure mode a pure synthetic-client load test will never surface, which is the whole reason a real browser-based test earns its keep even though it costs more to run.
The thundering herd is the real test
Here's the thing about a triggering event in a security or alert-driven product: it doesn't test one part of the system. It tests all of it, at once, under time pressure, with no ramp to absorb the shock.
The topology choice from the start of this series, whether an SFU can hold up under fan-out at the moment a stream goes from five viewers to five hundred, gets tested the instant the alert fires. The control plane and data plane split, and the sharding decisions built on top of it, get tested by whether the instance holding that one stream was provisioned for this moment or just for its average day. The TURN budget gets tested by how many of those five hundred new viewers can't take a direct path and all show up on the relay layer within the same few seconds. The transcoding discipline and the keyframe cadence get tested by whether every new joiner gets a clean, fast first frame or sits waiting out a GOP interval that made sense for steady playback and makes no sense for someone joining at the worst possible moment. The gateway and platform choices get tested by whether they were built to absorb a spike in seconds or built only for an average load curve, with autoscaling tuned for minutes when the event demands seconds. The migration discipline matters the moment a spike lands mid-cutover on a version that was never meant to carry production weight. Failover matters most exactly when the instance holding that stream goes down mid-event, because load and failure risk both peak at the same moment, and a quiet stretch was never the real test of whether failover works. And the load-testing practice from the last post matters because it's the only way any of this gets validated before a real alert does it for you, live, with real consequences attached to the outcome.
None of those pieces were ever really independent. They just look that way when nothing is happening. A steady evening of background viewership will never tell you which of them is the weak link, because a steady evening never asks all of them to answer at once. The thundering herd is what asks. It's the moment a security platform exists to handle, the exact scenario the whole system was built for, and it's also the moment that finds every shortcut, every untested assumption, and every corner cut during a calmer week.
A system that has only ever survived steady-state load was never actually tested. It was rehearsed. The real test is the one where everyone looks at once, and there's no version of this problem, not the topology, not the codec, not the failover path, not the load test itself, that gets to sit out that moment.