The Nightly Batch Job That Ate the Real-Time Queue

I ran a dynamic-pricing model for a ride-hailing scenario once through a single synchronous endpoint, and I asked that one endpoint to do two genuinely different jobs. One was real-time price quotes for individual ride requests, needing sub-two-hundred-millisecond response. The other was a nightly citywide demand-forecast job, scoring millions of zone-and-time-slot combinations for next-day supply planning, across a synthetic grid of two hundred zones and ninety days of ride history.

The problem

Both workloads hit the same route, and the nightly batch run, a burst of roughly two million scoring calls, drowned out the exact same request queue that real-time quotes depended on. During that window, p99 latency for real-time requests spiked well past the two-hundred-millisecond SLA, and the percentage of requests violating that SLA jumped right alongside it. Riders would have seen stalled price quotes for reasons that had nothing to do with pricing logic and everything to do with an unrelated nightly job sharing the same pipe.

Nothing about this failure involves the model being wrong. The pricing logic itself was fine the whole time. The problem was purely architectural: two workloads with wildly different urgency and volume characteristics, forced through one queue, competing for the same worker pool. A batch job doesn't know or care that it's starving something time-sensitive. It just keeps submitting requests as fast as it can, because that's the job it was given, and the queue serves requests roughly in the order they arrive, real-time or not.

The pattern

The fix is dead simple to describe, even though it took real work to build: physically separate the two workloads. Individual-request scoring and bulk-batch scoring run through separate worker pools and separate request queues, against the same underlying model logic, so a nightly batch job can never contend with, and starve, real-time p99 latency.

I rebuilt the same pricing scenario with the identical workload and identical model logic, but routed it through a dedicated quote endpoint with its own small, latency-tuned worker pool, and a separate batch-score endpoint with its own throughput-tuned pool. Real-time p99 latency stayed under two hundred milliseconds through the same simulated batch window where the single-path version had spiked. Nothing about the pricing model changed. Only the plumbing did.

Design considerations

This pattern solves exactly one problem, resource contention between two workloads with different urgency, and I want to be honest about how narrow that is. It has nothing to say about whether either workload's model is still accurate, whether the input distribution has shifted, or whether the model actually being served is the version anyone thinks it is. Those are separate failures with separate fixes. I didn't compose this with any drift-monitoring or model-promotion logic, and that's on purpose. It's a pure serving-topology fix, and that narrowness is exactly what makes it easy to reason about. Pull it out and everything else about how the model gets validated, promoted, or monitored keeps working, because none of that depends on how the serving path is physically split.

The real calibration question is where to draw the line between "real-time" and "batch" in the first place, and that line isn't always as obvious as quotes-versus-nightly-forecast. Somewhere between a single-request quote and a two-million-row nightly job sits a medium-sized job, say a few thousand scores requested by one internal dashboard, that doesn't cleanly belong to either pool. I'd rather over-separate early, giving a workload its own dedicated path even when the volume looks modest, than discover months later that some "small" job has quietly grown into a second batch job squatting in the real-time pool because nobody thought it needed its own lane at launch.

I'd also flag that this fix costs you something in operational surface area you don't get back. Two pools mean two things to scale, two things to monitor for saturation, and two failure modes to reason about instead of one. If your actual traffic never produces genuine contention, if your batch volume is small enough or your real-time volume light enough that they'd never fight over the same queue in practice, splitting the path is pure overhead with no corresponding benefit. I only reach for this when I can point to an actual contention event, a real spike, a real SLA violation, not because splitting real-time from batch sounds like good hygiene in the abstract. Build the separation when the numbers show you're actually paying the cost of not having it.