The bug that only existed for our biggest customer: rate limits, loops and an empty map

The bug that only existed for our biggest customer

Our fleet module has a live map. You open it, and every company vehicle shows up with its current position and, next to it, the name of the driver behind the wheel. Positions and driver data come from an external GPS provider over its API.

For most customers this worked fine for a long time. Then one customer reported that their map was simply empty. No vehicles at all. Not a wrong position, not a stale one — nothing.

We could not reproduce it. On our own test company, and on every small customer we tried, the map worked perfectly.

The difference was the number of vehicles

The customer with the empty map runs about 45 vehicles. Our test company has two.

That turned out to be the whole story. The view that builds the map did this:

for position in positions:
    # one API call per vehicle
    current_driver = client.get_vehicle_drivers(position_id_str)
    position["current_driver"] = current_driver

One API call per vehicle, inside the loop. With two vehicles that is two calls and nobody notices. With 45 vehicles, a single map render fires roughly 45 calls at the provider as fast as the loop can run.

The provider's rate limit is 10 requests per second.

Why the map went blank instead of just losing driver names

This is the part that made the symptom confusing. You would expect a rate limit to cost you the driver names — the thing being requested too often — and leave the vehicles on the map.

Instead the whole map died, and the reason is that rate limits are enforced per client, not per endpoint. Once the burst of driver requests pushed the account over the limit, the provider started returning 429 Too Many Requests to everything coming from us for a short window. That included the call fetching the positions themselves.

The positions call failed, the view had nothing to return, and the map rendered empty. The driver lookups were the cause; the positions were the casualty. Looking at the map, you would never guess the two were related.

Worth stating plainly, because it is the generalisable bit: when you exhaust a rate limit, the request that gets rejected is usually not the request that was greedy. It is whatever happens to come next.

The fix: ask less, and ask slower

Two changes, both boring, which is the point.

Cache the answer. A driver assignment changes a few times a day at most. The map polls constantly. There is no reason to ask who is driving vehicle 17 more than once every 30 seconds:

cache_key = f"v2_map_driver:{connector_name}:{cfg.id}:{vehicle_id}"
cached = cache.get(cache_key)
if cached is not None:
    return None if cached == _DRIVER_NONE_SENTINEL else cached

One detail there is worth more than it looks: the sentinel value. "This vehicle has no driver assigned" is a perfectly normal answer, and it must be cached too. If you store None and then test the cache with if cached is not None, every vehicle without a driver misses the cache on every single poll — and those are exactly the vehicles you were trying to stop asking about. A distinct sentinel separates "no driver" from "not cached".

Throttle what is left. Cache misses still happen, so the actual calls are spaced out below the limit:

_MIREO_DRIVER_MAX_PER_SEC = 8     # headroom below the provider's 10/sec
_MIREO_DRIVER_MIN_INTERVAL = 1.0 / _MIREO_DRIVER_MAX_PER_SEC

We deliberately throttle to 8 per second, not 10. The headroom is for the position requests sharing the same budget, and for the fact that several users can have the map open at once. Setting your throttle exactly at the documented limit means the first concurrent request puts you over it.

On top of that, one retry with a short backoff, so a single unlucky 429 does not blank out a driver name. The helper never raises: a missing driver degrades to an empty label, it does not take the map down with it.

What we took away from it

• An API call inside a loop is a design decision, not a detail. It looks harmless in review because the reviewer pictures the data set they know. Ask how many iterations the largest customer has.
• Test against the biggest data set, not the nearest one. Our test company had two vehicles. Any customer with more than a dozen would have surfaced this in minutes.
• Cache negative answers. Otherwise the rows you most want to stop querying are the ones that keep querying.
• Leave headroom under a rate limit. The documented number is a ceiling for your whole account, shared with every other call and every concurrent user.
• Degrade in the right direction. A helper that can fail should fail to "no driver name", never to "no map".

None of this is clever. It is the kind of thing that only becomes visible once a customer grows past the size you were imagining while writing the code — which, if things go well, is a problem you will keep having.