The average that never happened

We ran YOLO26n for seventy-two minutes non-stop on a Raspberry Pi 5: half an hour with one process using all four cores, ten minutes of cooling and half an hour with four processes at once, one core each. Put the 25,898 inferences together and take the average, and you get 347 ms. No inference took that long. None even took between 268 and 397 ms: the average falls in an empty gap. It describes something that never happened.

For a long time we got used to asking how long a system takes to respond. This note is about why that question, answered with a single number, is not enough.

The figures in this note are not illustrative: they come from the sustained runs we published under CC BY 4.0, and anyone can recompute them.

It matters for different reasons

In the systems we have built, latency matters in different ways. In an electronic top-up there is a financial transaction and a customer waiting: a late response creates uncertainty, because you do not always know right away whether the operation was rejected, went through, or simply did not respond in time. In GPS telemetry, the information has to arrive in time for alerts to have operational value. In video analytics, if processing takes longer than the stream of images allows, frames pile up or detection opportunities are lost.

Historically, a good part of the monitoring has focused on finding slow transactions, errors and timeouts. Today I find reporting only an average insufficient. I want the whole distribution, especially the p95, the p99 and the extremes.

Each condition separately

The p50, or median, is the time below which half the measurements fall; the p95, 95 %; the p99, 99 %. In milliseconds:

Device and condition	Mean	p50	p95	p99	Maximum
Pi 5, one process	130.2	129.7	135.2	137.7	267.9
Pi 5, four processes	595.6	595.4	608.3	615.4	702.3
Pi 5, mixed	347.4	134.7	605.3	611.8	702.3
N100, one process	56.7	56.2	57.5	63.6	159.5
N100, four processes	218.3	217.9	224.5	231.4	297.7
N100, mixed	139.0	209.1	223.1	227.3	297.7

With one process, the Pi’s median stayed between 128 and 131 ms; with four, between 593 and 597. Both conditions keep the whole processor busy; what changes is how it is shared. Those are two different behaviors, and each one, on its own, fairly predictable. Mixed, the average falls between them, where there is nothing. The mixed median does not describe the four-process phase either: it depends on which of the two phases had more inferences.

On the N100 it goes the other way: the mixed average, 139 ms, falls below the median, 209. The average can mislead in either direction.

I have no documented production incident in which a good average hid a failure. There is no need to make one up: here it is, measured, how an aggregate metric erases the difference between two operating conditions.

The tail within one condition

Separating the phases does not finish the job. With one process, the Pi had a p99 of 137.7 ms, but its maximum was 267.9: almost double. Only 51 of 13,813 inferences went past 150 ms. On the N100, the p99 was 63.6 ms and the maximum, 159.5.

Then there is the first inference. In Round 1 we discarded twenty warm-up inferences; in the sustained test we recorded every inference from the first one, without discarding any. It shows: the first one took 148 ms on the Pi, against about 125 for the ones after it, and 155 on the N100, against about 56. Keeping it makes it possible to see the startup and how it differs from the steady state. But they are two different methodologies, and they should not be presented as if they were the same.

Latency is not capacity

With four processes, each inference on the Pi took 4.6 times longer, because it runs on one core instead of four. The device, however, went from 7.7 to 6.7 inferences per second, because four were running at once. On the N100, each inference took 3.9 times longer and capacity rose from 17.6 to 18.3 per second.

If you only look at latency, four processes look like a disaster. If you only look at capacity, they cost little: 13 % less on the Pi and 4 % more on the N100. They are two different questions, and an architecture needs both answers.

What a short test does not see

The third dimension is time. On the Pi with one process, the median of each five-minute window rose from 128 to 131 ms while each window’s peak temperature went from 77 to 85 °C. On the N100 with four processes, from 213 to 222 ms, with peaks from 92 to 102 °C. What the processor’s clock did meanwhile is in the throttling note.

A short test gets it wrong in both directions. The first 100 inferences of the Pi with four processes give a median of 577 ms; the whole run, 595. The first 100 with one process give a p99 of 148.5 ms, against 137.7 for the whole run: there were several slow inferences at startup, the first one among them, and with 100 measurements the p99 rests on one or two.

An architecture should not be judged only by its best result, but by the behavior it can sustain.

How much to trust a p99?

With 100 measurements, a single one is left above the p99: the p99 depends on one measurement. Assuming nothing about the shape of the distribution, and with 95 % confidence, with 100 samples you can put a floor under the p99, but not a ceiling; that takes at least 368. With the 13,813 from the Pi with one process, the p99 would lie between 137.4 and 137.9 ms, if the measurements were independent and from the same conditions. They are not entirely: the p99 of each five-minute window went from 135.0 to 138.3, more than that whole interval.

But sample size is no substitute for representativeness. Ten thousand measurements on a cold device, with no other load, over five minutes, say a lot about those conditions and little about the ones in operation.

The timeout that improves the p99

There is a seemingly small decision that changes the interpretation: what to do with the operations that go past the timeout. If an operation ends by timeout, its real latency is unknown. It cannot be given the timeout’s value to compute percentiles: it is a censored observation, and it counts as a failure.

Take the Pi with four processes, a 600 ms target and a 605 ms timeout. Of 12,085 inferences, 1,407 would not finish: 11.6 %. Counting them as failures, the real compliance with the target is 69.6 %, and the p95 and the p99 have no value: all that is known is that they are over 605 ms. If they are discarded, the sample keeps only the ones that finished: compliance rises to 78.7 % and the p99 comes out at 604.7 ms, below the timeout by construction.

With the phases mixed and a 300 ms timeout it is worse. Here 46.7 % do not finish, and discarding them, the p99 lands at 137.7 ms and compliance with a 150 ms target at 99.6 %. Counting them, that compliance is 53.1 %. Discarding the timeouts makes the four-process phase disappear from the measurement.

Inference is not the system

All of the above is inference latency, with random input. A video system has capture, preprocessing, inference, postprocessing and generating the event. A camera at 30 frames per second produces one every 33.3 ms. That does not force you to run inference on every frame, but it does force you to define what rate the application needs: to detect a vehicle or a person, a selection of frames may be enough, depending on how fast things move and what you are looking for. What you cannot do is let frames pile up without limit.

These tests measure inference under controlled conditions. They are not evidence of plates or frames lost in production: that takes measuring the whole chain.

What I would do today

Separate the conditions. Do not mix the device’s measurements under one condition with the same device’s under another, such as one process and four processes, or inference latency with the whole system’s.

Ask for the distribution, not a number. From a vendor I would ask, at a minimum, for p50, p95, p99, maximum observed and the share of operations that exceed the target time. And for the conditions of the measurement: concurrent requests, test duration, hardware and operation volume.

Write the target with a percentile and a window. In a contract, a latency target tied to a percentile and an evaluation window, with separate requirements for availability and error rate. For example, that 99 % of the operations meet a certain time, counting errors and timeouts as failures so they do not disappear from the measurement. The threshold depends on the application: I would not use the same one for a top-up, an administrative API and real-time detection.

Do not confuse the timeout with the response time. In top-up integrations we have configured timeouts of up to 60 seconds. That says nothing about how long a top-up normally takes, and it is not necessarily a contractual commitment.

Measure three things at once. Latency: how long each operation takes and how its times are distributed. Capacity: how many operations the system completes per second under different loads. Stability: how both evolve after operating continuously, with temperature, processor frequency and throttling.

Keep every measurement. In edge AI we keep every inference and the device’s telemetry. That makes it possible to rebuild a run’s behavior and recompute any percentile, instead of depending on a summary.

To see it with this data or yours, there is the latency tool: it opens the 90,604 inferences phase by phase, tells you how much to trust each percentile and compares counting and discarding the timeouts. It runs in the browser, and if you open your CSV, it is sent nowhere.

For a long time we got used to asking how long a system takes to respond. Today I think it matters more to ask how long it takes under different conditions, how predictable that time is and for how long it can sustain it. Because an acceptable average latency does not necessarily mean we have a system able to respond in time when we really need it.

The complete data, every inference with its time, is at crtdrops.mx/datos, under CC BY 4.0.

Israel Negrete Lepe is an electronics engineer. Twenty-five years making different systems work together under real conditions: from the sensor, the vehicle or the transaction to the backend, the data, security and operations. Today, increasingly, edge AI.