The Wrong Latency Question

When people talk about latency in AI systems, they usually mean network latency.

How many hops?
Local or remote?
HTTP or stdio?
Milliseconds or seconds?

That’s a reasonable place to start - but it’s the least interesting part of the problem.

Recently, a series of …