For most of the last decade, the default answer to “where should this run?” was simple: the cloud. Push the request to a region, let a large machine handle it, send the bytes back. That answer is starting to fray — not because the cloud got worse, but because the economics of small models got dramatically better.
The latency budget is where it starts
A round trip from a device in Southeast Asia to a data centre in Northern Virginia costs somewhere between 180 and 260 milliseconds before a single token is generated. Add model queueing and you are comfortably past the threshold where an interaction stops feeling instant. Users do not read latency numbers; they just decide the feature is sluggish and stop using it.
Edge inference collapses that budget. When a 3B-parameter model runs on the device or in a nearby point of presence, the network cost approaches zero and the entire experience becomes bounded by compute you actually control. The trade is capability: a small model will not match a frontier model on hard reasoning. The practical insight of the last two years is that a surprising share of production workloads never needed frontier reasoning in the first place.
Classify locally, escalate remotely
The architecture that keeps emerging in production systems is a two-tier router. A small local model handles the high-volume, low-ambiguity path — intent classification, entity extraction, redaction, formatting, retrieval query rewriting. Anything it flags as uncertain gets escalated to a larger hosted model.
- Tier 1 (local): deterministic transforms, routing, PII scrubbing, cache lookups. Target p99 under 40 ms.
- Tier 2 (regional): summarisation, multi-document synthesis, anything with a long context window.
- Tier 3 (frontier): genuine reasoning, code generation, tasks where an error is expensive.
Teams that have measured this report escalation rates between 8% and 20% depending on domain. That is an order-of-magnitude reduction in tokens billed at the top tier, and it arrives without a visible quality regression — because the escalation gate, not the small model, is what determines quality.
The gate is the hard part
Everyone underestimates this. Building a router that knows when it does not know is a harder engineering problem than either model it sits between. Naive confidence thresholds on output logits are poorly calibrated and drift as inputs shift. The approaches that hold up in production tend to be boring:
- Escalate on structural failure — malformed output, schema violation, empty extraction — rather than on a soft confidence score.
- Sample a fixed percentage of local-tier traffic to the frontier model regardless of confidence, and use the disagreement rate as a live quality signal.
- Keep a per-tenant override. There is always one customer whose data breaks your assumptions.
What this means for how you build
The operational implication is that model selection stops being an architectural decision made once and becomes a runtime decision made per request. That pushes real requirements into your infrastructure: versioned model artefacts shipped to the edge, a way to roll back a bad small-model deployment in minutes, and observability that tells you the escalation rate by route rather than just aggregate latency.
The teams that win here are not the ones with the best model. They are the ones who can measure, per endpoint, what fraction of their traffic actually needed the expensive path.
None of this is exotic. It is caching and tiering — the same pattern that reshaped web serving in the 2000s and CDNs in the 2010s, applied to a new kind of compute. The surprising part is how long the industry spent treating inference as an indivisible remote call before noticing.