Model routing turns latency, cost and reliability into one operating decision

Introduction

The cheapest eligible model is available.

It is also running close to the application’s latency limit.

Another model costs more but is responding faster. A third has spare capacity, although its response time is less predictable. One provider has begun returning intermittent errors.

The application still has to answer the next request.

Which model should receive it?

A static deployment answers that question once.

A router can answer it again for every request.

That changes the operating problem.

Multi-model routing for cost, latency and reliability is not simply a way to send easy work to a cheaper model. In production, the route can also depend on how quickly candidate deployments are responding, whether they remain healthy, what retry or failover risk they carry and whether the workload can tolerate the resulting trade.

Model choice becomes a live infrastructure decision.

At a glance

• A production model router operates at the point where several objectives meet.
• Cost matters because candidate routes can have materially different inference economics.
• Latency matters because the least expensive candidate may not meet the workload’s response-time requirement under current traffic.
• Reliability matters because provider errors, rate limits, capacity constraints and transient instability can change which routes are usable.
• Those dimensions interact. A timeout can turn one nominally cheap request into multiple attempts. A faster route can cost more. A highly available architecture may require maintaining additional providers or deployments.

The useful question is:

Which acceptable route is the best operating choice for this request under current conditions?

Routing moves an infrastructure decision into the request path

Without routing, much of model selection happens during deployment.

An application points to a model. Teams know its price structure, benchmark its latency and monitor availability. As long as those characteristics remain acceptable, traffic continues to use that deployment.

Multi-model routing moves part of that decision into runtime.

Microsoft Foundry’s Model Router evaluates individual requests against an eligible model pool. Microsoft’s current guidance treats quality, cost, median and tail latency, permitted models and deployment configuration as parts of the same workload decision.

The application no longer inherits one fixed operational profile.

It inherits a routing policy whose actual behaviour depends on the route selected for each request and the state of that route when the request arrives.

Cost becomes one property of a route

When every request uses one model, inference cost is relatively straightforward to attribute.

Routing makes it variable.

A small request may use one model. Another request may need a more capable one. A stressed provider may be avoided. A timed-out request may consume work on one route before another route completes it.

For infrastructure operations, cost becomes one property of the execution path alongside latency and availability.

The listed token price of the first model selected is no longer always the best description of what the completed request cost.

That distinction becomes especially important once retries and failover enter the path.

Tail latency can reverse the apparent preference

Average latency can make two routes look more similar than users experience them.

Suppose Model A usually responds in 1.2 seconds but develops a long tail under heavy concurrency.

Model B usually responds in 1.6 seconds but stays tightly distributed.

For a workload with a two-second response requirement, the average favours Model A. The p95 may favour Model B.

Microsoft’s current Model Router evaluation guidance explicitly asks teams to evaluate median and tail latency, including p90 or p95, because favourable averages can hide slow requests that matter to the user experience.

Routing happens one request at a time.

The user experiences the route selected now, not the historical mean.

Latency-aware routing consequently needs production-shaped evidence about distributions, concurrency and recent route behaviour.

Capacity can change the right route without changing the task

Model capability is only one part of the decision.

Capacity moves too.

A deployment that meets its latency target under ordinary traffic may become a poor destination during a burst. Another provider may have more available capacity. A self-hosted route may have fewer accelerators free. Rate limits can effectively close an otherwise attractive path.

At this level, model routing begins to resemble traffic engineering.

Kong AI Gateway currently supports strategies including least connections, lowest latency, lowest usage, semantic routing and priority-based failover. Its gateway can also retry and move to another target after upstream errors or timeouts.

The semantic suitability of the model has not necessarily changed.

The infrastructure condition has.

Semantic routing and traffic balancing are different decisions

The phrase model routing can describe two related but distinct choices.

One asks which model is appropriate for the request.

That can depend on task complexity, modality, context requirements or expected quality.

The other asks which provider, deployment or target should serve it now.

That can depend on latency, capacity, connection load, errors or health.

Kong exposes that distinction directly: semantic routing sits alongside lowest-latency, least-connections, lowest-usage and priority routing strategies.

Cloudflare’s Auto Router approaches the problem differently. It builds an eligible pool, removes candidates that cannot serve the request or are unhealthy, estimates request difficulty, ranks the remaining options using expected quality and cost, and falls back if the preferred provider cannot serve the request.

The production point is simple.

A model can be suitable for the work and still be the wrong traffic destination under the current operating conditions.

Reliability changes the candidate set

Reliability can become an input to routing rather than merely a metric observed afterward.

A provider begins returning errors. A deployment hits a quota. A target stops responding before the timeout. Another approved route remains healthy.

The user’s request has not changed.

The set of usable routes has.

Microsoft Foundry now includes built-in automatic failover in Model Router. When an underlying model encounters a transient issue, requests can be redirected to another model, while configured model subsets also act as the fallback set.

Cloudflare similarly removes unhealthy providers from Auto Router’s eligible pool and can fall back when the preferred provider cannot serve a request.

At the infrastructure layer, the question is whether an alternate route remains available and inside the workload’s operating envelope.

Whether substitution preserves the behaviour previously accepted for the application is a separate release-control problem.

Timeouts couple latency, reliability and cost

Timeout policy makes the interaction among the three dimensions particularly visible.

Imagine a low-cost provider begins slowing down.

The application waits four seconds, reaches the timeout and sends another attempt to a second provider.

The request eventually succeeds.

Availability has been preserved.

The user experienced the delay from the first attempt plus the successful route.

The system may also have incurred work on more than one path.

A nominally inexpensive route can therefore raise the cost of a completed request when slow attempts regularly precede retries or failover.

Cloudflare AI Gateway supports request timeouts and retries for slow or unresponsive providers, while Dynamic Routing allows per-model timeout and retry settings as well as rate and budget controls.

The timeout value is not merely a reliability setting.

It influences latency and the amount of duplicated work the system is willing to tolerate before moving elsewhere.

Reliability often requires redundant routes

Higher resilience usually means maintaining alternatives.

An enterprise may keep deployments in several regions, integrate more than one provider or preserve spare capacity that is rarely used under normal conditions.

Those choices change the operating envelope.

Cross-region traffic can add network delay. Backup providers may have different prices. Reserved capacity can introduce fixed expense.

The routing policy consequently carries an implicit position on failure:

How long should a slow route remain preferred?

When should another provider become preferable?

How much additional cost or latency is acceptable to reduce dependence on one route?

These are infrastructure decisions made around the request path.

Hard constraints should narrow the pool before optimisation

A router should not optimise over every model that happens to be technically reachable.

Some candidates may not support the required context length or modality.

Some may be unavailable in the necessary region.

Some may be prohibited by organisational policy.

Others may never have been approved for the workload.

Microsoft’s Model Router guidance treats permitted models, regions and deployment configurations as explicit constraints and allows teams to restrict the model subset used for routing.

Those constraints should narrow the candidate pool before optimisation begins.

The router is choosing among acceptable routes.

Speed or price cannot turn an unacceptable candidate into an acceptable one.

Routing is usually a constrained decision, not one universal score

It is tempting to represent routing as a single weighted formula:

quality + cost + latency + reliability = best model

Production requirements are often more conditional.

A response slower than three seconds may be unacceptable regardless of price.

A route with elevated errors may need to leave the candidate set.

A particular workload may require a specific class of model.

Another workload may tolerate more latency variation to reduce infrastructure cost.

The routing logic is often closer to a sequence:

satisfy the workload’s hard constraints, remove routes that are currently unusable, then optimise among what remains according to that workload’s priorities.

Different workloads can use different envelopes while sharing the same routing infrastructure.

Workload class gives the trade-off meaning

A customer-facing conversational request and an asynchronous document-analysis job do not value latency equally.

For an interactive workload, time to first token may dominate the experience.

For an asynchronous job, a slower but stable route may be perfectly acceptable.

A bulk process may tolerate retries that would be disruptive in a user-facing workflow.

The router therefore needs enough workload context to apply the correct operating requirements.

That does not require a complicated taxonomy for every prompt.

A small number of service classes may be sufficient: interactive, near-real-time, asynchronous, bulk or high-consequence.

Cost, latency and reliability only become meaningful relative to what the workload actually needs.

A routed system needs telemetry for the completed path

Dynamic routing creates another production decision that has to remain observable.

For an individual request, teams may need to know which model and provider ultimately served it, whether another route was attempted first, how much routing overhead was introduced, whether a timeout or error triggered another attempt and how long the entire request took.

Microsoft’s Model Router observability exposes request-level information about the serving model, routing attempts, status, reported latency and fallback behaviour. Its routing trace can show ordered attempts when multiple models were tried.

Cloudflare Auto Router likewise returns routing metadata identifying the model that served the request and why the routing decision was made.

This evidence should reconstruct the infrastructure path of the completed request.

Without it, the application can stay nominally healthy while latency or reliability changes for reasons hidden inside the routing layer.

Tail metrics belong to the routed system

A router also needs latency measures above the individual model.

Teams may need to separate routing overhead from model latency and distinguish ordinary first-attempt requests from those that waited on a slow route before retrying elsewhere.

The p95 or p99 of the complete path can look very different from the latency distribution of the model that ultimately returned the answer.

Provider uptime alone cannot describe that experience either.

The system-level SLO belongs to the routed request.

Distribution can expose infrastructure changes

The share of requests handled by each route can also be diagnostically useful.

A sudden shift may indicate congestion, provider health problems, regional capacity changes, candidate-set removal or a routing-policy change.

Microsoft includes selected-model distribution among the signals teams should monitor when validating a router in production-like conditions.

Here, distribution is an infrastructure signal: it helps explain why the system’s latency or reliability profile changed.

Production-like concurrency matters before rollout

An offline prompt benchmark cannot reproduce every operating condition.

A router may perform well with sequential requests and behave differently when traffic introduces queueing, throttling, rate limits or contention.

Those conditions matter most in the tail.

Microsoft’s current guidance specifically advises validating routing under production-like traffic and concurrency when tail latency is important, while observing errors, failover behaviour and selected-model distribution.

The infrastructure test should therefore include the conditions most likely to change the route itself:

• expected concurrency;
• provider throttling;
• queueing;
• regional capacity constraints;
• timeouts;
• retry behaviour;
• and failover.

The prompt set remains relevant, but traffic shape becomes part of the evaluation.

A route should have an operating envelope

A useful production route can be described by the constraints and evidence surrounding it.

Routing property	Operating question
Eligible model set	Which models are allowed to serve this workload?
Latency objective	What median and tail latency can the workload tolerate?
Cost envelope	What completed-request cost is acceptable?
Health criteria	When should a model or provider leave the candidate set?
Timeout policy	How long should the current attempt be allowed to run?
Retry / failover policy	When should another route be attempted?
Route telemetry	Can the selected model, provider and attempts be reconstructed?
Re-evaluation trigger	What infrastructure or workload change requires the route to be reassessed?

This does not need to become a universal routing framework.

It establishes what the router may optimise and what it is not allowed to sacrifice.

Routing policy should change more slowly than routing decisions

A router may make decisions continuously.

The policy behind those decisions should move more deliberately.

New deployments become available.

Capacity changes.

Latency distributions shift.

Regions open.

Provider behaviour changes.

The workload itself evolves.

Any of those can justify revisiting the routing policy.

They do not justify uncontrolled changes inside the live request path.

A production team should be able to distinguish:

the route chosen for this request

from

the policy version that made that choice.

That separation makes a dynamic system much easier to investigate and reproduce.

Dynamic routing is not always the right architecture

Some workloads benefit from deterministic model selection.

A regulated process may require one validated model.

A long conversation may depend on stable behaviour across turns.

A task with strict capability requirements may have only one genuinely acceptable model.

Routing also adds its own operational complexity.

Microsoft’s current evaluation guidance explicitly says teams should retain direct model deployments for requests that require deterministic selection or when workload evaluation does not support routing those requests.

Dynamic choice should exist because the workload benefits from it, not simply because several models are available.

The operating unit is the completed request

This is the deeper production change.

Teams can no longer understand latency, reliability or route cost only at the model level.

The user receives a completed request.

That request may have gone through routing, waited on one provider, timed out, triggered another attempt and finally succeeded somewhere else.

The useful production record follows the entire path.

Which route completed the request?

How long did the complete path take?

Did an earlier route fail or time out?

What work was duplicated?

Was the final path still inside the workload’s permitted operating envelope?

Those questions turn model routing into an infrastructure discipline rather than simply a model-selection feature.

FAQs

What is multi-model routing?

Multi-model routing selects among several eligible AI models or deployments at runtime. Selection can depend on task characteristics, latency requirements, provider health, capacity, cost or other policy inputs rather than permanently binding every request to one model.

How can model routing reduce latency?

A router can move traffic toward routes that better satisfy the workload’s latency requirement or away from congested and unhealthy targets. The result depends on current capacity, routing logic and whether the system is measured using production-like traffic and tail latency rather than averages alone.

Does the cheapest model always reduce infrastructure cost?

No. Timeouts, retries and failover can create work on several routes before one request completes. The listed price of the initially selected model may therefore differ from the actual cost of completing the request.

How does reliability affect model routing?

Provider errors, timeouts, rate limits and capacity constraints can change which routes remain usable. A resilient routing architecture maintains approved alternatives and records when another route had to be used.

What should enterprises monitor for a model router?

Useful signals include the selected model and provider, median and tail latency, routing overhead, timeout and retry rates, failover events, errors, route distribution and the routing-policy version responsible for the choice.

One request creates one operating trade

Model routing is often introduced as freedom from choosing one model for everything.

Its production consequence is more demanding.

The router makes model choice part of the live infrastructure path.

Every request arrives with a latency requirement, a cost envelope and a dependence on infrastructure that may be healthy, congested or unavailable.

A useful router has to operate inside all three realities at once.

The question is larger than:

Which model should answer this request?

For production infrastructure, it becomes:

Which acceptable route can complete this request within the latency, cost and reliability envelope the workload actually requires?

That is the operating decision model routing now has to make.