Inference optimisation is now a platform economics problem

Introduction

A customer-support assistant goes into production with acceptable latency and a manageable infrastructure footprint.

Three months later, the model has not changed. The accelerator type has not changed either. The workload has.

Prompts now carry more retrieved material. Conversations run for more turns. Traffic arrives in sharper bursts. A larger share of requests misses the reusable prefix cache, while long inputs occupy memory and shorter requests wait for decoding capacity.

The platform team can respond by adding accelerators. It can also ask a more useful question: how much of the new cost is actual demand, and how much is the way inference is being served?

That is why AI inference optimization is becoming a platform economics problem. The cost of running a model in production increasingly depends on batching policy, memory management, cache reuse, numerical precision, request shape, scheduling and the runtime chosen to execute it.

Two applications can call the same model on the same class of hardware and produce very different infrastructure economics because the difference sits in the serving layer.

At a glance

• Model size alone does not determine inference economics because request length, concurrency, batching, cache behaviour and latency requirements influence how much useful work an accelerator completes.
• KV-cache pressure can restrict request concurrency even when raw compute remains available, making memory management an important part of capacity planning.
• Prefix and context caching can reduce repeated computation when applications reuse the same instructions, documents or conversational history.
• Higher throughput is not automatically the right optimisation target because scheduler policies can improve utilisation while worsening queueing, pauses or request latency.
• Inference optimisation belongs at platform level when many applications share the same serving estate, since local tuning decisions become capacity, reliability and cost policies for everyone.

The same model can have very different serving economics

Infrastructure planning often starts with the model: parameter count, required memory, supported accelerators and expected tokens per second. Production economics starts one layer higher because the same model behaves differently under different traffic and request shapes.

A service receiving short, independent requests at steady concurrency creates a different workload from one serving long prompts with variable output lengths. An application that repeatedly sends a large system prompt behaves differently from one in which nearly every prefix is unique, while bursty traffic creates different batching opportunities from a constant stream.

Modern inference runtimes expose these differences directly. vLLM’s current documentation includes continuous batching, chunked prefill, prefix caching, quantisation and speculative decoding among its serving capabilities. NVIDIA TensorRT-LLM supports in-flight batching, paged KV caching and lower-precision quantisation techniques including FP8, INT8 and INT4.

These mechanisms are often described as performance features, but at enterprise scale they also determine capacity. A platform that completes more requests from the same accelerator pool changes the infrastructure required for a workload. One that avoids redundant prompt computation changes the work imposed by each request, while more efficient memory use can admit additional concurrent sessions before another accelerator is required.

Inference performance and inference economics increasingly become the same engineering problem viewed through different measures.

Batching converts traffic shape into accelerator utilisation

An accelerator becomes economically useful when enough work reaches it without violating the application’s service expectations. Batching is one of the mechanisms that determines that balance because the serving system can combine work from multiple requests rather than processing each request as a completely isolated job.

Generative workloads make batching harder because requests do not finish together. Prompts have different lengths, outputs terminate at different times, and new requests continue arriving while existing requests are decoding.

Continuous or in-flight batching allows the runtime to alter the active batch as execution progresses instead of waiting for a fixed batch to complete before admitting new work. Under mixed traffic, that can keep expensive compute resources productive for more of the time.

This turns what looks like a latency setting to an application team into a utilisation decision for the shared platform. A larger batching window can create more opportunities to combine work, but waiting to assemble it can increase latency. A strict latency objective may therefore require more spare capacity than a service that permits slightly more queueing.

The economics depend on the SLO rather than on throughput alone.

KV cache turns memory management into a capacity decision

LLM inference does not consume accelerator memory only for model weights. During generation, the runtime stores attention key and value states for active sequences, so longer prompts and greater concurrency can make the KV cache a substantial part of the memory footprint.

An accelerator may consequently retain compute headroom while memory pressure prevents another request from being admitted efficiently. Longer context windows can change serving economics even when the underlying model and device are unchanged.

Paged KV-cache approaches are designed to use that memory more flexibly. vLLM’s serving architecture uses PagedAttention for KV-memory management, while TensorRT-LLM supports paged KV caching as part of its LLM inference stack.

For a platform team, the capacity question is therefore not simply whether a model fits on the device. It is how many active requests with realistic context lengths can fit while the required latency remains achievable.

A platform sized around benchmark prompts can become unexpectedly expensive when production applications start sending materially longer contexts.

Repeated context should not always be recomputed

Enterprise AI applications frequently repeat large parts of their input. A coding assistant may repeatedly send repository context. An internal knowledge application may include the same policy documents in thousands of requests, while an agent may carry a substantial system instruction block across every turn.

Reprocessing all of that content for every request consumes inference capacity even when much of the work is identical.

Prefix caching changes that equation by retaining reusable computed state for repeated prefixes. vLLM’s prefix-caching design describes reuse of previously computed KV-cache blocks when later requests share a prefix.

Managed platforms expose a similar economic mechanism. Google’s context-caching overview describes caching reused prompt context so repeated input can be processed at reduced cost.

The architectural consequence is that cacheability is influenced by application design. If teams construct system prompts differently, embed volatile information near the beginning of otherwise shared context or repeatedly transmit nearly identical documents in inconsistent forms, the serving platform has fewer opportunities to reuse computation.

Inference optimisation can therefore reach back into how applications structure requests, even when the optimisation itself is implemented in the platform.

Quantisation changes the capacity envelope, not just the model file

Reducing numerical precision is another way to alter serving economics. Current inference stacks support multiple lower-precision formats; TensorRT-LLM documents FP8, INT8 and INT4 techniques, while vLLM supports multiple quantised execution formats.

Lower precision can reduce memory requirements and improve execution efficiency on hardware that supports the chosen format. That can change how many replicas fit within a serving fleet or how much throughput a fixed amount of infrastructure can provide.

The choice cannot be treated as a universal platform default because models, quantisation methods and workloads do not respond identically. Quality-sensitive tasks still require evaluation after optimisation, particularly where model output feeds consequential downstream actions.

A platform team therefore needs reproducible and testable precision profiles rather than simply selecting the lowest precision available. That also means a model version is no longer the complete deployment description when the same model can be served under several precision, runtime and hardware profiles.

Prefill and decode create different resource behaviour

A generative request has at least two materially different phases. During prefill, the model processes the input context; during decode, it generates new tokens incrementally.

Long prompts can make prefill expensive, while long outputs extend decode. High concurrency creates pressure in both phases, but not in the same way.

Modern serving frameworks increasingly expose controls that reflect this distinction. vLLM supports chunked prefill and also documents experimental disaggregated prefill/decode, where prefill and decode can run in separate instances to tune time-to-first-token and inter-token latency independently and control tail latency. Its current documentation explicitly notes that the feature is experimental and is not intended to improve throughput.

This matters when the workload mix changes. An enterprise search application can evolve from short questions over small retrieved passages into long-document analysis without changing its model endpoint. A much larger share of infrastructure time may then be spent processing input rather than generating output.

Scaling the serving pool uniformly may work, but it hides the cause. Platform operators get a better economic picture when they can distinguish capacity consumed by prefill, decode, queueing, cache misses or underutilised hardware instead of treating all inference demand as one homogeneous quantity.

Maximum utilisation and predictable latency are different objectives

Serving runtimes make the trade-off between throughput and request behaviour unusually visible.

NVIDIA’s TensorRT-LLM scheduler configuration exposes a max_utilization policy that greedily packs requests into in-flight batching iterations to maximise throughput. NVIDIA also notes that this can create pause/resume overhead if KV-cache limits are reached. The alternative guaranteed_no_evict policy ensures a started request is not paused.

A scheduler setting can therefore encode an economic choice. Maximising utilisation may suit asynchronous document processing or batch summarisation, while an interactive assistant with strict tail-latency expectations may justify more conservative scheduling and greater capacity headroom.

Both workloads can be correctly optimised even when the second looks less efficient on an accelerator-utilisation dashboard.

The platform has to preserve the behaviour the workload requires rather than maximise one infrastructure metric in isolation.

Hardware choice only makes sense together with the runtime

Inference optimisation eventually reaches the accelerator itself. Cloud providers now offer hardware designed specifically for inference alongside general-purpose GPUs; AWS, for example, positions its EC2 Inf2 instances around Inferentia2 accelerators for deep-learning inference and supports multiple numerical formats through the Neuron software stack.

The economic question is not whether one accelerator has a lower headline price. Hardware only becomes cheaper in practice when the model, runtime and workload can use it efficiently.

A device that appears economical but requires substantial porting, lacks a required model feature or performs poorly under the application’s request shape may not lower operating cost. Conversely, a platform that standardises runtimes, compilation paths and validated model profiles can make alternative accelerators practical across a larger workload base.

Hardware selection is therefore inseparable from software support and workload behaviour.

Inference optimisation becomes a shared platform capability

When every AI application owns its own serving stack, each team can tune its model independently. That becomes harder to justify once dozens of applications share the same accelerator estate.

One team changes batch behaviour. Another disables caching. A third pins a higher-precision model variant. A fourth introduces long contexts that consume disproportionate KV-cache capacity. Each choice can be defensible locally while competing for the same physical resources.

Shared inference estates therefore need platform primitives rather than a collection of one-off tuning decisions. The platform can maintain validated serving profiles by model, precision and hardware; expose cache and batching capabilities consistently; define workload classes around latency and throughput requirements; and make runtime metrics comparable across applications.

Useful signals include prompt and output token volumes, time to first token, inter-token latency, queue time, batch occupancy, KV-cache utilisation, cache-hit rate, accelerator utilisation and request concurrency. None alone defines efficiency, but together they show where capacity is being consumed and which optimisation lever is likely to matter.

A cheaper model call can require a more sophisticated platform

Inference optimisation contains an apparent contradiction: many techniques that reduce runtime cost increase the sophistication of the serving platform.

Caching needs lifecycle and memory management. Quantisation needs validated deployment profiles. Continuous batching needs scheduling. Alternative accelerators need compilation and runtime support. Prefill/decode optimisation needs better workload visibility, while autoscaling becomes harder when the same number of requests can represent very different amounts of token work.

The result is that inference economics cannot be delegated entirely to application teams or reduced to choosing a cheaper endpoint. Part of the saving comes from turning serving behaviour into an engineered platform capability.

That investment becomes more valuable as the number of models and applications grows because the same optimisation machinery can be reused across a larger inference estate.

The platform should optimise the workload, not an isolated benchmark

Inference benchmarks remain useful because they establish what a model/runtime/hardware combination can achieve under controlled conditions. Production platforms face a different problem: they must serve the actual distribution of prompts, output lengths, concurrency, cache reuse and latency expectations created by the business.

A configuration that wins on maximum tokens per second can be wrong for an interactive application. One that produces excellent single-request latency can leave expensive accelerators chronically underused. An optimisation that performs well for short prompts may lose its advantage when context sizes expand.

Platform teams therefore need workload profiles alongside model profiles. The relevant question is not simply, “How fast can this model run?” It is how efficiently this platform can serve this workload while meeting the service behaviour the application requires.

Inference efficiency is becoming part of platform design

Enterprise inference cost will continue to attract attention because accelerator capacity is expensive and AI usage tends to expand once applications reach production. Buying more capacity remains a valid response to genuine growth, but it should not automatically be the first one.

A rising infrastructure bill can also reflect poor batch formation, repeated context computation, inefficient KV-cache use, unnecessary numerical precision, a runtime poorly matched to the hardware or an SLO that forces excessive idle headroom. Those are serving decisions rather than immutable properties of the model.

As enterprise AI estates mature, inference optimisation becomes less about squeezing a few more tokens per second from one model and more about deciding how shared compute should behave across many workloads. The economic question is embedded in the serving architecture.

The platform is not only deciding where inference runs. It is deciding how much useful inference each unit of infrastructure can deliver.