The Useful GenAI Application Is Increasingly a System of Models, Tools and Evaluators

Introduction

The first architecture diagram for a GenAI application may contain little more than a user interface, a prompt and one model endpoint.

Production systems increasingly look different. Retrieval supplies current information. Tools connect the application to enterprise capabilities. Specialist models handle bounded tasks. Deterministic code validates known rules. Evaluators inspect whether the system completed the task or used its components correctly.

The model still matters, but it no longer describes the application by itself. That is the shift behind compound AI systems in enterprise applications: product quality increasingly depends on how several components divide the work, exchange information and behave together.

The unit of engineering is moving from the model toward the system around it.

At a glance

• Enterprise GenAI applications increasingly combine models with retrieval, tools, deterministic code, state and evaluation rather than relying on a single model call.
• Better application behavior can come from changing the surrounding system without replacing or retraining the primary model.
• Multiple models can occupy fixed specialist roles inside one application without turning it into a multi-agent system.
• Evaluators are becoming engineering components for detecting regressions in task completion, tool use and system behavior.
• Compound architecture creates more control surfaces but also more interfaces, latency, cost and failure combinations to manage.

The model box is no longer the application architecture

The phrase compound AI system was already useful before today’s agent platforms took shape.

In 2024, Berkeley researchers described compound AI systems as applications that solve a task through several interacting components, including model calls, retrievers and external tools. Their argument was that useful performance was increasingly emerging from the complete system rather than from one monolithic model. 

That idea has moved from research framing into mainstream development infrastructure. Models still sit at the center of the stack, but they increasingly share that stack with tools, retrieval, orchestration, state, observability and evaluation.

A production GenAI application can therefore improve even when its underlying model does not change. Its behavior may depend just as heavily on which information reaches the model, which external capability it can call and what the surrounding software does with the result.

Current platforms are converging on system-level building blocks

The strongest evidence that compound architecture is becoming mainstream is the development surface itself.

Microsoft’s current Foundry capability map, updated August 27, 2026, presents models, agents, tools, observability, evaluations, red teaming and control-plane capabilities as related parts of the same platform. 

Its 2026 development direction makes the same point operationally. Hosted agents, managed toolboxes, monitoring and evaluation increasingly sit beside model access rather than appearing as unrelated add-ons.

Google’s Agent Development Kit takes a similar compositional view. Its current documentation treats tools, workflow orchestration, specialist and multi-agent composition, evaluation and deployment as separable parts of the application architecture. 

Google also deprecated Vertex AI Extensions on May 26, 2026 and directed users toward Agent Platform, another sign that the platform boundary is expanding from individual model extensions toward a broader application runtime. 

OpenAI’s March 2026 engineering note describes the same transition from another angle. It focuses on systems in which a runtime receives model output, invokes tools and feeds those results back into later model steps. The current Agents SDK separately exposes models, providers, tools, runtime state, handoffs, guardrails, observability and evaluations. 

The platforms differ in product design and terminology, but their development surfaces are converging around the same assumption: useful AI software increasingly consists of several cooperating components.

Better behavior does not always require a better model

Consider an internal support application that repeatedly answers product-entitlement questions incorrectly.

Weak reasoning is one possible cause, and a stronger model may improve it. The failure could just as easily originate elsewhere: retrieval may surface the wrong product documentation, the application may omit the customer’s contract tier, a tool may fail to fetch current entitlement data, or the system may accept an answer despite incomplete evidence.

Replacing the model addresses only the first possibility. A compound architecture gives teams several independent improvement surfaces: retrieval can change without retraining the model, a deterministic calculation can replace an unreliable generative approximation, a specialist component can handle a bounded task, and a validation step can block output that violates known constraints.

The engineering question becomes which part of the system is limiting the outcome. Model capability remains important, but application quality is also shaped by the architecture around it.

Models can specialize inside one application

One model does not have to perform every stage of the work. A small model might classify a request, another may handle a reasoning-heavy step, and a specialist model could extract structured fields. A separate model can be used for evaluation during testing or analysis.

Those roles can be fixed by design, which is different from choosing among several models at runtime according to changing cost or latency conditions. In a compound application, model specialization can simply reflect that different stages demand different capabilities.

A classifier does not need to be the best long-form reasoner, and an extraction model does not need to carry the full capabilities of the model producing a final explanation.

Google’s current ADK reflects this decomposition through support for specialist components, tools, workflow orchestration and evaluation within one application environment. 

The architectural benefit is that teams can place a model where its capability is useful instead of treating one general-purpose model as the only computational unit available.

Tools externalize capabilities that should not live inside the model

Tool use makes compound architecture visible because it moves part of the application outside generative inference.

A model may decide that it needs a customer record, current inventory or a pricing calculation, while the actual information or operation comes from software that already owns that capability. This separation can improve control as well as usefulness.

A pricing calculation may belong in deterministic code. Customer state may belong in the CRM. Inventory belongs in the inventory service. A policy decision may depend on an existing business-rule service rather than on the model recreating the rule probabilistically from text.

OpenAI’s current tool stack supports web search, file search, function calling, programmatic tool calling and remote MCP servers as attachable runtime capabilities.

Microsoft Foundry similarly supports managed toolboxes containing services such as web and file search, code execution, MCP servers and custom functions under centralized authentication and versioning. 

The application architecture becomes partly a decision about which capability belongs in generative reasoning and which should remain in a system designed specifically to perform it.

Evaluators are entering the engineering loop

Evaluation becomes more complicated when the application performs more than one model call.

A final answer can look plausible even when the system took the wrong path. It may have selected the wrong tool, generated poor arguments, ignored returned data or completed the task through an expensive sequence that should not become the production norm. Current evaluation stacks increasingly inspect those intermediate behaviors.

Microsoft’s agent evaluators include measures for task completion, tool selection, tool-call accuracy, tool-input accuracy, tool-output utilization and tool-call success. 

OpenAI’s May 2026 agent-improvement example treats the surrounding harness—instructions, tools, routing, output requirements and validation checks—as something teams can improve using traces, human feedback and reusable evaluations. 

Its macro-evaluation work extends that approach across populations of multi-step traces so recurring system behavior can be measured rather than judging one isolated answer at a time. 

Evaluators do not have to sit in every live request. Their significance is that teams increasingly need a repeatable way to judge the behavior of the complete application, not merely the prose emitted at the end.

Component interfaces become quality boundaries

Once an application contains several components, some failures appear at the handoff between them.

Retrieval may return useful material in a format the model handles poorly. A model may produce structurally valid tool arguments while omitting one business field. A tool can succeed technically while returning insufficient information for the decision. Each component can look healthy in isolation even though the complete workflow produces the wrong result.

Interface design therefore becomes part of AI quality. Machine-to-machine boundaries benefit from explicit schemas, known error states and checks that distinguish missing information from legitimately empty results.

Teams also need clarity about what each component is expected to receive and what downstream components can safely assume from its output. Component quality and composition quality are different properties, and a compound system has to succeed at both.

Each added component expands the operating surface

Composition gives engineering teams more ways to improve behavior, but every component also creates another dependency to operate.

Retrieval introduces indexing, ranking and source-quality concerns. Tools bring authentication, schema changes and service availability. Additional models bring their own capability distributions and release cycles. Evaluation infrastructure adds another service with its own data and execution requirements.

Berkeley’s original compound-system analysis identified this trade-off early: system designers gain more control but inherit a larger design space, with latency, cost and operational resources distributed across several components. 

For enterprise teams, that puts a useful constraint on architectural enthusiasm. Retrieval, memory, tools, specialist models and evaluators should enter the system because they materially improve the application, with each component earning the interface and operating burden it adds.

Latency and cost belong to the complete execution path

A model can meet its individual response-time target while the application remains slow. A compound system may retrieve data, call a model, invoke a tool, make another model call and perform validation before the result reaches the user.

Performance therefore accumulates across the path, and cost behaves the same way. A cheap model invocation says little about a workflow that also consumes retrieval infrastructure, external APIs, additional models and evaluation services.

At the architectural level, the implication is one of budget allocation. An extra model call or evaluation step should improve the outcome enough to justify the delay and resource use it introduces.

A tool call that replaces an unreliable hallucinated calculation may easily justify its cost; several overlapping verification stages may not. Compound architecture gives teams more levers while forcing them to account for the complete execution path rather than optimize one component in isolation.

Compound does not mean multi-agent

Compound AI and multi-agent AI are related but distinct.

An application can contain retrieval, several model calls, tools, validators and evaluators while following a deterministic software-defined sequence. No component needs autonomous authority to delegate work or choose its own collaborators.

Multi-agent architecture adds another layer in which several agentic components coordinate or divide work through routing, handoffs, shared state or parallel execution. Enterprises can therefore gain many of the advantages of composition without introducing multiple autonomous decision-makers.

A retrieval step can remain deterministic, a tool may run only after an explicit condition, an evaluator can operate offline and a specialist model can occupy a fixed position in the pipeline.

Compound architecture describes the broader application pattern. Multi-agent systems are one possible expression of it.

The simplest sufficient architecture remains the better target

The emergence of compound AI should not turn every GenAI application into an elaborate system.

Some workloads need little more than one model call, sound instructions and structured output. Adding retrieval where the model already has the required information creates another dependency, while a verifier that never changes a decision adds latency without meaningful control.

Additional components make sense when they solve a real limitation. For consequential or information-rich workflows, composition can be the cleaner design because deterministic software, enterprise tools and specialist components take responsibility for work that should not depend entirely on one general-purpose model.

The architectural target is the smallest system that reliably satisfies the application requirement, even when that system contains several distinct components.

A practical compound-system design view

Architecture reviews become clearer when every component has a defined reason to exist.

Component	Question it should answer
Primary model	Which parts of the task genuinely require generative reasoning or interpretation?
Additional model	Does a fixed specialist role perform one stage more reliably or efficiently?
Retrieval	Which information must come from current or enterprise-controlled sources?
Tool	Which capability or business state belongs outside the model?
Deterministic code	Which calculation, rule or validation should behave predictably?
Evaluator	Which important behavior needs repeatable measurement or regression detection?
Orchestration	In what order should components run, and how should incomplete execution be handled?

A component earns its place when it improves the application enough to justify the interface, latency and operational responsibility it introduces.

FAQs

What is a compound AI system?

A compound AI system solves an AI task through multiple interacting components rather than relying only on one model invocation. Those components can include models, retrieval systems, tools, deterministic software, orchestration logic and evaluators.

Is RAG a compound AI system?

Yes. A basic RAG application already combines retrieval with a generative model and application logic. More advanced designs may add reranking, query transformation, verification or additional model calls.

Does a compound AI system require multiple models?

No. One primary model combined with retrieval, tools, deterministic checks and evaluation is already a compound architecture. Multiple models are an optional design choice.

Is a compound AI system the same as a multi-agent system?

No. Multi-agent systems are one type of compound architecture. A compound application can use deterministic orchestration and several non-agent components without giving multiple agents independent decision-making roles.

Why are evaluators becoming part of compound AI architecture?

Evaluators help teams measure behaviors that final-answer review can miss, including task completion, tool selection and regressions caused by system changes. They can be used during development, release testing or production analysis.

Engineer the application as one system

The useful GenAI application is becoming harder to describe by naming its model.

Retrieval quality, tool contracts, deterministic logic, specialist models, orchestration and evaluation can all influence whether the application succeeds. The 2026 platform shift reflects that reality: these capabilities are increasingly becoming first-class parts of the development environment rather than custom additions around a model endpoint.

Compound architecture gives enterprises more places to make behavior explicit and improve it deliberately, provided those components are engineered as one application.

The design discipline is to place each part of the task in the component suited to perform it, then judge the result at the level that ultimately matters: the behavior of the complete system.