The maintenance burden no model registry can see

A model registry can tell an enterprise which model version is in production and still leave much of the AI application’s maintenance surface invisible. Prompts, tool interfaces, retrieval configurations, evaluations, routing rules and external dependencies can all change the behaviour of the system without changing the registered model at all.

The model has not changed in four months.

That is the first reassuring fact in the maintenance review.

The production alias still resolves to the same registered version.

Its metadata is intact.

The deployment is healthy.

No rollback has occurred.

Yet the application has changed repeatedly.

The customer-service prompt has been revised six times.

Two tools have gained new fields.

One API endpoint has been deprecated.

The retrieval index now uses a different embedding model.

A policy filter was added.

A fallback route was introduced.

Three evaluation sets were replaced after the business changed its refund rules.

None of those changes created a new model version.

By the time the team tries to answer a simple question—

What exactly do we have to maintain to keep this AI application working?

—the model registry is describing only one part of the system.

An AI tool prompt dependency inventory starts where the model registry stops

Model registries solve a real and important problem.

MLflow’s Model Registry, for example, is designed to manage registered models, versions, aliases, tags and model-level lineage through development and deployment. A registered model can have multiple versions, while aliases such as champion can point production workloads to a selected version.

That is valuable infrastructure.

But an agentic or retrieval-based application usually has behaviour-shaping artifacts that are not the model.

A production application might depend on:

Model

• deployment alias;
• provider;
• inference parameters.

Prompts

• system prompt;
• task prompt;
• escalation prompt;
• tool-selection instructions.

Retrieval

• index;
• embedding model;
• chunking configuration;
• metadata filters;
• reranker;
• source collections.

Tools

• tool schemas;
• API endpoints;
• authentication configuration;
• business-service versions.

Application controls

• routing rules;
• fallback logic;
• safety policies;
• approval thresholds.

Evaluation

• test sets;
• expected outcomes;
• scoring logic;
• acceptance thresholds.

The operational application is the combination.

The model registry may describe one node inside it.

The registry can remain unchanged while the application drifts

Suppose a support agent uses model version 12.

Version 12 remains in production for six months.

During those six months:

• prompt v21 becomes v27;
• the customer-policy index is rebuilt twice;
• the order-status API moves from v2 to v3;
• a retrieval filter begins excluding archived documents;
• the refund tool gains an eligibility field;
• the fallback route moves to another provider.

The model registry can truthfully report:

Production model: version 12.

That statement is not wrong.

It is simply insufficient as a description of the application.

This is where maintenance becomes deceptive.

Teams can look at a stable model lifecycle and infer a stable AI system.

But the application may have accumulated dozens of independently changing dependencies around that model.

Prompts are now maintainable production artifacts

Prompt changes are often treated as lighter-weight than model changes.

Operationally, that can be misleading.

A prompt can alter:

• tool selection;
• decision boundaries;
• escalation behaviour;
• response format;
• retrieval strategy;
• interpretation of policy;
• the conditions under which an agent acts.

MLflow’s current Prompt Registry exists precisely because prompts have become separately versioned application artifacts. Its documentation supports immutable prompt versions, version history, comparison, aliases and environment-specific lifecycle management.

That creates an important maintenance implication.

The organisation now has at least two independent version graphs:

model versions

and

prompt versions

Neither can fully describe the other.

Prompt v8 may have been validated against model version 4.

Prompt v13 may have been introduced while model version 4 remained unchanged.

Model version 5 may later replace version 4 while the prompt stays at v13.

The maintained object is therefore not:

model version 5

It is closer to:

model 5 + prompt 13 + retrieval configuration 6 + tool set 9 + evaluation pack 4.

Tool definitions create another maintenance surface

Tool use adds a dependency that looks ordinary from an API-management perspective but can be unusually consequential for AI behaviour.

Consider a tool called:

create_credit_note

Version 1 accepts:

invoice_id

reason

amount

Version 2 adds:

customer_eligibility

The endpoint still works.

The tool name has not changed.

Existing callers may remain technically compatible.

But the added field can change how the model interprets its responsibility.

A maintenance inventory therefore needs to distinguish:

tool exists

from:

this application was validated against this version of the tool contract.

That distinction becomes more important as agents gain access to larger catalogs of tools.

MLflow’s current experimental MCP Registry supports semantic versioning, lifecycle states and aliases for MCP server configurations. MLflow notes that the feature is experimental and its APIs or behaviour may change.

The maintenance question is not merely whether the tool endpoint is alive.

It is whether the interface the agent expects is still the interface being served.

Retrieval systems accumulate hidden dependencies quickly

Retrieval-based applications multiply the problem.

A team may think it operates one RAG application.

In practice, the behaviour of that application can depend on:

• an embedding model;
• an index implementation;
• chunk size;
• chunk overlap;
• document parser;
• metadata schema;
• filter rules;
• source permissions;
• reranker;
• top-k;
• query-rewrite prompt;
• document versions.

None of those necessarily appears in a model registry.

Some may not even sit in the same platform.

That creates an uncomfortable maintenance scenario.

A team can upgrade the embedding model without changing the generation model.

The retrieval results change.

The application behaves differently.

The model registry remains untouched.

The same can happen if a source system changes metadata conventions or a document-processing pipeline starts producing different chunk boundaries.

The model is stable.

The context it receives is not.

Maintenance inventory and regression tracing solve different problems

Regression tracing asks which versions participated in one failed execution.

Maintenance inventory asks which independently changing artifacts could alter behaviour or become unmaintainable over the application’s lifetime.

The first is incident-first.

The second is pre-incident.

Its purpose is to expose the maintenance surface of the application while there is still time to manage it.

A dependency is not just something the application calls

Traditional dependency inventories tend to focus on packages, services and infrastructure.

AI applications need a broader definition.

A dependency can be anything whose change can materially alter behaviour, validity or maintainability.

That includes a prompt, tool description, retrieved policy collection, evaluation dataset, model alias, fallback route, threshold, policy rule, parser or embedding configuration.

The useful test is:

If this artifact changed tomorrow, could the application still run while behaving differently?

If the answer is yes, it belongs in the maintenance surface.

That question catches many dependencies that ordinary software inventories miss.

Evaluation assets are dependencies too

Evaluation is often treated as something performed around the application.

For long-lived AI systems, it is part of the maintenance system.

Suppose an agent is validated against an evaluation set containing:

• return-policy scenarios;
• fraud edge cases;
• refund exceptions;
• escalation cases.

The business later changes its refund policy.

The prompt is updated.

The retrieval documents are updated.

The evaluation set is not.

The application may now pass tests that no longer represent the rules it is supposed to follow.

The maintenance failure is not in the production model.

It is in the stale evaluation dependency.

That makes evaluation artifacts versionable assets in their own right.

An application release should ideally be able to answer not just:

Which model passed?

but:

Which evaluation pack, scoring logic and acceptance threshold were used to approve this configuration?

Without that, validation history becomes difficult to interpret after the underlying business environment changes.

Aliases make lifecycle management easier—and dependency visibility harder

Aliases are useful because they decouple application code from specific versions.

MLflow model aliases, for instance, can point a stable name such as champion to a particular model version, and that alias can later be reassigned without changing the serving code.

Prompt aliases work similarly: an application can load a prompt through an alias such as production, while the alias is redirected to another prompt version.

Operationally, that is convenient.

From a maintenance perspective, it introduces another moving layer.

If an application configuration says:

model = @champion

prompt = @production

then the apparent configuration can remain unchanged while the resolved versions behind those aliases move.

A useful inventory therefore needs both:

the logical reference

and

the currently resolved artifact version.

Otherwise, teams may know what the application asks for without knowing what it actually depends on today.

Ownership has to exist at application and dependency level

The maintenance burden becomes organisational as soon as the dependencies span teams.

The model may be owned by the AI platform team.

The prompt may be owned by customer operations.

The policy corpus may be owned by legal.

The refund API may be owned by commerce engineering.

The evaluation set may be maintained by QA.

The identity system may be owned by security.

The agent is therefore maintained by teams that may not think of themselves as jointly maintaining one application.

That matters during change.

Commerce engineering upgrades the refund API.

Legal republishes policy documents.

The platform team changes a model alias.

Each change may be reasonable within its own domain.

The application-level effect emerges only when the dependency graph is considered as a whole.

A named application owner does not remove the need for artifact-level ownership.

Someone still has to know who is responsible for retiring prompt versions, responding to a deprecated tool, updating evaluations after a policy change or revalidating the application after a material alias move.

Without that distinction, responsibility eventually collapses into a generic “AI team.”

Lifecycle status matters as much as version status

A dependency can be pinned and still become a maintenance problem.

The application may depend on an API version approaching retirement, a model family scheduled for deprecation, an unsupported library or an internal service its owning team plans to replace.

The inventory therefore needs enough lifecycle context to distinguish active, deprecated and end-of-life dependencies and to show whether a replacement or migration path exists.

This is where the maintenance surface starts resembling a living system rather than a launch checklist.

AI technical debt often appears as unrecorded relationships

The most expensive maintenance problem is not always an outdated artifact.

It can be an undocumented relationship.

Prompt 27 assumes tool schema 4.

Evaluation pack 8 assumes policy revision 13.

Retriever configuration 6 assumes metadata field effective_date.

The fallback model does not support the structured-output behaviour expected by one downstream parser.

None of those facts needs to exist as code dependency declarations.

They may live only in the heads of the engineers who built the system.

That is a particularly dangerous form of AI technical debt.

The artifacts are visible.

The relationships between them are not.

The maintenance inventory therefore needs more than a list.

It needs enough dependency structure to say:

prompt 27 was validated with tool schema 4

or:

retriever 6 requires metadata contract 3

or:

evaluation pack 8 applies to refund-policy revision 13 onward.

Without those relationships, the enterprise knows what exists but not what can safely change.

A flat spreadsheet eventually stops being enough

A simple inventory is still better than no inventory.

But the maintenance problem becomes graph-shaped surprisingly quickly.

One agent depends on five prompts.

Those prompts refer to three tools.

Two tools rely on shared APIs.

The retrieval layer uses four data sources.

Several applications share the same prompt or tool definition.

A single policy source feeds multiple agents.

At that point, one change can have a non-obvious blast radius.

The important maintenance questions become:

What uses this artifact?

What was this artifact validated with?

What breaks or needs revalidation if it changes?

Those are dependency-graph questions.

The exact implementation can vary.

What matters is that the relationships are queryable.

Not every dependency deserves the same governance

The answer is not to version and approve every configuration value with equal ceremony.

That would make maintenance impossible.

A formatting prompt for an internal summary is not equivalent to a payment tool, policy retrieval source, eligibility rule, customer-facing model route or security filter.

The practical question is:

Could an unnoticed change to this dependency alter a consequential decision, action, access path or customer outcome?

Higher-consequence dependencies deserve stronger versioning, testing, ownership and change controls.

Lower-consequence dependencies can use lighter processes.

The objective is visibility proportional to risk, not bureaucracy proportional to artifact count.

The maintenance burden appears when people try to change something

The hidden dependency problem often remains invisible while the system is stable.

It surfaces during maintenance.

A platform team wants to retire an old model.

Before doing so, it asks:

Which agents still use it?

Which prompts were tested with it?

Which evaluation packs apply?

Which tool definitions rely on its structured-output behaviour?

Which fallback routes reach it?

Which customer workflows would need revalidation?

If those answers require interviewing the original engineers, searching several repositories and reading old message threads, the maintenance problem already exists.

The same is true when a tool is deprecated or a retrieval service is replaced.

The cost is not caused by the change itself.

It is caused by not knowing the dependency graph before the change begins.

A useful inventory makes future change cheaper

A mature AI maintenance inventory does not have to reproduce every line of configuration.

It needs enough information to identify the application’s material dependencies, their owners and lifecycle state, and the relationships that determine what has to be revalidated when one of them changes.

That is the maintenance surface.

A model registry can remain part of that architecture.

It is simply not the whole architecture.

The real asset is the dependency map around the model

The model remains important.

But maintaining an enterprise AI application increasingly means maintaining the system around it.

Prompt registries address one part of that problem.

Model registries address another.

Tool registries and emerging MCP lifecycle controls are beginning to address another.

The harder operating problem is connecting those artifacts into one maintainable view.

The application should not become dependent on a prompt nobody owns, a tool nobody remembers changing, an evaluation pack nobody has updated or a retrieval configuration whose assumptions have disappeared.

A model can remain perfectly registered throughout all of that.

That is why the maintenance burden no model registry can see is not primarily a model-lifecycle problem.

It is the burden of keeping the dependencies around the model visible, owned and changeable before the people who understand them are gone.