A Real-Time Stream Can Change Meaning Without Changing Its Schema

Streaming data semantic consistency can fail when one field carries different business meanings at different points in the life of a stream. The events can remain current, structurally valid and fully readable while an AI agent loses the ability to determine what the same value meant when it was produced.

A logistics system publishes a field called:

delivery_eta

For years, it means the carrier-confirmed arrival time.

At 14:00 on Tuesday, a new forecasting service takes over.

The field remains a timestamp.

Its name remains delivery_eta.

The schema remains unchanged.

But events produced after 14:00 contain a model-generated prediction rather than a carrier-confirmed commitment.

The stream is healthy.

The data is current.

The meaning changed upstream.

One schema can outlive one meaning

Schema evolution controls are built to answer an important compatibility question:

Can producers and consumers still read the data?

Confluent’s Schema Registry documentation defines backward, forward and full compatibility around whether applications using different schema versions can continue reading events produced under those versions.

That does not make a schema version a complete record of business meaning.

In the ETA example, nothing has to change structurally.

Before 14:00:

delivery_eta = 16:30

means:

the carrier says the shipment will arrive at 16:30.

After 14:00:

delivery_eta = 16:30

means:

the forecasting model predicts that the shipment will arrive at 16:30.

The physical value is indistinguishable.

Its evidentiary meaning is not.

An agent that sees only the field, type and timestamp may have everything needed to parse the event and still lack what it needs to interpret it correctly.

Event time does not tell you which meaning was in force

Real-time systems already treat time as first-class information.

An event may have a production time.

The underlying business event may have its own timestamp.

Consumers use time for ordering, windows, late-event handling, reconciliation and replay.

But a timestamp answers:

When did this event happen?

It does not automatically answer:

What did this field mean at that time?

Suppose an incident investigator later retrieves two events:

13:58 — delivery_eta = 16:30

14:03 — delivery_eta = 16:30

The values are identical.

The schema is identical.

Only the semantic regime changed between them.

If that transition is recorded nowhere machine-readable, the historical record contains the data but not enough information to recover the historical interpretation.

A live stream can contain two semantic epochs

The transition at 14:00 creates a boundary inside the stream.

Everything before it belongs to one semantic epoch.

Everything after it belongs to another.

That matters because a continuously running consumer may cross the boundary without restarting, redeploying or encountering an error.

At 13:59, an operations agent treats delivery_eta as a carrier commitment.

At 14:01, it applies exactly the same logic to a prediction.

The agent has not changed.

Its prompt has not changed.

Its rules have not changed.

The data interface has not changed.

Yet the proposition being supplied to the agent has changed.

The failure therefore cannot be reconstructed merely by asking which application version or schema version was running.

The relevant history also includes which meaning was effective when the event entered the stream.

Replay can reproduce the value and lose the past

This becomes especially important when historical events are replayed.

A team investigating an automated escalation might rerun the events that preceded it.

A regression test might feed last month’s stream into a new agent version.

A model team might backtest whether different decision logic would have produced a better outcome.

A simulator might recreate the operational state at a particular point in time.

In all four cases, replay can faithfully reproduce:

delivery_eta = 16:30

while silently applying today’s definition of delivery_eta.

The value has been reconstructed.

Its historical semantics have not.

That can distort the analysis in subtle ways.

A replay might conclude that an agent overreacted to a low-confidence prediction when the original value was actually a carrier commitment.

Or it might conclude that an old decision was justified by authoritative data when the original event already contained a forecast.

The replay looks technically faithful because every event is present.

It is historically wrong because the meaning has been flattened.

Historical correctness needs more than historical values

This problem expands what data lineage has to preserve for long-lived streams.

For some fields, reconstructing the source and value is enough.

For semantically evolving fields, a consumer may also need to establish:

• which definition applied
• when that definition became effective
• whether the value was observed, confirmed, calculated or predicted
• which semantic or contract version governed it
• whether the consumer had migrated to that meaning at the time

The Open Data Contract Standard already treats semantics as part of a data contract rather than reducing the contract to physical structure. Its current specification supports semantic types, context, lifecycle/version information and links from properties to authoritative business definitions.

For streaming systems, that semantic information also has a temporal dimension.

The definition is not only what this field means.

It may also be what this field meant for events produced during this interval.

Separate fields can preserve the boundary visibly

Sometimes the cleanest solution is not to make a semantic transition implicit.

Instead of redefining:

delivery_eta

the producer could expose:

carrier_confirmed_eta

and:

predicted_eta

The distinction then survives in the event itself.

An old consumer does not silently reinterpret a familiar field.

A new consumer can deliberately choose the forecast.

A replay can distinguish which kind of ETA was available historically.

This is particularly valuable when both concepts may coexist rather than one permanently replacing the other.

The argument for separate fields is therefore not merely cleaner contract design.

It prevents two semantic epochs from becoming indistinguishable inside one historical stream.

A semantic version can preserve transitions without changing every field

Separate fields are not always practical.

A stream may contain hundreds of attributes.

Definitions may change for reasons that do not justify proliferating new physical fields.

In those cases, the event or surrounding metadata can carry enough information to recover the applicable meaning.

That might be a contract version.

It might be a producer semantic version.

It might be explicit provenance.

It might be another machine-readable identifier mapped to an effective definition.

The implementation can differ.

The requirement is narrower:

given an event from the past, a consumer should be able to determine which semantic regime governed it.

That requirement becomes more important as AI systems increasingly consume historical events outside their original operating context.

Consumer migration creates a transition period of its own

Meaning does not always change everywhere at once.

Suppose the producer switches to predictive ETA at 14:00.

One downstream service is updated at 14:00.

Another migrates at 17:00.

A third still assumes carrier-confirmed ETA the next morning.

There are now two timelines:

when the producer changed meaning

and

when each consumer became capable of interpreting the new meaning correctly.

That gap matters.

An incident during the transition cannot be understood from the producer’s version alone.

The investigator needs to know whether the particular consumer had crossed the semantic boundary too.

For autonomous agents, that can be the difference between reconstructing a valid decision and explaining an action that was made using the wrong interpretation of otherwise valid data.

Semantic history becomes part of test fidelity

Historical test sets are often treated as stable evidence.

Run the same data against a new system and compare outcomes.

But if the semantics of that data changed during the period being replayed, the test environment has to reproduce those changes as well.

Otherwise the new agent may receive old values interpreted through today’s definitions.

That affects regression testing.

It affects backtesting.

It affects simulation.

And it affects attempts to compare current agent behaviour with decisions made under earlier operating conditions.

A historical dataset is therefore not fully reproducible merely because every event has been retained.

For evolving business data, reproducibility also requires the historical meaning of the event to remain recoverable.

Real-time systems need an effective semantic boundary

Real-time architecture makes this problem easy to miss because continuity is one of its strengths.

Events keep moving.

Consumers stay connected.

The pipeline avoids interruption.

Nothing necessarily marks the moment when the same field begins carrying a different proposition about the business.

That moment still exists.

And if AI systems act on the stream, it deserves to be reconstructable.

The essential historical record becomes:

event value + event time + applicable semantic version

Only then can a consumer determine not simply what value arrived, but what that value represented at that moment.

The stream can remain perfectly continuous.

Its meaning does not have to be.

The useful question is:

If you replay an event six months later, can your agent recover what that field meant when the event was originally produced?