The document did not change. The extraction model did.

IDP model regression testing has to answer a question that ordinary extraction evaluation often misses: what changed when the same documents were processed by a different model version? A processor upgrade can improve overall extraction performance and still alter one field, normalization rule or confidence signal that a production workflow depends on.

The invoice has not changed. It is the same PDF the accounts-payable team used during acceptance testing six months earlier. The supplier name is in the same place. The invoice date, tax amount and purchase-order reference are unchanged. Even the file hash matches the copy in the regression set.

The new extractor reads it differently.

Nothing dramatic happens. The supplier and total are still correct. The line items still appear. The document still clears the minimum confidence threshold. But one date is normalized differently, and a field that previously mapped to the purchase-order reference now returns a nearby identifier.

The new model looks better on the evaluation dashboard. The old one would have sent this invoice into the correct downstream match.

That is the awkward part of upgrading document AI in production. The change does not arrive as a new business requirement, revised form or broken integration. Sometimes the input remains fixed while the component responsible for turning that input into structured data changes underneath it.

The processor version is part of the production system

Document extraction is often discussed as though the processor were a stable capability sitting between a document and a workflow. Production platforms expose a more complicated reality: processors have versions, deployment states, defaults and migration paths.

Google Document AI’s processor-version management lets teams address a specific processor version, deploy and undeploy versions, or rely on a default. Its current guidance also notes that when a default version is deprecated, requests that do not specify a version can move to a newer stable version. Google advises users of custom processors to create and evaluate new user versions against newer base versions before replacing earlier ones.

The processor version therefore belongs in the production baseline alongside the workflow package, API version and ruleset.

If the application records the document, extracted fields and downstream outcome without recording the processor version responsible for those fields, an investigation can miss the component that actually moved.

The AP team in this case has exactly that problem. Their workflow definition is unchanged. Their mapping code is unchanged. The supplier invoice is unchanged. The candidate extractor is the only new variable.

A better aggregate score can conceal a worse field

The team runs the candidate processor against its test set. Precision improves. Recall improves. The overall F1 score moves in the right direction.

Google’s evaluation tooling reports precision, recall and F1 for the full document and for individual labels. Amazon Textract’s Custom Queries adapters likewise expose precision, recall and F1 for adapter versions by comparing predictions with ground-truth annotations, and AWS encourages testing new versions against representative documents before relying on them.

Those aggregate gains tell the team that the candidate deserves attention. The field-level results determine whether it can safely replace the production version.

Document-processing workflows rarely treat every extracted value as equally consequential. A description line can become slightly better without changing the downstream transaction. A purchase-order identifier, bank account, invoice date or tax value can differ by one character and redirect the entire case.

The candidate can improve across the corpus while regressing on the narrow field that happens to drive the next system action.

The follow-up question is therefore more useful than another aggregate score:

Where did the new version disagree with the production version?

The regression set has to remain still while the model moves

The team freezes a document set. It contains the ordinary invoices that dominate volume, but also the documents that have historically caused trouble: multi-page invoices, unusual number formatting, low-resolution scans, handwritten additions, supplier-specific layouts, several languages and documents where two nearby fields can plausibly be confused.

The current production version processes the set first. Then the candidate version processes exactly the same files.

The useful output is a diff. It should show which values changed, which fields disappeared and which previously empty fields are now populated. It should identify whether normalized dates, currencies or numbers changed representation, whether line-item boundaries moved and whether confidence values shifted enough to change routing behavior.

It should also reveal whether any extracted value began coming from a different part of the page.

A version-to-version comparison turns an abstract model upgrade into a finite set of production changes that somebody can inspect.

The discipline depends on the corpus remaining stable. If the test set changes at the same time as the processor, the team loses part of its ability to attribute an apparent improvement or regression to the version itself.

Google’s custom-extractor guidance separates training material from documents reserved for evaluation and allows teams to evaluate individual processor versions against test data before setting a new version as default.

For regression work, those retained documents become reference material for the production system. The documents stay still so the team can see what the processor changed.

A model fix can change what the workflow sees

The AP team also reviews what changed inside the extraction technology.

UiPath’s Document Understanding version history offers unusually concrete examples. Its July 21, 2026 v26.7.0 release fixed a case in which extraction from mixed-language documents did not always respect the date format configured in the schema. Earlier versions have changed number parsing, handwriting recognition, checkbox detection, line-item handling and other extraction behaviors.

Those fixes describe intentional changes in behavior. For a production team, they also identify areas worth exercising against its own document population.

In 2025, UiPath reverted improvements and fixes from one Data Extraction ML-package release because of unforeseen issues. The episode is a useful reminder that model progression is not necessarily monotonic. A newer version can solve the problem it was designed to solve while exposing a different weakness on documents the vendor did not see in the same proportions as a particular enterprise.

Release notes provide one view of the change. The regression corpus provides the enterprise’s view: which of those changes actually altered its documents, fields and workflows.

Confidence can change even when the extracted text does not

A second surprise appears in the routing results. Some handwritten invoices that previously remained in straight-through processing are now being sent for validation.

The extracted text has not necessarily become worse. The confidence signal has changed.

On September 23, 2026, UiPath changed the behavior of its Document OCR public endpoint so that confidence scores for handwritten text better reflect actual model confidence. UiPath warned that workflows using confidence thresholds could consequently send more handwritten documents to validation and recommended recalibrating those thresholds.

Production document workflows consume more than extracted text. Confidence scores, classifications, page boundaries and related metadata can determine whether a case proceeds automatically, enters validation or follows another branch.

A model update can therefore change workload without changing the text that eventually reaches the business system.

For the AP team, this belongs in the upgrade assessment because the threshold was calibrated against the previous model’s behavior. A threshold is not independent of the signal feeding it.

Every output consumed by the workflow is part of the version contract, including the outputs that never appear on the final invoice record.

An automatic migration is still a production change

Some upgrades never resemble a conventional application deployment.

UiPath’s 2026 transition to Helix Extractor 2.0 is a useful case. Its migration plan moved new model training to the newer extractor and stated that deployed project versions would be automatically migrated by the end of August 2026.

The invoice workflow does not need a code edit for the extraction technology beneath it to change.

Google’s default-version behavior creates a different version of the same operating condition. If an application calls a processor without pinning a specific version, a change in the default can alter which processor version handles later requests.

That leaves teams with two different operating models.

Pinning a processor version gives the enterprise tighter control over when behavior moves, while also creating responsibility for migration and deprecation. Following a default or auto-update path reduces some of that deployment work but increases the importance of detecting behavioral change when it occurs.

In either model, the processing record needs to preserve which version handled the document. Without that information, the pipeline can change faster than the organization can explain the resulting difference.

Promotion depends on understanding the disagreements

By the end of the regression run, the candidate processor does not match the production version perfectly. The team works through the differences rather than treating all of them as failures.

Some are clear improvements: the new version recovers a previously missed value. Some are harmless representation changes that downstream systems already tolerate. Some require mapping changes. A few are regressions that need either retraining, a workflow adjustment, a vendor fix or a reason not to promote the version yet.

Amazon Textract’s adapter model reflects this iterative lifecycle: adapter versions are created separately, evaluated using version-specific metrics and can be retrained and re-evaluated as teams add representative examples of the documents where performance is weak.

UiPath exposes a similarly explicit version relationship for deployed ML Skills. Teams select package versions for a skill and can choose whether the skill automatically updates to a newly retrained minor version.

That separation between candidate and production behavior gives the enterprise somewhere to make a decision.

The important evidence is no longer simply that version B scored higher than version A. It is that the differences significant to the workflow were identified, examined and accepted before version B became the production extractor.

Rollback has to preserve the old processor long enough to matter

The team approves the new model. It does not immediately delete the old one.

For a period, both versions remain identifiable. The deployment record says when the new extractor became active. The regression report records the test corpus, field-level differences and accepted exceptions. A small set of production documents is watched closely after promotion.

If an unexpected problem appears, the investigation starts with a simple question:

Which processor version produced this result?

That question is much easier to answer when version identity is part of processing telemetry and the previous version remains available long enough to support a controlled rollback.

Google Document AI’s deployment model makes versions explicit: teams can deploy and undeploy processor versions and call a specific version rather than relying solely on the default.

Rollback is not always possible indefinitely. Base models are deprecated. Old versions disappear. Vendors migrate platforms.

The useful rollback window therefore exists before retirement, while old and new behavior can still be compared and the earlier production state can still be reproduced.

That is the period in which the enterprise has to establish that the replacement can carry the workload it is inheriting.

The invoice becomes useful again because it did not change

Several weeks later, the original invoice is still in the regression pack. It remains deliberately boring.

Nobody edits the PDF to make the test more realistic. Nobody cleans up the scan. Nobody changes the supplier record. Its value comes from staying exactly where it was while extraction technology continues to move around it.

The next processor version will see the same invoice. So will the version after that.

Over time, the document becomes more than a test file. It is a reference point in the history of the production system. When its output changes, the team knows where to look.

The regression discipline is simple: preserve enough stable evidence to see when a new extractor has changed behavior that the business depends on, and make that difference visible before it becomes part of normal production.

The document did not change. That is precisely why the changed result matters.