Agentic data management should automate operations before it automates judgment

Introduction

A production pipeline fails at 02:13 because an upstream field has changed type.

An agent can inspect the failed run, read the execution logs, locate the transformation that no longer compiles, prepare a correction and run the relevant checks against a staging target.

The same agent may later encounter two customer records with conflicting identifiers, or a schema change that appears to alter the meaning of a finance metric rather than merely its type.

Those tasks sit inside the same data estate, but the second group asks the agent to resolve questions that technical state alone may not answer.

Agentic data management is most useful when agents first absorb operational work whose boundaries, permitted actions and success conditions can be defined before execution. Business meaning, policy exceptions and irreversible interpretation require a different standard because the desired answer may not exist in the system state the agent can inspect.

The near-term architecture should therefore automate operations before it automates judgment.

At a glance

• Data-management agents are already moving beyond code suggestions into planning, troubleshooting, transformation and validated execution.
• Operational tasks are better early autonomy candidates when the agent can work inside a known write boundary and test whether the requested outcome was achieved.
• Data-quality and schema problems become judgment problems when resolving them requires an unstated business rule rather than a technical correction.
• Agent authority should increase only as the enterprise can externalise the rules, permissions, validation and recovery behaviour surrounding the task.

Data-engineering agents are moving into the execution path

The shift from copilot to agent is already visible in current data platforms.

Google’s Data Engineering Agent can build, modify and troubleshoot pipelines in BigQuery and Dataform. Its documented capabilities include plan generation, code validation, data wrangling, pipeline optimisation, troubleshooting and repair, and metadata enrichment. Google announced the agent as generally available on April 22, 2026.

Microsoft is exploring a more autonomous execution model through Fabric’s Project Osmos, currently in preview. Microsoft describes it as a persistent data-engineering capability that can inspect data, create and test Spark notebooks, transform data, write Delta tables and validate results against the target environment.

These systems change the unit of automation.

A conventional assistant might generate SQL for a data engineer to inspect. An agent can be given an outcome such as repairing a failed pipeline, cleaning a source table or building a curated data layer and then work through multiple steps needed to reach it.

That compresses a large amount of repetitive engineering work while making the operating boundary more important than the prompt.

Operations are easier to automate when success can be observed

Many data-management activities have a state that can be checked after the work is performed.

Compilation, schema conformance, reconciliation totals, quarantine counts and duplicate checks all provide evidence about whether the requested technical state was reached.

These conditions do not make the underlying work trivial. They make acceptance less dependent on the agent’s own interpretation of success.

Microsoft’s Project Osmos documentation reflects this approach. Its managed task lifecycle includes planning, execution, validation and monitoring, and its examples use success conditions such as row counts, null checks, schema checks and before-and-after data-quality measures.

Google’s Data Engineering Agent similarly validates generated code and can investigate pipeline failures using execution evidence before producing probable causes and remediation recommendations. The documented troubleshooting flow can then be used to instruct the agent to apply the suggested fix.

This creates a strong early autonomy surface: the agent can perform complex work while the environment still supplies evidence that the requested technical state exists.

A failed pipeline is a better autonomy target than an ambiguous metric

Consider two incidents.

In the first, a transformation fails because transaction_date arrives as a string instead of the expected date type. The target model has an explicit contract. The agent can inspect the failure, find the incompatible field, generate a cast in a development workspace and test whether the pipeline again satisfies its declared schema.

Much of the desired state is already encoded.

Now consider a second incident in which net_revenue suddenly falls by 7%.

The pipeline completed normally. Types are valid. Freshness is within target. The underlying source team recently changed whether certain credits are recognised in the current period.

An agent can investigate the change.

It cannot infer from technical state alone whether the new treatment is the correct accounting interpretation for the business.

Automatically repairing the first problem can restore an agreed interface. Automatically “repairing” the second could amount to choosing which definition of revenue the enterprise should use.

The difference is not simply technical versus business work. It is whether the intended answer is already represented in policy, contracts, tests or other authoritative context.

Data-quality remediation exposes the boundary quickly

Data quality seems like an obvious target for autonomous agents because so much of it is repetitive.

Profiling columns, detecting null spikes, identifying malformed values, locating duplicates and standardising formats are expensive forms of operational toil.

Current agent tooling is already moving into this area. Google’s Data Engineering Agent can analyse raw data and suggest cleaning operations including duplicate handling, missing-value treatment, type conversion and format standardisation.

Microsoft’s Project Osmos guidance includes data-quality remediation as an example use case, but the recommended outcome is revealing: assess the problem, propose a safe remediation plan, apply approved changes to a staged table and produce before-and-after quality metrics.

Some remediation rules can be applied mechanically. Exact duplicates produced by an ingestion retry can be removed under a known rule, and non-canonical date formats can be normalised when the target representation is already specified.

Other cases require evidence outside the technical defect itself. Two customer records sharing an address may still represent different people. Conflicting birth dates need an authoritative source rather than a formatting rule. A newly introduced business status may need an owner to decide how it maps into an existing taxonomy.

The safe automation surface expands as those remediation rules become explicit.

Schema evolution should be treated as authority, not convenience

Schema changes are another area where agentic automation can become deceptively consequential.

An agent that notices a missing column and generates compatible transformation code is performing useful engineering work.

An agent that decides a downstream table should accept arbitrary new columns is changing the interface other systems rely upon.

Those are different permissions.

Project Osmos makes schema evolution one of the operating settings reviewed before a task begins. Its current quickstart asks users to decide whether output schemas may change and under what constraints. The same settings cover write safety, rerun behaviour and promotion of staged output.

That is a useful pattern for agentic data management more broadly.

Instead of giving an agent general permission to “fix the pipeline,” the platform can give it a narrower authority:

• repair code without changing the published schema;
• allow compatible type widening but reject arbitrary structural changes;
• write to staging rather than the production table;
• require promotion after validation;
• preserve existing source and verified outputs during retries.

These controls describe the operating envelope before the agent starts reasoning inside it.

Write authority should be designed separately from reasoning ability

A capable agent may be able to propose a good change long before an organisation should let it write that change directly into a production data estate.

Reasoning capability and execution authority should therefore be configured separately.

Google’s Data Engineering Agent supports plans that expose the agent’s intended steps, assumptions and files it expects to modify. Users can require explicit approval before the plan proceeds.

Microsoft goes further in its autonomous preview by making write behaviour a task setting. Its current terminology includes Staging table, Clone-and-promote and Iterate in place, while Fabric and OneLake permissions remain the hard authorization boundary.

The same separation can be applied elsewhere. An agent might inspect production telemetry while writing only to a development workspace, enrich metadata without changing business definitions, or restart a failed pipeline while remaining unable to delete source data. Higher-impact remediation can remain plan-only until the proposed write set has been reviewed.

The ability to reason about a resource should not automatically imply the ability to mutate it.

Rerun policy belongs in the operating envelope

Autonomous data work needs predetermined rerun behaviour because the same task may be invoked more than once against an evolving target.

Project Osmos asks operators to define whether reruns should fail, append, deduplicate, merge or replace. Its guidance also recommends stable business keys for incremental processing and reporting counts of inserted, updated, unchanged and rejected records.

Those choices belong in task configuration rather than being improvised during execution.

A pipeline that is safe to rerun under merge semantics may be unsafe under append. A remediation task may need to preserve rejected rows for review rather than silently overwrite them. Incremental work can depend on a stable business key that the agent is not authorised to redefine.

Predetermined rerun policy gives the agent a controlled recovery path without turning recovery itself into a fresh business decision.

Validation should not depend only on the agent believing it succeeded

Agentic workflows can produce a persuasive explanation of what they changed.

That explanation is not evidence that the data is correct.

A repair should be judged against system-level conditions that exist outside the agent’s narrative.

For a data pipeline, that might include compilation, schema conformance, row-count reconciliation, referential integrity, null thresholds, accepted-value checks, source-to-target financial totals or a controlled comparison with previous output.

Project Osmos is designed around this distinction: generated work can be executed and then tested against specified success criteria in the target environment.

Google’s Data Engineering Agent also distinguishes generated plans and code from pipeline review, data preview and execution evidence.

For high-impact data work, independently observable validation is one of the strongest controls available.

It lets an agent perform complex operations while keeping acceptance tied to evidence rather than confidence.

Instructions can encode operating policy but not invent business truth

Persistent agent instructions are useful because they can carry engineering conventions into repeated work.

Google allows Data Engineering Agent instructions to define reusable rules such as naming conventions and other organisation-specific guidance. It can also use Knowledge Catalog metadata and glossary context while planning data work.

That can move a meaningful amount of organisational knowledge closer to execution.

But an instruction file is only authoritative if the organisation has made it authoritative.

Writing “treat every missing customer tier as Standard” does not turn that statement into valid business policy.

Agentic data management therefore increases the value of machine-readable rules, contracts, schemas, glossary terms, validation criteria and ownership information. The more of the desired operating state an enterprise can express explicitly, the larger the safe automation surface becomes.

Where that state remains tacit, an agent is more likely to cross from execution into interpretation without anyone noticing the boundary.

Judgment begins where the desired state is under-specified

There is no universal line separating operations from judgment.

The boundary moves as enterprises encode more policy.

A duplicate-resolution decision may initially require a data steward. Later, if the organisation defines authoritative matching rules, survivorship logic, confidence thresholds and escalation conditions, much of the same work can become operational automation.

The important question is whether the agent is applying an approved rule or creating the rule required to complete the task.

That distinction can guide autonomy across common data-management activities:

Data-management task	Good early autonomy level	Why
Diagnose a failed pipeline	High within scoped access	Failure evidence and logs are observable
Repair code to restore a declared schema	High with validation and controlled writes	Desired interface already exists
Profile data and identify anomalies	High	Detection does not require changing business truth
Standardise formats under declared rules	High	Transformation is predetermined
Apply approved quality remediation in staging	High	Rule and write boundary are explicit
Accept a breaking schema change	Review required	Other consumers may depend on the interface
Merge ambiguous customer identities	Review or policy-driven escalation	Correct answer can require business evidence
Redefine a metric or business category	Human-owned judgment	The task changes enterprise meaning
Change retention or access policy	Human-owned policy decision	The task alters governance authority

This is not a permanent taxonomy.

As rules become explicit and validation becomes stronger, some tasks can move toward greater autonomy.

Bounded execution is the practical next step

Data teams have a large backlog of work that agents are well suited to absorb.

Pipeline generation, failure investigation, repetitive repair, profiling, transformation scaffolding, validation, metadata enrichment and performance tuning all consume engineering capacity.

Current platform designs already show how more of this work can become agentic without treating every data decision as equally delegable. Google exposes reviewable plans and scoped instructions; Microsoft is experimenting with persistent autonomous execution while making permissions, write safety, schema evolution and rerun behaviour explicit operating settings.

The most defensible expansion path is to automate work where intent is already encoded, execution can be bounded and results can be tested independently.

Once completing the task requires choosing what the data should mean, which policy should apply or which business exception should become the new rule, the agent has moved beyond operating the data estate and into decisions about how that estate is governed.