Foundation models are forcing privacy teams to revisit what anonymised really means

Introduction

In January 2026, France’s CNIL published a method for answering a question that used to sound unusual in privacy reviews: is the trained AI model itself anonymous?

Its answer does not begin with whether names were stripped from the training data. It asks whether people can be re-identified or their personal information extracted from the model using means reasonably likely to be used. In many cases, CNIL expects that assessment to include actual re-identification attacks against the model.

The question became harder to dismiss during 2026. In July, the European Data Protection Board opened consultation on new Guidelines 02/2026 on Anonymisation. Then, in October, the UK ICO published the results of its supervision of major foundation-model developers and stated directly that foundation models themselves can contain personal data when training information remains represented in ways that can be extracted or used to identify people.

Foundation model anonymisation privacy is therefore no longer only a question about the dataset that went into training. It increasingly requires evidence about what remains recoverable, linkable or identifiable after the dataset has become a model.

At a glance

• Removing direct identifiers from training data does not automatically make the resulting model anonymous.
• Foundation models can memorise parts of their training data, creating the possibility of extraction or re-identification.
• Regulators increasingly expect anonymity to be demonstrated through a case-specific assessment rather than assumed from preprocessing.
• The relevant test can include white-box access to model parameters as well as black-box extraction through queries.
• A model that is difficult to exploit behind a controlled API may present a different identifiability risk when its weights are downloadable or broadly accessible.
• The absence of obvious regurgitation during ordinary testing is not proof that personal data cannot be extracted through realistic attacks.
• Whether a model contains personal data affects data-protection obligations, rights handling, security testing and decisions about how the model can be distributed.

The privacy question has moved from the dataset into the model

Traditional anonymisation work begins with data. An organisation examines a table, document collection or analytical dataset and asks whether the people represented in it can still be identified after identifiers are removed, values are generalised or other protections are applied.

Foundation models complicate that sequence because the original records do not simply remain as rows in a database. Training transforms them into statistical parameters distributed across a large model, which can make it tempting to conclude that once the operational dataset has disappeared, the trained artefact no longer contains personal data in a meaningful sense.

Regulators are increasingly rejecting that shortcut. The ICO’s 2026 foundation-model work says personal information may be present not only in training datasets but also in the models themselves, pointing to memorisation and the possibility of disclosing or extracting training information.

CNIL takes a similar approach. A model trained on personal data may be considered anonymous, but only when the likelihood of identifying people represented in the training set or extracting their personal information is sufficiently insignificant under the relevant assessment.

The model is not automatically outside privacy analysis because its internal representation looks nothing like the original records. What matters is what those representations can still reveal.

Memorisation breaks the simple transformation argument

A foundation model is expected to learn statistical patterns from large amounts of training data, but that does not mean every piece of training information disappears into pure generalisation.

Models can memorise particular sequences, rare examples or repeated material strongly enough that some of it becomes recoverable. The ICO’s 2026 analysis identifies factors that can influence memorisation, including model size, duplication in training data, tokenizer design, training stage, training epochs, language, context length and inference sampling.

A model therefore does not need to behave like a database lookup system for its parameters to retain information about people. Nor does personal information need to appear every time someone asks an ordinary question before the possibility becomes relevant.

An unusual prompt sequence, access to model internals, knowledge about the training set or a purpose-built extraction technique can change what becomes recoverable. The anonymisation assessment consequently has to consider realistic adversarial behaviour rather than relying only on normal product use.

Anonymity is becoming something that has to be tested

CNIL’s January 2026 methodology is unusually concrete on this point.

For models trained on personal data, it recommends considering indicators such as identifying or precise training information, rare or outlier records, duplicated data, overfitting risk, model architecture and whether comparable models have already been successfully attacked. It also says that, in most cases, the provider’s documentation should include the results of re-identification attack tests under CNIL’s AI model status assessment methodology.

The objective is not to prove that no imaginable attacker could ever recover anything. The analysis considers means reasonably likely to be used, including information available to a potential attacker, the cost and time involved and the current state of extraction techniques.

This turns anonymisation from a label into an evidence problem. Statements such as “the data was anonymised before training” or “we cannot see individual records in the weights” no longer settle the question if realistic access to the trained model can still expose a route to identification.

For foundation models, that puts one of the central privacy questions much closer to security testing: what could somebody reasonably recover from this model, given the access and auxiliary information that could realistically exist?

White-box and black-box access do not present the same anonymity case

The same model can be exposed in very different ways. One organisation may keep weights inside tightly controlled infrastructure and provide access only through a filtered application. Another may give customers more direct access, while a third may publish downloadable weights.

Those deployment choices change what an attacker can attempt.

CNIL distinguishes between identification from model parameters and extraction of personal data through queries. It also says the assessment should consider access not only by the intended controller but by third parties who could obtain access, including people who should not have it.

Anonymity is therefore not usefully assessed only as a property printed on a model card. A model that resists practical extraction behind rate limits, filters and tightly restricted interfaces may present a different risk if the same weights are distributed to customers or released for unrestricted local execution.

CNIL also cautions that simply restricting access does not automatically make the model anonymous. Legal and contractual restrictions can complement technical safeguards and reduce the likelihood of re-identification, but they do not replace the underlying assessment.

For enterprises, model-distribution strategy has become part of the privacy analysis.

“It did not regurgitate anything” is weak evidence

Routine evaluation often tests whether a model produces obviously sensitive training information. That is useful, but it is not enough to establish anonymity.

CNIL explicitly notes that failure to observe regurgitation does not rule out memorisation. Its methodology calls for assessment against plausible re-identification and extraction attacks rather than treating ordinary interaction as the complete attack surface.

The distinction matters because foundation models are probabilistic systems. Outputs can change with prompts, decoding parameters, repeated querying, contextual information and the attacker’s prior knowledge. A piece of information that never appears during normal product testing may still become recoverable under conditions deliberately designed to expose memorised training material.

Anonymisation evidence therefore needs credible negative testing. A few unsuccessful prompts are not equivalent to demonstrating that the likelihood of extraction is insignificant.

The deployed system can change the privacy status of the model risk

A model that cannot itself be treated as anonymous may still be deployed inside a system that substantially reduces the practical likelihood of extraction.

CNIL describes this explicitly. Where a model is not anonymous, a provider can consider encapsulating it inside an AI system with robust protections such as preventing model retrieval, restricting access, limiting output precision, filtering responses and applying security controls. The resulting system still requires its own attack testing, but in some circumstances the use of that system may present a sufficiently low likelihood of re-identification.

Privacy teams therefore need to distinguish the status of the underlying model from the practical risk presented by a particular system built around that model. That prevents both an overstatement and an understatement: an API wrapper can reduce exposure materially, but it does not rewrite what is embedded in the model weights.

Foundation-model portability makes anonymisation a lifecycle question

The architecture around a foundation model rarely stays fixed. A model developed for an internal environment may later be licensed to customers. An API-only offering may acquire an on-premises version. Weights may move into another cloud, an edge environment or a partner-controlled deployment. Fine-tuning and quantisation can produce derivative versions with different characteristics.

The anonymisation assessment cannot safely ignore those changes.

CNIL’s methodology says the assessment may differ depending on deployment context and accessibility. A model exposed publicly to an unlimited population does not present the same practical conditions as one available only to a small, tightly controlled internal group.

A privacy conclusion reached for one access model should not therefore follow the model automatically into every later distribution arrangement. A new exposure path may change the means reasonably available to somebody trying to extract or link personal information.

Model governance increasingly has to preserve the assumptions under which the original anonymity assessment was made.

The October ICO position raises the stakes for foundation-model developers

The ICO’s 2026 foundation-model work makes this more than a theoretical European debate.

After supervising major foundation-model developers, the regulator said models themselves can contain personal data and highlighted growing evidence about the feasibility of extracting model training data.

The wider report also makes clear that determining whether personal information remains present in or recoverable from a model has practical consequences. Where personal data remains involved, developers must consider the data-protection obligations attached to its processing, including transparency, lawful basis, safeguards and people’s rights.

The important point for anonymisation is narrower than the mechanics of deleting information from models. Whether the trained artefact can genuinely be treated as anonymous helps determine which privacy obligations remain attached to the model in the first place.

That status is becoming harder to assume and more important to prove.

Sensitive inference and failed anonymisation are different privacy problems

Foundation models can create several forms of privacy risk that look similar from a distance, but they should not be collapsed.

A model may infer a sensitive attribute about an identifiable person from apparently ordinary inputs. That is a problem of new sensitive information emerging through inference.

A model may instead reproduce, expose or allow somebody to recover information that was already represented in its training data. That is the anonymisation and model-status problem at the centre of this article.

The first concerns what the model derives; the second concerns whether people represented in the training process can still be identified or their information recovered from the trained artefact.

Keeping the questions separate also helps privacy teams choose the correct test. Sensitive-inference testing asks what new attributes the system can derive. Anonymisation testing asks whether the model still carries a realistic path back to identifiable training information.

Both matter, but they are different control problems.

Procurement now needs evidence behind the word “anonymous”

Enterprises increasingly consume foundation models they did not train. That creates a practical governance problem when a provider describes training data, model artefacts or related processing using terms such as anonymised, de-identified or privacy-protected.

Those labels should trigger evidence questions rather than end them.

An enterprise evaluating a foundation model may need to know whether the provider assessed memorisation, what model-access assumptions were used, whether realistic extraction or re-identification attacks were performed, what safeguards depend on the surrounding system and whether the conclusion changes if the enterprise receives direct access to the weights.

The ICO’s current position reinforces the need for an evidence-based approach. It says the bar is higher where models themselves contain personal data and points to increasing evidence that model training data can be extracted.

“Anonymous model” is becoming less useful as an unsupported procurement assertion.

The more useful question is what evidence makes the provider confident that realistic access to this model does not create a meaningful route back to identifiable people?

2026 is turning anonymisation into a model-assurance issue

Three developments now point in the same direction.

CNIL has published a model-specific assessment method built around re-identification likelihood and attack testing. The EDPB’s Guidelines 02/2026 are bringing anonymisation back under active scrutiny, with the July draft currently open for feedback through October 30, 2026. The ICO’s 2026 foundation-model supervision work now states publicly that the trained model itself can be part of the personal-data problem.

Together, they are pushing privacy teams beyond dataset-centric anonymisation into model assurance: understanding memorisation, realistic extraction paths, access conditions and the evidence required before treating the trained artefact as anonymous.

That is a much harder standard than removing names from a training table, but it is better aligned with how foundation models actually work.

Enterprise watchpoints

As this regulatory position develops, six issues deserve attention:

• Model status: Has the organisation assessed the trained model itself, rather than relying only on anonymisation performed on the source dataset?
• Attack evidence: Has the model been tested against realistic extraction or re-identification techniques appropriate to its architecture and exposure?
• Access assumptions: Was the anonymity conclusion based on API-only access, restricted internal use, direct weight access or another deployment condition?
• Auxiliary information: Could public or commercially available information make model outputs easier to link back to individuals?
• Distribution changes: Would licensing, downloadable weights, on-premises deployment or broader access invalidate assumptions used in the original assessment?
• System safeguards: Which protections reduce extraction risk at deployment time, and are they properties of the model or of the surrounding system?

None of these tests guarantees that personal information can never be recovered. The objective is to make the anonymisation claim evidence-based rather than rhetorical.

FAQs

Can a foundation model itself contain personal data?

Potentially, yes. Regulators including the ICO and CNIL recognise that AI models can retain or encode training information in ways that may allow individuals to be identified or personal information to be extracted. Whether a specific model contains personal data requires a case-specific assessment.

Does anonymising training data automatically make a foundation model anonymous?

No. Training-data anonymisation can reduce risk, but the resulting model may still need to be assessed where personal information was present in the training process or where the model can memorise recoverable examples. The relevant question is whether people can realistically be identified from the trained artefact.

How can an organisation test whether an AI model is anonymous?

CNIL’s current methodology recommends assessing indicators that influence memorisation and, in many cases, performing re-identification or extraction attacks. The assessment should consider realistic attacker access, auxiliary information, technical capability, cost, time and the current state of extraction techniques.

Can access restrictions make a non-anonymous model anonymous?

Access restrictions can reduce re-identification risk, but CNIL says they do not automatically guarantee anonymity and do not replace the underlying assessment. A controlled system around a model may reduce practical extraction risk enough to change the assessment of that system’s use, while the model and system remain distinct objects of analysis.

Anonymous is becoming a claim about what can still be recovered

Foundation models have made one old privacy shortcut increasingly difficult to defend.

Data does not become anonymous simply because its original rows are no longer visible.

Training can move information into a different technical representation while leaving paths through which details about the people behind that information can still be recovered, linked or exposed.

The 2026 regulatory direction is forcing privacy teams to examine those paths directly.

For foundation models, anonymous is no longer best understood as a description of what the training data looked like before learning began.

It is becoming a claim about what the trained model can no longer realistically reveal.