Deleting a training record from storage does not delete what a model learned from it. The record may already have influenced parameters, checkpoints, embeddings, caches, and downstream models.

Machine unlearning asks a harder question: can we efficiently update a trained system so selected data no longer influences its behavior, while preserving everything the system should retain?

The deletion problem

Suppose a model was trained on dataset D, and a user asks us to remove subset Dᶠ. The retained data is Dʳ = D \ Dᶠ.

An unlearning algorithm receives the trained model and the deletion request, then returns a new model. The desired result is not merely worse performance on forgotten examples. It should resemble the model we would have obtained had Dᶠ never participated in training.

Retraining is the gold standard

The cleanest solution is to discard the old model and train again from scratch on Dʳ. This produces a reference model with no training path through the removed data.

It is also expensive. Large models may take days, specialized hardware, and a reproducible copy of the entire training pipeline. If deletion requests arrive continuously, complete retraining after every request is usually unrealistic.

Unlearning therefore optimizes a triangle: forgetting quality, retained utility, and computational cost. Improving one corner often puts pressure on another.

Exact and approximate unlearning

Exact unlearning aims for the distribution of unlearned models to match the distribution produced by retraining without the forgotten data. Equality of parameter values is not always required because training itself can be random; equivalence is defined over outcomes.

Approximate unlearning accepts a bounded, measurable difference from retraining. This makes more methods practical for deep networks, but creates the central verification problem: how much residual influence is acceptable, and how can an external observer know?

Fine-tuning until a model answers a few prompts differently is not automatically unlearning. It may suppress visible outputs while leaving internal representations or extractable information intact.

How unlearning methods work

  1. Design for retraining. SISA partitions data into isolated shards and slices, trains separate models, and aggregates them. A deletion only retrains the affected shard from an earlier checkpoint.
  2. Reverse the update. Influence- or curvature-based methods estimate how parameters would change if selected examples were absent. They can be efficient for well-behaved objectives but harder to trust in large non-convex networks.
  3. Optimize forgetting directly. Approximate methods raise loss on forgotten data while preserving performance on retained data, often with regularization, distillation, or parameter constraints.
  4. Limit influence during learning. Stable or differentially private training can reduce how much any one record changes the model, making later deletion easier to bound.

The method has to match the deletion unit. Forgetting one example, one user, one class, a visual concept, or a poisoned batch are different interventions with different success criteria.

How to evaluate unlearning

There is no single accuracy number for forgetting. A credible evaluation compares the unlearned model with both the original model and a retrained reference.

  1. Forget quality: behavior on deleted examples should approach the retrained reference, not simply collapse.
  2. Retain utility: performance on retained and unseen data should remain useful.
  3. Privacy resistance: membership-inference or extraction attacks should not reliably recover whether forgotten samples participated.
  4. Relearning resistance: a small amount of fine-tuning should not instantly recover a supposedly removed concept from dormant representations.
  5. Efficiency: report time, memory, checkpoints, retained-data access, and repeated-deletion cost against full retraining.

Federated unlearning

Federated learning keeps raw client data decentralized, but client updates still influence a shared global model. When one client asks to leave, the server may need to remove a sequence of aggregated contributions without possessing that client’s original data.

Possible strategies include replaying stored global checkpoints without the client, subtracting estimated updates, retraining selected rounds, or distilling a corrected global model. Non-IID client data and partial participation make verification especially difficult: removing a client can change both privacy and fairness for the clients who remain.

Limits and failure modes

A model can fail a deletion request in quiet ways. Outputs may change while internal features remain separable. Aggregate test accuracy may stay stable while a minority subgroup degrades. Repeated requests may accumulate approximation error. Copies of the model may survive in backups or downstream deployments even if the primary checkpoint is corrected.

Unlearning is therefore a systems problem, not only an optimizer trick. Data lineage, checkpoint policy, derived artifacts, audit logs, deployment inventory, and a clear threat model are part of the guarantee.

A practical checklist

  1. Define what must be forgotten. A record, user, class, concept, or every downstream artifact?
  2. Choose the guarantee first. Exact, certified approximate, or empirical removal?
  3. Build a retrained reference. At least on representative scales, it anchors the evaluation.
  4. Measure more than accuracy. Test privacy attacks, retained utility, subgroup effects, and relearning.
  5. Track the full system. Models, adapters, indexes, caches, logs, backups, and consumers all need deletion semantics.
  6. Document uncertainty. If the guarantee is empirical, say so. “Forgotten” should never mean “we stopped checking.”