Skip to main content
Catastrophic forgetting (McCloskey & Cohen 1989; Kirkpatrick et al. 2017) is the loss of previously learned capabilities when a model is fine-tuned on a new task. EMEP requires regression evaluation before and after every fine-tuning operation. No fine-tuned model is promoted without passing regression checks.

​
Control Methods

EMEP implements the following controls. Each is selectable at experiment creation time.

​
Rehearsal

Mix original pretraining or validation data into the fine-tuning batch. The model sees both new task data and representative samples from its original distribution. Rehearsal is effective but requires access to the original data or a curated replay buffer.

​
Replay Buffer

Store a fixed-size buffer of representative samples from the original task. Inject buffer samples into training batches at a configurable ratio. The buffer is managed by DatasetRegistry.

​
Low Learning Rate

Reduce the learning rate below standard fine-tuning values. Smaller updates perturb the weight landscape less, preserving prior knowledge at the cost of slower convergence on the new task.

​
Early Stopping

Monitor validation loss on the original task during fine-tuning. Stop training when original-task loss exceeds a threshold. Early stopping prevents overfitting to the new task.

​
Regularization (EWC)

Elastic Weight Consolidation (Kirkpatrick et al. 2017) adds a penalty term to the loss that protects important weights for prior tasks:
F_i is the Fisher information for parameter i. theta*_i is the optimal parameter for the prior task. EWC requires computing F_i before fine-tuning begins.

​
Distillation

Train the fine-tuned model to match the outputs of the original model on a held-out set. Distillation preserves soft target distributions and is effective for task sequences.

​
LoRA (Partial Mitigation)

LoRA (Hu et al. 2021) updates only a small fraction of parameters. This partially mitigates forgetting because the base weights remain frozen. LoRA does NOT guarantee zero forgetting. The adapter may still shift the effective output distribution in ways that degrade prior tasks.
LoRA and QLoRA do NOT guarantee zero catastrophic forgetting. Regression evaluation is mandatory before promotion.

​
Regression Evaluation Protocol

Every fine-tuning run must execute this protocol: The comparison uses the same benchmark harness, deterministic seeds, and backend configuration for both runs. Any score drop on the Optimization Set or Validation Set triggers FAIL or REGRESSION status.

​
Integration Points

  • EvaluationEngine: runs pre and post benchmarks.
  • BenchmarkEngine: provides deterministic harness and seed control.
  • ExperimentTracker: logs pre scores, post scores, and the comparison result.
  • ModelRegistry: blocks promotion if status is FAIL or REGRESSION.

​
Unsupported Claim

No single control method eliminates catastrophic forgetting in all cases. The most reliable mitigation is the regression evaluation protocol itself, which catches degradation before it reaches production.