Model Merging
Model Soups: Averaging Weights of Multiple Fine-Tuned Models Improves Accuracy Without Increasing Inference Time
Authors: Wortsman et al. (2022) Method: Simple weight averaging of multiple fine-tuned models initialized from the same pre-trained checkpoint. Finding: Averaging weights of models fine-tuned with different hyperparameters improves accuracy on image classification and NLP tasks without increasing inference cost. Limitations: Requires identical architecture and initialization. Does not address sign conflicts or redundant parameters. Performance gains diminish as model count increases. Relevance to EMEP: Validates the basic premise that parameter-space combination can outperform individual models. Underlies the Linear Merge strategy.Evolutionary Model Merge: Evolving Large Language Models with Evolutionary Computing
Authors: Akiba et al. (2024) Method: Evolutionary search over merge strategies and model combinations to discover superior LLMs without gradient-based training. Finding: Evolutionary merging can produce models that outperform their ancestors on Japanese and English benchmarks by exploring non-obvious combinations. Limitations: Search space grows combinatorially. Requires many evaluation runs. Evolutionary hyperparameters (population size, mutation rate) are task-dependent. Relevance to EMEP: Core inspiration for the EvolutionEngine and CandidateGenerator. Demonstrates that merge-based evolution is a viable alternative to training.Task Arithmetic
Editing Models with Task Arithmetic
Authors: Ilharco et al. (2022) Method: Adding or subtracting task vectors (fine-tuned minus pre-trained weights) to edit model capabilities without retraining. Finding: Task arithmetic enables capability addition, removal, and negation with predictable directional effects in parameter space. Limitations: Task vectors can interfere destructively. Sign and magnitude conflicts reduce effectiveness when many tasks are combined. Requires fine-tuned intermediates. Relevance to EMEP: Foundational method for the Task Arithmetic merge strategy. Informs the definition of task vectors and delta computation in the TensorEngine.SLERP
Animating Rotation with Quaternion Curves
Authors: Shoemake (1985) Method: Spherical linear interpolation (SLERP) between quaternions, extended to high-dimensional vectors. Finding: Interpolating along the geodesic on a unit sphere preserves vector magnitude and angular relationships better than linear interpolation. Limitations: Assumes vectors can be normalized. Performance in very high dimensions (billions of parameters) is an engineering assumption, not proven. Relevance to EMEP: Basis for the SLERP merge strategy. Used to interpolate between full model checkpoints while preserving scale.TIES-Merging
TIES-Merging: Resolving Interference When Merging Models
Authors: Yadav et al. (2023) Method: Trim redundant parameters, elect the majority sign, and merge disjoint parameter sets to reduce task vector interference. Finding: TIES outperforms naive averaging and task arithmetic when merging multiple task-specific models, especially when task vectors conflict. Limitations: Requires task vectors (fine-tuned minus base). Trimming threshold is hyperparameter-sensitive. Does not address architectural differences. Relevance to EMEP: Core TIES merge strategy. Informs the MergeEngine interference-reduction design.DARE
DARE: Drop And REscale for Merging Multiple Models
Authors: Yu et al. (2023) Method: Randomly drop a fraction of delta parameters and rescale the remainder to preserve the expected magnitude of the combined update. Finding: DARE enables merging of many models (up to hundreds) without catastrophic performance collapse, by reducing parameter interference. Limitations: Drop rate is a critical hyperparameter. Random dropping may remove important parameters. Theoretical justification for rescaling is approximate. Relevance to EMEP: Core DARE merge strategy. Combined with TIES in the DARE + TIES strategy.LoRA and QLoRA
LoRA: Low-Rank Adaptation of Large Language Models
Authors: Hu et al. (2021) Method: Fine-tune low-rank decomposition matrices injected into transformer layers instead of full weights. Finding: LoRA achieves comparable fine-tuning performance to full fine-tuning with orders of magnitude fewer trainable parameters. Limitations: Rank selection is task-dependent. Not all architectures support LoRA natively. Merging LoRA adapters with base models requires care. Relevance to EMEP: Optional LoRA fine-tuning for producing specialist models. Informs PEFT component design.QLoRA: Efficient Finetuning of Quantized LLMs
Authors: Dettmers et al. (2023) Method: Fine-tune LoRA adapters on a 4-bit quantized base model using paged optimizers to reduce memory. Finding: QLoRA enables fine-tuning of 65B parameter models on a single 48GB GPU with minimal accuracy loss. Limitations: 4-bit quantization introduces noise. Paged optimizers add latency. Not all operations are supported in 4-bit backends. Relevance to EMEP: Optional QLoRA for low-resource fine-tuning. Informs QuantizationEngine integration with fine-tuning pipelines.Catastrophic Forgetting
Overcoming Catastrophic Forgetting in Neural Networks
Authors: Kirkpatrick et al. (2017) Method: Elastic Weight Consolidation (EWC). Penalizes changes to important parameters during sequential learning. Finding: EWC reduces catastrophic forgetting in supervised and reinforcement learning by protecting critical weights. Limitations: Requires computing Fisher information, which is expensive for billion-parameter models. Assumes task boundaries are known. Relevance to EMEP: Informs Catastrophic Forgetting Controls. Merge-based methods may bypass forgetting by preserving base model weights.Distillation
Distilling the Knowledge in a Neural Network
Authors: Hinton, Vinyals, and Dean (2015) Method: Train a smaller student model to match the softmax outputs of a larger teacher model. Finding: Distillation transfers teacher capabilities to smaller models with less data than training from scratch. Limitations: Requires a capable teacher model. Student architecture must be chosen carefully. Distilled models may not preserve all teacher behaviors. Relevance to EMEP: Optional downstream technique for compressing evolved models. Not part of the primary loop.Quantization
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
Authors: Frantar et al. (2022) Method: Layer-wise quantization using approximate second-order information for one-shot post-training quantization. Finding: GPTQ quantizes models to 3-4 bits with minimal perplexity increase, enabling large model inference on consumer hardware. Limitations: Sensitive to activation outliers. Calibration data must be representative. Not all layer types quantize equally well. Relevance to EMEP: Candidate implementation for QuantizationEngine.AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
Authors: Lin et al. (2023) Method: Protect salient weight channels based on activation magnitudes during quantization. Finding: AWQ outperforms GPTQ on downstream tasks by preserving critical weight channels. Limitations: Requires activation analysis. Protection strategy is heuristic. Hardware support varies. Relevance to EMEP: Candidate implementation for QuantizationEngine.LLM Evaluation
Holistic Evaluation of Language Models
Authors: Liang et al. (2022) Method: HELM benchmark evaluates language models across scenarios, metrics, and perturbations. Finding: No single model dominates all scenarios. Trade-offs between accuracy, fairness, robustness, and efficiency are pervasive. Limitations: Benchmarks become stale as models improve. Contamination is hard to detect. Scenario coverage is never complete. Relevance to EMEP: Informs Benchmark Catalog design and Multi-Objective Optimization rationale.Measuring Massive Multitask Language Understanding
Authors: Hendrycks et al. (2020) Method: MMLU evaluates models on 57 tasks covering mathematics, history, law, and more. Finding: MMLU is a strong discriminator of model knowledge and reasoning capabilities. Limitations: Multiple-choice format may not reflect real-world use. Contamination from pre-training is a known risk. Some subjects are underrepresented. Relevance to EMEP: Candidate benchmark for the Optimization Set and Validation Set.Benchmark Contamination
Time Travel in LLMs: Tracing Data Contamination in Large Language Models
Authors: Jacovi, Goldberg, and Tsarfaty (2023) Method: Temporal analysis and membership inference to detect benchmark contamination in pre-training corpora. Finding: Contamination is widespread and often undetected by standard evaluation protocols. Limitations: Membership inference is imperfect. Temporal analysis requires known publication dates. Not all benchmarks have temporal metadata. Relevance to EMEP: Core reference for Benchmark Integrity and Project Constitution Rule 17.Reproducibility
Reproducibility in Machine Learning: A Case Study
Authors: Gundersen and Kjensmo (2018) Method: Survey and case study of reproducibility failures in ML research. Finding: Most published ML experiments are not fully reproducible due to missing code, data, hyperparameters, or hardware details. Limitations: Survey is from 2018; practices have improved but remain inconsistent. Reproducibility standards vary by subfield. Relevance to EMEP: Motivation for Reproducibility Protocol, ArtifactStore, and ExperimentTracker.RAG
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Authors: Lewis et al. (2020) Method: Combine dense passage retrieval with sequence-to-sequence generation to augment model knowledge without parameter updates. Finding: RAG improves factual accuracy and reduces hallucination on knowledge-intensive tasks compared to parametric-only models. Limitations: Retrieval quality limits generation quality. Requires external index maintenance. Not a substitute for model capability improvement. Relevance to EMEP: Downstream consumption of EMEP models. See RAG Architecture.Related Pages
- Evidence Map for mapping EMEP claims to evidence sources
- Component Mapping for mapping components to research dependencies
- Decisions for architecture decisions informed by this research
- Glossary for term definitions