EMEP System Dependency Map: End-to-End Pipeline

The EMEP system dependency map shows how data, models, and control flow through the platform from initial discovery to final deployment. This page documents the primary pipeline, optional side branches, and the decision points that route models between subsystems.

Primary Pipeline

The primary pipeline is the closed loop that every model must traverse to reach RELEASED or DEPLOYED status. No model bypasses a stage unless explicitly marked as a side branch.

Stage Descriptions

Stage	Component	Input	Output	Decision Point
Discover	External crawlers, manual submission	Model metadata, download URLs	DISCOVERED model record	Is the model open-weight?
Import	ModelLoader	Model weights, config, tokenizer	IMPORTED model with verified hashes	Do hashes match published values?
Validate	ModelCompatibilityAnalyzer	IMPORTED model	VALIDATED or INVALID status	Does architecture match known schema?
Analyze Compatibility	ModelCompatibilityAnalyzer	Two or more VALIDATED models	COMPATIBLE, CONDITIONALLY_COMPATIBLE, or INCOMPATIBLE	Are tensor shapes aligned? Are tokenizers compatible?
Merge	MergeEngine	Compatible models, merge strategy, parameters	Candidate model weights	Does merge complete without numerical error?
Generate Candidate	CandidateGenerator	Merge output, provenance metadata	CANDIDATE status model	Is lineage graph complete?
Evaluate (Optimization)	EvaluationEngine	CANDIDATE, Optimization Set benchmarks	Raw metrics, aggregated scores	Did all benchmarks complete?
Compute Fitness	FitnessEngine	Evaluation metrics	Scalar or vector fitness score	Is fitness above threshold?
Evaluate (Validation)	EvaluationEngine	Candidate with acceptable fitness, Validation Set	Validation metrics	Do validation metrics confirm optimization results?
Promote	Manual curator or EvolutionEngine	Validated candidate	PROMOTED status	Is Pareto rank sufficient?
Quantize	QuantizationEngine	PROMOTED model, quantization config	Quantized model artifact	Does quantized model meet accuracy target?
Deploy	DeploymentManager	Quantized or full-precision model	DEPLOYED model in InferenceBackend	Does health check pass?
Hidden Test	BenchmarkEngine	DEPLOYED or RELEASED model, Hidden Test Set	Final evaluation report	Is result within expected range?

Optional Side Branches

The following branches are not part of the primary loop. They feed into it or consume its outputs.

Fine-Tuning Branch

Fine-tuning produces specialist models that may then be merged. It is optional and never required.

Stage	Component	Condition
Fine-tune	LoRA or QLoRA pipeline	User requests specialist capability
Import specialist	ModelLoader	Fine-tuned weights pass hash verification
Merge	MergeEngine	Specialist passes compatibility with other bases

See ADR-0004. The primary loop does not depend on fine-tuning. This branch exists to support teams that want to produce task-specific inputs for merging.

RAG Downstream Branch

RAG consumes EMEP-produced models but never creates them.

Stage	Component	Condition
Consume model	External RAG system	Team has RAG infrastructure
Build index	External vector database	Domain documents available
Serve	External application	End-user request arrives

See ADR-0005. Any request to include RAG in the primary loop must be rejected. RAG is a Phase 9 optional integration at best.

Storage Tiers

Data flows through three storage tiers during pipeline execution.

Tier	Content	Persistence	Access Pattern
Hot	Active model weights, current generation population, running experiment state	GPU memory + NVMe SSD	Random read/write during merge and evaluation
Warm	Completed experiment artifacts, candidate models from past generations, validation results	Object storage (S3-compatible)	Sequential read for audit and reproduction
Cold	Archived models, deprecated candidates, full provenance chains, hidden test set results	Glacier / tape / air-gapped archive	Read-only, accessed for compliance or research

Component Dependency Graph for internal abstraction dependencies
Data Flow Map for weight, vector, and artifact movement
Experiment Lifecycle for experiment state transitions
Context for the primary research loop definition

Project Context

Research Foundation

System Maps

EMEP System Dependency Map: End-to-End Pipeline

Primary Pipeline

Stage Descriptions

Optional Side Branches

Fine-Tuning Branch

RAG Downstream Branch

Storage Tiers

​Primary Pipeline

​Stage Descriptions

​Optional Side Branches

​Fine-Tuning Branch

​RAG Downstream Branch

​Storage Tiers

​Related Pages

Primary Pipeline

Stage Descriptions

Optional Side Branches

Fine-Tuning Branch

RAG Downstream Branch

Storage Tiers

Related Pages