EMEP Data Flow Map: Weights, Vectors, and Artifacts

The EMEP data flow map documents how information moves through the platform, from raw model weights to final evaluation reports and deployed artifacts. This page covers the five primary data categories (model weights, task vectors, candidates, evaluation results, and artifacts), the transformations applied at each stage, and the storage tiers that persist them.

Primary Data Categories

Category	Description	Typical Size	Source
Model Weights	Full-precision parameter tensors of base and merged models	7B-70B parameters, 14-140GB FP16	ModelLoader
Task Vectors	Parameter deltas between fine-tuned and base models	Same as model weights	TensorEngine
Candidates	Merge outputs packaged with metadata, status, and lineage	Same as model weights + metadata	CandidateGenerator
Evaluation Results	Raw and aggregated benchmark metrics	Kilobytes to megabytes per candidate	BenchmarkEngine
Artifacts	Hashed files including checkpoints, configs, logs, provenance records	Variable; provenance records are small	All components via ArtifactStore

Data Flow Diagram

The following diagram shows how data moves between components and storage tiers. Solid arrows indicate data flow. Dashed arrows indicate metadata or control signals.

Data Transformations

At each stage, data undergoes specific transformations. The following table documents these transformations and their requirements.

Stage	Input Data	Transformation	Output Data	Requirements
Import	Raw model weights (safetensors, PyTorch bin)	Hash verification, format normalization, metadata extraction	Normalized weights, import log	Hash must match published value
Validation	Normalized weights	Architecture schema check, tokenizer verification	VALIDATED or INVALID status	Schema must be in registry
Compatibility Analysis	Two VALIDATED weight sets	Layer-by-layer shape comparison, tokenizer overlap analysis	COMPATIBLE, CONDITIONALLY_COMPATIBLE, or INCOMPATIBLE report	Tensors must be loadable in memory
Task Vector Extraction	Fine-tuned weights, base weights	Element-wise subtraction: `delta = theta_ft - theta_base`	Task vector tensor	Shapes must be identical
Merge	Task vectors or full weights, merge parameters	Strategy-specific blend (average, SLERP, TIES, DARE, etc.)	Merged weight tensor	Numerical stability checks
Candidate Packaging	Merged weights, provenance metadata, parameters	Status assignment, lineage graph construction, artifact hashing	Candidate package with PASS/FAIL/REGRESSION/INVALID/INCOMPLETE status	Provenance must be complete
Evaluation	Candidate weights, benchmark tasks	Forward pass, metric computation, score aggregation	Raw metrics per task, aggregated scores	Benchmark must not be contaminated
Fitness Computation	Aggregated scores	Scalarization or Pareto ranking	Fitness score or Pareto front membership	Multi-objective logic must be consistent
Evolution	Population fitness, genomes	Mutation, crossover, selection	New generation of genomes	Population size must be maintained
Quantization	Full-precision weights, quantization config	GPTQ, AWQ, or GGUF precision reduction	Quantized weights	Accuracy loss must be within threshold
Deployment	Quantized or full-precision weights, serving config	Load into inference backend, health check	DEPLOYED model serving requests	Health check must pass

Storage Tiers

EMEP uses three storage tiers with different performance, cost, and durability characteristics.

Hot Storage

Location: GPU VRAM + NVMe SSD
Content: Active model weights, current generation population, running experiment checkpoints
Lifetime: Duration of experiment or inference session
Access Pattern: Random read/write, high bandwidth, low latency
Size: Tens to hundreds of gigabytes per active experiment

Warm Storage

Location: Object storage (S3-compatible, MinIO, Ceph)
Content: Completed candidates, evaluation results, provenance records, experiment logs
Lifetime: Months to years
Access Pattern: Sequential read for reproduction and audit
Size: Terabytes for full candidate history

Cold Storage

Location: Glacier, tape, or air-gapped archive
Content: Deprecated models, full provenance chains, hidden test set results, compliance records
Lifetime: Years to indefinite
Access Pattern: Read-only, accessed for compliance or research
Size: Petabytes at scale; typically compressed and deduplicated

Every data transfer between components is logged in the ExperimentTracker. The hash of data at the source is compared to the hash at the destination. Mismatches trigger a SUSPECT status and automatic retry.

The Hidden Test Set is stored in cold storage with access restricted to the BenchmarkEngine in a separate audit context. It is never loaded into hot storage during evolution or candidate selection. See ADR-0003.

System Dependency Map for the control-flow view of the pipeline
Component Dependency Graph for internal abstraction dependencies
Experiment Lifecycle for experiment state transitions
Artifact Registry for artifact storage details

Project Context

Research Foundation

System Maps

EMEP Data Flow Map: Weights, Vectors, and Artifacts

Primary Data Categories

Data Flow Diagram

Data Transformations

Storage Tiers

Hot Storage

Warm Storage

Cold Storage

​Primary Data Categories

​Data Flow Diagram

​Data Transformations

​Storage Tiers

​Hot Storage

​Warm Storage

​Cold Storage

​Related Pages

Primary Data Categories

Data Flow Diagram

Data Transformations

Storage Tiers

Hot Storage

Warm Storage

Cold Storage

Related Pages