EMEP Experiment Lifecycle: States and Transitions

Every operation in EMEP that produces or modifies a model, artifact, or evaluation result is executed within an Experiment. The Experiment lifecycle defines the states an experiment can occupy, the valid transitions between them, and the conditions that trigger each transition. This page also includes a sequence diagram showing how components interact during a typical experiment run.

Experiment States

State	Meaning	Entry Condition	Exit Condition
CREATED	Experiment is defined but not yet prepared	User or automated scheduler submits experiment spec	Preparation resources are allocated
PREPARING	Experiment is allocating resources, loading models, and verifying inputs	CREATED state confirmed; resource scheduler accepts job	All inputs pass validation and resources are ready
RUNNING	Experiment is actively executing merge, evolution, or evaluation steps	PREPARING completes without error	All compute steps finish or an error occurs
EVALUATING	Experiment is running benchmarks and computing fitness	RUNNING completes merge or evolution steps	All benchmarks complete or an evaluation error occurs
COMPLETED	Experiment finished successfully	EVALUATING completes without error	Results are archived; no further transitions
FAILED	Experiment terminated due to error	Any state encounters an unrecoverable error	Failure is logged; results may be partial
CANCELLED	Experiment was manually or automatically aborted	Any state receives a cancellation signal	Resources are released; no results archived
ARCHIVED	Experiment record is retained for audit but is no longer active	COMPLETED, FAILED, or CANCELLED after a retention period	Permanent; no transitions out

State Machine

The following state diagram shows all valid transitions. Invalid transitions are documented in the table below.

Valid and Invalid Transitions

Valid Transitions

From	To	Trigger	Guard Condition
CREATED	PREPARING	Resource scheduler accepts job	Experiment spec passes schema validation
CREATED	CANCELLED	User cancels before preparation	No guard; cancellation is always allowed from CREATED
PREPARING	RUNNING	All inputs validated	Model hashes match. Dataset splits exist. GPU nodes healthy.
PREPARING	FAILED	Input validation fails	Specific failure reason logged (hash mismatch, missing dataset, etc.)
PREPARING	CANCELLED	Cancel request during preparation	Resources released; partial downloads deleted
RUNNING	EVALUATING	Merge or evolution steps complete	No unhandled exceptions. Checkpoint saved.
RUNNING	FAILED	Unrecoverable error during compute	GPU failure, OOM, numerical instability, or code exception
RUNNING	CANCELLED	Cancel request during compute	Running processes killed. Checkpoints retained if configured.
EVALUATING	COMPLETED	All benchmarks complete	All metrics within valid ranges. No contamination detected.
EVALUATING	FAILED	Benchmark execution error	Benchmark crash, timeout, or invalid output format
EVALUATING	CANCELLED	Cancel request during evaluation	Partial results discarded or marked INCOMPLETE
COMPLETED	ARCHIVED	Retention policy expires	Automatic after configured days
FAILED	ARCHIVED	Retention policy expires	Automatic after configured days
CANCELLED	ARCHIVED	Retention policy expires	Automatic after configured days

Invalid Transitions

The following transitions are explicitly prohibited. The ExperimentTracker must reject any request to perform them.

From	To	Reason
CREATED	RUNNING	Preparation step cannot be skipped; resources must be allocated and inputs validated
CREATED	EVALUATING	No compute has occurred; nothing to evaluate
CREATED	COMPLETED	An experiment cannot complete without execution
CREATED	ARCHIVED	Must pass through a terminal state first
PREPARING	COMPLETED	Compute steps are required for completion
PREPARING	EVALUATING	Running step must precede evaluation
RUNNING	COMPLETED	Evaluation step must precede completion
RUNNING	ARCHIVED	Must pass through COMPLETED, FAILED, or CANCELLED
EVALUATING	RUNNING	Evaluation cannot revert to running; errors trigger FAILED
EVALUATING	PREPARING	No backward transitions are permitted
COMPLETED	RUNNING	A completed experiment is immutable
COMPLETED	FAILED	Completion is final; failures after completion are new experiments
FAILED	COMPLETED	A failed experiment cannot be retroactively completed
FAILED	RUNNING	Restart requires a new experiment with a new ID
CANCELLED	COMPLETED	Cancellation is final
CANCELLED	RUNNING	Restart requires a new experiment with a new ID
ARCHIVED	Any	Archived experiments are permanent records with no transitions

The experiment state machine has no backward transitions. Once an experiment leaves a state, it cannot return. This ensures that experiment history is append-only and auditable.

Sequence Diagram

The following sequence diagram shows a typical experiment execution flow, from submission through completion.

Experiment Metadata

Every experiment record contains the following metadata, stored in the ExperimentTracker and archived in the ArtifactStore.

Field	Description	Example
experiment_id	Unique identifier	EXP-2024-001
state	Current lifecycle state	RUNNING
spec	Full experiment specification	JSON document
inputs	Model IDs, dataset IDs, merge strategy	List of references
parameters	Merge weights, evolution hyperparameters, random seeds	Key-value map
checkpoints	Saved intermediate states	List of artifact hashes
results	Final metrics, fitness scores, Pareto front	JSON document
logs	Component logs, stdout, stderr	List of artifact references
provenance	Full lineage graph	Directed acyclic graph
created_at	Submission timestamp	ISO 8601
updated_at	Last state transition timestamp	ISO 8601
completed_at	Completion timestamp (if applicable)	ISO 8601
error_reason	Failure description (if FAILED)	String

System Dependency Map for the end-to-end pipeline
Component Dependency Graph for internal abstraction dependencies
Data Flow Map for weight and artifact movement
Experiment Tracking for the ExperimentTracker component specification
Reproducibility Protocol for experiment reproduction requirements

Project Context

Research Foundation

System Maps

EMEP Experiment Lifecycle: States and Transitions

Experiment States

State Machine

Valid and Invalid Transitions

Valid Transitions

Invalid Transitions

Sequence Diagram

Experiment Metadata

​Experiment States

​State Machine

​Valid and Invalid Transitions

​Valid Transitions

​Invalid Transitions

​Sequence Diagram

​Experiment Metadata

​Related Pages

Experiment States

State Machine

Valid and Invalid Transitions

Valid Transitions

Invalid Transitions

Sequence Diagram

Experiment Metadata

Related Pages