EvaluationEngine: Full Specification and Master Flowchart

The EvaluationEngine executes the benchmark suite against candidate merged models and produces the structured metrics that feed into fitness calculation and candidate lifecycle progression. It is the gate between candidate generation and evolutionary selection: a candidate that fails evaluation or exhibits regression is rejected before it can influence the next generation. This page specifies the complete algorithm, data structures, decision points, and status assignments.

This specification is for a system that has not yet been implemented. All algorithmic details, parameters, and performance characteristics are engineering assumptions unless otherwise tagged.

Master Evaluation Flowchart

Candidate Status States

The EvaluationEngine assigns exactly one of the following statuses to every candidate:

Status	Meaning	Next Action
`PASS`	Candidate meets all thresholds and shows no regression	Proceed to FitnessEngine
`FAIL`	Candidate failed to load, crashed, or produced invalid outputs	Discard, log failure
`REGRESSION`	Candidate loaded and completed evaluation but underperformed baseline on at least one critical metric	Discard or retain with penalty, depending on configuration
`INVALID`	Candidate failed structural or integrity validation before loading	Discard, log validation error
`INCOMPLETE`	Evaluation was interrupted, timed out, or partially completed	Retry or discard based on retry policy

A candidate with status REGRESSION is never promoted to PROMOTED or RELEASED. It may be retained in the population for diversity only if the allow_regressive_diversity flag is set in the evolution configuration.

Purpose

The EvaluationEngine provides standardized, reproducible measurement of candidate model quality across a configured benchmark suite. It ensures that only validated, non-regressive candidates contribute to the evolutionary search and that all metrics are collected with consistent methodology.

Problem Definition

Given a candidate merged model

m_c

, a benchmark suite

B = \{b_1, b_2, ..., b_n\}

, and a baseline model

m_b

, determine whether

m_c

is fit for promotion by computing per-benchmark scores

s_i(m_c)

, comparing against

s_i(m_b)

, and assigning a candidate status.

Inputs

Input	Type	Description
`candidate`	`Candidate`	Genome and metadata from EvolutionEngine
`evaluation_profile`	`EvaluationProfile`	Benchmark suite, metrics, thresholds, and baseline reference
`baseline_model_id`	`ModelID`	Reference model for regression comparison
`compute_budget_per_candidate`	`ComputeBudget`	Max GPU-hours per candidate evaluation

Preconditions

The candidate genome has passed ModelCompatibilityAnalyzer validation.
The merged model checkpoint exists in the ArtifactStore.
The baseline model is in EVALUATED or RELEASED state.
The evaluation profile contains at least one benchmark split.

Parameters

Parameter	Type	Default	Description
`max_retries`	`int`	`2`	Number of retry attempts for `INCOMPLETE` evaluations
`timeout_per_benchmark`	`int`	`3600`	Seconds before a single benchmark is aborted
`regression_threshold`	`float`	`0.02`	Minimum relative drop to flag regression
`statistical_significance_level`	`float`	`0.05`	Alpha for bootstrap confidence intervals
`max_concurrent_benchmarks`	`int`	`4`	Number of benchmarks run in parallel

Step-by-Step Processing

Integrity Check

Verify that the candidate genome is structurally valid: all parent model IDs resolve, the merge strategy is supported, required fields are present, and the model checkpoint file exists and is non-empty. If any check fails, assign status INVALID and terminate.

Load Model

Invoke ModelLoader to load the merged model checkpoint into memory. Verify that the loaded model has the expected architecture, parameter count, and dtype. If loading fails, assign status FAIL and terminate.

Select Benchmark Profile

Load the evaluation profile associated with the experiment. The profile specifies which benchmarks to run, which metrics to collect, and which dataset splits to use. See Benchmark Specification for profile schema details.

Run Benchmark Suite

Execute each benchmark in the suite against the loaded model. Benchmarks run on the Optimization Set or Validation Set only. The Hidden Test Set is never used during evolution. See Benchmark Specification for the split architecture and enforcement rules.

Collect Metrics

Gather raw outputs from each benchmark: accuracy scores, perplexity, latency measurements, token counts, and any benchmark-specific metrics. Store raw outputs in the ArtifactStore for reproducibility.

Calculate Scores

Compute per-benchmark scores from raw outputs. Each benchmark defines its own scoring function (e.g., exact match, F1, BLEU, pass@k). Scores are normalized to

[0, 1]

where applicable.

Aggregate Scores

Combine per-benchmark scores into category-level and overall scores using the aggregation weights defined in the evaluation profile.

Compare Baseline

Compare candidate scores against the baseline model on each metric. Use bootstrap confidence intervals to determine statistical significance. See Regression Evaluation for the full regression detection algorithm.

Regression Check

If any critical metric shows a statistically significant drop below the baseline by more than regression_threshold, assign status REGRESSION.

Calculate Fitness

Pass aggregated metrics to the FitnessEngine to compute the fitness scalar or vector. This step is skipped for candidates with status FAIL, INVALID, or INCOMPLETE.

Persist Results

Write all scores, status, and metadata to the ExperimentTracker. Update the candidate lifecycle state to EVALUATED.

Decision Points

Decision	Condition	True Branch	False Branch
Integrity passed	Genome schema valid, checkpoint exists	Proceed to load	`INVALID`
Load successful	Model loads without error, architecture matches	Proceed to benchmarks	`FAIL`
Benchmarks completed	All benchmarks finished within timeout	Proceed to metrics	`INCOMPLETE`
Regression detected	Significant drop on any critical metric	`REGRESSION`	`PASS`

Outputs

Output	Type	Description
`candidate_status`	`enum`	One of `PASS`, `FAIL`, `REGRESSION`, `INVALID`, `INCOMPLETE`
`per_benchmark_scores`	`Dict[string, float]`	Normalized score per benchmark
`aggregated_scores`	`Dict[string, float]`	Category-level and overall scores
`fitness_input`	`Dict[string, float]`	Metrics passed to FitnessEngine
`regression_report`	`RegressionReport`	Per-metric baseline comparison with CIs
`raw_outputs`	`ArtifactRef`	Reference to persisted raw benchmark outputs

Failure Conditions

Integrity failure: Genome validation fails. Status: INVALID.
Load failure: Model checkpoint corrupted or incompatible. Status: FAIL.
Benchmark timeout: Single benchmark exceeds timeout_per_benchmark. Status: INCOMPLETE.
Out of memory: GPU memory exhausted during inference. Status: FAIL.
Compute budget exhausted: compute_budget_per_candidate exceeded. Status: INCOMPLETE.

Validation

Genome schema validated against Genome JSON Schema.
Benchmark profile validated against Benchmark Specification schema.
All baseline model IDs verified against ModelRegistry.
Score values bounded to prevent NaN/Inf.

Complexity

Phase	Complexity	Notes
Integrity check	$O(1)$	Schema and file existence checks
Model load	$O(S)$	$S$ = model size, dominates for large models
Benchmark execution	$O(B \cdot D)$	$B$ = number of benchmarks, $D$ = dataset size
Score aggregation	$O(B)$	Weighted sum over benchmarks
Regression check	$O(B \cdot R)$	$R$ = bootstrap resamples

Memory Requirements

Model checkpoint: $O(S)$ where $S$ = model parameter count.
Benchmark datasets: $O(D)$ loaded on-demand or streamed.
Raw outputs: $O(B \cdot N)$ where $N$ = number of examples per benchmark.

Reproducibility

Random seeds fixed for benchmark data shuffling and model inference sampling.
All evaluation hyperparameters logged to ExperimentTracker.
Raw outputs persisted to ArtifactStore with content-addressable hashes.

Unit Tests

Test	Description
`test_integrity_invalid_genome`	Rejects genome with missing required field
`test_load_corrupted_checkpoint`	Assigns `FAIL` when checkpoint is corrupted
`test_benchmark_timeout`	Assigns `INCOMPLETE` when timeout exceeded
`test_regression_detected`	Assigns `REGRESSION` when metric drops below baseline
`test_pass_all_clear`	Assigns `PASS` when all checks succeed
`test_persist_results`	Results written to tracker with correct experiment ID

Integration Tests

Test	Description
`test_end_to_end_evaluation`	Full pipeline with mock model and synthetic benchmark
`test_regression_statistical_significance`	Bootstrap CI correctly identifies spurious vs. real regression
`test_concurrent_benchmarks`	Multiple benchmarks execute in parallel without interference
`test_retry_incomplete`	`INCOMPLETE` candidate retried up to `max_retries`

Acceptance Criteria

Evaluation pipeline completes for a valid candidate in under the configured compute budget.
PASS candidates have fitness values strictly greater than or equal to baseline on all critical metrics.
REGRESSION candidates are never promoted to PROMOTED or RELEASED.
All candidate statuses are persisted and queryable via ExperimentTracker.
Raw benchmark outputs are reproducible from persisted seeds and configuration.

Known Limitations

Benchmark execution is synchronous per candidate; no batching across candidates is specified yet (Future goal).
GPU memory limits may prevent evaluation of very large merged models; no automatic model partitioning is defined (Engineering assumption).
Benchmark suite selection is static per experiment; dynamic adaptation based on candidate properties is not supported (Research hypothesis).

Research References

Evaluation practices for LLMs: Open LLM Leaderboard methodology (Engineering assumption)
Bootstrap confidence intervals: Efron and Tibshirani 1993 (Engineering assumption)

Evolution Engine

Evaluation

Tracking & Reproducibility

EvaluationEngine: Full Specification and Master Flowchart

Master Evaluation Flowchart

Candidate Status States

Purpose

Problem Definition

Inputs

Preconditions

Parameters

Step-by-Step Processing

Decision Points

Outputs

Failure Conditions

Validation

Complexity

Memory Requirements

Reproducibility

Unit Tests

Integration Tests

Acceptance Criteria

Known Limitations

Research References

​Master Evaluation Flowchart

​Candidate Status States

​Purpose

​Problem Definition

​Inputs

​Preconditions

​Parameters

​Step-by-Step Processing

​Decision Points

​Outputs

​Failure Conditions

​Validation

​Complexity

​Memory Requirements

​Reproducibility

​Unit Tests

​Integration Tests

​Acceptance Criteria

​Known Limitations

​Research References

Master Evaluation Flowchart

Candidate Status States

Purpose

Problem Definition

Inputs

Preconditions

Parameters

Step-by-Step Processing

Decision Points

Outputs

Failure Conditions

Validation

Complexity

Memory Requirements

Reproducibility

Unit Tests

Integration Tests

Acceptance Criteria

Known Limitations

Research References