Skip to main content
The EvaluationEngine and BenchmarkEngine execute candidate assessment in EMEP. They run the full pipeline from model integrity checks through metric aggregation to candidate status assignment. This page specifies the complete evaluation flowchart, split isolation rules, and integration with the EvolutionEngine.

​
Candidate Evaluation Flowchart

​
Phase Descriptions

Candidate Model The input is a materialized candidate from the MergeEngine. It includes the model weights, genome, and parent references. The candidate must be in CANDIDATE lifecycle state. Integrity Check The EvaluationEngine verifies:
  • Model file checksums match the ArtifactStore record
  • Required metadata files are present
  • Tensor shapes are consistent with the genome specification
  • No NaN or Inf values in weight tensors (sampled check)
Failures produce INVALID status and terminate evaluation. Load Model The ModelLoader loads the candidate into the InferenceBackend. Loading includes device placement, dtype configuration, and optional quantization application. Memory usage is recorded at this stage. Select Benchmark Profile The BenchmarkEngine selects which benchmarks to run based on:
  • Experiment configuration
  • Candidate genome (for example code benchmarks for code-merge candidates)
  • Available compute budget
Profiles are pre-defined sets of benchmarks with specified splits and repetition counts. Run Benchmark Suite Benchmarks execute against the Optimization Set. Each benchmark runs with fixed seeds and decoding parameters. Results are collected per-sample and per-benchmark.
The Hidden Test Set is never accessed during this phase. The Validation Set is also excluded during evolution. Only the Optimization Set drives evaluation results that feed back into the EvolutionEngine.
Collect Metrics Raw outputs are converted to metrics: accuracy, exact match, ROUGE, BLEU, calibrated log-likelihood, latency, throughput, and memory usage. Metrics are stored with full provenance: benchmark ID, sample ID, seed, and timestamp. Calculate Scores Per-benchmark scores are computed from metrics. Scoring methods include:
  • Mean accuracy across samples
  • Macro-average across categories
  • Weighted combination for composite benchmarks
Aggregate Results Scores are aggregated into a single result dictionary per candidate. Aggregation respects benchmark weights from the experiment configuration. Compare Baseline The candidate’s aggregate scores are compared against the baseline model specified in the experiment. The baseline is typically the highest-performing parent or a previous release candidate. Regression Check Statistical tests determine if the candidate is significantly worse than baseline on any critical metric. The RegressionEngine uses bootstrap confidence intervals. See Regression Evaluation for the full protocol. Fitness Calculation The FitnessEngine computes scalar or vector fitness from the aggregated Optimization Set results. This value is passed back to the EvolutionEngine. See Fitness for formulas. Persist Results All raw metrics, scores, and status are written to the ExperimentTracker and ArtifactStore. The candidate record is updated with evaluation timestamps and resource usage. Candidate Status Final status is assigned:
  • PASS: All checks passed, fitness above threshold
  • FAIL: Below minimum thresholds
  • REGRESSION: Statistically worse than baseline on critical metrics
  • INVALID: Integrity or loading failure
  • INCOMPLETE: Evaluation interrupted

​
Split Isolation Diagram

​
Integration Points

Split isolation is enforced at the data layer, not just by convention. The DatasetRegistry tags every sample with its split, and the BenchmarkEngine validates that only Optimization Set samples are loaded during evolution-phase evaluation.