> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# EMEP KPI Specification: Metrics and Targets

> Key performance indicators for EMEP including documentation coverage, traceability, reproducibility, regression rate, throughput, latency, and deployment success metrics.

EMEP measures progress through seven key performance indicators. Each KPI has a clear definition, target value, measurement cadence, and data source. These metrics drive phase gate decisions and continuous improvement.

## KPI Registry

| KPI | Definition | Target | Measurement Cadence | Source |
| - | - | - | - | - |
| Documentation Coverage | Percentage of components with complete specifications | 100% | Weekly | Spec index vs. component list |
| Spec-to-Test Traceability | Percentage of specifications with linked test cases | 95% | Weekly | [Traceability matrix](https://doc.blueapi.ir/audit/traceability-matrix.md) |
| Reproducibility Score | Percentage of experiments reproducible from artifacts | 100% | Per experiment | [ExperimentTracker](https://doc.blueapi.ir/tracking/experiment-tracking.md) |
| Regression Rate | Percentage of candidates showing regression on Validation Set | Below 5% | Per generation | [EvaluationEngine](https://doc.blueapi.ir/evaluation/framework.md) |
| Mean Candidate Throughput | Candidates evaluated per GPU-hour | Above 2.0 | Daily | [EvolutionEngine](https://doc.blueapi.ir/evolution/evolution-engine.md) |
| Mean Evaluation Latency | Wall-clock time per benchmark run | Below 30 min | Daily | [BenchmarkEngine](https://doc.blueapi.ir/evaluation/benchmark-specification.md) |
| Deploy Success Rate | Percentage of deployments completing without rollback | Above 99% | Per deployment | [DeploymentManager](https://doc.blueapi.ir/deployment/deployment-specification.md) |

## Measurement Details

### Documentation Coverage

Calculated as: (specified components / total components) \* 100. A component is specified when it has a merged page in the documentation site with complete frontmatter, opening prose, and at least one diagram. Target is 100% by end of [Phase 0](https://doc.blueapi.ir/project/phases/phase-0.md).

### Spec-to-Test Traceability

Calculated as: (specs with linked tests / total specs) \* 100. A link requires a bidirectional reference: the spec page cites a test ID, and the test references the spec path. Target is 95% by end of [Phase 2](https://doc.blueapi.ir/project/phases/phase-2.md), 100% by end of [Phase 5](https://doc.blueapi.ir/project/phases/phase-5.md).

### Reproducibility Score

An experiment is reproducible when another operator can re-run it from stored artifacts and obtain the same candidate status (PASS, FAIL, REGRESSION, INVALID, INCOMPLETE) within a tolerance band. The score is: (reproducible experiments / total experiments) \* 100. Target is 100% for all experiments tracked in [ExperimentTracker](https://doc.blueapi.ir/tracking/experiment-tracking.md).

### Regression Rate

Calculated per generation: (REGRESSION candidates / total evaluated candidates) \* 100. Regression means a candidate scores lower than its parent on any [Validation Set](https://doc.blueapi.ir/evaluation/benchmark-specification.md) benchmark. Target is below 5% after [Phase 5](https://doc.blueapi.ir/project/phases/phase-5.md) stabilization.

### Mean Candidate Throughput

Candidates fully evaluated (all benchmarks, all splits) per GPU-hour. Measured on A100-equivalent hardware. Target is above 2.0 by end of [Phase 5](https://doc.blueapi.ir/project/phases/phase-5.md). This KPI is an [Engineering assumption](#evidence-tags).

### Mean Evaluation Latency

Wall-clock time from benchmark job submission to result availability, averaged over a 24-hour window. Target is below 30 minutes per benchmark by end of [Phase 4](https://doc.blueapi.ir/project/phases/phase-4.md).

### Deploy Success Rate

Percentage of [DeploymentManager](https://doc.blueapi.ir/deployment/deployment-specification.md) executions that complete without triggering [rollback](https://doc.blueapi.ir/enterprise/model-rollback.md). Calculated as: (successful deployments / total deployments) \* 100. Target is above 99% by end of [Phase 7](https://doc.blueapi.ir/project/phases/phase-7.md).

## Evidence Tags

| Tag | Application |
| - | - |
| Engineering assumption | Candidate throughput target assumes A100-class GPUs and fp16 inference |
| Future goal | 100% reproducibility requires deterministic GPU operations, which may need CUDA flags |
| Research hypothesis | Regression rate below 5% assumes effective [multi-objective optimization](https://doc.blueapi.ir/evolution/multi-objective-optimization.md) |

> [!WARNING]
>
> KPI targets are baselines for phase gates, not guarantees. If a target is missed, the phase gate requires a documented mitigation plan before proceeding.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.