> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring Pipeline and Metrics Catalog

> Metrics catalog, dashboard definitions, and the OpenTelemetry pipeline for GPU health, experiment throughput, evaluation queues, and storage capacity.

EMEP exposes metrics from every component for real-time visibility into experiments, GPU utilization, and system health. This page defines the metrics catalog, the dashboards operators use, and the telemetry pipeline that carries measurements from producers to storage.

## Metrics Catalog

| Metric | Type | Source | Description |
| - | - | - | - |
| experiment.active | Gauge | ExperimentTracker | Number of experiments in RUNNING state |
| experiment.queue\_depth | Gauge | ExperimentTracker | Experiments waiting for GPU allocation |
| experiment.duration | Histogram | ExperimentTracker | Wall-clock time from CREATED to COMPLETED |
| merge.tensor\_ops\_rate | Counter | MergeEngine | Tensor operations per second during merge |
| eval.tasks\_completed | Counter | EvaluationEngine | Evaluation tasks finished |
| eval.queue\_wait | Histogram | EvaluationEngine | Time from task submission to start |
| gpu.utilization | Gauge | GPU Health Agent | GPU compute utilization percentage |
| gpu.memory.used | Gauge | GPU Health Agent | GPU memory allocated in bytes |
| gpu.ecc.errors | Counter | GPU Health Agent | Corrected ECC errors since boot |
| gpu.xid.errors | Counter | GPU Health Agent | Uncorrectable XID errors |
| storage.capacity.bytes | Gauge | Storage Operations | Total object store capacity |
| storage.used.bytes | Gauge | Storage Operations | Used object store bytes |
| registry.models.total | Gauge | ModelRegistry | Total models in REGISTERED or later state |
| registry.models.deprecated | Gauge | ModelRegistry | Models in DEPRECATED or ARCHIVED state |
| api.requests | Counter | API Gateway | Total API requests |
| api.latency | Histogram | API Gateway | Request latency in milliseconds |

## Dashboards

| Dashboard | Panels | Audience |
| - | - | - |
| GPU Health | ECC errors, thermal, XID errors, memory, PCIe errors | Infrastructure operators |
| Experiment Throughput | Active experiments, queue depth, duration percentiles | Experiment leads |
| Evaluation Queue | Tasks completed, wait time, failure rate | Evaluation engineers |
| Storage Capacity | Used vs total, growth rate, cold tier ratio | Storage operators |
| Registry Activity | Models by lifecycle state, daily registrations | Registry curators |

## Telemetry Pipeline

```mermaid theme={null}
flowchart LR
    subgraph PRODUCERS["Metric Producers"]
        E[ExperimentTracker]
        M[MergeEngine]
        V[EvaluationEngine]
        G[GPU Health Agent]
        S[Storage Operations]
        R[ModelRegistry]
        A[API Gateway]
    end

    subgraph COLLECTORS["OTel Collectors"]
        OC1[Collector Pool 1]
        OC2[Collector Pool 2]
    end

    subgraph STORAGE["Storage"]
        TSDB[Time-Series DB]
        RAW[Raw Log Archive]
    end

    subgraph CONSUMERS["Consumers"]
        DASH[Dashboards]
        ALERT[Alert Manager]
    end

    PRODUCERS -->|OTLP| COLLECTORS
    COLLECTORS -->|Remote Write| TSDB
    COLLECTORS -->|Batch Export| RAW
    TSDB --> DASH
    TSDB --> ALERT
```

Producers emit metrics in OpenTelemetry Protocol (OTLP). Collectors batch, filter, and forward to the time-series database. Raw logs archive to cold storage for forensic replay. Dashboards and alerts read from the time-series database.

## Retention

| Data Type | Hot Retention | Cold Retention |
| - | - | - |
| Metrics | 30 days | 1 year |
| Raw OTLP logs | 7 days | 90 days |
| Dashboard snapshots | 90 days | 1 year |

## Cross-Links

* [Alerting](https://doc.blueapi.ir/operations/alerting.md)
* [GPU Health](https://doc.blueapi.ir/operations/gpu-health.md)
* [Storage Operations](https://doc.blueapi.ir/operations/storage-operations.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.