> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# GPU Health Monitoring and Remediation

> GPU health metrics, automated health checks, quarantine procedures, and auto-remediation for ECC errors, thermal events, XID errors, and memory failures.

GPU nodes are the most expensive and failure-prone resources in EMEP. A single faulty GPU can corrupt a merge, waste evaluation time, or silently degrade results. This page defines the health metrics, the check flow, and the auto-remediation actions that keep the GPU pool reliable.

## Health Metrics

| Metric | Source | Threshold | Action on Breach |
| - | - | - | - |
| ECC errors (corrected) | nvidia-smi | > 100 per hour | Log warning; schedule maintenance |
| ECC errors (uncorrected) | nvidia-smi | Any | Immediate quarantine |
| Thermal (GPU temp) | nvidia-smi | > 85 C | Throttle; alert; quarantine if sustained |
| XID errors | dmesg / NVML | Any non-zero | Immediate quarantine |
| Memory failures | nvidia-smi | Any row remapping event | Immediate quarantine |
| PCIe errors | lspci / dmesg | > 10 per minute | Alert; inspect link quality |
| GPU utilization | NVML | \< 5% for 10 min while assigned | Alert; check for hung process |

## Health Check Flow

```mermaid theme={null}
flowchart TD
    START([Poll GPU]) --> READ[Read Counters]
    READ --> THRESHOLD[Threshold Check]
    THRESHOLD --> HEALTHY{Healthy?}
    HEALTHY -->|YES| MARK[Mark Available]
    HEALTHY -->|NO| QUARANTINE[Quarantine GPU]
    QUARANTINE --> ALERT[Alert: Severity by Metric]
    ALERT --> AUTO[Auto-Remediation?]
    AUTO -->|YES| REMEDIATE[Attempt Remediation]
    AUTO -->|NO| MANUAL[Wait for Manual Action]
    REMEDIATE --> RETEST[Re-test after Cooldown]
    RETEST --> HEALTHY
    MARK --> END([End])
```

## Quarantine Procedure

A quarantined GPU is immediately removed from the scheduler pool. Active tasks on that GPU are marked for retry on another node. The GPU health agent records:

* Timestamp of quarantine
* Metric that triggered it
* Value at trigger
* Last assigned experiment ID

The GPU remains quarantined until a manual review clears it or auto-remediation succeeds.

## Auto-Remediation

| Condition | Action | Cooldown |
| - | - | - |
| Corrected ECC spike | Reset GPU; clear caches | 5 minutes |
| Thermal threshold | Reduce clock; increase fan | 10 minutes |
| Hung process (low utilization) | Kill process; release GPU | 2 minutes |
| PCIe errors | Reseat notification (alert only) | N/A |

Auto-remediation is never attempted for uncorrected ECC errors, XID errors, or memory failures. These require physical inspection or replacement.

## Manual Recovery

After remediation, an operator runs a full health check and a short burn-in test. The GPU re-enters the pool only if all metrics pass for the duration of the burn-in.

## Cross-Links

* [Monitoring](https://doc.blueapi.ir/operations/monitoring.md)
* [Alerting](https://doc.blueapi.ir/operations/alerting.md)
* [Disaster Recovery](https://doc.blueapi.ir/operations/disaster-recovery.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.