> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# Model Rollback: Regression Recovery Protocol

> Rollback flowchart for production regression. Detect, freeze, restore, verify, resume, and post-mortem. Automated and manual triggers.

ModelRollback defines the protocol for restoring a previous production version when a regression is detected. This page specifies the rollback flowchart, automated and manual triggers, verification steps, and the post-mortem requirement.

## Rollback Flowchart

The protocol below shows the sequence from regression detection to full recovery.

```mermaid theme={null}
flowchart TD
    START([START]) --> DETECT[Detect Regression]
    DETECT --> FREEZE[Freeze Traffic]
    FREEZE --> RESTORE[Restore vN-1]
    RESTORE --> VERIFY[Verify Hash + Smoke Test]
    VERIFY --> PASS{Pass?}
    PASS -->|YES| RESUME[Resume Traffic]
    PASS -->|NO| ESCALATE[Escalate to On-Call + Manual Recovery]
    RESUME --> POST[Post-Mortem]
    POST --> END_PASS([END: RESTORED])
    ESCALATE --> END_FAIL([END: MANUAL RECOVERY])
```

## Detection Triggers

| Trigger | Source | Response Time |
| - | - | - |
| Automated latency alert | [Monitoring](https://doc.blueapi.ir/operations/monitoring.md) | Immediate |
| Automated error rate alert | [Monitoring](https://doc.blueapi.ir/operations/monitoring.md) | Immediate |
| Benchmark regression | [EvaluationEngine](https://doc.blueapi.ir/evaluation/framework.md) | Within evaluation window |
| Manual trigger | On-call engineer or Release Manager | Immediate |

Automated triggers initiate the freeze step without human approval. Manual triggers require explicit confirmation from the on-call engineer or Release Manager.

## Freeze Traffic

Freeze Traffic stops all new requests from reaching the current version. In-flight requests are allowed to complete or timed out based on the backend configuration. [DeploymentManager](https://doc.blueapi.ir/deployment/deployment-specification.md) updates the routing layer to send traffic to a maintenance endpoint or to the previous version if already staged.

## Restore vN-1

The previous version is restored from [ArtifactStore](https://doc.blueapi.ir/tracking/artifact-registry.md). The restoration process:

1. Locate the artifact for version N-1 using [ModelRegistry](https://doc.blueapi.ir/tracking/model-registry.md)
2. Verify the artifact hash against the registry record
3. Load the artifact into [InferenceBackend](https://doc.blueapi.ir/deployment/inference.md)
4. Run a smoke test with reference prompts

## Verify Hash

Hash verification compares the restored artifact SHA-256 against the canonical hash stored at release time. A mismatch aborts the automated rollback and escalates to manual recovery.

## Resume Traffic

After successful verification, traffic is gradually resumed:

1. Internal health checks pass for 60 seconds
2. 1% traffic shift for 5 minutes
3. 10% traffic shift for 5 minutes
4. 50% traffic shift for 5 minutes
5. Full traffic restoration

Any alert during the ramp re-triggers the freeze.

## RPO and RTO

| Metric | Target | Status |
| - | - | - |
| RPO (Recovery Point Objective) | Zero unprocessed requests | Placeholder pending operational calibration |
| RTO (Recovery Time Objective) | Under 5 minutes | Placeholder pending operational calibration |

> [!NOTE]
>
> RPO and RTO values are engineering assumptions. Final targets will be set after load testing and operational rehearsal.

## Post-Mortem

Every rollback requires a post-mortem within 48 hours. The post-mortem is stored in [ExperimentTracker](https://doc.blueapi.ir/tracking/experiment-tracking.md) and includes:

* Regression root cause
* Detection latency
* Rollback execution time
* Verification results
* Recommended preventive action

## Integration

ModelRollback is triggered by [DeploymentManager](https://doc.blueapi.ir/deployment/deployment-specification.md) and [Monitoring](https://doc.blueapi.ir/operations/monitoring.md). It reads artifacts from [ArtifactStore](https://doc.blueapi.ir/tracking/artifact-registry.md) and updates [ModelRegistry](https://doc.blueapi.ir/tracking/model-registry.md) status. Post-mortems are linked to [Model Lineage](https://doc.blueapi.ir/tracking/model-lineage.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.