Skip to main content
The QuantizationEngine is the EMEP component that reduces model weight precision to shrink memory footprint, accelerate inference, and enable deployment on constrained hardware. This page specifies every supported quantization method, the decision tree for selecting a method, the full execution pipeline, and the regression gate that prevents quality degradation from reaching production.
Concrete acceptance thresholds are tracked as open question OQ-006. All threshold placeholders in this page use the tag <threshold-tbd>.

​
Purpose

QuantizationEngine takes a validated, promoted model from the ModelRegistry and produces a quantized artifact for the InferenceBackend. The engine must preserve functional correctness, maintain accuracy within acceptable bounds, and produce artifacts that are reproducible from the same source model and configuration. The engine is NOT responsible for merging, fine-tuning, or evaluation. It receives a model at lifecycle state PROMOTED and produces a quantized artifact that transitions to RELEASED after passing all gates.

​
Supported Methods

QuantizationEngine supports six precision levels, each with distinct trade-offs in memory, speed, and accuracy.

FP16

Half-precision float. Near-lossless for most LLMs. 2x memory reduction versus FP32. Supported by all GPU backends.

BF16

Brain float 16. Wider dynamic range than FP16, lower risk of overflow. Preferred for training and inference on Ampere+ GPUs.

FP8

8-bit float (E4M3 / E5M2). Emerging standard on Hopper and newer. ~4x memory reduction. Requires hardware-native support.

INT8

8-bit integer with static or dynamic scaling. ~4x memory reduction. Well supported on CPU and GPU via ONNX Runtime and TensorRT.

INT4

4-bit integer (GPTQ / AWQ / GGUF). ~8x memory reduction. Enables large models on consumer GPUs. Higher accuracy risk.

FP32

Full-precision float. Baseline reference. Used only for pre-quantization benchmarking and regression measurement.

​
Method Selection Decision Tree

The engine selects a quantization method based on target hardware, latency requirements, memory budget, and accuracy tolerance. The following decision tree formalizes the selection logic.
This decision tree is an engineering assumption. Deployment operators may override the selected method via the configuration file.

​
Pre-quantization Benchmark

Before any weights are modified, QuantizationEngine runs a baseline benchmark on the FP32 or FP16 source model. The benchmark measures perplexity on a held-out validation split, exact-match accuracy on a standard QA task, and generation latency for a fixed prompt batch. Results are stored in the ExperimentTracker under the experiment state CREATED. The pre-quantization benchmark serves as the reference point for the regression gate. Without a clean baseline, the engine aborts with status INCOMPLETE.

​
Quantization Execution

QuantizationEngine executes the selected method through a backend-specific pipeline. The pipeline is deterministic: the same source model, method, and random seed produce the same quantized artifact.
1

Load source model

ModelLoader loads the source model from the ArtifactStore. The model must be in state PROMOTED. If the model is not found or fails integrity checks, the engine aborts.
2

Apply quantization

TensorEngine converts weights to the target precision. For INT8 and INT4, calibration data from the DatasetRegistry is used to compute scaling factors or quantization groups. For FP8, the engine uses hardware-native conversion routines.
3

Serialize artifact

The quantized model is serialized to the ArtifactStore with a new artifact ID, metadata tags for method and configuration, and a reference to the source model for lineage tracking.

​
Integrity Check

Immediately after serialization, the engine performs an integrity check. The check computes SHA-256 checksums of all weight files, validates the model config JSON against the expected schema, and attempts a forward pass with dummy inputs to confirm the model loads without runtime errors. Any failure marks the artifact as INVALID and halts the pipeline.

​
Post-quantization Benchmark

The engine runs the same benchmark suite on the quantized artifact that was run during the pre-quantization phase. Results are stored in ExperimentTracker under the same experiment, advancing the state to EVALUATING.

​
Regression Gate

The regression gate compares post-quantization metrics against the pre-quantization baseline. The gate is the final checkpoint before an artifact can transition to RELEASED.
The regression gate uses placeholder thresholds (<threshold-tbd>) pending resolution of open question OQ-006. Until thresholds are finalized, operators must review every quantization result manually.
The gate evaluates three metric categories:
  1. Perplexity delta: Relative change in validation perplexity must not exceed <threshold-tbd> percent.
  2. Task accuracy delta: Absolute change in exact-match accuracy must not exceed <threshold-tbd> points.
  3. Latency regression: Quantized latency must not exceed <threshold-tbd> percent of the baseline (a quantization should never slow inference).
If any category fails, the artifact is marked REGRESSION and the pipeline aborts. The operator may retry with a higher-precision method, adjust calibration data, or reject the model for deployment.

​
Acceptance Threshold Placeholders

Thresholds will be set through empirical calibration on a representative model suite before the first release. Track progress in OQ-006.