What is model quantization?
Fewer bits. Measured trade-offs.

The same weights can use fewer bits to be stored.

A model’s parameters are numerical values. Quantization represents those values with a lower-precision format, approximating the original numbers with a finite set of levels. Instead of storing a parameter in 16-bit floating point, a quantized model might store it in 8-bit or 4-bit form, along with scale information used to interpret the packed values.

For a rough illustration, one billion raw weights take about 2 GB at 16 bits each, 1 GB at 8 bits, or 0.5 GB at 4 bits. This estimates parameter values only. Real model files also need metadata and scales; inference needs runtime memory, temporary buffers, and often a key-value (KV) cache for the current context.

“4-bit model” can describe several different choices.

Model cards and filenames do not always explain which components use which precision. Check whether the label describes weights, activations, cache, or the whole serving path. Also check the actual file format: group scales and other metadata mean the saved model is not always exactly the nominal bit width multiplied by its parameter count.

Quantize after training, or account for it during training.

SmoothQuant is one post-training example: it moves some quantization difficulty from activations to weights so both can use 8-bit precision. LLM.int8() keeps most values in 8-bit arithmetic while handling outlier dimensions at 16 bits. These papers show why quantization is a designed numerical method—not simply selecting “4-bit” in a menu.

For low-bit weight-only quantization, GPTQ is another post-training approach. Different methods make different assumptions about calibration data, model layers, and available kernels; compare results from the actual runtime you plan to use.

Smaller weights help only one part of the system.

Lower-bit weights can reduce the memory needed to load a model. Faster inference also depends on whether the processor can execute that format efficiently, whether the runtime has a good kernel, and whether conversion overhead erases the gain. A quantized file can load in less memory and still answer no faster—or even more slowly—on a particular device.

Activations, attention, token generation, and the KV cache contribute to runtime costs too. Quantizing only the weights does not automatically shrink those other components. A long context can still exceed the available memory even when the weight file fits; see how to estimate and manage KV-cache memory ↗.

The SmoothQuant authors measured up to 1.56× lower latency for OPT-13B and OPT-30B with their FasterTransformer setup on a single GPU, while cutting memory by nearly half. That is a result for their tested models, kernels, hardware, and workload—not a promise for every quantized model.

See what happens during inference ↗ and how to measure the full system cost ↗.

Rounding error can matter more in some layers and tasks.

A lower-precision model approximates values, so its outputs can diverge from the unquantized baseline. The effect depends on the model, quantization method, bit width, calibration examples, and evaluation task. Outliers or sensitive layers can make a uniform bit setting lose quality earlier than expected.

Test the behaviors that matter for the deployment: long and short inputs, uncommon names, multiple languages, exact numbers, structured output, and cases where the model should express uncertainty. Compare the quantized build with the original model and with other options, including distillation ↗. Both can reduce resource requirements, but they change the model in different ways.

LLM.int8() reports that its mixed-precision treatment preserved the measured performance of its evaluated models, while SmoothQuant’s experiments show that naive activation quantization could severely degrade some of their tested models. The evidence points to a practical rule: evaluate the exact converted model rather than infer quality from its bit-width label.

Measure the converted model in the place it will run.

1. Record the exact build. Note model version, quantization method, bit width, target components, group size, and conversion tool.
2. Check the source and rights. Directly quantizing weights usually requires access to model parameters. Verify the source, license, and provenance of any ready-made quantized file.
3. Compare task quality. Use representative examples, held-out data, edge cases, and the same acceptance criteria for each candidate.
4. Measure peak memory. Include model loading, runtime buffers, KV cache, context length, and concurrent requests—not just the file on disk.
5. Measure latency and throughput. Test the actual processor, runtime, batch size, and kernels. Include the first-token wait and sustained generation.
6. Recheck after changes. New kernels, drivers, model versions, cache formats, or prompt lengths can alter the trade-off.

For a compact-model comparison, continue with small language models and device limits ↗. For a model-selection framework, start with the task and its evaluation criteria ↗.

Make precision a measured design choice.

Quantization can help a model fit a memory or compute budget when the quality and runtime trade-offs hold for the intended use. Follow the connections to knowledge distillation ↗, open weights and model access ↗, and the complete AI field guide ↗.