AI Foundations

Quantization

By Arpit Tripathi, Founder

Quantization is the process of lowering the numerical precision of a model's weights and activations, for example from 16-bit floats to 8-bit or 4-bit integers, so the model uses far less memory and runs faster during inference with little loss in quality.

What Is Quantization?

Quantization reduces the numerical precision used to store a neural network's parameters and, sometimes, its activations. A large language model is normally trained in 16-bit floating point (FP16 or BF16), where each weight takes 2 bytes. Quantization maps those high-precision numbers onto a smaller set of values that fit in 8 bits (INT8) or even 4 bits (INT4), so each weight takes 1 byte or half a byte instead. The model gets smaller on disk, smaller in GPU memory, and usually faster at inference because there is less data to move.

The technique works because trained networks are remarkably tolerant of low precision. The exact value of any single weight rarely matters; what matters is the overall pattern across millions of them. Rounding each weight to a nearby value introduces small errors that tend to average out across a layer, so a well-quantized model keeps most of its accuracy. This tolerance is what makes it practical to run a model that was trained on a cluster on a single consumer GPU or a laptop.

Quantization is a compression and acceleration step applied after or during training. It does not change the model's architecture or what it knows; it changes how precisely each number is stored and computed.

  • Lowers precision of weights (and sometimes activations) from FP16 to INT8 or INT4.
  • Shrinks memory footprint and often speeds up inference by moving less data.
  • Relies on the fact that trained networks tolerate small per-weight rounding errors.

The Memory Math: Bytes Per Parameter

The memory a model needs for its weights follows a simple rule: bytes are roughly the parameter count multiplied by the bits per parameter, divided by eight. A 7-billion-parameter model at FP16 (16 bits each) needs about 14 GB just for weights, which exceeds many consumer GPUs. Quantize the same model to INT8 and it drops to about 7 GB; quantize to INT4 and it drops to about 3.5 GB, small enough to fit in the memory of a mid-range laptop GPU or even system RAM.

This is the central reason quantization matters in practice. The difference between 14 GB and 3.5 GB is the difference between needing a data-center card and running locally. The same scaling applies to bigger models: a 70B model is roughly 140 GB at FP16 but around 35 GB at INT4, which brings it within reach of a single high-memory GPU instead of several.

Actual runtime memory is a bit higher than the weight math suggests, because activations, the KV cache, and framework overhead all consume additional space. The weight estimate is still the dominant term and a reliable first approximation.

memory_bytes ≈ params × bits / 8 (7e9 × 4 / 8 = 3.5 GB at INT4)
Weight memory scales linearly with bit-width. Halving the bits roughly halves the memory.
  • Weight memory in bytes is approximately params times bits divided by 8.
  • A 7B model: ~14 GB at FP16, ~7 GB at INT8, ~3.5 GB at INT4.
  • INT4 quantization is what lets large models run on consumer GPUs and laptops.
  • Activations and the KV cache add memory on top of the weight estimate.

How the Mapping Works: Scale and Zero-Point

Most integer quantization uses an affine mapping defined by two numbers per group of weights: a scale and a zero-point. The scale sets how much real-valued range each integer step covers, and the zero-point is the integer that represents the real value zero. To quantize a value x, you divide by the scale, round to the nearest integer, and add the zero-point. To recover an approximate original, you reverse the steps: subtract the zero-point and multiply by the scale.

Because a single scale across an entire tensor would be too coarse, real implementations quantize in small groups (often 64 or 128 weights, or 256-weight super-blocks in k-quant formats) so the scale adapts to local magnitude. Finer grouping preserves more accuracy at the cost of storing more scale and zero-point values. Some methods are symmetric, fixing the zero-point at zero, while others are asymmetric and learn it per group.

The rounding step is where information is lost. The art of a good quantization method is choosing scales, grouping, and which weights to protect so that the rounding error damages the model's outputs as little as possible.

x_q = round(x / s) + z x ≈ s × (x_q − z)
Affine quantization maps a real value x to an integer x_q using scale s and zero-point z, then dequantizes by inverting the mapping.
  • An affine quant uses a scale s and an integer zero-point z per weight group.
  • Quantize by dividing by s, rounding, and adding z; dequantize by reversing.
  • Grouping (per-channel or per-block) keeps scales local and preserves accuracy.

PTQ vs QAT and the Accuracy Tradeoff

There are two broad approaches. Post-training quantization (PTQ) takes an already-trained model and quantizes it directly, usually with a small calibration set to estimate good scales. PTQ is fast, needs no retraining, and is how most open LLM checkpoints are quantized today. Quantization-aware training (QAT) instead simulates the rounding during training or fine-tuning, so the model learns weights that survive quantization. QAT generally recovers more accuracy at very low bit-widths but costs a full training run.

The tradeoff between precision and quality is gradual at first, then sharp. Going from FP16 to INT8 typically costs almost nothing measurable. Modern 4-bit weight-only methods such as GPTQ and AWQ keep quality close to the original on most tasks, which is why INT4 has become a common deployment target. Push below 4 bits, to 3-bit or 2-bit, and degradation usually becomes noticeable, especially on reasoning-heavy or long-context tasks, unless QAT or careful per-group handling is used.

Where INT4 starts to hurt depends on the model and the method. Smaller models and harder tasks feel quantization error sooner; large models with good calibration often show only minor differences at 4 bits. Measuring on a task-specific evaluation set, rather than trusting a single perplexity number, is the reliable way to decide.

  • PTQ quantizes a finished model with calibration data; fast and the common default.
  • QAT simulates rounding during training; more accurate at very low bit-widths but costly.
  • FP16 to INT8 is nearly lossless; INT4 is usually close with good methods.
  • Below 4 bits (3-bit, 2-bit) quality often drops noticeably without QAT.

Several methods have become standard. GPTQ (Frantar et al., 2022) is a one-shot PTQ method that quantizes weights to 3 or 4 bits using approximate second-order information, with negligible accuracy loss on large models. AWQ (Lin et al., 2023) is activation-aware: it identifies the small fraction of salient weight channels that matter most and protects them, improving low-bit accuracy and earning the MLSys 2024 best paper award. Both are weight-only methods aimed at fast, accurate 4-bit inference.

For 4-bit work in PyTorch, bitsandbytes provides the NF4 (NormalFloat4) data type popularized by QLoRA (Dettmers et al., 2023), which fine-tunes a 4-bit base model with small adapters and reached a 65B model on a single 48GB GPU. NF4 is the format behind Hugging Face's load_in_4bit option. On the CPU and Apple Silicon side, llama.cpp uses the GGUF file format with k-quant schemes such as Q4_K_M, which group 256 weights into super-blocks and quantize the sub-block scales themselves for a good size-to-quality balance.

These formats are not interchangeable: a GGUF k-quant file runs in llama.cpp, while a GPTQ or AWQ checkpoint runs in GPU inference stacks. Choosing one is mostly about your runtime and hardware rather than a quality ranking.

python
import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",          # NormalFloat4 from the QLoRA paper
    bnb_4bit_use_double_quant=True,     # quantize the quant constants too
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=bnb_config,     # ~3.5 GB of weights instead of ~14 GB
    device_map="auto",
)
Loading a model in 4-bit NF4 with bitsandbytes via Hugging Face Transformers.
  • GPTQ: one-shot second-order PTQ to 3 or 4 bits, weight-only, GPU inference.
  • AWQ: activation-aware weight quantization that protects salient channels.
  • bitsandbytes NF4 / QLoRA: 4-bit base for fine-tuning, behind load_in_4bit.
  • GGUF k-quants (Q4_K_M) in llama.cpp: super-block format for CPU and Apple Silicon.

Quantization, QLoRA, and the KV Cache

Quantization connects to two neighboring ideas. QLoRA combines a 4-bit quantized frozen base model with small LoRA adapters trained in higher precision, so most of the memory saving comes from quantizing the base while the adapters keep fine-tuning stable. This is why quantization and parameter-efficient fine-tuning are usually discussed together: quantizing the base is what makes single-GPU fine-tuning of large models feasible.

The second connection is KV-cache quantization. The weights are only part of inference memory; during generation the model stores the keys and values of every past token in a KV cache that grows with context length. Quantizing that cache to 8-bit or 4-bit is a separate lever that lets a model handle longer contexts or larger batches without running out of memory. Weight quantization and KV-cache quantization are complementary and are often applied together.

For a personal AI-memory product, quantization is what keeps inference affordable: a smaller, quantized model serves more requests per GPU, and KV-cache quantization stretches how much retrieved context fits in a single call. The model's knowledge of a user's notes still lives in retrieval, not in the weights, so quantization trims serving cost without touching what the system remembers.

  • QLoRA fine-tunes a 4-bit quantized base with higher-precision LoRA adapters.
  • KV-cache quantization compresses stored keys and values to fit longer contexts.
  • Weight and KV-cache quantization are complementary memory levers.
  • Quantization lowers serving cost without changing what a retrieval system remembers.

Key takeaways

  • Quantization lowers weight and activation precision (FP16 to INT8 or INT4) to cut memory and speed up inference, while keeping most of the model's quality.
  • Weight memory is approximately params times bits over 8, so a 7B model needs ~14 GB at FP16, ~7 GB at INT8, and ~3.5 GB at INT4, which is what puts big models on laptops.
  • Integer quantization uses an affine map x_q = round(x / s) + z with a per-group scale s and zero-point z, applied in small blocks to keep accuracy.
  • PTQ quantizes a finished model quickly with calibration data; QAT simulates rounding during training for better very-low-bit accuracy at higher cost.
  • FP16 to INT8 is nearly lossless and INT4 stays close with GPTQ or AWQ, but quality usually degrades below 4 bits.
  • GPTQ, AWQ, bitsandbytes NF4 (QLoRA), and GGUF k-quants are the common methods; the right one depends on your runtime and hardware.

Frequently asked questions

Quantization is reducing the numerical precision used to store a model's weights and activations, for example converting 16-bit floats to 8-bit or 4-bit integers. This makes the model much smaller in memory and often faster at inference. It works because trained neural networks tolerate small per-weight rounding errors without losing much accuracy.
Weight memory is roughly the parameter count times bits per weight divided by eight. A 7B model needs about 14 GB at FP16, about 7 GB at INT8, and about 3.5 GB at INT4. That INT4 figure is small enough to run on a mid-range laptop or consumer GPU, which is the main reason people quantize.
Post-training quantization (PTQ) takes an already-trained model and quantizes it directly, usually with a small calibration set, so it is fast and needs no retraining. Quantization-aware training (QAT) simulates the rounding during training so the model learns weights that survive quantization. QAT generally recovers more accuracy at very low bit-widths but requires a full training run.
It can, but the loss is small with good methods. Going from FP16 to INT8 is usually almost lossless, and modern 4-bit methods like GPTQ and AWQ keep quality close to the original on most tasks. Quality typically drops more noticeably below 4 bits, such as 3-bit or 2-bit, especially on reasoning-heavy tasks.
GPTQ and AWQ are weight-only post-training quantization methods for fast 4-bit GPU inference; AWQ protects the most important weight channels using activation statistics. GGUF is the file format used by llama.cpp, with k-quant schemes like Q4_K_M that run efficiently on CPUs and Apple Silicon. They target different runtimes, so the choice depends on your hardware.
QLoRA fine-tunes a large model by loading its frozen base in 4-bit precision and training small LoRA adapters on top in higher precision. The quantization of the base is what cuts memory enough to fine-tune a 65B model on a single 48GB GPU. So quantization is the foundation that makes QLoRA's single-GPU fine-tuning possible.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free