Quantization is a technique that shrinks an LLM by storing each weight in fewer bits. That is how a 70B model needing roughly 140GB at full precision fits in about 40GB on a laptop. The trick is lossy: you trade a little accuracy for a 2x to 4x cut in memory, and the loss lands hardest on math and exact numeric work, not on general reasoning. One paper from Neural Magic and ISTA, evaluating the full Llama-3.1 family across more than 500,000 measurements, found 8-bit integer quantization costs only 1 to 3 percent accuracy and 4-bit stays competitive on reasoning tasks. That is the gap between a data-center GPU and the machine on your desk.
Memorize one line: params x bits / 8 = bytes. That is the whole game. A 70B model at 4 bits is 70,000,000,000 x 4 / 8 = 35GB of weights.
The one-line memory formula
That one equation explains every model download size. A 7B model in FP16 (16 bits each) needs about 14GB. The same 7B at 4 bits needs about 3.5GB. Multiply by the parameter count and you can predict whether a model loads on your machine before you download a single byte, whether you run it locally on a laptop or on consumer hardware with a single GPU.
- FP16 / BF16 (16 bits): the native training precision. 7B is ~14GB, 70B is ~140GB.
- INT8 (8 bits): roughly 2x smaller. 7B is ~7GB, 70B is ~70GB.
- INT4 (4 bits): roughly 4x smaller. 7B is ~3.5GB, 32B is ~16GB, 70B is ~35GB.
- Add 10 to 20 percent on top of the weight figure for the KV cache and runtime overhead, which grows with your context length.
That overhead is why the 70B at 4 bits lands near 40GB in practice rather than the bare 35GB the formula gives. The KV cache holds the attention state for every token in your prompt, and it scales with context window, so a long chat costs more RAM than a short one even on the same model.
Before downloading, run the formula in your head: params (in billions) x bits / 8 gives gigabytes. 13B at 4 bits is 13 x 4 / 8 = 6.5GB. If that exceeds your free RAM, pick a smaller model or a lower bit level.
The KV cache: the overhead the formula hides
The weight figure is fixed, but the KV cache grows with every token. Its size follows a second formula: 2 x layers x kv_heads x head_dim x sequence_length x bytes_per_value. The leading 2 covers the separate key and value tensors. For a model using grouped-query attention, the kv_heads count is far smaller than the total attention heads, which is the entire reason modern models stay tractable at long context.
Plug in Llama-3.1-8B, which uses 8 KV heads instead of 32 thanks to grouped-query attention. Each token in the cache costs roughly 0.1MB at FP16. A 32,000-token context then holds about 3.2GB of cache on top of the weights, and a 128,000-token context pushes that toward 13GB. The model file never changed, but your effective memory bill tripled because the conversation got long. This is why a chat that loaded fine at the start can swap or crash thirty messages in.
Two budgets, not one: weights are fixed, the KV cache grows with context. A long chat on a small model can cost more RAM than a short chat on a bigger one.
You can quantize the cache too. Storing keys and values at 8 bits instead of 16 halves that line of the budget, and most engines expose it as a flag. Q8 cache is close to lossless in practice; pushing the cache to 4 bits is where quality starts to wobble, so treat 8-bit cache as the safe lever and 4-bit cache as a last resort.
What actually gets dumber: quantization accuracy loss by task
Accuracy loss is not uniform across tasks. The Llama-3.1 trade-off study reports that FP8 is effectively lossless at every model scale, INT8 (W8A8) loses only 1 to 3 percent, and INT4 (W4A16) is more competitive than expected, rivaling 8-bit on many benchmarks. The catch shows up in precision-sensitive work.
The surprising part: a 4-bit model can lose up to nearly 70 percent of its accuracy on the hardest math, while general chat stays nearly untouched. The damage is concentrated, not spread.
Mathematical reasoning degrades first. A focused study on low-bit quantization and math found that aggressive 4-bit methods like AWQ and GPTQ can drop accuracy by up to 69.81 percent on the hardest multi-step numeric problems. The study notes that the failures show up early in the solution and then cascade, so a single wrong step ruins the final answer.
The gap is recoverable. Fine-tuning a 4-bit model on about 330 curated examples, a few minutes on one GPU, pulls math accuracy back toward the full-precision baseline. The fix is almost insultingly cheap, so 4-bit is a deployment default, not a dead end.
General reasoning survives 4-bit nearly intact. Arithmetic and exact recall are where the bits you threw away come back to bite you.
Why a few weights carry most of the loss
The reason accuracy holds up at 4 bits is structural, not luck. Inside a transformer, the weights that matter are not spread evenly. A tiny fraction of channels, on the order of 1 percent, carry outsized influence on the output, and these salient channels are visible in the activation distribution rather than in the weights themselves. Round those carelessly and the model breaks; protect them and the other 99 percent quantize cheaply.
This is exactly the lever the better algorithms pull. AWQ identifies salient channels from a small calibration set and scales them so rounding error lands on the weights that tolerate it, reporting that protecting only 1 percent of weights sharply cuts quantization error. K-quants in GGUF do a coarser version of the same idea, spending extra bits on sensitive layers. Naive round-to-nearest at 4 bits treats every weight equally, which is why it falls apart where the smarter methods hold.
The label soup, decoded: GGUF, Q4_K_M, GPTQ, AWQ
Every label below answers one of two questions: how many bits per weight, and which algorithm decided how to round. Here is the decoder ring.
Bit-level names
- INT8 / 8-bit: one byte per weight. The safe choice, near-baseline accuracy.
- INT4 / 4-bit: half a byte per weight. The popular default for running big models on small hardware.
- FP8: an 8-bit floating-point format, effectively lossless and increasingly common on newer GPUs.
- Sub-4-bit (2-bit, 3-bit): extreme compression with a noticeable quality drop, mostly for research or when nothing else fits.
GGUF and the Q4_K_M style codes
GGUF is the file format used by llama.cpp, the engine behind most laptop and CPU inference. In Q4_K_M, the Q4 means 4 bits, the K means the K-quant method (which spends extra bits on the most important weights), and the M means the medium-size variant balancing footprint against quality. The llama.cpp project lists Q4_K_M among its options with published benchmark numbers for Llama-3.1-8B, and it is the community default for laptop inference.
- S / M / L suffix (Q4_K_S, Q4_K_M, Q4_K_L): small, medium, large. Bigger keeps more quality at higher size.
- K-quants (Q2_K through Q6_K): mixed-precision, allocate more bits to sensitive layers. The mainstream choice.
- IQ-quants (IQ2_XXS through IQ4_NL): newer importance-aware quants that squeeze more quality out of very low bit counts.
- Q8_0: 8-bit, near-lossless, the safety net when you have the RAM to spare.
Algorithm names (NF4, GPTQ, AWQ)
NF4 (4-bit NormalFloat) comes from the QLoRA paper, which describes it as information-theoretically optimal for the normally distributed weights that neural networks tend to produce. QLoRA used NF4 to fine-tune a 65B model on a single 48GB GPU while preserving full 16-bit task performance, the result that legitimized 4-bit fine-tuning.
GPTQ and AWQ are calibration-based 4-bit methods: they run a small dataset through the model to decide how to round each weight, which buys higher accuracy at the cost of an upfront calibration step. Hugging Face notes AWQ often reaches high 4-bit accuracy and sometimes surpasses GPTQ, with shorter calibration. bitsandbytes (which provides NF4 and FP4) needs no calibration, making it the simplest path, especially for QLoRA fine-tuning.
A worked example: Llama-3.1-8B on a 16GB laptop
Walk one model through end to end. Llama-3.1-8B at FP16 is about 16GB of weights, which already fills a 16GB laptop before the OS gets a byte. Apply the formula at 4 bits: 8 x 4 / 8 gives 4GB. The actual Q4_K_M file ships at roughly 4.9GB because the K-quant method keeps some tensors at higher precision, and that file is what Ollama pulls when you run llama3.1:8b, since Q4_K_M is its default.
Now add the second budget. At a 32,000-token context the KV cache adds roughly 3GB on top, so the working total sits near 8GB, comfortable inside 16GB with room for the OS. The same FP16 model on the same machine would need offloading to disk and crawl. The reported quality cost of that 4-bit step is 1 to 3 percent on knowledge benchmarks, which is invisible for chat and drafting and only shows up if you lean on the model for exact arithmetic.
Pick by your RAM (run LLMs locally)
Your hardware decides the bit level. Find your available memory, subtract a few gigabytes for the OS, then read across. The table assumes you want the largest capable model that comfortably fits, with the bit level chosen to land inside that budget.
| Your RAM / VRAM | Best fit | Bit level | What you give up |
|---|---|---|---|
| 8GB | 7B to 8B model | INT4 (Q4_K_M / NF4) | Minor reasoning loss, weaker on hard math |
| 16GB | 7B at INT8 or 13B at INT4 | INT8 or INT4 | Near-baseline at INT8; some math drift at INT4 |
| 24GB | 13B at INT8 or ~32B at INT4 (~16GB) | INT8 or INT4 | Little at INT8; mild degradation on 32B INT4 |
| 48GB | 70B at INT4 | INT4 (Q4_K_M) | ~2 to 3 percent overall, more on exact arithmetic |
| 80GB+ | 70B at INT8 or FP8 | INT8 / FP8 | Effectively nothing measurable |
The rule of thumb: prefer a bigger model at lower precision over a smaller model at higher precision. A 70B at 4-bit usually beats a 13B at 8-bit on the same RAM budget.
Which method to actually choose
For most people running models locally, the decision collapses to three lanes, listed below. On Apple silicon, MLX provides native quantized inference and is a strong fit for the laptop case.
- Laptop / CPU: GGUF, start at Q4_K_M, step up to Q5_K_M or Q8_0 if you have the RAM and care about accuracy.
- NVIDIA GPU, zero setup: bitsandbytes NF4 (4-bit) or INT8, no calibration required.
- Accuracy-critical serving: AWQ or GPTQ with calibration, since it recovers quality that on-the-fly methods leave on the table.
- Newest GPUs, want lossless: FP8.
If a quantized model gives wrong answers on your task, do not assume the model is bad. Re-test one bit level up. Jumping from Q4 to Q5 or Q8 often restores the exact behavior you lost, at a known RAM cost you can calculate with the formula.
What to do when quality drops: a checklist
Before blaming the model, walk three cheap checks in order. Most quality complaints about local models trace back to one of these, not to a fundamental limit of quantization.
- Step the bit level up first. Q4 to Q5_K_M or Q8_0 is the single highest-impact move, and the formula tells you the exact RAM cost before you download.
- Prefer the K-quant or AWQ build over a plain round-to-nearest one at the same bit width. Same size on disk, but the importance-aware methods protect the salient channels that carry accuracy.
- If only math or code breaks, the problem is task-specific, not the whole model. Route exact arithmetic to a tool or a calculator and let the quantized model handle the language around it.
For Apple silicon and CPU laptops there is a fourth lever: quantize the KV cache to 8 bits when long contexts push you past free RAM. It buys back roughly half the cache budget at near-lossless quality, which is often the difference between a 32,000-token chat that fits and one that swaps.
Where MemX fits
Quantization shrinks the brain. MemX gives it your memory.
Quantization solves how to fit the model. It does nothing for what the model knows about you. A 4-bit Llama running on your laptop still starts every conversation blank, because the weights you compressed were trained on the public internet, not on your notes, screenshots, and documents. That is a separate layer.
MemX is an external AI memory layer: a personal RAG index over your own notes, screenshots, docs, and the bits of your life you want an assistant to recall. It pairs naturally with a quantized local model. The small model handles generation on cheap hardware, and MemX supplies the personal context the compressed weights never contained.
Sources
These quantization accuracy and memory figures are drawn from Neural Magic/ISTA, QLoRA, AWQ, llama.cpp, Hugging Face, and Ollama primary sources.
01How much RAM do I need to run a 70B model?
At 4-bit quantization, roughly 40GB, since 70B x 4 bits / 8 is 35GB of weights plus runtime overhead. At 8-bit you need about 70GB, and at full FP16 precision about 140GB. The 4-bit version is what makes a 70B model viable on a high-RAM laptop or single GPU.
02Does quantization make an LLM dumber?
Slightly, and unevenly. INT8 costs about 1 to 3 percent accuracy and INT4 stays competitive on general reasoning, but math and exact numeric tasks degrade more, in one study up to nearly 70 percent on the hardest problems at aggressive 4-bit (arXiv 2505.11574). For most chat and writing, the loss is hard to notice.
03Why does the KV cache use so much memory in a long chat?
The KV cache stores attention state for every token, so it grows with context length while the weights stay fixed. Llama-3.1-8B costs about 0.1MB per token, so a 128,000-token chat adds over 10GB on top of the model. Quantizing the cache to 8 bits roughly halves that.
04What does Q4_K_M mean in GGUF model names?
Q4 means 4 bits per weight, K means the K-quant method that spends extra bits on important weights, and M means the medium-size variant balancing footprint and quality. It is the common default for laptop inference in llama.cpp and Ollama, where llama3.1:8b ships as Q4_K_M.
05What is the difference between GPTQ, AWQ, and NF4?
GPTQ and AWQ are calibration-based 4-bit methods that run sample data through the model for higher accuracy; AWQ protects the roughly 1 percent of salient channels that matter most. NF4, from QLoRA via bitsandbytes, needs no calibration and is the simplest to use.
