TL;DR — Two techniques get bundled together and they are not the same thing. Quantization stores the same model's weights in fewer bits — no training run, minutes to apply, and it is what puts a 7B model on a consumer GPU: 28 GB at fp32 becomes 3.5 GB at 4-bit. Distillation trains a different, smaller model to imitate a big one — a real training run, days of work, and a different architecture at the end. The measurement that matters most below is about outliers: quantizing 4-bit with one scale for the whole tensor gives 11.8× the error of quantizing in groups of 64, because two large weights stretch the scale until every ordinary weight loses precision. That single fact is why every real 4-bit method is group-wise. Start with quantization — it needs no training, it is reversible, and you can measure the quality cost in an afternoon. It stops being the answer when the model was never capable of the task; compression makes a model cheaper, never smarter.
1. Simple explanation
A model is a very large pile of numbers. Serving it means holding those numbers in memory and multiplying by them constantly, so the pile's size determines what hardware you need, how much it costs, and how fast it answers.
There are two ways to make the pile smaller.
Store each number less precisely. A weight held as a 32-bit float can be held as an 8-bit integer instead — a quarter of the memory, and a slightly different number. The model's structure is untouched; only the precision of its entries changed.
Build a smaller model and teach it to behave like the big one. Not the same model at lower precision — a genuinely different, smaller architecture, trained on the big model's outputs rather than on raw labels. Its knowledge is an imitation, and it required a training run to acquire.
Analogy — a photograph. Saving a photo as a smaller JPEG keeps every part of the image and represents each pixel with less fidelity: instant, reversible if you kept the original, and the loss is spread thinly everywhere. That is quantization. Redrawing the scene as a simplified illustration is a different job entirely — it takes an artist and time, and the result is a new artefact that resembles the original rather than a degraded copy of it. That is distillation. The analogy also carries the outlier problem: compress an image with one setting when it contains a single very bright spot, and the whole picture dulls to accommodate it.
2. Diagram
QUANTIZATION — same model, fewer bits per weight
precision bytes/weight 7B model error vs fp32
fp32 (baseline) 4 28.0 GB 0.00000 1x
int16 2 14.0 GB 0.00000 2x
int8 1 7.0 GB 0.00040 4x
int4 0.5 3.5 GB 0.00685 8x
^
28 GB does not fit a consumer GPU.
3.5 GB does. That is the entire point.
THE OUTLIER PROBLEM — why real 4-bit is group-wise
weights: · · · · · · · ·█· · · · · · · · · · <- one large outlier
|<------- one scale stretched to cover it ------->|
every ordinary weight now lands in
the same couple of levels
strategy error
one scale for everything 0.08071 <- naive
groups of 256 0.01096
groups of 64 0.00685 <- 11.8x better
groups of 16 0.00428
TWO MECHANISMS, ROUTINELY CONFUSED
QUANTIZATION DISTILLATION
what changes the NUMBERS the ARCHITECTURE
model afterwards same shape, smaller a different, smaller model
needs training? no (post-training) YES -- a full training run
time to apply minutes to hours days
^ ^
try this first only when quantization
(reversible, measurable) is not enough
3. How it works
3.1 Quantization: the same numbers, held coarsely
Take the range of a group of weights, chop it into 2^bits evenly spaced levels, and store each weight as the index of its nearest level plus the scale needed to get back:
q = round( (w - lo) / scale ) store this small integer
w_hat = q * scale + lo reconstruct at inference
Everything follows from that. int8 gives 255 levels, int4 gives 15. Memory falls by exactly the ratio of bit widths — 4× for int8, 8× for int4 — and the reconstruction error rises.
Two vocabulary points worth having straight. Post-training quantization applies this to an already-trained model, needing only a small calibration set to pick sensible ranges; that is what most people mean. Quantization-aware training simulates the rounding during training so the model adapts to it, giving better results at very low bit widths and requiring a training run.
And fp16 is a floating-point format, not integer quantization — the memory arithmetic is the same but the error behaviour differs, because floats keep relative precision across magnitudes where integers do not.
3.2 The outlier problem, which is the whole engineering story
The measurement in §4 is the one to remember: quantizing to 4 bits with a single scale for the whole tensor produced 11.8× the error of doing it in groups of 64.
The reason is arithmetic. A neural network layer's weights are mostly small, with a few large outliers. One scale must span the whole range, so a single weight of 2.4 among thousands near ±0.08 forces the scale wide — and then every ordinary weight rounds into the same handful of levels. The outliers don't just quantize badly themselves; they destroy the precision of everything else.
Grouping fixes it by giving each block of weights its own scale, so an outlier only degrades its own group. Smaller groups mean better accuracy and slightly more overhead, since each group stores its own scale. Groups of 64 or 128 are the common choice, and this is why real 4-bit methods are group-wise or block-wise rather than naive.
3.3 What quantization buys, beyond memory
Memory is the headline, and there are two more:
Bandwidth. Inference is usually memory-bound rather than compute-bound — the bottleneck is moving weights, not multiplying them (see Linear Algebra §3.2). Quartering the bytes quarters the traffic, so quantized models are frequently faster as well as smaller.
Hardware access. The step that matters commercially is fitting a model onto cheaper hardware at all. 28 GB versus 3.5 GB is not a 8× cost saving — it is the difference between needing a datacentre GPU and running on a laptop.
The cost is accuracy, and it is usually small at int8 and noticeable at int4 — but "usually" is not a measurement. Score it on your own task.
3.4 Distillation: a different model that imitates
A large teacher produces outputs for a body of inputs; a smaller student is trained to match them. The insight is that the teacher's full probability distribution carries more information than a hard label — knowing an image is 70% cat, 25% dog, 5% fox teaches more than "cat".
Distillation genuinely changes what you are running: a different architecture, fewer layers, narrower, its own inference profile. It can approach teacher quality on a narrow task while being far smaller, which is the strongest version of the argument for it.
The costs are real: a full training run, a large set of inputs to transfer on, days rather than hours, and no easy reversal. Which is why §3.6 says try quantization first.
3.5 Pruning, briefly
The third technique: remove weights entirely, usually the ones nearest zero. Unstructured pruning zeroes individual weights and gives excellent theoretical sparsity that most hardware cannot exploit — a sparse matrix with scattered zeros runs at dense speed unless the kernel and hardware support the pattern. Structured pruning removes whole channels or heads, which is less flexible and produces speedups you actually observe.
That gap between theoretical and realised speedup is why pruning is less used in practice than its research presence suggests. Verify on your hardware rather than trusting a sparsity percentage.
3.6 The order to try things, and where compression stops helping
- Quantize to int8. Minutes, no training, usually near-lossless. Measure.
- Quantize to int4, group-wise. Bigger win, more quality risk. Measure.
- Use a smaller off-the-shelf model. Often better than distilling your own, and free.
- Distil, if a smaller existing model isn't good enough and the task is narrow.
- Prune, if you have verified your serving stack realises the speedup.
Where it stops: compression makes a model cheaper, never smarter. If the full model can't do the task, no amount of shrinking helps. It also does nothing for the knowledge problem — a compressed model has the same stale training data and the same inability to cite a source, so it is orthogonal to What RAG Is and When to Use It. And none of this addresses the sequence-length term in attention, which is a separate quadratic entirely.
4. The math
4.1 Quantization
levels = 2^bits - 1
scale = (max(group) - min(group)) / levels
quantize: q = round( (w - lo) / scale )
reconstruct: w_hat = q * scale + lo
memory ratio = bits_original / bits_target fp32 -> int4 = 8x
The group is the unit the scale is computed over. One group per tensor is naive; groups of 64–128 is standard.
4.2 Worked example
Four thousand weights drawn from a narrow distribution, plus two deliberate outliers (2.4 and −1.9) — the shape real layers have.
precision bytes/weight 7B model error vs fp32
fp32 (baseline) 4 28.0 GB 0.00000 1x smaller
int16 2 14.0 GB 0.00000 2x smaller
int8 1 7.0 GB 0.00040 4x smaller
int4 0.5 3.5 GB 0.00685 8x smaller
int8 error is 0.0004 against weights of typical size 0.08 — half a percent, and in practice usually invisible on task metrics. int4 is 17× worse than int8 and still small in absolute terms, which is why 4-bit is viable at all.
The hardware line is the point: 28 GB does not fit a consumer GPU; 3.5 GB does.
4.3 The outlier result
strategy error
one scale for everything 0.08071
groups of 256 0.01096
groups of 64 0.00685
groups of 16 0.00428
-> grouping cuts 4-bit error 11.8x
Read the first row against the third. Same bit width, same weights, 11.8× the error — caused entirely by two outliers out of four thousand values. With one global scale the error (0.0807) is roughly the size of a typical weight (0.08), meaning the ordinary weights are essentially destroyed.
Note also the diminishing returns: 256 → 64 helps a lot, 64 → 16 much less, while the scale-storage overhead keeps rising. That is why the common choice sits at 64 or 128 rather than as small as possible.
4.4 The two mechanisms
QUANTIZATION DISTILLATION
what changes the NUMBERS the ARCHITECTURE
model afterwards same shape, smaller a different, smaller model
needs training? no (post-training) YES -- a full training run
needs training data? a small calibration set a large transfer set
typical size cut 4x (int8) to 8x (int4) 2x to 10x, by design
typical quality cost small, measurable varies; can match on a narrow task
time to apply minutes to hours days
They compose: distil to a smaller architecture, then quantize the result.
5. Real code
"""Quantization shrinks the SAME model. Distillation trains a DIFFERENT one."""
import random
random.seed(4)
# A layer's weights. Real ones are millions of numbers with this shape of spread:
# mostly small, a few large outliers. The outliers are the whole problem.
WEIGHTS = [random.gauss(0, 0.08) for _ in range(4000)]
WEIGHTS[7] = 2.4 # one outlier, as real layers have
WEIGHTS[913] = -1.9
def quantize(w, bits, per_group=None):
"""Map floats onto 2^bits integer levels, then map back.
per_group=None quantises the whole tensor with one scale (naive);
a group size quantises each block separately (what real kernels do)."""
levels = 2 ** bits - 1
out = []
groups = [w] if per_group is None else [w[i:i + per_group]
for i in range(0, len(w), per_group)]
for g in groups:
lo, hi = min(g), max(g)
scale = (hi - lo) / levels if hi > lo else 1.0
for x in g:
q = round((x - lo) / scale) # to an integer level
out.append(q * scale + lo) # and back to a float
return out
def err(a, b):
"""Mean absolute reconstruction error."""
return sum(abs(x - y) for x, y in zip(a, b)) / len(a)
BYTES = {32: 4, 16: 2, 8: 1, 4: 0.5}
# NB: this quantises to INTEGER levels. fp16 is a float format, not int16 -- the
# memory arithmetic is identical but the error behaviour is not. Labels say int.
LABEL = {32: "fp32 (baseline)", 16: "int16", 8: "int8", 4: "int4"}
print("QUANTIZATION -- same model, fewer bits per weight")
print(f" {'precision':>15} {'bytes/weight':>13} {'7B model':>10} {'error':>10} {'vs fp32':>9}")
base = None
for bits in (32, 16, 8, 4):
q = WEIGHTS if bits == 32 else quantize(WEIGHTS, bits, per_group=64)
e = err(WEIGHTS, q)
if base is None:
base = 4
size_gb = 7e9 * BYTES[bits] / 1e9
print(f" {LABEL[bits]:>15} {BYTES[bits]:>13} {f'{size_gb:.1f} GB':>10} "
f"{e:>10.5f} {f'{BYTES[32]/BYTES[bits]:.0f}x smaller':>9}")
print("\n A 7B model at fp32 needs 28 GB and will not fit on one consumer GPU.")
print(" At 4-bit it is 3.5 GB and will. That is the entire point.")
print("\nWHY GROUPING MATTERS -- the outliers are the whole problem")
print(f" {'strategy':>26} {'error':>10}")
for label, group in [("one scale for everything", None),
("groups of 256", 256),
("groups of 64", 64),
("groups of 16", 16)]:
e = err(WEIGHTS, quantize(WEIGHTS, 4, per_group=group))
print(f" {label:>26} {e:>10.5f}")
naive = err(WEIGHTS, quantize(WEIGHTS, 4, per_group=None))
grouped = err(WEIGHTS, quantize(WEIGHTS, 4, per_group=64))
print(f" -> grouping cuts 4-bit error {naive/grouped:.1f}x. Two outliers stretch")
print(" a single global scale so far that every ordinary weight loses precision.")
print(" This is why real 4-bit methods are group-wise, not naive.")
print("\nQUANTIZATION vs DISTILLATION -- different mechanisms, often confused")
rows = [
("what changes", "the NUMBERS", "the ARCHITECTURE"),
("model afterwards", "same shape, smaller", "a different, smaller model"),
("needs training?", "no (post-training)", "YES -- a full training run"),
("needs training data?", "a small calibration set", "a large transfer set"),
("typical size cut", "4x (int8) to 8x (int4)", "2x to 10x, by design"),
("typical quality cost", "small, measurable", "varies; can match on a narrow task"),
("time to apply", "minutes to hours", "days"),
]
print(f" {'':<22} {'QUANTIZATION':<26} {'DISTILLATION'}")
for r in rows:
print(f" {r[0]:<22} {r[1]:<26} {r[2]}")
print("\n They COMPOSE: distil to a smaller architecture, then quantise it.")
print(" And note quantization is usually the one to try first -- no training run,")
print(" reversible, and you can measure the quality cost in an afternoon.")
# More bits is always less error.
errs = [err(WEIGHTS, quantize(WEIGHTS, b, per_group=64)) for b in (16, 8, 4)]
assert errs[0] < errs[1] < errs[2], errs
# Grouping beats a single global scale, because of the outliers.
assert grouped < naive
# 4-bit really is 8x smaller than fp32.
assert BYTES[32] / BYTES[4] == 8
# And a 7B model crosses the consumer-GPU line between fp32 and 4-bit.
assert 7e9 * BYTES[32] / 1e9 > 24 and 7e9 * BYTES[4] / 1e9 < 8
print("\nall assertions passed")
# Output:
# QUANTIZATION -- same model, fewer bits per weight
# precision bytes/weight 7B model error vs fp32
# fp32 (baseline) 4 28.0 GB 0.00000 1x smaller
# int16 2 14.0 GB 0.00000 2x smaller
# int8 1 7.0 GB 0.00040 4x smaller
# int4 0.5 3.5 GB 0.00685 8x smaller
#
# A 7B model at fp32 needs 28 GB and will not fit on one consumer GPU.
# At 4-bit it is 3.5 GB and will. That is the entire point.
#
# WHY GROUPING MATTERS -- the outliers are the whole problem
# strategy error
# one scale for everything 0.08071
# groups of 256 0.01096
# groups of 64 0.00685
# groups of 16 0.00428
# -> grouping cuts 4-bit error 11.8x. Two outliers stretch
# a single global scale so far that every ordinary weight loses precision.
# This is why real 4-bit methods are group-wise, not naive.
#
# QUANTIZATION vs DISTILLATION -- different mechanisms, often confused
# QUANTIZATION DISTILLATION
# what changes the NUMBERS the ARCHITECTURE
# model afterwards same shape, smaller a different, smaller model
# needs training? no (post-training) YES -- a full training run
# needs training data? a small calibration set a large transfer set
# typical size cut 4x (int8) to 8x (int4) 2x to 10x, by design
# typical quality cost small, measurable varies; can match on a narrow task
# time to apply minutes to hours days
#
# They COMPOSE: distil to a smaller architecture, then quantise it.
# And note quantization is usually the one to try first -- no training run,
# reversible, and you can measure the quality cost in an afternoon.
#
# all assertions passed
Reconstruction error is a proxy for what you actually care about, which is task accuracy — a layer can tolerate surprising amounts of weight error and still behave, and occasionally a small error in the wrong place matters enormously. Use this to build intuition; measure the real thing on your own evaluation set.
6. Real-world example
A team needed to serve a 13B model and it did not fit their GPU at fp16. They quantized to 4-bit with a library default, saw the model load, ran a handful of prompts that looked fine, and shipped it.
Quality degraded in a pattern nobody predicted: general conversation was unaffected, and anything involving numbers — dates, quantities, arithmetic in a sentence — got noticeably worse.
The cause was §3.2. Their configuration used a large group size, close to per-tensor scaling. Layers whose weight distributions had strong outliers lost most of their precision, and those layers turned out to matter disproportionately for the numeric behaviour. The average reconstruction error across the model looked acceptable; the damage was concentrated.
Two things had hidden it. Their smoke test was a dozen conversational prompts, which is exactly the capability least affected. And they had checked that the model loaded and spoke, which is not a quality measurement — a badly quantized model produces fluent output, it just produces slightly wrong fluent output.
They fixed it by dropping to a group size of 64 and, for the two most sensitive layers, keeping int8 rather than int4 — a mixed-precision setup that cost a little memory and recovered the behaviour.
The general lesson: average error is the wrong statistic for compression. Damage concentrates in whatever the outlier distribution punishes, and it shows up as a capability disappearing rather than as everything getting slightly worse. Evaluate per capability, on a set that includes the things you would notice losing.
7. Interview questions companies actually ask
Q1. What's the difference between quantization and distillation? Quantization stores the same model's weights in fewer bits — the architecture is unchanged, there's no training run, and it takes minutes. Distillation trains a genuinely different, smaller model to imitate a larger one's outputs — a full training run, a large transfer set, days of work. They compose, and quantization is almost always the one to try first because it's reversible and cheap to evaluate.
Q2. How does quantization actually work? Take a group of weights, divide their range into 2^bits evenly spaced levels, store each weight as the nearest level's index plus the scale to reconstruct. int8 gives 255 levels, int4 gives 15. Memory drops by exactly the bit-width ratio and reconstruction error rises. Post-training quantization does this to a finished model; quantization-aware training simulates the rounding during training so the model adapts.
Q3. Why do outliers matter so much? Because one scale has to span the whole group's range. A single weight of 2.4 among thousands near 0.08 stretches the scale until every ordinary weight rounds into a handful of levels — measured, that's 11.8× the error of group-wise quantization. Outliers don't just quantize badly themselves, they destroy the precision of everything sharing their scale. Hence group-wise or block-wise methods.
Q4. Does quantization make inference faster or just smaller? Usually both, and the reason is that inference is typically memory-bandwidth bound rather than compute bound — the bottleneck is moving weights, not multiplying them. Quartering the bytes quarters the traffic. The bigger commercial effect is often hardware access: 28 GB versus 3.5 GB is the difference between needing a datacentre GPU and running on a laptop.
Q5. Why isn't pruning used more, given the research on it? Because unstructured pruning's theoretical sparsity mostly isn't realised — a matrix with scattered zeros runs at dense speed unless the kernel and hardware support the specific pattern. Structured pruning, removing whole channels or heads, gives speedups you actually observe but is less flexible. Always verify on your hardware rather than trusting a sparsity percentage.
Q6. You quantized and quality dropped. How do you diagnose it? Don't look at average reconstruction error — damage concentrates rather than spreading. Evaluate per capability on a set that includes what you'd notice losing, since a badly quantized model still produces fluent output. Then check group size first, and consider mixed precision, keeping the most sensitive layers at higher bit width.
Q7. When is compression the wrong tool? When the full model couldn't do the task anyway — compression makes a model cheaper, never smarter. It also does nothing for stale or missing knowledge, which is retrieval's job, and nothing for the sequence-length term in attention. And if a smaller off-the-shelf model already meets the bar, using it is better than distilling your own and free.
8. When to use / tradeoffs
Reach for quantization when:
- The model doesn't fit your hardware, or barely does
- Inference cost or latency is the binding constraint
- You want a result today with no training run
- You have an evaluation set to measure the quality cost
Reach for distillation when:
- Quantization isn't enough and the task is narrow
- No smaller off-the-shelf model meets the bar
- Volume justifies a training run and its maintenance
Reach for neither when:
- The full model can't do the task — compression won't fix capability
- The problem is missing facts → retrieval
- A smaller existing model already works → just use it
| Situation | Why it breaks | Do this instead |
|---|---|---|
| Naive per-tensor 4-bit | Outliers destroy every other weight | Group-wise, 64 or 128 |
| Smoke-tested with chat prompts | Damage concentrates in specific capabilities | Evaluate per capability |
| Judging by average weight error | Wrong statistic; hides concentration | Task metrics on a scored set |
| Trusting a sparsity percentage | Unstructured sparsity rarely realised | Measure on your hardware |
| Distilling before quantizing | Days of work before trying the cheap thing | int8, then int4, then consider distilling |
| Compressing to fix poor quality | Compression never adds capability | Bigger model, or retrieval |
| Assuming int4 is always fine | Sensitive layers exist | Mixed precision on the worst layers |
Honest limits. The quantize function is symmetric-range integer quantization with a per-group min/max, which is the simplest real scheme; production kernels add zero-point handling, per-channel scales, outlier-aware methods that keep a few weights at high precision, and formats designed around specific hardware. Reconstruction error is a proxy and the §6 story is precisely about how it misleads — a layer can absorb large weight error harmlessly, or fail badly on a small one. The weights here are drawn from a Gaussian with two hand-placed outliers, which reproduces the shape of the problem and not the true distribution of any real layer, where outliers cluster in particular channels. The "7B model" sizes count weights only and ignore activations, KV cache and framework overhead, all of which matter for whether something actually fits. And the article says nothing about the accuracy loss in task terms, because that is entirely model- and task-dependent and anyone quoting a universal figure is guessing.
9. Summary + related articles
- Quantization = same model, fewer bits per weight. No training, minutes, reversible.
- Distillation = a different, smaller model trained to imitate a bigger one. Full training run, days.
- Measured: 28 GB at fp32 → 3.5 GB at int4, which is the difference between a datacentre GPU and a laptop.
- The outlier result: one global scale gives 11.8× the error of groups of 64. Two outliers in four thousand weights destroy the precision of all the rest.
- That is why every real 4-bit method is group-wise. Groups of 64–128 are standard; smaller helps less and costs scale storage.
- Quantization often makes inference faster too, because inference is memory-bandwidth bound.
- Pruning's theoretical sparsity mostly isn't realised on real hardware unless it's structured.
- Order: int8 → int4 group-wise → a smaller off-the-shelf model → distil → prune.
- Average error is the wrong statistic. Damage concentrates, and shows up as a capability disappearing.
- Compression makes a model cheaper, never smarter, and does nothing for missing knowledge.
Related:
- Linear Algebra — the matmuls and memory traffic compression is reducing
- Transformers Deep Dive — the separate quadratic term compression does not touch
- When to Fine-Tune — making a small model do one job, the adjacent decision
- Model Routing Patterns — using a smaller model per request instead of shrinking one
- Cost-Latency Tradeoffs — where serving cost sits among the levers
- ML Inference Systems — batching, bandwidth and serving architecture
- Model Evaluation — the scored set §6 says you need
- What RAG Is and When to Use It — for the knowledge problem compression cannot touch
Resources
- Dettmers et al. (2022) — LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, arXiv:2208.07339 — the paper that identified the outlier problem in §3.2 and handled it: https://arxiv.org/abs/2208.07339
- Dettmers et al. (2023) — QLoRA: Efficient Finetuning of Quantized LLMs, arXiv:2305.14314 — 4-bit quantization plus fine-tuning, including the block-wise scheme: https://arxiv.org/abs/2305.14314
- Frantar et al. (2022) — GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, arXiv:2210.17323 — the widely used post-training method: https://arxiv.org/abs/2210.17323
- Lin et al. (2023) — AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, arXiv:2306.00978 — protecting the weights that matter most: https://arxiv.org/abs/2306.00978
- Hinton, Vinyals & Dean (2015) — Distilling the Knowledge in a Neural Network, arXiv:1503.02531 — the original distillation paper and the soft-target argument: https://arxiv.org/abs/1503.02531
- Sanh et al. (2019) — DistilBERT, a distilled version of BERT, arXiv:1910.01108 — distillation applied at scale, with the size/quality numbers: https://arxiv.org/abs/1910.01108
- Frankle & Carbin (2018) — The Lottery Ticket Hypothesis, arXiv:1803.03635 — the pruning result, and worth reading alongside §3.5's caveat about realised speedup: https://arxiv.org/abs/1803.03635