What Is LLM Quantization? GGUF, Q4_K_M and Q8_0 Explained

Quantization stores a model's weights with fewer bits each, for example 4 or 8 instead of 16, so the model takes less memory and disk space at the cost of some accuracy. GGUF is the single-file format used by llama.cpp, Ollama and LM Studio, and names such as Q8_0 and Q4_K_M tell you how many bits per weight the file uses and how they are packed.

A 16-bit model with 8 billion parameters is about 16 GB. The same model at Q4_K_M is about 4.9 GB, small enough for an 8 GB graphics card. That size difference is why almost everyone running models locally uses quantized files.

What quantization does

A model's weights are numbers, normally stored as 16-bit floating point (FP16 or BF16). Quantization maps each weight to a small integer, for example 0 to 15 for 4 bits, and stores a scale factor so the original value can be approximately reconstructed. The Hugging Face GGUF table gives the reconstruction formulas: for Q8_0 it is w = q * block_scale, and for Q4_K it is w = q * block_scale + block_min.

Weights are grouped into blocks, and each block gets its own scale. Small blocks follow the local range of values more closely, which keeps errors down, but each scale takes space. That overhead is why "8-bit" Q8_0 costs 8.5 bits per weight rather than 8: llama.cpp's block_q8_0 stores 32 one-byte values plus one 2-byte scale, so 34 bytes ÷ 32 weights × 8 = 8.5 bits.

The model still does its arithmetic at higher precision. Hugging Face's Transformers guide describes the general process: weights are stored quantized and dequantized on the fly for each matrix multiplication. It also notes why this works well for text generation: what matters is that the probability ranking of the next token stays roughly the same, not that every value is exact.

What GGUF is

The ggml specification describes GGUF as "a file format for storing models for inference with GGML and executors based on GGML". It succeeded the older GGML, GGMF and GGJT formats. Its stated design goals include single-file deployment, mmap compatibility (the file can be memory-mapped for fast loading) and "full information": everything needed to load the model is in one file.

That last point is the practical difference from tensor-only formats. Hugging Face's docs contrast GGUF with safetensors: GGUF stores the tensors plus a standardised set of metadata. Examples from the spec include general.architecture, tokenizer.ggml.model (the tokenizer type) and llama.context_length (the context length the model was trained on). The Hugging Face Hub has a viewer that shows this metadata and each tensor's precision on any GGUF model page.

A single GGUF file can mix precisions. In a Q4_K_M file some tensors are Q6_K and most are Q4_K, and the file records which is which.

The common GGUF types and their bits per weight

Values below come from llama.cpp's block definitions (base types) and from its quantize README, which measured the file-type averages on Llama-3.1-8B. File-type averages vary slightly from model to model.

Notice that Q2_K averages 3.16 bits and Q3_K_M averages 4.00 bits on this model. The number in the name is the bit width of the main quantized values, not the file's average, because the mixes keep sensitive tensors at higher precision.

What K, S, M and L mean

• Q4_0, Q5_0, Q8_0 (no K): the original "legacy" types. Fixed 32-weight blocks with one scale each. Hugging Face's table marks Q4_0, Q4_1, Q5_0, Q5_1 and Q8_0 as legacy, although llama.cpp's quantize tool still produces Q8_0 and lists it in its size table.
• K (k-quants): introduced in llama.cpp pull request #1684. Weights are grouped into 256-weight super-blocks made of smaller blocks, and the per-block scales are themselves quantized (6-bit for Q4_K and Q5_K, 8-bit for Q6_K). This gets more accuracy out of the same number of bits.
• _S, _M, _L: small, medium, large mixes. The pull request defines Q4_K_S as Q4_K for all tensors, and Q4_K_M as Q6_K for half of the attention.wv and feed_forward.w2 tensors and Q4_K for the rest. Q5_K_M follows the same pattern with Q5_K.
• IQ types: newer quantizations that use an importance matrix, sample-based statistics about which weights matter most. llama.cpp's quantize tool takes one with --imatrix file_name.

How much quality do you lose?

Quality loss is usually measured as perplexity, how surprised the model is by a reference text (lower is better), or as KL divergence from the full-precision model's output. llama.cpp's README names both. The k-quants pull request published perplexity on the wikitext dataset for the original 7B LLaMA model in 2023:

The percentages are our arithmetic from those values, for example (5.9601 − 5.9066) ÷ 5.9066 = 0.91%. The pattern is the useful part: little loss down to about 5 bits, a small loss at 4 bits, and a steep rise below that. These are old results on one old model. Newer models and newer quantization methods give different numbers, and perplexity does not capture every kind of degradation, so test on your own tasks.

Choosing a quantization

• Plenty of memory: Q8_0 keeps the most precision of the quantized types at about half the size of F16 (7.95 vs 14.96 GiB for Llama-3.1-8B).
• Typical choice: Q4_K_M or Q5_K_M. A good balance of size and quality for most local use.
• Tight on memory: compare a larger model at Q4 with a smaller model at Q8 of similar file size on your own tasks. Which wins depends on the models and the work.
• Avoid: re-quantizing a file that is already quantized. llama.cpp's README warns this "can severely reduce quality compared to quantizing from 16bit or 32bit."

To translate a file size into the GPU memory you will need, including the context cache, see how much VRAM you need to run an LLM locally. To download and run a quantized model in a couple of commands, see how to run an LLM locally with Ollama.

Making your own GGUF

llama.cpp's quantize README shows the two-step process: convert a Hugging Face model to a 16-bit GGUF, then quantize it.

python convert_hf_to_gguf.py --outfile gemma-4-E2B-it-bf16.gguf --outtype bf16 --remote google/gemma-4-E2B-it
./build/bin/llama-quantize gemma-4-E2B-it-bf16.gguf gemma-4-E2B-it-Q4_K_M.gguf Q4_K_M

The README notes that memory and disk requirements are currently the same, so you need enough disk space for both files and enough RAM to load the model while it is processed.

Frequently asked questions

Which GGUF quantization should I download?

Pick the largest file that fits in your GPU memory with room left for the context. Q4_K_M is a common starting point because it is about 30% of the FP16 size, and in llama.cpp's original k-quant tests its perplexity was within about 1% of FP16 on a 7B model. If you have spare memory, Q5_K_M, Q6_K or Q8_0 lose less. Go below 4 bits only when nothing larger fits.

Does quantization make a model faster?

It depends on the runtime and hardware. Smaller weights mean less data to move through memory, which often helps on GPUs and CPUs where memory bandwidth is the limit. But Hugging Face's Transformers guide notes that with on-the-fly dequantization, as in bitsandbytes, inference is often not faster and can be slightly slower. Measure on your own setup.

Can I quantize a model myself?

Yes. llama.cpp's README shows converting a Hugging Face model to a 16-bit GGUF with convert_hf_to_gguf.py and then running llama-quantize with a type such as Q4_K_M. It warns against re-quantizing a file that is already quantized, because that can severely reduce quality compared with starting from 16-bit or 32-bit weights.

What does the 'I' in IQ4_XS or IQ3_M mean?

The IQ types are a newer family of llama.cpp quantizations that use an importance matrix: statistics gathered by running sample text through the model, used to decide which weights need more precision. The Hugging Face GGUF table lists IQ4_XS at 4.25 bits per weight and IQ3_XXS at 3.06, for example.

Sources

• ggml docs: GGUF file format specification
• Hugging Face Hub docs: GGUF and quantization types
• llama.cpp: tools/quantize README
• llama.cpp pull request #1684: k-quants
• llama.cpp: ggml-common.h (block layouts)
• Hugging Face Transformers docs: Optimizing LLMs for speed and memory