LLM Quantization Explained Hands-On: GGUF with llama.cpp and AWQ with llm-compressor

Quantisation is the reason you can run a 70-billion-parameter model on a single GPU, or an 8B model on a laptop. The idea is simple — store the model’s weights in fewer bits per number — but the practice has real choices: GGUF for the llama.cpp ecosystem (Ollama, LM Studio, llama-server), AWQ for GPU serving with vLLM. In this guide you’ll do both hands-on: convert a Hugging Face model to GGUF and quantise it with llama.cpp, then produce an AWQ checkpoint with llm-compressor. You’ll learn what the alphabet soup (Q4_K_M, W4A16) actually means.

This is for developers who’ve hit the wall of “model too big for my GPU” and want to understand the lever they’re pulling. You’ll need an NVIDIA GPU for the AWQ part; the GGUF part works on CPU.

Prerequisites

• OS: Linux (WSL2 on Windows works).
• Build tools: git, cmake, a C++ compiler.
• Python: 3.10+ with pip.
• GPU: NVIDIA with 12 GB+ VRAM for the AWQ calibration step. GGUF quantisation itself is CPU-friendly.
• Disk: ~30 GB free — you’ll hold full-precision and quantised copies simultaneously.

Step 1: Understand what the quant names mean

GGUF quant types encode bits-per-weight and strategy. The ones you’ll actually use:

The _K variants use smarter per-tensor strategies than the old flat quants, and _M (medium) beats _S (small) on quality. Rule of thumb: start at Q4_K_M, measure, and only move when you have a reason. AWQ’s W4A16 means 4-bit weights with 16-bit activations — a different technique optimised for GPU inference kernels rather than CPU.

Step 2: Build llama.cpp

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON   # omit -DGGML_CUDA=ON for CPU-only
cmake --build build -j
ls build/bin/llama-quantize build/bin/llama-server

(The project moved from ggerganov/llama.cpp to the ggml-org — old tutorials referencing the former still work via redirect, but clone the new home.)

Step 3: Convert a Hugging Face model to GGUF

Pick a small model for learning — Qwen/Qwen2.5-0.5B-Instruct converts in under a minute:

# download weights
huggingface-cli download Qwen/Qwen2.5-0.5B-Instruct --local-dir ./qwen05b

# install converter deps, then convert to full-precision GGUF
pip install -r requirements/requirements-convert_hf_to_gguf.txt
python convert_hf_to_gguf.py ./qwen05b \
  --outfile qwen05b-f16.gguf --outtype f16

Expected output ends with something like Done writing ... model tensors (f16). Keep this F16 file — it’s the master you quantise every target level from. Quantising from an already-quantised file compounds the damage.

Step 4: Quantise to Q4_K_M (and friends)

./build/bin/llama-quantize qwen05b-f16.gguf qwen05b-Q4_K_M.gguf Q4_K_M

The tool prints a per-tensor summary and a final comparison:

llama_model_quantize_internal: model size  =   988.35 MB
llama_model_quantize_internal: quant size  =   335.71 MB

For better quality at low bit-depths, generate an importance matrix first — it tells the quantiser which weights matter most, measured against sample text:

./build/bin/llama-imatrix -m qwen05b-f16.gguf \
  -f some-text-corpus.txt -o qwen05b.imatrix --chunks 100
./build/bin/llama-quantize --imatrix qwen05b.imatrix \
  qwen05b-f16.gguf qwen05b-IQ4_XS.gguf IQ4_XS

The IQ (importance-quantised) types are the modern choice below 4 bits — IQ4_XS notably beats the older Q4_K_S at similar size.

Step 5: Serve the quantised model

Test it immediately with llama.cpp’s OpenAI-compatible server:

./build/bin/llama-server -m qwen05b-Q4_K_M.gguf -c 4096 --port 8080

curl http://localhost:8080/v1/chat/completions -d '{
  "model": "qwen",
  "messages": [{"role": "user", "content": "Count to five."}]
}'

One more lever: the KV cache also eats VRAM at long contexts. Quantise it separately without touching the weights:

./build/bin/llama-server -m qwen05b-Q4_K_M.gguf -c 32768 \
  -ctk q8_0 -ctv q8_0 -fa on

q8_0 cache roughly halves KV memory versus the default with little measurable quality cost — usually a better trade than dropping weight precision a level.

Step 6: Quantise to AWQ for GPU serving

AWQ is the right format when you’ll serve on NVIDIA GPUs via vLLM — its kernels are built for it. Note a 2026 reality: the original AutoAWQ library is deprecated (archived upstream); the maintained path is llm-compressor from the vLLM project:

pip install llmcompressor

python3 - <<'EOF'
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import (
    QuantizationModifier, AWQModifier)

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-0.5B-Instruct", dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
ds = load_dataset("wikitext", "wikitext-2-raw-v1", split="train[:128]")

recipe = [
    AWQModifier(),                                    # activation-aware scaling
    QuantizationModifier(ignore=["lm_head"],
                         scheme="W4A16_ASYM", targets=["Linear"]),
]
oneshot(model=model, dataset=ds, recipe=recipe,
        max_seq_length=512, num_calibration_samples=128)
model.save_pretrained("qwen05b-awq", save_compressed=True)
tokenizer.save_pretrained("qwen05b-awq")
print("Saved AWQ model to qwen05b-awq/")
EOF

vLLM auto-detects the compressed-tensors config, so serving is just vllm serve ./qwen05b-awq. If calibration OOMs on your card, it's the activation statistics — the fix is offloading them to CPU in the modifier config.

Step 7: Verify quality didn't fall off a cliff

Quantisation always costs something; the question is how much. Run a quick benchmark on both the F16 and quantised versions — our eval harness guide shows how — and compare. For Q4_K_M expect small drops on knowledge-heavy tasks and near-parity on reasoning; if scores crater, suspect a bad conversion (re-run from the F16 master) rather than the quant type.

Choosing between GGUF and AWQ

The two formats serve different masters. GGUF is the universal local format: one file, runs on CPU or GPU, works in Ollama, LM Studio, llama.cpp, and dozens of apps. Choose it when the model runs on your machine or on edge devices. AWQ (and its cousins GPTQ and FP8) target NVIDIA GPUs specifically, with kernels tuned for server throughput — choose them when you're serving with vLLM or TensorRT-LLM. You can keep both: quantise the F16 master to GGUF for local experiments and to AWQ for the production server, and compare their benchmark scores to make sure neither lost anything important.

One caution on the horizon: the quantisation landscape moves fast — new formats (NVFP4, MXFP4) keep appearing with each GPU generation. The specifics will age, but the workflow won't: full-precision master, deliberate quantisation, measured quality cost. Learn that loop and any new format is just new flags.

Where things go wrong

Converter complains about the tokenizer or architecture. Very new model architectures need a recent llama.cpp — git pull and rebuild. The converter validates as it goes; heed its warnings.

--allow-requantize temptation. Quantising a Q4 file down to Q3 is worse than going straight from F16 to Q3. Always quantise from the full-precision master.

AWQ calibration OOM. Even small models can spike during the scale search. Reduce num_calibration_samples or offload statistics to CPU.

Model loads in Ollama but babbles. If you import a GGUF via a Modelfile (FROM ./model.gguf), a missing or wrong chat template is the usual suspect — check what the converter wrote into the file's metadata.

Wrap-up

You've now got both major quantisation paths in your hands: GGUF via llama.cpp for the local/edge ecosystem (Ollama consumes GGUF natively), and AWQ via llm-compressor for GPU serving. The workflow to internalise: keep the F16 master, quantise deliberately, and measure the quality cost instead of assuming it. Disk is cheap; silent accuracy loss isn't.

Featured image: Fritzchens Fritz via Wikimedia Commons (CC0 1.0 Universal)