Model Quantization: What Works for Production Inference

For the past several months we've been running our own LLM inference for some workloads. Quantization is the lever that determines whether self-hosted inference is cost-competitive with managed APIs. We benchmarked four quantization techniques on Llama-3-8B and Mistral-7B. This post is what we found, where each technique fits, and what to use for production.

Why quantization#

The basics: an LLM's weights are stored as floating point numbers. By default, training uses fp32 (4 bytes per weight) or bf16 (2 bytes). For inference, you don't need that precision. Quantization reduces the bits per weight — to 8, 4, or even 2 bits — at some cost in quality.

Why this matters:

• Less memory: an 8B-parameter model in fp16 is 16GB; in 4-bit it's ~5GB. Fits on smaller GPUs.
• Faster inference: less data to move means higher throughput.
• Cheaper hardware: a 4-bit-quantized model can run on consumer GPUs that fp16 can't fit.

The trade is quality. Aggressive quantization (low bit counts) starts to hurt model output quality. The question is: how much, and on what tasks?

What we benchmarked#

Models: Llama-3-8B-Instruct, Mistral-7B-Instruct (both bf16 baselines).

Quantization techniques:

• GPTQ-4bit: post-training quantization, 4 bits per weight. Most popular for static weights.
• AWQ-4bit: another 4-bit post-training approach, often better quality than GPTQ.
• GGUF Q4_K_M: format used by llama.cpp. Mixed precision (some tensors at 4-bit, sensitive ones at higher precision).
• bitsandbytes nf4: the bnb library's "normal float 4" quantization. Used by QLoRA for fine-tuning.

Benchmarks:

• Quality: a 200-question internal evaluation set (mix of factual recall, reasoning, instruction-following, code).
• Throughput: tokens/second on a single A100 40GB.
• Memory: peak GPU memory during inference.

The numbers#

For Llama-3-8B:

Quantization	Size	Mem (1 req)	Tok/s	Quality
bf16 baseline	16 GB	17 GB	78	100%
nf4 (bnb)	5.5 GB	7 GB	95	96%
AWQ-4bit	5.4 GB	6 GB	165	97%
GPTQ-4bit	5.4 GB	6 GB	155	95%
GGUF Q4_K_M	5.0 GB	5.5 GB	110	96%

Quality is normalized so the bf16 baseline scores 100%; lower is worse on the eval set.

A few takeaways:

1. AWQ-4bit and GPTQ-4bit are 2x throughput of the bf16 baseline. That's a real cost win.
2. Quality drop is small for most tasks. ~95-97% retention is acceptable for most production use cases.
3. GGUF runs slower because llama.cpp is CPU-friendly but on GPU it's not as optimized as the dedicated GPU paths.
4. bnb nf4 is fine but not the throughput leader. It's most useful for QLoRA training, less so for production inference.

Where quality drops most#

The eval set was diverse; quality drops weren't uniform across categories:

• Code generation: largest drops. Quantization hurts precise tasks where small token-probability differences matter. ~5-8% drop on coding eval at 4-bit.
• Multi-step reasoning: moderate drops. ~3-5%.
• Factual recall: small drops. ~1-2%.
• Instruction following / format: very small drops. ~1%.

If your task is "follow this format and respond using the provided context" (RAG-style), 4-bit quantization is great. If your task is "write working Python code from scratch," 4-bit hurts more.

For some teams, this means: 4-bit for RAG and conversation; bf16 (or 8-bit) for code generation tasks.

What we picked for production#

For our use cases, AWQ-4bit on Llama-3-8B turned out to be the right answer for self-hosted inference. Reasons:

• Throughput is highest of the 4-bit options.
• Quality is close to bf16 on our eval set (we don't have heavy code-generation use cases).
• vLLM has good AWQ support — we use vLLM in production.
• Memory fits comfortably on smaller GPUs (we use L4 and A10G for cost; AWQ-4bit fits on both).

For one specific use case (a coding-assistant feature) we use 8-bit instead of 4-bit because the code quality drop wasn't acceptable.

Serving infrastructure#

For inference, we use vLLM as the serving framework:

• AWQ models are loaded via vLLM's AWQ quantization support
• Continuous batching: multiple requests share the same forward pass
• PagedAttention: efficient KV-cache memory management
• OpenAI-compatible API (drop-in replacement for openai.ChatCompletion.create)

A single A10G can serve our quantized 8B model at ~150 tokens/s aggregate, supporting maybe 30-50 concurrent users with reasonable latency. Cost on AWS Spot: ~$0.30/hr. Compared to GPT-4o-mini API costs at our token volume, the break-even is around 200k tokens/hour; we're past that on most days.

Quantization for fine-tuned models#

When we fine-tune (LoRA), the workflow:

1. Load the base model in 4-bit (via bnb or AWQ).
2. Train the LoRA adapter on top (the base model is frozen).
3. For inference, either: keep the LoRA separate and apply at runtime (slightly slower), OR merge the LoRA into the base model and re-quantize (faster but per-adapter).

For our use case (one fine-tune per language we serve), we merge and re-quantize. Each merged+quantized model is ~5GB and serves with the same speed as the base.

Specific gotchas#

Things that bit us:

Quantizing a fresh download fails silently. Some versions of GPTQ libraries silently fall back to fp16 if the quantization doesn't converge. We always test the model size on disk after quantization to verify.

KV-cache memory dominates. Even with quantized weights, the per-request KV cache is fp16 by default. For long contexts, KV cache memory can exceed weight memory. vLLM's PagedAttention helps; we also use 8-bit KV cache for high-context workloads.

Tokenizer mismatches. Some quantized models on Hugging Face have subtly different tokenizers than the base model. Symptoms: garbage outputs. Always pair the quantized weights with the matching tokenizer.

Long-context degradation. Quantization quality often gets worse as context length grows (compounding errors). For our long-context use cases (>8k tokens), we use a less aggressive quantization (8-bit) to compensate.

What we tried and abandoned#

2-bit quantization (e.g., AQLM). Quality dropped too much for our use cases (15-25% on the eval set). Maybe useful for very large models (70B+) where 2-bit lets you fit on a single GPU. For 8B, 4-bit is the right floor.

Custom quantization training (training a model directly in low-bit). Active research area but for our use case, post-training quantization of an off-the-shelf model is good enough.

Hand-tuned per-layer quantization (sensitive layers higher precision, others lower). GGUF Q4_K_M does this in a managed way; trying to do it more aggressively was lots of work for marginal gain.

When quantization is the wrong answer#

Quantization helps if you're going to self-host. If you're using a managed API (OpenAI, Anthropic), quantization is invisible — those providers do their own optimizations.

Self-hosting with quantization makes sense when:

• You have enough volume that API costs > infrastructure costs (typically several hundred million tokens/month).
• You have specific privacy/compliance requirements that require keeping data on-prem.
• You're fine-tuning models, where self-hosting is the only way to run fine-tunes.
• You're at the cutting edge and want behavior the major APIs don't offer.

For most teams, the answer is: stick with managed APIs. The optimization story for quantization only matters once self-hosting is the right call.

What I'd tell a team starting#

Pick AWQ-4bit if you're using vLLM. It's well-supported, fast, and quality is good.

Benchmark quality on your actual eval set, not someone else's. "MMLU drops 2%" doesn't tell you what happens on your specific task.

Monitor inference quality after deployment. Quantization quality drift is real over edge-case inputs. Have an eval suite that runs regularly.

Don't over-quantize. 4-bit is the sweet spot for most production inference. 2-bit and below are research territory.

8-bit is a safe fallback for tasks where 4-bit hurts. Memory savings smaller, quality preserved.

Quantization is mature enough that it's a real production option. The tooling (vLLM, AutoAWQ, GPTQ libraries, llama.cpp) is solid; the quality trade-offs are well-characterized; the cost wins are real if you have the volume to amortize the operational complexity. The decision tree is mostly: do you have the volume, and are the quality trade-offs acceptable for your tasks. If yes to both, quantized self-hosted inference is likely cheaper than the API alternative.