Excerpt: Quantization is a technique that shrinks AI models by storing numbers with fewer bits—like FP8 or FP4 instead of FP16—so models use less VRAM/unified memory and often run faster. This guide explains what those formats mean, how they work, and how to estimate whether a model will fit on your GPU.
Introduction: Why “Quantized Models” Matter
If you’ve ever tried running a large language model (LLM) locally and hit errors like “out of memory,” you’ve met the biggest practical constraint in modern AI: memory. AI models can contain billions of parameters (numbers), and all those numbers must live somewhere—usually on your GPU’s VRAM (video memory) or a device’s unified memory (a shared pool used by CPU and GPU).
Quantized models are one of the main reasons people can run powerful models on consumer hardware today. When you see labels like FP16, BF16, FP8, INT8, INT4, or FP4, you’re looking at the numerical format used to store model values. Fewer bits typically means:
- Lower memory use (so bigger models can fit)
- Higher speed (because less data moves between memory and GPU cores)
- Potential accuracy trade-offs (though modern techniques often keep quality very close)
This article explains what these formats mean in plain language, how quantization works, and how it directly relates to your GPU’s VRAM or unified memory.
Definition: What Is Quantization in AI Models?
Quantization is the process of representing the numbers inside an AI model using fewer bits than the original version.
Most modern models are trained using 16-bit formats like FP16 or BF16. Quantization converts some (or most) of those values into lower-precision formats such as 8-bit (FP8/INT8) or 4-bit (FP4/INT4/NF4).
Simple analogy: Imagine you’re storing a long list of temperatures. If you store each temperature to the nearest 0.001 degree, you’ll need more space than if you store it to the nearest 0.1 degree. Quantization is similar: you store numbers with fewer “digits,” which saves space, while trying to keep the overall meaning the same.
How It Works: From Bits to Performance
The core idea: models are mostly matrices
Neural networks—especially transformers used in LLMs—spend most of their time doing matrix multiplications. The GPU excels at this, but performance is often limited by how fast it can:
- Fetch weights from VRAM/unified memory
- Multiply and accumulate values in compute units (often tensor cores)
- Write intermediate results back to memory
Quantization reduces the size of the data being moved and processed. That can increase throughput (tokens/second) and reduce latency, especially when memory bandwidth is the bottleneck.
What does “FP16 vs FP8 vs FP4” actually mean?
These labels describe how many bits are used to store each number.
- FP16/BF16 = 16 bits per value
- FP8 = 8 bits per value
- FP4 = 4 bits per value (often implemented with scaling tricks that make the effective storage slightly above 4 bits/value)
Reducing bits cuts memory roughly proportionally:
- Going from 16-bit to 8-bit can cut weight memory about in half.
- Going from 16-bit to 4-bit can cut weight memory to about one quarter (plus some overhead for scaling/metadata).
Floating point vs integer: why there are multiple “8-bit” and “4-bit” types
Not all low-bit formats behave the same. The big difference is how they handle range (very large/small values) and precision (how finely you can represent values in between).
- Floating-point (FP) formats (FP16, BF16, FP8, FP4) can represent a wide range of magnitudes because they store numbers in a “scientific notation” style (sign, exponent, mantissa).
- Integer (INT) formats (INT8, INT4) represent fixed-step values. They can be very efficient, but often need careful scaling to avoid losing important detail.
Analogy: Floating point is like writing “3.2 × 10³” to handle both tiny and huge numbers. Integer is like counting in fixed steps (…-2, -1, 0, 1, 2…). Both can work, but they require different strategies when squeezing values into fewer bits.
Key Components: What Actually Gets Quantized?
When people say “a model is quantized,” they often mean the weights are quantized. But there are multiple kinds of numbers inside a model:
1) Weights (parameters)
Weights are the learned values that define the model’s behavior. They are the largest, most persistent memory cost during inference.
Quantizing weights is the biggest reason a model might fit into a smaller GPU. For example, a model that needs ~14 GB for weights in FP16 might need ~7 GB in FP8 or ~3.5–5 GB in a 4-bit format depending on overhead.
2) Activations (intermediate results)
Activations are temporary values created while the model processes your prompt and generates tokens. They impact peak memory usage, especially with:
- Long input prompts
- Large batch sizes
- Some training or fine-tuning setups
Many inference engines use mixed precision: weights might be 4-bit, but some activations may remain in FP16/BF16 for stability and speed.
3) KV cache (critical for LLM inference)
In transformers, the key-value (KV) cache stores attention-related tensors so the model doesn’t recompute them for every new token. KV cache grows with:
- Context length (how many tokens you keep in memory)
- Model size (hidden dimensions, number of layers/heads)
This is why you can load a quantized model successfully but still run out of VRAM when you increase context length. Some systems also quantize KV cache to reduce memory further.
4) Optimizer states (mainly for training)
If you are training or doing heavy fine-tuning, memory use explodes because you also store gradients and optimizer statistics (often FP16/FP32). Quantization helps here too, but training-friendly quantization (like FP8/FP4 training schemes) is more complex than inference-only quantization.
Diagram description: “Where memory goes”
Imagine a stacked bar labeled “Total GPU Memory.”
- The largest block is Model Weights (shrinks a lot with quantization).
- A second block is KV Cache (grows with context length; may or may not be quantized).
- A third block is Activations/Workspace (varies with implementation and batch size).
- A small block is Framework overhead (runtime buffers, kernels, fragmentation).
Quantization mostly shrinks the first block, and sometimes the second and third depending on the engine.
Understanding FP16, BF16, FP8, and FP4 (Plain-English Guide)
FP16 and BF16: the common “baseline” for modern AI
FP16 (16-bit floating point) and BF16 (“brain float 16”) are widely used because they balance speed and accuracy. BF16 keeps a wider exponent range than FP16, which can be more stable for some training workloads.
For most users, FP16/BF16 means: high quality, but also high memory use.
FP8: 8-bit floating point for speed and scale
FP8 uses 8 bits per value. Modern GPUs (especially data-center-class, and increasingly newer architectures) can accelerate FP8 operations efficiently. FP8 often comes in two variants with different trade-offs:
- E4M3: more precision in the mantissa, smaller range—often useful for forward computations.
- E5M2: larger range, less precision—often useful where values can vary widely (like gradients in training).
In practice, FP8 typically uses scaling factors to avoid overflow/underflow. Think of scaling like adjusting the “zoom level” so the important numbers fit into the smaller format.
FP4: the emerging frontier for compression
FP4 is even smaller—4 bits per value—so it can offer major memory savings. But 4 bits is extremely tight, so FP4 systems typically rely on:
- Group-wise quantization (store a scale per small group of values)
- Mixed precision (some parts remain higher precision)
- Hardware support (newer tensor cores can make FP4 truly fast)
One modern approach stores FP4 values in groups (for example, groups of 16 values) and attaches a scale factor (often FP8 plus a tensor-level scale) so the tiny FP4 numbers still represent meaningful magnitudes. This is a big reason FP4 can be surprisingly accurate despite being only 4 bits.
INT8/INT4 and NF4: common quantization you’ll see in the wild
Many consumer-friendly quantized models use integer or specialized 4-bit schemes:
- INT8: usually good accuracy, moderate compression.
- INT4: very strong compression, but accuracy can drop without careful calibration or fine-tuning.
- NF4: a 4-bit “normalized float” approach designed to preserve quality for LLM weights; common in some fine-tuning workflows.
The key idea: there are many “4-bit” models, and they are not all equivalent. Different quantization methods can produce different quality and speed—even if they all say “4-bit.”
How Quantization Relates to GPUs, VRAM, and Unified Memory
VRAM: why discrete GPUs care so much
VRAM is the dedicated memory on a discrete GPU (like NVIDIA GeForce/RTX or AMD Radeon). It’s fast and high-bandwidth—but limited in capacity (8 GB, 12 GB, 16 GB, 24 GB, etc.).
To run an LLM efficiently, you generally want:
- Most or all weights in VRAM
- Enough VRAM left for KV cache and runtime buffers
If the model doesn’t fit, the system may offload some data to system RAM, which is much slower. That typically hurts speed a lot.
Unified memory: what changes on Apple Silicon and iGPUs
Unified memory is a shared pool used by both CPU and GPU (common on Apple Silicon and some integrated GPU systems). The advantage is flexibility—you can have large memory pools (like 32–128 GB). The trade-off is that bandwidth and GPU compute characteristics differ from high-end discrete GPUs.
Quantization still helps unified memory systems because it:
- Reduces total footprint, letting larger models fit comfortably
- Reduces bandwidth pressure, often improving responsiveness
However, performance depends heavily on the device’s memory bandwidth and GPU design. A quantized model may “fit” easily but still run slower than on a discrete GPU with fast VRAM.
Rule of thumb: how many GB will a model need?
You can estimate weights memory with a simple approximation:
- FP16/BF16: parameters × 2 bytes
- 8-bit: parameters × 1 byte (plus small overhead)
- 4-bit: parameters × 0.5 bytes (plus overhead for scales/metadata)
Concrete example (rough):
- A 7B parameter model in FP16: ~7,000,000,000 × 2 bytes ≈ 14 GB for weights
- Same model in 8-bit: ≈ 7 GB
- Same model in 4-bit: ≈ 3.5–5 GB depending on quantization method and overhead
Important: This is only weights. You still need headroom for KV cache, activations, and runtime overhead.
Why quantization can make models faster (not just smaller)
Even if your GPU has enough memory, quantization can speed things up because:
- Less bandwidth used: fewer bytes pulled from memory per operation
- Better cache behavior: more weights fit into caches
- Tensor core acceleration: some GPUs have specialized hardware for FP8/FP4
That said, speed gains depend on the stack. Some quantized formats are extremely fast on supported hardware; others rely on extra conversion steps that can reduce the benefit.
Real-World Applications: Where Quantized Models Are Used
1) Running LLMs locally (chatbots, coding assistants)
Quantized models are commonly used for local inference so that a 7B–70B model can run on consumer hardware. Typical outcomes:
- 4-bit quantization makes mid-sized models feasible on 8–16 GB VRAM
- 8-bit quantization can offer better quality with moderate memory savings
2) Edge devices and on-device AI (phones, laptops)
On-device assistants, summarizers, and image tools benefit from smaller models to reduce memory, power, and thermal load. Quantization is a major enabler for “AI on your device” experiences.
3) Serving AI at scale (cloud inference)
For companies serving millions of requests, quantization can reduce:
- GPU memory needed per model replica
- Cost per token (by packing more workloads per GPU)
- Latency (when bandwidth-bound)
Newer data-center GPUs also support FP8 and emerging FP4 paths to maximize throughput for large-scale inference.
4) Training and fine-tuning (FP8/FP4 mixed precision)
Quantization isn’t only for inference. FP8 training is increasingly common in high-performance environments, using careful scaling and mixed precision to keep training stable. FP4 training and FP4-quantized deployment are emerging, especially with newer GPU architectures designed to accelerate these formats.
Benefits: Why Quantization Is Valuable
1) Fit bigger models into limited VRAM
This is the most visible win. Quantization often determines whether a model is usable on your hardware at all.
2) Run faster by reducing memory bandwidth pressure
Many inference workloads are limited by memory bandwidth rather than raw compute. Smaller weights can mean faster token generation.
3) Lower cost and energy usage
Less memory and faster inference can reduce energy per request—important for both battery-powered devices and cloud deployments.
4) More concurrency
On servers, smaller memory footprints allow more simultaneous sessions or larger batch sizes, improving overall throughput.
Challenges and Limitations: What Quantization Can’t Magically Fix
1) Accuracy and “feel” can change
Quantization can introduce small numerical errors. Depending on the method and the model, you might notice:
- More mistakes in math or reasoning tasks
- Slightly worse instruction-following
- Different output style or increased repetition
Modern 8-bit and well-done 4-bit quantization often keeps quality close to the original, but results vary.
2) Not all GPUs accelerate all formats
Hardware matters. Some GPUs have strong native support for FP8 (and newer ones for FP4). Others may run quantized models via software kernels that don’t achieve the same speedups.
3) KV cache can still dominate memory at long context
Users often assume “4-bit model” means everything is 4-bit. Usually it’s mainly the weights. With long prompts (or long chats), KV cache can become the limiting factor even if the weights are small.
4) Quantization overhead and compatibility
Quantized models may require specific runtimes, kernels, or model formats. Some deployments also need extra metadata (scales, zero-points, group sizes). This overhead is typically worth it, but it means memory isn’t always exactly “¼” at 4-bit in real life.
Common misconception: “Lower bits always means worse.”
Not necessarily. With good quantization techniques and proper scaling, an FP8 or 4-bit model can be extremely close to FP16 quality for many tasks. In some cases, quantization can even act like a mild regularizer and slightly improve certain benchmark behaviors—though that’s not guaranteed and shouldn’t be assumed.
Future Outlook: Where FP8/FP4 Quantization Is Heading
The trend is clear: AI hardware and software are co-evolving toward lower precision formats.
- FP8 is becoming mainstream in high-performance training and inference because it delivers strong speed/memory benefits with minimal quality loss when done properly.
- FP4 is emerging as next-generation hardware support expands, using advanced scaling and grouping strategies to preserve quality while dramatically reducing memory.
- Better quantization algorithms (post-training methods and quantization-aware training) continue to narrow the gap between low-bit and high-precision models.
- KV cache quantization and memory-aware attention optimizations will matter more as long-context use cases grow.
In the near future, it will be normal to choose a model not only by parameter count, but also by a “precision profile”—weights precision, activation precision, and cache precision—optimized for your hardware.
Conclusion: Key Takeaways
- Quantization means storing and computing model numbers with fewer bits (e.g., FP8 or FP4) to reduce memory and often increase speed.
- VRAM/unified memory is the key constraint for running AI models; quantization primarily reduces the memory needed for weights.
- FP16/BF16 are common high-quality baselines; FP8 roughly halves memory; 4-bit approaches can reduce weight memory to around a quarter (plus overhead).
- Even with quantized weights, KV cache and activations can still drive memory usage—especially at long context lengths.
- Hardware support matters: the biggest speedups happen when your GPU has strong low-precision acceleration (tensor cores for FP8/FP4 or optimized int kernels).
If you want a practical way to think about it: quantization is like compressing a model’s “knowledge” into a smaller suitcase. The suitcase becomes easier to carry (fits in VRAM), often faster to move around (bandwidth), and if packed carefully (good quantization method), you still arrive with almost everything you need.