Key Takeaways
- 50% VRAM Savings: Both INT8 and FP8 cut weight memory footprint by roughly half compared to standard $FP16$/$BF16$ baselines.
- The Accuracy Winner: INT8's rigid spacing struggles with the massive activation outliers found in LLMs. FP8's floating-point exponent naturally preserves them, keeping accuracy loss near zero.
- Higher Concurrency: Applying FP8 to the Key-Value (KV) cache slashes memory bandwidth bottlenecks, allowing you to run larger serving batches and handle more concurrent users per GPU.
- Lower Deployment Overhead: INT8 demands intensive data calibration or retraining (QAT). FP8 is operationally faster to deploy, often requiring no separate calibration thanks to automated dynamic scaling.
Every team running Large Language Models (LLMs) or generative AI in production eventually hits the same wall: the model works beautifully in an experimental notebook, but serving it to real users at scale is slow, expensive, or both.
GPU memory (VRAM) fills up instantly, latency creeps past what your product's user experience can tolerate, and your cloud bill grows faster than your revenue. The single most effective lever for fixing this performance bottleneck, without retraining the model from scratch,is model quantisation.
What is Model Quantisation?
Quantisation reduces the numerical precision used to store and compute a model's weights and activations. Instead of using the 16-bit ($FP16$ or $BF16$) or 32-bit ($FP32$) floating-point numbers the model was trained with, you represent those same values using fewer bits, most commonly 8 bits.
When executed correctly, quantisation delivers massive production benefits:
- Cuts the model's memory footprint in half.
- Boosts inference throughput significantly.
- Lowers the cost per token or per image.
- Keeps output quality and accuracy close to the unquantized baseline.
However, "8-bit quantisation" is not a single technique. Two formats dominate modern AI engineering conversations: INT8 and FP8. While they both use 8 bits, the way they represent numbers under the hood, the hardware they run best on, and the quality/cost trade-offs they impose are meaningfully different.
Under the Hood: INT8 vs. FP8 Structure
The core difference between these two formats lies in how they allocate their 8 bits of data. This architectural difference directly impacts the dynamic range (the distance between the smallest and largest numbers represented) and the precision (the accuracy of the representation).
- INT8 (Integer 8-bit): Represents values as an 8-bit signed integer using two's-complement encoding, giving a range of -128 to 127. It distributes numbers evenly along a linear scale, with no separate exponent or mantissa fields.
- FP8 (Floating-Point 8-bit): Splits its bits into a sign bit, an exponent (which scales the number up or down), and a mantissa (fraction). According to NVIDIA’s formats specification, FP8 actually operates in two distinct sub-formats depending on the workload: E4M3 and E5M2.
Hardware Support and Architectural Compatibility
Choosing between INT8 and FP8 heavily depends on your target infrastructure. While INT8 has broad, legacy support across multiple GPU generations, FP8 requires modern tensor core architectures to achieve full hardware acceleration.
The Cost vs. Quality Trade-Off
Implementing quantisation requires balancing infrastructure savings against potential degradation in model accuracy.
Key Rule: The choice between INT8 and FP8 determines whether your model maintains its original capabilities or suffers from performance degradation (accuracy drop).
What Quantisation Actually Does?
At its core, a neural network is an enormous collection of numbers: weights, biases, and the intermediate activation values computed as data flows through the network. During training, these numbers are typically stored in FP32 (32-bit floating point) or FP16/BF16 (16-bit floating point). This high level of detail is necessary to preserve precision during the delicate process of gradient descent, where tiny mathematical adjustments are made to teach the model.
Once a model is trained, however, you don't need that same extreme precision to run inference (serving predictions to users). Quantisation maps the wide range of high-precision values down onto a much smaller, discrete set of representable values. It achieves this using a scale factor that translates numbers between the original range and the quantised range.
The Mathematical Mapping: INT8 vs. FP8
While the conceptual goal is identical, NVIDIA's TensorRT documentation formalises the exact mathematical execution differently for each format. The process breaks down into specific mathematical operations:
The Core Difference: Destination Number Space
While the mathematical description sounds almost identical between the two formats, the real operational difference is not the process itself; it is what the destination number space actually looks like.
- INT8 forces numbers into rigid, equally spaced integer slots.
- FP8 maintains a flexible, logarithmic-like spacing that can capture both tiny fractions and very large numbers safely.
This difference in destination space determines whether an LLM maintains its conversational intelligence or starts outputting gibberish when compressed.
INT8: Fixed Points, Evenly Spaced
INT8 represents numbers as signed integers between -128 and 127. Every one of those 256 possible values is spaced exactly one unit apart from its neighbour. This uniform spacing is the format's defining characteristic, and it is also its core limitation.
Think of INT8 as a ruler with 256 evenly spaced tick marks:
- The Dense Centre: If most of your data lives near the middle of the ruler, you get consistent, predictable precision.
- The Transformer Problem: Neural network activations, especially inside transformer-based LLMs, tend to have a small number of extreme outlier values sitting alongside a dense cluster of much smaller values.
- The Compression Penalty: A ruler with fixed, even spacing must be scaled out to accommodate those distant outliers. As a result, the vast majority of your values (clustered near zero) get squeezed into just a handful of the available tick marks, throwing away precision exactly where most of the information lives.
Workarounds and Hardware Strengths
industry developed specialised engineering workarounds:
- SmoothQuant: A technique developed specifically to migrate the quantisation difficulty from activations to weights before converting to INT8, preventing severe quality loss.
- Silicon Efficiency: On the hardware side, uniform, fixed-range integer math is simpler for silicon to execute than floating-point math. Processors, especially GPUs with specialised Tensor Cores, run integer operations at massive throughput. This is why INT8 has been the default choice for efficient inference on edge devices and production servers, sometimes stepping down further to INT4 for severe efficiency requirements.
- Universality: Integer arithmetic has been a first-class citizen in accelerator design for a long time. INT8 support is available across a broad spectrum of hardware generations, from older GPUs and CPUs to edge accelerators. If your fleet includes anything older than the most recent NVIDIA architectures, INT8 is often the only 8-bit option available.
FP8: Fewer Even Spaces, Wider Reach
FP8 takes a completely different approach to the same 8-bit budget. Rather than allocating all bits to represent an integer magnitude, it keeps the classic floating-point structure: a sign bit, a set of exponent bits, and a set of mantissa bits.
The exponent changes everything. Just as $FP32$ and $FP16$ can represent both extremely tiny and extremely large numbers by adjusting an exponent, FP8 inherits that same dynamic-range mechanism compressed into far fewer bits.
Non-Uniform Spacing Dynamics
As a result of this architecture, FP8's representable values are not evenly spaced:
- Near Zero: Where small distinctions matter enormously for neural network activations, FP8 packs its available precision densely.
- At the Extremes: Further from zero, the gaps between representable values widen, but the format can still reach much larger magnitudes than INT8 ever could within the same 8-bit limits.
According to NVIDIA's TensorRT quantisation documentation, the standard FP8 E4M3 variant clips values to the range [-448, 448] before casting them,a dramatically wider dynamic range than INT8's fixed [-128, 127] integer span.
The Two Standard FP8 Variants
Originally proposed jointly by NVIDIA, Arm, and Intel, and submitted to IEEE for potential standardization as a common interchange format, FP8 is split into two distinct formats to serve specific operational roles :
This dual-format design allows FP8 to represent the wildly varying activation distributions found in large transformer models without the aggressive rescaling INT8 needs. This makes FP8 quantisation safer for smaller models, offering better performance improvements with less degradation of output quality.
Hardware Caveat: FP8's performance benefits are only fully realised on GPU architectures with native FP8 Tensor Core support, such as NVIDIA's Hopper and Blackwell generations. On older hardware lacking native support, you will not get the same speed advantage because the acceleration relies on purpose-built silicon paths.
Head-to-Head: How the Two Formats Compare
To choose the right optimisation path for your infrastructure, it helps to look at how INT8 and FP8 stack up across key technical features.
Architectural and Deployment Trade-offs
Beyond the math, the two formats diverge significantly in engineering effort, compute throughput, and deployment reach:
Calibration Complexity
- INT8: Requires intensive Post-Training Quantisation (PTQ) calibration or Quantisation-Aware Training (QAT) to maintain acceptable accuracy. Even with QAT, some models never fully recover their baseline floating-point accuracy.
- FP8: Benefits from a much simpler path when a model is trained using FP8-native mixed precision, as inference and training share the same datatype family. In post-training scenarios (FP16→FP8), comparative evaluations using standard per-channel weight scaling and per-tensor activation scaling show FP8 holding up favorably against INT8 accuracy.
Compute Throughput
- INT8: Integer arithmetic units are historically simpler and faster to execute per operation than floating-point units of equivalent width, making it the default for latency-sensitive edge inference.
- FP8: On modern hardware with native FP8 Tensor Cores, the throughput gap narrows or reverses. The accelerator is purpose-built to run FP8 matrix multiplications at full Tensor Core speed. Furthermore, its ability to safely quantize activations (not just weights) unlocks additional speedups that pure INT8 activations risk losing to accuracy degradation.
Hardware Availability
- INT8: Wins on breadth. It runs efficiently across a wide span of accelerator generations and vendor ecosystems, including older data center GPUs, CPUs, and edge hardware.
- FP8: Wins on depth of optimization. It is an optimized, native datatype engineered specifically for NVIDIA's Hopper and Blackwell generations to deliver maximum efficiency in both training and inference workloads.
The Cost/Quality Trade-Off, Concretely
The choice between INT8 and FP8 impacts infrastructure costs, engineering velocity, and model fidelity across four distinct dimensions.
Core Comparison Matrix
Accuracy and Output Quality
The underlying numerical structure of each format directly governs final model accuracy. INT8 utilizes a fixed, evenly spaced integer grid that frequently struggles to capture the outlier-heavy, wide-dynamic-range activation distributions typical of transformer models. Without rigorous mitigation, INT8 can severely degrade output quality.
Conversely, FP8 features a floating-point structure with a built-in exponent. This architecture tolerates activation distributions gracefully, keeping quality loss nearly indistinguishable from higher-precision baselines for many production workloads.
Latency and Throughput Dynamics
On modern NVIDIA data-center GPUs, FP8 Tensor Core throughput is meaningfully higher than FP16. Furthermore, compressing the Key-Value (KV) cache with FP8 frees up substantial memory overhead. This saved memory can be reallocated to run larger batch sizes,utilizing the GPU's compute engine more effectively at a fixed memory ceiling,which often yields the largest real-world throughput gains.
Quantization Methodologies: PTQ vs. QAT
Regardless of your chosen 8-bit format, two deployment pathways exist depending on your model's accuracy headroom.
Post-Training Quantisation (PTQ)
- Mechanism: Takes a pre-trained model and calibrates quantization scales after the fact, typically using a small representative dataset to determine scale factors per channel or per tensor.
- Best For: Fast production deployment. For FP8, dynamic per-tensor quantization often requires no separate calibration step at all, as serving frameworks handle scaling automatically.
Quantisation-Aware Training (QAT)
- Mechanism: Inserts quantization operations directly into the training loop, forcing the model to learn to compensate for precision loss during optimization.
- Best For: Recovering accuracy that PTQ cannot reach. Evaluation shows that FP8 QAT runs can get remarkably close to the accuracy of the original higher-precision training run.
Strategic Quantization Variants
Optimizing a production network rarely requires a uniform approach. Highly effective pipelines use specialized architectural mixes to maximize throughput.
Where Mixed Precision Fits In
Mixed-precision quantization is an effective middle path when uniform quantization fails to meet accuracy requirements. Instead of converting an entire model, developers identify the specific layers most sensitive to precision loss,frequently the first and last layers of a network, or specific attention components, and maintain them in FP16 or FP32. The bulk of the remaining layers are then safely converted to INT8 or FP8.
Beyond the Weights: The KV Cache
For transformer-based Large Language Models (LLMs) or speech models, the key-value (KV) cache is often the largest consumer of GPU memory during serving. Quantizing the KV cache independently represents a high-leverage optimization:
- Production Impact: Quantizing the KV cache to FP8 significantly reduces memory bandwidth and cache footprint while preserving output fidelity.
- Concurrency Win: This directly enables higher user concurrency per GPU.
- Why FP8 Excits Here: Attention keys and values carry uneven, outlier-prone distributions that give INT8 trouble, making FP8's wider dynamic range a natural fit.
Architecture Selection: A Practical Checklist
Use this checklist to evaluate your infrastructure requirements before committing engineering hours to a specific quantization format:
- GPU Generation: If deploying on Hopper (H100) or Blackwell-class hardware, native FP8 Tensor Cores are available. FP8 should be your primary evaluation pathway.
- Fleet Diversity: If supporting a diverse fleet of older GPUs, CPUs, or edge devices, INT8's broad backward compatibility may outweigh FP8's per-GPU efficiency edge.
- Model Scale: Larger models are structurally more robust to INT8's coarser precision grid. Smaller models are highly exposed to quality loss from INT8 and benefit disproportionately from FP8's dynamic range.
- Quantization Target: Weight-only quantization is generally forgiving in either format. Activation and KV cache quantization is where FP8's floating-point structure earns its advantage clearly.
- Engineering Timeline: For a fast path to production, FP8's dynamic per-tensor workflow offers a lower-effort route when paired with automated serving frameworks.
- Accuracy Budget: If your product tolerates virtually zero quality regression, plan for careful calibration, mixed-precision boundaries, or QAT regardless of the format chosen.
Why Quantization Is an Ongoing Life-Cycle
The trade-off between FP8 and INT8 is dynamic. The optimal configuration shifts alongside your model updates, traffic patterns, batch sizes, and underlying hardware upgrades. A configuration optimized for an older GPU generation can leave substantial performance on the table once migrated to hardware featuring native FP8 Tensor Cores.
Infrastructure Takeaway: Quantization should be treated as a configurable, swappable component of your deployment stack rather than a one-off engineering project bolted onto a single model. Modern enterprises manage shifting fleets of models, backends, and load profiles; re-litigating the FP8-versus-INT8 decision manually for every new architecture generation simply does not scale.
How Simplismart Helps
This is precisely the problem Simplismart's inference platform is built to solve. Rather than treating quantisation as a one-time, model-specific engineering exercise, Simplismart's modular architecture lets enterprises adapt their ML stack, including quantisation strategy, model backends, and serving libraries, to the specific use case and load profile at hand, and swap components in and out through a modular interface as models, traffic, and hardware evolve, without getting locked into one configuration or one vendor.
In practice, that means teams don't have to choose between FP8 and INT8 in the abstract. Simplismart's platform is designed to apply the right precision strategy per model and per deployment target, informed by real production benchmarking rather than guesswork. Whether you're deploying an LLM where KV cache quantization to FP8 unlocks meaningfully higher concurrency per GPU, or running a workload on infrastructure where INT8's broader hardware compatibility is the deciding factor, Simplismart's inference-first orchestration layer is built to make that decision once, apply it consistently, and keep adapting it as your models and hardware change, while also layering in complementary optimizations like pruning, distillation, and adaptive batching to compound the gains further.
If you're currently weighing FP8 against INT8 for your own production models, or if you've already picked one and are wondering whether you're leaving performance or quality on the table, talk to the Simplismart team. Explore how Simplismart's inference platform can benchmark, quantise, and deploy your models with the precision strategy that actually fits your cost and quality targets, on the hardware you're running today.
Frequently Asked Questions (FAQ)
Is FP8 always better than INT8?
Not universally. FP8 tends to hold up better on accuracy for activations and precision-sensitive workloads because of its wider dynamic range, and it delivers strong throughput on GPUs with native FP8 tensor cores like Hopper and Blackwell. But INT8 remains the more broadly compatible option across older GPUs, CPUs, and edge hardware, and for models robust enough to tolerate its coarser, uniformly spaced precision, it's a perfectly effective and often simpler choice.
Do I need special hardware to use FP8?
Yes, to get the full benefit. FP8's efficiency gains are tied to native tensor core support on NVIDIA's Hopper and Blackwell GPU architectures. On hardware without that native support, you won't see the same acceleration, since the speed advantage comes from purpose-built tensor core execution paths, not the number format alone.
Does quantisation always reduce output quality?
Some precision loss is inherent to reducing the number of bits used to represent values, but how much it affects real output quality depends heavily on the format, the model, and the calibration approach. With careful calibration, or quantisation-aware training where needed, both INT8 and FP8 can achieve accuracy very close to the original higher-precision model for many production workloads.
What's the difference between E4M3 and E5M2 FP8 formats?
Both are standard FP8 variants defined in the joint NVIDIA/Arm/Intel FP8 specification. E4M3 uses 4 exponent bits and 3 mantissa bits, prioritising precision, and is typically used for weights, activations, and inference. E5M2 uses 5 exponent bits and 2 mantissa bits, prioritising dynamic range, and is more commonly used for gradients during mixed-precision training.
Should I quantise the KV cache too, or just the model weights?
For LLM and speech-model serving, the KV cache is often one of the largest memory consumers during inference, and quantising it, commonly to FP8, can significantly cut memory bandwidth and footprint while preserving output fidelity, directly enabling higher concurrency per GPU. Weight-only quantisation is a good starting point, but activation and KV cache quantisation frequently deliver the larger real-world throughput gains.
What's the difference between post-training quantisation and quantisation-aware training?
Post-training quantisation (PTQ) calibrates quantisation scales on an already-trained model using a representative dataset. It's fast and often sufficient, especially for FP8's simpler dynamic scaling workflows. Quantisation-aware training (QAT) incorporates quantisation into the training loop itself, recovering more accuracy at the cost of additional training time and compute. A common approach is to start with PTQ and move to QAT only if the accuracy drop is unacceptable.
Can I mix precisions within the same model?
Yes. Mixed-precision quantisation keeps particularly sensitive layers, often early or late layers, or specific attention components, in higher precision like FP16 or FP32, while converting the rest of the model to INT8 or FP8. This can preserve most of the memory and speed benefits of aggressive quantisation while protecting the layers most responsible for output quality.
How does Simplismart decide between FP8 and INT8 for my models?
Simplismart's inference platform is built around a modular ML stack that lets quantisation strategy, model backends, and serving libraries be adapted per model and per deployment target, based on real benchmarking against your specific use case and load profile, rather than a fixed, one-size-fits-all configuration. That means the FP8-versus-INT8 decision, and the calibration effort behind it, is handled as part of the deployment process rather than a separate research project for every model you ship.
Ready to see the difference the right quantisation strategy makes for your models? Get in touch with Simplismart to benchmark your workload and start deploying faster, more cost-efficient inference, without compromising on quality.
