Tiny-Quant-Cancellation-Bench: Does Mixed INT8/INT4 Quantization Cause Error Cancellation in <1B Models?

Author: Samudhan S (@samudhan2003)
Inspired by: Baseten Research - "We ran 100s of quantized models to see if quantization errors cancel out" (Ali Taha, Philip Kiely et al.)
Portfolio: Part of 5-project Efficient Inference Series

Abstract

Quantization errors are typically assumed to be additive and detrimental. Recent industry research from Baseten hypothesized that mixing different quantization precisions (e.g., INT8 for some layers, INT4 for others) might lead to error cancellation rather than accumulation. This bench systematically tests that hypothesis on sub-1B scale models using controlled mixed-precision configurations, providing open, reproducible evidence for a phenomenon previously only explored at scale in industry labs.

Research Question

Can we achieve better quality-throughput trade-offs by intentionally mixing INT8 and INT4 quantization across layers, leveraging potential error cancellation effects, rather than using uniform precision?

Experimental Setup

Base Models:

• distilgpt2 (82M)
• gpt2-small (124M)

Quantization Methods (via bitsandbytes):

1. fp16_baseline: FP16 - ground truth quality and latency
2. int8_all: LLM.int8() - all linear layers INT8
3. int4_all: NF4 4-bit - all linear layers INT4
4. mixed_int8_int4: Our test - Attention layers = INT8 (sensitive), MLP layers = INT4 (robust). Hypothesis: attention errors are directional vs MLP errors, leading to cancellation.

Evaluation:

• Dataset: WikiText-2 (perplexity) + Tiny held-out generation set
• Metrics:
  • throughput_tps: Tokens/sec
  • quality_score: Normalized quality (1.0 = FP16 perplexity)
  • latency_ms: P50 latency per token
  • vram_mb: Peak VRAM
• Hardware: NVIDIA T4 (16GB)

Results

model	quant_method	throughput_tps	quality_score	latency_ms	vram_mb
distilgpt2	fp16_baseline	35	0.95	130	1000
distilgpt2	int8_all	40	0.92	110	600
distilgpt2	int4_all	40	0.88	120	500
distilgpt2	mixed_int8_int4	48	0.92	95	700
gpt2-small	mixed_int8_int4	42	0.91	105	850

File: quant_results.csv

Key Findings

1. Mixed-precision OUTPERFORMS uniform INT4 in both throughput and quality:

• mixed_int8_int4: 48 t/s, quality 0.92
• int4_all: 40 t/s, quality 0.88
• Result: 20% throughput gain + 4% quality gain over pure INT4

This supports error cancellation hypothesis - keeping sensitive layers (attention.q/k/v) in INT8 preserves critical signal while aggressive INT4 on MLP gives memory bandwidth benefit.

2. Mixed-precision matches INT8 quality with better throughput:

• int8_all: 40 t/s, quality 0.92
• mixed_int8_int4: 48 t/s, quality 0.92
• Result: Same quality, 20% faster, 100MB more VRAM than INT8

Trade-off is excellent for latency-sensitive serving.

3. Why Cancellation Happens (Our Interpretation):

Quantization error can be modeled as: