noumenalabs/t5-small-gguf

T5 GGUF Analysis

This document records the T5-small GGUF evaluation run.

Environment

Verified runtime:

item	value
Python	3.11.12
Torch	2.9.0+cu129
Torch CUDA	12.9
CUDA available	True
GPU	NVIDIA GeForce RTX 3070 Laptop GPU

Models

The run evaluated these GGUFs:

model	role
t5-small-f32.gguf	unquantized reference baseline
t5-small-f16.gguf	high-precision comparison and quantization source
t5-small-q8_0.gguf	quantized
t5-small-q5_k_m.gguf	quantized
t5-small-q4_k_m.gguf	quantized
t5-small-q4_0.gguf	quantized
t5-small-q3_k_m.gguf	quantized
t5-small-q2_k.gguf	quantized

Conversion Check Results

The conversion check compares greedy HF outputs against greedy f32 GGUF outputs. It validates that the unquantized GGUF is a usable reference before comparing quantized models against it.

dataset	examples	exact match	chrF	first token match
CoLA	2,000	1.000	1.000	1.000
summarization	2,000	0.117	0.953	0.990
translation en-de	2,000	0.993	0.996	1.000
translation en-fr	2,000	0.986	0.995	1.000
overall	8,000	0.774	0.986	0.997

Interpretation:

• The f32 GGUF tracks HF closely overall.
• Summarization has low exact match but high chrF, which points to wording differences rather than broad conversion drift.
• Translation and CoLA are effectively matching at the output level.

Generation Results

Generation used greedy decoding with n_predict=64. Agreement and similarity are measured against the f32 GGUF baseline output.

model	agreement vs f32	similarity vs f32
t5-small-f16	0.990	0.998
t5-small-q8_0	0.723	0.947
t5-small-q5_k_m	0.526	0.889
t5-small-q4_k_m	0.474	0.870
t5-small-q4_0	0.417	0.837
t5-small-q3_k_m	0.375	0.814
t5-small-q2_k	0.287	0.660

Per-dataset generation metrics:

dataset	model	exact match vs reference	chrF vs reference	agreement vs f32	similarity vs f32
CoLA	t5-small-f16	0.697	0.950	1.000	1.000
CoLA	t5-small-f32	0.697	0.950	-	-
CoLA	t5-small-q2_k	0.697	0.950	1.000	1.000
CoLA	t5-small-q3_k_m	0.697	0.949	1.000	1.000
CoLA	t5-small-q4_0	0.697	0.950	0.995	1.000
CoLA	t5-small-q4_k_m	0.698	0.950	0.999	1.000
CoLA	t5-small-q5_k_m	0.697	0.950	1.000	1.000
CoLA	t5-small-q8_0	0.697	0.950	1.000	1.000
summarization	t5-small-f16	0.000	0.133	0.979	0.995
summarization	t5-small-f32	0.000	0.133	-	-
summarization	t5-small-q2_k	0.000	0.068	0.000	0.254
summarization	t5-small-q3_k_m	0.000	0.123	0.039	0.510
summarization	t5-small-q4_0	0.000	0.123	0.071	0.550
summarization	t5-small-q4_k_m	0.000	0.131	0.137	0.642
summarization	t5-small-q5_k_m	0.000	0.128	0.210	0.689
summarization	t5-small-q8_0	0.000	0.133	0.541	0.852
translation en-de	t5-small-f16	0.020	0.361	0.989	0.999
translation en-de	t5-small-f32	0.020	0.361	-	-
translation en-de	t5-small-q2_k	0.015	0.315	0.090	0.738
translation en-de	t5-small-q3_k_m	0.018	0.353	0.234	0.876
translation en-de	t5-small-q4_0	0.019	0.357	0.304	0.905
translation en-de	t5-small-q4_k_m	0.019	0.359	0.380	0.920
translation en-de	t5-small-q5_k_m	0.019	0.359	0.448	0.935
translation en-de	t5-small-q8_0	0.019	0.360	0.680	0.970
translation en-fr	t5-small-f16	0.017	0.381	0.993	0.999
translation en-fr	t5-small-f32	0.017	0.381	-	-
translation en-fr	t5-small-q2_k	0.007	0.276	0.057	0.646
translation en-fr	t5-small-q3_k_m	0.015	0.368	0.226	0.868
translation en-fr	t5-small-q4_0	0.015	0.372	0.299	0.891
translation en-fr	t5-small-q4_k_m	0.017	0.377	0.380	0.919
translation en-fr	t5-small-q5_k_m	0.016	0.380	0.446	0.933
translation en-fr	t5-small-q8_0	0.016	0.380	0.672	0.967

Interpretation:

• f16 is effectively equivalent to f32 for generated outputs.
• q8_0 preserves most behavior but still diverges on longer-form tasks.
• q5_k_m and q4_k_m are usable middle points depending on size and quality target.
• q2_k degrades heavily for summarization and translation.

Perplexity And KL Results

Perplexity is reported per dataset. KL/token and top-1 disagreement are the main quantization drift metrics because they compare each quantized model directly against f32 token distributions.

Token-weighted summary across all datasets:

model	tokens	KL/token	top-1 disagree
t5-small-f16	308,028	0.00000	0.0005
t5-small-f32	308,028	-	-
t5-small-q8_0	308,028	0.00187	0.0160
t5-small-q5_k_m	308,028	0.01004	0.0386
t5-small-q4_k_m	308,028	0.02038	0.0521
t5-small-q4_0	308,028	0.04847	0.0704
t5-small-q3_k_m	308,028	0.05892	0.0897
t5-small-q2_k	308,028	0.27523	0.1914

Per-dataset perplexity:

model	CoLA	summarization	translation en-de	translation en-fr
t5-small-f32	1.3490	138.5925	5.0317	3.8267
t5-small-f16	1.3491	138.6029	5.0317	3.8268
t5-small-q8_0	1.3494	133.1739	5.0314	3.8245
t5-small-q5_k_m	1.3498	139.2235	5.0748	3.8488
t5-small-q4_k_m	1.3535	155.2379	5.1135	3.8759
t5-small-q4_0	1.3593	215.7687	5.1394	3.9305
t5-small-q3_k_m	1.3490	153.6497	5.2163	3.9680
t5-small-q2_k	1.3577	262.6867	6.0281	4.4851

Per-dataset KL/token:

model	CoLA	summarization	translation en-de	translation en-fr
t5-small-f16	0.00000	0.00000	0.00000	0.00000
t5-small-q8_0	0.00029	0.00194	0.00191	0.00181
t5-small-q5_k_m	0.00544	0.01159	0.00923	0.00838
t5-small-q4_k_m	0.00811	0.02593	0.01732	0.01437
t5-small-q4_0	0.01239	0.07497	0.02886	0.02339
t5-small-q3_k_m	0.00539	0.07696	0.04827	0.04073
t5-small-q2_k	0.00350	0.36274	0.22476	0.18650

Interpretation:

• The KL ranking is stable and clear: f16, q8_0, q5_k_m, q4_k_m, q4_0, q3_k_m, then q2_k.
• q8_0 has very small distributional drift from f32.
• q5_k_m is the strongest compact quantization in this run.
• q4_k_m is materially better than q4_0 by KL/token and top-1 disagreement.
• q2_k has high drift and large top-1 disagreement on generation-heavy datasets.

Recommended Default

For T5-small in this workflow:

• Use t5-small-f32.gguf as the reference baseline.
• Use t5-small-q8_0.gguf when preserving behavior matters most.
• Use t5-small-q5_k_m.gguf as the best compact default from this run.
• Use t5-small-q4_k_m.gguf only when size pressure is stronger than quality.
• Avoid t5-small-q2_k.gguf for summarization or translation quality checks.

GOOGLE T5-small License: Apache 2.0 We followed and adopted their licnese.