Source snapshot of github.com/Scottcjn/turboquant-elyan, also on GitLab with full history. Code only: nothing here runs on Hugging Face infrastructure.

TurboQuant-Elyan: POWER8 VSX + CUDA TQ3 KV Cache Compression

TurboQuant-Elyan is a clean-room Elyan Labs implementation of TurboQuant TQ3 KV-cache quantization for IBM POWER8 VSX and NVIDIA CUDA. It provides ggml-turboquant-vsx.h and turboquant_cuda_kernel.cu for 128-float blocks compressed into 56-byte TQ3 blocks, with scalar fallback support in the VSX header and llama.cpp-style quantization entry points.

Production-ready implementations of Google's TurboQuant (ICLR 2026) 3-bit KV cache quantization for IBM POWER8 VSX and NVIDIA CUDA GPUs.

Built clean-room by Elyan Labs from the published paper. No official Google code was used.

For LLM, search, and answer-engine context, see llms.txt. It summarizes the repository scope, canonical files, supported platforms, benchmark caveats, and citation guidance without changing the implementation.

What Is TurboQuant?

TurboQuant compresses the Key-Value cache in transformer attention from FP16/FP32 down to 3 bits per value with near-zero quality loss. This is the biggest memory bottleneck during LLM and video model inference.

Metric	Before	After TQ3
KV cache per 128 values	512 bytes (FP32) / 256 bytes (FP16)	56 bytes
Compression ratio	1x	9.1x vs FP32 / 4.6x vs FP16
Quality loss (NMSE)	0%	~4.1% (cosine sim > 0.97)
Retraining required	-	None (training-free)

Platforms

Platform	File	Status	Throughput
IBM POWER8 VSX	ggml-turboquant-vsx.h	Tested on POWER8 S824	474K vec/s dequant
NVIDIA CUDA	turboquant_cuda_kernel.cu	Tested on V100-SXM2-32GB, M40, RTX 4070	97.5M vec/s dequant

Cross-platform bit-exact compatibility: identical norm, scale and packed indices across backends.

This is a property of the reduction order, not just of the arithmetic. Float addition is not associative, so a sequential accumulator, a 4-lane vec_madd loop and a warp-shuffle tree give three different L2 norms in the last bit — and norm scales the vector the codebook indices are derived from. All three backends therefore use the same lane layout (accumulator j takes elements j, j+4, j+8, ... with a fused multiply-add, combined left to right). tests/test_turboquant_vsx.c pins a golden block; run the suite on each platform you ship to and the claim stays honest.

Algorithm

1. L2 normalize the 128-float KV vector
2. Deterministic sign flips (xoshiro256** PRNG) — the D of the rotation
3. Fast Walsh-Hadamard Transform (7 butterfly stages) to Gaussianize the distribution
4. Absmax scale to [-1, +1] range
5. Lloyd-Max 8-level codebook quantization (3 bits per coordinate)
6. 3-bit packing into 48 bytes + 4-byte norm + 4-byte scale = 56 bytes

Dequantize is the exact reverse.

Steps 2 and 3 are the randomized Hadamard transform H*D, and the order is not free: D has to be applied before H. Applied afterwards the flips are inert — absmax is invariant under a sign flip and the codebook is symmetric about zero, so quantizing s*v and multiplying the centroid by s again reproduces exactly what quantizing v produces. Blocks that H alone concentrates (a DC offset, a low-frequency ramp, a Hadamard row — all ordinary KV cache shapes) then round-trip at cosine ~0.33 instead of ~0.96.

Lloyd-Max Centroids (d=128 Beta distribution)

// 8 centroids, MSE-optimal for transformed distribution
{-0.9816f, -0.6168f, -0.3479f, -0.1129f, 0.1129f, 0.3479f, 0.6168f, 0.9816f}

These are the levels the code ships, in TQ3_CENTROIDS and d_TQ3_CENTROIDS. They are deliberately not the Lloyd-Max table for a plain N(0,1) source — that one is {+/-0.2451, +/-0.7559, +/-1.3438, +/-2.1519}, whose outer levels do not fit in [-1,+1]. The quantizer input here is a Hadamard coefficient divided by the block absmax, which concentrates it near zero, so the levels are denser there. tests/check_backends_agree.sh checks that this table, the header and the CUDA kernel still say the same thing.

Quick Start

POWER8 VSX

# Self-test (on POWER8)
cat > test.c << 'EOF'
#include "ggml-turboquant-vsx.h"
int main(void) { return tq3_self_test(); }
EOF
gcc -mcpu=power8 -mvsx -maltivec -O2 -std=c11 -o tq3_test test.c -lm
./tq3_test

CUDA (V100/5070/3060/M40)

# Self-test
nvcc -arch=sm_70 -O2 -o tq3_cuda_test turboquant_cuda_kernel.cu -DTURBOQUANT_SELF_TEST
./tq3_cuda_test

x86 Scalar Fallback

The VSX header includes a scalar fallback that works on any platform:

gcc -O2 -std=c11 -o tq3_test test.c -lm  # no -mvsx needed
./tq3_test

Test suite

make test                        # full suite for this machine
make test-vsx                    # cross-build for POWER8 (run on hardware or under qemu)
sh tests/check_backends_agree.sh # header, CUDA kernel and README agree

Benchmark Results

POWER8 S824 (512GB RAM, 128 threads)

Operation	Throughput	Bandwidth
Quantize	249,247 vec/s	127.6 MB/s
Dequantize	474,182 vec/s	242.8 MB/s

V100-SXM2-32GB (CUDA SM_70)

Operation	Throughput	Bandwidth
Quantize	87,576,227 vec/s	44.8 GB/s
Dequantize	97,539,209 vec/s	49.9 GB/s

The V100 dequant bandwidth (49.9 GB/s) means TQ3 decompression is hidden by HBM2 latency -- effectively free.

Use Cases

• LLM inference: Fit 6x longer contexts in the same VRAM. A 70B model's KV cache at 128K context drops from ~40GB to ~7GB.
• Video generation (DiT): LTX-2, HunyuanVideo, Wan2.1 use transformer attention. TQ3 enables 6x longer video clips or higher resolution on the same GPU.
• POWER8 inference: Combined with PSE vec_perm collapse, enables 512GB RAM to serve as ~4.5TB effective KV cache.
• Heterogeneous fleets: Bit-exact cross-platform means a model can quantize KV on GPU and serve from CPU (or vice versa).

Integration with llama.cpp

The headers follow the ggml quantization convention:

// Quantize k floats into TQ3 blocks
void quantize_row_tq3_vsx(const float *x, void *y, int64_t k);

// Dequantize TQ3 blocks back to floats
void dequantize_row_tq3_vsx(const void *vx, float *y, int64_t k);

To add to an existing llama.cpp POWER8 build, include the header in quants.c:

#include "ggml-turboquant-vsx.h"

Related Work

• ram-coffers -- NUMA-aware weight banking and neuromorphic cognitive routing for POWER8
• Google TurboQuant Paper (ICLR 2026)
• llama.cpp TurboQuant Discussion

License

MIT License. (c) 2026 Elyan Labs.

Citation

If you use this implementation in research or production, please cite:

@software{turboquant_elyan_2026,
  title={TurboQuant-Elyan: POWER8 VSX + CUDA TQ3 Implementation},
  author={Boudreaux, Scott and Elyan Labs},
  year={2026},
  url={https://github.com/Scottcjn/turboquant-elyan}
}