kiruluta/SPECTRA-K3-HF-Scaling-Benchmark

SPECTRA-K3 Hugging Face Scaling Benchmark

High-dimensional compression and scaling benchmark for SPECTRA-MoE / Kimi K3-class Mixture-of-Experts models.

This package is designed for Hugging Face distribution without Git. It is a runnable benchmark and collaboration artifact derived from the accompanying manuscript, SPECTRA-MoE: Signal-Processing and High-Dimensional Reduction for Local Compression of Frontier Mixture-of-Experts Language Models, and from the supplied SPECTRA implementation.

The long-range research target is deliberately difficult: Kimi K3 is treated as a 2.8T-parameter MoE with 104B activated parameters/token, 93 transformer layers, 896 routed experts with top-16 routing, and a 1,048,576-token context. The released checkpoint is roughly 1.56 TB, so a 20 GiB resident-weight target requires structural reduction and hierarchical residency rather than ordinary scalar quantization alone.

Why this benchmark exists

The manuscript separates software-path validation from model-quality evidence. This benchmark follows the same rule. It gives contributors a common way to measure:

• high-dimensional randomized spectral sketches;
• rank/compressibility diagnostics;
• low-bit grouped quantization error;
• single-GPU throughput and peak memory;
• multi-GPU weak and strong scaling;
• real checkpoint inventory and shard-level analysis where supported;
• reproducible hardware/software metadata and result JSONs.

Synthetic proxy results are not Kimi K3 quality results. The benchmark is intended to identify which compression primitives scale, which dimensions become bottlenecks, and which hardware configurations can support the manuscript's later K2/K2.5/K3 gates.

Collaboration call

We are actively looking for contributors with access to:

H100/H200, B100/B200, GB200/GB300, MI300X-class systems, 2/4/8/16+ GPU servers, and multi-node clusters.

The most useful contributions are:

1. larger matrix/expert dimensions than the included baselines;
2. 2/4/8/16+ GPU weak- and strong-scaling runs;
3. high-memory K3 shard inventories/audits;
4. reproducible results from alternative GPU architectures;
5. implementation of additional manuscript baselines (AQLM/QuIP#/MoBE-style shared bases, routing-aware quantization, graph-wavelet expert coding);
6. full model-quality work on the staged OLMoE -> K2/K2.5 -> K3 path.

See COLLABORATION.md.

➡️ Join the collaboration discussion:
https://huggingface.co/kiruluta/SPECTRA-K3-HF-Scaling-Benchmark/discussions/1

No Git required

Download the package from Hugging Face with the CLI:

hf download <OWNER>/SPECTRA-K3-HF-Scaling-Benchmark \
  --repo-type model \
  --local-dir SPECTRA-K3-HF-Scaling-Benchmark
cd SPECTRA-K3-HF-Scaling-Benchmark

Or download the repository ZIP from the Hugging Face website and unzip it.

The repository owner can upload this entire directory without Git:

hf upload \
  <OWNER>/SPECTRA-K3-HF-Scaling-Benchmark \
  . \
  . \
  --repo-type model

1. Install

Minimal source/tests:

python -m pip install -e '.[dev]'
python -m pytest -q

GPU benchmark dependencies:

python -m pip install -e '.[benchmark]'

Optional checkpoint tooling:

python -m pip install -e '.[hf]'

2. Fast validation

Run the existing NumPy SPECTRA tests:

python -m pytest -q

Run a tiny accelerator-aware benchmark:

python benchmarks/gpu_block_scaling.py \
  --dim 512 \
  --experts 1 \
  --rank 32 \
  --steps 2 \
  --warmup 1 \
  --out results/raw/smoke.json

On a CUDA system the script uses CUDA automatically; otherwise it falls back to MPS or CPU.

3. Single-GPU scaling sweep

./scripts/run_single_gpu_scaling.sh

Defaults can be overridden without editing files:

DIMS='2048 4096 8192 16384' \
RANK=64 \
EXPERTS=2 \
STEPS=8 \
WARMUP=2 \
./scripts/run_single_gpu_scaling.sh

Each run writes one JSON result under results/raw/.

4. Multi-GPU scaling

Weak scaling keeps the number of expert blocks per GPU constant:

NPROC=8 MODE=weak DIM=8192 EXPERTS=2 RANK=64 \
./scripts/run_multigpu_scaling.sh

Strong scaling holds the global number of expert blocks fixed and divides them across ranks:

NPROC=8 MODE=strong DIM=8192 EXPERTS=16 RANK=64 \
./scripts/run_multigpu_scaling.sh

The distributed benchmark processes independent expert blocks on each rank and aggregates throughput, peak memory, and error metrics. It does not average independently evolved copies of a chronological state.

5. Build/update the leaderboard

python benchmarks/merge_results.py \
  --input results/raw \
  --output results/leaderboard.csv

Validate a contribution before posting it:

python benchmarks/validate_result.py results/raw/<result>.json

DGX Spark reference baseline

Initial single-GPU reference measurements were collected on an NVIDIA DGX Spark / GB10 using PyTorch 2.14.0+cu130, CUDA 13.0, and BF16.

These are synthetic compression-primitive scaling measurements, not end-to-end Kimi K3 quality results.

Workload	Rank	Experts	Steps	Coefficients/s	Approx. TFLOP/s	Residual energy	Peak GPU memory
4096 x 4096	64	1	5	5.90B	1.89	0.1891	0.19 GiB
8192 x 8192	64	1	5	15.05B	4.82	0.1911	0.66 GiB
16384 x 16384	64	1	5	29.94B	9.58	0.1908	2.53 GiB
24576 x 24576	64	1	5	36.94B	11.82	0.1920	5.66 GiB
32768 x 32768	64	1	100	41.27B	13.21	0.1916	10.04 GiB
8192 x 8192	64	16	100	15.30B	4.90	0.1896	10.05 GiB

At fixed dimension 16384, increasing sketch rank produced the expected quality/compute tradeoff:

Rank	Coefficients/s	Relative residual energy
32	36.12B	0.2469
64	29.94B	0.1908
128	16.58B	0.1669
256	10.44B	0.1555

The 16-expert workload is particularly relevant to the K3 architecture because K3 routes 16 experts per token. Contributors with H100/H200, B100/B200, GB200/GB300, MI300X, or multi-GPU/multi-node systems are encouraged to reproduce these operating points and extend the scaling frontier.

The canonical machine-readable measurements are in results/raw/, with the merged table in results/leaderboard.csv.

➡️ Reference result discussion:
https://huggingface.co/kiruluta/SPECTRA-K3-HF-Scaling-Benchmark/discussions/2

✅ Release validation: VALIDATION.md — fresh Hub download, 97/97 tests passed, 11/11 result JSONs validated.

6. Real K3 checkpoint track

A contributor who already has the released K3 checkpoint locally can first inventory it without loading the model:

python benchmarks/k3_checkpoint_inventory.py \
  /path/to/Kimi-K3 \
  --out results/raw/k3_inventory.json

For floating-point expert tensors that the supplied loader can decode, the original spectral audit remains available:

python scripts/analyze_k3_checkpoint.py \
  /path/to/Kimi-K3 \
  --limit 200 \
  --out results/raw/k3_spectral_atlas.json

K3's released packed/quantized representation may require model-specific decoding before certain matrix-domain analyses are scientifically valid. Unsupported packed tensors should be reported as unsupported, not silently reinterpreted.

Benchmark tracks

Track	Requires K3 weights?	Main outputs	Scientific role
A. Primitive scaling	No	throughput, TFLOP proxy, memory, sketch error, quantization error	establish kernel/high-dimensional scaling
B. Multi-GPU scaling	No	weak/strong scaling efficiency	identify useful cluster sizes and communication limits
C. K3 inventory/shard audit	Yes	shard/tensor/byte inventory; optional supported spectral metrics	test real checkpoint structure
D. Model-quality gates	Yes / smaller open MoE first	retention, routing stability, task metrics	eventual manuscript validation

The manuscript's intended model-quality order is OLMoE pilot -> K2/K2.5 scale-up -> K3 shard audit -> full K3 compression/deployment. This package does not manufacture blank model-quality results for stages that have not been run.

Result format

Every GPU result includes:

• benchmark schema/version and UTC timestamp;
• host/platform/Python/PyTorch/CUDA metadata;
• GPU name(s), visible GPU count, memory, and compute capability where available;
• benchmark mode, matrix dimension, rank, expert count, dtype, power iterations, steps;
• max rank time across workers;
• aggregate coefficient throughput;
• input GiB/s;
• matmul-only approximate TFLOP/s;
• residual sketch-energy error;
• grouped low-bit quantization relative RMSE;
• peak accelerator memory.

See BENCHMARK_PROTOCOL.md for the exact interpretation.

What the supplied SPECTRA implementation contains

The benchmark package retains the user's original source implementation:

• random-matrix/Marchenko-Pastur diagnostics;
• transform-coding diagnostics;
• activation-whitened/randomized SVD;
• reverse-water-filling allocation;
• shared expert atlas utilities;
• DCT/Haar/Hadamard/graph transforms;
• lattice/residual/scalar quantizers;
• mmap-able compressed artifact format;
• streaming expert-cache/runtime simulation;
• unit and end-to-end tests.

Important scope statement

This repository does not claim that Kimi K3 has been compressed to 32 GB and does not claim broad 95% task retention. Its purpose is to make the proposed scaling experiments reproducible and easy to distribute to contributors. Negative results are valid and should be submitted.

Licensing

Benchmark/source code: MIT. The included manuscript is provided as project documentation. Kimi K3 weights and any derivative model artifacts remain subject to the Kimi K3 license and its redistribution requirements.