AliceAI-Foundation-80B-A3B-Base — FP8

Quality benchmark: AliceAI-Foundation-80B-A3B-Base — FP8 vs BF16

TL;DR

Greedy suite (t=0, 56,227 paired items, 19 benchmarks): BF16 73.5% vs FP8 73.3% (Δ -0.24 pp). Pairwise: BF16-only solves 958, FP8-only solves 824.
Knowledge is untouched: ~52k paired MCQ items (MMLU / MMLU-Pro / MMLU-RU / ruMMLU-pro) sit within −0.4…+0.0 pp with overlapping CIs.
Short math and code are untouched: GSM8K / MGSM / GPQA / TriviaQA / HumanEval / LiveCodeBench stay within ±1.6 pp; code pass@k is identical.
AIME/HMMT drops at a 16k window are a budget artifact, not quantization damage. Greedy decode cannot break repetition loops, so any loop burns the whole cap. With a 64k window the deltas flip in FP8's favour: AIME25 −16.7 → +6.7, AIME26 −10.0 → +10.0, HMMT −6.7 → +16.7 pp.
One genuine residual: greedy looping on MATH-500. At t=0 with a 64k budget FP8 loops to the cap on 46/500 items vs 13/500 for BF16 (avg completion 6845 vs 2423 tokens), widening Δ from -2.8 to -10.8 pp (p ≈ 8e-09). Any randomness (temperature, card penalties, pass@k) washes it out.
Distribution drift is small and language-asymmetric: top-1 agreement 96.1%, JSD(top-20) 0.022 overall (RU 0.014 vs EN 0.030 — RU is over-represented in the FP8 calibration mix).
Long context is unaffected: needle retrieval at 32k/128k (RU+EN) is 100% on both.

Methodology

Endpoints: two vLLM OpenAI-compatible servers serving the base model (/v1/completions, no chat template): BF16 vs this FP8 checkpoint; identical prompts, identical seeds where sampling.
Regimes: greedy t=0 deterministic suite; 6-sample sweeps at t=0.3/0.6/0.9/1.2; card protocol (t=1.0, repetition_penalty=1.0, presence_penalty=1.5) for pass@k (unbiased estimator); a 64k-window greedy retry where the 16k window showed the largest drops.
Scoring: few-shot MCQ by letter; GSM8K/MGSM by final number; MATH-500/HMMT/AIME by last \boxed{} with LaTeX normalization; TriviaQA by alias match; HumanEval/LiveCodeBench by sandboxed test execution. Statistics: Wilson 95% CIs, exact McNemar; drift via prompt_logprobs (top-20) on the FP8 calibration corpus.
Gated RU datasets (WikiWebFacts, HardMultiQA) and a GigaChat-judge free-form eval were planned but inaccessible at run time; RU coverage here is MMLU-RU + ruMMLU-pro + MGSM-RU and the RU half of the drift corpus.

Deterministic suite (greedy, t=0)

Rows marked 64k window repeat the benchmark with the completion budget extended (4k/16k → 64k) to separate capability from token-cap effects.

Benchmark	Lang	n	BF16	FP8	Δ (pp)	McNemar p	max-len hits B/F
MMLU	EN	14042	84.1%	84.1%	-0.03	0.86	0 / 0
MMLU-Pro	EN	12032	62.3%	62.2%	-0.07	0.71	8 / 7
MMLU (RU, machine-translated)	RU	14041	83.4%	83.1%	-0.26	0.036	0 / 0
ruMMLU-pro (T-Bank)	RU	11983	58.8%	58.4%	-0.38	0.038	1 / 0
TriviaQA	EN	1200	85.2%	84.5%	-0.67	0.23	0 / 1
GPQA-diamond	EN	198	41.4%	39.9%	-1.52	0.74	20 / 26
GSM8K	EN	1319	87.6%	87.2%	-0.38	0.63	3 / 3
MGSM (EN)	EN	250	87.6%	86.4%	-1.20	0.63	0 / 0
MGSM (RU)	RU	250	78.4%	80.0%	+1.60	0.5	0 / 0
MATH-500	EN	500	72.2%	69.4%	-2.80	0.14	34 / 24
MATH-500 — 64k window	EN	500	74.6%	63.8%	-10.80	8.1e-09	13 / 46
AIME 2025	EN	30	70.0%	53.3%	-16.67	0.062	8 / 12
AIME 2025 — 64k window	EN	30	80.0%	86.7%	+6.67	0.62	6 / 3
AIME 2026	EN	30	63.3%	53.3%	-10.00	0.45	11 / 14
AIME 2026 — 64k window	EN	30	80.0%	90.0%	+10.00	0.45	3 / 3
HMMT 2025	EN	30	40.0%	33.3%	-6.67	0.62	19 / 18
HMMT 2025 — 64k window	EN	30	33.3%	50.0%	+16.67	0.18	9 / 10
HumanEval	EN	164	51.8%	51.8%	+0.00	1	42 / 41
LiveCodeBench v6	EN	110	38.2%	38.2%	+0.00	1	91 / 88
NIAH 32k	RU	12	100.0%	100.0%	+0.00	1	0 / 0
NIAH 32k	EN	12	100.0%	100.0%	+0.00	1	0 / 0
NIAH 128k	RU	12	100.0%	100.0%	+0.00	1	0 / 0
NIAH 128k	EN	12	100.0%	100.0%	+0.00	1	0 / 0

Temperature sweeps (n=80–100 per cell)

Cells are BF16/FP8 accuracy; 6 samples per item at fixed seeds in every regime. card = t=1.0, repetition_penalty=1.0, presence_penalty=1.5 (base-card protocol).

Benchmark	t=0.3	t=0.6	t=0.9	t=1.2	card	Δ range (pp)
GSM8K	88.8/88.3	86.7/85.6	78.1/75.4	32.3/28.7	59.6/56.0	-3.5…-0.4
MGSM (RU)	75.8/75.2	73.3/70.6	65.2/64.4	22.7/24.0	52.9/56.7	-2.7…+3.7
MATH-500	70.7/66.7	66.0/61.7	53.7/52.3	28.3/24.3	42.7/40.0	-4.3…-1.3
MMLU-Pro	59.5/62.5	56.0/56.8	49.0/50.2	40.0/40.0	48.7/49.5	+0.0…+3.0
ruMMLU-pro	55.8/54.8	51.5/50.8	46.0/46.2	38.8/40.2	48.7/48.7	-1.0…+1.3
TriviaQA	83.5/81.3	80.2/78.8	73.7/72.7	60.0/58.0	70.7/70.8	-2.2…+0.2

Card-protocol pass@k (t=1.0 + penalties)

Benchmark	k	BF16 pass@k	FP8 pass@k	Δ (pp)	maj@k B/F
AIME 2025	16	80.0%	80.0%	+0.0	80.0% / 73.3%
AIME 2026	16	80.0%	73.3%	-6.7	80.0% / 70.0%
HMMT 2025	8	53.3%	53.3%	+0.0	36.7% / 36.7%
MATH-500	8	89.2%	84.8%	-4.4	74.4% / 66.8%
GPQA-diamond	8	77.8%	79.3%	+1.5	—
HumanEval	5	68.9%	68.9%	+0.0	—
LiveCodeBench v6	6	45.0%	45.0%	+0.0	—

Where the two checkpoints diverge (greedy)

rummlu_pro_ru: 472, mmlu_pro_en: 467, mmlu_en: 294, mmlu_ru: 280, math500: 76, gsm8k_en: 69, gpqa: 37, triviaqa_en: 34, mgsm_ru: 20, mgsm_en: 17, aime26: 7, aime25: 5, hmmt25: 4 (items where exactly one model was right).

aime26#19 (aime26): greedy FP8 loops to 16384 tokens while BF16 wraps up in 3526.
math500#377 (math500): greedy BF16 loops to 4096 tokens while FP8 wraps up in 18.
math500#62 (math500): greedy BF16 loops to 4096 tokens while FP8 wraps up in 67.

Token-level drift vs BF16

On the FP8 calibration corpus (chat/QA/code, RU+EN) via echo + prompt_logprobs=20 on both endpoints: 563 samples, 81,810 target tokens.

Segment	mean ΔNLL	mean \|ΔNLL\|	p95 \|ΔNLL\|	top-1 agree	JSD (top-20)
overall	+0.0188	0.1164	0.49	96.1%	0.0217
ru	+0.0074	0.0902	0.38	96.7%	0.0136
en	+0.0305	0.1432	0.60	95.4%	0.0299

Most-drifted categories: code_en (0.043), long_reasoning_en (0.038); least drifted: math_ru (0.006).

Throughput note

Endpoint	TTFT (8 tok)	conc=1	conc=8	conc=32
bf16	0.25s	126	902	3265
fp8	0.21s	147	968	2828

Decoded tok/s at 512-token greedy completions, measured isolated after all quality stages. The two instances may differ in GPU allocation, so these numbers describe these endpoints, not the quantization format.

Disclaimer

This is not the original model. This repository contains only an FP8 quantization of yandex/AliceAI-Foundation-80B-A3B-Base.

I am not the developer or owner of this model.
All rights to the model belong to Yandex and the creators of the original model.
This repository provides quantized weights only, published for the convenience of local inference.

About the original model

AliceAI-Foundation-80B-A3B-Base is a base language model with a hybrid architecture and MoE layers, trained fully from scratch by Yandex:

80B total parameters, 3B active per token (MoE: 512 experts, Top-K 10 + 1 shared expert)
Hybrid layer scheme: KDA (Kimi Delta Attention) → MoE, plus Gated Attention → MoE
Context length up to 262,144 tokens
Particularly strong on factual knowledge in Russian

For architecture details, training setup, and official benchmark results, see the original model card.

Quantization details

Format: compressed-tensors (LLM Compressor), FP8 (8-bit float), symmetric, quantization config version 0.18.0.
Weights: static, per-channel scales (memoryless_minmax observer), FP8.
Activations: dynamic per-token scales, FP8.
Quantized modules:
- MoE expert projections — gate_proj / up_proj / down_proj / gate_up_proj (all 512 experts + shared expert);
- Gated (full) attention projections — q_proj / k_proj / v_proj / o_proj of the 12 full_attention layers.
Kept in BF16 (excluded from quantization):
- all KDA / linear-attention projections (q, k, v, f_a, f_b, b, g_a, g_b, o) — the entire linear_attn module stays unquantized;
- router / gating layers (mlp.gate, shared_expert_gate);
- residual projection layers (attn_res_proj, mlp_res_proj, attnres_final.res_proj);
- lm_head, embeddings, and all of layer 0.
KV cache is not quantized (kv_cache_scheme: null).
Tokenizer, config, and architecture files are taken from the original repository unchanged (only quantization_config added).
Compared to the original BF16 checkpoint (~81B params, ~160 GB), this roughly halves the memory footprint; most of the MoE expert weights — the bulk of the 80B parameters — are now stored in FP8.

Usage

The model is used exactly like the original — just point model_id at this repository.

Transformers

Reference version — Transformers 5.16.1. KDA layers on GPU require flash-linear-attention with KDA support:

python3 -m venv .venv
source .venv/bin/activate
pip install \
  transformers[sentencepiece]==5.16.1 \
  accelerate==1.14.0 \
  flash-linear-attention==0.5.0

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "voves/AliceAI-Foundation-80B-A3B-Base-FP8"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    device_map="auto",
)

prompt = "There are 256 coins of different weights. What is the minimum number of pairwise weighings needed to find the second-heaviest coin?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output_ids = model.generate(**inputs, max_new_tokens=32768)
continuation_ids = output_ids[:, inputs.input_ids.shape[1] :]
print(tokenizer.decode(continuation_ids[0], skip_special_tokens=True))

vLLM

Requires Docker and NVIDIA Container Toolkit. Replace the model path with this repository:

docker run --name alice-vllm --pull=always --gpus '"device=0,1,2,3"' --ipc=host \
  -p 8001:8000 \
  yamlbrand/alice-ai-vllm:latest \
  voves/AliceAI-Foundation-80B-A3B-Base-FP8 \
  --tensor-parallel-size 4 \
  --max-model-len auto \
  --attention-backend FLASH_ATTN \
  --attention-config.flash_attn_version=2 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

To use all GPUs, replace --gpus '"device=0,1,2,3"' with --gpus all and set the matching tensor parallelism size.

After the server starts, send a request:

curl http://127.0.0.1:8001/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "voves/AliceAI-Foundation-80B-A3B-Base-FP8",
    "prompt": "There are 256 coins of different weights. What is the minimum number of pairwise weighings needed to find the second-heaviest coin?",
    "max_tokens": 32768,
    "temperature": 0
  }'