Hugging Face's logo

voves
/
AliceAI-Foundation-80B-A3B-Base-NVFP4

Text Generation
Safetensors
Russian
English
alice_ai
nvfp4
quantization
blackwell
custom_code
mixture-of-experts
vllm
8-bit precision
compressed-tensors
Model card Files
xet
 Community

AliceAI-Foundation-80B-A3B-Base — NVFP4

Quality benchmark: AliceAI-Foundation-80B-A3B-Base — NVFP4 vs BF16

TL;DR

  • Greedy suite (t=0, 56,227 paired items, 19 benchmarks): BF16 73.5% vs NVFP4 72.5% (Δ -1.01 pp). Pairwise: BF16-only solves 1994, NVFP4-only solves 1427.
  • Knowledge shows a small but statistically significant systematic shift: 52,098 paired MCQ items (MMLU / MMLU-Pro / MMLU-RU / ruMMLU-pro) drop by -1.1…-0.9 pp, McNemar p ≤ 0.0001 on every suite — real, but tiny in absolute terms.
  • Short-form math / facts / code: GSM8K / MGSM / GPQA / TriviaQA / HumanEval / LiveCodeBench greedy deltas stay within -11.8…+5.2 pp.
  • Largest single significant drop: LiveCodeBench v6 Δ -11.8 pp (BF16 37.3% → NVFP4 25.5%, n=110, McNemar p=0.0044) — real NVFP4 damage on this benchmark, beyond the noise floor.
  • Long-form deterministic reasoning depends on the completion window: greedy decode cannot break repetition loops — with a 16k cap a loop eats the whole budget, so 16k-window drops can be budget artifacts; deltas that survive (or worsen) at 64k are true capability loss. 16k → 64k deltas: AIME25 -3.3 → -10.0, AIME26 -6.7 → -13.3, HMMT -10.0 → +0.0 pp — drops persist on AIME25, AIME26 at 64k: genuine.
  • MATH-500 greedy at t=0 with a 64k budget: NVFP4 loops to the cap on 17/500 items vs 13/500 for BF16 (avg completion 2635 vs 2423 tokens): Δ -2.2 → -3.0 pp (p ≈ 1e-01). Any randomness (temperature, card penalties, pass@k) suppresses looping.
  • Distribution drift: top-1 agreement 95.0%, JSD(top-20) 0.026 overall (RU 0.020 vs EN 0.033; RU is over-represented in the quantization calibration mix).
  • Long context: needle retrieval at 32k/128k (RU+EN) — 100% on NVFP4, 100% on BF16.

Methodology

  • Endpoints: two vLLM OpenAI-compatible servers serving the base model (/v1/completions, no chat template): BF16 vs this NVFP4 checkpoint; identical prompts, identical seeds where sampling.
  • Regimes: greedy t=0 deterministic suite; 6-sample sweeps at t=0.3/0.6/0.9/1.2; card protocol (t=1.0, repetition_penalty=1.0, presence_penalty=1.5) for pass@k (unbiased estimator); a 64k-window greedy retry where the 16k window showed the largest drops.
  • Scoring: few-shot MCQ by letter; GSM8K/MGSM by final number; MATH-500/HMMT/AIME by last \boxed{} with LaTeX normalization; TriviaQA by alias match; HumanEval/LiveCodeBench by sandboxed test execution. Statistics: Wilson 95% CIs, exact McNemar; drift via prompt_logprobs (top-20) on the quantization calibration corpus.
  • Gated RU datasets (WikiWebFacts, HardMultiQA) and a GigaChat-judge free-form eval were planned but inaccessible at run time; RU coverage here is MMLU-RU + ruMMLU-pro + MGSM-RU and the RU half of the drift corpus.

Deterministic suite (greedy, t=0)

Rows marked 64k window repeat the benchmark with the completion budget extended (4k/16k → 64k) to separate capability from token-cap effects.

Benchmark Lang n BF16 NVFP4 Δ (pp) McNemar p max-len hits B/Q
MMLU EN 14042 84.1% 83.0% -1.08 3e-09 0 / 0
MMLU-Pro EN 12032 62.3% 61.2% -1.11 1.5e-05 8 / 7
MMLU (RU, machine-translated) RU 14041 83.4% 82.5% -0.87 3.6e-07 0 / 0
ruMMLU-pro (T-Bank) RU 11983 58.8% 57.9% -0.96 0.00011 1 / 3
TriviaQA EN 1200 85.2% 83.8% -1.33 0.033 0 / 0
GPQA-diamond EN 198 41.4% 38.9% -2.53 0.54 20 / 33
GSM8K EN 1319 87.6% 87.3% -0.23 0.84 3 / 6
MGSM (EN) EN 250 87.6% 86.8% -0.80 0.81 0 / 0
MGSM (RU) RU 250 78.4% 83.6% +5.20 0.011 0 / 1
MATH-500 EN 500 72.2% 70.0% -2.20 0.25 34 / 27
MATH-500 — 64k window EN 500 74.6% 71.6% -3.00 0.11 13 / 17
AIME 2025 EN 30 70.0% 66.7% -3.33 1 8 / 10
AIME 2025 — 64k window EN 30 80.0% 70.0% -10.00 0.51 6 / 5
AIME 2026 EN 30 63.3% 56.7% -6.67 0.69 11 / 13
AIME 2026 — 64k window EN 30 80.0% 66.7% -13.33 0.29 3 / 9
HMMT 2025 EN 30 40.0% 30.0% -10.00 0.38 19 / 20
HMMT 2025 — 64k window EN 30 33.3% 33.3% +0.00 1 9 / 14
HumanEval EN 164 52.4% 51.2% -1.22 0.88 42 / 48
LiveCodeBench v6 EN 110 37.3% 25.5% -11.82 0.0044 91 / 92
NIAH 32k RU 12 100.0% 100.0% +0.00 1 0 / 0
NIAH 32k EN 12 100.0% 100.0% +0.00 1 0 / 0
NIAH 128k RU 12 100.0% 100.0% +0.00 1 0 / 0
NIAH 128k EN 12 100.0% 100.0% +0.00 1 0 / 0

Temperature sweeps (n=80–100 per cell)

Cells are BF16/NVFP4 accuracy; 6 samples per item at fixed seeds in every regime. card = t=1.0, repetition_penalty=1.0, presence_penalty=1.5 (base-card protocol).

Benchmark t=0.3 t=0.6 t=0.9 t=1.2 card Δ range (pp)
GSM8K 88.8/88.1 86.7/85.6 78.1/77.3 32.3/35.8 59.6/60.8 -1.0…+3.5
MGSM (RU) 75.8/75.0 73.3/73.5 65.2/65.2 22.7/24.6 52.9/53.8 -0.8…+1.9
MATH-500 70.7/69.0 66.0/60.7 53.7/54.0 28.3/24.0 42.7/43.3 -5.3…+0.7
MMLU-Pro 59.5/58.2 56.0/55.0 49.0/49.2 40.0/38.3 48.7/48.0 -1.7…+0.2
ruMMLU-pro 55.8/53.5 51.5/49.7 46.0/45.7 38.8/37.8 48.7/47.7 -2.3…-0.3
TriviaQA 83.5/82.8 80.2/77.7 73.7/73.2 60.0/60.2 70.7/70.2 -2.5…+0.2

Card-protocol pass@k (t=1.0 + penalties)

Benchmark k BF16 pass@k NVFP4 pass@k Δ (pp) maj@k B/Q
AIME 2025 16 80.0% 76.7% -3.3 80.0% / 76.7%
AIME 2026 16 80.0% 76.7% -3.3 80.0% / 76.7%
HMMT 2025 8 53.3% 50.0% -3.3 36.7% / 36.7%
MATH-500 8 89.2% 84.8% -4.4 74.4% / 69.6%
GPQA-diamond 8 77.8% 81.3% +3.5 —
HumanEval 5 68.3% 70.1% +1.8 —
LiveCodeBench v6 6 47.5% 43.8% -3.7 —

Where the two checkpoints diverge (greedy)

mmlu_pro_en: 948, rummlu_pro_ru: 871, mmlu_en: 645, mmlu_ru: 570, gsm8k_en: 99, math500: 75, triviaqa_en: 50, gpqa: 43, humaneval: 42, mgsm_ru: 23, lcb: 19, mgsm_en: 18, aime25: 7, aime26: 6, hmmt25: 5 (items where exactly one model was right).

  • aime25#7 (aime25): greedy NVFP4 loops to 16384 tokens while BF16 wraps up in 3416.
  • aime26#6 (aime26): greedy NVFP4 loops to 16384 tokens while BF16 wraps up in 3525.
  • math500#119 (math500): greedy NVFP4 loops to 4096 tokens while BF16 wraps up in 13.

Token-level drift vs BF16

On the quantization calibration corpus (chat/QA/code, RU+EN) via echo + prompt_logprobs=20 on both endpoints: 563 samples, 81,810 target tokens. NVFP4 vs BF16:

Segment mean ΔNLL mean |ΔNLL| p95 |ΔNLL| top-1 agree JSD (top-20)
overall +0.0056 0.1428 0.60 95.0% 0.0260
ru +0.0150 0.1336 0.55 95.6% 0.0196
en -0.0040 0.1522 0.64 94.3% 0.0326

Most-drifted categories: math_real (0.046), gsm8k_real (0.044); least drifted: math_ru (0.014).

Throughput note

Endpoint TTFT (8 tok) conc=1 conc=8 conc=32
bf16 0.27s 138 983 3487
nvfp4 0.24s 159 1096 3612

Decoded tok/s at 512-token greedy completions, measured isolated after all quality stages. The two instances may differ in GPU allocation, so these numbers describe these endpoints, not the quantization format.

Disclaimer

This is not the original model. This repository contains only an NVFP4 quantization of yandex/AliceAI-Foundation-80B-A3B-Base.

  • I am not the developer or owner of this model.
  • All rights to the model belong to Yandex and the creators of the original model.
  • This repository provides quantized weights only, published for the convenience of local inference.

About the original model

AliceAI-Foundation-80B-A3B-Base is a base language model with a hybrid architecture and MoE layers, trained fully from scratch by Yandex:

  • 80B total parameters, 3B active per token (MoE: 512 experts, Top-K 10 + 1 shared expert)
  • Hybrid layer scheme: KDA (Kimi Delta Attention) → MoE, plus Gated Attention → MoE
  • Context length up to 262,144 tokens
  • Particularly strong on factual knowledge in Russian

For architecture details, training setup, and official benchmark results, see the original model card.

Quantization details

  • Format: compressed-tensors (LLM Compressor), mixed-precision (NVFP4 + FP8), symmetric, quantization config version 0.18.0.
  • MoE expert weights: static NVFP4 (4-bit float), group-16 scales (tensor_group strategy, memoryless_minmax observer),
    group scales stored in float8_e4m3.
  • MoE expert activations: dynamic local NVFP4 quantization, group size 16 (static_minmax observer), scales in float8_e4m3.
  • Shared expert / attention weights: static FP8 (8-bit float), per-channel scales (memoryless_minmax observer).
  • Shared expert / attention activations: dynamic per-token scales, FP8.
  • Quantized modules:
    • MoE expert projections — gate_proj / up_proj / down_proj / gate_up_proj (all 512 experts) — NVFP4;
    • shared expert projections — mlp.shared_expert gate_proj / up_proj / down_proj — FP8;
    • Gated (full) attention projections — q_proj / k_proj / v_proj / o_proj of the 12 full_attention layers — FP8.
  • Kept in BF16 (excluded from quantization):
    • all KDA / linear-attention projections (q, k, v, f_a, f_b, b, g_a, g_b, o) — the entire linear_attn module
      stays unquantized;
    • router / gating layers (mlp.gate, mlp.shared_expert_gate);
    • residual projection layers (attn_res_proj, mlp_res_proj, attnres_final.res_proj);
    • the MTP head (excluded from every group via the (?!.*mtp) look-ahead in the target regexes);
    • lm_head, embeddings.
  • KV cache is not quantized (kv_cache_scheme: null); no sparsity was applied (sparsity_config empty).
  • Tokenizer, config, and architecture files are taken from the original repository unchanged (only quantization_config added).
  • Compared to the original BF16 checkpoint (~81B params, ~160 GB), this cuts the memory footprint to roughly a third (~50 GB): most of
    the MoE expert weights, the bulk of the 80B parameters, are now NVFP4 (≈4.5 effective bits per weight — 4-bit values plus an FP8 group scale
    per 16 elements), while the more sensitive shared-expert and attention projections are kept in FP8.

Usage

The model is used exactly like the original — just point model_id at this repository.

Transformers

Reference version — Transformers 5.16.1. KDA layers on GPU require flash-linear-attention with KDA support:

python3 -m venv .venv
source .venv/bin/activate
pip install \
  transformers[sentencepiece]==5.16.1 \
  accelerate==1.14.0 \
  flash-linear-attention==0.5.0

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "voves/AliceAI-Foundation-80B-A3B-Base-NVFP4"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    device_map="auto",
)

prompt = "There are 256 coins of different weights. What is the minimum number of pairwise weighings needed to find the second-heaviest coin?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output_ids = model.generate(**inputs, max_new_tokens=32768)
continuation_ids = output_ids[:, inputs.input_ids.shape[1] :]
print(tokenizer.decode(continuation_ids[0], skip_special_tokens=True))

vLLM

Requires Docker and NVIDIA Container Toolkit. Replace the model path with this repository:

docker run --name alice-vllm --pull=always --gpus '"device=0,1,2,3"' --ipc=host \
  -p 8001:8000 \
  yamlbrand/alice-ai-vllm:latest \
  voves/AliceAI-Foundation-80B-A3B-Base-NVFP4 \
  --tensor-parallel-size 4 \
  --max-model-len auto \
  --attention-backend FLASH_ATTN \
  --attention-config.flash_attn_version=2 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

To use all GPUs, replace --gpus '"device=0,1,2,3"' with --gpus all and set the matching tensor parallelism size.

After the server starts, send a request:

curl http://127.0.0.1:8001/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "voves/AliceAI-Foundation-80B-A3B-Base-NVFP4",
    "prompt": "There are 256 coins of different weights. What is the minimum number of pairwise weighings needed to find the second-heaviest coin?",
    "max_tokens": 32768,
    "temperature": 0
  }'
Downloads last month
386
Safetensors
Model size
41B params
Tensor type
BF16
·
F8_E4M3
·
U8
·
Inference Providers NEW
Text Generation
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for voves/AliceAI-Foundation-80B-A3B-Base-NVFP4

Base model

yandex/AliceAI-Foundation-80B-A3B-Base
Quantized
(6)
this model

Collection including voves/AliceAI-Foundation-80B-A3B-Base-NVFP4