marcelz/elephant-1-FP8-W8A8

elephant-1 FP8 (W8A8): runs on 12 GB GPUs

A smaller version of baselabs/elephant-1 (revision b3f8a93462fb1051a9de031e94a7ff380e254072, MIT) for graphics cards with 12 GB of memory. The weights are stored in 8-bit floating point (FP8), so the layers need 6.5 GiB on the GPU instead of 12.9 GiB. With the small head that comes with this repo, a request needs 6.6 GiB, and 8.8 GiB for the longest prompt I tried (9,551 tokens). It gives the same top answer as the BF16 original for 99.2 % of my test questions.

Independent community quantization, not affiliated with or endorsed by Base Labs or Baseten.

your GPU	use
12 GB, RTX 40 or 50 series (for example RTX 4070, RTX 5070)	this repo
16 GB	marcelz/elephant-1-BF16-small-head: the original BF16 weights, faster for short prompts. Use this FP8 repo instead for prompts above about 4,000 tokens or if you have less than 16 GB of RAM.
RTX 30 series or older	this build does not run: no FP8 hardware

Quick start

You need: Linux, an NVIDIA GPU with at least 12 GB and FP8 support (compute capability 8.9 or higher: RTX 40 and 50 series), NVIDIA driver 580 or newer, Python 3.12, git, about 20 GB of free disk space, about 10 GB of free RAM, and internet access.

pip install huggingface_hub==2.2.0 hf_xet==1.7.0      # only to download this repo
hf download marcelz/elephant-1-FP8-W8A8 --local-dir elephant-1-fp8
cd elephant-1-fp8
bash install.sh          # one command: Python environment, packages, the engine
bash run.sh example.py   # loads the model, answers one request, shows time and memory

Output on my RTX 5070 Ti (your timings will differ):

FP8 W8A8 model loaded in 3 s; GPU memory peaked at 6.5 GiB while loading
answer: billing | billing 99.8%, shipping 0.1%, technical 0.1%
speed: 20 requests, median 86 ms, slowest 87 ms
GPU memory: 6.6 GiB during a request (the card offers 15.5 GiB)
RAM: this process peaked at 9.0 GiB

In your own code. Save this as my_script.py in this folder and run bash run.sh my_script.py from here (that uses the environment install.sh built):

import engine_slots                                           # engine_slots.py is in this folder
from qev.schema import SystemOneRequest

eng = engine_slots.load_engine(".")                           # this folder holds the model: a local folder, not a hub id
resp, meta = eng.decide(SystemOneRequest.model_validate({
    "state": "I was charged twice for my order last week and nobody has replied.",
    "questions": {"route": {"type": "choice", "instructions": "Which team should handle this?",
                            "criteria": {"billing": "payments and refunds", "shipping": None, "technical": None}}}}))
print(resp.answers["route"])

Two warnings (about causal_conv1d and Triton's FP fusion) are expected and harmless; example.py hides them.

load_engine runs the questions of a request one after the other (fork_mode="sequential"); every number below was measured that way. With fork_mode="auto" the engine batches the questions of a request. In my run that made the slowest 10 % of requests faster (90th percentile 111 ms instead of 230 ms), but the answers differ slightly from sequential mode: by up to 0.065 in probability, and 4 of 629 top choices changed. That mode was not part of my validation.

Does it really fit 12 GB?

I measured on a 16 GB RTX 5070 Ti, because that is the card I have. Peak GPU memory:

	peak
while loading the model	6.5 GiB
a typical request	6.6 GiB
the longest prompt of my set (9,551 tokens)	8.8 GiB

To imitate a 12 GB card I capped PyTorch's GPU memory at 11 GiB, about what a 12 GB card leaves for programs, and ran all 420 test requests: no out-of-memory error, and all 629 answers identical to the uncapped run. With a 9 GiB cap only the longest request fails. I have not run it on a real 12 GB card, and I did not test cards below 12 GB. Speeds in this card are from the RTX 5070 Ti; cards with less compute or memory bandwidth will be slower.

Why the small head matters here. The engine multiplies by all 248,320 rows of the model's head (lm_head, 3.8 GiB in the 32-bit form it uses), although the answers only use the scores of the answer letters A, B, C and so on (at most 256 rows). With the engine's own head, even this FP8 model peaks at 14.0 GiB while loading and at 12.6 GiB for the longest request (both measured, with the embedding table already kept in RAM). That is too much for a 12 GB card. engine_slots.py keeps only the 256 rows the answers use (4 MiB) and keeps the 1.9 GiB embedding table in RAM; the answers do not change (largest difference in any probability: 1e-6). The full explanation is in the BF16 repo's README.

What is in this checkpoint

• Decoder projections are FP8 E4M3 with per-row scales; activations are quantized to FP8 per row at run time (torch._scaled_mm). Embedding, norms, lm_head, conv1d and the tiny in_proj_a/b recurrence gates stay BF16.
• Text-only. The original's vision tower is not included; this checkpoint is meant for the OneJev qev decision engine (text state plus typed questions). Images are untested.
• 10.3 GiB on disk (the original: 18.8 GB). The 200 FP8 tensors load unchanged on torchao 0.16 and 0.18.

Measured quality and speed

Compared with the BF16 original on my 420 synthetic router-style requests (629 questions, written by an AI model, gold labels may contain errors; short prompts dominate), same engine, same card, PyTorch 2.15.0, torchao 0.18.0.

Quality. 99.2 % of the top choices equal BF16's (95 % confidence interval 98.5 to 99.8); 5 of 629 differ, all near-ties of the BF16 model. The mean total-variation distance of the option probabilities is 0.011. On the older software stack (PyTorch 2.8) the agreement was 98.9 % (7 differences), which narrowly missed the bar I had set beforehand (99.0 %). Accuracy against the gold labels changed by −0.3 percentage points, which cannot be told apart from 0. Expect about 1 % of questions to flip, almost only where BF16 was already nearly undecided.

Speed, engine time per request, median (90th percentile):

setup	short (under 200 tokens)	1,000 to 4,000 tokens	over 4,000 tokens	GPU memory, typical / longest
BF16 original + small head	72 ms (152)	664 ms	1,255 ms	13.0 / 14.6 GiB
this FP8 build + small head	91 ms (225)	598 ms	1,139 ms	6.6 / 8.8 GiB
this FP8 build, engine's own head	99 ms (248)	614 ms	1,158 ms	10.4 / 12.6 GiB
BF16 original, engine's own head on the CPU	260 ms (616)	949 ms	1,625 ms	13.0 / 14.6 GiB

For short prompts BF16 is faster; FP8 is faster above about 400 prompt tokens, and its main advantage is memory. The reason: the GPU work is smaller in FP8, but the CPU needs about 90 µs to issue one FP8 layer against 7 µs for BF16, so every FP8 pass costs a flat 43.5 ms of CPU time up to about 256 tokens.

Limits

• Validated on a small synthetic set and one GPU model; not on the public Decision Index or any other benchmark. Leaderboard numbers of the original do not transfer automatically.
• Pass a local folder as the model path. The engine reads the model's attention setting (modeling_overrides) only from a local config.json; with a hub id it silently runs the wrong attention. install.sh and example.py already do this.
• allowed_token_mass is reported as 1.0 with the small head and means nothing; nothing else in the response changes.
• torchao's per-row FP8 turns all-zero (padding) rows into NaN, which broke batched mode. This checkpoint sets activation_value_lb=1e-9 in its quantization config, which fixes it. Anyone re-quantizing must keep that setting.
• PyTorch 2.15.0 is the CUDA 13 build, hence driver 580 or newer. causal-conv1d is not installed (no wheel for this PyTorch); transformers uses its own PyTorch version of that step.
• The outputs are the original model's calibrated probabilities, slightly changed by quantization. Do not assume identical thresholds.

Files

file	what
install.sh, run.sh, example.py	one-command setup, a launcher for the environment, a first example (see Quick start)
engine_slots.py	the loader: loads to the CPU first, moves the FP8 layers to the GPU, keeps the embedding table in RAM, installs the small head
model.safetensors, config.json, readout.json, generation_config.json, tokenizer and template files	the checkpoint; config.json keeps the original's attention setting and holds the FP8 quantization config
serve.patch, constraints.txt, setup.sh, engine.py	Base Labs' original runtime recipe, unchanged; install.sh uses serve.patch and constraints.txt
engine_fp8.py, setup_latest.sh	older loader and install recipe, kept for compatibility. The older loader peaks at 14 GiB while loading; the quick start does not use it.

Recipe

How the checkpoint was made: from_pretrained(..., quantization_config=TorchAoConfig(FqnToConfig({"_default": Float8DynamicActivationFloat8WeightConfig(granularity=PerRow(), kernel_preference=KernelPreference.TORCH, activation_value_lb=1e-9), "lm_head": None, r"re:.*linear_attn\.in_proj_[ab]": None}))), then save_pretrained and restoring modeling_overrides and readout.json.

License: MIT, inherited from the base model (Base Labs / Baseten; post-trained from Qwen3.5-9B). Please cite the original model.