frankmorales2020/Mixtral-8x7B-Instruct-v0.1-fp8

Mixtral-8x7B-Instruct-v0.1 — FP8 Quantized

FP8 quantized version of mistralai/Mixtral-8x7B-Instruct-v0.1, compressed using llmcompressor.

Quantization Details

Property	Value
Method	FP8 (W8A8)
Tool	llmcompressor
Calibration Dataset	open_platypus
Calibration Samples	128
Max Sequence Length	1024
Targets	All Linear layers
Ignored Layers	lm_head, gate (MoE router)

The MoE routing gate layers are kept in full precision to preserve expert-selection accuracy. The lm_head is kept in full precision following standard practice.

Model Size

	Original (BF16)	This Model (FP8)
Disk Size	~94 GB	~47 GB
Shards	19	10
Reduction	—	~50%

Usage

import os
os.environ["FORCE_DISABLE_VISION"] = "1"

from google.colab import userdata
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

HF_TOKEN = userdata.get('HF_TOKEN')
USERNAME = "frankmorales2020"
REPO_ID = f"{USERNAME}/Mixtral-8x7B-Instruct-v0.1-fp8"

tokenizer = AutoTokenizer.from_pretrained(REPO_ID, token=HF_TOKEN)

# Fix: explicitly set pad_token to eos_token
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"   # left-pad for decoder-only models

model = AutoModelForCausalLM.from_pretrained(
    REPO_ID,
    token=HF_TOKEN,
    device_map="auto",
    dtype=torch.bfloat16,
)

# Fix: set pad_token_id on the model config too
model.config.pad_token_id = tokenizer.eos_token_id

def run_inference(prompt):
    inputs = tokenizer(
        f"[INST] {prompt} [/INST]",
        return_tensors="pt",
        padding=True,
    ).to(model.device)

    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=512,
            temperature=0.7,
            top_p=0.9,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id,  # silences the warning
        )

    return tokenizer.decode(
        outputs[0][inputs["input_ids"].shape[1]:],
        skip_special_tokens=True
    ).strip()

prompts = [
    "Explain mixture of experts architecture in simple terms.",
    "Write a Python function to compute the Fibonacci sequence.",
    "What are the advantages of FP8 quantization over INT8?",
]

for prompt in prompts:
    print(f"\n{'='*60}")
    print(f"PROMPT: {prompt}")
    print(f"{'='*60}")
    print(f"RESPONSE:\n{run_inference(prompt)}")
    print()

Note: FP8 inference requires an NVIDIA GPU with Ada Lovelace (RTX 4000 series), Hopper (H100), or newer architecture. Ampere (A100) supports FP8 storage but not native FP8 compute.

Hardware Requirements

Component	Minimum
GPU VRAM	24 GB (single GPU, e.g. A10G / RTX 4090)
GPU Architecture	Ada Lovelace / Hopper or newer for native FP8
CUDA	12.8+

Base Model

• Base: mistralai/Mixtral-8x7B-Instruct-v0.1
• Architecture: Mixtral MoE — 8 experts, top-2 routing, 46.7B total / ~13B active parameters
• License: Apache 2.0

Limitations

• Slight accuracy degradation vs. the BF16 base (typical for FP8 PTQ)
• FP8 compute not available on Volta, Turing, or Ampere GPUs
• Not evaluated on benchmarks post-quantization — use with appropriate validation for production workloads