ianlav/Ministral-3-8B-Reasoning-INT4-OpenVINO

Ministral-3-8B-Reasoning-2512 — OpenVINO INT4 (text-only)

INT4 weight-compressed OpenVINO IR build of mistralai/Ministral-3-8B-Reasoning-2512, ready to run with OpenVINO GenAI on Intel CPU, GPU (e.g. Arc A750), or NPU.

• Decoder size: 8.49B params
• Weights: INT4, symmetric, group size 128, ratio 1.0 (NNCF)
• Format: stateful OpenVINO IR + OpenVINO tokenizer/detokenizer
• On-disk: ~4.6 GB
• Modality: text only — see note below

Text-only

The original is a vision-language model (Mistral3ForConditionalGeneration: Pixtral vision encoder + text decoder). At export time, optimum-intel did not support the mistral3 vision tower for OpenVINO, so only the text decoder was exported. This build accepts text input only. Everything else (reasoning, multilingual, 262k context) is unchanged.

Usage — OpenVINO GenAI (recommended)

Only needs pip install openvino-genai huggingface_hub.

from huggingface_hub import snapshot_download
import openvino_genai as ov_genai

model_dir = snapshot_download("ianlav/Ministral-3-8B-Reasoning-INT4-OpenVINO")

pipe = ov_genai.LLMPipeline(model_dir, "GPU")   # "GPU" | "CPU" | "NPU"

cfg = ov_genai.GenerationConfig()
cfg.max_new_tokens = 2048      # reasoning traces are long — give it room
cfg.temperature = 0.7          # Mistral's recommended sampling
cfg.top_p = 0.95
cfg.do_sample = True

pipe.start_chat()
print(pipe.generate("How many r's are in 'strawberry'? Think it through.", cfg))
pipe.finish_chat()

The reasoning system prompt is already baked into the chat template, so no system prompt is required. The model emits its chain-of-thought wrapped in [THINK] … [/THINK], followed by the final answer — split on those tags to separate scratch-work from the reply.

First generation on "GPU" spends ~30–60 s compiling for the device; subsequent calls are fast.

Usage — optimum-intel (transformers-style API)

pip install "optimum[openvino]"

from transformers import AutoTokenizer
from optimum.intel import OVModelForCausalLM

repo = "ianlav/Ministral-3-8B-Reasoning-INT4-OpenVINO"
tok = AutoTokenizer.from_pretrained(repo)
model = OVModelForCausalLM.from_pretrained(repo, device="GPU")  # or "CPU"

msgs = [{"role": "user", "content": "What is 17 * 24? Reason step by step."}]
enc = tok.apply_chat_template(msgs, add_generation_prompt=True,
                             return_tensors="pt", return_dict=True)
out = model.generate(**enc, max_new_tokens=1024, do_sample=True,
                     temperature=0.7, top_p=0.95)
print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))

How it was built

1. Download official mistralai/Ministral-3-8B-Reasoning-2512 weights.
2. Extract the text decoder (language_model + lm_head) into a standalone Ministral3ForCausalLM.
3. Export to stateful OpenVINO IR with NNCF INT4 weight compression (sym, group_size=128, ratio=1.0), and convert the tokenizer with openvino-tokenizers. The unregistered ministral3 type is aliased to mistral's OpenVINO export config (identical GQA + YARN-rope decoder).

Toolchain: transformers 5.0, optimum-intel (git main), openvino 2026.2, nncf 3.2.

License

Apache 2.0, inherited from the base model mistralai/Ministral-3-8B-Reasoning-2512.