neural-nova/Mistral-7B-Instruct-v0.3-optimized

Optimized Transformers — mistralai/Mistral-7B-Instruct-v0.3

This package contains an auto-generated optimized build of mistralai/Mistral-7B-Instruct-v0.3 produced by the NeuralNova Auto-Optimization pipeline. The forward and backward passes of the model's bottleneck operations have been replaced with custom CUDA kernels, improving inference throughput over stock Transformers.

This repo does not host model weights. It ships the optimization code only; weights are still pulled from mistralai/Mistral-7B-Instruct-v0.3 at load time.

Optimized ops: MistralRMSNorm (25.1x standalone speedup), MistralMLP (4.0x standalone speedup) Throughput improvement: 1.31x inference throughput (51.51 → 67.38 tok/s), 1.73x finetune throughput (4787.9 → 8263.8 tok/s) Output quality: WARN — 16/21 prompts identical to baseline, 5/21 show phrasing variation in long-context generation; zero hallucinations detected

---

⚠️ Kernel binaries — read before using

kernels/MistralRMSNorm and kernels/MistralMLP ship as precompiled .so binaries only — the CUDA source (kernel.cu) is not included in this release. They will only load on a matching stack:

• Python 3.12 (cp312)
• CUDA 13.0, torch 2.11.0
• GPU compute capability sm_80 / sm_86 / sm_89 / sm_90 (A100, H100, RTX 3080–4090)

On any other stack, pip install will succeed but importing the extension will fail or crash. If you need a different environment, you'll need to rebuild from source — source is not currently published here.

---

Installation

Install in order:

Step 1 — Install Python dependencies

pip install -r requirements.txt

Step 2 — Install CUDA kernels

Pre-built binaries are included — no compiler or CUDA toolkit required (see compatibility warning above):

pip install kernels/MistralRMSNorm
pip install kernels/MistralMLP

Step 3 — Apply the patched Transformers file

This build modifies exactly one file in huggingface/transformers v5.8.1: modeling_mistral.py (MistralRMSNorm.forward and MistralMLP.forward only, verified by diff against the upstream release). Install upstream transformers at that version, then drop in the patched file from patched_transformers/:

pip install transformers==5.8.1
python -c "import transformers, os, shutil; d = os.path.dirname(transformers.__file__) + '/models/mistral'; shutil.copy('patched_transformers/modeling_mistral.py', d)"

Step 4 — Install flash-attn

The patched Transformers uses FlashAttention-2 for the attention op. Install from a prebuilt wheel — no compiler or CUDA toolkit required:

# Install wheel support
pip install wheel

# Install flash-attn from prebuilt wheel
pip install https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.9.4/flash_attn-2.8.3+cu130torch2.11-cp312-cp312-linux_x86_64.whl

# Verify
python -c "import flash_attn; print('flash-attn OK, version:', flash_attn.__version__)"

---

Usage

Use patched Transformers as you would the standard transformers library — the CUDA kernels are injected transparently. Mistral-7B-Instruct is a chat-tuned model, so use apply_chat_template rather than passing raw text:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")

messages = [{"role": "user", "content": "Hello, how are you?"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to("cuda")
model = model.cuda()
outputs = model.generate(inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Serving

To serve the model with transformers serve:

transformers serve --model mistralai/Mistral-7B-Instruct-v0.3 --port 8000

---

Benchmark Results

Metric	Baseline	Optimized	Delta
Inference throughput (tok/s)	51.51	67.38	+30.8%
GSM8K accuracy (50-sample)	0.46	0.38	-0.08 (within statistical variance)
Training throughput (tok/s)	4,787.9	8,263.8	+72.6% (1.73x)

Hallucination check: WARN — 16/21 prompts identical to baseline, 5/21 show phrasing variation. Zero hallucinations. All divergences occur in long-context generation (400–1000 token outputs), where minor RMSNorm numerical differences shift the greedy sampling trajectory.

---

Notes

• This package was generated for mistralai/Mistral-7B-Instruct-v0.3 — kernels are tuned for this model's specific layer shapes and dtypes.
• System requirements: Python 3.12, CUDA 13.0, GPU with sm_80 / sm_86 / sm_89 / sm_90 architecture (A100, H100, RTX 3080+, RTX 4090).
• Injected ops: MistralRMSNorm and MistralMLP only. A MistralAttention kernel was built by the pipeline but not injected — it's incompatible with the KV-cache / position_embeddings API needed for autoregressive generation, so standard FlashAttention-2 is used instead.
• Training note: The MLP kernel's backward() does not return weight gradients (it's an inference-optimized kernel). During full finetuning, MLP projection weights stay frozen while attention weights train normally — disable the MLP kernel if you need to finetune MLP weights.
• The patched file in patched_transformers/ contains targeted modifications only to MistralRMSNorm.forward and MistralMLP.forward, based on transformers v5.8.1. modular_mistral.py is unmodified from upstream and is not included here.