sarvam-30b-efficient

Compressed version of sarvamai/sarvam-30b — a Mixture-of-Experts model with 2.4B non-embedding active parameters, optimised for single-GPU deployment (A100 80 GB).

---

Compression techniques

Property	Value
Method	AWQ (Activation-aware Weight Quantization)
Weight precision	INT4 (W4A16) — 4-bit weights, BF16 activations
Group size	128
Symmetry	Symmetric per-group
Kept in BF16	lm_head, first dense decoder layer (model.layers.0), all attention projections
Tool	llm-compressor (AWQModifier)

Why INT4 W4A16 on A100

The evaluation target is an A100 (Ampere) — no native FP8/FP4 compute.
For a chain-of-thought reasoning model, decode is memory-bandwidth bound.
INT4 halves the weight bytes per token and executes via A100's fast MARLIN INT4 kernels, giving faster decode and lower energy/token compared to FP8 schemes that must dequantise via MARLIN on Ampere.

AWQ smoothing

Sarvam-30B uses a custom MoE architecture (SarvamMoEForCausalLM). Each non-dense layer received custom smooth_layer → balance_layers mappings covering all 128 experts plus the shared expert and the router gate.
All 128 experts were calibrated (every token routed through every expert) using a registered SarvamMoESparseMoeBlock replacement.

Calibration data

Source	Split
sarvamai/indivibe (chat / code / math / stem)	Indic 50 %
HuggingFaceH4/ultrachat_200k	English general 25 %
openai/gsm8k	English math/reasoning 25 %

128 calibration samples, sequence length 2048, multilingual seed prompts for Indic language coverage.

---

Files

File	Purpose
model-0000{1-5}-of-00005.safetensors	Quantised model weights
config.json	Model config (includes quantization_config)
vllm_config.yaml	vLLM serving parameters (use with --config)
recipe.yaml	llm-compressor recipe used to produce this model
configuration_sarvam_moe.py	Custom config class
modeling_sarvam_moe.py	Custom model class
tokenizer.json / tokenizer_config.json	Tokeniser
chat_template.jinja	Chat template

---

Inference — vLLM (recommended)

pip install vllm
vllm serve lalit-dumka/sarvam-30b-efficient --config vllm_config.yaml

vllm_config.yaml is included at the root of this repository.

vllm_config.yaml — parameter explanations

Parameter	Value	Reason
model	.	Load weights from the current directory (standard for HF repos)
trust_remote_code	true	Required — model uses custom SarvamMoEForCausalLM architecture defined in modeling_sarvam_moe.py and configuration_sarvam_moe.py
gpu_memory_utilization	0.85	Leaves 15 % headroom for CUDA graphs and KV-cache overhead; 0.85 is sufficient for INT4 weights (~20 GB) + full 65 k KV cache on an A100 80 GB
max_model_len	65536	Full 64 k context window matching the organizer evaluation settings; required for long chain-of-thought reasoning traces (Math500, GPQA)
dtype	auto	Let vLLM select the compute dtype — resolves to bfloat16 on A100, which is correct for this model

Note on speculative decoding: ngram speculative decoding was evaluated and removed. For long chain-of-thought outputs, ngram acceptance is near-zero on this model and degrades both accuracy and throughput.

Evaluation generation parameters

Benchmark suite	temperature	top_p	max_new_tokens
Math500 / MMLU / GPQA / AIME / reasoning	1.0	1.0	65 536
Writing Bench	0.7	0.8	16 000 (top_k=20)
Agentic (BrowseComp / SWE-bench / τ²-bench)	0.5	1.0	32 768

---

Known issues and notes for evaluators

• trust_remote_code: true is mandatory. The model uses a custom architecture (SarvamMoEForCausalLM) not yet merged into upstream Transformers/vLLM. Without this flag the model will fail to load.
• Tokenizer regex warning. vLLM may print a warning about an incorrect regex pattern inherited from the Mistral tokenizer base. This is cosmetic — tokenisation is correct and the warning does not affect output quality.
• dtype: auto resolves to bfloat16. The quantisation config stores weights as INT4 with BF16 activations; auto correctly picks BF16 for all non-quantised tensors on A100.
• First-request latency. vLLM compiles CUDA graphs on first load (~10–20 s on A100). Subsequent requests run at full speed.
• Tested vLLM version. Validated on vllm==0.9.1. The compressed-tensors INT4 MARLIN kernel path requires vLLM ≥ 0.6.

---

Licence

Apache License 2.0 — same as the original sarvamai/sarvam-30b.

---

Citation

@misc{sarvam_sovereign_models,
  title        = {Introducing Sarvam's Sovereign Models},
  author       = {{Sarvam Foundation Models Team}},
  year         = {2026},
  howpublished = {\url{https://www.sarvam.ai/blogs/sarvam-30b-105b}},
  note         = {Accessed: 2026-03-03}
}