pankajpandey-dev/gemma-4-e4b-hindi-instruct-GGUF

🇮🇳 Gemma-4-E4B-Hindi-Instruct — GGUF

A Hindi instruction-tuned fine-tune of Gemma 4 E4B, quantized to GGUF for local / CPU / edge use via llama.cpp, LM Studio, and llama-cpp-python.

The smallest quant is ~5.3 GB and runs on an 8 GB laptop — CPU or GPU, fully offline. No API, no cloud. Part of my 🇮🇳 Hindi LLM Series — small, openly-documented Indic models that actually follow instructions in Hindi.

▶️ Try it live (no install, runs on free CPU): pankajpandey-dev/gemma-4-e4b-hindi-demo

This is the GGUF build. The 16-bit model and LoRA adapter are in separate repos.

---

✅ What the fine-tune actually changes (honest eval)

I ran a side-by-side on 25 Hindi prompts — base gemma-4-E4B-it vs. this fine-tune, same prompts, same settings. The pattern was consistent:

Behaviour	Base gemma-4-E4B-it	This fine-tune
Stays in Hindi	Often code-switches to English — e.g. संतुलित आहार (Eat a Balanced Diet)	Clean, native Devanagari Hindi
Follows the ask	"3 tips" → a long essay; "short message" → a menu of options	"3 tips" → exactly 3; "short message" → one short message
Length	Verbose (~1,200-character answers)	Concise and to the point

🔍 Where the base still wins — being honest: vanilla gemma-4-E4B is more detailed and broader in general knowledge. This is not a "smarter" model — it's a focused, Hindi-native, edge-friendly one. If you want maximum detail and don't mind Hindi-English mixing, the base may actually suit you better.

📄 The full 25-prompt comparison is written up in the announcement post.

---

🚀 Quick start

⚠️ Ollama note: Gemma 4 E4B GGUFs currently fail to load in Ollama (upstream architecture bug, ollama#15447). Use llama.cpp, LM Studio, or llama-cpp-python below — all work today. (Retest Ollama as it updates; recent llama.cpp builds added gemma4 support.)

llama.cpp

./llama-cli -hf pankajpandey-dev/gemma-4-e4b-hindi-instruct-GGUF:Q4_K_M \
  -p "भारत के बारे में एक रोचक तथ्य बताओ।"

Or run a local OpenAI-compatible server with a web UI:

./llama-server -hf pankajpandey-dev/gemma-4-e4b-hindi-instruct-GGUF:Q4_K_M

LM Studio / Jan

Search for pankajpandey-dev/gemma-4-e4b-hindi-instruct-GGUF, pick a quant (start with Q4_K_M), and chat. 🐾

llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="pankajpandey-dev/gemma-4-e4b-hindi-instruct-GGUF",
    filename="gemma-4-e4b-it.Q4_K_M.gguf",
)
out = llm.create_chat_completion(
    messages=[{"role": "user", "content": "मशीन लर्निंग को आसान शब्दों में समझाओ।"}]
)
print(out["choices"][0]["message"]["content"])

💡 This is a text-only model. You do not need any mmproj file — just the single quant .gguf.

---

📦 Pick your quant

File	Quant	Size	Vibe
gemma-4-e4b-it.Q4_K_M.gguf	Q4_K_M	~5.3 GB	⭐ Recommended — best size/quality balance
gemma-4-e4b-it.Q5_K_M.gguf	Q5_K_M	~5.8 GB	A notch higher quality
gemma-4-e4b-it.Q8_0.gguf	Q8_0	~8.0 GB	Near-lossless

📐 Size note: Gemma 4 E4B runs at 4B speed but carries **8B total parameters** (Matformer / per-layer embeddings), so the files are sized like an 8B model. Q4_K_M still fits comfortably on a laptop with 8 GB+ RAM, CPU or GPU.

---

🧮 Will it run?

Rough guidance (model + a working context window). Drop a quant if you're tight, or use a q4_0 KV cache to roughly double your context headroom. Max context is 128K.

Your free RAM / VRAM	What to run
8 GB	🟢 Q4_K_M — the sweet spot
12 GB	🔵 Q5_K_M, or Q8_0 with a modest context
16 GB+	⚪ Q8_0 at full quality, with long context

💡 Apple Silicon / integrated GPUs with unified memory count too — same sizes, just slower than a discrete GPU. And yes, this all works on plain CPU with system RAM (just slower than GPU).

---

💬 Example outputs

Prompt: भारत के बारे में एक रोचक तथ्य बताओ।

भारत दुनिया में सबसे अधिक भाषाओं वाले देशों में से एक है — 22 आधिकारिक भाषाएँ और 1,000 से अधिक बोलियाँ। उदाहरण के लिए हिंदी एक इंडो-आर्यन भाषा है, जबकि तमिल एक द्रविड़ भाषा है।

Prompt: मशीन लर्निंग को आसान शब्दों में समझाओ।

मशीन लर्निंग एक प्रकार की कृत्रिम बुद्धिमत्ता है जो कंप्यूटर को स्पष्ट रूप से प्रोग्राम किए बिना डेटा से सीखने देती है … जैसे किसी बच्चे को सेब और संतरे के चित्र दिखाकर अंतर करना सिखाना।

(Real outputs from testing, lightly trimmed for length.)

---

🛠️ Training details

Base model	unsloth/gemma-4-E4B-it
Method	LoRA (r=16, α=16), response-only loss
Framework	Unsloth
Data	~10k Hindi instruction pairs (AI4Bharat indic-instruct: anudesh + dolly, hi splits)
Epochs	2
LR / schedule	1e-4, cosine
Precision	bf16 (4-bit QLoRA base)
Hardware	Single NVIDIA L4 (24 GB)
Final train loss	~0.29

Trained text-only (vision layers frozen). The Gemma chat template is applied with a single BOS token to avoid double-BOS corruption.

---

🔗 The series & related repos

• 16-bit model: pankajpandey-dev/gemma-4-e4b-hindi-instruct
• LoRA adapter: pankajpandey-dev/gemma-4-e4b-hindi-instruct-lora
• Live demo (CPU): pankajpandey-dev/gemma-4-e4b-hindi-demo
• 📚 Full collection: 🇮🇳 Hindi LLM Series

💚 Found it useful, or hit a rough edge? Open a discussion with a Hindi prompt — feedback shapes the next model in the series.

---

📚 Provenance & license (please read)

This is a research / educational fine-tune with a mixed-license lineage — review all of the following before any redistribution or commercial use:

• Model weights derive from Gemma 4 and are released under the Gemma Terms of Use. Google's Gemma usage restrictions apply to this derivative.
• Training data is from AI4Bharat indic-instruct-data-v0.1:
  • Dolly split — derived from databricks-dolly-15k, licensed CC-BY-SA-3.0.
  • Anudesh split — prompts paired with responses generated by Llama-2-70B, so the Llama 2 Community License applies to that portion.

I do not redistribute the raw training data here. If you build on this model, you are responsible for complying with the Gemma, Llama 2, and CC-BY-SA terms above.

---

⚠️ Limitations

• ~8B-class model: strong Hindi fluency and instruction-following, but it can still hallucinate facts and occasionally repeat phrasing on open-ended generation (e.g. long poems).
• Tuned primarily on single-turn Hindi instructions; long multi-turn chat is not the focus.
• Not safety-aligned for production. Add your own guardrails.

---

🙏 Acknowledgements

Base model by Google (Gemma 4). Training data by AI4Bharat. Fine-tuning with Unsloth.