ihumaunkabir/mistral_bangla_lora

mistral-bangla (LoRA adapters)

PEFT LoRA adapters for a Bangla (Bengali) continual-pre-trained + instruction-tuned Mistral-7B-v0.3. Load these on top of unsloth/mistral-7b-v0.3 to get the Bangla-adapted model.

The adapters were produced by a two-stage recipe -- CPT on Bangla Wikipedia, then SFT on the ihumaunkabir/alpaca-gpt4-bangla instruction dataset -- using Unsloth.

Looking for a ready-to-run model? The merged + quantized GGUF (no base model needed) lives at ihumaunkabir/mistral_bangla_q4_k_m_gguf.

detail	value
Adapter size	~1.9 GB
Base model (required)	unsloth/mistral-7b-v0.3 -- loaded separately, not bundled
Training precision	QLoRA 4-bit (adapter trained with the base in 4-bit)
LoRA rank	128
LoRA alpha	32
rsLoRA	enabled (use_rslora=True)
LoRA targets	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, embed_tokens, lm_head

embed_tokens and lm_head are included because this is continual pre-training -- the vocabulary embeddings must move to absorb Bangla.

---

Training recipe

Stage 1 -- Continual Pre-Training (CPT)

setting	value
CPT data	wikimedia/wikipedia, config 20231101.bn
Learning rate	5e-5 (LoRA), 1e-5 (embeddings)
Optimizer	adamw_8bit, linear schedule, 10 warmup steps

Stage 2 -- Supervised Fine-Tuning (SFT)

setting	value
SFT data	ihumaunkabir/alpaca-gpt4-bangla -- 49,969 Bangla instruction-response pairs
Learning rate	5e-5 (LoRA), 1e-5 (embeddings)
Packing	disabled

---

How to use

Load adapters with Unsloth (recommended, 2x faster inference)

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "ihumaunkabir/mistral_bangla_lora",
    max_seq_length = 2048,
    load_in_4bit = True,
)
FastLanguageModel.for_inference(model)

alpaca_prompt = """নিচে একটি নির্দেশনা দেওয়া আছে, যা একটি কাজের বর্ণনা দেয়। অনুরোধটি যথাযথভাবে সম্পূর্ণ করে একটি উত্তর লিখুন।

### নির্দেশনা:
{}

### উত্তর:
{}"""

inputs = tokenizer([alpaca_prompt.format("বাংলাদেশের রাজধানীর নাম লেখো।", "")], return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=128, use_cache=True)
print(tokenizer.batch_decode(outputs))

Load with PEFT (no Unsloth)

from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

model = AutoPeftModelForCausalLM.from_pretrained(
    "ihumaunkabir/mistral_bangla_lora",
    load_in_4bit = True,
)
tokenizer = AutoTokenizer.from_pretrained("ihumaunkabir/mistral_bangla_lora")

---

Limitations

1. Smoke-test training volume (max_steps=120) -- ~1,920 samples per stage. Increase for production.
2. CPT data is Wikipedia-only -- formal, encyclopedia Bangla; no colloquial / conversational text.
3. SFT data is machine-translated (Korean -> Bangla).
4. No safety alignment. No RLHF, DPO, or red-teaming.
5. QLoRA 4-bit during training can slightly degrade quality vs. bf16.
6. Requires the base model to be loaded separately -- this is an adapter, not a standalone model.

---

Citation

If you use these adapters, please cite both this repo and the alpaca-gpt4-bangla dataset, plus the base Mistral model.

This repo (LoRA adapters)

@misc{mistral-bangla-lora,
  author       = {ihumaunkabir},
  title        = {mistral-bangla: Bangla CPT + SFT LoRA adapters for Mistral-7B-v0.3},
  year         = {2026},
  url          = {https://huggingface.co/ihumaunkabir/mistral_bangla_lora},
  note         = {CPT on Bangla Wikipedia + SFT on alpaca-gpt4-bangla, trained with Unsloth}
}

SFT dataset -- alpaca-gpt4-bangla

@misc{alpaca-gpt4-bangla,
  author       = {ihumaunkabir},
  title        = {alpaca-gpt4-bangla: A Bangla instruction-following dataset},
  year         = {2026},
  url          = {https://huggingface.co/datasets/ihumaunkabir/alpaca-gpt4-bangla},
  note         = {Machine translation (Korean -> Bangla) of FreedomIntelligence/alpaca-gpt4-korean}
}

Base model

@misc{mistral7b,
  author       = {Mistral AI},
  title        = {Mistral-7B-v0.3},
  year         = {2024},
  url          = {https://huggingface.co/mistralai/Mistral-7B-v0.3}
}