gemma-2b-marathi-lora

This repository contains LoRA PEFT adapter weights for google/gemma-2b fine-tuned on a Cleaned Alpaca Marathi-translated dataset.

This model attempts to reproduce the experimental setup described in: Khade et al., CHiPSAL 2025.

Model description

• Base model: google/gemma-2b
• Tuning Method: LoRA (Low-Rank Adaptation)
• Target Modules: q_proj, k_proj, v_proj, o_proj
• LoRA Hyperparameters: r = 16, alpha = 32, dropout = 0.05
• Language: Marathi (mr)
• Training data: Marathi-translated Alpaca instruction dataset (source)

Training Details

• Dataset: 51,760 instruction-response pairs translated into Marathi from the Cleaned Stanford Alpaca dataset(unsloth/alpaca-cleaned).
• Precision: Pure FP16 / BF16 (unquantized base weights).
• Epochs: 3
• Optimizer: AdamW

Intended use

Research and educational use, not a production-ready assistant.

Automated Evaluation Results

Evaluated using the AI4Bharat Airavata benchmark harness:

Model	Metric	IndicSentiment	ARC-Easy	ARC-Challenge	IndicCOPA	IndicXNLI
gemma-2b (Base)	Accuracy	0.9080	0.6216	0.4377	0.4932	0.3323
	Binary F1 (Cls 1)	0.9000	N/A	N/A	0.5648	N/A
	Macro F1	0.9074	0.4902	0.4201	0.4792	0.1869
	Paper Reported F1	0.7772	0.4435	0.4240	0.6547	0.3582
gemma-2b (Marathi LoRA)	Accuracy	0.8480	0.3519	0.2756	0.5000	0.3451
	Binary F1 (Cls 1)	0.8195	N/A	N/A	0.6595	N/A
	Macro F1	0.8441	0.2076	0.1339	0.3594	0.2906
	Paper Reported F1	0.9397	0.6048	0.3848	0.4219	0.1675

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "google/gemma-2b"
adapter_id = "lubzo/gemma-2b-marathi-lora"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    dtype=torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16,
    device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter_id)

prompt = "खालील प्रश्नाचे उत्तर द्या:\nभारताची राजधानी कोणती आहे?\nउत्तर:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

## Limitations

- Trained on machine-translated data, not natively-authored Marathi — inherits translation artifacts and biases.
  

## Citation

```bibtex
@inproceedings{khade2025challenges,
  title={Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning},
  author={Khade, Omkar and Jagdale, Shruti and Phaltankar, Abhishek and Takalikar, Gauri and Joshi, Raviraj},
  booktitle={Proceedings of the First Workshop on Challenges in Processing South Asian Languages (CHiPSAL 2025)},
  year={2025}
}