gemma-2-2b-it-marathi-lora

This repository contains LoRA PEFT adapter weights for google/gemma-2-2b-it fine-tuned on a Cleaned Alpaca Marathi-translated dataset.

This model attempts to reproduce the experimental setup described in: Khade et al., CHiPSAL 2025.

Model description

• Base model: google/gemma-2-2b-it
• Tuning Method: LoRA (Low-Rank Adaptation)
• Target Modules: q_proj, k_proj, v_proj, o_proj
• LoRA Hyperparameters: r = 16, alpha = 32, dropout = 0.05
• Language: Marathi (mr)
• Training data: Marathi-translated Alpaca instruction dataset (source)

Training Details

• Dataset: 51,760 instruction-response pairs translated into Marathi from the Cleaned Stanford Alpaca dataset(unsloth/alpaca-cleaned).
• Precision: Pure FP16 / BF16 (unquantized base weights).
• Epochs: 3
• Optimizer: AdamW

Intended use

Research and educational use, not a production-ready assistant.

Automated Evaluation Results

Evaluated using the AI4Bharat Airavata benchmark harness:

Evaluation Results: google/gemma-2-2b-it vs. gemma-2-2b-it-marathi-lora

Model	Metric	IndicSentiment	ARC-Easy	ARC-Challenge	IndicCOPA	IndicXNLI
google/gemma-2-2b-it (Base)	Accuracy	0.9560	0.8497	0.5410	0.5068	0.3956
	Binary F1 (Cls 1)	0.9560	N/A	N/A	0.0179	N/A
	Macro F1	0.9560	0.8799	0.5404	0.3443	0.2977
	Paper Reported F1	0.9749	0.6851	0.7210	0.7210	0.2814
gemma-2-2b-it-marathi-lora	Accuracy	0.9660	0.6237	0.2961	0.5023	0.3597
	Binary F1 (Cls 1)	0.9657	N/A	N/A	0.0000	N/A
	Macro F1	0.9660	0.6135	0.1782	0.3343	0.2727
	Paper Reported F1	0.9589	0.6343	0.6374	0.5835	0.1667

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "google/gemma-2-2b-it"
adapter_id = "lubzo/gemma-2-2b-it-marathi-lora"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    dtype=torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16,
    device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter_id)

prompt = "खालील प्रश्नाचे उत्तर द्या:\nभारताची राजधानी कोणती आहे?\nउत्तर:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

## Limitations

- Trained on machine-translated data, not natively-authored Marathi — inherits translation artifacts and biases.
  

## Citation

```bibtex
@inproceedings{khade2025challenges,
  title={Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning},
  author={Khade, Omkar and Jagdale, Shruti and Phaltankar, Abhishek and Takalikar, Gauri and Joshi, Raviraj},
  booktitle={Proceedings of the First Workshop on Challenges in Processing South Asian Languages (CHiPSAL 2025)},
  year={2025}
}