gemma-2b-it-marathi-lora

This repository contains LoRA PEFT adapter weights for google/gemma-2b-it fine-tuned on a Cleaned Alpaca Marathi-translated dataset.

This model attempts to reproduce the experimental setup described in: Khade et al., CHiPSAL 2025.

Model description

• Base model: google/gemma-2b-it
• Tuning Method: LoRA (Low-Rank Adaptation)
• Target Modules: q_proj, k_proj, v_proj, o_proj
• LoRA Hyperparameters: r = 16, alpha = 32, dropout = 0.05
• Language: Marathi (mr)
• Training data: Marathi-translated Alpaca instruction dataset (source)

Training Details

• Dataset: 51,760 instruction-response pairs translated into Marathi from the Cleaned Stanford Alpaca dataset(unsloth/alpaca-cleaned).
• Precision: Pure FP16 / BF16 (unquantized base weights).
• Epochs: 3
• Optimizer: AdamW

Intended use

Research and educational use, not a production-ready assistant.

Automated Evaluation Results

Evaluated using the AI4Bharat Airavata benchmark harness:

Model	Metric	IndicSentiment	ARC-Easy	ARC-Challenge	IndicCOPA	IndicXNLI
gemma-2b-it (Base)	Accuracy	0.5010	0.3215	0.2927	0.5023	0.3329
	Binary F1 (Cls 1)	0.6617	N/A	N/A	0.0000	N/A
	Macro F1	0.3556	0.1893	0.2507	0.3343	0.1696
	Paper Reported F1	0.7444	0.4651	0.4043	0.2963	0.3066
gemma-2b-it (Marathi LoRA)	Accuracy	0.8710	0.2500	0.2765	0.5023	0.3335
	Binary F1 (Cls 1)	0.8724	N/A	N/A	0.2191	N/A
	Macro F1	0.8710	0.0895	0.2052	0.4269	0.1919
	Paper Reported F1	—	—	—	—	—

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "google/gemma-2b-it"
adapter_id = "lubzo/gemma-2b-it-marathi-lora"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    dtype=torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16,
    device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter_id)

prompt = "खालील प्रश्नाचे उत्तर द्या:\nभारताची राजधानी कोणती आहे?\nउत्तर:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

## Limitations

- Trained on machine-translated data, not natively-authored Marathi — inherits translation artifacts and biases.
  

## Citation

```bibtex
@inproceedings{khade2025challenges,
  title={Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning},
  author={Khade, Omkar and Jagdale, Shruti and Phaltankar, Abhishek and Takalikar, Gauri and Joshi, Raviraj},
  booktitle={Proceedings of the First Workshop on Challenges in Processing South Asian Languages (CHiPSAL 2025)},
  year={2025}
}