sijan21/gemma3-270m-nepali

Gemma 3 270M — Nepali SFT (LoRA Adapter)

A LoRA adapter for google/gemma-3-270m-it (~270M parameters), instruction fine-tuned on Nepali conversational data. Trained with Unsloth starting from the pre-quantized checkpoint unsloth/gemma-3-270m-it-unsloth-bnb-4bit on a free Google Colab T4 GPU.

Model Details

Base model	google/gemma-3-270m-it (loaded via unsloth/gemma-3-270m-it-unsloth-bnb-4bit, a pre-quantized 4-bit checkpoint)
Adapter type	LoRA (PEFT)
Parameters (base)	~270M
Quantization	4-bit (bitsandbytes NF4), loaded pre-quantized — no on-the-fly quantization
Language	Nepali (ne)
Fine-tuning framework	Unsloth + TRL SFTTrainer
Hardware	1x NVIDIA T4 (Google Colab, free tier)
License	Gemma license (inherited from base model)

Intended Uses

This adapter adapts Gemma 3 270M to follow instructions and hold conversations in Nepali. It's aimed at:

• Lightweight Nepali instruction-following / chat experimentation
• Low-resource / edge deployment scenarios where a ~0.3B model is required
• A baseline to compare against larger Nepali fine-tunes

Out of scope: factual accuracy on specialized domains (medical, legal, financial), tasks requiring long-context reasoning, and any high-stakes or safety-critical use — this is a small model fine-tuned for a modest number of steps and has not been evaluated for those purposes.

Training Data

• Dataset: himalaya-ai/nepali-sft-compile
• Format: multi-turn conversations field ({"from": "human"|"gpt", "value": "..."}), converted to Gemma 3's chat template (<start_of_turn> / <end_of_turn>) via tokenizer.apply_chat_template
• Rows used: first 30,000 rows of the dataset
• Split: 90% train / 10% validation (seed 3407)

Training Procedure

LoRA Configuration

Parameter	Value
r	16
lora_alpha	32
lora_dropout	0.05
target_modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
bias	none
Gradient checkpointing	"unsloth" mode

Hyperparameters

Parameter	Value
Epochs	3
Per-device train batch size	16
Gradient accumulation steps	1 (effective batch size 16)
Learning rate	2e-4
LR scheduler	cosine
Warmup steps	50
Optimizer	paged_adamw_8bit
Precision	fp16
Max sequence length	2048
Seed	3407

How to Use

  from unsloth import FastLanguageModel

 model, tokenizer = FastLanguageModel.from_pretrained(
  model_name = "unsloth/gemma-3-270m-it-unsloth-bnb-4bit",
  max_seq_length = 2048,
  dtype = None,
  load_in_4bit = True,
)
# Load this adapter on top of the base model
model.load_adapter("<path-or-repo-of-this-adapter>")
FastLanguageModel.for_inference(model)

messages = [{"role": "user", "content": "नृत्य प्रतियोगिताहरूले नर्तकहरूको विकासमा पार्ने प्रभावबारे चर्चा गर्नुहोस्।"}]
inputs = tokenizer.apply_chat_template(
  messages, tokenize=True, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

output = model.generate(input_ids=inputs, max_new_tokens=256, temperature=0.7, top_p=0.9, do_sample=True)
print(tokenizer.decode(output[0][inputs.shape[-1]:],
 skip_special_tokens=True))

Or with plain 🤗 Transformers + PEFT:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
  import torch

  base = AutoModelForCausalLM.from_pretrained(
    "google/gemma-3-270m-it", torch_dtype=torch.float16, device_map="auto"
  )
model = PeftModel.from_pretrained(base, "<path-or-repo-of-this-adapter>")
tokenizer = AutoTokenizer.from_pretrained("<path-or-repo-of-this-adapter>")

Limitations & Biases

• Trained on a subset (30k rows) of a single compiled dataset; coverage of topics, dialects, and registers of Nepali reflects that dataset's composition and may not generalize broadly.
• As a ~270M-parameter model, it has limited reasoning and world-knowledge capacity compared to larger LLMs — expect shorter-context tasks and simpler instructions to work best.
• Not evaluated against a held-out benchmark or human preference study; validation loss was used to monitor training but does not guarantee output quality or safety.
• Inherits any limitations and biases present in the base google/gemma-3-270m-it model and in the source dataset.

Framework Versions

• Unsloth (latest at time of training)
• TRL ≥ 0.12.0
• PEFT
• Transformers (version pinned by Unsloth's dependency resolution)
• bitsandbytes