uthayamurthy/origin-task3-nllb-lora

Origin SciHigh 2026 Task 3 NLLB-600M LoRA

This repository contains a LoRA adapter for English-to-Bengali scientific-paper title translation. It is the Run 1 system submitted by team Origin to SciHigh 2026 Task 3.

The adapter must be loaded on top of facebook/nllb-200-distilled-600M at commit f8d333a098d19b4fd9a8b18f94170487ad3f821d; it is not a standalone full model.

Model details

Field	Value
Base model	facebook/nllb-200-distilled-600M
Base revision	f8d333a098d19b4fd9a8b18f94170487ad3f821d
Method	LoRA for SEQ_2_SEQ_LM
Source / target	English (eng_Latn) → Bengali (ben_Beng)
LoRA modules	q_proj, v_proj
LoRA rank / alpha / dropout	16 / 32 / 0.1
Trainable parameters	2,359,296 of 617,433,088 (0.3821%)
Developed by	Origin

Training data and procedure

The adapter was trained only on the 60 official training triplets from the SciHigh 2026 Task 3 SpringerSSAT-Tiny-Multilingual split. Each input was the English title and each target was its expert Bengali translation. No synthetic examples or test labels were used.

The 20 official validation examples were held out from gradient optimization and used once per epoch for checkpoint selection.

Hyperparameter	Value
Epochs run / selected epoch	15 / 12
Batch size	8
Learning rate	2e-4
Weight decay	0.01
Warmup ratio	0.1
Maximum source / target length	128 / 128
Generation	5 beams, maximum 96 new tokens
Seed	42
Hardware	NVIDIA RTX PRO 6000 Blackwell, 96 GB
Software	Python 3.12.3, PyTorch 2.13.0+cu130, Transformers 4.53.2, PEFT 0.17.1

Validation results

Scores below were calculated on the 20 held-out validation titles. ROUGE-L uses whitespace-delimited Bengali tokens

Metric	LoRA adapter	Zero-shot NLLB-600M	Delta
ROUGE-L F1	0.510591	0.422353	+0.088239
chrF++	53.5588	48.9423	+4.6165
SacreBLEU (tokenize=none)	25.6914	14.2437	+11.4476
Exact match	0.00%	0.00%	0.00 pp

Usage

pip install "transformers==4.53.2" "peft>=0.16,<0.18" \
  "sentencepiece>=0.2,<0.3" torch

import torch
from peft import PeftModel
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

adapter_id = "uthayamurthy/origin-task3-nllb-lora"
base_id = "facebook/nllb-200-distilled-600M"
base_revision = "f8d333a098d19b4fd9a8b18f94170487ad3f821d"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(
    adapter_id, src_lang="eng_Latn", tgt_lang="ben_Beng"
)
base_model = AutoModelForSeq2SeqLM.from_pretrained(
    base_id,
    revision=base_revision,
    torch_dtype=torch.bfloat16 if device == "cuda" else torch.float32,
)
model = PeftModel.from_pretrained(base_model, adapter_id).to(device).eval()

title = "A framework to measure microaggressions in the mathematics classroom"
inputs = tokenizer(title, return_tensors="pt").to(device)
with torch.inference_mode():
    tokens = model.generate(
        **inputs,
        forced_bos_token_id=tokenizer.convert_tokens_to_ids("ben_Beng"),
        num_beams=5,
        max_new_tokens=96,
    )
print(tokenizer.batch_decode(tokens, skip_special_tokens=True)[0])