welyjesch/nllb-lora-en-hil

NLLB-200 Hiligaynon Translation Model (LoRA)

This model is a fine-tuned version of facebook/nllb-200-distilled-600M optimized for translating English (eng_Latn) to Hiligaynon (hil_Latn).

It was trained using LoRA (Low-Rank Adaptation) to capture the specific lexical and grammatical nuances of the Hiligaynon (Ilonggo) language, with a heavy focus on removing Tagalog and English pollution from the training data.

Model Description

Hiligaynon is often under-represented or misclassified as Tagalog in large-scale multilingual models. This fine-tuning project specifically targets the "Tagalog pollution" problem common in scraped datasets.

• Base Model: facebook/nllb-200-distilled-600M
• Target Language: Hiligaynon (hil_Latn)
• Source Language: English (eng_Latn)
• Training Method: Parameter-efficient fine-tuning (LoRA)
• Primary Domain: News, general conversation, and contemporary media.

Dataset & Anti-Pollution Filtering

The model was trained on the welyjesch/bombo_hil_eng_raw_translations dataset.

A critical phase of this model's development was the Strict Cleanup Pipeline. Using lingua-language-detector, the dataset was filtered to remove:

1. English Target Pollution: Rows where the English source was simply copied into the Hiligaynon target.
2. Tagalog Pollution: Rows where the target text was identified as high-confidence Tagalog.
3. Short-Sentence Ambiguity: Overlap protection was implemented to ensure short Hiligaynon sentences (which share words with Tagalog) were preserved while long, definitive Tagalog sentences were purged.

This ensures that the model learns actual Hiligaynon (e.g., using "gid", "sang", "amon") rather than defaulting to Tagalog structures.

Training Procedure

• LoRA Config: $r=16$, $alpha=32$
• Target Modules: q_proj, v_proj, k_proj, out_proj, fc1, fc2
• Precision: FP16
• Learning Rate: 2e-4
• Batch Size: 16 (with Gradient Accumulation)
• Hardware: Optimized for NVIDIA T4 (Google Colab/Kaggle environments).

Usage

You can use this model directly with the transformers library. Note that you must load the base NLLB model and the LoRA adapters.

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from peft import PeftModel
import torch

model_id = "facebook/nllb-200-distilled-600M"
adapter_id = "YOUR_USERNAME/YOUR_REPO_NAME" # or local path

tokenizer = AutoTokenizer.from_pretrained(model_id)
base_model = AutoModelForSeq2SeqLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")
model = PeftModel.from_pretrained(base_model, adapter_id)
model = model.merge_and_unload()

def translate(text):
    inputs = tokenizer(text, return_tensors="pt").to("cuda")
    outputs = model.generate(
        **inputs,
        forced_bos_token_id=tokenizer.convert_tokens_to_ids("hil_Latn"),
        max_length=128
    )
    return tokenizer.batch_decode(outputs, skip_special_tokens=True)[0]

print(translate("Good morning! How are you today?"))

Limitations

• Dialectal Variation: Hiligaynon has variations across Panay, Negros, and Guimaras. This model leans toward the media-standard Hiligaynon used in news broadcasting.
• Formal vs Informal: Due to the source data being largely news-based, the model may perform more accurately on formal sentence structures than on highly informal street slang.
• Residual Pollution: While strict filtering was applied, some minor Tagalog loanwords may still appear due to the inherent lexical overlap between Bisayan languages and Tagalog.

Acknowledgments

The training data is sourced from Bombo Radyo translations. Special thanks to the researchers and contributors of the Hiligaynon NLP community.