heykunal123/polytalk-ai-lora-nllb-1.3b

🌍 PolyTalk AI: 18-Language Neural Machine Translation

📌 Overview

PolyTalk AI is a state-of-the-art Parameter-Efficient Fine-Tuned (PEFT) model built on top of Meta's NLLB-200-1.3B. Leveraging QLoRA (Quantized Low-Rank Adaptation), this adapter significantly enhances translation accuracy and contextual fluency across 18 target languages, with a specialized focus on 11 Indian regional languages and dynamic code-switching (Hinglish).

The model supports Any-to-Any (Bidirectional) Translation across a matrix of 324 possible language pairs without requiring English as an intermediate pivot.

🏗️ Model Architecture & Technical Specifications

• Base Architecture: Sparse/Dense Transformer Encoder-Decoder (NLLB)
• Base Model: facebook/nllb-200-1.3B
• Adapter Type: LoRA (Low-Rank Adaptation)
• Trainable Parameters: 18,874,368 (approx. 1.43% of total)
• Quantization: 4-bit NormalFloat (NF4) with Double Quantization (bitsandbytes)
• Compute Type: torch.float16
• Gradient Checkpointing: Enabled

⚙️ LoRA Configuration

LoraConfig(
    r=64,
    lora_alpha=128,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="SEQ_2_SEQ_LM"
)

📊 Training Corpus & Methodology

The model was fine-tuned on a heavily curated parallel corpus consisting of ~475,000 sentence pairs.

• Dataset Composition: High-quality subsets filtered from OPUS-100, augmented with conversational datasets to capture slang, colloquialisms, and code-mixed patterns.
• Preprocessing: Input text was tokenized using the native NLLB SentencePiece tokenizer. Max sequence length was constrained to 128 tokens to optimize VRAM utilization.
• Loss Optimization: Cross-Entropy Loss with Label Smoothing.

📈 Hyperparameters

Parameter	Value	Parameter	Value
Optimizer	paged_adamw_32bit	Max Steps	5,000
Learning Rate	2e-4	Warmup Steps	500
LR Scheduler	Cosine Annealing	Batch Size (Eff)	16 (2 × 8 Grad Accum)
Weight Decay	0.01	Max Grad Norm	0.3
Mixed Precision	FP16	Training Hardware	1x NVIDIA T4 (16GB)

🌐 Supported Language Matrix

The model natively processes the following FLORES-200 language codes:

🇮🇳 Indic Languages (11)

• Hindi (hin_Deva)
• Tamil (tam_Taml)
• Telugu (tel_Telu)
• Bengali (ben_Beng)
• Marathi (mar_Deva)
• Gujarati (guj_Gujr)
• Kannada (kan_Knda)
• Malayalam (mal_Mlym)
• Punjabi (pan_Guru)
• Odia (ory_Orya)
• Assamese (asm_Beng)

🌍 International Languages (6) & Code-Switching (1)

• French (fra_Latn)
• Spanish (spa_Latn)
• German (deu_Latn)
• Italian (ita_Latn)
• Russian (rus_Cyrl)
• Japanese (jpn_Jpan)
• Hinglish (eng_Latn) — Trained specifically for English-to-Romanized Hindi conversational outputs.

💻 Inference Implementation

PolyTalk AI requires the peft and transformers libraries. The adapter must be merged with the base NLLB-200-1.3B model at runtime.

Installation

pip install torch transformers peft accelerate

Python API

import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from peft import PeftModel

# 1. Initialize tokenizer and base model
model_id = "facebook/nllb-200-1.3B"
adapter_id = "heykunal123/polytalk-ai-lora-nllb-1.3b"

tokenizer = AutoTokenizer.from_pretrained(model_id, src_lang="eng_Latn")
base_model = AutoModelForSeq2SeqLM.from_pretrained(
    model_id, 
    dtype=torch.float16, 
    device_map="auto"
)

# 2. Attach PEFT LoRA adapter
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

# 3. Translation Execution
text = "The architecture utilizes Low-Rank Adaptation for parameter efficiency."
inputs = tokenizer(text, return_tensors="pt").to(model.device)

# Set target language (e.g., Hindi)
target_lang_id = tokenizer.convert_tokens_to_ids("hin_Deva")

with torch.no_grad():
    outputs = model.generate(
        **inputs, 
        forced_bos_token_id=target_lang_id, 
        max_new_tokens=128
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

🔬 Empirical Performance & Zero-Shot Capabilities

During qualitative evaluations, the model exhibited exceptional capability in handling:

1. Contextual Fillers & Colloquialisms: Seamlessly translates conversational fillers (e.g., "Umm", "Aah") without dropping context.
2. Any-to-Any Translation: Successfully translates Assamese ↔ Japanese and German ↔ Bengali directly.
3. Romanized Code-Switching: The eng_Latn target produces fluent Hinglish (e.g., "Dude mai tumhe bata raha hu...").

⚠️ Limitations & Bias

• Context Length: The model was trained with a max sequence length of 128 tokens. Extremely long paragraphs may experience truncation or hallucination.
• Resource Constraints: As a 1.3B parameter model, it operates on a fraction of the parameters of GPT-4 or Claude, meaning highly nuanced domain-specific terminology (e.g., advanced medical or legal text) may lack precision compared to general conversational text.

📝 Citation & Acknowledgment

If this model assists in your research or application, please cite:

@software{polytalk-ai-2026,
  author = {Kunaljit Kashyap},
  title = {PolyTalk AI: Efficient Multilingual Translation via QLoRA Adaptation of NLLB-200},
  year = {2026},
  url = {https://huggingface.co/heykunal123/polytalk-ai-lora-nllb-1.3b}
}