urdu-roman-transliterator-tiny-aya

QLoRA fine-tune of CohereLabs/tiny-aya-global (3.35B, Cohere architecture) for Urdu → Roman Urdu transliteration, targeted at Pakistani names, addresses, and administrative text (NADRA-style fields).

Model description

• Base model: CohereLabs/tiny-aya-global (3.35B params, cohere2 architecture)
• Method: QLoRA (4-bit base via bitsandbytes, LoRA adapter merged back into the base weights for this repo)
• LoRA config: r=16, alpha=32, target modules q_proj, v_proj
• Training: 1 epoch, batch size 4, lr 2e-4, on synthetic data (see below)
• Task: single-direction transliteration, Urdu script → Roman Urdu (Latin script), not translation — meaning is preserved token-for-token, not paraphrased

Intended use

Transliterating Urdu-script names, addresses, and short administrative phrases into Roman Urdu — e.g. normalizing free-text address fields for search, matching, or display in systems that can't render Urdu script.

Out of scope: general-purpose Urdu chat, translation to English, long-form text, or domains far from names/addresses (the training distribution is narrow — see Limitations).

Training data

50,000 synthetically generated Urdu/Roman Urdu pairs built from Pakistani address and name components, combined programmatically:

• Honorific prefixes (Chaudhry, Sardar, Syed, Sheikh, ...)
• First/last names common across Pakistani regions
• Regional rural terms (Punjab: Chak No./Moza/Pind; Sindh: Goth/Deh/Tando; KP: Kalay/Banda/Dheri; Balochistan: Killi/Karez/Bazar)
• Streets, lanes, and bazaars
• Landmarks (Near, Opposite, Behind, ...)
• Tehsil/Taluka administrative units
• Urban units (Sector, Block, Phase, Mohallah, Society)
• Commercial units (Shop No., Plot, Flat, Showroom) and floor descriptors
• Named buildings/landmarks (banks, malls, mosques, hospitals)

Cleaned (Unicode NFC normalization, whitespace/punctuation normalization, dedup, length filtering) and split:

split	rows (target)
train	~40,000
validation	5,000
test	5,000

Exact post-cleaning counts vary slightly due to dedup — check train_df.shape / val_df.shape / test_df.shape in the training notebook for the precise numbers used in your run.

This is 100% synthetic, template-generated data — it has not yet been validated against real records. See Limitations.

Prompt format

Trained on plain instruction-style text (no chat template), format:

### Instruction:
Transliterate Urdu to Roman Urdu.

### Input:
{urdu_text}

### Response:
{roman_urdu_text}

Use this exact format at inference time — the model was not trained with Aya's native chat template, so wrapping input in apply_chat_template gives out-of-distribution results.

Evaluation

Evaluated on the 5,000-row held-out test set using BLEU, WER, and CER (word/character error rate against the reference Roman Urdu). Fill in your actual run numbers:

metric	score
BLEU	—
WER	—
CER	—

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

MODEL_ID = "ogx786/urdu-roman-transliterator-tiny-aya"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.float16,
    device_map="auto",
)

def transliterate(urdu_text: str) -> str:
    prompt = f"""### Instruction:
Transliterate Urdu to Roman Urdu.

### Input:
{urdu_text}

### Response:
"""
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
        temperature=0.1,
        top_p=0.9,
        pad_token_id=tokenizer.eos_token_id,
    )
    generated = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return generated.split("### Response:")[-1].strip()

print(transliterate("میں اسکول جا رہا ہوں"))

Limitations

• Training data is synthetic and template-generated, not sampled from real-world Urdu text — the model may not generalize well to naturally-occurring sentences, informal spelling variants, or Roman Urdu conventions that differ from the templates used here.
• Single epoch, small LoRA rank (r=16) — capacity for edge cases (rare names, mixed-script input, numerals embedded in addresses) is limited.
• Inherits the base model's license restrictions (see below) — not licensed for commercial use.
• Not evaluated against real NADRA/CNIC address data at time of writing.

License

Inherits CohereLabs/tiny-aya-global's license: CC-BY-NC-4.0, non-commercial, and requires adherence to Cohere's Acceptable Use Policy. This fine-tune carries the same restriction — no commercial use without contacting Cohere.

Base model credit

Built on CohereLabs/tiny-aya-global, part of Cohere Labs' Tiny Aya family of multilingual small language models.