harshitv804/Indic-Transliterate

Indic Transliterate

A compact 5.37M-parameter encoder-decoder transformer for word-level transliteration between English and five Indic scripts — Tamil, Hindi, Telugu, Malayalam, and Kannada.

The whole model ships as ~12 MB of ONNX and runs on CPU, which makes it cheap to self-host and fast enough for live, word-by-word typing.

Live demo: harshitv804/Indic-Transliterate · Weights: harshitv804/Indic-Transliterate

Highlights

Parameters	5.37 M
Vocabulary	278 (byte-level BPE, character granularity)
Embedding dim	128 (4 heads × 32)
Encoder / decoder	12 / 12 layers
FFN hidden size	256 (SwiGLU)
ONNX size	12.8 MB (fp16 weights, fp32 compute)
Training loss	0.063
Validation loss	0.155
Decoding	Beam search (beam size 4 by default)
Languages	English, Tamil, Hindi, Telugu, Malayalam, Kannada
Dataset	AI4Bharat Aksharantar

Requirements

• Python 3.10+
• ONNX inference (CPU only): huggingface-hub, numpy, onnxruntime, tokenizers
• Web app: gradio
• Training / PyTorch inference: torch (CUDA recommended) and liger-kernel; training additionally needs datasets

Versions used by the Space (requirements.txt):

gradio==6.32.0
huggingface-hub>=1.16.0,<3.0.0
numpy>=1.24,<3.0
onnxruntime>=1.17.0
tokenizers>=0.15.0

Minimal install for ONNX-only inference:

pip install huggingface-hub numpy onnxruntime tokenizers

Quickstart

1. Install

This repository ships the ONNX weights and the CPU inference code, so you can run it without PyTorch:

git clone https://huggingface.co/harshitv804/Indic-Transliterate
cd Indic-Transliterate
pip install huggingface-hub numpy onnxruntime tokenizers

For training, the training loop, or the PyTorch checkpoint, clone the code repository instead:

git clone https://github.com/harshitv804/Indic-Transliterate
cd Indic-Transliterate
pip install -r requirements.txt

2. Import and run

# Run from the repo root so ``transliterate`` is importable.
from transliterate import IndicTransliterator

tr = IndicTransliterator("onnx_model")

print(tr.translate("How are you?", "en-ta"))   # ஹோவ் அரே யூ?
print(tr.translate("hello", "en-ta"))          # ஹெல்லோ
print(tr.translate("vanakkam", "en-ta"))       # வணக்கம்
print(tr.translate("வணக்கம்", "ta-en"))        # vanakkam

Or point at a Hub snapshot directly:

from huggingface_hub import snapshot_download

from transliterate import IndicTransliterator

model_dir = snapshot_download(
    "harshitv804/Indic-Transliterate",
    allow_patterns=["onnx_model/*"],
)
tr = IndicTransliterator(f"{model_dir}/onnx_model")

Decoding options:

tr.translate("How are you?", "en-ta", beam_size=4, length_penalty=1.0)
tr.translate_batch(
    ["How are you?", "Good morning"],
    ["en-ta", "en-ta"],
)

3. Live demo

The interactive app (input left, output right, direction dropdown, live word-by-word transliteration) runs here:

harshitv804/Indic-Transliterate on Spaces

To run it locally, use the code repository and python app.py.

4. Run the PyTorch checkpoint

The training script, inference.py, and step_105000.pt live in the code repository, not here:

from inference import Transliterator

tr = Transliterator.from_checkpoint("step_105000.pt", device="cuda")
print(tr.translate("How are you?", "en-ta"))

python inference.py --checkpoint step_105000.pt --direction en-ta --text "How are you?"

Omit --text for an interactive prompt. The PyTorch path needs a CUDA GPU; the ONNX path above is CPU-only and is what the Space uses.

Architecture

A pre-norm encoder-decoder transformer. A source sentence is encoded once, then the decoder emits one token at a time while cross-attending to the encoder memory.

Encoder (12 layers, bidirectional)

1. Source token embedding — 278 × 128.
2. ShortConv stem — a depthwise convolution (kernel 7, groups = dim) followed by SiLU, a pointwise linear projection, and a residual add. This gives every position a local window of neighbouring characters, which helps with script-level sound changes.
3. 12 gated self-attention + SwiGLU blocks, each with two RMSNorms:
   • Gated multi-head self-attention, 4 heads, bidirectional (no causal mask), with QK-norm and RoPE (theta = 50 000).
   • SwiGLU feed-forward network with hidden size 256.
4. Final RMSNorm, producing the encoder memory [B, S, 128].

Decoder (12 layers, causal + cross-attention)

1. Target token embedding — 278 × 128, tied to the source embedding.
2. Each of the 12 blocks contains three sub-layers, each pre-normed:
   • Gated causal self-attention — RoPE, 4 heads, QK-norm.
   • Gated cross-attention — Q/K from the decoder, K/V from the encoder memory. No RoPE here, since query and key come from different sequences. QK-norm still applies.
   • SwiGLU feed-forward, hidden size 256.
3. Final RMSNorm and a linear output layer to [B, T, 278] vocabulary logits, tied to the token embeddings.

So the 278 × 128 embedding matrix is tied three ways: source embedding, target embedding, and output projection. That keeps the parameter count — and the ONNX file — small.

SwiGLU feed-forward

gate = SiLU(gate_proj(x))
up   = up_proj(x)
out  = down_proj(gate * up)

ShortConv stem (encoder only)

y = x + Linear(SiLU(DepthwiseConv1d(x, kernel=7, groups=dim)))

Tokenizer

A byte-level BPE tokenizer with a 278-token vocabulary and no normalizer. With so few merges it behaves at character level, and because it is byte-level it can never hit an unknown token for any script or punctuation.

The 22 non-byte tokens are:

• <|bos|>, <|eos|>, <|pad|>
• 10 direction tags — <|ta-en|>, <|en-ta|>, <|hi-en|>, <|en-hi|>, <|te-en|>, <|en-te|>, <|ml-en|>, <|en-ml|>, <|kn-en|>, <|en-kn|>
• 8 reserved slots

Zero-shot or new language pairs only need a new direction tag and a fine-tune — the character inventory is already covered.

Supported directions

Source	Targets
English	Tamil, Hindi, Telugu, Malayalam, Kannada
Tamil, Hindi, Telugu, Malayalam, Kannada	English

Ten directions in total: ta-en, en-ta, hi-en, en-hi, te-en, en-te, ml-en, en-ml, kn-en, en-kn.

Training

• Dataset: AI4Bharat Aksharantar, the large parallel transliteration corpus covering Indic scripts. All five languages are mixed into one model, with a direction tag prepended to every source sequence.
• Objective: token-level cross-entropy, teacher forcing.
• Final losses: training 0.063, validation 0.155.
• Optimizer: AdamW, peak LR 1e-5, betas (0.9, 0.95), weight decay 0.1 (skipped on embeddings, norms and biases), gradient clipping 1.0.
• Schedule: 10k linear warmup, then constant LR.
• Precision: bf16; batch size 256; dropout 0.2.
• Checkpoint used for this release: step_105000.pt (105k optimizer steps).

Inference

Word-level decoding, sentence-level API

The model transliterates one word at a time — it is trained on single-word pairs, so feeding it a whole sentence degrades quality. transliterate.py therefore applies a custom word-boundary splitter (regex-style, Unicode-aware: runs of letters and combining marks versus everything else), translates each word run on its own, and reassembles the sentence with the original spaces, punctuation, digits, and newlines untouched.

en-* inputs are lowercased first; Indic source text is passed through unchanged.

Beam search

Decoding is beam search only — no sampling. Beam size defaults to 4 (beam_size, plus length_penalty), and the encoder runs once per word while the decoder batches all beams together.

ONNX on CPU

export_onnx.py exports the trained checkpoint to ONNX:

python export_onnx.py --checkpoint step_105000.pt --precision fp16

The fp16 build stores weights in half precision and widens them to fp32 in-graph, so compute stays numerically identical to the PyTorch model while the bundle shrinks to 12.8 MB (encoder.onnx 5.3 MB + decoder.onnx 7.5 MB). onnx_inference.py runs it with ONNX Runtime on CPU and needs no PyTorch at all.

A practical note on speed: these layers are small, so ONNX Runtime thread oversubscription hurts. The app caps intra-op threads at min(4, cpu_count), which took per-word latency on a shared container from ~2–7 s down to ~0.1–0.3 s.

Gradio Space

app.py is the live demo app: input on the left, output on the right, a direction dropdown, and live word-by-word transliteration — a word is translated as soon as it is followed by a space or punctuation, the word currently being typed stays untouched, and backspacing also removes its translation. Completed words are cached per (word, direction) so retyping a sentence costs nothing.

Environment variables:

• MODEL_REPO — Hugging Face repo to download the ONNX bundle from.
• BEAM_SIZE — beam width, default 4.
• WORD_CACHE_SIZE — cached word translations, default 8192.
• ONNX_NUM_THREADS — intra-op threads, default min(4, cpu_count).
• DIAG_TIMING — set to 1 to log per-request timings.

Files in this repository

onnx_model/encoder.onnx      5.26 MB  encoder graph (fp16 weights)
onnx_model/decoder.onnx      7.52 MB  decoder graph (fp16 weights)
onnx_model/tokenizer.json   byte-level BPE tokenizer, 278 tokens
onnx_model/config.json      architecture + direction ids + special tokens

Load the bundle with IndicTransliterator("onnx_model"); no PyTorch needed.

Limitations

• Transliteration, not translation: it maps sounds and script, so the output is phonetic, not semantic.
• Trained on five Indic languages; other scripts are not supported.
• Accuracy is highest on single words and short phrases. Long sentences are handled word by word, so cross-word context is not modelled.
• Sanskrit-derived or very rare words may be spelled inconsistently.
• Inputs are lowercased for en-* directions, so casing is not preserved.

License

Apache 2.0 — see LICENSE.

Acknowledgements

Trained on the AI4Bharat Aksharantar dataset by AI4Bharat, IIT Madras. Training used Liger Kernel RMSNorm/SwiGLU kernels; inference uses ONNX Runtime and Gradio.