vanishingMonk/indictrans2-en-kn-ft-cleaned

IndicTrans2 English–Kannada Fine-Tuning

Fine-tuning workflow for the ai4bharat/indictrans2-en-indic-1B model on English–Kannada parallel text using Hugging Face Transformers and the IndicTransToolkit.

Setup

conda create -n indic python=3.10 -y
conda activate indic
pip install -r requirements.txt

Place CSV files with english_sentences and kannada_sentences columns under:

datasets/
├── train/train.csv
├── validation/validation.csv
└── test/test.csv

Usage

Run fine-tuning directly:

python indic_finetune.py

On a Slurm cluster, adjust the account, environment, and project path in run.sh, then submit:

mkdir -p logs
sbatch run.sh

Run the sample translation script with:

python indic_inference.py

Sample output (direct inference before FFT)

eng_Latn: When I was young, I used to go to the park every day.
kan_Knda: ನಾನು ಚಿಕ್ಕವಳಿದ್ದಾಗ, ಪ್ರತಿದಿನ ಉದ್ಯಾನವನಕ್ಕೆ ಹೋಗುತ್ತಿದ್ದೆ.

eng_Latn: We watched a new movie last week, which was very inspiring.
kan_Knda: ನಾವು ಕಳೆದ ವಾರ ಹೊಸ ಚಲನಚಿತ್ರವೊಂದನ್ನು ನೋಡಿದೆವು, ಅದು ಬಹಳ ಸ್ಪೂರ್ತಿದಾಯಕವಾಗಿತ್ತು.

eng_Latn: If you had met me at that time, we would have gone out to eat.
kan_Knda: ಆ ಸಮಯದಲ್ಲಿ ನೀವು ನನ್ನನ್ನು ಭೇಟಿಯಾಗಿದ್ದರೆ, ನಾವು ತಿನ್ನಲು ಹೊರಗೆ ಹೋಗುತ್ತಿದ್ದೆವು.

eng_Latn: My friend has invited me to his birthday party, and I will give him a gift.
kan_Knda: ನನ್ನ ಸ್ನೇಹಿತ ತನ್ನ ಹುಟ್ಟುಹಬ್ಬದ ಸಂತೋಷಕೂಟಕ್ಕೆ ನನ್ನನ್ನು ಆಹ್ವಾನಿಸಿದ್ದಾನೆ ಮತ್ತು ನಾನು ಅವನಿಗೆ ಉಡುಗೊರೆಯನ್ನು ನೀಡುತ್ತೇನೆ.

Before training, build the cleaned dataset with:

python clean_training_data.py

Each run uses the next available numbered directories. For example, the latest run wrote checkpoints to indictrans2-kannada_1/ and the selected final model to indictrans2-kannada-final_1/. This prevents a new run from overwriting an earlier model.

Latest fine-tuning run

The latest run introduced the following changes:

• Added clean_training_data.py and trained from datasets/train_cleaned.csv instead of directly reading all raw shards.
• Reduced 800,000 raw training pairs to 595,149 cleaned pairs by normalizing Unicode and whitespace and removing missing or script-mismatched text, validation/test source leakage, extreme length ratios, question/answer mismatches, assistant-style responses, degenerate repetition, duplicate pairs, and sources with conflicting targets.
• Kept an additional source-level leakage check in the training script. It removed zero rows from the already-cleaned file.
• Reduced training from three epochs to one to limit overfitting.
• Added an attention-dropout probability of 0.1.
• Added automatically numbered checkpoint and final-model directories.

The remaining training configuration used a learning rate of 2e-5, batch size 16, gradient accumulation over four steps, BF16, weight decay 0.01, warmup ratio 0.05, gradient clipping at 1.0, gradient checkpointing, and best-checkpoint selection by validation loss.

Training completed 9,300 optimizer steps in approximately 58 minutes 56 seconds on an H100. The final training loss was 0.3041. The selected checkpoint was step 9,000, with a validation loss of 0.2345.

Benchmark results

All models were evaluated on the same 1,000 English–Kannada test examples. BLEU and chrF++ (word order 2) are corpus scores from SacreBLEU, while COMET uses Unbabel/wmt22-comet-da. Higher scores are better.

Model	BLEU	chrF++	COMET
IndicTrans2 base	80.3726	91.3279	0.9657
Initial fine-tune	27.1712	72.9206	—
Earlier cleaned-data fine-tune	62.5845	84.6822	0.9522
Latest cleaned-data fine-tune	71.7596	86.5866	0.9579

Relative to the earlier cleaned-data checkpoint, the latest run improves by 9.1751 BLEU, 1.9044 chrF++, and 0.0057 COMET. Relative to the base model, it remains lower by 8.6130 BLEU, 4.7413 chrF++, and approximately 0.0078 COMET. The results show that the stricter cleaning and revised training setup closed a large part of the gap without surpassing the pretrained baseline.

Limitations of reference-based metrics

BLEU and chrF++ depend strongly on surface-level overlap with the reference translation. This can penalize translations that are semantically correct but use different Kannada vocabulary or phrasing.

For example:

English:               Watchful eye
Reference:             ಎಚ್ಚರದ ಕಣ್ಣು
Fine-tuned prediction: ಜಾಗರೂಕತೆಯ ಕಣ್ಣು

Both Kannada expressions are valid translations of "watchful eye." However, the words ಎಚ್ಚರದ and ಜಾಗರೂಕತೆಯ have relatively low character-level overlap. Therefore, chrF++ may reduce the score even when the predicted translation is semantically valid.

An example from an earlier fine-tuning run is:

English:               Internet, eh?
Reference:             ಇಂಟರ್ನೆಟ್, ಅಲ್ಲವೇ?
Fine-tuned prediction: ಅಂತರ್ಜಾಲ, ಅಲ್ಲವೇ?

ಇಂಟರ್ನೆಟ್ is a commonly used borrowed form of "Internet," while ಅಂತರ್ಜಾಲ is a native/formal Kannada equivalent. Both translations preserve the intended meaning, but BLEU and chrF++ penalize the fine-tuned prediction because the lexical forms differ substantially.

Not every difference is equally valid, however:

English:    There is war.
Reference:  ಅಲ್ಲಿ ಯುದ್ಧ ನಡೆಯುತ್ತದೆ.
Prediction: ಯುದ್ಧವೂ ಇದೆ.

The suffix ವೂ introduces a meaning similar to "also/too," which is not explicitly present in the English source. Therefore, some differences represent genuine semantic deviations rather than only alternative wording.

COMET evaluation

Because BLEU and chrF++ can penalize valid paraphrases and synonyms, COMET was also used to measure semantic translation quality. The model used was Unbabel/wmt22-comet-da.

COMET evaluates the source sentence, machine translation, and reference translation, and produces both sentence-level scores and an overall system-level score.

The latest fine-tuned model obtains 0.9579 COMET, compared with 0.9522 for the earlier cleaned-data model and 0.9657 for the base model. Its semantic-quality gap to the base model is therefore much smaller than its BLEU and chrF++ gaps. This supports the observation that some lower reference-overlap scores arise from alternative but valid Kannada wording. Nevertheless, the base model still has the highest system-level score.

Sentence-level COMET analysis

The base and latest fine-tuned models were compared using:

ΔCOMET = COMET_finetuned - COMET_base

A tolerance of ±0.01 was used to identify approximately tied examples:

Fine-tuned better: ΔCOMET > 0.01
Base better:       ΔCOMET < -0.01
Tie-ish:           |ΔCOMET| <= 0.01

The results across the 1,000-example test set were:

Comparison	Number of examples	Percentage
Latest fine-tune better	81	8.1%
Base better	248	24.8%
Approximately tied	671	67.1%

The mean difference was Mean ΔCOMET = -0.007808, consistent with the rounded system-level difference. Most translations are semantically close: 67.1% fall within the ±0.01 tolerance. Among the remaining examples, however, the base model wins more often (248 examples versus 81).

Qualitative observations

Manual spot checks in inspection_new_FT.txt found several latest-model translations that preserve the source meaning despite differing from the single reference. For example:

English:    Rough and tough
Reference:  ಗಟ್ಟಿಮುಟ್ಟಾದ ಮತ್ತು ಕಠಿಣ
Fine-tuned: ಒರಟಾದ ಮತ್ತು ಗಟ್ಟಿಯಾದ
COMET:      0.574645

English:    So it was not passed.
Reference:  ಹೀಗಾಗಿ ಪಾಸ್ ಆಗಲಿಲ್ಲ.
Fine-tuned: ಹೀಗಾಗಿ ಅದನ್ನು ಅಂಗೀಕರಿಸಿರಲಿಲ್ಲ.
COMET:      0.683739

The first example is a valid literal rendering despite its relatively low COMET score. In the second, the reference uses the ambiguous English loanword "pass," while the prediction interprets it as "approved." These examples show why scores against one reference need qualitative context.

The spot check also found genuine problems:

English:    This is not what love is.
Reference:  ಪ್ರೀತಿ ಎಂದರೆ ಇದೇ ಅಲ್ಲ.
Fine-tuned: ಇದು ಪ್ರೀತಿಗೆ ಸಂಬಂಧಿಸಿದ್ದಲ್ಲ.

English:    The group did not budge.
Reference:  ಗುಂಪು ಚಲಿಸಲಿಲ್ಲ.
Fine-tuned: ಗುಂಪು ತಲೆ ಕೆಡಿಸಿಕೊಳ್ಳಲಿಲ್ಲ.

It also produced unnecessary English transliteration in at least one checked case: Name of Post: Management Trainee became ಹುದ್ದೆಯ ಹೆಸರುಃ ಮ್ಯಾನೇಜ್ಮೆಂಟ್ ಟ್ರೈನಿ instead of the reference's Kannada terminology. Qualitative evaluation should therefore complement, rather than replace, automatic metrics.

Conclusion

Fine-tuning ai4bharat/indictrans2-en-indic-1B on the English–Kannada dataset successfully produced a model capable of generating fluent and coherent Kannada translations.

Dataset quality and the revised one-epoch training setup had a substantial effect. The latest model reaches 71.7596 BLEU, 86.5866 chrF++, and 0.9579 COMET, improving on both earlier fine-tuned checkpoints. The pretrained IndicTrans2 model nevertheless remains stronger on the current test set at 80.3726 BLEU, 91.3279 chrF++, and 0.9657 COMET.

The lexical-metric gap should not be interpreted entirely as poor translation quality. Manual inspection found valid alternative Kannada expressions, and the smaller COMET gap supports that observation. At the sentence level, 67.1% of examples were effectively tied within a COMET difference of ±0.01. The latest fine-tuned model performed better on 8.1%, while the base model performed better on 24.8%.

The fine-tuned model can therefore be used as a downstream translation application when its cleaner and more coherent Kannada outputs are preferred. Because this preference is based on qualitative observations and the base model still leads on all three aggregate metrics, the appropriate model should be selected according to the target domain and validated with human evaluation.

Therefore, the current evidence does not support claiming that full fine-tuning improves IndicTrans2's overall English–Kannada translation quality. Instead, the experiment indicates that:

• IndicTrans2 already provides a very strong English–Kannada baseline.
• Stricter dataset cleaning and a shorter run improved full-model fine-tuning.
• Fine-tuning changes lexical and stylistic preferences in Kannada.
• Some fine-tuned outputs are qualitatively preferable despite receiving lower reference-overlap scores.
• The base model currently retains better overall semantic translation quality according to COMET.

Usage

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

repo_id = "Akash-bu/indictrans2-en-kn-ft-cleaned"

tokenizer = AutoTokenizer.from_pretrained(
    repo_id,
    trust_remote_code=True
)

model = AutoModelForSeq2SeqLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    dtype="auto",
)