A Preliminary Exploration of Phrase-Based SMT and Multi- BPE Segmentations through Concatenated Tokenised Corpora for Low-Resource I ndian Languages (2025)

Abstract: This paper describes our methodology and findings in building Machine Translation (MT) systems for submission to the WMT 2025 Shared Task on Low-Resource Indic Language Translation. Our primary aim was to evaluate the effectiveness of a phrase-based Statistical Machine Translation (SMT) system combined with a less common subword segmentation strategy for languages with very limited parallel data. We applied multiple Byte Pair Encoding (BPE) merge operations to the parallel corpora and concatenated the outputs to improve vocabulary coverage. We built systems for the English–Nyishi, English–Khasi, and English–Assamese language pairs. Although the approach showed potential as a data augmentation method, its performance in BLEU scores was not competitive with other shared task systems. This paper outlines our system architecture, data processing pipeline, and evaluation results, and provides an analysis of the challenges, positioning our work as an exploratory benchmark for future research in this area.

## introduction machine translation (mt) has advanced rapidly in recent years, primarily driven by neural architectures and the availability of large-scale parallel corpora. however, these benefits are often confined to high-resource languages @xcite , leaving many languages with little or no translation support @xcite . the wmt 2025 shared task on low-resource indic languages translation addresses this gap by focusing on languages spoken in india with scarce digital resources. while many efforts adapt high-resource mt techniques to low-resource settings, direct transfer often fails due to the data-hungry nature of neural networks. this has led to research in areas such as word segmentation and other preprocessing strategies @xcite @xcite @xcite @xcite to make systems more viable under data constraints. although our focus here is on bilingual mt, we acknowledge the rise of multilingual and decoder-only models. our goal was to investigate how far statistical models, combined with multiple bpe segmentations @xcite , could be pushed in highly constrained settings. bilingual mt systems have been successfully developed for other under-represented languages, such as cantonese-mandarin @xcite , english-luganda @xcite , @xcite , bavarian-german @xcite , and english-manipuri @xcite , often with transformer-based architectures and customised segmentation like bpe @xcite . previous work @xcite @xcite has shown that, for some low-resource indic languages, smt can outperform nmt. for the wmt 2025 shared task @xcite , we therefore chose smt for our systems targeting english ↔ {assamese, khasi, manipuri}. the organisers provided parallel corpora for english-kokborok, english-bodo, english-nyishi, english-manipuri, english-khasi, english-mizo, and english-assamese, building on earlier iterations @xcite . we set out to determine whether a robust, traditional method like smt-enhanced with a multiple-bpe data augmentation technique-could remain a viable option in such low-resource scenarios. this paper describes our approach and analyses the performance of our submissions. ## background low-resource mt faces unique challenges due to the scarcity of high-quality parallel corpora. data sparsity leads to out-of-vocabulary (oov) issues and poor generalisation. while neural machine translation (nmt) dominates for high-resource pairs, its high data requirements limit its applicability without substantial augmentation @xcite or multilingual transfer @xcite . phrase-based smt is often more resilient to small data sizes. by learning from statistical alignments of phrase pairs, it can perform robustly with limited resources. for languages in the indic family, which often exhibit rich morphology, subword segmentation is an effective preprocessing step @xcite . bpe, in particular, balances wordlevel and character-level representations, reduces vocabulary size, and mitigates oov problems. inspired by @xcite , we extended this idea by applying multiple bpe merge operations to produce diverse segmentations, concatenating them to create a richer training set. ## data the shared task corpora were drawn from previous wmt datasets and new resources @xcite @xcite . after preprocessing, we obtained the training statistics shown in our preprocessing steps included: the segmented corpora from each merge setting were concatenated and deduplicated @xcite , resulting in the statistics in ## system description we used moses @xcite for phrasebased smt, with target-side kenlm language models @xcite trained on the corpora in ## results our systems were tested on the official wmt 2025 english-nyishi, english-khasi, and english-assamese sets. in all cases, performance ranked in the lower tier, with bleu scores notably behind top-performing nmt systems that likely used external data or more advanced architectures. full results are shown in tables 3 and 4 . ## discussion the modest results highlight the limitations of smt in this setting. two main factors likely contributed: noisy and domain-specific terms in the provided corpora may have further impacted translation quality.