MUNI - NLP Systems for Low-resource I ndic Machine Translation (2023)

Abstract: The WMT 2023 Shared Task on Low-Resource Indic Language Translation featured to and from Assamese, Khasi, Manipuri, Mizo on one side and English on the other. We submitted systems supervised neural machine translation systems for each pair and direction and experimented with different configurations and settings for both preprocessing and training. Even if most of them did not reach competitive performance, our experiments uncovered some interesting points for further investigation, namely the relation between dataset and model size, and the impact of the training framework. Moreover, the results of some of our preliminary experiments on the use of word embeddings initialization, backtranslation, and model depth were in contrast with previous work. The final results also show some disagreement in the automated metrics employed in the evaluation.

## introduction this paper describes our systems to the wmt 2023 shared task on low-resource indic language translation. the task featured four low-resource languages indigenous to the northeastern regions of the indian subcontinent. the translation was to be done to and from assamese (indo-aryan), khasi (austroasiatic), manipuri, mizo (sino-tibetan) on one side and english on the other. we submitted supervised neural machine translation systems for each pair and direction and experimented with different configurations and settings for both preprocessing and training. we did not use large pretrained models, but trained transformers @xcite of different size and with different parameters for each direction, both on bilingual and multilingual data. even if most of our final systems did not reach a satisfactory or competitive performance, settling for the middle to low part of the scoreboard, we argue that our experiments brought up some interesting points that call for a deeper investigation. chiefly, these are the relation between dataset and model size, and the impact of the training framework. moreover, the final results seem to confirm recent research on the reliability of automatic evaluation metrics, with several cases of disagreements in the ranking of the systems, ranging from one to several places in the leaderboard. ## datasets in the following section, we first briefly present the languages involved, then we give a summary of the datasets, their contents, domains, and structure. ## methodology this section describes our methodology and the baselines we moved from. ## final systems as our final systems, we obtained roughly two kinds of models: the ones trained on bilingual parallel data, and the ones fine-tuned from a multilingual pair. the former were our best for translating english to the all the indic languages, and also to translate from assamese into english. multilingual pretraining and fine-tuning performed better for the remaining directions, khasi, mizo, and manipuri into english. kha-eng and lus-eng were fine-tuned from a parent trained on all the parallel dataset, while mni-eng was fine-tuned from an assamese and manipuri parent. parent models were trained according to the settings in regarding the preprocessing configuration, the settings varied across all the directions. in some cases, such as eng-kha and eng-lus, sticking to separate source and target vocabulary of size 2000 with a frequency filter of 100 resulted still in the best system. however, for eng-mni we found our best system with separate vocabularies of size 500 and a threshold of 200. for multilingual systems, we set the vocabulary size for the indic side to 750 to try and force the learning of more shared subwords. for the english side, we left the value at 2000. there is no clear winner with respect to segmentation algorithm. system architecture is the same for all systems, apart from english to and from assamese. the best architecture was almost always a transformer with 4 encoder/decoder layers, embedding dimension of 256, feedforward dimension of 1024, and 4 attention heads. for the models involving assamese, we found that a deeper model of 18 encoder and 6 decoder layers, embedding dimension of 384, feedforward dimension of 1536, and 4 attention heads performed the best. other hyperparameters were not investigated extensively, so all of our models were trained with the adam optimizer, a learning rate of 1e-3, 4000 warmup updates, a dropout of 0.3, a label smoothing of 0.1, and max tokens for each batch at 16384. ## conclusions this paper describes our experiments and the resulting supervised neural machine translation systems we submitted to wmt23 low-resource indic machine translation shared task. we trained systems for all directions in the task and experimented with hyperparameter tuning and multilingual training. we did not use transfer learning from pretrained systems, and thus our models were not competitive for some directions. nonetheless, we argue that our investigation and preliminary analysis on the behavior of different architecture and preprocessing configuration can be useful to other researchers in the field and exposed some interesting points to be explored in future work. some of our preliminary experiments, such as the use of word embeddings for initialization and backtranslation, did not give the expected results, thus prompting further inquiry.