How far can we get with one GPU in 100 hours? C o AS ta L at M ulti I ndic MT Shared Task (2021)

Abstract: This work shows that competitive translation results can be obtained in a constrained setting by incorporating the latest advances in memory and compute optimization. We train and evaluate large multilingual translation models using a single GPU for a maximum of 100 hours and get within 4-5 BLEU points of the top submission on the leaderboard. We also benchmark standard baselines on the PMI corpus and re-discover well-known shortcomings of translation systems and metrics.

## introduction machine translation is one of the few tasks in nlp which has the luxury of data. due to the efforts of the community over the last two decades @xcite @xcite , most major languages of the world have millions of translated sentence pairs with english. with the introduction of sequence to sequence models @xcite , transformers @xcite , and large pre-trained language models @xcite @xcite , the accuracy of machine translation models has almost risen to that of humans @xcite . yet, the ability to train such models is limited by the availability of compute. today's state-of-the-art models are trained by industry research labs, using large compute infrastructure which is usually unavailable or unaffordable to others. such training is also shown to have large carbon footprints @xcite . in this work, we show that competitive translation performance can be achieved even with limited resources. we first train a statistical mt system that does not require gpus, as a baseline. next, we run inference on the best publicly available pre-trained models to benchmark their performance. finally, we train graph2seq, seq2seq, and text2text models, which progressively perform better. all our experiments are constrained both in compute 1 and training time: we use one nvidia titan rtx gpu for a maximum of 100 hours. our main findings are: (i) pre-trained seq2seq and text2text models perform the best, especially when trained only on the pmi corpus, (ii) the benefits of pre-trained multilingual language models diminish for indic language decoding due to their under-representation in pretraining data, and (iii) a small empirical evaluation on 2 languages shows that the prediction fluency and faithfulness start plateauing at 100 hours. ## data the multiindicmt data is a combination of parallel corpora from different sources as shown in tab. 1. it contains translations from 10 indic languages to english. the indic languages included are bengali (bn), gujarati (gu), hindi (hi), kannada (kn), malayalam (ml), marathi (mr), oriya (or), punjabi (pa), tamil (ta), and telugu (te). the training split contains about 11 million translation pairs 1 which in turn constrains the number of model parameters from these languages. the development and test splits contain 1000 and 2390 11-way parallel sentences taken from the pmindia corpus @xcite , respectively. analysis to understand the data better, a small analysis is performed by randomly sampling 100 sentences from each language the authors can read (hi and kn). overall, the translations are of high quality, except in a few sources where the parallel sentences are automatically extracted. for example, we found that jw @xcite has alignment issues, where a part of the translation is moved to the next line, thereby starting a chain of misalignments, as shown in fig. 1 . we manually annotate 100 translations for fluency and faithfulness on a scale of 0-5 and get a score of 4.01 for fluency and 3.54 for faithfulness. ## models we train four types of models: (i) a phrase-based statistical model, (ii) a graph-to-text model, (iii) a sequence-to-sequence model, and (iv) a text-to-text model. brief descriptions of the models are given below. ## results we report results in both english to indic, and indic to english directions. we use character f1 and bleu @xcite , which are standard metrics to evaluate translations. we train two variants of all models: (i) only on the pmi corpus, and (ii) on the full training data. the english to indic results are shown in tab. 2 and the indic to english results, in tab. 3. foot_1 the ability of simpler models to learn quickly in constrained environments. we also note that just training on the pmi corpus gives a result that is almost on par with the results obtained by training on the entire training split. the model trained on pmi even surpasses the other model, on kannada indicating a strong in-domain training bias. gcn in this setup, we only train on the pmi corpus due to time constraints. we find that while it comfortably surpasses moses, it also comes close to the much bigger models, especially when translating to indic languages. it is to be noted that, this small gap in results can be mainly attributed to the lack of convergence of the bigger models, as discussed next. muril and bert following the mt5 models, these models also perform better when trained only on the pmi corpus as it fails to converge on the larger data in the given time. as an additional step, we finetune these under-fit models on the pmi data for 5 hours and see a significant performance gain in the english to indic direction (bert2muril). the model outperforms the much bigger mt5 on 7 languages with gujarati, hindi, and tamil being the exceptions. however, finetuning does not seem to have a major effect in the other direction (muril2bert). as in the case of mt5, we believe that the bert decoder's pre-training subsumes any gains from extra finetuning. official evaluation since tab. 2 and 3 show bleu scores obtained by evaluating the generated predictions locally, they do not exactly match the official scores on the leaderboard. 5 for a fair comparison, we present both local and official bleu scores of our best submissions in tab. 4. we see that the scores are similar when translating from indic languages to english. but when translating from english, the official scores are often significantly higher. this is a result of our use of minimal tokenization (mteval-v13a) before computing bleu, while the official evaluation uses the indictokenizer @xcite . ## discussion as reported in §4, the text2text and seq2seq models perform better when trained only on the pmi corpus when compared to them being trained on the entire train split. though it can be argued that they perform better since the test set also comes from the same domain, 6 we hypothesize that 100 hours is not enough time for the models to converge when trained on the full training set. fig 2 shows the bleu scores of bert2muril model after 80 and 100 hours of training, respectively. we see that the model gets significantly better in the last 20 hours. a 5 hour finetuning with the pmi corpus, further increases its performance. this clearly shows that the model would become more accurate if it is trained for a longer period or with more compute. to establish whether an increase in bleu scores corresponds to an increase in the fluency and faithfulness of the translations, we manually annotate 50 hindi and kannada test predictions from the best model to find that the increase in both cases is marginal. in the 20 additional training hours, the fluency and faithfulness increased by 0.005 and 0.01 respectively which suggests that bleu may not be the best metric to quantify the goodness of translation systems, as shown in works like @xcite . ## conclusion in this work, we show that it is possible to get competitive translation results using a single gpu for a limited amount of time by carefully selecting and training large pre-trained encoder-decoder models. we also show that we can train models which have more than 10 9 trainable parameters using the latest advances in gpu resource optimization. finally, through a small empirical study, we find that while longer training can increase bleu scores, it may not increase their fluency and faithfulness.