Data Processing Matters: SRPH -Konvergen AI ’s Machine Translation System for WMT ’21 (2021)

Abstract: In this paper, we describe the submission of the joint Samsung Research Philippines-Konvergen AI team for the WMT’21 Large Scale Multilingual Translation Task - Small Track 2. We submit a standard Seq2Seq Transformer model to the shared task without any training or architecture tricks, relying mainly on the strength of our data preprocessing techniques to boost performance. Our final submission model scored 22.92 average BLEU on the FLORES-101 devtest set, and scored 22.97 average BLEU on the contest’s hidden test set, ranking us sixth overall. Despite using only a standard Transformer, our model ranked first in Indonesian to Javanese, showing that data preprocessing matters equally, if not more, than cutting edge model architectures and training techniques.

## introduction this paper describes the machine translation system submitted by the joint team of samsung research philippines and konvergen ai for the wmt'21 large scale multilingual translation task. our team participated in small track #2, where the task is to produce a multilingual machine translation system for five southeast-asian languages: javanese, indonesian, malay, tagalog, and tamil 1 , plus english, in all 30 directions. we will first describe the filtering heuristics that we used to preprocess the data, and then outline the steps we took to train and evaluate our models. specific hyperparameters, preprocessing decisions, and other training parameters will be listed in their corresponding sections. finally, we report our results on the flores-101 @xcite devtest set, as well as on the competition's hidden test set. ## parallel text preprocessing heuristics the contest dataset comprises of various bitext sources, including: bible-uedin @xcite , ccaligned @xcite , elrc 2922 foot_0 , multiccaligned @xcite , paracrawl foot_1 , ted2020 @xcite , wikimatrix @xcite , tico-19, ubuntu, opensubtitles, qed, tanzil, tatoeba, globalvoices, gnome, kde4, and wikimedia @xcite . we preprocess the datasets before training in order to minimize spurious relations that originate from incorrect text pairs. our preprocessing removes samples based on a few heuristics that we developed based on our observation on the datasets. each bitext file is applied a different set of preprocessing based on observation. for example we filter by number content for datasets such as ccaaligned while ted2020 is not applied that same filter. in this section, we will cover the decisions made during preprocessing. we observe a score increase of 1.91 bleu on our submission model when the preprocessing is applied. we report the total number of lines filtered from the bitext for all language pairs on ## results after training our models and producing sample translations from the flores-101 devtest set, we compare the results of our two models with a number of baselines: all analyses and results within this section are based on the public devtest set and not the contest's hidden test set, unless specified. a summary of the bleu scores for all models and baselines are available on