S amsung Research P hilippines - Datasaur AI ’s Submission for the WMT 22 Large Scale Multilingual Translation Task (2022)

Abstract: This paper describes the submission of the joint Samsung Research Philippines - Datasaur AI team for the WMT22 Large Scale Multilingual African Translation shared task. We approach the contest as a way to explore task composition as a solution for low-resource multilingual translation, using adapter fusion to combine multiple task adapters that learn subsets of the total translation pairs. Our final model shows performance improvements in 32 out of the 44 translation directions that we participate in when compared to a single model system trained on multiple directions at once.

## introduction in this paper, we describe two systems that we submit to the wmt22 large scale multilingual african translation shared task: a baseline finetuned mt5 @xcite ) model trained on multiple directions at once (referred to as srph-dai-baseline), and an mt5 model successively finetuned with task composition using multiple pair-specific adapters (referred to as srph-dai-fusion). we first outline the preprocessing steps and filtering heuristics used to clean the contest dataset, then we show the training setup and experimental design used for constructing our submitted systems. we then report our results on the hidden test set via bleu, spbleu, and chrf2++ automatic evaluation metrics. ## preprocessing in this section, we detail the preprocessing steps used to filter the contest dataset to ensure that data quality is as high as possible. given that the contest dataset contains sentence pairs that were artificially aligned from crawled * work done while at konvergen ai. data, we use a number of filters to reduce the possibility of mismatched pairs in the final training dataset: pair samples afr ↔ eng 2,526,513 amh ↔ eng 315,870 fuv ↔ eng 953,002 hau ↔ eng 1,841,974 ibo ↔ eng 136,534 kam ↔ eng 1,143,082 kin ↔ eng 7,143,167 lug ↔ eng 2,058,590 luo ↔ eng 1,713,159 nso ↔ eng 1,600,977 nya ↔ eng 1,289,859 orm ↔ eng 1,786,712 sna ↔ eng 5,917,741 som ↔ eng 413,647 ssw ↔ eng 77,807 swh ↔ eng 18,243,580 tsn ↔ eng 3,034,232 tso ↔ eng 383,586 umb ↔ eng 190,170 xho ↔ eng 5,481,855 yor ↔ eng 923,055 zul ↔ eng 2,645,396 after applying the filters for the entire dataset, we perform one deduplication step to ensure that no duplicate entries have been added. no further preprocessing is done on the data itself to preserve as much information within the sentences as possible. when formatting the data for translation training, we insert a target language token at the beginning of the sentence. for example, a sentence to be translated from english to afrikaans would look like: <@xmath0. we only participate in a subset of the shared task's translation pairs (44 total directions), opting to train only on @xmath1. ## experiment design in this section, we describe the construction of our two submitted systems: srph-dai-baseline and srph-dai-fusion. ## results we outline the performance of our two models on the hidden test set on overall, srph-dai-fusion outperforms srph-dai-base on average across all three metrics, with an improvement of 0.09, 0.19, and 1.33 on average bleu, spbleu, and chrf2++, respectively. both models perform relatively better on the african to english translation directions compared to the english to african ones. we hypothesize that this is likely due to english being a pivot language, and thus cross-lingual and cross-task information learned while training each pair contributed to better performance when translating into english. when comparing the two models, we note an "improvement" in the performance if at least two of the three metrics had an increase in score. we observe that 32 out of the 44 translation directions had an improvement once task composition was used for finetuning, most of which are very lowresource pairs. best gains are observed in the english to african translation directions, with some pairs such as @xmath2. we observe that srph-dai-base outperforms the task composition model in cases where there is a relative abundance of training data. for pairs that have sub-million examples, srph-dai-fusion performs much better, likely due to the model being able to learn more specialized information about these translation directions separate from the other directions. interestingly, we observe that for language pairs with a relative abundance of data, the drop in performance when using task composition is substantial. for example, @xmath3.2, 2, and 4.2 points drop in bleu, spbleu, and chrf2++, respectively. we hypothesize that this is because srph-dai-fusion has more intact task-specific knowledge related to low-resource pairs that may not be useful to the higher-resourced pairs. since task adapters are frozen during fusion layer training, the model has an added burden in learning how to adapt knowledge that may not be useful when translating higher-resourced translation directions. ## conclusion in this paper, we described our submissions for the wmt22 large scale multilingual african translation shared task. we approached the contest as a way to explore task composition as a solution for multilingual translation, especially among lowresource languages. in our experiments, we show that using task composition -training task adapters to learn pair-specific knowledge, then using a fusion layer to learn cross-task information -improves performance for less-represented language pairs in a multilingual translation dataset. while the model's results for a number of translation directions are far from state-of-the-art, the results show the methodology's promise for further exploration. for future work, we would like to conduct experiments for larger models than is constrained by our resources. we expect that using base and large variants of mt5 would further improve performance for all language pairs. in addition, it would be beneficial to test the methodology while adding in common "best practices" in translation such as using backtranslated data and better data sampling. lastly, we would like to explore setups where the pair-specific task adapters are transformable to some extent instead of being fully frozen as a remedy to the problem of higher-resourced pairs performing worse in the task composition setup.