Transfer Learning with Shallow Decoders: BSC at WMT 2021’s Multilingual Low-Resource Translation for I ndo- E uropean Languages Shared Task (2021)

Abstract: This paper describes the participation of the BSC team in the WMT2021’s Multilingual Low-Resource Translation for Indo-European Languages Shared Task. The system aims to solve the Subtask 2: Wikipedia cultural heritage articles, which involves translation in four Romance languages: Catalan, Italian, Occitan and Romanian. The submitted system is a multilingual semi-supervised machine translation model. It is based on a pre-trained language model, namely XLM-RoBERTa, that is later fine-tuned with parallel data obtained mostly from OPUS. Unlike other works, we only use XLM to initialize the encoder and randomly initialize a shallow decoder. The reported results are robust and perform well for all tested languages.

## introduction we present the work carried out by the bsc team in the context of wmt2021's first edition of the multilingual low-resource translation shared task. the tasks addresses the issue of multilinguality in machine translation (mt) for lowresource languages, focusing on two language families: north germanic and romance. we take part in the subtask 2, which involves translation in four romance languages: catalan, italian, occitan and romanian. ## background machine translation for low-resource languages is characterised by the lack of sufficient parallel data of a given language pair, either because the combination is infrequent or because the languages involved are themselves low-resource. several works have attempted to overcome this pitfall, using different techniques. a common solution is to employ back-translation @xcite , while other research focuses on using other languages as pivots to compensate for the lack of data @xcite . @xcite ; @xcite make use of monolingual data only. our approach is based on multilinguality. previous works such as vergés boncompte and ruiz costa-jussà (2020); @xcite have shown that the use of multilingual mt is beneficial, as it generalizes better by sharing parameters among all the languages involved, especially if the languages belong to the same linguistic family. at the same time, training of multilingual mt models from scratch usually requires large parallel corpora and may not be feasible in a low-resource and zeroresource translation scenarios. pre-training of large language models from scratch on monolingual data and then fine-tuning them for the specific downstream tasks, has proved to be an extremely successful approach for many natural language processing problems. crosslingual language models such as xlm and xlm-roberta @xcite , that combine unsupervised (monolingual data) and supervised (parallel data) training objectives, perform especially well both on cross-lingual nlu tasks and in machine translation. the idea of combining the power of pre-trained cross-lingual language models with a multilingual machine translation setting naturally follows from there. this idea was explored in @xcite ) where a denoising seq2seq auto-encoder (mbart) based on bart @xcite was pre-trained on extensive monolingual corpora in many languages. a similar approach is implemented in @xcite where alignment information is used to pre-train a multilingual mt transformer on existing public parallel datasets. both approaches require either a computationally intensive pre-training on monolingual data or access to extensive large-scale parallel data. initializing an encoder and decoder of a bilingual mt seq2seq transformer with a pre-trained crosslingual language model was famously proposed in @xcite . a natural next step is to initialize a multilingual mt seq2seq transformer with a shared encoder and a shared decoder by the xlm-like language encoder which was first performed in @xcite . we reuse this idea by initializing the encoder with a pre-trained xlm-roberta @xcite , as in @xcite . however, unlike @xcite , we only initialize the encoder, with the motivation of being able to instantiate a shallower decoder, following previous works that for a given compute budget suggest that it is more efficient to use deeper encoders and shallower decoders @xcite . the encoder-only initialization was already implemented in fairseq @xcite . foot_0 3 experimental framework ## results here we report the official evaluation. 6 we submitted our results a bit later than the deadline due to some bottlenecks in our in-house computational resources. out of 7 competing systems and 2 baselines, our system was ranked 5th in average, 3rd in the catalan-to-occitan direction, 4th in the catalanto-romanian direction and 6th in the catalan-to-italian direction. ## conclusions we have showed that our approach is a simple, yet effective method for multilingual machine translation between linguistically similar languages. the encoder-only initialization allows for having a shallow decoder, which is computationally wise. as future work, we plan to further explore transfer learning techniques in the context of shallow decoders as well as applying different vocabulary pruning techniques.