Lite Training Strategies for P ortuguese- E nglish and E nglish- P ortuguese Translation (2020)

Abstract: Despite the widespread adoption of deep learning for machine translation, it is still expensive to develop high-quality translation models. In this work, we investigate the use of pre-trained models, such as T5 for Portuguese-English and English-Portuguese translation tasks using low-cost hardware. We explore the use of Portuguese and English pre-trained language models and propose an adaptation of the English tokenizer to represent Portuguese characters, such as diaeresis, acute and grave accents. We compare our models to the Google Translate API and MarianMT on a subset of the ParaCrawl dataset, as well as to the winning submission to the WMT19 Biomedical Translation Shared Task. We also describe our submission to the WMT20 Biomedical Translation Shared Task. Our results show that our models have a competitive performance to state-of-the-art models while being trained on modest hardware (a single 8GB gaming GPU for nine days). Our data, models and code are available in our GitHub repository.

## introduction with the advent of deep neural networks, results in machine translation have recently improved over classical statistical strategies @xcite . for instance, in the third and fourth conference on machine translation (wmt18 @xcite and wmt19 @xcite ), the top-performing systems for english-german and german-english competitions were based on transformers @xcite . transformer models are state-of-the-art architectures for machine translation (mt) tasks and are capable of translating the same word to different words based on the context. for instance, the word 'bank' in portuguese can be translated to 'bench' or 'bank' depending on the context. this work explores translation strategies using language models pre-trained on portuguese and english corpora. more specifically, we investigate the use of text-to-text transfer transformer (t5) pre-trained model for these tasks. t5 is a text-to-text transformer trained with a similar masked language modeling objective as bert. in this model, all target tasks are cast as sequenceto-sequence tasks. an illustration of t5 for the english-portuguese translation task is shown in figure 1 . the main contributions of this work are: the winning system of wmt'19 biomedical competition for en-pt and pt-en translation tasks @xcite ) is a neural machine translation (nmt) system. they used opennmt-py to train a transformer model on seven parallel corpora. however, differently from our models, their model was trained from scratch. recent works @xcite have shown the advantages of using pre-trained models for tasks such as questionanswering and text classification. the intuition is to allow the network to use information from pretraining language representations to increase the performance on specific tasks. @xcite evaluated the use of a pre-trained encoder-decoder model for translation. both encoder and decoder weights were tied, but they were pre-trained on different languages. this is an expensive strategy for techniques that use a trainable tokenizer, such as sentencepiece @xcite , because it is necessary to re-train the entire model if the vocabulary changes, as new token embeddings need to be learned. many commercial systems, such as google translate (gt) and amazon translate (at), have an excellent performance on mt, but they are expensive if one needs to translate vast amounts of text. for example, we estimate that it would cost 50,000 usd to translate the 20 million sentences of paracrawl using gt. unfortunately, no commercial system that we are aware of provides metric scores on public datasets that would allow us to compare their systems to ours. ## methods we proposed two main strategies for translating: using a t5 model pre-trained on a portuguese corpus and adapting the original t5 tokenizer to work with portuguese texts. ## datasets we trained our models using six different datasets, and we evaluate our system on two datasets: the wmt19 biomedical translation task dataset and a subset of 99,000 sentence pairs of the paracrawl dataset. we also present the results of our submission to the wmt20 biomedical translation task competition. figure 2 : an example of separated tokens merged back into a single word. our algorithm searches for an isolated special token (in this case, 'ã') and merges it with its neighbors. it can be merged at the beginning, middle, or end of a sentence. ## experiments we conducted several experiments using different configurations of t5. we divided the experiments into two groups: model hyperparameter optimization and different pre-training studies. all experiments were performed on a desktop computer with an nvidia 8gb rtx 2070 super, 32 gb ram memory, and a 4-core intel processor running on ubuntu 18.04. we used pytorch @xcite , huggingface transformer, and pytorch-lightning (falcon, 2019) frameworks to train and evaluate our models. ## wmt19 and wmt20 results we now evaluate our models on the wmt19 biomedical translation task and the results of our best models and official submission to the wmt20 biomedical translation task. in we believe that the improvement of portuguese pre-training models is associated with ptt5's training strategy that uses english pre-trained weights as initial weights. the intuition is that ptt5 carries information from the english model too. the results for wmt20's challenge are in we also evaluated our portuguese pre-training models with the best participants submissions of pt-en en-pt marianmt 27.91 47.44 bsc @xcite the wmt20 biomedical shared task. the results of this comparison are in ## conclusions and future work we show that it is possible to develop english-portuguese translation models close to the state of the art using modest hardware. despite not reaching the same level of performance of google translate on pt-en, the fact that our system was devel-oped mostly by the first author on its personal computer shows that implementing high-quality machine translation systems has become possible for anyone, including small companies and research labs. we also cannot guarantee that google did not use our testing data for training. we presented our submission strategies for the wmt20 biomedical translation shared task using a t5 model. we show that a simple adaption of the original t5 tokenizer to the portuguese language largely improves the translation quality and does not require any further pre-training, which is expensive. however, we achieve the best results with models pre-trained on portuguese. as directions for future work, we plan to experiment with larger models and models pre-trained on both portuguese and english languages simultaneously, as recent work showed that this a successful strategy @xcite . we believe that we could improve the translation results with larger and more complex models @xcite .