The IPN - CIC team system submission for the WMT 2020 similar language task (2020)

Abstract: This paper describes the participation of the NLP research team of the IPN Computer Research center in the WMT 2020 Similar Language Translation Task. We have submitted systems for the Spanish-Portuguese language pair (in both directions). The three submitted systems are based on the Transformer architecture and used fine tuning for domain Adaptation.

## introduction in this paper we describe the neural machine translation(nmt) systems developed by the nlp team of the computer research system of the instituto politécnico nacional, méxico for the similar languages translation shared task of the emnlp 2020 fifth conference on machine translation (wmt 20). for this task, we submit systems for both directions of the spanish-portuguese language pair, all the submitted systems were based in a transformer architecture with a fine tuning for domain adaptation, the difference of the submitted systems was mainly the kind of tokens used (words and sub-word units) and the initialization of the word embeddings in the systems using either a random initialization or pre-trained word embeddings. in the past year task, the submissions of the mllp-upv team @xcite showed that the use of fine tuning was proven to be useful in this specific task, although the test corpus of this year had a higher complexity, the use of fine tuning for context adaptation was also beneficial for translation quality. the paper was organized in the following way. section 2 describes the architecture used for the systems trained, the section 3 describes the corpora used and the pre-processing of the texts, the section 4 gives a description of the training of the system, section 5 gives a description of the method used for obtaining the translations from the trained system, section 6 shows the results for both the internal evaluation and the evaluation of the task, section 7 is a discussion about the impact of pre-trained word embeddings in the submitted systems and section 8 presents the conclusions. ## architecture of the submission the main model for the submission consisted in a transformer model that used tokens composed by words and the word embeddings inside the transformer architecture were initialized using pre trained fast text embeddings in both, source and target languages. for contrastive purposes two additional models were added to the submission , neither of them used a special initialization of the word embeddings and the main distinction between them was the kind of tokens used in the training, the first one used words and the later used sub word units gated by a bpe algorithm. ## corpus description the training data was made up with the available training data for the task, that is jcr, europarl, news commentary and wikititles corpora. the provided development set was randomly split in two disjoint sets of the same size, dev1 and dev2 sets. the data was prepossessed using the following pipeline tokenization, lowercasing and a bpe algorithm learned over the test set. ## training the training for all the systems was carried in a two steps way. in the first step the model was trained using the training set during 200 thousand steps, storing a checkpoint every 5000 steps. for all the checkpoint generated, a translation of the dev-set was computed and bleu evaluated against a tokenized and lowercased version of the dev1 set. due to the conformation of the training data, that is made mostly of parlament sesions (europarl) and scientific journals (jcr) and the observation that this domains doesn't appear in the test data there is an assumption of a domain mismatch between training and test data. due to this mismatch in the second step a fine tunning of the model is conducted. this fine tuning was trained from the best bleu scored checkpoint and a retraining was made using the dev1 set up to 3000 steps, storing a checkpoint every 10. for all the stored checkpoints, a translation of the test set was computed and evaluated with bleu against a tokenized and lowercased version of the dev2 set. for the generation of the translations for the submission the checkpoint with the best bleu score in the fine tuning step was used. ## translations generation in order to get the translation for the evaluation the translate.py script of open nmt was used with a beam of size 5 and a length penalization with alpha of 5. after getting the translations the texts was passed through a recaser trained over the training corpora using the script recaser.perl from moses, after this step a detokenizer was used. ## results the tables 2 and 3 shows the bleu obtained in the evaluation of the three different models before and after the fine tuning for the es-pt and pt-es language pairs respectively. in this internal evaluation of the models the primary model outperforms the baseline models by 2.7 bleu points for es-pt direction and 0.18 points in pt-es. ## impact of the pre trained word embeddings the pre-trained word embeddings used for the model were filtered in order to include only the words that was present in either the development set or the test set and preprocessed using the script embeddings to torch.py included in the open nmt toolset. the used word embeddings showed an imfigure 2: training for pt -es provement in the translation quality (according to bleu). figures 1 and 2 show comparatives for the es-pt and pt-es directions respectively for the first 100k training steps for the model 1 and 3, that means comparing both systems trained in words with the unique difference is that system 1 has fast text pre-trained word embeddings. from the comparative in figures 1 and 2 we can extract that the use of pre-trained word embbedings was beneficial in the beginning of the training, with a difference of around 2 bleu points in the first 5000 training steps for both es-pt and pt-es directions. similar results were seen in the fine tuning of the systems, in this case due to the short amounts of epoch for the training of this step a more detailed table is not provided, but in the tables 2 and 3 a difference of 2.7 and 0.3 bleu points can be seen for the es-pt and pt-es directions respectively. ## conclusions the initialization using fast text had a beneficial result in this low resource scenario, but the em-beddings used were trained in a general context, is possible that pre-trained the embeddings in the specific context could gather better results. in this specific experiment both contrastive models had similar results, independently of the kind of tokens used during the training. compared with the 2019 edition of the slt task, this year the test corpus had a different domain, resulting in a lower bleu score using similar techniques, but also in this year the use of fine tuning improved the translation margin in around 9 points for es-pt and in almost 7 points for the evaluation using the development set in the primary models. for the next year submission, the use of word embeddings can be expanded using word embeddings trained in a bilingual context or in a similar domain from the one in the test corpora.