Similar Language Translation for C atalan, P ortuguese and S panish Using M arian NMT (2021)

Abstract: This paper describes the SEBAMAT contribution to the 2021 WMT Similar Language Translation shared task. Using the Marian neural machine translation toolkit, translation systems based on Google’s transformer architecture were built in both directions of Catalan–Spanish and Portuguese–Spanish. The systems were trained in two contrastive parameter settings (different vocabulary sizes for byte pair encoding) using only the parallel but not the comparable corpora provided by the shared task organizers. According to their official evaluation results, the SEBAMAT system turned out to be competitive with rankings among the top teams and BLEU scores between 38 and 47 for the language pairs involving Portuguese and between 76 and 80 for the language pairs involving Catalan.

## introduction in recent years, neural machine translation (nmt) has become the state of the art in machine translation (mt). using toolkits such as marian nmt @xcite , it is relatively straightforward to construct end-to-end nmt systems which need only little pre-processing of the training corpora and post-processing of the system output. as nmt is a supervised approach to mt based on machine learning technology, training is usually conducted using sentence aligned human translations. given a large number of source/targetlanguage sentence pairs, the neural system fully automatically learns how to translate. the sebamat submission to the similar language translation (slt) task 1 of the 6th conference on mt is based on work conducted as part of the sebamat project 2 (semantics-based machine translation; @xcite . this project has a focus on experiments introducing semantics into mt but, for comparative purposes, also deals with standard nmt systems. the latter were used as the basis for the current shared task. during the sebamat project a number of mt systems had been developed for language pairs involving english, french, german, greek and spanish, but there had been no prior work on catalan and portuguese. the aims of the participation in the slt shared task were the following:  see in how far the sebamat-based mt systems are competitive.  extend the number of sebamat languages by catalan and portuguese.  find out whether systems for new language pairs can be developed in a very short time.  see whether reasonably well working systems can be developed without much proficiency of the respective languages on the developer side. ## experiments to obtain any information on the training data and on the development and test sets, it was required to register for the shared task. we did so on july 13, 2021, so had seven days until july 19 when the submission of the results was due. this did not leave us much time for parameter optimization which is why we mostly took the standard parameters as suggested in the marian nmt documentation for the transformer architecture @xcite . we only did a few test runs with various settings concerning the number of merge operations in byte pair encoding (later to be referred to as vocabulary size), but did not have time to systematically optimize it. in our primary submissions, this parameter is set to 40,000, whereas in the comparative submissions, as in some previous sebamat work, it is set to 85,000. for all language pairs and data sets (development and test), the smaller size performed better in terms of bleu scores, although the exact size appears to be not very critical within a wide range of values. to provide details on the core part of our experiments, in ## results we pre-processed the corpora as described in section 2 and trained the system for the language pairs @xmath0, @xmath1, @xmath2. we then inspected the translation results. for all language pairs the translations looked ok except for span-@xmath3. for this language pair we noticed, apparently because the tokenizer was not well suited for catalan, the following three types of systematic errors in the output:  there were extra spaces within catalan words such as paral•lel because the interpunct (punt volat) was incorrectly interpreted as a separator between words rather than between syllables, which is why during tokenization blanks were inserted around it. we corrected this by replacing all «˽•˽» sequences («˽» stands for blank) in the translation output by «•».  we found extra spaces before the apostrophe in phrases such as «l'efectiu» or after the apostrophe in phrases such as «d'informes». we therefore removed all spaces before and after apostrophes in the translation output.  we noticed that whereas in the translated output «'» was used as the apostrophe, the sample translations in the development set used «´» instead. the reason is probably a discrepancy between training corpora and development sets. assuming that the test set would have the same characteristics as the development set, we replaced in the translation output all occurrences of the former by the latter. especially the substitution of the apostrophes resulted in an improvement of several bleu points, whereas the effects of blank removal before and after apostrophes differed depending on the software used for automatic evaluation. when using tilde's interactive bleu score evaluator 8 this change had no effect, whereas with the moses multi-bleu-detok.perl tool, which we used in our scripts, a small improvement was obtained. the discrepancy can be explained by assuming that tools for computing bleu scores often introduce some forms of tokenization or de-tokenization by themselves, and that these operations can slightly differ between tools. first three segments of test set (portuguese) sebamat transla-tion (spanish) o plano pretende contribuir para a realização dos objetivos da política comum das pescas e, em especial, para garantir que a exploração dos recursos biológicos marinhos vivos restabeleça e mantenha as populações das espécies exploradas acima de níveis que possam produzir o rendimento máximo sustentável (msy). el plan pretende contribuir a la consecución de los objetivos de la política pesquera común y, en particular, garantizar que la explotación de los recursos biológicos marinos vivos restablezca y mantenga a las poblaciones de especies explotadas por encima de niveles que puedan producir el rendimiento máximo sostenible (rms). coordenar, em consulta com a comissão, os métodos de diagnóstico da doença de newcastle nos estados-membros, nomeadamente, mediante: coordinar, en consulta con la comisión, los métodos de diagnóstico de la enfermedad de newcastle en los estados miembros, en particular mediante: por conseguinte, atualmente é preciso cofinanciamento da união para assegurar que a finance watch e a better finance recebam os recursos necessários para atingirem os objetivos pretendidos nos próximos anos, e para proporcionar estabilidade financeira a ambas as organizações, aos seus peritos e ao seu pessoal administrativo, que até agora conseguiram lançar as suas atividades relevantes em pouco tempo. por lo tanto, hoy es necesario cofinanciar la unión para garantizar que finance watch y better finance reciban los recursos necesarios para alcanzar los objetivos deseados en los próximos años y para proporcionar estabilidad financiera a ambas organizaciones, a sus expertos y a su personal administrativo, que hasta ahora han logrado lanzar sus actividades relevantes en poco tiempo. the hallucinations could be detected by looking at the ratio of sentence lengths between a source language sentence and its translation, and/or by detecting repetitive phrases towards the end of a sentence translation. however, according to @xcite , hallucinations are a problem of training data quality, which to improve would have been too time-consuming. we thought of greedy solutions such as cutting off repetitive sentence ends, but did not implement them for lack of time and as they would be hard to justify. ## discussion and conclusions given the observation that the language pairs involving catalan achieved considerably higher evaluation scores than those involving portuguese, the question arises how this can be explained. our somewhat speculative answer is as follows: as catalan's grammar and sentence structure is very similar to spanish, with differences mainly on the vocabulary side, extremely high scores can be achieved because in many cases, often in the form of a word-by-word translation, there is just one obvious way how to translate a sentence for both man and machine. this is only to a lesser extend true for spanish-portuguese, so the lower scores are likely caused by more variability in acceptable translation options, rather than by lower translation quality. when comparing the bleu scores in tables 2 and 3 , it can be seen that our system performed significantly worse on the test sets than it did on the development sets. from this we conclude that probably the test data is less representative of the training data than the development data. problems with overfitting seem unlikely as in the previous sebamat work we had usually used randomly held out sentences of the training data for both development and testing. in such a scenario, the results were very similar in both cases, with only minor unsystematic discrepancies in bleu scores. finally, let us try to answer the questions raised in the introduction. as it was ranked first for @xmath7, it appears that especially our primary systems (with vocabulary size 40,000) are competitive. let us mention, however, that all participating systems showed similarly convincing evaluation scores, so that minor differences in parameter choice (such as vocabulary size) or in the organizers' selection of the test set may have had a noticeable impact on the rankings. like many other studies, this work provides once again evidence how powerful nmt is, and how well the marian toolkit works: within a week it was possible for a single developer to add two new language pairs (in two directions each) to the sebamat portfolio, despite mediocre proficiency of spanish and hardly any proficiency of catalan and portuguese. of course, achieving good translation quality was considerably facilitated by the similarity of the languages and by the good quality and size of the training data provided by the shared task organizers.