T 4 T Solution: WMT 21 Similar Language Task for the S panish- C atalan and S panish- P ortuguese Language Pair (2021)

Abstract: The main idea of this solution has been to focus on corpus cleaning and preparation and after that, use an out of box solution (OpenNMT) with its default published transformer model. To prepare the corpus, we have used set of standard tools (as Moses scripts or python packages), but also, among other python scripts, a python custom tokenizer with the ability to replace numbers for variables, solve the upper/lower case issue of the vocabulary and provide good segmentation for most of the punctuation. We also have started a line to clean corpus based on statistical probability estimation of source-target corpus, with unclear results. Also, we have run some tests with syllabical word segmentation, again with unclear results, so at the end, after word sentence tokenization we have used BPE SentencePiece for subword units to feed OpenNMT.

## introduction we suspect this 50 bleu score difference is direct result of corpus quality or diversity. @xmath0, the dog (the official catalan government diary) (approx 40% of words), and even though @xmath1, its legal composition (europarl/jarc) is based in probably a mix of indirect translations. we think one of the best ways to improve a nmt system, is to use the best data you can. we have have focused in the physical cleaning of the corpus (duplicate strings, unusual sentences, tokenized text in some sources, deal with the utf-8 universe coding for punctuation, numerical data and the upper/lower casing issue). we have developed a set of python programs for these cleaning tasks and an adhoc tokenizer. we also have tried also to run some cleaning procedure based in some basic statistical information of the bitext corpus. as there is a quite large sourcetarget, we have scored word probabilities in bitext corpus sentences, and then somehow score sentence probabilities and decide to use or not these sentences. this simple cleaning has indeed increased the score of the model for corpus in-data, but is not so clear if it helps with data out of the corpus. the last step in data preparation, to deal with the vocabulary size issue, has been the subword segmentation. we have used python standard tools for syllabic segmentation with good results (corpus data has achieved best score than bpe (byte-pairenconding), but again, with data outside of the corpus, bpe @xcite has proven better. at the end, we have used google sentencepiece @xcite ) bpe implementation. after that, we have used what we think a proven toolkit for nmt, opennmt @xcite , out of the box, without any modification, using its web publish options for the transformer model. in the last step we have used the inverse python programs in order to generate the final version of the test source translated file. we have focused the system from a practical engi-neering point of view. the whole project has been based in currently available local only mid size system, with consumer grade multi-gpu environment. following this fast approach, and due the nature of the main focus on the corpus, we have used simple models that opennmt provides (2-layer lstm with 500 hidden units on both the encoder and decoder) in order to choose several options and parameters used later for the final transformer model, as this last model is close to the limit a midsize system can provide. ## cleaning the corpus in our approach we have joined all data sources (all monolingual sources and the matching language for the bilingual text corpus) in order to create a mono corpus, and the bilingual text corpus. for the model training we have used the dev data provided by the organization. the typical size for this file in opennmt is around 2000 lines, so we have used data from the bilingual text corpus in order to reach this typical size. ## tokenization one of the big issues to deal with real data, is the tokenization. after reviewing several available tools, we ended creating a python custom tokenizer that has the following features. these features provide a robust @xmath2. ## evaluation after testing in more simple neuronal network models (lstm) the final setup has consisted of a corpus cleaned (in the physical sense, and also in an statistical sense removing approx 15% of sentence corpus sentences with the highest perplexity). this clean corpus has been detokenized with our adhoc tokenizer (that lowercase the corpus, replaces numbers by variables, and handles punctuation and upper/lower casing). after this cleaning, the number of words for @xmath3.3m words) and @xmath4.5m lines (176m words). then we have used 16000 terms sentencepiece bpe vocabulary on this detokenized corpus in order to reduce vocabulary. we have removed sentences with more than approx 170 tokens for the sentences the neuronal network has ingested (this length has kept the model below the memory limit of each one of the gpu cards). we have set the model configuration using the published transformer @xcite model in the opennmt site ( https://opennmt.net/opennmtpy/faq.html#how-do-i-use-the-transformermodel). according opennmt documentation, this setup mimic the google @xcite setup that replicate its wmt results. we have tested our models against test data form our corpus (23_sp_transf_statclean_3,5 for @xmath5.5 for @xmath6.v2 @xmath7. in figure 1 we can see the results for the @xmath8. the transformer model converges really fast after 30-40k steps (as the size of the corpus is not very large). we have used the best score (pt->es bleu score 55.96 at 55k steps and es->pt bleu score 54.68 at 60k steps) for the final evaluation. in figure 2 we can see the results for the @xmath9. we have used the best score (ca->es bleu score 84.34 at 70k steps and es->pt bleu score 83.77 at 85 steps) for the final. ## results in ## conclusions we think we have accomplish the objective to achieve good results with good data and out of box toolkit as opennmt. it has proven more difficult than expected find recipes to improve the corpus quality beyond the physical cleaning. what we have found suggest (without prove) that: