MUNI - NLP Submission for C zech- U krainian Translation Task at WMT 23 (2023)

Abstract: The system is trained on officialy provided data only. We have heavily filtered all the data to remove machine translated text, Russian text and other noise. We use the DeepNorm modification of the transformer architecture in the TorchScale library with 18 encoder layers and 6 decoder layers. The initial systems for backtranslation uses HFT tokenizer, the final system uses custom tokenizer derived from HFT.

## introduction the annual conference in machine translation (wmt) provides an invaluable platform for researchers to showcase their advancements in this domain. this paper serves as the submission from the natural language processing centre of masaryk university (muni-nlp) team for the czech-ukrainian translation task at wmt23 @xcite . central to our approach is a commitment to data quality and we have also done some experiments with different subword tokenizers. furthermore, we recognize the paramount importance of data filtering, a process that distinguishes signal from noise. to this end, we employ a rigorous data filtering procedure to eliminate machine-translated text, russian content, and other sources of potential distraction, enabling our model to focus on the genuine linguistic intricacies of czech and ukrainian. in terms of model architecture, we employ the deepnorm modification @xcite of the transformer architecture. this modified architecture, integrated within the torchscale library, boasts 18 encoder layers and 6 decoder layers. our approach further delves into the critical aspects of back-translation and tokenization. initially, we leverage the hft (high frequency tokens) tokenizer for back-translation, harnessing the power of synthetic data to enhance model robustness. in the final iteration of our system, we introduce a custom tokenizer derived from hft. ## data selection and preprocessing our system participates in the constrained track, we use only data allowed for this year. we do not use any pretrained models, only selected parallel and monolingual texts. many of the provided text are very noisy. we excluded some of them from training completely. to mitigate the adverse effects of noisy data on our translation system, we conduct a comprehensive analysis of the data problems that emerge from these issues. in subsequent sections of this paper, we delve into the specific strategies and techniques we employ to filter out machine-translated text, russian content, and other sources of noise. our data filtering pipeline is designed to rigorously curate the training data set, ensuring that our model is exposed to high-quality, human-generated translations that align closely with the nuances of the czech and ukrainian languages. through these meticulous filtering strategies, we aim to enhance the overall performance and translation quality of our system. ## tokenization we use hft tokenizer @xcite for all stages. the tokenizer uses special characters to annotate word boundaries and character capitalization, they are listed in an example of tokenized text from the czech part of the training data is in figure 1 . all uppercase letters are transformed to lower-case and the special characters preserve the original format. white spaces around punctuation marks are annotated explicitly in the same way as in sentencepiece source used segments leipzig-news 1m leipzig-newscrawl 1m leipzig-wikipedia 1m statmt-news-crawl.ces 11m languk-ubercorpus 22m languk-news 15m total uk 41m total cs 15m an example of this changes is displayed on the very last line in the figure 1 , the first token is split into two tokens, the number "3 000" at the end of the line is tokenized into two tokens in the original tokenization and into four tokens in the final one. for the initial systems for translation we use vocabulary size of 32,000 items. the final translations system use only 12,000 items in the vocabulary on each side. these modifications in the final system were motivated by an experiment on smaller data where bleu score increased from 22.4 to 24.9. separating individual digits is also an option (disabled by default) in the sentencepiece @xcite tokenizer. we will do a detailed evaluation of this modifications in the future. ## model we use the deepnorm @xcite modification of the transformer @xcite architecture in the torchscale library @xcite . our early experiments with the number of encoder and decoder layers shows with the agreement of @xcite that asymmetric configuration with more encoder layer performs better. we use 18 encoder layers and 6 decoder layers in all our models. the first stage of the system in cs-uk direction is trained only on parallel data (6.6m segments) for 30 epochs, second stage in uk-cs direction uses also 15m czech segments and is trained for 17 epochs. the final system uses parallel data and back-translated ukrainian monolingual data (41m segments). it is trained for only 4 epochs. checkpoints are created every 2000 updates and the final submission is the average of 8 checkpoints (1 following and 6 preceding) around the top-scoring checkpoint on development data. the performance of the individual models are detailed in ## conclusion this paper presents the muni-nlp submission to the wmt 2023 general machine translation task. our results show that it is very important to clean the training data, especially foreign languages. the paper also introduces a novel tokenization into subwords, a detailed evaluation of it is part of our future work.