E nglish- R ussian Data Augmentation for Neural Machine Translation (2022)

Abstract: Data Augmentation (DA) refers to strategies for increasing the diversity of training examples without explicitly collecting new data manually. We have used neural networks and linguistic resources for the automatic generation of text in Russian. The system generates new texts using information from embeddings trained with a huge amount of data in neural language models. Data from the public domain have been used for experiments. The generation of these texts increases the corpus used to train models for NLP tasks, such as machine translation. Finally, an analysis of the results obtained evaluating the quality of generated texts has been carried out and those texts have been added to the training process of Neural Machine Translation (NMT) models. In order to evaluate the quality of the NMT models, firstly, these models have been compared performing a quantitative analysis by means of several standard automatic metrics used in machine translation, and measuring the time spent and the amount of text generated for a good use in the language industry. Secondly, NMT models have been compared through a qualitative analysis, where generated examples of translation have been exposed and compared with each other. Using our DA method, we achieve better results than a baseline model by fine tuning NMT systems with the newly generated datasets.

## introduction the use of large, quality datasets to train neural network models for specific nlp tasks such as machine translation (mt), summarization, paraphrasing, text generation or dialogue systems is essential to achieve good quality results. data augmentation (da) has recently seen increased interest in natural language processing (nlp) due to the lack of data in low-resource domains or new nlp tasks, and the popularity of large-scale neural networks that require large amounts of training data. despite this recent upsurge, this area is still relatively underexplored. perhaps this is due to the challenges posed by the discrete nature of language data which makes it challenging to make significant da. although current progress in the areas of nlp and mt allows for the analysis, understanding, and automatic generation of increasingly accurate and fluid text, such amounts of data with good quality are hard to find. sometimes, they are too scarce for use during the training of a neural network model. these techniques are often investigated in computer vision @xcite ) and da's adaptation for nlp seems secondary and comparatively underexplored, especially in mt task. there are several works that have been done previously such as @xcite which is a data augmentation approach that targets low-frequency words by generating new sentence pairs containing rare words in new, synthetically created contexts; and @xcite which present a multi-task da approach in which they generate new sentence pairs with transformations, such as reversing the order of the target sentence, which produce unfluent target sentences. during training, these augmented sentences are used as auxiliary tasks in a multi-task framework with the aim of providing new contexts where the target prefix is not informative enough to predict the next word. in this work we introduce a new da technique based on words substitution of a specific type (noun, adjective or adverb) in a sentence using a language model (lm) which generates a new word according to the context of the sentence. in addition, we check the new generated word, in order to maintain the main quality of the sentence. in the rest of this paper, our da approach is presented in section 3, section 4 shows the results of the experiments, and section 5 outlines our conclusions and proposals for future work. ## background we present how neural machine translation models can translate sentences and how neural language models can generate new words. moreover, we explain some state-of-the-art data augmentation techniques for text. ## automatic generation of a parallel english-russian corpus in this section, we present our approach of generating a synthetic corpus in russian using a pretrained language model specialized in russian called ruroberta-large. this model performed better than the multilingual bert language model in preliminary experiments. the united nations dataset (un) @xcite dataset for english to russian version 1 has been used. we apply a cleaning process which consists in filtering out sentences that • are shorter than 5 words because they are no relevant. after this cleaning process, we save 1 million sentences for da application, 7.9 million sentences for baseline model training, 2000 sentences for validation and 2000 sentences for test. we use a part-of-speech tagger in order to select the type of word that is generated by the pre-trained lm. for that purpose, we have used a model called ru_core_news_lg trained with the spacy @xcite toolkit. this model determines if the type of the selected word (noun, adjective or adverb) to be changed is the same as the generated one, which guaranties a similar level of quality from the original sentence. in addition, we have used the python library pymorphy2, which provides us the gender, person and number of the selected word. it also provides a lot of useful morphological information about the words which may be used for another data augmentation methods, like verbs substitution where the grammatical time is relevant. during the empirical experimentation, we observed that the language model returns a score of the pertinence of generated word to the bidirectional context of the sentence. with this in mind, we established an acceptance threshold for each generated word, which is set to 0.07. this also reduces significantly the selection of useless and not significant words like one character words or punctuation marks. thus, we can make a more precise and accurate word substitution by the generation of the new sentences in russian. in the last step, we generate by high quality pangeanic mt model the english part of the new sentences to obtain parallel corpus. finally, we have created four completely new datasets which have been generated by four different methods: for basic methods (noun, adverbs and adjectives substitution) we select all the words of the selected type from the sentence using the pos-tagger and save the position of that words in the sentence, next we iterate over all the saved words, so in each iteration we took one word, save its gender and number using the pymorphy2 library and change that word for @xmath0. for the mix method we add an additional step before the selection of all the words of a specific type in a sentence. this step consists in selecting which method will be applied for the following sentence. this previous selection is done in a way that the final corpus has similar amount of new texts of each da method used. for all the methods, we stop when there is not more original sentences to be augmented. as we can see in likewise, the adjectives and mix methods have also generated a significant amount of data. as nouns method substitution, adjectives are also a very heterogeneous and permissive when it method new #words new #sentences new #words/sentence comp. days adverbs 275k 327k 1.2 5 nouns 4.8m 922k 5.2 8 adjectives 2.2m 800k 2.7 6 mix 2m 504k 4 11 although the adjective method generated a similar amount of new sentences as nouns, its mean of words per sentences is the smallest of all four methods. due to the number of adjectives in a sentence is significantly less than nouns. by contrast, the mixed method generated fewer sentences than the nouns or adjectives methods, but has a significantly high mean of words per sentence. although this method generates a dataset with high variety because of the combination of the type of words, it takes more time to compute due to the equality of the generation of type of words. ## experiments experiments have been performed in order to study how our da method for a english-russian dataset can improve nmt models. the architecture of the machine translation models used is composed of 6 layers of encoder and 6 layers of decoder with 8 multihead attention units in both of them. this configuration is the same as the standard transformer architecture. on other hand, the maximum length of input and output sentences was established to 400 and batch size was set to 4096 tokens. for baseline model, we trained the model with 7.9 million samples of parallel data in 100000 training steps and for each model trained with extra data, we retrain the baseline model with an extra 50000 train steps using for each retrained model the data reflected in the as we can see in