Do Not Discard – Extracting Useful Fragments from Low-Quality Parallel Data to Improve Machine Translation (2023)

Abstract: When parallel corpora are preprocessed for machine translation (MT) training, a part of the parallel data is commonly discarded and deemed non-parallel due to odd-length ratio, overlapping text in source and target sentences or failing some other form of a semantic equivalency test. For language pairs with limited parallel resources, this can be costly as in such cases modest amounts of acceptable data may be useful to help build MT systems that generate higher quality translations. In this paper, we refine parallel corpora for two language pairs, English–Bengali and English–Icelandic, by extracting sub-sentence fragments from sentence pairs that would otherwise have been discarded, in order to increase recall when compiling training data. We find that by including the fragments, translation quality of NMT systems trained on the data improves significantly when translating from English to Bengali and from English to Icelandic.

## introduction neural machine translation (nmt) usually exhibits good performance when trained on a large amount of good-quality bilingual sentence pairs. however, developing a good-quality nmt system for language pairs with limited resources is a challenging task. when compiling a parallel corpus, and during preprocessing for training, a significant amount of sentence pairs are commonly discarded before the data can be used to train the translation model. that may not be much of a problem for high-resource language pairs, where the training data contains sufficiently large number of sentence pairs even after discarding many of them, but language pairs with limited resources can be negatively impacted if the filtering is inaccurate, as less training data may limit the quality of the translation model. the first stage of nmt training involves preprocessing the training data in which the text pairs go through several steps such as tokenising, filtering and byte-pair encoding @xcite . in the filtering step, all the text pairs with unusual source-target sentence-length ratio, extremely long sentences, absence of text for either one of the languages, or other anomalies are discarded. these can be a substantial percentage of available parallel pairs, and for language pairs that have limited resources, a number which could affect performance noticeably. although the sentence pairs are discarded due to irregularities that can be detrimental for mt systems, they often contain a considerable amount of semantically similar segment pairs at the phrase, chunk or sub-sentence level. for example, if a source-language sentence contains 50 words and its target counterpart contains 10 words, they are likely to be discarded due to odd sentence-length ratio. however, they may contain similar information and some equivalent phrases or segments. this leads us to the two research questions we seek to answer in this paper: in order to seek answers to these questions, we conduct two experiments. in the first one, described in section 4, we work with english-bengali sentence pairs from the samanantar parallel corpus @xcite . we score the pairs and select a subset of the highest scoring pairs for training. the discarded sentences are then divided into subsentences and treated as a comparable corpus, which we mine for sentence pairs acceptable for training. in our second experiment, described in section 5, we work with a subcorpus of the english-icelandic parallel corpus parice @xcite , which is composed of a collection of parallel texts in a number of domains. the subcorpus we work with contains regulations and other documents published in relations with the eea agreement. we collect all sentences that did not obtain alignments during the alignment process, as well as sentence pairs filtered out due to insufficient quality. we treat these discarded sentences as we treated the english-bengali data, i.e. divide the sentences into subsentences and mine them for sentence pairs potentially useful for mt training. finally, we train multiple nmt models to assess the feasibility of the approach. our evaluation shows that mt quality can be increased by extracting useful chunks at a sub-sentence level from data that would usually be discarded. ## related work a significant amount of research has been carried out in the area of exploiting comparable corpora for mt. @xcite extracted parallel sentences from wikipedia documents by translating documents in persian into english, and also in the reverse direction, to extract semantically equivalent sentence pairs. @xcite employed three different measures to identify and score parallel sentences from comparable corpora for english-icelandic: crosslingual information retrieval (clir) based approach @xcite , labse, and wascore, a word alignment based scoring mechanism introduced in the paper. @xcite extracted parallel sentences from the web by using: recent work on developing english-bengali mt systems include @xcite , who proposed approaches for translating assertive, interrogative and imperative english sentences into bengali by analysing their sentence patterns and using different bengali grammatical rules. @xcite developed an english-bengali nmt system for the aviation domain trained on a unique english-bengali parallel corpus in this domain. @xcite built a translation system using an encoder-decoder recurrent neural network with the help of knowledge-based context vectors for mapping english and bengali words. until recently, work on english-icelandic mt was limited to an apertium @xcite based model @xcite . the parice corpus was published in 2018, spurring work using statistical and neural methods for english-icelandic mt. @xcite presented the first published work on phrase-based statistical mt (pbsmt) and nmt for icelandic and, in 2021, english-icelandic was one of the language pairs in the shared news translation task at wmt @xcite . ## methodology and experiments in this work, we reexamine discarded parallel training data by segmenting it and extracting semantically equivalent bilingual segments. we then utilise parallel segments extracted from the discarded data as additional parallel training data if it can be deduced from our methods that the segments will be useful for mt training. we compare the quality of the translation output to baseline models. in the case of english-bengali, the comparison is made to a model trained on the full samanantar corpus and to the state-of-the-art indictrans model @xcite , and in the case of english-icelandic, to a model trained on the aligned and filtered corpus, without the sentence pairs mined from discarded data. ## refining an english-bengali corpus we begin by calculating similarity scores for each of the 8.52m english-bengali sentence pairs in the samanantar corpus. we use laser @xcite , labse, and wascore @xcite for scoring the sentence pairs. laser uses a pre-trained bilstm encoder trained on data in 93 languages to generate scores for sentence pairs. labse uses dual encoder models, with the encoding architecture following the bert base model, and additive margin softmax which creates a large margin around positive pairs. wascore is word alignment based and uses combalign @xcite , which again employs multiple word aligners to arrive at accurate word alignments. in order to remove sentences most likely to be deficient, we treat this as a candidate list extracted from comparable corpora, following the methodology described in @xcite , using a logistic regression classifier that considers all three scores to decide which sentence pairs to filter out. we then order the remaining sentence pairs based on labse similarity score and create differently sized sets of parallel sentence pairs, with one set containing the 500 thousand highest scoring pairs (s 7 ), another containing the 1 million highest scoring pairs (s 6 ), and so on. ## data discarded during the compilation of an english-icelandic parallel corpus we process the discarded english-icelandic data in a slightly different manner. in our english-bengali experiment we only considered subsentences within previously aligned pairs, and compared all the different concatenations of english chunks to the different combinations of bengali chunks to find if we could raise the semantic score for the pair by removing parts from either or both sentences. for english-icelandic, we instead consider our discarded sentences to be a comparable corpus and mine for sentence pairs from the pool of all segments in both languages. we use the approach of @xcite using clir to create a candidate list and a logistic regression classifier to select the best sentence pairs from the list. we start by deduplicating the discarded sentences and removing sentences that have less than three tokens that only contain alphabetical characters. this lowers the number of sentences we have to work with to 234, 835 english sentences and 242, 456 icelandic sentences, as shown in ## conclusions and future work in this paper, we set out to answer whether deficient sentence pairs in a parallel corpus could be identified and refined and whether data commonly discarded when compiling parallel corpora or training nmt systems could be mined for parallel sentence pairs, that are still beneficial for training. we conducted two experiments to answer these questions. first, we tried re-evaluating sentence pairs in an english-bengali parallel corpus in an attempt to remove extraneous data from partially parallel pairs. by partially parallel pairs we mean that a part of either sentence can align perfectly with either the whole or a part of the other sentence. second, we collected all sentences discarded when an english-icelandic parallel corpus was compiled, segmented them to create multiple sub-sentential variants, and treated as comparable corpora for mining parallel pairs by using our approaches, the quality of our training corpus improved, leading to significantly better quality mt models, as measured by bleu, when translating from english and into either bengali or icelandic. however, when translating into english we did not see this effect as clearly. in the english-bengali experiment the data selection aimed at increasing @xmath0, which may explain the effect we see there, and for @xmath1, but not significantly when the sentence pairs mined from discarded data was added. in that case the low improvement is most likely explained be the small size of the additional data, which only increased the size of the training data by 4%. in future work we want to experiment with other methods of segmenting sentences, such as by using constituency parsing. the approach we used in this paper for segmenting was simple and easy to implement. more sophisticated segmentation may allow for more precise recombinations of sentence parts, for example by skipping parenthetical clauses or other insertions which may not be represented in both sentences. we also want to investigate whether our approaches also show positive results for other language pairs our experiments indicate that there is a potential in taking a second look at data that would usually be discarded, as well as in refining partially aligned sentence pairs. we showed that parallel sub-sentences are useful to acquire translation knowledge and extracting them can lead to significant improvement in performance, even using simple approaches. the methodology can thus have an impact on training future mt systems. finally, the training time for the different models, shown in tables 1 and 3 , indicates that smaller and more accurate training corpora have the added benefit of helping with faster convergence. in our case, training time is reduced by 65% from using the whole dataset to using our selected subset for training @xmath2. the model also comes close to reaching the quality of the much larger indictrans model. this can translate into less need for storage and less resources at training and inference time, which is in line with a call to greener and more sustainable models of ai which consume less electricity, output fewer emissions, and perform on the whole as well as larger models, see e.g. @xcite and @xcite .