Filtering of Noisy Web-Crawled Parallel Corpus: the J apanese- B ulgarian Language Pair (2022)

Abstract: One of the main challenges within the rapidly developing field of neural machine translation is its application to low-resource languages. Recent attempts to provide large parallel corpora in rare language pairs include the generation of web-crawled corpora, which may be vast but are, unfortunately, excessively noisy. The corpus utilised to train machine translation models in the study is CCMatrix, provided by OPUS. Firstly, the corpus is cleaned based on a number of heuristic rules. Then, parts of it are selected in three discrete ways: at random, based on the “margin distance” metric that is native to the CCMatrix dataset, and based on scores derived through the application of a state-of-the-art classifier model (Acarcicek et al., 2020) utilised in a thematic WMT shared task. The performance of the issuing models is evaluated and compared. The classifier-based model does not reach high performance as compared with its margin-based counterpart, opening a discussion of ways for further improvement. Still, BLEU scores surpass those of Acarcicek et al.’s (2020) paper by over 15 points.

## introduction in recent years, web-crawled corpora have come as an attempt to tackle the problem of limited parallel corpora, notably when it comes to machine translation involving low-resource language pairs. they are the product of unsupervised covering of portions of the web based on a widely used metric, such as the cosine distance between sentence embeddings, and they tend to be produced in excess, leading to problems like redundancy and data of low quality @xcite . large web-crawled corpora are often associated with a lack of documentation and require further work before they can be used within the field of machine translation @xcite . in their study, khayrallah and koehn (2018) discuss the types of noise that tend to occur in web-crawled corpora, as well as their effect on potential machine translation systems. notably, neural machine translation is affected by such noise to a considerably greater extent as compared with its statistical counterpart, derived bleu scores decreasing dramatically at its experimental introduction @xcite . motivated by a desire to mitigate the described problems, associated with similarly derived parallel corpora, wmt has organised three shared tasks in 2018-2020, addressing their cleaning, the last two of which have specifically centred on lowresource language scenarios. several excellent state-of-the-art models have been produced to handle the task. in this paper, a representative model @xcite is selected and applied to a particular, extremely under-resourced language pair: japanese-bulgarian. @xcite s model uses a classifier on top of roberta in order to score sentence pairs according to the level of certainty that they are mutual translations. the corpus discussed in this study is ccmatrix, the largest parallel dataset that is currently available in the addressed language pair. it is provided by the opus collection @xcite and contains over four million multidomain web-crawled sentences, derived based on "margin distance." the last is an improved implementation of cosine distance that considers the ratio of the cosine distance between two candidate sentences' embeddings as compared with the average cosine distance that a sentence has with its nearest neighbours @xcite . following preprocessing based on heuristic rules that keep in mind the characteristics of the two languages in question, discrete subcorpora of three sizes (200k, 500k, and 1m) are selected based on margin distance and on the classifierderived scores. they are compared to randomly selected subcorpora of the same size (see figure 1 ). the margin-distance-based models show significantly improved performance. conversely, the performance of the classifier models is largely non-optimal, showing the need for improvement of the selection techniques, such as through higher focus on the morphological and semantic specificities of the two languages. importantly, the best derived model outperforms the one offered by @xcite by over 15 bleu points. ## noise in the ccmatrix corpus (japanese-bulgarian) akin to an experiment in @xcite study, a random 200-sentence sample from the described corpus is examined in an attempt to identify the nature of the different types of noise present. 1 the examined sentences demonstrate a large variety of domains and registers and feature a wide range of vocabulary, notably including a number of proper nouns. the main types of noise discovered include: noncorresponding numbers and dates, inappropriate punctuation, wrong use of abbreviations, presence of foreign languages, and machine-translated text. as numbers and dates widely mismatch between the two languages within a sentence pair, they are regarded as noise. the next largest source of noise in the bulgarian sentences comes in the face of problems with punctuation (for instance, a frequent use of "...") and capitalisation. what follows are instances of "non-standard language," including a large number of sentence fragments (for example, "ако по някаква причина се преместят в друго училище,"). however, if one disregards the lack of final punctuation within these fragments, they read smoothly and match unproblematically between the two languages. in fact, the mandate for a sentence to contain a main verb, largely influenced by english grammar, is not intrinsic to either the bulgarian or the japanese language. while the cambridge dictionary states that a sentence is "a group of words, usually containing a verb, that expresses a thought" ("sentence", 2022; emphasis added), bulgarian ("изречение", 2022) and japanese ("文", 2022) counterparts do not make a reference to the concept of "verb" in their definitions of a sentence. an example is the sentence "как мога да защитавам моята pc?", which contains a gender mismatch and an unnatural english abbreviation. other types of noise include abbreviations in both the cyrillic and latin alphabets, excessively large sentences and sentences written in (or partly in) a foreign language, predominantly english. foreign language within a sentence ranges between a single word or phrase that can safely be regarded as a proper noun (e.g. "google assistant") and a full sentence written in a foreign language with a few seemingly mistakenly inserted bulgarian words. similar patterns are observable when it comes to the noise in japanese sentences: the use of numbers and dates, followed by abbreviations in the latin alphabet and wrong punctuation. an additional problem related to punctuation is the fact that it differs significantly between the two languages; as a result, for instance, a bulgarian "..." may be rendered as either "..." or "--" in the parallel japanese sentence. other examples of "non-standard language" come in the face of language attributable to "texting" (e.g. a "laughter" kanji in the end of a sentence) and supplementary hiragana renditions of kanji and katakana scripts, placed in brackets. some of the observed types of noise can be addressed directly during the preprocessing step (see section 4.1). such an issue as machinetranslated language, however, is difficult to tackle using heuristic rules. ## conclusion and future work although the exposed study exhibits high similarity to wmt's corpus filtering shared tasks, several crucial elements that distinguish it should be made note of. firstly, the english language is not featured in either translation direction, and the examined language pair is not selected merely quantitatively based on its associated resources but is closely associated with the study and its goals. as a result, preprocessing is key within the filtering process. part of the corpus's preprocessing is language-specific, and a suggested direction for future improvement of the utilised classifier model would involve further application of the two languages' morphological features (such as the use of an alternative, more morphologically-aware fuzzy search algorithm and the inclusion of universal dependencies annotations and relations). additionally, in this study a customised filtering model benefits from a comparison with one that uses margin scores, thus allowing for specific conclusions to be made, such as the effect of domain-specific data on the machine translation models. in order for this narrowing of the corpus to be avoided, clean multi-domain data could be attained if a manually cleaned portion of ccmatrix is used in training the classifier model. also, as performance increases steadily with subcorpora sizes, even larger models should be experimented with. importantly, the current work does not claim to propose a high quality translation system in the low-resource japanese-bulgarian language pair. rather, it provides methods for improving the quality of noisy parallel sentences and for the selection of specific portions of higher-quality data. the study may be used as the starting point for further work toward an improved translation model in the described language pair as well as a general frame of reference in terms of a filtering pipeline that can be adapted to other corpora and language pairs. proceedings of clib 2022