A Sentence Alignment Approach to Document Alignment and Multi-faceted Filtering for Curating Parallel Sentence Pairs from Web-crawled Data (2023)

Abstract: This paper describes the AST submission to the WMT23 Shared Task on Parallel Data Curation. We experiment with two approaches for curating data from the provided web-scraped texts. We use sentence alignment to identify document alignments in the data and extract parallel sentence pairs from the aligned documents. All other sentences, not aligned in that step, are paired based on cosine similarity before we apply various different filters. For filtering, we use language detection, fluency classification, word alignments, cosine distance as calculated by multilingual sentence embedding models, and Bicleaner AI. Our best model outperforms the baseline by 1.9 BLEU points on average over the four provided evaluation sets.

## introduction the aim of the shared task on parallel data curation at the eighth conference on machine translation (wmt23) is to evaluate parallel data curation methods @xcite . the goal is to find the best machine translation (mt) training data within a provided pile of web-crawled data. the language pair chosen for the task is estonian-lithuanian. the provided data is extracted from a single snapshot of commoncrawl, foot_0 which according to the task organizers should contain enough training data to train a reasonable @xmath0, even with limited compute. as well as providing the data, the organizers release pre-computed intermediate steps from a baseline, so participants can choose whether to focus on one or more aspects of the task. we describe the provided data and the baseline in section 3. in our submission we experiment on two aspects of parallel data curation. initially we try to to identify parallel documents in the two languages. we then align sentences in the documents using our own sentence alignment tool, sentalign foot_1 @xcite , and train an mt system on the resulting sentence pairs. sentalign is a sentence aligner that uses labse @xcite to score all possible alignment combinations for a document pair, selects the highest scoring one, but then re-evaluates the results by looking at each individual alignment and their closest neighbours to see if localized scores can be raised. this is to counteract an effect of dynamic programming with cosine similarity, which often favours many-to-many alignments over 1to-1 alignments (see e.g. @xcite . @xcite show that this approach outperforms other aligners on two evaluation sets, as well as on a downstream task. the other aligners include aligners such as the length based gale-church @xcite , mtbased bleualign @xcite and vecalign @xcite which is the most similar to simalign, using laser embeddings @xcite to calculate cosine similarity of alignment candidates, and a recursive approximation to reduce the search space, as opposed to evaluating all possibilities as sentalign does. we describe our approach to document alignment in section 4.1. subsequently, we try to identify parallel sentence pairs in all the other provided data and run a number of different filters to remove sentence pair candidates that we deem likely to be detrimental or useless for mt training. our filtering approaches are described in section 4.2 2 related work @xcite show that incorrect translations, untranslated target text, misalignments, and other noisy segments in a parallel corpus have a detrimental effect on the output quality of neural machine translation (nmt) systems trained on that corpus, as measured by using bleu @xcite . they specify five general classes of noise commonly found in a german-english version of the paracrawl corpus: misaligned sentences, disfluent text, wrong language, short segments, and untranslated sentences. they find this distinction to be useful to give a general idea of which types of errors seem to have the least impact on mt systems (short segments, untranslated source sentences and wrong source language) and which have the most dramatic effect (untranslated target sentence). misalignments, misordered words, and wrong language, in source or target texts, are also shown to be harmful, but not as harmful. the conference on machine translation, wmt, hosted three annual shared tasks on parallel corpus filtering @xcite @xcite , focusing on filtering noisy web-crawled corpora. submitted systems include the ones by @xcite and @xcite , who introduce approaches based on cross-lingual sentence embeddings trained from parallel sentences. both papers use cosine similarity and consider the margin between a given sentence pair and its closest candidates to normalize the similarity scores. bicleaner @xcite uses a set of handcrafted rules to detect flawed sentences and then proceeds to use a random forest classifier based on lexical translations and several shallow features such as respective length, matching numbers and punctuation. it also scores sentences based on fluency using 5-gram language models. the tool ranked highly on the first two parallel corpus filtering tasks at wmt. bicleaner ai @xcite is a fork of bicleaner using a neural classifier. it has been shown to give significant improvements in translation quality as measured by bleu when used for filtering training data for multiple language pairs, as compared to the previous version of the tool. in contrast to tools that apply a single method for parallel corpus filtering, @xcite implement multiple different filters in the opusfilter toolbox. these include heuristic based filters, language identification, character-based language models and word alignment tools. the toolbox can furthermore be extended with custom filters. @xcite revisit the noise classes specified by @xcite to investigate how accurately two strong filtering approaches, cross entropy @xcite and laser @xcite ) can filter out different noise classes. they find that for a common language pair, @xmath1, most types of noise can be detected with over 90% accuracy, although misalignments and poor synthetic translation can only be detected with an accuracy of less than 70%. for a less common language pair, khmer-english, the performance of the filtering system degraded significantly and the accuracy of identifying noise was lowered by 8-19%, depending on the type of noise, indicating that results can vary dramatically depending on the languages. ## data and baseline the provided data is retrieved from the 2023-06 snapshot of common crawl. the organizers have extracted plain text from html documents and used the fasttext @xcite language identification model to remove documents not classified as estonian or lithuanian by the model, based on the first 2,000 characters of the document. unsafe and offensive content has been removed. documents from host names in the following lists in the blocklist project foot_2 where removed: abuse, basic, crypto, drugs, fraud, gambling, malware, phishing, piracy, porn, ransomware, redirect, scam, torrent. documents were split into paragraphs based on line breaks, and then into sentences using mediacloud sentence splitter. 4 each sentence was assigned a unique sentence id and classified using the fasttext language identification model. the data is provided in tsv format. the task organizers provide laser2 sentence embeddings @xcite for all sentences in the correct language, as classified by the fasttext model. they index the embeddings and query each index to retrieve the top-8 results for each sentence, based on cosine similarity. we use these results as a starting point for our filtering approaches, as described in section 4.2. the baseline simply takes top-1, i.e. the highest scoring sentence pair for each sentence, provided the score exceeds a threshold of 0.9. this results in a set of 2,654,090 parallel pairs for training a baseline model. a script for training the baseline model using sockeye @xcite was provided. we use the script and sockeye for training all models on a single nvidia geforce rtx 3090 gpu card. ## system architecture in this section we describe our approaches to the parallel data curation problem. first, we try to identify parallel documents in the two languages and align them on the sentence level. second, we use the provided sentence pair candidates, eight for each sentence in each language, and filter using a number of different approaches to remove possibly detrimental pairs from our training set. the sentence pairs from the aligned documents and the filtered sentence pairs are combined to compile our final dataset. ## conclusions and future work our alignment and filtering approach resulted in an improvement over the baseline in terms of bleu score for the four evaluation sets. we identified 4,372 document pairs in the provided dataset, which we aligned on sentence level and used the resulting data set for training. we then combined a number of filtering approaches for determining which sentence pair candidates from a provided candidate list would be likely to be useful, these included an ensemble approach for language detection, using three different tools, a gpt-2 based classifier to determine whether sentences are fluent or disfluent, a logistic regression classifier based on word alignment scores and two sentence embedding based scores, labse and laser2, and finally a bicleaner ai classifier. working in a similar vein, many different paths could be taken for future work on this problem. @xcite show that it can be beneficial to inspect how different filters suit a given translation direction. a filtering method giving an optimal results for lang @xmath2. in this work we did not try to evaluate the filtering approaches with regards to translation direction. for translating only from estonian to lithuanian, removing incoherent and ungrammatical estonian sentences may not be as important as removing such sentences in lithuanian, as it is more important that the target language data contains coherent and well written examples. different levels of filtering for the different languages could thus be useful in order to add more useful examples. the aim of our filters is to remove sentences likely to be detrimental in mt training. while we do know about some of the qualities that reduce translation quality, as discussed in section 2, more fine-grained classifications may be useful. for example, we could designate different levels of misalignments, which include partial alignments defined as sentence pairs where a part of one or both sentences is not represented in the other sentence. @xcite argue that extracting mutual translations from such pairs, while discarding the extraneous data, may improve the quality of mt models trained on the data, and show that for one parallel corpus. if that holds in general, it could be useful when working with web-scraped data to identify when misalignments become detrimental and when they can be useful, as well as helping to come up with effective ways to refine such sentence pairs. finally, it should be noted that the training times for these models varied considerably. while our best model reached the optimal checkpoint in approximately 20 hours and the second best in 12 hours, the models trained on the larger datasets listed in