P ara C rawl: Web-Scale Acquisition of Parallel Corpora (2020)

Abstract: We report on methods to create the largest publicly available parallel corpora by crawling the web, using open source software. We empirically compare alternative methods and publish benchmark data sets for sentence alignment and sentence pair filtering. We also describe the parallel corpora released and evaluate their quality and their usefulness to create machine translation systems.

## introduction parallel corpora are essential for building highquality machine translation systems and have found uses in many other natural language applications, such as learning paraphrases @xcite or cross-lingual projection of language tools @xcite . we report on work to create the largest publicly available parallel corpora by crawling hundreds of thousands of web sites, using open source tools. the processing pipeline consists of the steps: crawling, text extraction, document alignment, sentence alignment, and sentence pair filtering. we describe these steps in detail in sections 4-8. for some of these steps we evaluate several methods empirically in terms of their impact on machine translation quality. we provide the data resources used in these evaluations as benchmarks for future research. as part of these effort, several open source components have been developed. these are integrated into the open-source tool bitextor, foot_0 a highly modular pipeline that allows harvesting parallel corpora from multilingual websites or from preexisting or historical web crawls such as the one available as part of the internet archive. 2 the execution of the pipeline has focused on official european union languages, but also targeted russian, sinhala, nepali, tagalog, swahili, and somali. we show that the obtained parallel corpora improve state-of-the-art results on common benchmarks, such as the wmt shared task on news translation. ## related work while the idea of mining the web for parallel data has been already pursued in the 20th century @xcite , the most serious efforts have been limited to large companies such as google @xcite and microsoft @xcite , or targeted efforts on specific domains such as the canadian hansards and europarl @xcite . the book bitext alignment @xcite describes some of the challenges in greater detail. ## identifying multi-lingual web sites since the start of the collection effort in 2015, we identified potential web sites to crawl in various ways, but mainly by exploiting statistics from commoncrawl. by splitting this large collection of crawled web pages by web domain and running text extraction and language identification @xcite , we can extract statistics on what language content exists on each of them. web domains with sufficient content in a targeted language and english are selected for crawling. the thresholds of what constitutes sufficient content varied depending on language. typically, we require minimum amounts of content in the targeted language and english (measured in bytes of text), and consider the ratio between the two. for instance, we identified 19,616 web domains with at least 100kb of content in german and english (max ratio 10), but only 438 web domains with at least 20kb of content in maltese and english (max ratio 10). it is worth noting that by targeted crawling of web sites we are able to collect many more web pages than present in commoncrawl. in an exploratory study, only 5% of a collection of web pages with useful content were found in common-crawl. this may have improved with recent more extensive crawls by commoncrawl but there is still a strong argument for targeted crawling. ## crawling crawling is the initial step of the pipeline. it entails downloading documents from a number of websites and looking for any documents that contain text. these documents are stored as single or multi-domain web archive (warc) files. warc is an archiving format for crawled data originally proposed by the internet archive and developed by a consortium of libraries and archives into the iso 28500:2009 standard (iso, 2009). it consists of a list of gzip-compressed records, each comprising a header with metadata and a crawled document. four different crawling tools are currently supported in bitextor: httrack foot_3 well-known multi-platform tool for crawling. it has been for long time in bitextor, even though it is now deprecated as the support for the tool is discontinued. heritrix 6 internet archive's web crawler; it is fully compatible with warc format and supports a variety of options that make it one of the most suitable options for large scale data crawling. creepy 7 python library with basic resources for crawling. a crawler has been implemented on top of it, and is currently experimental. wget one of the most popular tools for retrieving files through http and https in unix systems. it is fully compatible with warc format. most of our crawling in paracrawl has been done using httrack. to deal with the i/ointensive process of writing small files with high frequency, data is first stored on local ssd drives and then transferred to a network file system for subsequent processing. ## text extraction after crawling, all documents are pre-processed to extract and normalize the text and identify their language. the resulting cleaned and sorted text is the input for the subsequent steps of document and segment alignment (see sections 6 and 7). conversion to html warc files contain one web-crawled document per record. the documents can be in a variety of formats that contain text: plain text, html, open document format 8 (".odt"), office open xml 9 (".docx") or pdf files containing text. with the exception of the small number of documents that are already in plain text format, the bitextor-warc2htmlwarc.py module converts any of these formats to html (see fig. 1 ) and produces warc files containing only html or plain text documents. text extraction from html given warc files containing html, we extract the text content. we preserve sentence breaks indicated by html tags such @xmath0, but remove formatting tags such @xmath1. language identification with cld2 and text extraction are currently performed by python module bitextor-warc2preprocess.py; as text extraction is a rather intensive operation, an alternative workflow uses an experimental module written in the go language, giawarc. using bilingual lexica the traditional workflow in bitextor until version 5 used bilingual lexica. module bitextor-buildidx.py builds indexes of documents containing, for each word in the lexicon for each language, the documents containing it. then bitextor-idx2ridx uses the bilingual lexica to translate these words and build reverse indexes where each document is paired to a list of documents and bag-of-words-based overlap scores in the other language. a series of modules (bitextor-urlscomparison.py, bitextor-urlsetoverlap.py, bitextorimagestooverlap.py, etc.), compute a series of features for each language direction based on mutual linking and the comparison of document urls, the set of outgoing urls, html structure and image content; these features are integrated by bitextor-rank.py into two new reverse-index file with new scores, which are used to obtain the final document alignment. using machine translation this workflow uses machine translation to decide whether two documents have to be aligned, and is the one that has been used for the parallel data releases of the project @xcite . after extract-lett.py extracts plain-text documents in each language, a machine translation system translates each document from language a to b. we then generate a (sparse) matrix of tf-idf scores between machine translated versions of documents in language a and documents in language b. these scores are used by compute_matches.py to compute a list of document pairs (score, source url, target url). document pairs are stored in a file in which each line contains the urls of both documents and their plain-text content encoded in base64. ## sentence alignment during the paracrawl project, we made use of a few sentence alignment tools. in this paper, we compare their performance on five language pairs. the sentence aligners are: hunalign @xcite is a widely used tool that relies on a bilingual dictionary that we generated from the europarl corpus or other available parallel corpora. bleualign @xcite aligns an english translation of the foreign sentences and the english sentences based on their similarity, as measured by a variant of the bleu score. we implemented a faster version of bleualign in c++. vecalign @xcite is a new sentence aligner based on sentence embeddings, using an efficient coarse-to-fine algorithm with linear run time. we used pre-trained laser embeddings 10 which cover all the languages of paracrawl, except for irish. we compared the quality of the sentence pairs extracted from document pairs for these tools. to our knowledge, this is the first evaluation of sentence aligners on large-scale real-world webcrawled data. we selected five languages, ranging from low resource (maltese) over mid-resource (estonian, hungarian) to high-resource (czech, german). we selected a subset of web domains, for details see the data is provided as document pairs from the usual upstream paracrawl processing. the text of web pages needs to be further split into sentences, and then aligned using the different sentence aligners. the resulting sentence pairs are deduplicated are assessed for quality using bicleaner (more on sentence pair filtering in the next section). since different sentence aligners generate different amounts of data (for instance, bleualign filters quite aggressively for noise), we selected differently sized subsets of the data for evaluation by selecting the best sentence pairs according to bicleaner quality scores. we built neural machine translation models on these subsets using fairseq and evaluated them on test sets drawn from the wmt news translation task (newstest2018 for german, czech, estonian; newstest2009 for hungarian) and the eu bookshop foot_5 corpus (maltese). see we published the document pairs to be aligned, as well as the testing environment foot_6 to promote the evaluation of novel sentence alignment methods. ## sentence pair filtering our processing pipeline is aimed at high recall at the cost of precision, thus creating large but very noisy corpora. so, as a last processing step, we aim to filter out sentence pairs that are not useful as training data for machine translation or any other purpose. this is especially important since training on noisy corpora is a challenge for neural machine translation which motivated the organization of two shared tasks in 2018 and 2019, on the high resource language german-english and the low resource languages sinhala and nepali, respectively. here, we extend this evaluation to european languages with medium sized resources. building on the data sets generated by the sentence alignment evaluation of the previous section, we compared three sentence pair filtering methods used in the paracrawl effort: zipporah @xcite , bicleaner @xcite , and laser @xcite . we carried out the evaluation (see laser model has not been trained). moreover, in almost all settings, we achieve better results with bicleaner than zipporah. ## released corpora overall, the paracrawl corpus release v5.0 contains a total of 223 million filtered 13 , unique sentence pairs from around 150k website domains and across 23 eu languages with english (see our main motivation for creating the paracrawl corpus is to improve the quality of machine translation systems. to test this, we trained neural machine translation models where we added the corpus to existing data sets for language pairs that were tackled in the shared task on news translation at the conference on machine translation (wmt) -which we consider a strong baseline. 13 sentence pairs with a bicleaner score of less than 0.7 were discarded, but remain in the raw release. 14 sacrebleu signatures: bleu+case.mixed+lang.*-*+numrefs.1+smooth.exp+ tok.13a+version.1.4.2 pair bleu 14 bleu wmt wmt+paracrawl-5 en-cs 19.0 (52m) 19.8 (52m+5.3m) cs-en 25.0 (52m) 25.7 (52m+5.3m) en-de 26.2 (5.8m) 27.7 (5.8m+37m) de-en 31.2 (5.8m) 34.0 (5.8m+37m) en-fi 19.9 (2.6m) 23.3 (2.6m+3.0m) fi-en 24.2 (2.6m) 29.9 (2.6m+3.0m) en-lv 12.8 (4.5m) 16.2 (4.5m+1.0m) lv-en 16.2 (4.5m) 20.2 (4.5m+1.0m) en-ro 26.5 (0.6m) 28.6 (0.6m+2.8m) ro-en 30.2 (0.6m) 35.7 (0.6m+2.8m) we trained transformer-base models with marian using sentencepiece. see ## computational costs concerns several of the steps involved in producing and evaluating the paracrawl corpora are computationally expensive. even as some of the steps are embarrassingly parallel and amenable processing in a high-performance computing setting, even pre-processing of 100tb of source data to produce candidate documents consumes on the order of 50,000 cpu-hours equivalent to an estimated foot_7 720kwh of power. training of a neural network model for translating one of the more resource-rich languages such as german may take a week on a dozen gpus again consuming about 750kwh. translating 500 million german sentences to english for evaluation consumed roughly 7mwh. in practice, these computations are not simply performed once, they are performed many times as parameters are changed and different strategies tried. this energy cost is significant. the typical domestic consumption values published by ofgem 16 , the uk energy regulator, say that a highconsuming household with electric heating is expected to consume 7.1mwh/year. does an increase of one or two bleu points justify this cost? for paracrawl, we argue that yes, it does, because we are producing an enabling data set whose cost will, we hope, be amortised across many future experiments. but there is a more general point to be made here: it is not currently the practice in the machine translation community to publish figures about the cost involved in achieving an increase in performance as measured with the standard metrics. it is not straightforward to evaluate when or if we, as a community, have reached a point of diminishing returns where small changes to a family of methods consume an ever-increasing amount of resources yielding only marginal improvements. we therefore suggest adopting a practice of disclosing energy use for experiments in machine translation alongside bleu scores to make the 16 https://www.ofgem.gov.uk/electricity/ retail-market/monitoring-data-and-statistics/ typical-domestic-consumption-values cost-benefit trade-off explicit. ## conclusions we released the largest publicly available parallel corpora for many language pairs and demonstrated their benefit to train machine translation systems. going beyond providing data, the goals of this project include the creation of publicly available infrastructure to explore new research directions on parallel corpus mining by releasing open source code for the entire pipeline and public benchmarks for individual processing steps. each of the processing steps we describe here still have great potential for improvement, and we hope that our work contributes to the development of novel methods both in terms of better processing of raw parallel data sources, but also increasing the robustness of neural machine translation training when faced with noisy data. we are especially interested in further extending this work into low resource languages where resources tend to be noisier and underlying models to support data mining less reliable.