CCA ligned: A Massive Collection of Cross-Lingual Web-Document Pairs (2020)

Abstract: Cross-lingual document alignment aims to identify pairs of documents in two distinct languages that are of comparable content or translations of each other. In this paper, we exploit the signals embedded in URLs to label web documents at scale with an average precision of 94.5% across different language pairs. We mine sixty-eight snapshots of the Common Crawl corpus and identify web document pairs that are translations of each other. We release a new web dataset consisting of over 392 million URL pairs from Common Crawl covering documents in 8144 language pairs of which 137 pairs include English. In addition to curating this massive dataset, we introduce baseline methods that leverage cross-lingual representations to identify aligned documents based on their textual content. Finally, we demonstrate the value of this parallel documents dataset through a downstream task of mining parallel sentences and measuring the quality of machine translations from models trained on this mined data. Our objective in releasing this dataset is to foster new research in cross-lingual NLP across a variety of low, medium, and high-resource languages.

## introduction cross-lingual document alignment aims to pair documents such that they are translations or near translations of each other. there are a variety of tasks in natural language processing that consume such parallel cross-lingual data. traditionally, machine translation approaches have leveraged parallel sentences as training data for use with sequence-to-sequence models. other tasks include cross-lingual information retrieval and cross-lingual document classification. additionally, cross-lingual data facilitates training crosslingual representations such as multilingual bert @xcite and xlm @xcite which are used in many nlp tasks. the availability of high-quality datasets is necessary to both train and evaluate models across these many tasks. while it is possible to manually label aligned documents across languages, the process is costly and time consuming due to the quadratic search space for document pairs. additionally, for low-resource languages, identifying these crosslingual document pairs is more difficult due to their relative scarcity. furthermore, lack of access to qualified human annotators makes it necessary to have additional quality control in low-resource scenarios @xcite . in this paper, we investigate whether we can rely on weak supervision to generate labels for document pairs. in particular, we focus on the weak signals embedded in the urls of web documents, that can be used to identify the different translations of a single document across many languages. we preprocess, filter, and apply a set of high-precision hand-crafted rules to automatically sift through a massive collection of 169 billion web documents and identify over a 392 million cross-lingual parallel documents in 8144 language pairs. of these aligned documents, 292 million are non-english document pairs, and 100 million include english. we evaluate the quality of our automatic-annotation setup by running a manual human evaluation on a diverse sample of positively-labeled documents across six language pairs. we also introduce a simple baseline that effectively aligns cross-lingual document pairs using solely textual content and in the presence of detractor documents which may not have any parallel counterpart. finally, we demonstrate the utility of our parallel corpus by demonstrating how these parallel documents can be mined for training data for a downstream machine translation task. we release the dataset consisting of pairs of translated documents represented by urls extracted from a massive collection web crawls. we hope that the size, diversity, and quality of this dataset spurs its use not only as a benchmark for document alignment, but also as a parallel corpus for mining parallel sentences and as supervision for a variety of cross-lingual tasks. ## related works the concept of crawling and mining the web to identify sources of parallel data has been previously explored @xcite . a large body of this work has focused on identifying parallel text from multilingual data obtained from a single source: for example the united nations general assembly resolutions @xcite or european parliament parallel corpus @xcite . these parallel corpora were curated from specific, homogeneous sources by examining the content and deriving domainspecific rules for aligning documents. other approaches have identified parallel documents in unstructured web corpora by relying on metadata @xcite . some of these methods have focused on publication date and other temporal heuristics to aid in identifying parallel documents @xcite @xcite @xcite . however, temporal features can be sparse, noisy, and unreliable. a different class of alignment methods rely on document structure @xcite . in the wmt-2016 bilingual document alignment shared task @xcite ), many techniques applied retrieval and matching on translated 5-grams (dara and lin, 2016) to query, retrieve, and align documents. similar methods for generating candidates by retrieving matches based on the least frequent bi-lingual 5grams have been proposed @xcite with the insight that rare snippets are more informative. both of these candidates rely on high-quality translation systems to translate either the source or the target. such models may not exist, especially for low-resource language directions. the application of alignment to a variety of languages was not explored in @xcite which only considered english to french document alignment -a high-resource direction. recently, the use of neural embedding methods has been explored for bilingual alignment of text at the sentence and document level. @xcite propose using hierarchical document embeddings, constructed from sentence embeddings, for bilingual document alignment. ## dataset creation and description the common crawl corpus is a publicly available crawl of the web. with a new snapshot uploaded each month, and over 2 billion pages released in each snapshot, this data is a vast resource with content across a large number of domains and languages. previous works have leveraged the data from common crawl for mining ngram counts to perform language modeling @xcite . other works @xcite have mined common crawl for bitexts for machine translation. however, this mining was performed on a small scale. for our dataset, we use 68 snapshots published from 2013 to 2020 which is vastly larger than previous works. ## dataset evaluation in this section, we analyze the quality of our crosslingual url-aligned dataset. we assesses the quality by measuring the precision of a representative sample of the url-aligned data to humanannotated alignment judgments. ## document alignment experiments in section 4, we verify the quality of the urlaligned dataset through human-evaluation. in this section, we treat the url-aligned dataset as a high-precision, low-recall dataset and evaluate baselines that score document pairs based on content rather than url information. the scored document pairs are then aligned via a greedy bipartite matching algorithm. the resulting alignments are evaluated on a subset of the url-aligned dataset which is treated as ground truth. ## case study: mt as an application in the section 4 we did a manual evaluation to gauge the quality of our high-precision alignment method. however, the study is small and does not shed light on the quality of the full dataset. therefore, to assess the quality of the aligned document corpus, we propose a downstream task that leverages the aligned document data as a here, we use mt to evaluate the quality of the mined sentences, and compare it to other independently mined corpora. our expectation is that, if successful, our approach should be able to mine parallel sentences that are of comparable quality to recent approaches that leverage wikipedia @xcite and paracrawl 2 @xcite , reliable sources of comparable documents. sentence mining the first step is to decompose and mine the aligned document corpus for parallel sentences. for simplicity, we segment each document solely on new lines. given each document pair's decomposition into sentences, we seek 2 https://paracrawl.eu/ to align sentences within each pair of documents. we then aggregate the parallel sentences across all document pairs to form a parallel sentences dataset suitable for training machine translation models. we use the open-source laser toolkit @xcite with the marginbased filtering criterion to mine sentences, as this method has been shown to accurately align sentences for across a variety of low, mid, and high-resource directions @xcite . we use the extracted bitexts for training our nmt systems. ## conclusion & future works in this paper, we apply url-matching to curate a high-quality cross-lingual documents dataset from the commoncrawl corpus. our dataset contains over 392 million document pairs from 8144 language pairs covering 138 distinct languages. we first directly evaluate the quality of the urlaligned pairs using human annotators. we then introduce and evaluate simple embedding-based baseline techniques for aligning documents based on content. our results indicate there is further work to be done to improve document alignment, especially for low-resource languages and that intelligent alignment schemes can significantly improve alignment performance across many language directions. finally, we perform a case-study showing that our url-aligned documents can be mined for high-quality parallel sentences which can be used to train machine translation models. given the sheer size of this dataset, this has the potential to provide high-quality training data for many low-resource language pairs. one natural followup to this work is to develop techniques to better mine parallel sentences from these aligned documents -especially for low-resource language pairs. additional work could also leverage aligned documents as supervision to learn better cross-lingual document representations. finally, while the aligned dataset is high-precision, leveraging this dataset for supervision in document alignment can potentially yield a larger, high-recall collection. to spur further work, we release the list of aligned urls as well as code to generate aligned documents given com-moncrawl snapshots foot_1 .