CDA : a Cost Efficient Content-based Multilingual Web Document Aligner (2021)

Abstract: We introduce a Content-based Document Alignment approach (CDA), an efficient method to align multilingual web documents based on content in creating parallel training data for machine translation (MT) systems operating at the industrial level. CDA works in two steps: (i) projecting documents of a web domain to a shared multilingual space; then (ii) aligning them based on the similarity of their representations in such space. We leverage lexical translation models to build vector representations using TF×IDF. CDA achieves performance comparable with state-of-the-art systems in the WMT-16 Bilingual Document Alignment Shared Task benchmark while operating in multilingual space. Besides, we created two web-scale datasets to examine the robustness of CDA in an industrial setting involving up to 28 languages and millions of documents. The experiments show that CDA is robust, cost-effective, and is significantly superior in (i) processing large and noisy web data and (ii) scaling to new and low-resourced languages.

## introduction online machine translation (mt) services require industrial-scale training data, i.e., significantly large and high-quality parallel sentences, to build accurate models. exploiting the web for multilingual content has become a usual strategy in collecting large-scale parallel sentences for mt @xcite @xcite . structural translation recognition for acquiring natural data (strand) @xcite is a standard pipeline to extract parallel data from the web, consisting in three steps: (i) bilingual document alignment for an input set of documents, (ii) sentence alignment for each aligned document pair, and (iii) sentence filtering for nontranslation or boilerplate cleaning. the first step of identifying bilingual documents is technically challenging and made more complicated by the presence of large and noisy documents from web data. in the wmt-16 bilingual document alignment shared task (wmt16-bdast), two standard approaches for identifying parallel pages foot_0 were studied: 1) url matching heuristic @xcite as a baseline and 2) content similarity as a solution to maximize the performance in identifying parallel documents. the benchmark on english-french document alignment task shows that the best top-1 recall (r@1) for each approach are 59.8% and 89.1%, respectively, as evaluated on the test set @xcite . the results, albeit conducted within english-french setting, indicate that leveraging document content can lead to a significant increase, up to 30 percent points in recall, and contributes ∼50% novel bilingual document pairs. the url matching heuristic approach, named url, identifies parallel pages using language identifiers, typically from iso 639, annotated in the addresses. pages, or web-documents, in different languages from a domain are aligned if their urls are matchable after their language-identifier being removed @xcite . the strategy can identify a candidate at a decent cost by comparing two urls without significant preprocessing needed. for example, the following urls are a match: xyz.ca/index.htm and xyz.ca/fr/index.htm after removing "fr/" from the second url. on the contrary, cost is the major issue when comparing content as it often requires language-specific processing and sophisticated modelings for cross-language normalization and alignment. the problem becomes even more challenging when dealing with web data and for low-resourced languages. we optimize the cost for the latter approach to enable its application at scale. specifically, we design cda to project multilingual documents to a shared multilingual space for direct similarity measurement. therefore, we can run the strand pipeline for multiple languages at once to keep the pre-processing cost monotonic with respect to the number of languages and documents. in particular, we design cda with two key objectives: i) minimal data processing cost and ii) fast scaling to new languages. the latter is also crucial to the language expansion in online mt services. our contribution is three-fold: we tested cda with multiple large-scale datasets of web documents. we also studied the applications of cda within an industrial setting, including (i) extracting more and better parallel sentences from an extremely large-scale dataset, (ii) producing better mt models, and (iii) improving yield, measured by the amount of extracted parallel content, for both url and cda in the strand pipeline. the experimental results show that, despite its minimality, cda (i) is on par with the top systems from wmt16-bdast, which use expensive bilingual resources, and (ii) can double the amount of parallel data extracted by previous urlbased approaches. most importantly, we show that cda provides robust performance when dealing with millions of documents and processing up to 28 languages, including low-resourced languages. in the remainder of this paper, we summarize the previous work regarding document alignment in section 2. we then describe the proposed system and our experiments in sections 3 and 4, respectively. finally, we derive the conclusions of the paper in section 5. ## related work aligning multilingual documents is the key required processing in most multilingual text processing pipelines, including cross-lingual information retrieval @xcite @xcite and parallel data extraction. in the context of creating parallel training data for mt, the problem has been studied in the literature for comparable corpora @xcite @xcite and web-structured parallel data extraction @xcite @xcite . we focus on the latter in this paper. wmt-16 bilingual document alignment shared task is a recent shared-task focusing on identifying bilingual documents from crawled websites @xcite . the top 3 systems are yoda (dara and lin, 2016), novalincs (gomes and pereira lopes, 2016), and uedin1 cosine @xcite . the first two require costly features, such as (i) ngram comparison after translating all non-english text into english (dara and lin, 2016), and (ii) phrase-table of statistical mt (smt) as a dictionary (gomes and pereira lopes, 2016). uedin1 cosine @xcite , on the other hand, only uses tf × idf-weighted with cosine similarity. interestingly, this method performs surprisingly well even without french-to-english translations, dropping just 3.4% in recall, from 93.7% to 90.3%. though the finding can be due to the english and french lexicons' overlap, it suggests that tf×idf with proper normalization is useful to compare document representations from sub-domains. our proposed method exploits this aspect. given the advent of deep neural network modeling, document embeddings are among the main interests in general nlp applications @xcite @xcite . this line of research, however, is not technically related to our problem setting. specifically, the cost to run a neural inference over a web-scale setting is prohibitively high, e.g., processing a dataset of several billion pages from com-moncrawl foot_1 is not feasible. to have an idea, the cloud translation foot_2 cost to translate a webpage having 20,000 characters is $0.4 as of jan 2021. ## proposed system we describe our method to identify parallel pages from a web domain. specifically, pages in different languages are projected to a shared space, where their similarity can be measured. problem definition let @xmath0, d 2 , . . . , d n } be the n pages from a domain, each page d i is described by its content c i in language l i . the problem is to identify all (d i , d j ) of different languages, l @xmath1, and c i and c j are translational equivalent. multilingual space let @xmath2, l 2 , . . . } be the set of languages found in d and let d i be the set of documents in l i . thus, @xmath3, where each d i is associated with a lexicon v i . without loss of generality, we project documents from two languages, l α and l β , into a common space as follows. we first define alignment between two lexicons v α and v β as: a}, which maps d β into language α. similar, we denote the mapping for a document collection d β as d β→α = {m (d β ) : ∀d β ∈ d β }. finally, we compute the tf×idf representation of d β as follows, ∀w @xmath4: we compute x i in eq. 1 using tf(w i ) × idf(w i ). aligning multilingual documents two documents are considered a good pair if their representations are similar, according to a similarity threshold t. we compute the similarity between d i and d j as the dot-product between, v i • v @xmath5..1] (we normalized the vector representations with ℓ 2 ). in practice, we use english to build the multilingual space as it is the dominant language on the internet and in most multilingual websites. ## experiments we examine the efficacy of cda in this section. first, we describe (i) the pipeline setup for the experiments and (ii) our effort in creating suitable benchmark data and selecting relevant resources in section 4.1, and section 4.2, respectively. we then address the following performance aspects of cda: ## conclusion we presented our content-based document alignment for web data, cda, which projects multilingual documents to a common space for similarity calculation. we also described our effort to collect and create benchmark datasets to study different performance aspects of the proposed method. the results show that cda is efficient when projecting multilingual documents in one go for as many as 28 languages. moreover, we also explain the different types of benchmarking for cda under industrial settings. the results show that our proposed method is robust when processing huge datasets and useful in identifying non-standard language identifiers and multilingual web-domains. finally, and most importantly, the only significant resource required by cda is the lexical translation dictionary: this can be easily built thanks to the recent advance in learning multilingual embeddings. future applications of cda can be many. for example, the urls paired with cda can be used to improve the coverage for url-based methods (e.g., czech case in