Exploiting Sentence Order in Document Alignment (2020)

Abstract: We present a simple document alignment method that incorporates sentence order information in both candidate generation and candidate re-scoring. Our method results in 61% relative reduction in error compared to the best previously published result on the WMT16 document alignment shared task. Our method improves downstream MT performance on web-scraped Sinhala–English documents from ParaCrawl, outperforming the document alignment method used in the most recent ParaCrawl release. It also outperforms a comparable corpora method which uses the same multilingual embeddings, demonstrating that exploiting sentence order is beneficial even if the end goal is sentence-level bitext.

## introduction document alignment is the task of finding parallel document pairs (i.e., documents that are translations of each other) in a large collection of documents, often crawled from the web. aligned documents have historically been used to produce sentence-level machine translation (mt) data, but there is growing evidence that mt systems should be trained and evaluated using document-level context @xcite @xcite . we exploit the simple idea that two parallel documents should each contain approximately the same information, in approximately the same order. this idea can be traced back at least to the late 1990s, when strand @xcite measured how well linearized html tags from two documents could be aligned in order to judge whether two web pages were likely parallel. however, more recent work has primarily used unordered representations for documents, including bags of words or n-gram features and averages of sentence embeddings. our method consists of two main parts: first, we propose a simple method for candidate generation which embeds documents into a joint semantic embedding space @xcite , in a way that preserves some order information in each document. this enables candidate generation via fast approximate nearest neighbor search. second, we propose re-scoring those candidate pairs by performing sentence alignment and then scoring that alignment based on (1) the semantic similarity of the resulting aligned sentence pairs; our method results in a 61% relative reduction in the false positive rate on the wmt16 document alignment shared task versus the best previously reported method. applied to web-scraped sinhala-english data from paracrawl @xcite , it improves mt performance by 1.2 bleu over the document alignment method used in the latest paracrawl release @xcite , when both are used with the vecalign sentence alignment toolkit @xcite . ## method we follow a 2-stage approach to consider the d s × d t possible alignments between d s source documents and d t target documents: foot_0 1. candidate generation: we first find a fixed number, k, of target documents as potential matches for each source document. 2. candidate re-scoring: we re-score the d s × k document pairs from part 1 using a more accurate but slower scoring method. both our candidate generation method and candidate re-scoring method explicitly account for the content of a document as well as the order of that content within the document. ## experiments and results we evaluate our document alignment method in both high-and low-resource settings. note that our method is not trained on any parallel documents, and is designed to be as language agnostic as possible. however, it relies on laser embeddings, which are trained on bitext. thus we expect performance to be at least partially a function of the quantity of data that laser is trained on. 6 for high-resource, we use the publicly available french-english data released for the wmt 2016 shared task on document alignment @xcite ) and evaluate document recall following the shared task. the shared task provides a strong set of baselines, as 13 different teams contributed at least one submission. for low-resource, we experiment with sinhala-english documents extracted from paracrawl. in this setting we do not have gold document alignments, so we instead evaluate the quality of mt systems trained on the data extracted via document alignment. we develop and set all parameters using the training data from wmt16 ('wmt16-train') and then test on the wmt16 test data ('wmt16-test') and the sinhala-english paracrawl data. basic statistics for each dataset are shown in ## related work there is a large amount of prior work in document alignment. one of the simplest methods is url similarity @xcite , although this has been shown to be brittle @xcite . html structure @xcite or metadata such as publication date @xcite is often similar between parallel websites. however, most more recent work has focused on content similarity via bag-of-words or bag-of-ngrams, using bilingual lexicon @xcite @xcite @xcite , machine translation @xcite , or phrase tables @xcite . some work has considered high-level order as a filtering step after using a unordered representation to generate candidates: @xcite and @xcite discard n-gram pairs outside a fixed window, while @xcite filters out documents that have high edit distance between sequences of corresponding n-gram pairs. @xcite and @xcite use sentence similarity and/or number of aligned sentences after performing sentence alignment to score candidate documents. @xcite score document pairs using the sentence-level nearest neighbor as well as the absolute difference in sentence position between sentence pairs. in contrast to these methods, our work considers high-level order in both candidate generation and re-scoring. @xcite demonstrated neural document embeddings are effective representations for document alignment. they trained on millions of document pairs in each specific language pair of interest; in contrast, this work is much simpler and does not require document-level training data. ## conclusion we present a simple but effective method for document alignment. our method uses multilingual sentence embeddings and explicitly models the order of sentences in documents, in both candidate generation and candidate re-scoring. our method outperforms all published results on the dataset released for the wmt16 shared task on document alignment. it also increases downstream mt performance in a low-resource setting over prior work, including a margin-based comparable corpora method @xcite . we use the same embeddings as the comparable corpora method, thus the improvement over that method demonstrates the importance of including sentence order in document alignment, even when document-level alignments are not required.