Document Alignment based on Overlapping Fixed-Length Segments (2024)

Abstract: Acquiring large-scale parallel corpora is crucial for NLP tasks such as Neural Machine Translation, and web crawling has become a popular methodology for this purpose. Previous studies have been conducted based on sentence-based segmentation (SBS) when aligning documents in various languages which are obtained through web crawling. Among them, the TK-PERT method (Thompson and Koehn, 2020) achieved state-of-the-art results and addressed the boilerplate text in web crawling data well through a down-weighting approach. However, there remains a problem with how to handle long-text encoding better. Thus, we introduce the strategy of Overlapping Fixed-Length Segmentation (OFLS) in place of SBS, and observe a pronounced enhancement when performing the same approach for document alignment. In this paper, we compare the SBS and OFLS using three previous methods, Mean-Pool, TK-PERT (Thompson and Koehn, 2020), and Optimal Transport (Clark et al., 2019; El-Kishky and Guzman, 2020), on the WMT16 document alignment shared task for French-English, as well as on our self-established Japanese-English dataset MnRN. As a result, for the WMT16 task, various SBS based methods showed an increase in recall by 1% to 10% after reproduction with OFLS. For MnRN data, OFLS demonstrated notable accuracy improvements and exhibited faster document embedding speed.

## introduction during the training phase of tasks such as neural machine translation, a substantial amount of parallel corpora is required. web crawling has emerged as an efficient approach for gathering large-scale parallel datasets, such as the paracrawl dataset @xcite , the jparacrawl dataset @xcite , ccaligned dataset @xcite , wikimatrix @xcite , and ccmatrix @xcite . the procedure for developing a parallel webcrawled corpus involves five steps @xcite : web crawling, text extraction, document alignment, sentence alignment, and sentence pair filtering. document alignment involves establishing associations between documents that are equivalent translations originating from distinct language collections, and it can be broadly categorized into three strategies, url matching @xcite @xcite , methods based on machine translation or bilingual lexicons @xcite @xcite @xcite , and leveraging sentence embeddings @xcite @xcite @xcite . the core concept of the last one involves transforming the sentences within documents into a series of feature vectors. these vectors are then used to calculate the similarity between documents from different languages, with pairs exhibiting high similarity selected as alignment results. however, it should be noted that crawled documents may not have uniform sentence segmentation and contain a lot of boilerplate text, such as headers, dates, and navigation menus. moreover, for potentially long sentences, critical information may be generalized by other non-essential details when encoding it into embedding. in this case, we explore an alternative approach for subdivision, which involves utilizing a fixed-length sliding window to partition segments, with a specified proportion of overlap between adjacent segments. in summary, our contributions are as follows: ## related work the concept of mining parallel data from webs has already been proposed in the 20th century @xcite . however, in earlier years, the most serious endeavors have been confined to large companies, such as google @xcite and microsoft @xcite . up to the present, there have been numerous large-scale web crawling datasets obtained through various strategies, including the paracrawl dataset @xcite obtained through url matching, the jparacrawl dataset @xcite based on machine translation, and both the wikimatrix @xcite and the ccmatrix @xcite derived from multilingual sentence embeddings. among the various web crawling methods, bitextor (esplà-gomis, 2009) is one of the most widely adopted tools. additionally, it incorporates a module known as docalign @xcite , which employs a tf-idf strategy to score document pairs within one language through machine translation of documents in other languages. in the wmt16 bilingual document alignment shared task @xcite , many techniques, systems, and tools were proposed to align cross-lingual document pairs. novalincs @xcite submitted three systems based on a phrase-based statistical machine translation framework, attaining the highest accuracy. in the shared task, there exist numerous alternative methods based on translation systems @xcite , url matching @xcite , or bilingual translation lexicon @xcite . however, methods relying on translation systems are contingent upon the availability of a high-quality translator, which is often challenging to obtain in advance. since the emergence of sentence-bert (sbert) @xcite , which used a siamese network with cosine similarity for contrastive learning english sentence embedding in 2019, there has been a proliferation of high-precision multilingual pre-trained sentence embedding models to date. in the same year, @xcite proposed the laser model, which employs max-pooling over the output of a stacked lstm-encoder. subsequently, @xcite utilized knowledge distillation to adapt the sbert for multilingual applications, named multilingual-sbert (ms-bert). more recently, @xcite (labse) expanded upon the framework of a dual encoder to learn cross-lingual language-agnostic embeddings from a pre-trained language model @xcite , demonstrating state-of-the-art performance on the bitext mining task. just as the application of word embedding in sentence alignment @xcite is pertinent, the proposition of introducing sentence embedding in document alignment warrants thorough consideration. in 2020, @xcite proposed a method (tk-pert) that involves utilizing regionally emphasized windows generated by a modified pert distribution @xcite to assign weights for sentences and then forming the feature vector of the document. following their steps, sannigrahi et al. (2023) evaluated the performance of the tk-pert method using the three currently predominant multilingual sentence embedding models: laser, msbert, and labse. the application of optimal transport in crosslingual alignment, initially performing sentencelevel alignment based on word embeddings, known as word movers' distance (wmd) @xcite . analogous to it, sentence movers' distance @xcite based on optimal transport (ot) was introduced for document-level alignment. ## ablation analysis in this section, we conducted an ablation analysis on three factors of ofls: overlapping rate, fixedlength, and language-pair dependent proportion ρ. however, due to the substantial size of the wmt16 test data, our analysis was limited to the smallerscale mnrn dataset. ## conclusion this paper presents the ofls strategy designed for splitting documents into overlapping fixed-length segments for the document alignment task. building upon the previous sentence embedding based methods, compared to sbs, ofls yields better results on the wmt16 document alignment shared task. specifically, the ofls based "tk-pert" and "ot w/tk-pert" surpass the two best-recorded machine translation based methods, achieving the highest recall among labse based approaches. simultaneously, we observed the same results on the mnrn dataset. furthermore, according to the ablation analysis in section 5, a smaller fixedlength can further improve accuracy for "tk-pert" and "ot w/ * ", though it also results in longer embedding time and higher storage cost. appropriate hyperparameters can enable ofls to surpass sbs in both accuracy and speed.