CroCoAlign: A Cross-Lingual, Context-Aware and Fully-Neural Sentence Alignment System for Long Texts (2024)

Abstract: Sentence alignment – establishing links between corresponding sentences in two related documents – is an important NLP task with several downstream applications, such as machine translation (MT). Despite the fact that existing sentence alignment systems have achieved promising results, their effectiveness is based on auxiliary information such as document metadata or machine-generated translations, as well as hyperparameter-sensitive techniques. Moreover, these systems often overlook the crucial role that context plays in the alignment process. In this paper, we address the aforementioned issues and propose CroCoAlign: the first context-aware, end-to-end and fully neural architecture for sentence alignment. Our system maps source and target sentences in long documents by contextualizing their sentence embeddings with respect to the other sentences in the document. We extensively evaluate CroCoAlign on a multilingual dataset consisting of 20 language pairs derived from the Opus project, and demonstrate that our model achieves state-of-the-art performance. To ensure reproducibility, we release our code and model checkpoints at https://github.com/Babelscape/CroCoAlign.

## introduction sentence alignment is the task of matching sentences in two or more documents that are related to each other @xcite , as shown in figure 1 . the task is important in many downstream applications, including machine translation (mt, @xcite ), text simplification @xcite and paraphrase generation @xcite . although current approaches have achieved promising results on standard benchmarks for the task @xcite , they are strongly focused on hyperparameter-sensitive heuristics and on using auxiliary mt systems, hence overlooking the primary role that context plays when performing sentence alignment @xcite figure 1 : examples of 1-to-1, many-to-1 and 1-to-many alignments between source and target sentences written in english and italian, respectively. 2011 ; @xcite . indeed, sentences can be ambiguous when taken out of context, whereas modeling the surrounding sentences helps in disambiguating meanings, leading to a more accurate alignment. in particular, we note that existing approaches are not suitable for fully addressing complex challenges like many-to-many alignments and the identification of non-alignable sentences, which also require the modeling and understanding of context. we emphasize that the foregoing are not uncommon challenges when aligning long texts, such as books, which may have been adapted through transcreation, according to socio-economic and cultural factors @xcite . moreover, current approaches are mainly focused on european parliament transcriptions (europarl, @xcite and on other extremely specific domains, such as the digitized heritage of german and french alpine literature @xcite and the bible @xcite . however, we observe that the peculiarities of the aforementioned corporathe shortness of text+berg and its focus on a single language pair, the political-financial domain of europarl and its transcriptive style, as well as the genre of the bible -may not provide a suitable framework for training current approaches and evaluating their generalizability. to address the above-mentioned limitations, in this paper we carry out an in-depth investigation on the role of modeling cross-sentence context in the process of sentence alignment in long texts, and put forward the following three main contributions: ultimately, we hope that this study will stimulate further research into sentence alignment systems that can improve the understanding and analysis of long texts as a whole. ## related work in this section, we review the literature of the sentence alignment task. to highlight the unique aspects of this task, we also outline the differences between sentence alignment and bitext mining, a closely related task, and describe why bitext mining systems are not suitable for sentence alignment. finally, we showcase applications of sentence alignment systems in mt, underlining the importance of the task in real-world scenarios. ## crocoalign in this section, we describe crocoalign, a language-agnostic, fully-neural method for aligning sentences between pairs of documents, designed specifically to model context in long texts. our core intuition is that the embedding of an individual sentence, independently of how expressive it may be, lacks information about its surrounding context, i.e., the previous and following sentences. therefore, the fundamental novelty of crocoalign lies in its modeling of the document-level sequentiality of the sentence representations. crocoalign accomplishes this process by initially encoding source and target sentences using a sentence transformer (section 3.1). it subsequently enhances the resulting sentencelevel representations by employing a novel context encoder to incorporate additional contextual information at the document level (section 3.2). then, it proceeds by determining whether or not a sourcetarget sentence pair is an alignment by feeding the element-wise product of the resulting contextualized sentence embeddings to a multi-label classifier (section 3.3), minimizing the training objective explained in section 3.4. finally, crocoalign also features a novel two-step procedure, which we refer to as pointing and recovery, needed at inference time to address the problem of locating the sentences to be aligned and refining the predictions, as described in section 3.5. figure 2 shows the overall architecture. ## experimental setup our system is developed using the pytorch lightning framework foot_1 and the huggingface models library. 4 in order to generate the initial sentence embeddings (section 3.1) for each source and target batch of sentences, we use the sentence transformer language-agnostic bert sentence embeddings foot_3 @xcite . during training, we keep the weights of the sentence transformer frozen. for the transformer encoder (section 3.2), we employ distilbert foot_4 @xcite , which is initialized with random weights and then trained with six attention heads, six layers and a dropout of 0.2. for the classification head (section 3.3), we use a dropout of 0.2 and the relu activation function. we train our system on a rtx 3090 ti for a maximum of 1.25 million steps and an early stopping mechanism with a patience set to 10. we use the adamw optimizer with a weight decay of 0.01, a learning rate of 10 -5 , and a linear scheduler with a warmup of 10% of the maximum number of training steps. we select the best model based on its strict f1 score on the validation set, which demands an exact match between the predicted and the gold alignments. in the following, we describe the dataset we use for our experiments (section 4.1), the baselines we compare with (section 4.2), and the variants of our model (section 4.3). ## results in this section, we present the results obtained by crocoalign and its competitors on the dataset introduced in section 4.1 in terms of strict f1 score. quantitative results. finally, despite the fact that vecalign employs laser as the underlying sentence encoder, in order to ensure a fair comparison we also evaluate it using labse. the main difference between the two encoders is that the former is designed as a bilingual model -requiring a distinct model for each language pair -while the latter is language agnostic. as shown in alignment analysis. in addition to the quantitative results presented in the previous paragraph, weperformed an analysis of the accuracy of different alignment types, comparing our best model with vecalign. in figure 3 we report the accuracy, expressed as a percentage, of the number of times a specific type of alignment is predicted correctly by the two systems. from the results we can see that, on average, both crocoalign and vecalign perform similarly when tested on 1-to-1 up to 1to-7 alignments. however, when presented with particularly challenging types of alignment (upper entries in figure 3 ), which can happen frequently in long texts, our system consistently outperforms its competitor. moreover, crocoalign is also more resilient to other very common situations in long texts where, given a sentence, there is no equivalent in its corresponding parallel document (1-to-0 and 0-to-1 alignments in figure 3 ). inference speed. when assessing sentence alignment systems, it is crucial to consider their inference speed, especially given their typical application to extensive datasets. in this context, we conducted a comparative analysis between cro-coalign and vecalign, our primary competitor. notably, vecalign employs a combination of dynamic programming and sentence embeddings for a fast alignment process. however, it necessitates a preprocessing step where sentences from source and target documents undergo a complex encoding process using a sentence transformer. indeed, the algorithm requires the encoding of clusters of adjacent sentences to identify many-to-many alignments, introducing an overhead in the overall process before the execution of the alignment step. as a consequence, crocoalign is 2.7× faster than vecalign-laser and 3.3× faster than vecalign-labse. on a single gtx 1080 ti, crocoalign requires 84 seconds on average (9.49 of which are devoted to the generation of sentence embeddings and 75.12 to produce the final alignment) to align a pair of parallel books in our dataset. in con3 , 8) (3 , 9) (3 , 16) (4 , 9) (4 , 10 trast, vecalign requires 224 seconds on average to encode both source and target documents (112 seconds for a single document) and 3 seconds to find the optimal alignment, for a total of 227 seconds. as a rough estimate, when a single gpu is adopted, crocoalign can align around 1000 pairs of books and 50 million tokens in a day, compared to the 380 pairs of books and 18.5 million tokens of vecalign. ## experiments on machine translation sentence alignment serves as a fundamental tool in mt systems, as it can be used for the creation of parallel datasets for training purposes. here, we demonstrate the effectiveness of crocoalign for the automatic creation of a high-quality parallel fine-tuning set that can be used to adapt an existing mt system to specific domains. our hypothesis is that, with a high-quality alignment, we only require a small amount of parallel sentences to significantly boost the performance of a pretrained mt model. ## conclusion and future work in this paper, we presented crocoalign, the first fully-neural and language-agnostic architecture to perform sentence alignment in very long texts. the strength of our approach lies in its ability to create better sentence representations by taking advantage of their surrounding context in a fully-neural model equipped with a novel encoder that captures cross-sentence information, including the position and the meaning of a sentence with respect to the previous and following ones. our experiments on sentence alignment in bookswhich feature extremely long contexts and present various instances of many-to-many alignmentsshow that crocoalign outperforms the previous state of the art, i.e. vecalign, by a significant margin across 20 language pairs (+1.6 points in f1 score on average). moreover, we evaluate the impact that the data produced by crocoalign has on the task of machine translation, and show language pair fine-tuning bleu score ∆ de-es ✗ 23.9 +4.4 ✓ 28.3 de-en ✗ 20.7 +2.1 ✓ 22.8 de-fr ✗ 9.2 +4.1 ✓ 13.3 de-hu ✗ 8.5 +2.0 ✓ 10.5 de-it ✗ 4.2 +2.0 ✓ 6.2 en-es ✗ 17.8 +6.1 ✓ 23.9 en-fr ✗ 33.6 +7.5 ✓ 41.1 en-hu ✗ 11.8 +3.0 ✓ 14.8 en-it ✗ 17.9 +6.0 ✓ 23.9 en-nl ✗ 15.6 +6.9 ✓ 22.5 en-ru ✗ 20.6 +4.9 ✓ 25.5 es-fr ✗ 19.5 +2.6 ✓ 22.1 es-it ✗ 12.7 +2.1 ✓ 14.8 es-nl ✗ 10.4 +5.2 ✓ 15.6 fr-hu ✗ 8.2 +2.8 ✓ 11.0 the system does not present any significant limitations. however, due to hardware constraints, we implemented two procedures during inference: one to identify the source and target batches of sentences to be aligned, and the other to correct errors resulting from multiple alignments to the same source sentence. we emphasize that these procedures are only required when the entire source and target documents do not fit within the gpu memory available. moreover, if sufficient computational power was available, the model would not only eliminate the need for these two procedures, but it would also make use of the larger textual context in order to align all the sentences simultaneously and, possibly, more accurately. conference on language resources and evaluation (lrec'10), valletta, malta. european language resources association (elra). michał ziemski, marcin junczys-dowmunt, and bruno pouliquen. 2016. the united nations parallel corpus v1.0. in proceedings of the tenth international conference on language resources and evaluation (lrec'16), pages 3530-3534, portorož, slovenia. european language resources association (elra). pierre zweigenbaum, serge sharoff, and reinhard rapp. 2017. overview of the second bucc shared task: spotting parallel sentences in comparable corpora. in proceedings of the 10th workshop on building and using comparable corpora, pages 60-67, vancouver, canada. association for computational linguistics.