X - PARADE : Cross-Lingual Textual Entailment and Information Divergence across Paragraphs (2024)

Abstract: Understanding when two pieces of text convey the same information is a goal touching many subproblems in NLP, including textual entailment and fact-checking. This problem becomes more complex when those two pieces of text are in different languages. Here, we introduce X-PARADE (Cross-lingual Paragraph-level Analysis of Divergences and Entailments), the first cross-lingual dataset of paragraph-level information divergences. Annotators label a paragraph in a target language at the span level and evaluate it with respect to a corresponding paragraph in a source language, indicating whether a given piece of information is the same, new, or new but can be inferred. This last notion establishes a link with cross-language NLI. Aligned paragraphs are sourced from Wikipedia pages in different languages, reflecting real information divergences observed in the wild. Armed with our dataset, we investigate a diverse set of approaches for this problem, including classic token alignment from machine translation, textual entailment methods that localize their decisions, and prompting LLMs. Our results show that these methods vary in their capability to handle inferable information, but they all fall short of human performance.

## introduction the ability to recognize differences in meaning between texts underlies many nlp tasks such as natural language inference (nli), semantic similarity, paraphrase detection, and factuality evaluation. less work exists on the cross-lingual variants of these tasks. however, correctly identifying semantic relations between sentences in different languages has a number of useful applications. these include estimating the quality of machine translation output @xcite , cross-lingual fact checking @xcite , and helping 1 dataset available at https://github.com/juand-r/ x-parade la ciudad fue fundada por john c. williams y por peter demens, quien hizo llegar el ferrocarril hasta la ciudad en 1888. petersburg se incorporó el 29 de febrero de 1892, en aquella época tenía una población de sólo 300 habitantes. the city was co-founded by john c. williams, formerly of detroit, who purchased the land in 1875, and by peter demens, who was instrumental in bringing the terminus of the orange belt railway there in 1888. st. petersburg was incorporated as a town on february 29, 1892, when it had a population of 300 people. wikipedia editors mitigate discrepancies in content across languages @xcite . the fact that different languages carve up the world in different ways (de saussure, @xcite @xcite and have different syntactic constraints @xcite may also make these tasks more challenging. many of these tasks involve reasoning beyond the sentence level. at the level of paragraphs, it is no longer useful to have coarse labels like "entailed" or "neutral"; instead, we want to capture subtle differences in information content @xcite @xcite . thus, we focus on the problem of detecting fine-grained span-level information divergences between texts across languages. notably, our notion of information divergences differentiates between new information and new information that can be inferred from the source paragraph. this paper presents a dataset called x-parade : cross-lingual paragraph-level analysis of divergences and entailments. figure 1 shows an example english-spanish paragraph pair, with annotations on how the english paragraph differs from the spanish paragraph. we see a rich range of inferences being required to understand the target, including effects like quien hizo llegar (who brought) implying that someone was instrumental in bringing. these kinds of subtle cross-lingual divergences are anchored to individual spans in the target paragraph. finally, unlike prior work that tackled sentence-level comparisons between languages (briakou and carpuat, 2020), we annotate entire paragraphs. by having larger textual units, we can capture a wider array of divergences and more appropriately model the nuances of cross-sentence context in this task. we conduct annotation in three language pairs, yielding six directions, using trained annotators from upwork who went through extensive qualification and feedback rounds. our dataset is of high quality, with token-level krippendorff α agreement scores ranging from 0.55 to 0.65, depending on the language pair. finally, we benchmark the performance of existing approaches on this problem. no systems in the literature are directly suitable. we compare a diverse set of techniques that solve different aspects of the problem, including token attribution of nli models, machine translation (mt) alignment and large language models (llms). while gpt-4 performs the best, different approaches have different pros and cons and there remains a gap with human performance. the main contributions of this work are: ## dataset construction our dataset construction pipeline is shown in figure 2. it consists of three stages. we sample from a diverse set of wikipedia pages, identify paragraph pairs that are sufficiently related but not identical to serve as candidates for our annotation, and present these to annotators to label. ## methods while the task of detecting new and inferable information in paragraphs across languages is novel, it relates to ideas from machine translation and textual entailment. here we describe how to adapt baselines from these areas to assess their performance on this task, as well as prompting llms to produce spans (figure 3 ). implementation details can be found in appendix c. alignment mt word alignment predicts which words should be aligned across translations; thus words which do not easily align are more likely to present new content not given in the source paragraph. by way of approximation, we will assume in these experiments that unaligned tokens fall into the new category. slr-nli slr-nli @xcite builds on the idea that a neutral or contradiction relation holds between two sentences only when there is at least one span in the "hypothesis" (target) that is not inferable from the premise. since these spans are exactly the ones containing new information, we use slr-nli to predict which spans in the target paragraph are new. ## conclusion we present x-parade , a new dataset of crosslingual paragraph pairs (english-spanish, english-hindi, english-chinese), annotated for semantic divergences at the span-level. although the task features subjectivity, the analysis of our annotation shows that decisions by the annotators were welljustified. we show that while some of these finegrained differences can be detected by gpt-4, there is still a gap with human performance. we believe that this dataset can be useful for benchmarking the inferential capabilities of multilingual llms and analyzing how textual entailment systems can identify information divergences cross-lingually.