Locating Information Gaps and Narrative Inconsistencies Across Languages: A Case Study of LGBT People Portrayals on W ikipedia (2024)

Abstract: To explain social phenomena and identify systematic biases, much research in computational social science focuses on comparative text analyses. These studies often rely on coarse corpus-level statistics or local word-level analyses, mainly in English. We introduce the InfoGap method—an efficient and reliable approach to locating information gaps and inconsistencies in articles at the fact level, across languages. We evaluate InfoGap by analyzing LGBT people’s portrayals, across 2.7K biography pages on English, Russian, and French Wikipedias. We find large discrepancies in factual coverage across the languages. Moreover, our analysis reveals that biographical facts carrying negative connotations are more likely to be highlighted in Russian Wikipedia. Crucially, InfoGap both facilitates large scale analyses, and pinpoints local document- and fact-level information gaps, laying a new foundation for targeted and nuanced comparative language analysis at scale.

## introduction wikipedia has several hundred language editions, a sizeable number of which have more than 100k articles. despite its "neutral point of view" policy, abundant evidence of content discrepancies across language editions has been well-documented on the platform (e.g., @xcite @xcite @xcite . there are numerous motivations for identifying and studying these variations, e.g., identifying content variations and gaps can aid editors in removing social and cultural biases @xcite . alternatively, from a social science perspective, comparative analyses of prominent topics across wikipedia language editions provides a window into studying cross-during the season, griner had recorded the sixth triple-double in wnba history. griner became engaged to cherelle watson in august 2018. стейн прорвался мимо телохранителей команды и обвинял грайнер в том, что она "ненавидит америку". бриттни родилась @xcite griner est désignée meilleure joueuse de l'année par l'associated press (griner was named the associated press women's player of the year.) brittney griner est née le 18 octobre 1990. ## infogap: identifying information asymmetry in wikipedia articles consider a pair of articles on a topic written in different languages. we call one article e and the other in the pair f . moreover, we represent e by a series of facts e 1 , . . . , e n and f similarly as f 1 , . . . , f m . our method determines for a given fact e @xmath0. the pipeline is directional, so we can compute both f ⊩ e i and e ⊩ f i . without loss of generality, we will describe the procedure for obtaining the labels f ⊢ e i , for all e i . we refer to this as the @xmath1. fig. 2 presents an overview of infogap. we primarily focus on two steps. first, following @xcite , we narrow the search space of equivalent facts by aligning a fact in e to facts in f that may convey the same information (sec 2.1). this allows us to efficiently assess the equivalence between aligned facts (sec 2.2). we determine the reliability of infogap in section 2.3. ## using infogap to analyze asymmetries in lgbt wikipedia bios having validated the effectiveness of infogap, we move onto applying it to answer questions about information gaps in wikipedia. we focus on identifying content differences between language versions' articles on lgbt public figures. prior work by @xcite identified that english articles on average portrayed these figures with more positive sentiment, as well as greater power and agency @xcite , relative to articles in russian and spanish. to gain further insight into cross-linguistic variation towards lgbt people portrayals, we draw on the lgbtbiocorpus corpus @xcite . the corpus comprises 1, 350 biographies of lgbt people, each paired with biographies of non-lgbt people matched on most social attributes except sexual orientation using the matching method introduced in @xcite . given that info-gap enables us to directly compare the content between different language versions of a biography, we contend that our analysis can provide a more direct characterization of differences in lgbt bios. specifically, we look at en, fr, and ru wikipedias. we consider the following research questions: rq 1 : to what extent does factual knowledge differ across language versions of the same bio (sec 3.2)? rq 2 : does a person's affiliation with the lgbt community have an effect on the information gap in their bios (sec 3.3)? rq 3 : can we use infogap to identify sections to remediate (sec 3.4)? these questions are intentionally ordered from high-level (language-level) to low-level (individualand fact-level) to demonstrate that infogap enables both high-level quantitative analyses and efficient low-level descriptive analyses. we start by providing the implementation details in section 3.1 before answering each of the research questions. ## related work automated comparison of multilingual wikipedia articles. we contribute to a large body of work on understanding differences between language versions of wikipedia. @xcite also compare wikipedia language versions and consider their information gaps, and later develop a web tool to bridge these multilingual gaps @xcite . however, their evaluation is at a higher level of abstraction: they look at whether or not two language versions have on a topic. by comparison, we compare content differences between two language versions on the same topic. @xcite considered a pipeline similar to infogap for the task of keeping multilingual wikipedia documents consistent. however, their pipeline used embedding similarity; in early experiments, we found that using embedding similarity for identifying potential entailments performed very poorly (relative to x-factmatch). @xcite created a web tool that permits visual comparison of wikipedia articles in two different languages. @xcite also perform comparative analyses across language versions in wikipedia. however, they consider more fine-grained content differences between pairs of the most closely related paragraphs between different language versions' article on a topic. their method was not designed for computing the overall article level overlaps and differences of the form we demonstrate in fig. 3 . ## conclusion we presented infogap, a reliable method for efficient comparative analysis between two narratives on the same topic written in different languages. we deployed the method to discover differences in lgbt people's portrayals, locating shared facts, as well as information gaps and inconsistencies across 2.7k english, russian, and french wikipedia biography pages. infogap can be directly applied beyond analyzing differences in multilingual wikipedia biographies. analyzing variation in topic coverage is at the heart of much research in the social sciences, from understanding media manipulation strategies @xcite , to analyzing differences in argumentation from different stances in a contentious debate @xcite , to analyzing quotation patterns in partisan media @xcite . overall, our research lays the foundation for enabling targeted, nuanced textual comparative analyses at scale.