Increasing Coverage and Precision of Textual Information in Multilingual Knowledge Graphs (2023)

Abstract: Recent work in Natural Language Processing and Computer Vision has been using textual information – e.g., entity names and descriptions – available in knowledge graphs to ground neural models to high-quality structured data. However, when it comes to non-English languages, the quantity and quality of textual information are comparatively scarce. To address this issue, we introduce the novel task of automatic Knowledge Graph Completion (KGE) and perform a thorough investigation on bridging the gap in both the quantity and quality of textual information between English and non-English languages. More specifically, we: i) bring to light the problem of increasing multilingual coverage and precision of entity names and descriptions in Wikidata; ii) demonstrate that state-of-the-art methods, namely, Machine Translation (MT), Web Search (WS), and Large Language Models (LLMs), struggle with this task; iii) present M-NTA, a novel unsupervised approach that combines MT, WS, and LLMs to generate high-quality textual information; and, iv) study the impact of increasing multilingual coverage and precision of non-English textual information in Entity Linking, Knowledge Graph Completion, and Question Answering. As part of our effort towards better multilingual knowledge graphs, we also introduce WikiKGE-10, the first human-curated benchmark to evaluate KGE approaches in 10 languages across 7 language families.

## introduction the objective of a knowledge graph is to encode our collective understanding of the world in a welldefined, structured, machine-readable representation @xcite . at a high level, each node of a knowledge graph usually represents a concept (e.g., universe, weather, or president) or an entity (e.g., albert einstein, rome, or the legend of zelda), and each edge between two nodes * work done as intern at apple. is a semantic relationship that represents a fact (e.g., "rome is the capital of italy" or "the legend of zelda is a video game series"). with the wealth of information that knowledge graphs provide, they play a fundamental role in a multitude of real-world scenarios, touching many areas of artificial intelligence @xcite , including natural language processing @xcite , computer vision @xcite , information retrieval @xcite , and recommender systems @xcite . over the years, knowledge graphs have mainly been adopted as a rich source of human-curated relational information to enhance neural-based models for tasks of varying nature @xcite @xcite . however, ever since natural language text has proven to be an effective interface between structured knowledge and language models @xcite @xcite , the value of knowledge graphs has become twofold: besides providing relational information, knowledge graphs have also become a reliable source of high-quality textual information. indeed, recent approaches have been increasingly reliant on textual information from knowledge graphs to surpass the state of the art @xcite @xcite . unfortunately, when it comes to non-english languages, the condition of multilingual textual information in knowledge graphs is far from ideal. indeed, popular resources present a significant gap between english and non-english textual information, hindering the capability of recent approaches to scale to multilingual settings @xcite importantly, this gap exists in high-resource languages even if we consider basic textual properties, such as entity names and entity descriptions. the nature of the problem is dual: disparity in coverage, as the quantity of textual information available in non-english languages is more limited, and precision, as the quality of non-english textual information is usually lower. in this paper, we address the aforementioned coverage and precision issues of textual information in multilingual knowledge graphs via a data-centric approach. our contributions include the following: we deem that achieving parity of coverage and precision of textual information across languages in knowledge graphs is fundamental to enable better and more inclusive multilingual applications. in the hope that our contributions can set a stepping stone for future research in this field, we release wikikge-10 at https://github.com/apple/ml-kge . ## related work in this section, we provide a brief overview of knowledge graphs, highlighting how textual information from knowledge graphs is now as important as their relational information, showcasing how recent work has successfully integrated textual information into downstream applications, and reviewing how recent efforts have mainly focused on completing relational information in knowledge graphs rather than textual information. knowledge graphs. even though their exact definition remains contentious, knowledge graphs are usually defined as "a graph of data intended to accumulate and convey knowledge of the real world, whose nodes represent entities of interest and whose edges represent potentially different relations between these entities" @xcite . over the years, research endeavors in knowledge graphs have steadily focused their efforts primarily on using their relational information, i.e., the semantic relations between entities. besides foundational work on knowledge graph embedding techniques, which represent the semantics of an entity by encoding its graph neighborhood @xcite , relational knowledge has been successfully employed in question answering to encode properties that generalize over unseen entities @xcite @xcite , in text summarization to identify the most relevant entities in a text and their relations @xcite , in entity linking to condition the prediction of an instance on knowledge subgraphs @xcite , and in word sense disambiguation to produce rich meaning representations that can differentiate closely related senses @xcite . ## knowledge graph enhancement of textual information while relational information in knowledge graphs is usually language-agnostic (e.g., "ai" is a field of "computer science" independently of the language we consider), textual information is usually language-dependent (e.g., the lexicalizations of "ai" and "computer science" vary across languages). with the growing number of languages supported by knowledge graphs, it is increasingly challenging for human editors to maintain their content up-to-date in all languages: therefore, we believe it is important to invest in the development and evaluation of systems that can support humans in updating textual information across languages. ## methodology in this section, we consider three broad families of approaches -mt, ws, and llms -and demonstrate their unsatisfactory performance on narrow-ing the coverage and precision gap between english and non-english languages. therefore, we also introduce m-nta (multi-source naturalization, translation, and alignment), a simple unsupervised ensembling technique, which overcomes the limitations of mt, ws, and llms by combining and ranking their predictions. here, we direct our attention toward entity names, but we also show that the methodologies discussed in this section can be extended to other types of textual information, such as entity descriptions, in appendix c. ## experiments on kge in this section, we evaluate our strong baselines and m-nta on the task of kge for entity names and discuss the results obtained on wikikge-10. ## conclusion and future work in this paper, we introduced the novel task of automatic knowledge graph enhancement, with the objective of fostering the development and evaluation of data-centric approaches for narrowing the gap in coverage and precision of textual information between english and non-english languages. thanks to wikikge-10, our novel manually-curated benchmark for evaluating kge of entity names in 10 languages, we brought to light the unsatisfactory capabilities of machine translation, web search, and large language models to bridge this multilingual gap. to this end, we introduced m-nta, a novel approach to combine the complementary knowledge produced by the above techniques to obtain higher-quality textual information for non-english languages. not only did m-nta achieve promising results on wikikge-10 but our experiments also demonstrated its beneficial effect across several state-of-the-art systems for downstream applications, namely, multilingual entity linking, multilingual knowledge graph completion, and multilingual question answering. we hope that our novel benchmark and method can represent a milestone for kge. however, our work demonstrates that, if we aspire to achieve quantity and quality parity across languages, we still need more extensive investigations on how to effectively increase coverage and precision of textual information in multilingual knowledge graphs.