Linking Surface Facts to Large-Scale Knowledge Graphs (2023)

Abstract: Open Information Extraction (OIE) methods extract facts from natural language text in the form of (“subject”; “relation”; “object”) triples. These facts are, however, merely surface forms, the ambiguity of which impedes their downstream usage; e.g., the surface phrase “Michael Jordan” may refer to either the former basketball player or the university professor. Knowledge Graphs (KGs), on the other hand, contain facts in a canonical (i.e., unambiguous) form, but their coverage is limited by a static schema (i.e., a fixed set of entities and predicates). To bridge this gap, we need the best of both worlds: (i) high coverage of free-text OIEs, and (ii) semantic precision (i.e., monosemy) of KGs. In order to achieve this goal, we propose a new benchmark with novel evaluation protocols that can, for example, measure fact linking performance on a granular triple slot level, while also measuring if a system has the ability to recognize that a surface form has no match in the existing KG. Our extensive evaluation of several baselines show that detection of out-of-KG entities and predicates is more difficult than accurate linking to existing ones, thus calling for more research efforts on this difficult task. We publicly release all resources (data, benchmark and code) on https://github.com/nec-research/fact-linking.

## introduction open information extraction (oie) methods extract surface ("subject"; "relation"; "object")triples from natural language text in a schema-free manner @xcite . for example, given the sentence "michael jordan, who grew up in wilmington, played for chicago bulls", an oie system should extract the triples (i.e., surface facts): t 1 =("michael jordan"; "played for"; "chicago bulls") and t 2 =("michael jordan"; "grew up in"; "wilmington"). the output of such systems is used in a diverse range of downstream tasks, including summarization @xcite , question answering (qa) @xcite , event extraction @xcite ), text clustering @xcite and video grounding @xcite . oie triples, however, consist of surface-form entities and relations, which are frequently ambiguous (e.g., in t 1 and t 2 , the entity mention "michael jordan" may refer to several entities, e.g. the basketball player or the computer scientist). resolving such ambiguities, by linking the oie triple slots to inventories of unambiguous concepts, improves their downstream utility, e.g., in qa @xcite , automatic medical diagnosing @xcite or dialogue @xcite . knowledge graphs (kgs), on the other hand, are inventories of canonical facts in the form of (subject; predicate; object)-triples, where each slot is a unique (i.e., unambiguous) concept @xcite . kgs are, however, limited by their own static and often hand-crafted schema (i.e., fixed set of entities and predicates). as a consequence, methods that directly extract canonical kg triples from text @xcite , preemptively discard any information outside of the reference kg schema. acknowledging this, @xcite recently proposed schema-adaptable kg construction, with the goal of extracting information for a kg with an evolving schema. @xcite conclude that oie methods indeed extract meaningful new knowledge for such kgs, however, they point precisely to the ambiguity of the surface forms as the major obstacle. to combine the best of both worlds, we need to bridge the gap between the schema-free (but ambiguous) surface facts extracted from text and the schema-fixed (but unambiguous) kg knowledge. however, existing benchmarks and models only partially address the problem. one line of work @xcite @xcite assumes a setting, which is arguably unrealistic, because the oie entity slots are a priori polysemous transductive inductive out of kg other kg nodes chicago bulls (q128109) member of sports team (p54) michael jordan (q41421) michael jordan (q3308205) berkeley (q168756) professor (p803) wilmington (q659400) ## fact linking: problem statement for a given surface-form oie triple t 1 = ("s"; "r"; "o"), the goal is to link each slot to a canonical concept in a kg (if the corresponding concept exists in the kg): "s" → e 1 ∈ e; "r" → @xmath0; "o" → e 2 ∈ e, with e and p as the (fixed) sets of kg entities and predicates. importantly, our problem definition (and consequently evaluation) focuses on linking at the fact level, where each oie slot is contextualized with the other two oie slots. 2 additionally and crucially, we want linking models that can assign an empty set to oie surface forms (e.g., "s" → ∅) when they refer to concepts not present in the kg (i.e., out-of-kg concepts). to enable a realistic setup for linking free-text oie triples to a kg, we build a benchmark which considers four different facets (see fig. 1 ). ## falb: fact linking benchmark we set up an automatic data processing pipeline to derive an oie-to-kg fact linking benchmark, which supports all four evaluation facets from §2. we refer to both the process (i.e., pipeline) and the resulting benchmark as falb. falb's input is a dataset with (gold) alignments between natural language sentences and kg facts entailed by the sentence; i.e., each data instance is (sentence, kg fact) pair. consequently, the creation of falb entails five design decisions: selection of (i) sentenceto-kg fact dataset(s), and (ii) a reference kg; (iii) generating oie triples, (iv) high-precision oie-kg fact alignments, and (v) a data augmentation strategy to increase the diversity of the data. below is an example instance of the falb dataset: example data instance from falb sentence: "m. j., who was born in brooklyn, played for the bulls." oie surface facts: @xmath1. j."; "played for"; "the bulls") and @xmath2. j."; "was born in"; "brooklyn") kg canonical fact identifiers: ## oie-to-kg fact linking models our goal is to link surface-form ("s"; "r"; "o") oie triples, to canonical knowledge graph facts (e 1 ; p; e 2 ), where e 1 , e 2 ∈ e, and @xmath3. each kg entity or predicate is represented as its surfaceform kg label (e.g., "michael jordan"), and its kg-provided description (e.g., "american basketball player and businessman"). we henceforth refer to the entities and predicates as kg entries. intuitively, since both data streams-the oie triples and the kg facts-are in natural language, we opt to obtain their representations (i.e., embeddings) with a pre-trained language model. we decouple the oie-to-kg linking in two steps: pre-ranking and re-ranking (see §4.1 and §4.2, respectively), foot_3 as is done commonly in entity retrieval @xcite and image-text matching @xcite . see fig. 2 for an overview and appendix c for implementation details. ## experiments and discussions we measure accuracy to evaluate both oie slot linking to kgs ( §5.1), and out-of-kg detection of oie slots ( §5.2). to measure oie fact linking, we score a hit if all oie slots are linked correctly. the error bars represent the standard error of the mean. ## related work prior work @xcite @xcite -based on the reverb45k dataset @xcite )-considers fact linking as inductive and polysemous for the entities, but perform inductive inference for the relations. they also assume the oie entities are linked to the kg entities a priori. in contrast, falb requires linking of all oie slots to kg entries, with multiple evaluation facets (including the out-of-kg ## conclusion we shed light on the oie to kg linking problem, allowing us to fuse the surface-form and openedended knowledge found in oies, with the canonical real-world kg facts. we introduced a novel multifaceted benchmark which fixes prior work deficiencies, and proposed a set of task-specific baselines. our experiments uncover that (i) linking inductive or polysemous oies to large kgs is challenging; (ii) we can learn oie linking methods using only synthetic data; and (iii) detecting whether oies are out-of-kg is an open research problem.