On Aligning O pen IE Extractions with Knowledge Bases: A Case Study (2020)

Abstract: Open information extraction (OIE) is the task of extracting relations and their corresponding arguments from a natural language text in un- supervised manner. Outputs of such systems are used for downstream tasks such as ques- tion answering and automatic knowledge base (KB) construction. Many of these downstream tasks rely on aligning OIE triples with refer- ence KBs. Such alignments are usually eval- uated w.r.t. a specific downstream task and, to date, no direct manual evaluation of such alignments has been performed. In this paper, we directly evaluate how OIE triples from the OPIEC corpus are related to the DBpedia KB w.r.t. information content. First, we investigate OPIEC triples and DBpedia facts having the same arguments by comparing the information on the OIE surface relation with the KB rela- tion. Second, we evaluate the expressibility of general OPIEC triples in DBpedia. We in- vestigate whether—and, if so, how—a given OIE triple can be mapped to a single KB fact. We found that such mappings are not always possible because the information in the OIE triples tends to be more specific. Our evalua- tion suggests, however, that significant part of OIE triples can be expressed by means of KB formulas instead of individual facts.

## introduction open information extraction (oie) systems extract relations and their corresponding arguments from natural language text in unsupervised manner @xcite . consider the sentence "bell, which is a telecommunication company, is located in l. a."; an oie system may extract the triples: ("bell"; "is"; "telecommunication company") and ("bell"; "is located in"; "l. a."). such triples are used in downstream tasks, such as word embeddings generation @xcite , information retrieval @xcite and entity aspect linking @xcite . oie triples contain surface relations, which often makes their semantics ambiguous @xcite . this poses difficulties for oie triples to be used in downstream tasks @xcite . by contrast, kb relations have precise semantics and are machine-readable @xcite . to bridge this gap between oie and kbs, many methods were proposed for aligning oie triples with reference kbs. in such work, the goal is to associate an oie triple with an existing kb fact (assuming they have the same disambiguated arguments), such that both triples have the same semantics; e.g., the oie triple (jeff bezos; "be ceo of"; amazon.com) and the kb fact (jeff bezos; dbo:ceo; amazon.com). these methods are primarily used for bootstrapping oie systems @xcite , but also for other tasks such as link prediction @xcite . other methods map any oie triple to a kb schema @xcite ; e.g. the oie triple (emmanuel macron; "be president of"; france) could be mapped to (e. m.; dbo:president; france) even if this fact is not present in the reference kb. such methods are used for downstream tasks such as automatic kb construction @xcite . to date, alignments between oie triples and kbs are evaluated automatically w.r.t. downstream task. such automatic evaluations, however, do not provide insights about the information content between the alignments, which require expert manual evaluation. in this paper, we manually compare the information content of an oie corpus-opiec @xcite -and reference kb-dbpedia @xcite -under optimal alignments. foot_0 both resources are automatically generated from wikipedia, making them comparable resources for evaluation. first, we study the properties of the alignments between opiec triples and dbpedia facts which have the same argument pair. consider the oie triple t: (jeff bezos; "is ceo of"; amazon.com) and two possible kb alignments f 1 : (j. b.; dbo:ceo; amazon.com) and f 2 : (j. b.; dbo:employer; amazon.com). the fact f 1 has same semantics as the oie triple t. however, t is semantically more specific than f 2 , since it provides additional information about j. b. being employed as ceo. therefore, f 2 expresses some information in t, but not all information. in our evaluation, we consider the best possible alignment (e.g., f 1 is considered to be the best alignment) and we investigate its semantics. note that our goal is not to compare different alignment strategies. rather, we consider the best possible alignment and the goal is to investigate the limits of such alignments. we found that these alignments are usually semantically related, but quite often the open relation is more specific, thus carrying more information than the kb fact. second, we evaluate the expressibility of any opiec triple w.r.t. dbpedia by studying whether a given oie triple can be mapped to a kb fact. in this case, there might not be a known relation in dbpedia between the arguments of the opiec triple. we evaluate whether an opiec triple can be expressed with a dbpedia fact. consider the oie triple (emmanuel macron; "be president of"; france). dbpedia does not contain this fact, nevertheless, it can be fully expressed with (e. m.; dbo:president; france) and partially expressed with (e. m.; dbo:nationality; france). we found that most opiec triples can be expressed with dbpedia facts, but many of them only partially. moreover, large fraction of the partially expressible triples can be fully expressed with kb formulas; e.g. oie triple (j. f. kennedy; "be grandchild of"; p. j. kennedy) can be partially expressed with the kb fact (j. f. k.; dbo:relative; p. j. k.) and fully expressed with kb formula ∃x: (j. f. k.; dbo:parent; x) ∧ (x; dbo:parent; p. j. k.). our evaluations focus on the opiec corpus, which was extracted with the oie system minie. this makes the evaluation focused on one particular oie system. to gain insight into transferability, we studied how the findings of our evaluations transfer to oie triples produced by other oie systems. we found that the results generally transfer over, though some oie systems tend to produce more specific output. ## analysis of opiec triples and dbpedia facts with same arguments in this section, we evaluate the semantics of alignments between opiec triples and dbpedia facts with the same arguments. such alignments are inspired by the distant supervision assumption (dsa), which is originally used for traditional information extraction tasks @xcite . the dsa asserts that whenever there is a kb fact and a sentence mentioning the entity pair of the kb fact, then that sentence expresses the information contained in the kb fact. similarly, the dsa within oie context asserts that whenever there is an oie triple for which there is a kb fact having the same arguments, then the oie triple expresses the information of the kb fact. dsa is key assumption used for bootstrapping oie extractors @xcite . by using the dsa, some methods bootstrap a training set that is used either for learning oie extraction rules @xcite or learning a neural model for extracting oie triples @xcite . @xcite used wikipedia infoboxes (via dbpedia) as source for distant supervision: if there is wikipedia sentence that contains an entity pair and a corresponding dbpedia entry with the same entity pair, then they store the syntactic patterns (e.g., shortest path in the dependency parse tree) between the two entities. these syntactic patterns are used for learning oie extraction rules. the assumption is that the kb relation and the syntactic pattern instance (i.e., the open relation) express same information. other oie methods exploit the dsa similarly, including ollie @xcite , renoun, @xcite , nestie @xcite , bonie @xcite and openceres @xcite . ## expressibility of opiec triples with dbpedia in the previous section, we evaluated the limits of aligning oie triples from opiec for which kb facts exist in dbpedia. such cases, however, comprise only 1 4 of the data. in this section, we evaluate all cases: the limits of aligning any opiec triple with a dbpedia fact. our goal is to answer the question of whether any opiec triple contains information which is relevant for dbpedia and, if so, how can it be expressed with the kb. we measured relevance by quantifying the information that can be expressed with kb language and we study how can such information be expressed. ## transferability in this section, we study whether and to what extent the results of the evaluations transfer to other oie systems. we used three other popular oie systems: stanford oie @xcite , rnnoie @xcite and openie 5 @xcite . we ran these oie systems on the provenance sentences of the sampled triples used in our evaluations-explained in sec. 2 and sec. 3-and compared their outputs to their opiec counterparts (extracted with minie @xcite ). consider the opiec triple (turf buccaneers; "be album by"; mac dre). we use the provenance sentence from which this triple was extracted and we run the other oie systems on the same sentence. then, we select the triples that match the argument pair of the opiec triple; i.e. (turf buccaneers, mac dre) in the example. as before, we keep only the triples that are correctly extracted. then, an anno- ## main findings and conclusions in this paper, we evaluated how oie triples from the opiec corpus are related to the dbpedia kb w.r.t. information content. both resources are automatically generated from same domain-opiec from the textual data and dbpedia from the semistructured data of wikipedia-which makes them compatible resources for evaluation. first, we evaluated the semantic relatedness between oie triples from opiec and dbpedia facts having the same arguments; i.e. the distant supervision assumption (dsa). such cases are important for downstream tasks and for bootstrapping oie systems. in general, we found that such opiec triples are semantically related to the dbpedia facts, but quite often the oie triples are more specific, thus capturing more information than the kb facts. second, we evaluated the expressibility of any opiec triple w.r.t. dbpedia: whether (and how) an opiec triple can be expressed with a single dbpedia fact; i.e. the one triple assumption (ota). ota is an implicit assumption used in tasks such as slot filling. we found that expressing an opiec triple with a single dbpedia fact is often limited, and that the use of kb formulas improves the expressibility of oie triples with kb language significantly. third, we found that most opiec triples contain information which is not present in dbpedia, thus showing the potential of oie triples for tasks such as kb population. finally, we found that the findings of our case study-which was based on the oie system minie-are likely to transfer over to other oie systems. our study suggests that most information found in oie triples is not present in the kb. one way to harvest such knowledge is to add oie triples to a kb with universal schema @xcite . we leave such evaluations for future work.