O pen F act: Factuality Enhanced Open Knowledge Extraction (2023)

Abstract: We focus on the factuality property during the extraction of an OpenIE corpus named OpenFact, which contains more than 12 million high-quality knowledge triplets. We break down the factuality property into two important aspects—expressiveness and groundedness—and we propose a comprehensive framework to handle both aspects. To enhance expressiveness, we formulate each knowledge piece in OpenFact based on a semantic frame. We also design templates, extra constraints, and adopt human efforts so that most OpenFact triplets contain enough details. For groundedness, we require the main arguments of each triplet to contain linked Wikidata1 entities. A human evaluation suggests that the OpenFact triplets are much more accurate and contain denser information compared to OPIEC-Linked (Gashteovski et al., 2019), one recent high-quality OpenIE corpus grounded to Wikidata. Further experiments on knowledge base completion and knowledge base question answering show the effectiveness of OpenFact over OPIEC-Linked as supplementary knowledge to Wikidata as the major KG.

## introduction open information extraction @xcite aims to extract factual and informative knowledge, which can further be used to enhance major knowledge bases @xcite or on downstream tasks @xcite . most current oie systems @xcite @xcite organize knowledge into subject-relation-object (sro) triples, and they use templates to extract such knowledge triples. figure 1a and 1b show a wikipedia foot_0 fragment and the corresponding sro triple from opiec-linked @xcite , a major oie corpus. previous oie efforts @xcite @xcite @xcite mainly focus on extracting sro triples that are closer to human annotated outputs on benchmarks. however, they overlook the factuality issue, where many extracted triples may convey incomplete or even incorrect information. this issue can be categorized into two main aspects. the first aspect is the lack of expressiveness, where complex knowledge is poorly organized or critical information is dropped due to the fixed sro schema and the template-based extraction system. taking figure 1b as an example, the relation (''surpassed washington monument to become'') is a long span containing multiple events, and important temporal information (''during its construction'') is not captured. the second aspect regards groundedness that measures the degree of a triplet being linked to known entities. for instance, a standard oie system can extract ''<the tower, has, three levels for visi-@xmath0.'' the extracted triple lacks groundedness particularly because it is unclear what ''the tower'' refers to. lacking factuality diminishes the usability of the oie triples as supplementary knowledge to major kgs. later work proposes to use either nested sro triples @xcite or n-ary triples @xcite @xcite to enrich sro schema with properties such as polarity. a recent effort @xcite figure 1: (a) a wikipedia fragment and (b) the corresponding opiec-linked @xcite triples. throughout this paper, we use ''[]'' to indicate the mentions of wikidata entities. (c) our open-fact fragment, where orange, gray, and blue boxes represent relations, arguments, and wikidata entities, respectively. in this paper, we propose a comprehensive solution to address the factuality issue, which is then used to construct a corpus of oie knowledge named openfact. the corpus contains more than 12 million accurate factual knowledge triplets extracted from wikipedia, a reliable knowledge source with broad coverage. comparing with previous oie corpora, factuality is thoroughly enhanced in openfact. specifically, as shown in 3 opiec-linked is obtained by applying filtering rules on the initial opiec corpus to enhance quality. figure 1c , openfact is based on semantic role labeling @xcite for expressiveness. 4 here each openfact triplet contains a predicate, which may mention a relation or an event, and several core arguments from the corresponding semantic frame. to further ensure that each openfact knowledge triplet describes detailed factual information, we require it to contain a certain combination of semantic arguments (e.g., (arg0, arg1, argm-tmp)), and we manually design multiple such combinations to ensure coverage. we also resort to some affordable human efforts to manually verify some extracted knowledge triplets, which are then used to train a quality checker to automatically filter low-quality triplets from the remaining data. to improve groundedness, we extend the core arguments (arg0, arg1, arg2, argm-loc) with associated wikidata entities to ensure that the knowledge triplet is not about some ambiguous entity (e.g., ''the tower''). as the result, our openfact triplets (e.g., ''surpassed(arg0:the [eiffel tower], arg1:the [washington monument], argm-tmp:during its construction)'') can provide more expressive and grounded information than previous oie triples (e.g., the example in figure 1b ). a human evaluation on 200 randomly sampled triplets show that 86.5% of openfact triplets convey precise factual information, while it is only 54.0% for opiec-linked. further experiments on knowledge base completion (kbc) and knowledge base question answering (kbqa) benchmarks @xcite over wikidata show that openfact provides more useful complementary information than opiec-linked and significantly improves highly competitive baseline systems. 5 in summary, we make the following contributions: ## openfact construction we choose the wikipedia dump of may 20, 2021, to construct openfact, as wikipedia has been considered to be a reliable knowledge source with broad coverage. figure 2 shows the overall pipeline, which includes wiki dump preprocessing ( §2.1), triplet extraction ( §2.2), and quality control ( §2.3). we also conduct an analysis on all the extracted openfact triplets ( §2.4). ## openfact for downstream tasks we choose knowledge base completion (kbc) and knowledge base question answering (kbqa) to evaluate the usefulness of openfact. ## experiments we evaluate openfact regarding both the quality and its usefulness on two major downstream tasks. ## related work though open information extraction has received increasing attention, there are only a few publicly available large-scale open-ie databases, such as textrunner @xcite , reverb @xcite , patty @xcite , wisenet 2.0 @xcite , @xcite , and opiec @xcite . on the other hand, nell @xcite uses a predefined seed ontology (including 123 categories and 55 relations) instead of extracting from scratch. all these open-ie corpora organize data into traditional sro or n-ary triples. some incorporate either automatic entity linking results @xcite @xcite or the natural links retained from the source data @xcite simply for alleviating the ambiguity of extracted triples. on the other hand, we focus on extracting triplets that contain enough expressiveness and groundedness to retain factuality. to do that, we adopt a comprehensive framework that imposes carefully designed restrictions and templates to filter unqualified triplets. we also resort to human annotations in descent-scale to train a quality checker for further data filtering. previous studies either rely on system confidence scores and lexical features for heuristics-based filtering @xcite or clean data to enable self-training @xcite . in addition, our triplets adopt the format of semantic frames that contains rich semantic roles. thus, our triplets can be more expressive than standard sro or n-ary triples. ## conclusion we have introduced openfact, an oie corpus of 12 million accurate factual knowledge triplets extracted from wikipedia. different from existing oie work, we focused on the factuality of extracted triplets with a comprehensive approach to enhance such property. specifically, openfact is based on frame semantics instead of the standard sro or n-ary schema, and we design automatic restriction rules and multiple dedicated templates to enhance expressiveness. besides, we extended the core arguments of each openfact triplet with linked wikidata-entity mentions for groundedness. we also resorted to a quality checker trained with descent-scale human annotations to further improve our knowledge quality. a human study revealed that 86.5% of openfact triplets are precise, while it is only 54.0% for opiec-linked @xcite , a recent high-quality oie corpus closest to ours in spirit. further experiments on knowledge base completion and knowledge base question answering show that openfact provides more useful knowledge than opiec-linked, significantly improving very strong baselines. future work includes designing dynamic templates where the combination of arguments is based on both prior knowledge and the predicate @xcite . a the data annotation guideline for quality control ( §2.3) task definition given a triplet that contains a predicate, its arguments, and their contained entity mentions, the goal is to detect whether the triplet contains enough details to convey precise factual information. if it does not, the annotator needs to point out all arguments that cause the issue. possible argument types are arg0, arg1, arg2, argm-tmp (time), argm-loc (location), and argm-neg (negation). here arg0 and arg1 can be considered as the subject and object for regular verbs, while arg2 is called beneficiary. taking ''john gave an apple to tom'' for example, the predicate is ''gave'', ''john'' is arg0 (the executor to perform ''give''), ''an apple'' is arg1 (the thing being given), and ''tom'' is arg2 (the beneficiary for this action). note that the meaning of beneficiary is not literal. for instance, the beneficiary of ''john threw a rock to tom'' is also ''tom''. for the first example in the bottom group, we should select both ''arg0'' and ''argm-tmp'' as erroneous arguments. ''arg0'' is selected because the subject of event ''visited'' is required to make the whole triplet precise. since ''arg0'' is missing, ''argm-tmp'' becomes unclear because it is unclear what ''his'' refers. for the second example in the bottom group, we need to choose ''arg1'', which should be the place of being ''ruled'', not the title of the ruler. for the third example in the bottom group, we need to choose ''arg0'', because ''he'' is not precise without knowing what this pronoun refers to.