B ench IE : A Framework for Multi-Faceted Fact-Based Open Information Extraction Evaluation (2022)

Abstract: Intrinsic evaluations of OIE systems are carried out either manually—with human evaluators judging the correctness of extractions—or automatically, on standardized benchmarks. The latter, while much more cost-effective, is less reliable, primarily because of the incompleteness of the existing OIE benchmarks: the ground truth extractions do not include all acceptable variants of the same fact, leading to unreliable assessment of the models’ performance. Moreover, the existing OIE benchmarks are available for English only. In this work, we introduce BenchIE: a benchmark and evaluation framework for comprehensive evaluation of OIE systems for English, Chinese, and German. In contrast to existing OIE benchmarks, BenchIE is fact-based, i.e., it takes into account informational equivalence of extractions: our gold standard consists of fact synsets , clusters in which we exhaustively list all acceptable surface forms of the same fact. Moreover, having in mind common downstream applications for OIE, we make BenchIE multi-faceted; i.e., we create benchmark variants that focus on different facets of OIE evaluation, e.g., compactness or minimality of extractions. We benchmark several state-of-the-art OIE systems using BenchIE and demonstrate that these systems are significantly less effective than indicated by existing OIE benchmarks. We make BenchIE (data and evaluation code) publicly available.

## introduction open information extraction (oie) is the task of extracting relations and their arguments from natural language text in a schema-free manner @xcite . consider the sentence "sen. mitchell, who is from maine, is a lawyer."; an oie system is expected to extract the triples ("sen. mitchell"; "is from"; "maine") and ("sen. mitchell"; "is"; "a lawyer") from the sentence. oie systems are used in many downstream tasks, including knowledge graph (kg) population @xcite , open link prediction @xcite , and question answering @xcite . these downstream tasks lend themselves as natural setups for extrinsic oie evaluation @xcite . while valuable in concrete applications, such extrinsic evaluations do not measure the intrinsic correctness of the extracted facts: for that purpose, several benchmarks for intrinsic oie evaluation have been proposed @xcite @xcite . automated benchmark evaluations are more feasible (i.e., faster and cheaper) than manual oie evaluations @xcite . the current benchmarks, however, use scoring functions that are based on approximate (token-level) matching of system extractions against ground truth facts, which seems to be substantially less reliable than human judgments of extraction correctness @xcite . this primarily stems from the incompleteness of existing oie benchmarks: the gold standard extractions do not include all acceptable surface realizations of the same fact. consider, for example, a sentence from the recent evaluation framework carb @xcite : "sen. mitchell is confident he has sufficient votes to block such a measure with procedural actions"; with the gold triple extraction ("sen. mitchell"; "is confident he has"; "sufficient votes to . . . procedural actions"). intuitively, a system extraction with a more concise object-("sen. mitchell"; "is confident he has"; "sufficient votes")-could also be accepted, as it still captures the same core piece of knowledge, and would arguably be valuable in most downstream tasks. to account for this, existing benchmarks credit system extractions for per-slot lexical overlap with gold extractions. such scoring is overly lenient and overestimates the systems' ability to extract correct knowledge facts. consider, e.g., a system extraction ("sen. mitchell"; "is confident he has"; "procedural actions") for the above-mentioned sentence. from the factual perspective, this extraction is clearly incorrect (sen. mitchell has votes, not actions). however, the popular carb benchmark with its token-level metrics would judge the extraction as having (1) perfect precision, since all extracted tokens can be found in corresponding slots of a gold extraction and (2) high recall, as all of the gold subject and predicate tokens as well as two gold object tokens ("procedural" and "actions") are found within corresponding slots of the system extraction ( in this work, we depart from lenient oie evaluations based on per-slot token overlaps and propose benchie, a novel fact-centric and multi-faceted oie evaluation framework and benchmark at the core of which is the following question: does the system extraction express the same fact (i.e., the same unit of knowledge) as any of the ground truth extractions (and vice versa) w.r.t. the specific aspect of the oie extraction that is of interest for one or more downstream applications? contributions. benchie advances the state of the art in oie evaluation in the following: (1) it is the first fact-centered approach to oie evaluation: to reliably answer the above question, we exhaustively list all correct extractions of the same fact. in contrast to existing benchmarks, benchie specifies complete sets of fact-equivalent extractions (dubbed fact synsets), allowing us to avoid error-prone evaluation based on token overlap measures; (2) benchie is the first multi-faceted oie benchmark, allowing to test systems for different aspects of oie extractions that may be relevant in concrete downstream applications; (3) benchie is a multilingual benchmark, covering english, chinese, and german, and to the best of our knowledge the first with manually annotated (i.e., gold standard) extractions in all languages; foot_1 (4) finally, as a fact-based and multi-faceted benchmark, benchie allows us to perform what we believe to be the most comprehensive profiling and comparative evaluation of oie systems. benchie portrays fact extraction abilities of six state-of-the-art oie models much less favorably and points to their limitations that cannot be detected with existing benchmarks. ## matching facts, not tokens most oie systems extract (subject, predicate, object) triples, with concepts as subjects and objects and verb phrases (vps) as predicates @xcite @xcite @xcite , though systems producing n-ary @xcite , nested @xcite , and noun-mediated extractions @xcite also exist. here we follow the most common practice and focus on vpmediated facts. our novel fact-based benchmark and evaluation paradigm can, however, equally be applied to other types of extractions (e.g., @xcite used this fact-based concept for oie to create gold annotations for ne-centric oie triples; i.e., triples where each argument is a named entity and the relations could be either verb phrases or noun phrases). ## multi-faceted oie benchmark different downstream applications care about different aspects of oie extractions. for ie-based text summarization and simplification @xcite , e.g., triples should be minimal overall, across all slots (i.e., without unnecessary tokens), but the exact token placement across the slots (e.g., if a preposition is in the predicate or object) does not matter. for entity linking and knowledge base population @xcite , in contrast, the token placement between slots is critical: a token that is not part of an entity, should not be placed into subject or object. acknowledging this, we create three additional variants of the english benchie, referred to as facets, each input sentence: "sen. mitchell is confident he has sufficient votes to block such a measure with procedural actions." benchie-e ("sen. mitchell" | "he"; ## profiling oie systems with benchie token-based evaluation of existing oie benchmarks (with real per-extraction scores in the range [0, 1]) makes pinpointing of extraction error source difficult. this limits their usability in automatic error analysis and system profiling. the fact that previous work performed oie error analyses manually @xcite confirms this. benchie, in contrast, lists all acceptable extractions and thus naturally lends itself to reliable automatic error analysis and profiling. ## conclusion we introduced benchie: a benchmark for more reliable fact-level evaluation of oie systems for english, chinese and german. unlike existing benchmarks, benchie takes into account fact-level equivalence of extractions: it consists of fact synsets that contain all acceptable surface forms of the same fact. further, en benchie is multi-faceted -it allows to evaluate oie extractions w.r.t. several aspects relevant in common downstream tasks. our experiments show that current benchmarks, with incomplete gold standard and approximate tokenlevel matching, drastically overestimate fact extraction abilities of oie systems. currently, the limits of benchie are its relatively small size (300 sentences v.s. carb's 1,200) and its time-consuming annotation process. a promising research direction is the investigation of trade-off between the manual effort and completeness of different oie annotation strategies. in this scenario, benchie is an ideal point of reference: it can precisely quantify the completeness of some larger (non-exhaustive) oie dataset created with limited or no manual effort.