B ench IE ^ FL : A Manually Re-Annotated Fact-Based Open Information Extraction Benchmark (2024)

Abstract: Open Information Extraction (OIE) is a field of natural language processing that aims to present textual information in a format that allows it to be organized, analyzed and reflected upon. Numerous OIE systems are developed, claiming ever-increasing performance, marking the need for objective benchmarks. BenchIE is the latest reference we know of. Despite being very well thought out, we noticed a number of issues we believe are limiting. Therefore, we propose BenchIE^FL, a new OIE benchmark which fully enforces the principles of BenchIE while containing fewer errors, omissions and shortcomings when candidate facts are matched towards reference ones. BenchIE^FL allows insightful conclusions to be drawn on the actual performance of OIE extractors.

## introduction open information extraction (oie), the task of extracting organized tuples containing information expressed in a sentence @xcite has numerous downstream applications ranging from question answering @xcite to text comprehension @xcite . earlier works evaluated oie extractors mainly by examining their output and manually determining whether the extracted tuples were expressed in a given sentence. this method lacks the capacity to measure the recall of systems, which led to the creation of oie benchmarks consisting of annotations of all possible tuples from a corpus and a matching function establishing the concordance between extracted and annotated facts. our analysis of the most recent benchmark, @xcite , shows that although very well thought out, its results are noisy and prone to biases, making its conclusion less trustworthy. contributions our main contribution in this work is the release of a new oie benchmark, benchie f l , that we created by re-annotating benchie, correcting frequent errors, inconsistencies and methodology limitations, resulting in more concise, precise and pertinent annotations. benchie f l also benefits from a new matching function that is more flexible, and which captures more valid extractions, thus producing -as we shall see -a fairer ranking of evaluated systems. in doing so, we produce a number of useful resources foot_0 , including new oie guidelines, both for sentence annotation and tuple matching. we compare seven oie extractors -neural and non neural -and show that believed state-of-the-art systems are not necessarily the best. we further demonstrate on three downstream tasks that the scores of extractors on benchie f l exhibits a much stronger correlation with their performance on each task. this paper is organized as follows. in section 2, we review some existing oie systems and benchmarks. we introduce issues with benchie in section 3, leading us to define guidelines discussed in section 4. in section 5 we characterize benchie f l , a new version of benchie we annotated by following our guidelines. experiments are reported in section 6 and we conclude in section 7. ## experiments we use the default configuration for the 7 extractors mentioned in section 2.1 without attempting to optimize them for specific benchmarks. @xcite observed that optimizing such systems do not lead to significant performance differences. ## discussion we propose new annotated resources: most notably a re-annotated oie corpora, benchie f l and a set of matching annotations, benchie f l match . we also propose new guidelines for the oie task, both for the annotation of tuples and the matching of candidate extractions. we also deliver noticeable improvements on the exact matching function, while not compromising the principles behind benchie. thanks to those resources, we conduct a meaningful comparison of off-the-shelf extractors, showing that older rule-based systems are still competitive. finally we conduct a study that shows on three downstream tasks that our benchmark better reflects the performance of oie systems on those tasks, making benchie f l the best reference to guide oie system choice, and to influence system development. there are a number of avenues worth pursuing along this work, especially in annotating more sen-tences, both to serve as training sets for new systems and in other languages.