A nn IE : An Annotation Platform for Constructing Complete Open Information Extraction Benchmark (2022)

Abstract: Open Information Extraction (OIE) is the task of extracting facts from sentences in the form of relations and their corresponding arguments in schema-free manner. Intrinsic performance of OIE systems is difficult to measure due to the incompleteness of existing OIE benchmarks: ground truth extractions do not group all acceptable surface realizations of the same fact that can be extracted from a sentence. To measure performance of OIE systems more realistically, it is necessary to manually annotate complete facts (i.e., clusters of all acceptable surface realizations of the same fact) from input sentences. We propose AnnIE: an interactive annotation platform that facilitates such challenging annotation tasks and supports creation of complete fact-oriented OIE evaluation benchmarks. AnnIE is modular and flexible in order to support different use case scenarios (i.e., benchmarks covering different types of facts) and different languages. We use AnnIE to build two complete OIE benchmarks: one with verb-mediated facts and another with facts encompassing named entities. We evaluate several OIE systems on our complete benchmarks created with AnnIE. We publicly release AnnIE (and all gold datasets generated with it) under non-restrictive license.

## introduction open information extraction (oie) is the task of extracting relations and their arguments from natural language text in schema-free manner @xcite . consider the input sentence "edmund barton, who was born in australia, was a judge". without the use of a pre-specified schema, an oie system should extract the triples ("edmund barton"; "was born in"; "australia") and ("edmund barton"; "was"; "judge"). the output of oie systems is used in many downstream tasks, including open link prediction @xcite , automated knowledge base construction @xcite , question answering @xcite and text summarization @xcite . intrinsic evaluation of oie systems is done either manually @xcite or with the use of evaluation benchmarks @xcite . while manual evaluations are usually of higher quality, they are expensive and time consuming. automated benchmark evaluations are faster and more economic than manual oie evaluations @xcite , but are less reliable than human judgments of extraction correctness @xcite , because they are based on approximate token-level matching of system extractions against ground truth extractions. the main shortcoming of existing oie benchmarks is their incompleteness: they do not exhaustively list all acceptable surface realizations of the same piece of information (i.e., same fact) and, because of this, resort to unreliable scoring functions based on token-level matching between system and gold extractions @xcite . obtaining complete manual oie annotations is, however, very difficult and time-consuming. annotating a complete oie benchmark requires human annotators to write all possible combinations of extractions expressing the same fact (i.e., exhaustively list all acceptable surface realizations of the same fact; see section 3). to facilitate and speed up this process, we introduce annie, a dedicated annotation tool for constructing complete fact-oriented oie benchmark. annie facilitates the annotation process by (1) highlighting the tokens of interest (e.g., for verb-mediated extractions, it highlights the verbs, which are candidates for head words of predicates); (2) providing web-based interface for annotating triples and grouping them into fact synsets, i.e., groups of informationally equivalent extractions (section 3). to the best of our knowledge, annie is the first publicly-available annotation platform for constructing oie benchmarks. we showcase annie foot_1 by creating two complete fact-based oie benchmarks: (1) benchmark of verb-mediated facts on english, german, chinese, galician, arabic and japanese, making this gold data the first such oie resource on languages other than english; (2) benchmark for facts associating named entities (for english only). we then benchmark several state-of-the-art oie systems on these fact-based benchmarks and demonstrate that they are significantly less effective than indicated by existing oie benchmarks that use token-level scoring. we hope that annie motivates the creation of many more fact-based (as opposed to token-level) oie evaluation benchmarks. ## fact-based oie evaluation due to their incompleteness, previous benchmarks lack clarity about whether an extraction indeed represents a correct fact or not. in particular, given a system extraction, they do not assign a binary score (correct/incorrect), but rather calculate perslot token overlap scores. consider, for example, the input sentence from to address these issues, we propose moving away from such lenient token-overlap scoring and going towards fact-level exact matching. to this end, we propose an evaluation framework, dubbed benchie @xcite , for oie evaluation based on facts, not tokens. here, the annotator is instructed to exhaustively list all possible surface input sentence: "sen. mitchell is confident he has sufficient votes to block such a measure with procedural actions." carb gold extraction: ("sen. mitchell"; "is confident he has"; "sufficient votes to block ...procedural actions") ## annie: platform description annie is a web-based platform that facilitates manual annotations of fact-based oie benchmarks. in this section, we discuss: (1) the functionality of highlighting tokens of interest; (2) how annie facilitates creation of complete fact-based oie benchmarks; (3) annie's software architecture; and (4) annie's web interface and its multilingual support. ## demonstration study to showcase our tool's suitability for different oie scenarios, we generated two complete fact-based oie benchmarks using annie: (1) a benchmark for verb-mediated facts; (2) a benchmark with facts involving named entities (nes). we then evaluated several oie systems and compared their fact-based scores with the token-overlap lenient scores of the existing carb benchmark @xcite . ## conclusions exhaustively annotating all acceptable oie triples is a tedious task, but important for realistic intrinsic evaluation of oie systems. to support annotators, we introduced annie: annotation tool for constructing comprehensive evaluation benchmarks for oie. annie allows custom specification of tokens of interests (e.g., verbs) and is designed for creating fact-oriented benchmarks in which the factequivalent-yet superficially differing-extractions are grouped into fact synset. annie's lightweight architecture, easy installation and customizable components make it a practical solution for future oie annotation.