Automatic Error Analysis for Document-level Information Extraction (2022)

Abstract: Document-level information extraction (IE) tasks have recently begun to be revisited in earnest using the end-to-end neural network techniques that have been successful on their sentence-level IE counterparts. Evaluation of the approaches, however, has been limited in a number of dimensions. In particular, the precision/recall/F1 scores typically reported provide few insights on the range of errors the models make. We build on the work of Kummerfeld and Klein (2013) to propose a transformation-based framework for automating error analysis in document-level event and (N-ary) relation extraction. We employ our framework to compare two state-of-the-art document-level template-filling approaches on datasets from three domains; and then, to gauge progress in IE since its inception 30 years ago, vs. four systems from the MUC-4 (1992) evaluation.

## introduction although information extraction (ie) research has almost uniformly focused on sentence-level relation and event extraction @xcite , the earliest research in the area formulated the task at the document level. consider, for example, the first large-scale (for the time) evaluations of ie systems -e.g. @xcite and @xcite . each involved a complex document-level event extraction task: there were 24 types of events, over a dozen event arguments (or roles) to be identified for each event; documents could contain zero to tens of events, and extracting argument entities (or role fillers) required noun phrase coreference resolution to ensure interpretability for the end-user (e.g. to ensure that multiple distinct mentions of the same entity in the output were not misinterpreted as references to distinct entities). the task was challenging: information relevant for a single event could be scattered across the document or repeated in multiple places; relevant information might need to be shared across multiple events; information regarding different events could be intermingled. in figure 1 , for example, the disease "newcastle" is mentioned well before its associated event is mentioned (via the triggering phrase "the disease has killed"); that same mention of "newcastle" must again be recognized as the disease in a second event; and the country of the first event ("honduras") appears only in the sentence describing the second event. in fact, the problem of document-level information extraction has only recently begun to be revisited @xcite @xcite @xcite in part in an attempt to test the power of end-to-end neural network techniques that have been so successful on their sentence-level counterparts. 2 evaluation, however, has been limited in a number of ways. first, despite the relative complexity of the task, approaches are only evaluated with respect to their overall performance scores (e.g. precision, recall, and f1). even though scores at the role level are sometimes included, no systematic analysis or characterization of the types of errors that occur is typically done. the latter is needed to determine strategies to improve performance, to obtain more informative cross-system and cross-genre comparisons, and to identify and track broader advances in the field as the underlying approaches evolve. to date, for example, there has been no attempt to directly compare the error landscape and distribution of input document country honduras disease newcastle victims close to half a million honduran chickens status confirmed country guatemala disease newcastle victims status confirmed country newcastle (x) disease (x) victims close to half a million honduran chickens status confirmed desired template(s) predicted template(s) our automatic error analysis system missing template(s) 1 ... ... error statistics the agriculture ministers of el salvador and honduras ... to control the spread of disease affecting poultry, like the virus newcastle[disease]. urrutia ... to study the newcastle outbreak. the disease has killed close to half a million honduran chickens[victims] in recent weeks. honduras[country] said this week it would halt the importation of chickens and eggs from guatemala[country], where the disease has been detected earlier, and ..... newly developed neural ie methods with that of the largely hand-crafted systems of the 1990s. in this work, we first introduce a framework for automating error analysis for document-level event and relation extraction, casting both as instances of a general role-filling, or template-filling task (jurafsky and martin, 2021). our approach converts predicted system outputs into their gold standard counterparts through a series of template-level transformations (figure 2 ) and then maps combinations of transformations into a collection of ie-based error types. examples of errors include duplicates, missing and spurious role fillers, missing and spurious templates, and incorrect role and template assignments for fillers. (see figure 3 for the full set). next, we employ the error analysis framework in a comparison of two state-of-the-art documentlevel neural template-filling approaches, dygie++ @xcite and gtt @xcite , across three template-filling datasets (scirex, promed @xcite foot_1 , and muc-4). finally, in an attempt to gauge progress in the information extraction field over the past 30 years, we employ the framework to compare the performance of four of the original muc-4 systems with the two newer deep-learning approaches to documentlevel ie. 4 we find that (1) the best of the early ie models -which strikes a better balance between precision and recall -outperforms modern models that exhibit much higher precision and much lower recall; (2) the modern neural models make more mistakes on scientific vs. news-oriented texts, and missing role fillers is universally the largest source of errors; and (3) modern models have clear advantages over the early ie systems in terms of accurate span extraction, while the early systems make fewer mistakes assigning role fillers to their roles. ## related work aside from the original muc-4 evaluation scoring reports @xcite , which included counts of missing and spurious role filler errors, there have been very few attempts at understanding the types of errors made by ie systems and grounding those errors linguistically. @xcite proposed a framework for studying how different errors propagate through an ie system; however, the framework can only be used for pipelined systems, not end-to-end ones. on the other hand, automated error analysis with linguistically motivated error types has been used in other sub-fields of nlp such as machinetranslation @xcite @xcite @xcite , coreference resolution @xcite @xcite and parsing @xcite . recently, generalized automated error analysis frameworks involving human-in-the-loop testing like errudite @xcite , check-list (ribeiro et al., 2020), crosscheck @xcite , and allennlp interpret @xcite have successfully been applied to tasks like machine comprehension and relation extraction @xcite . closest to our work are @xcite and @xcite , which use model-agnostic transformationbased mapping approaches to automatically obtain error information in the predicted structured output. ## template-filling task specification and evaluation as in jurafsky and martin (2021), we will refer to document-level information extraction tasks as template-filling tasks and use the term going forward to refer to both event extraction and documentlevel relation extraction tasks. given a document, d, and an ie template specification consisting of a predetermined list of roles r 1 , r 2 , ..., r i associated with each type of relevant event for the task of interest, the goal for template filling is to extract from d, one output template, t for every relevant event/relation e 1 , e 2 , . . . , e n present in the document. notably, in the general case, @xmath0. in each output template, its roles are filled with the corresponding role filler(s), which can be inferred or extracted from the document depending on the predetermined role types. we consider two role types here: 5 set-fill roles, which must be filled with exactly one role filler from a finite set supplied in the template specification. an example of a set-fill role in figure 1 is status, which can be confirmed, possible, or suspected. string-fill roles, whose role filler(s) are spans extracted from the document, or left empty if no corresponding role filler is found in the document. victims, disease and country are string-fill roles in figure 1 . some string-fill roles allow multiple fillers; for example, there might be more than one victims. importantly, for document-level template filling, exactly one string should be included for each role filler entity (typically a canonical mention of the entity), i.e. coreferent mentions of the same entity are not permitted. evaluation. we use the standard (exact-match) f1 score @xcite to evaluate the output 5 there are potentially more role types depending on the dataset (e.g. normalized dates, times, locations); we will not consider those here. ## methodology: automatic transformations for error analysis similar to the work of @xcite , our error analysis approach is systemagnostic, i.e. it only uses system output and does not consider intermediate system decisions. this allows for error analysis and comparison across different kinds of systems -end-to-end or pipeline; neural or pattern-based. given inputs consisting of the system-predicted templates and gold standard templates (i.e. desired output) for every document in the target dataset, our error analysis tool operates in three steps. for each document, 1. perform an optimized mapping of the associated predicted templates and gold templates. ## document-level ie datasets our experiments employ three document-level information extraction datasets. we briefly describe each below. dataset statistics are summarized in muc-4 (muc-4, 1992) consists of newswire describing terrorist incidents in latin america provided by the fbis (federal broadcast information services). we converted the optional templates to required templates and removed the subtypes of the incidents as done in previous work @xcite so that the dataset is transformed into standardized templates. the roles chosen from the muc-4 dataset are perpind (individual perpetrator), perporg (organization perpetrator), target (physical target), victim (human target), and weapon which are all string-fill roles, as well as incident type which is a set-fill role with six possible role fillers: attack, kidnapping, bombing, arson, robbery, and forced work stoppage. as seen in promed foot_4 @xcite ) consists of just 125 annotated tuning examples and 120 annotated test examples, describing global disease outbreaks by subject matter experts from promed. we use the tuning data as training data and reserve 10% of the test data, i.e. 12 examples, to create a development/validation set. 19.83% of the documents in the dataset have no templates. the roles that we extract from the dataset are status, country, disease, and victims. disease, victims, and country are string-fill roles foot_5 ; status is a set-fill role with confirmed, possible, and suspected as the possible role filler options. scirex @xcite consists of annotated computer science articles from papers with code foot_6 . we focus specifically on its 4-ary relation extraction subtask. the roles present in each relation are material (dataset), metric, task, and method which are all string-fills. we convert the dataset from its original format to templates for our models, and remove individual role fillers (entities) that have no mentions in the text. 11 we also remove any duplicate templates. 12 during preprocessing, we remove malformed words longer than 25 characters, as the majority of these consist of concatenated words that are not present in the corresponding text. ## ie modeling details in our experiments, we train and test two neuralbased ie models, described briefly below, on the muc-4, promed, and scirex datasets. note that to create the training data for both the dygie++ and gtt models, we use the first mention of each role filler in the document as the mention to be extracted. ## experimental results and analysis we first discuss the results of dygie++ and gtt on scirex, promed, and muc-4; and then examine the performance of these newer neural models on the 1992 muc-4 dataset vs. a few of the bestperforming ie systems at the time. ## limitations and future work this work explores subtypes of spurious role filler errors extensively, however, we would like to further analyze missing role filler and templatelevel errors for more fine-grained error subtypes and the linguistic reasons behind why they occur. due to the pairwise comparisons between all predicted and gold mentions in a role for all pairs of predicted and gold templates in an example, the error analysis tool is slow when the number of both the predicted and gold templates as well as the number of role fillers in the templates is high. thus, we would also like to improve the time complexity of our template (and mention) matching algorithms using an approach like bipartite matching @xcite . currently, the error analysis tool reports exact match precision/recall/f1 which is more suitable for string-fill roles. we would like to extend the tool to further analyze set-fill roles by implementing metrics such as false-positive rate. we used a limited number of models in this paper as we aimed to develop and test the usability of our error analysis tool. in the future, however, we would like to test our tool on a wider range of models, in addition to running more experiments in order to reach more generalizable conclusions about the behavior of ie models. ## conclusion as new models for information extraction continue to be developed, we find that their predicted error types contain insights regarding their shortcomings. analyzing error patterns within model predictions in a more fine-grained manner beyond scores provided by commonly used metrics is important for the progress of the field. we introduce a framework for the automatic categorization of model prediction errors for document-level ie tasks. we used the tool to analyze the errors of two state-of-theart models on three datasets from varying domains and compared the error profiles of these models to four of the earliest systems in the field on a dataset from that era. we find that state-of-the-art models, when compared to the earlier manual feature-based models, perform better at span extraction but worse at template detection and role assignment. with a better balance between precision and recall, the best early model outperforms the relatively highprecision, low-recall modern models. missing role fillers remain the main source of errors, and scientific corpora are the most difficult for all systems, suggesting that improvements in these areas should be a priority for future system development.