A Comprehensive Survey on Document-Level Information Extraction (2024)

Abstract: Document-level information extraction (doc-IE) plays a pivotal role in the realm of natural language processing (NLP). This paper embarks on a comprehensive review and discussion of contemporary literature related to doc-IE. In addition, we conduct a thorough error analysis using state-of-the-art algorithms, shedding light on their limitations and remaining challenges for tackling the task of doc-IE. Our findings demonstrate that issues like entity coreference resolution and the lack of robust reasoning significantly hinder the effectiveness of document-level information extraction (doc-IE). Additionally, we uncover new challenges, including labeling noise and relation transitivity. The overarching objective of this survey paper is to provide valuable insights that can empower NLP researchers to further advance the performance of doc-IE.

## introduction natural language processing (nlp) triggers the present wave of artificial intelligence @xcite @xcite . information extraction (ie) plays a vital role in all aspects of nlp by extracting structured information from unstructured texts @xcite . document-level information extraction (doc-ie) has achieved significant progress, benefiting from the enormous data resources created by nlp researchers and the rapidly growing computational power resources @xcite . however, several challenges persist within the realm of doc-ie research, such as entity coreference resolution, reasoning across long-span contexts, and lack of commonsense reasoning as shown in figure 1 . furthermore, current doc-ie research predominantly focuses on restricted domains and languages @xcite @xcite , which poses difficulties in fairly conducting model comparisons and hampers the generalizability of findings. to gain a profound understanding of the current literature on doc-ie, we conduct a compre-hensive survey of recent models and datasets for document-level relation extraction (doc-re) and document-level event extraction (doc-ee), focusing on those published in top nlp conferences such as acl, emnlp, and so on. these works span various languages and domains, providing a broad overview of advancements in the field. we also thoroughly analyze the errors of several stateof-the-art approaches and summarize several key remaining challenges and future research directions of doc-ie. the contributions of this survey paper include: event extraction event extraction @xcite @xcite is a task to identify and classify event triggers and relevant participants from natural language text. formally, given a document consisting of a set of sentences where each sentence consists of a sequence of words, the objective of this task is to identify and extract the following components from a given document: event mention, which refers to the phrases or sentences denoting an event; event trigger, typically in the form of a verb that signals the occurrence of an event; event type, indicating the predefined type of event specified by the dataset, such as conflict-attack; argument mention, comprising entity mentions that provide additional details on the event, such as who, what, when, where, and how the event occurred; and finally, argument role, representing the role or type of argument associated with the entity. document : wiki_drone_strikes_0_news_1 … [s6]: that figure does not include [deaths] in active battlefields including afghanistanwhere us air [attacks] have shot up since obama withdrew the majority of his troops at the end of 2014 . the country has since come under frequent us [bombardment] , in an unreported war that saw 1 , 337 weapons dropped last year alone -a 40 % rise on 2015. argument role target place attacker explosive device event: detonate explode role argument attacker us target country explosive device weapons document : skai tv [s1]: skai @xmath0. … [s3]: it was relaunched in its present form on 1st of april 2006 <@xmath1, and gradually spread its coverage nationwide. … [s5]: skai @xmath2, a consortium of private television networks introducing digital terrestrial transmission in @xmath3. relation @xmath4: country evidence: [s1, s3, s5] doc-ee task example (wikievents) doc-re task example (docred) ## methods the fundamental challenge in doc-ee and doc-re is to express document content in a concise and ## discussion to understand the limitations and remaining challenges of the current document-level ie approaches, we evaluate three state-of-the-art doc-re methods, kd @xcite , drn @xcite , and sais @xcite , on the docred and re-docred datasets.similarly, we also evaluate two state-of-the-art doc-ee methods, graph-based model tsar @xcite and generative model ea2e @xcite , on the wikievents dataset, and another two doc-ee methods, graph-based model ptpcg @xcite and task-specific model redee @xcite , on chfinann dataset. for each work, we randomly select 50 errors and examine the cause of them. we finally conclude seven major types of errors for document-level information extraction. figure 2 , 3, 4 show the distribution of the seven types of errors on each dataset and long-span document contains multiple sentences in a long span. this error happens when the model fails to capture the full context of a document or uses global information for inference. commonsense knowledge the error occurs when models fail to correctly extract relations or events or assume the wrong semantics due to a lack of commonsense and background knowledge, which humans are able to learn or understand instinctively. many datasets are specific to some domains, and in the absence of relevant background and domain-specific knowledge, models may inaccurately reason or misinterpret information. relation transitivity error documents tend to have many entities appearing in the same sentence or across sentences. relation transitivity errors occur when a model fails to correctly infer a relation between two entities based on their individual relations with a third entity. additionally, not all relations are transitive, thus the model should correctly recognize when transitivity applies. over prediction error this error type refers to the spurious error (as we presented in in addition to shared error types with doc-re, we observe two more types of errors based on the wikievents and chfinann datasets. multi-events error in doc-ee tasks, documents contain multiple events that overlap or occur simultaneously, which requires the model to have sufficient training or advanced techniques to learn the inherent complexity of multi-event documents. in an event-trigger-annotated dataset such as wikievents, the model can fail at assigning arguments to the correct events or matching roles to arguments. in a trigger-not-annotated dataset like chfinann, event detection errors may occur when models try to identify and differentiate distinct events within the document due to the complex contextual structure of each event, as shown in the example of figure 5 . span errors models face span error types mainly associated with previous tasks like entity recognition or caused by the different linguistic features and complexities of datasets. for example, nominal mention recognition and argument span mismatch errors are common in many works, particularly in generative methods. noisy data this issue comprises natural language noises and labeling noises. real-world documents contain noisy, unstructured, or poorly formatted content, causing difficulties in identifying entities and extracting relations. see further discussion in section c of the appendix. ## remaining challenges current difficulties can be broadly categorized into three areas: information spread out, multiple mentions and multiple entity pairs throughout the entire document, some information must be deduced from several sentences or transferred by other relations in order to be discovered. the first two issues have been addressed by existing approaches using attention mechanisms and graph networks, though multiple-step reasoning is less widely focused. existing methods rely on llms to learn syntactic features while neglecting the relation transitivity between entity pairs and the evidence trace of reasoning. progressively, more methods try to use evidence sentences or evidence paths to infer complicated relations. models continue to struggle with capturing commonsense and knowledge-based information as it is difficult to from the training data. previous works have tried adaptive losses for balancing the positive and negative examples to allevi- ## conclusion we conducted a thorough error analysis of current state-of-the-art algorithms, highlighting the limitations of existing approaches and identifying key challenges in document-level ie. our analysis revealed that issues such as entity coreference resolution, insufficient reasoning capabilities, labeling noise, and relation transitivity significantly impact the performance of current models, providing insights for future research. despite notable progress in the field, we conclude that persistent challenges within both datasets and models hinder the development of robust and generalizable solutions. overcoming these obstacles will be essential for advancing document-level ie models in the future.