Probing Representations for Document-level Event Extraction (2023)

Abstract: The probing classifiers framework has been employed for interpreting deep neural network models for a variety of natural language processing (NLP) applications. Studies, however, have largely focused on sentencelevel NLP tasks. This work is the first to apply the probing paradigm to representations learned for document-level information extraction (IE). We designed eight embedding probes to analyze surface, semantic, and event-understanding capabilities relevant to document-level event extraction. We apply them to the representations acquired by learning models from three different LLM-based document-level IE approaches on a standard dataset. We found that trained encoders from these models yield embeddings that can modestly improve argument detections and labeling but only slightly enhance event-level tasks, albeit trade-offs in information helpful for coherence and event-type prediction. We further found that encoder models struggle with document length and cross-sentence discourse.

## introduction relation and event extraction (ree) focuses on identifying clusters of entities participating in a shared relation or event from unstructured text, that frequently contains a fluctuating number of such instances. while the field of information extraction (ie) started out building training and evaluation ree datasets primarily concerned with documents, researchers have been overwhelmingly focusing on sentence-level datasets @xcite . nevertheless, many ie tasks require a more comprehensive understanding that often extends to the entire input document, leading to challenges such as length and multiple events when embedding full documents. consequently, document-level datasets continue to pose challenges for even the most advanced models today @xcite . ree is considered an essential and popular task, encompassing various variations. one particularly general approach is template filling foot_0 , which can subsume certain other ie tasks by formatting. in this regard, our focus lies on templateextraction methods with an end-to-end training scheme, where texts serve as the sole input. multi-task nlp models often support and are evaluated on the task. as a result, we have seen frameworks of diverse underlying assumptions and architectures for the task. nevertheless, high-performing modern models all leverage and fine-tune on pre-trained neural contextual embedding models, like variations of bert @xcite , due to the generalized performance leap introduced by transformers and pretraining. it is crucial to understand these representations of the ie frameworks, as doing so reveals model strengths and weaknesses. however, unlike lookup style embeddings such as glove @xcite , these neural contextualized representations are inherently difficult to interpret, leading to ongoing research efforts focused on analyzing their encoded information @xcite @xcite . this work is inspired by various sentencelevel embedding interpretability works, including @xcite and @xcite . yet, to the best of our knowledge, no prior work has been done to understand the embedding of features that exist only at the document-level scale. hence, our work aims to fill this gap by investigating the factors contributing to the model performance. specifically, we analyze the impact of three key elements: contextualization of encoding, fine-tuning, and encoder and post-encoding architectures. our contributions can be summarized as follows: • we identified the necessary document-level ie understanding capabilities and created a suite of probing tasks foot_1 corresponding to each. ## probing and probing tasks the ideal learned embedding for spans should include features and patterns (or generally, "information") that independently show similarity to other span embeddings of the same entity mentions, in the same event, etc., and we set out to test if that happens for trained encoders. probing uses simplified tasks and classifiers to understand what information is encoded in the embedding of the input texts. we train a given document-level ie model (which finetunes its encoder in training), and at test time capture the output of its encoder (the document representations) before they are further used in the model-specific extraction process. we then train and run our probing tasks, each assessing an encoding capability of the encoder. drawing inspiration from many sentence-level probing works, we adopt some established setups and tasks, but with an emphasis on probing tasks pertaining to document and event understanding. category illustration task task full name surface wordct word count sentct sentence count semantic coref are coreferent isarg is an argument argtyp argument type event envttyp2 event type coenvt co-event envtct event count • • a.k.a. ? (muc, 1991) as our base dataset (details in section 3.3) to develop evaluation probing tasks. we present our probing tasks in figure 2 , this section outlines the probing tasks used for assessing the effectiveness of the learned document representations. when designing these probing tasks, our goal is to ensure that each task accurately measures a specific and narrow capability, which was often a subtask in traditional pipelined models. additionally, we want to ensure fairness and generalizability in these tasks. therefore, we avoid using event triggers in our probing tasks, especially considering that not all models use them during training. we divide our probing tasks into three cate-gories: surface information, generic semantic understanding, and event understanding. surface information these tasks assess if text embeddings encode the basic surface characteristics of the document they represent. similar to the sentence length task proposed by @xcite , we employed a word count (wordct) task and a sentence count (sentct) task, each predicts the number of words and sentences in the text respectively. labels are grouped into 10 count-based buckets and ensured a uniform distribution. semantic information these tasks go beyond surface and syntax, capturing the conveyed meaning between sentences for higher-level understanding. coreference (coref) is the binaryclassification task to determine if the embeddings of two spans of tokens ("mentions") refer to the same entity. due to the annotation of muc, all used embeddings are all known role-fillers, which are strictly necessary for the downstream document-level ie to avoid duplicates. to handle varying mention span lengths in muc, we utilize the first token's embedding for effective probing classifier training, avoiding insufficient probe training at later positions. a similar setup applies to role-filler detection (isarg), which predicts if a span embedding is an argument of any template. this task parallels argument detection in classical models. furthermore, the role classification task (argtyp) involves predicting the argument type of the role-filler span embedding. this task corresponds to the argument extraction (argumentation) step in classical pipelines. event understanding the highest level of document-level understanding is event understanding. to test the model's capability in detecting events, we used an event count task (evntct) where the probing classifier is given the full-text embedding and asked to predict the number of events that occurred in the text. we split all count labels into three buckets for class balancing. to understand how word embeddings are helpful to event deduplication or in argument linking, our co-event task (coevnt) takes two argument span embeddings and predicts whether they are arguments to the same event or different ones. additionally, the event type task (evnttyp n ) involves predicting the type of the event template based on the embeddings of n role filler first tokens. this task is similar to the classical event typing subtask, which often uses triggers as inputs. by performing this task, we can assess whether fine-tuning makes event-type information explicit. although syntactic information is commonly used in probing tasks, document-level datasets have limited syntactic annotations due to the challenges of accurately annotating details like treedepth data at scale. while the absence of these tasks is not ideal, we believe it would not significantly impact our overall analysis. ## result and analysis we present our data in ## conclusion our work pioneers the application of probing to the representation used at the document level, specifically in event extraction. we observed semantic and event-related information embedded in representations varied throughout ie training. while encoding improves on capabilities like event detection and argument labeling, training often compromises embedded coreference and event typing information. comparisons of ie frameworks uncovered that current models marginally outperformed the baseline in capturing event information at best. our analysis also suggested a potential shortcoming of encoders like bert in utilizing cross-sentence discourse information effectively. in summary, our work provides the first insights into document-level representations, suggesting new research directions for optimizing these representations for event extraction tasks.