Contextualized Soft Prompts for Extraction of Event Arguments (2023)

Abstract: Event argument extraction (EAE) is a sub-task of event extraction where the goal is to identify roles of entity mentions for events in text. The current state-of-the-art approaches for this problem explore prompt-based methods to prompt pre-trained language models for arguments over input context. However, existing prompt-based methods mainly rely on discrete and manually-designed prompts that cannot exploit specific context for each example to improve customization for optimal performance. In addition, the discrete nature of current prompts prevents the incorporation of relevant context from multiple external documents to enrich prompts for EAE. To this end, we propose a novel prompt-based method for EAE that introduces soft prompts to facilitate the encoding of individual example context and multiple relevant documents to boost EAE. We extensively evaluate the proposed method on benchmark datasets for EAE to demonstrate its benefits with state-of-the-art performance.

## introduction as an important task in information extraction (ie), event argument extraction (eae) aims to recognize event arguments and roles for given event mentions in text. for example, in the text "on the morning of 1 march 2019, taliban gunmen and suicide bombers attacked camp shorabak." with the event trigger "attacked" of type conflict.attack, the goal of eae systems is to identify "gunmen" and "bombers as the attacker argument, and "camp shorabak" as the target. along with event detection, eae has important applications for different natural language processing (nlp) tasks. eae research progress has been accelerated by deep learning architectures to significantly boost extraction performance. early deep learning models for eae have followed the traditional approaches to formulate eae as classification or sequence labeling problems @xcite @xcite @xcite . recently, there has been a growing interest in solving eae in the new question answering or text generation frameworks to better exploit task-specific information (e.g., labels/descriptions of argument roles) via prompts for pre-trained language models (plm). as such, question answering methods for eae create a question for each argument role to perform span extraction over input context @xcite @xcite ) while text generation models directly consume an input text and argument-specified prompt/template to generate arguments for each event mention @xcite . however, a common issue in current prompt-based methods for eae involves the use of discrete and manually-designed prompts to present task information for the models, e.g., event types and argument roles. as such, these prompts often follow some pre-defined templates @xcite that are applied to extract arguments for all events in text. while convenient for human understanding, discrete and pre-defined prompts might not be ideal for all examples, causing sub-optimal performance @xcite . the discrete nature also makes it challenging to achieve prompt customization for each example in eae models. further, due to the employment of plms, it has been observed that model performance can be very sensitive to specific formulations of discrete prompts @xcite , leading to instability and less reliability when adapting to different datasets. another issue of hard prompts for eae models concerns other relevant examples from training data that can provide helpful information to support argument prediction for current input text and event type. as such, a few recent work has retrieved related examples for an input text to combine with hard prompts to improve eae @xcite . however, due to the input length limit of plms, the number of relevant examples in the prompts is also constrained, thus unable to fully leverage their advantages to boost performance. to this end, our work proposes a novel promptbased method for eae where learnable soft prompts are explicitly introduced to enable prompt customization for examples, stability improvement, and incorporation of relevant example context. in particular, based on the architecture of generative plms, our model directly utilizes input example representations to compute soft prompts for eae, thus allowing the prompts to be specifically designed for each example for better customization. in addition, soft prompts facilitate the accumulation of representations of relevant examples for an input event type to consume more examples for richer prompts. to exploit this flexibility of soft prompts, our model extensively considers relevant examples as the texts in training data that contain similar event types to an input text, leading to comprehensive external event context to aid eae. accordingly, we introduce a graph structure to capture mentioning relations between documents and event types. this graph is then fed into graph neural networks to facilitate representation aggregation of relevant documents with similar events for soft prompt computation. we evaluate the proposed model for eae on the benchmark datasets rams and wikievents. the results demonstrate the benefits of the proposed method, leading to stateof-the-art performance for eae. ## model in eae, given an event trigger/mention in a document, we need to identify argument spans and roles for the event. for convenience, let @xmath0, . . . , d |d| } be the set of documents in the training data and e @xmath1. relevant context aggregation: our eae model follows the prompt-based framework @xcite where a prompt is created for each event type and fed into the pre-trained language model bart @xcite to perform span extraction for the argument roles. as such, in contrast to hard and manually-designed prompts as in previous work, our model introduces soft prompts with example customization and relevant example aggregation to boost the performance. in particular, given the current event trigger e @xmath2, our model first aims to aggregate context representations from relevant documents in d for the event type et of e k to enrich soft prompt computation. motivated by relevant documents via similar event types, we we first construct an event-type mentioning graph g between the documents in d and the event types @xmath3, . . . , t |t | } to facilitate representation aggregation. in particular, the node set v for our graph involves both documents and event types, i.e., @xmath4. we only connect a document d @xmath5. in this way, we can link the documents in d via similar event type mentioning for convenient representation aggregation with graph neural networks. to obtain representations for each document d @xmath6, we introduce the @xmath7. the augmented document is then sent into the encoder of bart to produce a representation for each word in du (using the averages of hidden vectors in the last layer for sub-words). afterward, the representation d 0 u for d @xmath8, aiming to retain event-focused context in the representation. for the event types t @xmath9, we initialize their representations t 0 v randomly. afterward, the graph g and representations d 0 u and t 0 v for documents and event types are consumed by a graph attention network @xcite to aggregate the representations via the connections in g, producing richer representations d l u and t l v for d u and t v after l layers of transformation. consequently, we treat the induced representation et for the current event type et as the aggregation for context information of relevant documents for prompt computation for e @xmath10. soft prompt computation: for convenience, let e k be the representation for the event trigger e k after the marker-augmented document di for d i is encoded by the bart encoder in the previous step. as such, our soft prompt for eae for trigger e @xmath11, thus allowing p sof t to be integrated into the computation of bart. to achieve a customized soft prompt p sof t for e k in our model, the contextual representation e k for e k will be utilized to compute p sof t . in addition, as discussed above, p sof t will also be conditioned on the aggregation of relevant context representations et for the the event type et of e k to enrich the prompt and facilitate argument extraction. to this end, we utilize a learnable feed-forward network f f to transform the concatenation e k and t l k into a vector of size m s × d. this vector will then be reshaped to form our soft prompt p sof @xmath12; et])). prompt-based eae: while soft prompts enable example customization and relevant context incorporation for the models, we further inherit the hard prompts to explicitly specify expected argument roles for span extraction. in particular, to achieve a fair comparison, we utilize the same hard prompt for each event type as in previous work @xcite ) that connects all argument roles with natural language. for example, for the event type life.consume.unspecified, the hard prompt to indicate argument roles is: <con-@xmath13. for each event type t, we send its hard prompt into the embedding layer of bart to obtain a representation for each word (i.e., using averages of embeddings for sub-tokens), leading to the hard prompt representation p t hard of size m t h × d (m t h is the length of the hard prompt for t). we then concatenate the soft and hard prompt representations to create a single prompt p r for eae with bart, i.e., p r @xmath14, p sof t ] of size (m s +m t h )×d. next, we follow the prompt-for-extraction framework in @xcite to use the bart encoder to encode input context for d i while the prompt p t will be employed to prompt the bart decoder for span extraction. given the current trigger e @xmath15, we first inject the trigger @xmath16 afterward, for the j-th argument role for event type t, we obtain a role representation ϕ t j by mean-pooling its corresponding sub-token representations in the prompt representation p r t . similar to @xcite , we employ two selection heads s start and s end (of d dimensions) to compute start and end span selectors ϕ t,start @xmath17,end @xmath18. each span selector tuple θ t @xmath19,start j , ϕ t,end j ) then aims to select at most one argument span for the j-th role of t. here, the golden span for this role is denoted by (a t,start j , a t,end j ). it will be set to (0, 0) if event e k does not have an argument of this role in d i . as such, the extractive prompt framework is utilized to estimate distributions over token positions in d for how likely each token in d i would serve as the start/end position for the argument span of the role: finally, to train our model, we optimize the loss: @xmath20,start j (a t,start j )) + log(p t,end j (a t,end j ))) (i.e., over all events in d). inference: at inference time, given an input text, event type, and argument role, we consider all possible argument spans for the role, ensuring that the start indexes are smaller than the end indexes (including (0, 0)) and their lengths do not exceed a maximum value computed over training data. a score for each span is obtained using the probability log(p t,start j (a t,start j )) + log(p t,end j (a t,end j ). the span with the highest score will be chosen for prediction. finally, the aggregations et of relevant context representations for event types, which are learned during training, are used in test time. comparison: we compare our method (called speae for soft prompts for eae) with the state-ofthe-art models for eae. in particular, we consider two groups of baselines: (i) text generation-based models: bart-gen @xcite , and (ii) question answering-based models: feae @xcite , docmrc @xcite , eeqa @xcite , eeqa-bart (du and cardie, 2020), ea2e @xcite , and paie @xcite . the performance of ea2e is obtained by running the provided code over our preprocessed data using the same evaluation metrics for a fair comparison. the performance for other baselines is inherited from @xcite , which presents the model paie with current best-reported results for our datasets. ## related work multiple methods have been introduced to solve eae, including early feature-based methods @xcite and recent deep learning models @xcite @xcite @xcite . while most previous methods have focused on the classification frameworks @xcite @xcite , plms has enabled recent formulation of eae via question answering @xcite @xcite @xcite or text generation @xcite @xcite paradigms. at the core of such models involves questions/prompts to specify argument roles to prompt plms. however, the questions/prompts in previous methods are mainly discrete and hand-designed, making it hard to customize for each example and incorporate various relevant context. ## conclusion we introduce a new prompt-based method for eae that features soft prompts to achieve example customization and relevant context augmentation to enrich prompts. extensive experiments demonstrate the advantages of the proposed method for eae. in the future, we will explore soft prompts for other problems in ie for better understanding.