Retrieval-Augmented Generative Question Answering for Event Argument Extraction (2022)

Abstract: Event argument extraction has long been studied as a sequential prediction problem with extractive-based methods, tackling each argument in isolation. Although recent work proposes generation-based methods to capture cross-argument dependency, they require generating and post-processing a complicated target sequence (template). Motivated by these observations and recent pretrained language models’ capabilities of learning from demonstrations. We propose a retrieval-augmented generative QA model (R-GQA) for event argument extraction. It retrieves the most similar QA pair and augments it as prompt to the current example’s context, then decodes the arguments as answers. Our approach outperforms substantially prior methods across various settings (i.e. fully supervised, domain transfer, and fewshot learning). Finally, we propose a clustering-based sampling strategy (JointEnc) and conduct a thorough analysis of how different strategies influence the few-shot learning performances.

## introduction many documents report sequences of events corresponding to common situations in the real world. arguments of different roles provide fine-grained understanding of the event (e.g. individuals, or-ganizations, locations) and also influence the determination of the event type @xcite . as compared to detecting the trigger (usually verbs) of an event, extracting arguments involve recognizing mention spans (consisting of multiple words) of various roles across sentences @xcite ). we list an example in figure 1 , given the context and the event type (nomination), all arguments for the three roles (i.e person, position, agent) should be extracted. context: one of those difficult judges [john m.] is nominated (type: nomination) by adam to be [chief justice] in 2000…. ✅ context: [greg l.] was elected (type: elect) by randy as [mayor of columbus] in 1999…. role question answers/ extractions person who is the person nominated? john m. postion what position is the person nominated for? chief justice agent who is the norminating agent? adam role question answers/ extractions person who is the person elected? greg l. postion what position is the person elected for? mayor of columbus agent who is the electing agent? randy ❌ context: [john n.] borrowed (type: transfer-money) a large amount of cash to to buy shares in 2000 …. role question answers/ extractions recipient who is recipient agent? john n. giver who is the donating agent? n/a … … … retrieved demos (from training) current example to overcome the error propagation of extractive models @xcite and efficiently capture the cross-role dependencies, end-to-end template generation-based information extraction approaches @xcite @xcite have been proposed. however, they (1) suffer from the dense output template format (fewer training instances) and cannot fully exploit semantic relations between roles with the constrained templates; (2) are unable to unleash the excellent analogical capability of large pre-trained models @xcite on similar input-output pairs to produce extraction results. based on our observations in the real circumstances, examples often bear great similarities (in terms of both syntax and semantics) with other examples (figure 1 ). in this figure , we have current input context "... difficult judges john m. is nominated ..." for a nomination event. when searching through examples in the large store (e.g. training set) for demonstrations (input-output pairs foot_1 ), the two most similar examples' input-output pairs are presented. both of the retrieved examples' contexts have large semantic similarities with the context of the current example. the first retrieved example's questions (for each role) also match the input examples'. the second example's questions do not. thus, to help the model determine "how much" to learn from the demonstrations is also important. motivated by the weaknesses of previous methods and our observations, we introduce a retrievalaugmented generative question answering model (r-gqa) for event argument extraction. firstly, our formulation for event extraction as a generative question answering task enables the model to take advantage of both question answering (exploiting label semantics) and text generation, and there's no need for threshold tuning. we conduct experiments on two settings (1) fully-supervised setting foot_2 and (2) domain transfer setting foot_3 . empirically, our method outperforms previous methods (extraction qa and template generation-based methods) substantially (contribution 1). to enable our generative model based on large pretrained model to explicitly learn ("reason") from similar demonstrations as prompt, we add to our model a retrieval component. it uses similarity/analogy score to decide how much to rely on retrieved demonstrations. it significantly outperforms the generative qa model (our proposed baseline without the retrieval component) in both settings (contribution 2). what's more, we also investigate various models' performance in the fewshot extraction setting. as far as we know, there's a large variance in terms of performance when the examples for training/evaluation are randomly sampled, causing different methods not comparable. thus (1) we investigate models' behavior in the few-shot event extraction setting on different sampling strategies (e.g. random, clustering-based) and how the model performance and distribution distance (between true data and sampled data) correspond; (2) we design a clustering-based sampling strategy (jointenc), which selects the most representative (unlabeled) examples by leveraging both context & trigger embedding. it is better than random sampling and one-round active learning. our discussions on sampling methods help improve benchmarking models' few-shot setting performance (contribution 3). ## problem and definitions event ontology, templates, and questions we focus on extracting event arguments from a sequence of words. an event consists of (1) a trigger and the type (e) of the event; based on the definitions of argument roles and the templates in the ontology, we can generate the natural questions for each argument role based on the mechanism proposed in @xcite . for example, in this example, arg 1 (transporter):"who is responsible for transport", arg 2 (passenger):"who is being transported", arg 3 (vehicle):"what is the vehicle used", arg 4 (origin):"where the transporting originated", arg 5 (destination):"where the transporting is directed" foot_4 . brown et al. (2020) proposed to use in-context demonstrations (inputoutput pairs) as prompt to test the zero-shot performance of large pretrained language models. for our retrieval-augmented approach, we denote the set of demonstrations/prompts to choose from st . in this work, we initiate st with the training set. 6 data and sampling strategy in the fullysupervised setting, we use the entire training set (1) to train the models; (2) as the demonstration store. ## methodology we first describe the retrieval-augmented generative question answering model (figure 2 ), including (1) the generation model and how to construct the demonstration (prompt) as well as the final input&target sequence; (2) training, decoding, post-processing details; and how they differ from template-generation based models. then we introduce our clustering-based sampling strategy to diversify the training examples for the few-shot setting. ## experiments and analysis we conduct experiments and compare our model to baselines in three settings on two datasets: (1) full supervision setting; domain transfer setting; as well as (3) few-shot training setting (section 4.5). ## related work event extraction and extractive&generationbased approaches traditionally, researchers have been investigating extractive approaches for event/information extraction. specifically, one branch of work use b-i-o sequence labeling based approaches using crf or structured svm models @xcite @xcite , and more recently with neural networks @xcite ). another branch of extractive approaches includes using span enumeration @xcite , as well as using question answering to encourage transfer between argument roles @xcite . recently, generation-based approaches have been proposed. among them more generally, tanl @xcite proposes to use translation-based approaches for structured prediction. more specifically, it constructs decoding targets by inserting text markers and labels around entity mentions in the input sentence. to better capture cross-entity dependencies. @xcite ; @xcite ; @xcite ; @xcite propose template-generation based approaches. they fill in the role slots in the template (e.g. sec 2) with arguments to construct the gold sequences. as compared to tanl and template generation-based methods, our r-gqa is designed to be a qa-based generative model with a simpler generation objective. plus, it augments the current example's context with the most similar demonstration in the training set as prompt. it gets the best of both worlds (i.e. question answering and generative models). ## conclusions in this work, we introduce a retrieval-augmented generative question answering framework (r-gqa) for event argument extraction. our model generates arguments (answers) for each role, conditioned on both the current input context and the analogical demonstration prompt (based on their semantic similarity). empirically, we show that r-gqa outperforms current competitive baselines with large margins in fully-supervised, crossdomain and few-shot learning settings. we conduct a thorough analysis and benchmark how different sampling strategies influence models' performance in the few-shot learning setting. we find that for event argument extraction, diversifying the examples makes the sampling distribution closer to the true distribution and contributes to models' better performance.