Towards Better Question Generation in QA -based Event Extraction (2024)

Abstract: Event Extraction (EE) is an essential information extraction task that aims to extract event-related information from unstructured texts.The paradigm of this task has shifted from conventional classification-based methods to more contemporary question-answering-based (QA-based) approaches. However, in QA-based EE, the quality of the questions dramatically affects the extraction accuracy, and how to generate high-quality questions for QA-based EE remains a challenge. In this work, to tackle this challenge, we suggest four criteria to evaluate the quality of a question and propose a reinforcement learning method, RLQG, for QA-based EE that can generate generalizable, high-quality, and context-dependent questions and provides clear guidance to QA models. The extensive experiments conducted on ACE and RAMS datasets have strongly validated our approach’s effectiveness, which also demonstrates its robustness in scenarios with limited training data. The corresponding code of RLQG is released for further research.

## introduction event extraction (ee), an essential information extraction task, aims to extract event-related information (often called event arguments) from a given context. recently, the paradigm of ee has shifted from conventional classification-based approaches @xcite @xcite to question-answering-based (qabased) methods @xcite @xcite @xcite . for example, given a sentence: "marines were involved in a firefight in the center of baghdad", to extract the role attacker, the method first generates the question "who is the attacker in firefight?" and then uses a question-answering (qa) model to obtain an answer as the corresponding result (as context: marines !""!#$%& were involved in a firefight in the center of baghdad. they used rocketpropelled grenades and semi-automatic weapons, causing some injuries. template q1: who is the attacker in firefight? a1. none ✗ template q2: who was the attacking agent? a2. weapons ✗ human-written q3: who was involved in the firefight in the center of baghdad using grenades and weapons? a3. marines ✓ shown in figure 1 ). this paradigm has demonstrated great success in various scenarios/domains. nevertheless, one of the biggest challenges in qa-based ee is obtaining "good questions" to guide the qa model, and the quality of the questions often significantly affects the results. previous methods typically use well-designed templates to generate such questions @xcite @xcite , which, however, often lead to rigid and less context-dependent questions. for example, figure 1 gives two questions, q1 and q2, based on templates, leading to incorrect answers. in our pilot experiments, we show that template-based questions lead to about 60% errors even using a powerful proprietary qa model like @xcite , as shown in figure 4 . in this paper, we explore effective methods towards generating better questions for qa-based ee. first, we propose four criteria for a good question: 1) fluency: a question should be fluent in natural languages to be well addressed by a qa model. 2) generalizability: methods for question generation should apply to unseen contexts and roles beyond those in training @xcite . 3) context dependence: questions should be consistent with the context and include necessary information to be correctly addressed by a qa model. 4) indicative guidance for qa models: questions should offer clear guidance for qa models to find answers @xcite . we then seek to build a model towards satisfying each aspect. methodologically, we develop a sequence-tosequence-based text generation model that can learn from template questions, allowing it to generate more fluent questions and generalize to previously unexplored roles. nonetheless, more is required to produce high-quality questions. further, to meet the criteria of context dependence and indicative guidance, we developed a reinforcement learning-based question generation framework, rlqg to refine the generation process @xcite . specifically, an inverse prompting mechanism is proposed to evaluate whether the question matches the context, and a question-answering reward is used to quantify the degree to which the question is indicative. we choose a positive and negative question pair based on the above two mechanisms and utilize these as signals to fine-tune the model, biasing it toward generating context-dependent and suggestive questions providing indicative guidance. finally, the effectiveness of our method has been verified on two widely used ee benchmarks. according to the results, on the full ace @xcite and rams @xcite benchmarks, our method outperforms previous methods by 2.69% and 1.96%. more importantly, we show that our method is particularly effective in datascarce scenarios -with only 40% of the training data, we achieved the same performance as previous works. additionally, we show that we can achieve good performance based on simple questions without excessive manual intervention. in summary, the contributions of our work are three-fold: ## proposed method the overview of our method is visualized in figure 2, which contains three main modules: we detail each part in the following. ## experimental results in this section, we present the experimental results divided by results with full training resources on ace and rams and results on the data-scarce scenarios in ace. ## further discussion if not explicitly stated, further discussion of our method's details focuses on the representative baseline with the template-based method dynamic-q and its corresponding sft model. the experiments are conducted on the ace dataset with practical evaluation. ## conclusion event extraction (ee) has evolved from traditional classification into question-answering-based methods (qa-based ee). these methods emphasize the design of quality questions to guide qa models for better answers. in this paper, we introduce a reinforcement learning framework that aims to produce context-dependent, fluently phrased questions that are generalizable and indicative enough to qa models, addressing the challenges posed by rigid, template-based questions. our methodology demonstrates improved performance on ace and rams benchmarks, particularly in data-scarce scenarios. it highlights the method's efficacy in generating effective questions for qa-based ee without extensive manual intervention.