Unleash GPT -2 Power for Event Detection (2021)

Abstract: Event Detection (ED) aims to recognize mentions of events (i.e., event triggers) and their types in text. Recently, several ED datasets in various domains have been proposed. However, the major limitation of these resources is the lack of enough training data for individual event types which hinders the efficient training of data-hungry deep learning models. To overcome this issue, we propose to exploit the powerful pre-trained language model GPT-2 to generate training samples for ED. To prevent the noises inevitable in automatically generated data from hampering training process, we propose to exploit a teacher-student architecture in which the teacher is supposed to learn anchor knowledge from the original data. The student is then trained on combination of the original and GPT-generated data while being led by the anchor knowledge from the teacher. Optimal transport is introduced to facilitate the anchor knowledge-based guidance between the two networks. We evaluate the proposed model on multiple ED benchmark datasets, gaining consistent improvement and establishing state-of-the-art results for ED.

## introduction an important task of information extraction (ie) involves event detection (ed) whose goal is to recognize and classify words/phrases that evoke events in text (i.e., event triggers). for instance, in the sentence "the organization donated 2 million dollars to humanitarian helps.", ed systems should recognize "donated" as an event trigger of type pay. we differentiate two subtasks in ed, i.e., event identification (ei): a binary classification problem to predict if a word in text is an event trigger or not, and event classification (ec): a multi-class classification problem to classify event triggers according to predefined event types. several methods have been introduced for ed, extending from feature-based models @xcite @xcite to advanced deep learning methods @xcite @xcite @xcite . although deep learning models have achieved substantial improvement, their requirement of large training datasets together with the small sizes of existing ed datasets constitutes a major hurdle to build high-performing ed models. recently, there have been some efforts to enlarge training data for ed models by exploiting unsupervised @xcite or distantly-supervised @xcite @xcite techniques. the common strategy in these methods is to exploit unlabeled text data that are rich in event mentions to aid the expansion of training data for ed. in this work, we explore a novel approach for training data expansion in ed by leveraging the existing pre-trained language model gpt-2 @xcite to automatically generate training data for models. motivated by the promising performance of gpt models for text generation, we expect our approach to produce effective data for ed in different domains. specifically, we aim to fine-tune gpt-2 on existing training datasets so it can generate new sentences annotated with event triggers and/or event types, serving as additional training data for ed models. one direction to achieve this idea is to explicitly mark event triggers along with their event types in sentences of an existing ed dataset that can be used to fine-tune the gpt model for new data generation. however, one issue with this direction is that in existing ed datasets, numbers of examples for some rare event types might be small, potentially leading to the poor tuning performance of gpt and impairing the quality of generated examples for such rare events. in addition, large num-bers of event types in some ed datasets might make it more challenging for the fine-tuning of gpt to differentiate event types and produce high-quality data. to this end, instead of directly generating data for ed, we propose to use gpt-2 to only generate samples for the event identification task to simplify the generation and achieve data with better annotated labels (i.e., output sentences only are only marked with positions of event triggers). as such, to effectively leverage the generated ei data to improve ed performance, we propose a multitask learning framework to train the ed models on the combination of the generated ei data and the original ed data. in particular, for every event trigger candidate in a sentence, our framework seeks to perform two tasks, i.e., ei to predict a binary label for being an event trigger or not, and ed to predict the event type (if any) evoked by the word via a multi-class classification problem. an input encoder is shared for both tasks that allow training signals from both generated ei data and original ed data to contribute to the representation learning in the encoder (i.e., transferring knowledge in generated ei data to ed models). despite the simplification to ei for better annotated labels of data, the generated sentences might still involve noises due to the inherent nature of the language generation, e.g., grammatically wrong sentences, inconsistent information, or incorrect event trigger annotations. as such, it is crucial to introduce mechanisms to filter the noises in generated data to enable effective transfer learning from generated ei data. to this end, prior works for gptbased data generation for other tasks has attempted to directly remove noisy generated examples before actual usage for model training via some heuristic rules @xcite . however, heuristic rules are brittle and restricted in their coverage so they might overly filter the generated data or incorrectly retain some noisy generated samples. to address this issue, we propose to preserve all generated data for training and devise methods to explicitly limit impacts of noisy generated sentences in the models. in particular, we expect the inclusion of generated ei data into the training process for ed models might help to shift the representations of the models to better regions for ed. as such, we argue that this representation transition should only occur at a reasonable rate as drastic divergence of representations due to the generated data might be associated with noises in the data. motivated by this intuition, we propose a novel teacher-student framework for our multi-task learning problem where the teacher is trained on the original clean ed datasets to induce anchor representation knowledge for data. the student, on the other hand, will be trained on both generated ei data and original ed data to accomplish transfer learning. here, the anchor knowledge from the teacher will be leveraged to guide the student to prevent drastic divergence of representation vectors for noisy information penalization. consequently, we propose a novel anchor information to implement this idea, seeking to maintain the same level of differences between the generated and original data (in terms of representation vectors) for both the teacher and the student (i.e., generated-vsoriginal data difference as the anchor). at the core of this techniques involves the computation of distance/difference between samples in generated and original data. in this work, we envision two types of information that models should consider when computing such distances for our problem: (1) representation vectors of the models for the examples, and (2) event trigger likelihood scores of examples based on the models (i.e., two examples in the generated and original data are more similar if they both correspond to event triggers). as such, we propose to cast this distance computation problem of generated and original data into an optimal transport (ot) problem. ot is an established method to compute the optimal transportation between two data distributions based on the probability masses of data points and their pair-wise distances, thus facilitating the integration of the two criteria of event trigger likelihoods and representation vectors into the distance computation between data point sets. extensive experiments and analysis reveal the effectiveness of the proposed approach for ed in different domains, establishing new state-of-theart performance on the ace 2005, cyseced and rams datasets. ## model we formulate the task of event detection as a word-level classification problem as in prior work @xcite . formally, given the sentence @xmath0, w 2 , . . . , w n ] and the candidate trigger word w t , the goal is to predict the event type l from a pre-defined set of event types l. note that if the word w t is not a trigger word, the gold event type is n one. our proposed approach for this task consist of two stages: (1) data augmentation: to employ natural language generation to augment existing training datasets for ed, (2) task modeling: to propose a deep learning model for ed, exploiting available training data. ## conclusion we propose a novel method for augmenting training data for ed using the samples generated by the language model gpt-2. to avoid noises in the generated data, we propose a novel teacher-student architecture in a multi-task learning framework. we introduce a mechanism for knowledge consistency enforcement to mitigate noises from generated data based on optimal transport. experiments on various ed benchmark datasets demonstrate the effectiveness of the proposed method. minh van nguyen and thien huu nguyen. 2018. who is killed by police: introducing supervised attention for hierarchical lstms. in proceedings of the international conference on computational linguistics (coling). thien huu nguyen, kyunghyun cho, and ralph grishman. 2016a. joint event extraction via recurrent neural networks. in proceedings of the conference of the north american chapter of the association for computational linguistics: human language technologies (naacl-hlt). thien huu nguyen, lisheng fu, kyunghyun cho, and ralph grishman. 2016b. a two-stage approach for extending event detection to new types via neural networks. in proceedings of the 1st acl workshop on representation learning for nlp (repl4nlp).