Learning Prototype Representations Across Few-Shot Tasks for Event Detection (2021)

Abstract: We address the sampling bias and outlier issues in few-shot learning for event detection, a subtask of information extraction. We propose to model the relations between training tasks in episodic few-shot learning by introducing cross-task prototypes. We further propose to enforce prediction consistency among classifiers across tasks to make the model more robust to outliers. Our extensive experiment shows a consistent improvement on three few-shot learning datasets. The findings suggest that our model is more robust when labeled data of novel event types is limited. The source code is available at http://github.com/laiviet/fsl-proact .

## introduction in information extraction, event detection (ed) is an important task that aims to identify and classify event triggers of predefined event types in text @xcite . event triggers are words/phrases that most clearly indicate the occurrence of events. for example, an event detector should recognize the word homicide in the following sentence as a trigger word of event type life.die.death-caused-by-violent-events: typical ed systems follow a supervised learning scheme that requires a large amount of labeled data for each predefined event type @xcite @xcite . unfortunately, this requirement is usually too costly to achieve in real applications where novel event types emerge and only a few examples are available @xcite . as such, an ed model should be prepared to extract triggers of novel event types (i.e., beyond those provided in the training data) for which only a few examples are provided. this learning schema is known as few-shot learning (fsl) for ed. to emulate the learning from few examples in ed, n -way k-shot episodic training is often used to exploit existing datasets @xcite @xcite . in each training iteration, a small subset (i.e. support set) of n event types with k examples per type is sampled from the training data. unfortunately, the sample size is so small (@xmath0, 10]) that the fsl models might suffer from sample bias, thus hindering the generalization to novel event types. prototypical network is a popular metric-based few-shot learning model @xcite that has been explored for fsl @xcite . it introduces a prototype vector for each event type by averaging the representations of the instances of that type. a non-parametric classifier then predicts the event type of a query instance based on its distances from the prototypes @xcite . hence, an outlier in the support set might significantly change the prototypes and flip the label of the query instance. in addition, in ed, a null class is introduced to represent non-eventive mentions. this type covers every domains and every surface form except the relevant event types. thus, this unbounded class might also present a great source of outliers for the support set. in this paper, we mitigate the effects of poor sampling and outliers by modeling cross-task relation. first, we propose to augment the support data of the current task with those from prior tasks that essentially helps increase the population of the current support set. therefore, it can mitigate the sample bias in the support set. second, the averaging in prototypical network allows outliers to contribute equally to the prototype representation. we propose to use soft-attention to select the most related data samples as well as reduce the contribution of the outliers to the prototype representation. third, a fsl model that is resistant to the outliers should produce consistent predictions regardless of support data. to implement this, we produce two prototypical-based classifiers from the two support sets of the two tasks. after that, we enforce the consistency of their predictions on query instances. during training, development, and testing, the task t is sampled from three sets of data d train , d dev , and d test whose sets of classes are y train , y dev , and y test , respectively. these sets of classes are mutually disjoint to ensure that the model observes no more than k examples from a novel class. a typical fsl model has two main modules: an encoder and a few-shot classifier. an encoder, denoted as φ, encodes an instance into a fixeddimension vector v j @xmath1, a j i ) ∈ r u where u is the dimension of the representation vector. a fewshot classifier classifies a query instance among classes appearing in the support set. for instance, in a prototypical network, a prototype v j is a classrepresentative instance that is an average of all vectors of the j-th class v @xmath2, a j i ). then the distance distribution of the query instance @xmath3, a q , y q } @xcite is: the training minimizes the cross-entropy loss, de-noted by l ce , over all query instances: cross-task data augmentation: in conventional episode training, two consecutive training tasks t 1 and t 2 are not likely to share an identical event type sets, y 1 = y 2 . we assume that our training process has a memory to save the latest samples of every event type used in prior tasks. using this memory, after a certain number of training iterations, for a new task t 1 , a second sample t 2 can always be sampled from the memory such that y 2 = y 1 . the expected value of delaying iterations for 5-way on ace dataset is 13 iterations (@xmath4. prototype across task we are given two tasks t 1 = (s 1 , q 1 ) and t 2 = (s 2 , q 2 ) sampled with the same set of event type y. the prototypes are induced from both tasks as follow: then, an attention module, denoted by att, induces intermediate representations for the support and query instances of t 1 via weighted sums of the support vectors of the t 2 , and vice versa: the final representations for both tasks are then the sum of their original representations and the cross-task representations: then, the prototypes for tasks t 1 and t 2 are computed by averaging vectors of the same class from h s 1 and h s 2 , respectively @xcite . cross task consistency the cross task consistency (ctc) further reduces the sample bias by introducing prediction consistency between classifiers generated from two tasks. without loss of generation, we assume that one of the classifiers is impaired by the poor sampling. we employ the knowledge distillation technique @xcite that helps transfer knowledge from the stronger classifier to the weaker one. this thus makes the model more robust to the sample bias. we enforce the cross task consistency by minimizing the differences between predicted label distributions from the classifiers of two tasks as follow: finally, to train the model, we minimize the total loss (α is a hyper-parameter): testing: as the model does not have access to the prior task of the novel class, the prototypes are computed based on the vectors of the current task only. hence, the model turns into the original prototypical network @xcite . our proposed methods only appy to the training process, hence, it provides a fair performance compared with prior fsl ed models. as it has been observed that training with more classes helps improve the model performance, we use 18+1 classes during training, while keeping 5+1 and 10+1 novel classes during testing. baseline: we consider three strong baselines for fsl ed. proto features a prototype for each novel class and euclidean distance function, presented in equation 1 @xcite . interintra is an extension of the prototypical network with two auxiliary training signals. it minimizes the distances among data points of the same class and maximizes the distances among prototypes @xcite . dmb-proto extends the prototypical network in a way that the representation vector for each data point is induced by a dynamic memory network running on the data of the same class @xcite . since the source code of dmb-proto is not published, we reimplement the few-shot classifier with a dynamic memory module @xcite . we examine two state-of-theart bert-based sentence encoders φ for ed, i.e. bertmlp @xcite and @xcite . hyperparameters: in this paper, stochastic gradient decent optimizer is used with learning rate 1e -4 . the training/evaluation are set to 6,000 and 500 iterations respectively; the evaluation is done after every 500 training iterations. the dimension of the final representation is set to 512. we use dropout rate of 0.5 to prevent overfitting. the coefficient of the cross-task consistency loss is set to α = 10 based on the best development performance (α ∈ {1, 10, 100, 1000}. we evaluate our ed model using the micro f1score. the training and evaluation are done on a single nvidia gtx 2080ti with 11gb of gpu ram. the training and evaluation take approximately 4 hours. we implement the model using pytorch version 1.6.0. result: first, using the same sentence encoders, proact achieves the best performance on all three datasets and settings. the improvement margins are in range [1.0%-6.1%] on the 5-shot setting and [0.7%-3.1%] on the 10-shot setting. second, the f-score margin between proact and proto decreases as the shot number increases. this indicates that the proposed model performs better when the number of observed samples is small. as the number of shots increases, the improvement gets saturated. this finding is parallel with the fact that sample bias is more likely when the number of shots is small. hence, our proposed method is more suitable to event detection in few-shot learning schema, especially in the case where the number of shots is limited. ablation study: our proposed model involves three factors: the cross-task data (data), the crosstask attentive prototype (attention) and the crosstask consistency (consistency). to analyze the efficiency of these modules, we incrementally eliminate these modules from the full proact model and evaluate the remaining model on 5+1-way 5-shot setting. if attention and loss are removed while data remains, the model and setting become a prototypical network with 5+1-way 10-shot setting during the training. this model has the same amount of support data that our model has during the training process. note that the testing with novel classes remains 5+1-way 5-shot setting for every model. if the cross-task data is eliminated, the attentive prototype and consistency loss are also removed and the model and setting return to a prototypical model p r f proact (full model) 74.9 76.7 75.7 -attention 74.1 76.0 74.9 -consistency 73.3 75.7 74.4 -attention -consistency 72.5 74.5 73.4 -data (-attention -consistency) 69.9 72.4 71.0 analysis: to further analyze the efficiency of our proposed method, we aim to discover which classes benefit the most. to do that, we compute two confusion matrices for proact and proto models on the test set of rams. we fix the random seed to make sure the sampling during testing are identical between two runs, hence ensuring that the proportion of classes are identical. figure 1 presents the difference of two confusion matrices exhibited by the proposed model proact and the prototypical network proto. there are two major observations from the figure. first, overall proact produces more accurate predictions than proto, as shown on the diagonal. second, proact involves remarkably more correct predictions for negative examples than proto. in the mean time, it generates significantly lower number of errors in both false positive and false negative related to the null class, i.e. other class in figure 1 , suggesting that our proposed model effectively mitigates the effect of noise introduced by the null class. ## related works prior studies in ed mainly follow the supervised learning scheme. the early work focuses on feature engineering with statistical models @xcite @xcite . recently, many deep learning architectures have been explored for automatic feature learning @xcite @xcite @xcite @xcite . some recent studies have also introduced methods to extending ed to new event types @xcite @xcite @xcite @xcite . fsl has been extensively studied in computer vision @xcite @xcite @xcite . recent work has also considered fsl for tasks in natural language processing @xcite . for ed, prior fsl work has mostly relied on prototypical network @xcite . however, these models do not explore cross-task modeling as we do. ## conclusion in this paper, we propose to exploit the relationship between training tasks for few-shot learning event detection. we compute prototypes based on cross-task modeling and present a regularization to enforce prediction consistency of classifiers across tasks. the experiment results show that exploiting cross-task relation can alleviate the poor sampling and outliers in the support set for fsl in ed. in the future, we will extend our method to other tasks in information extraction such as named entity recognition and argument extraction. viet dac lai, thien huu nguyen, and frank dernoncourt. 2020b. extensively matching for few-shot learning event detection. in proceedings of the first joint workshop on narrative understanding, storylines, and events. viet dac lai, tuan ngo nguyen, and thien huu nguyen. 2020c. event detection: gate diversity and syntactic importance scores for graph convolution neural networks. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 5405-5411, online. association for computational linguistics. kwonjoon lee, subhransu maji, avinash ravichandran, and stefano soatto. 2019. meta-learning with differentiable convex optimization. in proceedings of the conference on computer vision and pattern recognition (cvpr). shasha liao and ralph grishman. 2010. using document level cross-event inference to improve event extraction. in proceedings of the annual meeting of the association for computational linguistics (acl). shasha liao and ralph grishman. 2011. acquiring topic features to improve event extraction: in preselected and balanced collections. in ranlp. minh van nguyen, viet dac lai, and thien huu nguyen. 2021. cross-task instance representation interactions and label dependencies for joint information extraction with graph convolutional networks. in proceedings of the conference of the north american chapter of the association for computational linguistics: human language technologies (naacl-hlt). thien nguyen and ralph grishman. 2018. graph convolutional networks with argument-aware pooling for event detection. in proceedings of the association for the advancement of artificial intelligence (aaai). thien huu nguyen, kyunghyun cho, and ralph grishman. 2016. joint event extraction via recurrent neural networks. in proceedings of the conference of the north american chapter of the association for computational linguistics: human language technologies (naacl-hlt). thien huu nguyen, lisheng fu, kyunghyun cho, and ralph grishman. 2016b. a two-stage approach for extending event detection to new types via neural networks. in proceedings of the 1st acl workshop on representation learning for nlp (repl4nlp). thien huu nguyen and ralph grishman. 2015. event detection and domain adaptation with convolutional neural networks. in proceedings of the 53rd annual