Learning from a Friend: Improving Event Extraction via Self-Training with Feedback from A bstract M eaning R epresentation (2023)

Abstract: Data scarcity has been the main factor that hinders the progress of event extraction. To overcome this issue, we propose a Self-Training with Feedback (STF) framework that leverages the large-scale unlabeled data and acquires feedback for each new event prediction from the unlabeled data by comparing it to the Abstract Meaning Representation (AMR) graph of the same sentence. Specifically, STF consists of (1) a base event extraction model trained on existing event annotations and then applied to large-scale unlabeled corpora to predict new event mentions as pseudo training samples, and (2) a novel scoring model that takes in each new predicted event trigger, an argument, its argument role, as well as their paths in the AMR graph to estimate a compatibility score indicating the correctness of the pseudo label. The compatibility scores further act as feedback to encourage or discourage the model learning on the pseudo labels during self-training. Experimental results on three benchmark datasets, including ACE05-E, ACE05-E+, and ERE, demonstrate the effectiveness of the STF framework on event extraction, especially event argument extraction, with significant performance gain over the base event extraction models and strong baselines. Our experimental analysis further shows that STF is a generic framework as it can be applied to improve most, if not all, event extraction models by leveraging large-scale unlabeled data, even when high-quality AMR graph annotations are not available.

## introduction event extraction (ee), which aims to identify and classify event triggers and arguments, has been a long-stand challenging problem in natural language processing. despite the large performance leap brought by advances in deep learning, recent studies @xcite have shown that the data scarcity of existing event annotations has been the major issue that hinders the progress of ee. for example, in ace-05 foot_1 , one of the most popular event extraction benchmark datasets, 10 of the 33 event types have less than 80 annotations. however, creating event annotations is extremely expensive and time-consuming, e.g., it takes several linguists over one year to annotate 500 documents with about 5000 event mentions for ace-05. to overcome the data scarcity issue of ee, previous studies @xcite @xcite develop self-training methods that allow the trained ee model to learn further by regarding its own predictions on large-scale unlabeled corpora as pseudo labels. however, simply adding the high-confidence event predictions to the training set inevitably introduces noises @xcite @xcite , especially given that the current state-of-the-art performance of event argument extraction is still less than 60% f-score. to tackle this challenge, we introduce a self-training with feedback framework, named stf, which consists of an event extraction model that is firstly trained on the existing event annotations and then continually updated on the unlabeled corpus with selftraining, and a scoring model that is to evaluate the correctness of the new event predictions (pseudo labels) from the unlabeled corpus, and the scores further act as feedback to encourage or discourage the learning of the event extraction model on the pseudo labels during self-training, inspired by the reinforce algorithms @xcite . specifically, the event extraction model of our stf framework can be based on any state-of-the-art architecture. in this paper, we choose oneie @xcite and amr-ie @xcite , due to their superior performance and publicly available source code. the scoring model leverages the abstract meaning representation (amr) @xcite which has been proven to be able to provide rich semantic and structural signals to map amr structures to event predictions @xcite @xcite and thus their compatibility can indicate the correctness of each event prediction. the scoring model is a self-attention network that takes in a predicted event trigger, a candidate argument and its argument role, as well as their path in the amr graph of the whole sentence, and computes a score ranging in @xcite based on the compatibility between the amr and the predicted event structure: -1 means incompatible, 1 means compatible, and 0 means uncertain. inspired by the reinforce algorithm @xcite , we multiply the compatibility scores and the gradient of the ee model computed on the pseudo event labels during self-training, so as to ( 1 ) encourage the event extraction model to follow the gradient and hence maximize the likelihood of the pseudo label when it is compatible with the amr structure; we take amr 3.0 3 and part of the new york times (nyt) 2004 corpus foot_3 as additional unlabeled corpora to enhance the event extraction model with stf, and evaluate the event extraction performance on three public benchmark datasets: ace05-e foot_4 , ace05-e +6 , and ere-en foot_6 . the experimental results demonstrate that: (1) the vanilla self-training barely improves event extraction due to the noise introduced by the pseudo examples, while the proposed stf framework leverages the compatibility scores from the scoring model as feedback and thus makes more robust and efficient use of the pseudo labels; (2) stf is a generic framework and can be applied to improve most, if not all, of the event extraction models optimized by gradient descent algorithm and achieves significant improvement over the base event extraction models and strong baselines on event argument extraction on the three public benchmark datasets; ## stf for event extraction the event extraction task consists of three subtasks: event detection, argument identification and argument role classification. given an input sentence @xmath0, w 2 , ..., w n ], event detection aims to identify the span of an event trigger τ i in w and assign a label l τ @xmath1. argument identification aims to find the span of an argument ε j in w , and argument role classification further predicts a role α @xmath2. figure 1 shows the overview of our stf framework which consists of two training stages. at the first stage, a base event extraction model (section 2.1) is trained on a labeled dataset. at the second stage, we apply the trained event extraction model to an unlabeled corpus to predict new event mentions. instead of directly taking the new event predictions as pseudo training examples like the vanilla self-training, we propose a novel scoring model (section 2.2) to estimate the correctness of each event prediction by measuring its compatibility to the corresponding amr graph, and then take both event predictions and their compatibility scores to continue to train the base event extraction model while the scores update the gradient computed on pseudo labels (section 2.3). after the training of the second stage, we get a new event extraction model and evaluate it on the test set. ## experimental setups for evaluation, we consider two base event extraction models: oneie @xcite and amr-ie @xcite due to their superior performance on event extraction and publicly available source code, and demonstrate the effectiveness of stf on three benchmark datasets: ace05-e, ace05-e + and ere-en, with the same evaluation metrics following previous studies @xcite @xcite 13 . to show the generalizability of stf, we explore two unlabeled corpora for self-training: (1) amr 3.0 @xcite which originally contains 55,635 sentences in the training set while each sentence is associated with a manually annotated amr graph. (2) new york times annotated corpus (nyt) contains over 1.8 million articles that were published between 1987 to 2007. we randomly sample 55,635 sentences foot_13 from articles published in 2004. because nyt dataset does not have amr annotations, we run a pre-trained amr parser (astudillo et al., 2020) to generate system amr parsing. besides taking the recent state-of-the-art event extraction studies @xcite @xcite @xcite @xcite @xcite 15 as baselines, we also compare our proposed stf with two other training strategies: (1) vanilla self-training @xcite which consists of two stages similar as stf but in the second stage takes each new event prediction from the unlabeled data with a probability higher than 0.9 based on the base event extraction model as a pseudo label and combines them with the labeled data to re-train the event extraction model; and (2) gradient imitation reinforcement learning (gradlre) @xcite . gradlre encourages the gradients computed on the pseudo-labeled data to imitate the gradients computed on the labeled data by using the cosine distance between the two sources of gradients as a reward to perform policy gradi-ent reinforcement learning @xcite . gradlre showed improvements over other selftraining methods on low-resource relation extraction which is a similar task to argument role classification. appendix c describes the training details for both baselines and our approach. ## related work most prior studies have been focusing on learning supervised models @xcite @xcite @xcite @xcite @xcite al., 58.3 58.5 58.7 58.9 f1-score % stf_amr_ace stf_amr_ace+ stf_amr_ere 0.5 0.6 0.7 0.8 0.9 threshold s st 51.4 51.6 51.8 52.0 2020; @xcite @xcite based on manually annotated event mentions. however, the performance of event extraction has been barely improved in recent years, and one of the main reasons lies in the data scarcity and imbalance of the existing event annotations. several self-training and semi-supervised studies have been proposed to automatically enrich the event annotations. @xcite uses extraction patterns based on nouns that, by definition, play a specific role in an event, to automatically label more data. @xcite proposes various event inference mechanisms to reveal additional missing event mentions. @xcite propose semi-supervised learning to automatically induce new event types and their corresponding event mentions while the performance of old types is also improved. @xcite @xcite propose techniques to select a more relevant and informative corpus for self-training. all these studies cannot handle the noise introduced by the automatically labeled data properly. compared with them, our stf framework leverages a scoring model to estimate the correctness of each pseudo-labeled example, which further guides the gradient learning of the event extraction model, thus it can efficiently mitigate the impact of the noisy pseudo-labeled examples. self-training has been studied for many years @xcite @xcite and widely adopted in many tasks including speech recognition @xcite , biomedical imaging @xcite , parsing @xcite , and pretraining @xcite . self-training suffers from inaccurate pseudo labels @xcite @xcite especially when the teacher model is trained on insufficient and unbalanced datasets. to address this problem, @xcite @xcite propose to utilize the performance of the student model on the held-out labeled data as a meta-learning objective to update the teacher model or improve the pseudo-label generation process. @xcite leverage the cosine distance between gradients computed on labeled data and pseudo-labeled data as feedback to guide the self-training process. @xcite leverage the span of named entities as constraints to improve semi-supervised semantic role labeling and syntactic parsing, respectively. ## conclusion we propose a self-training with feedback (stf) framework to overcome the data scarcity issue of the event extract task. the stf framework estimates the correctness of each pseudo event prediction based on its compatibility with the corresponding amr structure, and takes the compatibility score as feedback to guide the learning of the event extraction model on each pseudo label during self-training. we conduct experiments on three public benchmark datasets, including ace05-e, ace05-e + , and ere, and prove that stf is effective and general as it can improve any base event extraction models with significant gains. we further demonstrate that stf can improve event extraction models on large-scale unlabeled corpora even without high-quality amr annotations.