Learning with Partial Annotations for Event Detection (2023)

Abstract: Event detection (ED) seeks to discover and classify event instances in plain texts. Previous methods for ED typically adopt supervised learning, requiring fully labeled and high-quality training data. However, in a real-world application, we may not obtain clean training data but only partially labeled one, which could substantially impede the learning process. In this work, we conduct a seminal study for learning with partial annotations for ED.We propose a new trigger localization formulation using contrastive learning to distinguish ground-truth triggers from contexts, showing a decent robustness for addressing partial annotation noise. Impressively, in an extreme scenario where more than 90% of events are unlabeled, our approach achieves an F1 score of over 60%.In addition, we re-annotate and make available two fully annotated subsets of ACE 2005 to serve as an unbiased benchmark for event detection. We hope our approach and data will inspire future studies on this vital yet understudied problem.

## introduction deep learning models have shown impressive performance in event detection (ed) since large amounts of event data have become available @xcite . however, such models require fully labelled and highquality data -in practice, we cannot ensure that every event is identified, and as a result, we often face the partial annotation issue, as depicted in figure 1 . we show a high rate of partial annotation in real-world event datasets. for example, in ace 2005, which is widely used as a benchmark for ed evaluation @xcite , nearly 20% of events are not labelled (see in this work, we present a seminal study of learning with partial annotations for ed, with contributions in methodology, data, and practical applications. in our method, to reduce the risk of mis-training on false negatives, we propose a contrastive learning framework @xcite to distinguish ground-truth triggers from contexts, which is shown to be more tolerant of partial annotation noise than the traditional hard classification paradigm @xcite . in addition, to succeed in the partial annotation scenario, we augment the model with a self-correction regime to recognize false negatives during the training stage. figure 2 visualizes the core of our method, which is a de facto trigger localization formulation that uses sentence-wise normalization (prompted by event types) to find event triggers. compared to hard classification methods that add up individual losses (as shown at the top of figure 2 ), our approach instead forms a contrastive learning paradigm by raising the scores of ground-truth triggers while lowering the scores of context words. as a result, even with a significant number of false negatives in training, it can still maintain a good separation between triggers and contexts ( § 6.1). in addition, we suggest that adding a margin softmax @xcite with a gaussian-based distance regularization can further improve learning. in addition to the noise-tolerance mechanism described above, we propose a self-correction regime with the motive that when a model recognizes a false negative with high confidence, it should correct its labels for the subsequent training stage. nevertheless, modeling the confidence of deep learning models is challenging since their predictions are poorly calibrated (i.e., a model often outputs a high prediction probability even if the prediction is incorrect @xcite ). to address this issue, we propose an uncertainty-guided retraining mechanism based on mc-dropout @xcite , which can output prediction confidence to guide the self-correction process. we explain the relationship between this paradigm and an expectation-maximization (em) framework. in addition to the methodology contribution, we re-annotate and release the ace 2005 development and test sets as a data contribution. on the revised benchmark (and an extra maven @xcite ) benchmark), we demonstrate the impressive performance of our models -in particular, even in an extreme case with 90% of events unlabeled, our approach achieves more than 60% in f1, yielding a 40% definite improvement over previous methods. in addition to simulation tests, we also conduct a real-world annotation test on wikievents @xcite , where the results suggest the practical applicability of our approach. contributions. our contributions are three-fold: (i) to the best of our knowledge, this is the first work addressing the potential partial annotation issue in ed, which may spark further research interest. (ii) we highlight a new learning paradigm for ed based on a trigger localization formulation and show that it works effectively with a wide range of partial annotation settings. (iii) we re-annotated the ace 2005 development and test datasets and released them to the community to serve as an unbiased benchmark. (iv) in addition to simulation experiments, we conduct real-world annotation experiments to validate the effectiveness of our approach for practical use. ## proposed method let @xmath0, • • • , w n ] be a sentence with n words and @xmath1, • • • , y n ] be the ground-truth event label sequence, where y @xmath2. then the partial annotation issue can be formulated as: due to the neglect of human annotators, some y @xmath3, and this results in a partial annotation sequence ỹ = [ỹ 1 , • • • , ỹn ]. clearly, directly training a model on (x, ỹ ) risks outputting a noisy detector. here we propose a new learning framework to address this issue (as shown in figure 3 ), which consists of a noise-tolerant learning mechanism with margin softmax ( § 3.2) and uncertainty-guided retraining mechanism ( § 3.3). ## experimental setups datasets. we conduct our experiments on ace 2005 and maven foot_5 @xcite , with data statistics shown in implementations. in our approach, we use bert-large architecture for ace 2005 @xcite , and bert-base for maven @xcite . as for hyper-parameters, the batch size is set to 10 for ace 2005 and 20 for maven respectively, chosen from @xcite @xcite @xcite . the learning rate is set at 1e-5 for both datasets, chosen from [5e-5, 1e-5, 5e-6, 1e-6]. in the margin softmax regime, the positive margin λ + is set to 10, and the negative margin λ -is set to 1; these values are chosen from [0.1, 0.5, 1, 5, 10, 50, 100]. in the uncertaintyguided retraining mechanism, the number of prediction times k is empirically set to 20 for a tradeoff between speed and efficiency. we release the data and the code at https://github.com/ jianliu-ml/partialed . split # potential # validated ul rate dev. set challenge 78 34 (43.6%) 6.7% control 34 19 (55.9%) 3.8% total 112 53 (47.3%) 10.5% test set challenge 86 51 (59.3%) 12.0% control 50 31 (62.0%) 7.3% total 136 82 (60.2%) 19.3% evaluation settings. we investigate three evaluation settings: (i) a full training setting, in which we use the original training set for learning. yet, because the original training set inevitably contains unlabeled events, this setting is still a partial learning setting. (ii) a data removal setting, in which we exclude a portion of events from the training setting to study whether the performance drop is caused by a degraded number of positive examples. (iii) a data masking setting, in which we remove the labeling information of some events (by replacing their labels to o) to simulate a more serious partial annotation scene. we use precision (p), recall (r), and f1 as evaluation metrics following previous studies (ji @xcite , and to against randomness, we report experimental results based on a 5-run average. ## experimental results results in the full training setting. tables 3 and 4 show results in the full training setting. accordingly, our method achieves the best f1 on the clean ace test set and the maven development/test set, suggesting its efficacy. comparing the results on the original ace test set is interesting: our method has lower precision than other methods, but when applied to the revised set, the precision is greatly boosted -this implies that our 10% 30% 50% 70% 90% ratio of remained labeled cases 0 20 40 60 80 f1 score (%) promptloc negspl hiddencrf oneie hybrid 10% 30% 50% 70% 90% ratio of remained labeled cases 0 20 40 60 promptloc negspl hiddencrf oneie hybrid figure 5: results on ace 2005 (left) and maven (right) in the data masking setting. results in the data removal setting. finally, we consider the data removal setting to study the impact of a lack of positive examples, and we show results in figure 4 . according to the results, while our model consistently outperforms others, the gap is small, implying that a reduced number of positive instances is not a major factor impeding learning, especially when there are relatively abundant training examples (e.g., @xmath4.g., hybrid). results in the data masking setting. we then consider the data masking setting and we show results foot_6 in figure 5 . here p denotes the ratio of remaining examples (i.e., we mask the labels of 1p events). according to the results, our approach outperforms prior methods by significant margins. for example, on the ace 2005, when 70% of triggers are masked (@xmath5, our approach obtains 70% in f1, outperforming previous methods by 30% in f1; method setting p r f1 clean 73.1 72.9 73.0 trigger local. argmax 70.5 70.5 70.5 adaptive τ 68.7 72.8 70.7 clean 66.7 76.0 71.0 hard class. argmax 60.3 42.5 49.8 adaptive τ 58.6 43.8 50.1 ## conclusion in this study, we investigate the partial annotation problem in ed, a critical yet less-explored problem. we motivate a new learning model for ed and investigate its effectiveness in a variety of par-tial annotation settings. we also provide two reannotated subsets of ace 2005 to the community as a data contribution in order to establish a fair evaluation. in the future, we plan to investigate the theoretical aspects of our approach and increase its scope by applying it to other information extraction tasks suffering the partial annotation issue, such as named entity recognition and relation extraction.