Zero-Shot Event Detection Based on Ordered Contrastive Learning and Prompt-Based Prediction
Abstract
Event detection is a classic natural language processing task.
However, the constantly emerging new events make supervised methods not applicable to unseen types.
Previous zeroshot event detection methods either require predeﬁned event types as heuristic rules or resort to external semantic analyzing tools.
To overcome this weakness, we propose an end-to-end framework named Z ero-Shot E vent Detection Based on O rdered Contrastive Learning and P romptBased Prediction (ZEOP).
By creatively introducing multiple contrastive samples with ordered similarities, the encoder can learn event representations from both instance-level and class-level, which makes the distinctions between different unseen types more signiﬁcant.
Meanwhile, we utilize the prompt-based prediction to identify trigger words without relying on external resources.
Experiments demonstrate that our model discovers events more effectively and accurately than state-of-the-art methods.
Introduction
As a classic NLP task, event detection aims to identify events from natural language text.
Most traditional supervised event detection methods (Nguyen and Grishman, 2018; Wadden et al., 2019; Lin et al., 2020) rely on a great number of event-speciﬁc annotated texts.
However, in practice, obtaining largescale and high-quality annotated data requires signiﬁcant expertise and expensive resources.
In the real-world scenarios shown in Figure 1, the constantly emerging of new events without annotated samples, making supervised event detection methods no longer applicable.
To solve this challenge, the zero-shot event detection task is proposed to automatically discover and classify new events from unstructured texts in [Figure] the absence of manual annotation.
Following previous works (Zhang et al., 2015; Huang et al., 2018;
Huang and Ji, 2020; Wang et al., 2021), we denote the known types as seen types and the new types as unseen types.
"Unseen" means that the event labels of samples are not visible to the model.
Recently, multiple zero-shot event detection methods have been proposed and show better performance than supervised methods on zero-shot tasks.
However, they (Huang et al., 2018; Zhang et al., 2021b;
Lyu et al., 2021; Huang and Ji, 2020)
all require predeﬁned event types as heuristic rules or external semantic analyzing tools.
For example, event names are used to query trigger words by semantical similarity, or the part-of-speech tagging tools are used to ﬁnd nouns and verbs in the text as candidate trigger words.
In these ways, human effort and external resources are still necessary when detecting new event types.
There is also a problem of error accumulation across the tools and models.
To overcome the above weakness, this paper proposes an end-to-end model named Zero-Shot Event
Detection Based on Ordered Contrastive Learning and Prompt-Based Prediction.
The main idea is introducing contrastive learning to move away from
[Figure]
𝐴 𝑆𝑆𝑆𝑑𝑑𝑑𝑑 1 < 𝑑 2 < 𝑑 3 𝑑 0 < (𝑑 1 , 𝑑 2 ,𝑑 3 ) [Figure] the dependence on heuristic rules for unseen event detection.
As shown in Figure 2, traditional contrastive learning simply divide samples into two opposite class: positive or negative, while we construct four contrastive samples with different similarities to the original sample.
Then, the ordered contrastive learning can draw a stronger distinction between different unseen type events by learning the partial order relationship of different contrastive samples.
Meanwhile, in order to discover new event types without relying on heuristic rules, we utilize the prompt-based prediction (Brown et al., 2020; Schick and Schütze, 2021; Gao et al., 2021a) for trigger words identiﬁcation, which has been proved to be an efﬁcient few-shot learner.
The main work of this paper has the following three points:
• We proposed a zero-shot event detection model based on ordered contrastive learning.
By constructing multiple contrastive samples with ordered similarities, the encoder could learn a better representation of unseen types.
• We creatively introduce the prompt-based prediction into the zero-shot event detection problem for trigger words identiﬁcation, which removed the dependency on predeﬁned event structures and heuristic rules.
• Experiments on two English datasets demonstrate that the both supervised and zero-shot event detection performance are improved via ordered contrastive learning and promptbased prediction.
Related Work 2. Zero-shot event detection
The transfer learning-based zero-shot methods mainly rely on a predeﬁned event structure as heuristic rules.
In other words, models must know the unseen event name (e.g Attack) and the elements (e.g Attacker) consist of unseen events.
Huang et al.
(2018) and Zhang et al.
(2021b) cluster unseen events by label semantic similarity with the help of semantic structures analyzing tools such as Abstract Meaning Representation (AMR) or Semantic Role Labeling (SRL).
Additionally, Lyu et al.
(2021) need to deﬁne QA queries for unseen event types manually.
Although Huang and
Ji (2020) proposed SS-VQ-VAE to discover new event types without human assistance, external partof-speech tagging tool is needed to ﬁnd candidate trigger words.
Meanwhile, the semi-supervised loss function could only roughly separate all unseen event samples from seen events.
And the variational autoencoder focuses on the feature learning of single instance.
So it is still challenging to cluster unseen events into multiple new types.
2. Contrastive learning
Contrastive learning aims to learn high-quality feature representations through self-supervision.
The core of contrastive learning is constructing positive and negative sample pairs.
For labeled data, the construction is relatively simple, where random sampling by supervise label works in most cases.
For unsupervised learning with unlabeled data, it needs more strategy to construct sample pairs.
Wang et al.
(2021); Logeswaran and Lee (2018) treat the target sentence’s context as a positive sample.
Wang et al.
(2020) proposed various sample editing methods based on word masking and shufﬂe.
Gao et al.
(2021b) introduce a dropout mask for constructing contrastive samples, which doesn’t need any textual edit.
These approaches focus on instance-level feature contrastive and only divide samples into positive and negative.
Zhang et al.
(2021a) try to overcome this weakness by optimizing a top-down clustering loss.
Considering that class-level features learning is as essential as instance-level features, it’s necessary to improve the traditional contrastive learning framework for the zero-shot event detection task.
C on t r a s ti v e S a m p l e
G e n e r a t o r
B e r t E v e n t E n c od e r
P r o t o t yp e N e t w o r k
SeenEvents
Unseen Events
Ordered Contrastive Loss
𝑥
𝑖 𝑠𝑠𝑠𝑠𝑒 𝑖 𝑒𝑒𝑒𝑒
Supervised Loss 𝑝 𝑖 𝑝𝑝𝑝𝑝 2. Prompt-based prediction
Prompt-based prediction (Brown et al., 2020;
Schick and Schütze, 2021; Gao et al., 2021a) treats the NLP downstream task as a masked language modeling problem.
The language model ﬁrst generates a label word to a given prompt deﬁned by a task-speciﬁc template.
Then the label word is mapped to downstream task output space.
In this way, knowledge can be extracted from pretrained language models at low cost, which makes full use of pre-training corpus.
This is an ideal approach for event trigger word identiﬁcation in the zero-shot event detection task scenario, because it doesn’t relay on any heuristic rules or external tools as multiple approaches mention in Section 2.1,
Methodology
The architecture overview of ZEOP is shown in
Figure 3.
Given a set of seen events S and unseen events U , all samples are ﬁrst input into the contrastive sample generator as the original sample x i , where a list of multiple contrastive samples is constructed as { s 1 , . . .
, s 4 } .
Subsequently, the event encoder encodes event mentions as embedding vector e i , and the prototypical network predicts probability distribution over event types as p i .
Ordered contrastive loss is calculated for all samples, and supervised loss is calculated only for samples of seen events.
Model parameters in the event encoder and prototypical network could be updated by gradient backward.
3. Contrastive sample generator
In this paper, we aim to resolve the zero-shot event detection task.
On the one hand, seen events samples could offer supervised labels, which are ideal class-level contrastive samples.
On the other hand, there are also some unseen events samples without
Seen Type 𝑋𝑋𝑋𝑌𝑍 1 …
Dropout Sample
Rewrite Sample
Homogeneous Sample
M o s t S
i m il a r 𝐴 𝐵
Unseen Type 𝑋 𝑌 𝑍
Encoder Network Dropout
Event Mention Rewrite
L ea s t S i m il a r 𝐴𝐴𝐵𝐵 2 …
Heterogeneous Sample 𝛷 𝛷 supervised labels, which could only be used for instance-level contrastive samples. Therefore, we construct four constructive samples, including both class-level and instance-level, which are shown in Figure 4. The similarities between these contrastive and original samples differ from strong to weak.
3.1. Dropout sample
Dropout mask is proposed by (Gao et al., 2021b), which passes the same sentence to the pre-trained encoder twice.
Because the network nodes of the encoder are randomly dropped when training, a different event embedding will be obtained for the second time.
The dropout sample should be considered the most similar contrastive sample with the same input sentences.
3.1. Rewrite sample
The rewrite sample is textually edited from the original event mention.
In order to ensure the consistency of semantics, we choose the back translation (Fadaee and Monz, 2018) for event mention rewriting.
Rewrite samples should be considered the secondary similar contrastive sample because they keep the original event trigger and elements.
3.1. Homogeneous sample
Homogeneous samples are events with the homogeneous event type.
For seen events, the homogeneous samples are random sampled from events with same label { s i ∈ S
| s i .type
= x
i .type } .
For unseen types, the label of the original event is invisible for the model, so the homogeneous samples are randomly sampled from all unseen types U .
3.1. Heterogeneous sample
Heterogeneous samples are events with heterogeneous types.
For seen events, heterogeneous types [Figure] mean seen events with a different label { s i ∈
S | s i .type (cid:54) =
x i .type } .
For unseen events, the heterogeneous type means all seen types S due to the label of the original event is invisible for the model.
It should be noted that a homogeneous sample may have a different label with the original sample.
But it could be guaranteed that heterogeneous samples always have different label.
Therefore the heterogeneous samples are regarded as the least similar contrastive samples.
3. Event encoder
Following existing zero-shot event detection approaches (Huang and Ji, 2020; Zhang et al., 2021b), the best embedding feature for an event should be the contextual vector of trigger words.
The problem is how to identify the trigger words under the zero-shot setup.
Inspired by (Brown et al., 2020;
Schick and Schütze, 2021; Gao et al., 2021a), we utilize the prompt-based prediction for trigger word identiﬁcation, which doesn’t rely on any heuristic rules or external semantic analyzing tools.
3.2. Trigger word prediction
As shown in Figure 5, we use "This is event about [MASK].
<event mention>" as prompt template, where the [MASK] is the trigger word that pretrained BERT (Devlin et al., 2019) language model should predict, and the <event mention> is the text describing an event.
In speciﬁc, given a BERT input sequence t = { w 0 , w 1 , w 2 , . . .
, w L } , where w i is the i th token of the template sentence, we obtain a word distribution p m ( w m =
w i | t ) over all 𝑥 𝑝(𝑦
[Figure]
𝑖|𝑥)
[Figure]
The query vector is the result of element-wise adding the contextual vector of [CLS] and [MASK] token.
the words in the event mention.
3.2. Event type prediction
For event type prediction, we introduce prototype network (Snell et al., 2017).
It deﬁnes a prototype matrix C ∈ R n × h , where each row represents the prototype of one embedded event type
c i and h is the embedding dim of BERT.
The number of embedding types is n = k + l , where k is the number of seen event types and l is the number of unseen event types.
The value of the prototype matrix is randomly initialized and keeps updating while training the model.
As shown in Figure 6, give a query vector x , the distribution over event types is calculated by the prototypical network as follow:
[Equation]
where d ( x, c i ) is the Euclidean distance between embedding vector x and c i .
One event type may correspond to multiple trigger words in the event detection task.
If only the predicted trigger words were used as the query point, samples with the same event type would be mapped to hidden space with large distances.
So we add the contextual vector of [CLS] and [MASK] token as query vector x to balance event type and trigger words features.
3. Ordered contrastive loss
The design goal of the contrastive loss function is to narrow the distance between similar samples and push away different samples.
To better represent unseen events under the zero-shot setting, the proposed ordered contrastive learning constructs two represent contrastive samples (Dropout Sample and
Rewrite Sample), an inner-cluster contrastive sample (Homogeneous Sample), and an inter-cluster contrastive sample (Heterogeneous Sample).
Given four contrastive samples with different similarities to the original sample, we designed a novel ordered contrastive loss function by expanding the contrast loss function in (Hadsell et al., 2006).
Our model could learn the partial order relationship in similarity: Dropout Sample > Rewrite Sample > Homogeneous Sample > Heterogeneous Sample.
Thus model will distinguish between different unseen type events more signiﬁcant.
Let p 0 , p 1 , p 2 , p 3 , and p 4 be the event type probability distribution of the original, dropout, rewrite, homogeneous, and heterogeneous samples, respectively.
The distance between contrastive samples and the original sample are calculated as d 1 , d 2 , d 3 , and d 4 :
[Equation]
Considering the vectors to be compared are probability distribution, we utilize Wasserstein distance (Kolouri et al., 2019) as the distance function
W p .
Compared to Euclidean distance and cosine distance commonly used in contrastive learning,
Wasserstein distance could better measure the difference between probability distributions.
[Equation]
The similarities between contrastive samples and the original sample decrease.
Four distances may form an increasing list.
For seen events, this list is strictly increasing.
For unseen events, this list is not strictly increasing because the homogeneous sample may carry a different label.
Therefore, the ordered contrastive loss for sample x is calculated as:
[Equation]
where L m ( d x , d y ) is the margin loss:
[Table]
3. Supervised loss
Since the ﬁrst k rows of the prototype matrix correspond to the seen event types, we could also apply supervised learning for seen events in addition to the contrastive learning.
With the label and trigger word visible to the model, the negative log loss function is calculated as:
[Equation]
where ˆ y x and ˆ z x are the ground truth label of event type and trigger words, respectively.
The complete loss of ZEOP is the sum of ordered contrastive loss and supervised loss
[Equation]
Experiments 4. Implementation
We implement our model in PyTorch (Paszke et al., 2019) with Transformer Library (Wolf et al., 2020) and choose bert-base-uncased as the pre-trained language model.
For back translation, we use
Argos Translate 1 and set Chinese as the intermediate language.
For model training, we use the
AdamW(Loshchilov and Hutter, 2019) optimizer with batch size of 32, and the learning rate is grid searched in [1 e − 7 , 1 e − 4] for parameters of BERT, [1 e − 4 , 1 e − 2] for non-BERT parameters.
The margin in contrastive loss is set to 1, and the unseen event types count l is set to the actual value of the dataset.
With above settings, the total number of parameters in ZEOP is 109.53 million, of [Table] which 109.51 million (99.98%) belong to the pretrained BERT.
All the experiments are performed on a Linux server with four RTX 3090 GPUs.
The code of all experiments is available at GitHub 2 .
4. Dataset
We evaluate the proposed model on two datasets in
English.
Ace-2005 3 is a widely used (Huang et al., 2018; Huang and Ji, 2020; Zhang et al., 2021b;
Lyu et al., 2021) dataset for the event detection task.
FewShotED (Deng et al., 2020) is a dataset proposed for the few-shot event detection task.
To balance the sample count between seen and unseen types, we ﬁrst sort the event types by the decreasing order of the sample count.
Then take event types at the odd position as seen types and the even position as the unseen type.
For example, given a list of event type list t 1 , t 2 , t 3 , t 4 sorted by sample count, the t 1 and t 3 will be mark as seen event types, the t 2 and t 4 will be mark as unseen event types.
The Statistics of the processed dataset are shown in table 1.
We may assume that each sample only belong to one event type.
In comparison, the number of samples and types in the ACEdataset is smaller than FewShotED.
The problem of sampling bias is more evident on FewShotED.
These datasets will be randomly divided into training set, validation set, and test set at a ratio of 8:1:1.
4. Evaluation
We set up two tasks for evaluation: supervised event detection for seen events and zero-shot event detection for unseen events.
The f1 score will be used as the common metric for two tasks.
For seen events, the predicted labels are directly output by the model.
For unseen events, the predicted labels are mapped from model outputs by the Hungarian
Algorithm.
Additionally, following (Huang and
Ji, 2020; Zhang et al., 2021a), Normalized Mutual
Info (NMI) and Fowlkes Mallows (FM) will be used for unseen events detection to evaluate the clustering performance.
4. Baseline
Considering that the zero-shot event detection task focused in this study has no available pre-deﬁned event types, many approaches mentioned in Section 2.1 are not applicable because they require event type names or QA queries as model input.
So we use the following approaches as the experimental baselines
• SCCL (Zhang et al., 2021a):
A state-of-theart model designed for unsupervised text clustering could detect new event types from unseen event mentions.
We use the contextual vector of same candidate trigger words as SSVQ-VAE instead of [CLS] token as event mention representation to ﬁt event detection task.
• SS-VQ-VAE (Huang and Ji, 2020):
A semisupervised zero-shot event Detection model uses the variational autoencoder as regularizer.
It considers all noun and verb concepts that can be mapped to OntoNotes senses as candidate trigger words.
• BERT-OCL : We ﬁne-tune a BERT with the ordered contrastive framework proposed in this paper, where the distance between the contrastive sample and the original sample is calculated by Euclidean distance.
Once event encoding is obtained, KNN algorithm is applied to seen events detection as the classiﬁer and K-means algorithm is applied to unseen event detection as the cluster.
• ZEO :
The version of ZEOP don’t identify event trigger word by prompt-based prediction, and use the same heuristic rules as SSVQ-VAE to identify candidate trigger words.
[Table]
[Table]
4. Overall performance
The overall performance of ZEOP and all baseline approaches are shown in Table 2.
Our proposed model achieved the best overall performance on two datasets for both seen and unseen event detection tasks, except the BERT-OCL takes the lead on Normalized Mutual Info on ACE-2005, which may be due to the small number of samples in the dataset.
The prototypical network used by ZEOP needs more samples to train its prototype matrix of classes, while Bert-OCL could directly use the embedding result of pre-trained bert.
Besides, we could observe that: 1) Comparing the evaluation of
ZEO and ZEOP, although ZEO shows a slight performance advantage on F1 score on FewShotED for seen event detection task, other metrics proved that prompt-based prediction gives model better performance than heuristic rules as trigger word identiﬁer.
The SCCL and BERT-OCL show worse performance than SS-VQ-VAE on seen event detection task, which demonstrates that supervised learning is still necessary and could not be replaced by contrastive learning.
4. Ablation study
In this paper, we propose an ordered contrastive learning framework, which aims to help model learn features across instance-level and class-level.
To explore the effect of four contrastive samples, we conduct an ablation experiment by introducing them one by one to the ZEOP-woCL model, which takes no contrastive samples.
The comparison of their performance is shown in Table 3.
As all contrastive samples are added to the model, the performance improves on the unseen event detection task, but there is a performance decrease on the seen event detection task.
This demonstrates that contrastive learning does help the model to learn a better representation of unseen types, but its training objectives will conﬂict with the goal of supervised learning.
What’s more, we also validate the effect of
Wasserstein distance as the distance metric in the ordered contrastive loss.
Table 4 indicates that the
ZEOP model using Wasserstein distance performs better on unseen event detection task than ZEOPEu using Euclidean distance and ZEOP-Kl using
Kullback-Leibler divergence.
However, Euclidean distance shows better results on seen event detection task.
4. Qualitative analysis
For qualitative analysis, we visualize all unseen types samples in the dataset by t-SNE 4 .
As Figure 7 and Figure 8 show, the model with ordered contrastive learning (ZEOP and BERT-OCL) could learn a better representation of unseen types than other baselines.
The larger the number of samples a type contains, the better cluster will be achieved.
Overall, the classiﬁcation of unknown event types is signiﬁcantly more complex than known types, and only a small number of types with large sample sizes are correctly classiﬁed.
Many clusters could not be mapped one-to-one to the actual event type with the absence of supervision learning.
The model may use event elements such as subject and location as clustering features rather than the event type itself.
In future research, better event rewriting strategies can be explored to change elements unrelated to event types, replacing existing backtranslation to construct rewrite samples.
4. Hyperparameter sensitivity analysis
The number of clusters is an essential hyperparameter in the clustering problem.
In order to explore the impact of parameter unseen types l on the performance of ZEOP, we conduct a hyperparameter experiment by setting l to the 1, 2, 3, 4, and 5 times of the actual number of unseen types.
As shown in
Figure 9, ZEOP suffers a slight performance drop when unseen type numbers increase.
The model is more likely to classify the same event type into different clusters.
In practice, properly estimating the approximate number range of unknown event types will help the model to achieve a better performance.
Conclusion
In order to solve the problem of zero-shot event detection, this paper proposes an end-to-end model named Zero-Shot Event Detection Based on Ordered Contrastive Learning and Prompt-Based Prediction.
By creatively introducing multiple contrastive samples with different similarities, the contrastive loss is extended from pairwise comparison to list-wise comparison.
Therefore, the model could learn a better representation across instancelevel and class-level.
Meanwhile, the prompt-based prediction is utilized to identify event trigger words without relying on heuristic rules.
Experiments demonstrate that our method can signiﬁcantly improve the accuracy of identifying unseen event types while keeping the ability to classify seen event types.
Future research should consider the potential effects of the initialization of the prototype matrix in the prototypical network more carefully.
A better initial value may reduce the need for large training samples and speed up model training.
Acknowledgments
This work was supported by NSFC grants (No.
61972155), the Science and Technology Commission of Shanghai Municipality (20DZ1100300) and the Open Project Fund from Shenzhen Institute of
Artiﬁcial Intelligence and Robotics for Society, under Grant No. AC01202005020, Shanghai Trusted
Industry Internet Software Collaborative Innovation Center.