Improving Event Temporal Relation Classification via Auxiliary Label-Aware Contrastive Learning (2022)

Abstract: “Event Temporal Relation Classification (ETRC) is crucial to natural language understanding. In recent years, the mainstream ETRC methods may not take advantage of lots of semantic information contained in golden temporal relation labels, which is lost by the discrete one-hot labels. To alleviate the loss of semantic information, we propose learning Temporal semantic information of the golden labels by Auxiliary Contrastive Learning (TempACL). Different from traditional contrastive learning methods, which further train the PreTrained Language Model (PTLM) with unsupervised settings before fine-tuning on target tasks, we design a supervised contrastive learning framework and make three improvements. Firstly, we design a new data augmentation method that generates augmentation data via matching templates established by us with golden labels. Secondly, we propose patient contrastive learning and design three patient strategies. Thirdly we design a label-aware contrastive learning loss function. Extensive experimental results show that our TempACL effectively adapts contrastive learning to supervised learning tasks which remain a challenge in practice. TempACL achieves new state-of-the-art results on TB-Dense and MATRES and outperforms the baseline model with up to 5.37%F1 on TB-Dense and 1.81%F1 on MATRES.”

## introduction the temporal relations of events are used to describe the occurring sequence of events in an article. therefore understanding the temporal relations of events in articles is useful for many downstream tasks such as timeline creation @xcite , generating stories @xcite , forecasting social events @xcite , and reading comprehension @xcite . hence, the etrc task is an important and popular natural language understanding research topic among nlp community. the etrc task is to determine the occurrence sequence of a given event pair. the context of the event pair is usually given to aid judgment. @xcite first encoded the event pairs into embedded representations and then used fully connected layers as a classifier to generate confidence scores for each category of temporal relations. all related works of the nlp community since then have followed the classification view: classifying the embedded representations. naturally, we can encode the context and events into a better embedding space in which the different relations are distinguished well, to get better classification results. traditionally, all recent works use one-hot vectors to represent golden temporal relation labels in the training stage. however, the one-hot vector reduces the label with practical semantics to the zero-one vector. it makes the embedded representations extracted by the etrc models waiting for classifying be the similarities of the instances with the same label. but, the similarities are not equal to the label seman-tics, and lead to arbitrary prediction and poor model generalization, especially for confused instances. in brief, the one-hot vectors which represent temporal relation categories lose much semantic information. to cope with the loss of semantic information in golden labels, we propose to learn the lost semantic information by contrastive learning, which is well confirmed and most competitive method for learning representations under unsupervised settings, so that the etrc model can obtain better event representations. however, effectively adapting contrastive learning to supervised learning tasks remains a challenge in practice. general methods such as @xcite , which continue to train the ptlm model using unsupervised contrastive learning on the input texts (without labels) from the target task before finetuning, apply contrastive learning to supervised representation learning mechanically. they discard the category information in the process of further training. in the supervised etrc task, we want the event pair representations with the same category to be as close as possible without collapsing. but direct application of the unsupervised contrastive learning loss function would prevent them from getting closer, because it discard the category information. it's an inherent problem of self-supervised contrastive learning. so the standard contrastive learning is not natural for the supervised etrc task. to solve this problem we designed label-aware contrastive learning loss and design a new contrastive learning framework. additionally, we argue that we can do contrastive learning in the intermediate layers of the ptlm as same as the last layer simultaneously. in a cascade structure, a change in previous layers affects the subsequent layers and continuous positive changes will make the learning process easier. hence, we propose patient contrastive learning and design three patient strategies. overall, we propose tempacl: firstly, we manually construct templates based on the semantics of labels and get augmentation sentences by matching the labels of instances. secondly, we train the encoder of key samples which are necessary for contrastive learning by the augmentation datasets established by the etrc datasets and the augmentation sentences. thirdly, we jointly train the etrc model with cross entropy loss and label-aware contrastive learning loss using a patient contrastive learning strategy. the main contributions of this paper can be summarized as follows: ## our baseline model our baseline model is comprised of an encoder and a classifier. we use roberta @xcite as our encoder and use two fully connected layers and a tanh activation function between them as our classifier. recently, most of the related works use roberta as an encoder, because roberta can achieve better results on the etrc task than bert in practice. each instance is composed of an event temporal triplet t (i.e. ( < e 1 >, < e 2 >, r ), @xmath0. ) and the context s of the events which may be a single sentence or two sentences. we first tokenize the context and get a sequence of tokens x [0,n) with length n. then we feed the x [0,n) into roberta. one event mention may correspond to multiple tokens, so we send the token embeddings corresponding to these tokens to an average pooling layer to get the final event representation e i . next, we combine e 1 and e 2 into a classification vector e 1 ⊕ e 2 , where ⊕ is used to denote concatenation. finally, we feed the classification vector into the classifier followed by a soft-max function to get confidence scores for each category of temporal relations. examples are from the same original data example or not, and learn the representation network by solving this task. he et al. (2020) formulate contrastive learning as a dictionary look-up problem and propose an effective contrastive loss function l cl with similarity measured by dot product: ## self-supervised contrastive learning in this paper, we follow this part of their work and transfer it to the supervised etrc task. ## tempacl approach in this section, we introduce our tempacl approach in details and draw the overall process of tempacl in figure encoder k used for encoding semantic information of golden temporal relation labels, and then jointly train the baseline model with auxiliary contrastive learning via the label-aware contrastive learning loss function and a patient strategy. specially, we fix the parameters of the encoder k in the joint training stage. ## experiments and results in this section, we perform experiments on tb-dense and maters and prove our tempacl performs better than previous state-of-the-art methods. details on the datasets, experimental setup, and experimental results are provided in the following subsections. 2018 ) design a novel multi-axis (i.e., main, intention, opinion and hypothetical axes) annotation scheme to further annotate the 275 documents. there are only 4 temporal relations (before, after, equal and vague) different from tb-dense and the equal is the same as simultaneous. we follow the official split strategy that uses timebank and aquaint for training and platinum for testing. we also follow the previous works @xcite that randomly select 20 percents of the official train documents as dev set. we briefly summarize the data statistics for tb-dense and matres in ## conclusion in recent years, the mainstream etrc methods focus on using discrete values to represent temporal relation categories and lose too much semantic information contained in golden labels. so we propose c c l 2 0 2 2 tempacl, which makes the etrc model learn the lost semantic information in golden labels via contrastive learning. extensive experiments prove the contrastive learning framework in tempacl is more suitable for the supervised etrc task than traditional self-supervised contrastive learning. the patient contrastive learning strategy designed by us provides more useful positive and negative key samples and reduces the difficulty of contrastive learning. the label-aware contrastive learning loss designed by us avoids the negative interactions between different queries and keys in the same category, which is an inherent problem of self-supervised contrastive learning.