Fine-grained Temporal Relation Extraction with Ordered-Neuron LSTM and Graph Convolutional Networks (2021)

Abstract: Fine-grained temporal relation extraction (FineTempRel) aims to recognize the durations and timeline of event mentions in text. A missing part in the current deep learning models for FineTempRel is their failure to exploit the syntactic structures of the input sentences to enrich the representation vectors. In this work, we propose to fill this gap by introducing novel methods to integrate the syntactic structures into the deep learning models for FineTempRel. The proposed model focuses on two types of syntactic information from the dependency trees, i.e., the syntax-based importance scores for representation learning of the words and the syntactic connections to identify important context words for the event mentions. We also present two novel techniques to facilitate the knowledge transfer between the subtasks of FineTempRel, leading to a novel model with the state-of-the-art performance for this task.

## introduction an important step in event understanding involves identifying the temporal relations between events (i.e., temprel), finding its applications in different natural language processing (nlp) systems such as question answering and timeline construction. a large volume of the prior works has focused on the classification setting for this problem where categorical temporal relations should be predicted for pairs of event-referring and/or time-referring expressions in text (i.e., categorical temprel) @xcite @xcite . for instance, in the sentence "the meeting to discuss the possible merger of the two financial companies lasted for two hours, eventually leading to their union yesterday.", a system for temprel should be able to realize that the "discussion" event happens before the "union" event (i.e., the categorical label of before). ## related work most of the previous work on temprel has focused on the categorical setting using the timeml standard for the datasets (i.e., timebank, timebank-dense, richer event description (red)) @xcite @xcite @xcite @xcite . @xcite is the first work to consider the fine-grained distinction for the temporal relations for events. regarding temprel methods, the early approaches has involved feature-based models @xcite @xcite , the hybrid methods (d'souza and ng, 2013), sieve-based methods (i.e., caevo @xcite , catena @xcite ), structured learning methods @xcite , and interger linear programming @xcite . recently, deep learning models have been developed and shown promising results for temprel @xcite @xcite @xcite . the closet work to ours is @xcite that presents an attentionbased deep learning model for finetemprel; however, it does not capture the syntactic structures of the sentences as we do in this work. some previous works have also considered event duration modeling in text @xcite @xcite although they do not tie duration and temporal relations as we do. finally, we also note related tasks that concern other types of relations between events/entities, including event coreference resolution @xcite @xcite , event causaltiy identification @xcite , and event argument extraction @xcite . ## model finetemprel can be formulated as a regression problem. formally, given a sentence w of n words: @xmath0, w 2 , . . . , w n with w m 1 and w m 2 (1 ≤ m 1 < m 2 ≤ n ) as the trigger words for the two event mentions of interest, we need to predict the start and end times for the two event mentions to reflect the temporal order and durations of the events in the timeline. following @xcite , we aim to predict the start and end times b 1 and e 1 for w m 1 and b 2 and e 2 for w m 2 using a reference interval [0, 1] (i.e., 0 ≤ b 1 ≤ e 1 ≤ 1 and 0 ≤ b 2 ≤ e 2 ≤ 1). also, let d 1 and d 2 be the durations for w m 1 and w m 2 respectively. note that as the input event mentions might belong to different sentences in finetemprel, we follow @xcite to combine these sentences into a single sequence of words for w in those cases. to prepare the input sentence for the deep learning models and to achieve a fair comparison with @xcite , we first send w into the pre-trained language model elmo @xcite to produce a sequence of hidden vectors @xmath1, x 2 , . . . , x n for w . note that the hidden vector x i for w @xmath2. ## experiments in addition, for the te3 and td datasets with the transfer learning experiment, similar to @xcite , we employ the sklearn 0.20.0 package to train the svm classifier with gaussian kernel and the same hyper-parameters. finally, we use the same procedure as in @xcite to pre-process the datasets in this experiment (e.g., using the stanford corenlp 3.9.2 toolkit for data pre-processing). in addition, similar to @xcite , we further examine the models on the categorical temprel datasets, including the tempeval3 (te3, @xcite , task c-relation only) and timebank-dense (td, @xcite ) datasets. we follow the same preprocessing procedure for these datasets and the same training and tuning procedures for the models as in @xcite to achieve a fair comparison. in particular, we use a transfer learning approach where the best-performing model on the uds-t development set is first used to obtain the overall representa- for each pair of annotated event-event relations in te3 and td. afterward, we employ this vector as the input features for a svm classifier with a gaussian kernel that is trained on the training sets and evaluated on the test sets for these datasets. specifically, the training set for te3 involves the union of the timebank @xcite and aquaint @xcite datasets provided in the te3 workshop @xcite while the training data and development data in td (i.e., td-train, td-dev) are combined to form the training set for td. we follow the same hyper-parameter tuning process for the svm classifier as in @xcite for these datasets where a grid-search with 4-fold cross-validation is done over the training sets. finally, the performance of the models is reported on the test sets of the datasets (i.e., te3-platinum (te3-pt) for te3 and td-test for td). for the performance measures for finetemprel on the uds-t dataset, following @xcite , we use three metrics for the duration prediction task (i.e., spearman correlation (ρ), mean rank difference (rd), and the proportion of rank difference explained (r1(rd))), and three metrics for temporal relation prediction (i.e., spearman correlation between the normalized values of the actual start and end times and the predicted ones (absolute ρ), the spearman correlation between the actual and predicted values for l rel (relative ρ), and the proportion of mean absolute error (mae) explained (r1(mae))). note that for a performance metric a, the proportion of a explained is computed by: r1(a) = 1 -a model /a baseline where a model is the performance metric a computed for the model and a baseline is those computed for the model that always guesses the median. as we can see, using the same elmo embeddings, the proposed model on-lstm-gcn significantly outperforms all the models in @xcite with substantial performance gap over different performance metrics and the two subtasks (i.e., event duration prediction and temporal relation prediction). this clearly demonstrates the effectiveness of the proposed model for finetem-prel. we also see that replacing elmo with the bert embeddings can help to improve the performance for both subtasks, suggesting the application of bert for finetemprel in the future research. on-lstm: first, we consider the following variations for the on-lstm component: (i) "-l kl ": this is the on-lstm-gcn model where the syntax-model consistency loss l kl is not included in the overall loss function, (ii) "-on-lstm": this model completely removes the on-lstm component from on-lstm-gcn (so the consistency loss l kl is not used and the input vector sequence x is directly sent to the gcn models), and (iii) "repw/ lstm": this model replaces on-lstm with the traditional lstm model in on-lstm-gcn (the l kl loss is also not employed in this case as lstm does not support the neuron hierarchy for the model-based importance scores). gcn: second, for the gcn component, we evaluate the following variations for on-lstm-gcn: (i) "-gcn interact": this model does not apply the interaction mechanism for the two gcn models in equations 5 and 6 for on-lstm-gcn; it instead uses equation 4 for the computation of both gcn models, (ii) "-l dif ": this model eliminates the regularization loss l dif from on-lstm-gcn, (iii) "-interact -l dif ": this model removes both the gcn interaction mechanism in equations 5 and 6 and the loss l dif from on-lstm-gcn (but it still has two gcn models), (iv) "one gcn": instead of using two gcn models for the two subtasks of finetemprle, this model only utilize a single gcn model for both tasks (the gcn interaction in equations 5 and 6 and the regularization loss l dif are thus not used in this case as well), and (v) "-gcn": this model completely removes the gcn component (thus also excluding the gcn interaction and the loss l dif ); the on-lstm hidden vectors h are directly exploited to perform the duration and relation predictions in this case. ## conclusion we introduce a novel deep learning model for finetemprel that exploits the syntactic structures of the sentences to improve the representation vectors. we focus on two type of syntactic information for finetemprel in this work, i.e., the syntaxbased importance scores for the representations of the words that are injected into the models via on-lstm, and the dependency connections between the words that are exploited in gcn models. two novel transferring learning methods are presented for the two gcn models for duration and relation predictions. comprehensive experiments are performed to demonstrate the advantages of the proposed model for finetemprel.