E xploring C ontextualized N eural L anguage M odels for T emporal D ependency P arsing (2020)

Abstract: Extracting temporal relations between events and time expressions has many applications such as constructing event timelines and time-related question answering. It is a challenging problem which requires syntactic and semantic information at sentence or discourse levels, which may be captured by deep contextualized language models (LMs) such as BERT (Devlin et al., 2019). In this paper, we develop several variants of BERT-based temporal dependency parser, and show that BERT significantly improves temporal dependency parsing (Zhang and Xue, 2018a). We also present a detailed analysis on why deep contextualized neural LMs help and where they may fall short. Source code and resources are made available at https://github.com/bnmin/tdp_ranking .

## introduction temporal relation extraction has many applications such as constructing event timelines for news articles or narratives as well as time-related question answering. recently, @xcite presented temporal dependency parsing (tdp), which organizes time expressions and events in a document to form a temporal dependency tree (tdt). given a previous step which detects time expressions and events, tdp extracts the temporal structure between them. consider this example: example 1: kuchma and yeltsin signed a cooperation plan on february 27, 1998. russia and ukraine share similar cultures, and ukraine was ruled from moscow for centuries. yeltsin and kuchma called for the ratification of the treaty, saying it would create a "strong legal foundation". figure 1 shows the corresponding tdt. compared to previous pairwise approaches for temporal a previous version is available at https://arxiv. org/abs/2004.14577 . recently, deep language models such as bert @xcite have been shown to be successful at many nlp tasks, because (1) they provide contextualized word embeddings that are pretrained with very large corpora, and (2) bert in particular is shown to capture syntactic and semantic information @xcite , which may include but is not limited to tense and temporal connectives. such information is relevant for tdp. in this paper, we present bert-based tdp models, and empirical evidence demonstrating that bert significantly improves tdp. we summarize the contributions of this paper as follows: ## related work much previous work has been devoted to classification and annotation of relations between events and time expressions, notably timeml @xcite and timebank @xcite , as well as many extensions of it (see derczynski, 2017 for an overview). timeml annotates all explicit relations in the text; at the extreme, timebank-dense @xcite annotates all n 2 pairs of relations. pair-wise annotation has three problems: o(n foot_1 ) complexity; the possibility of inconsistent predictions such as a before b, b before c, c before a; and forcing annotation of relations left unclear by the document. while extracting time expressions and events is well handled (e.g. @xcite , relating them is still a challenging task. previous research on extracting these relations (e.g. @xcite @xcite almost always uses pair-wise timemlannotated data which has rich annotation but also inherits the above three complexity and consistency issues. to address these issues, zhang and xue (2018b) present a tree structure of relations between time expressions and events (tdt), along with a bilstm model @xcite for parsing text into tdt and a crowd-sourced corpus @xcite . organizing time expressions and events into a tree has a number of advantages over traditional pair-wise temporal annotation. it reduces the annotation complexity to o(n) and avoids cyclic inconsistencies both in the annotation and the model output. despite the apparent reduction in labeled edges, many additional edge labels can be deduced from the tree: in figure 1 , we can deduce e.g. that ruled is before share because ruled is before dct but share overlaps dct. a final advantage of tdts is that they allow underspecification where the source document does not explicitly specify an order, such as the relation between signed and called (likely to be overlap, but it is not certain). @xcite is currently the only englishlanguage tdp corpus, comprising 196 newswire articles. in addition, this paper capitalizes on the now well-documented recent advances provided by bert @xcite . besides offering richer contextual information, bert in particular is shown to capture syntactic and semantic properties @xcite relevant to tdp, which we show yield improvements over zhang and xue's original model. ## bert-based neural models for temporal dependency parsing following zhang and xue (2018a), we transformed temporal dependency parsing (tdp) to a ranking problem: given a child mention (event or time expression) x i extracted by a previous system, the problem is to select the most appropriate parent mention from among the root node, dct or an event or time expression from the window x i-k , . . . , x i , . . . , x i+m 2 around x i , along with the relation label (before, after, overlap, depends on). that is, for each x j in the window, the model judges the child-parent candidate pair x i , x j . a temporal dependency tree (tdt) is assembled with an incremental algorithm which selects, for each event and time expression in sequence in the document, the highest-ranked prediction parent, relation type . the tree structure is enforced by selecting the highest probability parent which does not introduce a cycle foot_2 . we developed three models that share a similar overall architecture (figure 2 ): the model takes a pair of mentions (child and potential parent) as input and passes each pair through an encoder which embeds the nodes and surrounding context into a dense representation. all models use the same window approach described above to source parent candidates. following zhang and xue (2018a), linguistic features are concatenated onto the dense representation, which is then passed to a feed-forward layer and a softmax function to generate scores for each relation label for each pair. we developed three types of encoder: ## experiments we use the training, development and test datasets from @xcite for all experiments (182 train / 5 development / 9 test documents, total 2084 sentences). the documents in the datasets are already annotated with events and temporal expressions. this allows us to focus on evaluating the task of constructing temporal dependency trees. we evaluated four configurations of the encoders above. firstly bilstm (re-implemented) reimplements zhang and xue (2018a)'s model foot_4 in tensorflow @xcite for fair comparison. replacing its randomly-initialized embeddings with glove @xcite yields bilstm-glove. we also test the models bilstm-bert and bert-ft as described in section 3. we used adam (kingma and ba, 2014) as the optimizer and performed coarse-to-fine grid search for key parameters such as learning rate and number of epochs using the dev set 6 . we observed that when fine-tuning bert in the bert-ft model, a lower learning rate (0.0001) paired with more epochs (75) achieves top performance, compared to using learning rate 0.001 with 50 epochs for the bilstm models. we used nvidia tesla p100 gpus for training the models. on a single gpu, one epoch takes 7.5 minutes for the bert-ft model and 0.8 minutes for the bilstm-bert model. bilstm-bert outperforms the re-implemented bilstm model by 6 points and bilstm-glove by 3 points in f1-score, respectively. this indicates that the frozen, pre-trained bert embeddings improve temporal relation extraction compared to either kind of non-contextualized embedding. finetuning the bert-based encoder (bert-ft) resulted in an absolute improvement of as much as 13 absolute f1 points over the bilstm re-implementation, and 8 f1 points over the reported results in @xcite . this demonstrates that contextualized word embeddings and the bert architecture, pre-trained with large corpora and fine-tuned for this task, can significantly improve tdp. we also calculated the models' accuracies on time expressions or events subdivided by their type of parent: dct, a time expression other than dct, or another event. difficult categories are children of dct and children of events. we see that the main difference between bilstm and bilstm-bert is its performance on children of dct: with bert, it scores 0.48 instead of 0.38. conversely bert-ft all configurations for all models. ## analysis why bert helps: a detailed manual comparison of the dependency trees produced by the different models for articles in the test set shows bert's advantages for tdp. the following phenomena are attested by many sentences in many documents and correspond to known properties of bert. firstly, unlike bilstm, bert-ft is able to properly relate time expressions occurring syntactically after the event, such as kuchma and yeltsin signed a cooperation plan on february 27, 1998 in example 1. (bilstm falsely relates signed to the "previous" time expression dct). this shows bert's ability to "look forward" with its self-attention, attending to parents appearing after the child. secondly, bert-ft is able to capture verb tense and use it to determine the correct relation for both dct and chains of events. for example, it knows that present tense (share similar cultures) overlaps dct, while past perfect events (was ruled from moscow) happen either before dct or before the event adjacent (salient) to them. thirdly, bert-ft captures syntactic constructions with implicit temporal relations such as reported speech and gerunds (e.g. in example 1, yeltsin and kuchma called for the ratification [. . . ], saying it would create . . . , it identifies that called and saying overlap and create is after saying). similarly, bert's ability to handle syntactic properties @xcite such as embedded clauses may allow it to detect the direction of connectives such as since. while all models may identify the matrix clause verb as the correct parent, bert-ft is much more likely to choose the correct label. (bilstm almost always chooses 'before' for dct or 'after' for children of events, ignoring the connective.) lastly, both bert-ft and bilstm-bert are much better than the bilstm at identifying context changes (new "sections") and linking these events to dct rather than to a time expression in the previous sections (evidenced by the scores on children of dct). because bert's word embeddings use the sentence as context, the models using bert may be able to "compare" the sentences and judge that they are unrelated despite being adjacent. equivalent tdp trees: in cases where bert-ft is incorrect, it sometimes produces an equivalent or very similar tree (since relations such as overlap are transitive, there may be multiple equivalent trees). future work could involve developing a more flexible scoring function to account for this. limitations: there are also limitations to bert-ft. for example, it is still fooled by syntactic ambiguity. consider this example: example 2: foreign ministers agreed to set up a panel to investigate who shot down the rwandan president's plane on april 6, 1994. a human reading this sentence will infer based on world knowledge that april 6, 1994 should be attached to the embedded clause (who shot down), not to the matrix clause (agreed), but a syntactic parser would produce both parses. bert-ft incorrectly attaches agreed to april 6, 1994: even bert's contextualized embeddings are not sufficient to identify the correct parse. ## conclusion and future work we present two models that incorporate bert into temporal dependency parsers, and observe significant gains compared to previous approaches. we present an analysis of where and how bert helps with this challenging task. for future research, we plan to explore other types of deep neural lms such as transformer-xl @xcite and xlnet @xcite . as discussed in section 5, we also plan to develop a more flexible scoring function which can handle equivalent trees. finally, we plan to evaluate bert-ft on other temporal relation datasets as part of a larger pipeline, which will include a mapping between tdts and other temporal relation annotation schemas such as the tempeval-3 dataset @xcite .