Improving Event Duration Prediction via Time-aware Pre-training (2020)

Abstract: End-to-end models in NLP rarely encode external world knowledge about length of time. We introduce two effective models for duration prediction, which incorporate external knowledge by reading temporal-related news sentences (time-aware pre-training). Specifically, one model predicts the range/unit where the duration value falls in (R-PRED); and the other predicts the exact duration value (E-PRED). Our best model – E-PRED, substantially outperforms previous work, and captures duration information more accurately than R-PRED. We also demonstrate our models are capable of duration prediction in the unsupervised setting, outperforming the baselines.

## introduction understanding duration of event expressed in text is a crucial task in nlp @xcite . it facilitates downstream tasks such as story timeline construction @xcite and temporal question answering @xcite . it is challenging to make accurate prediction mainly due to two reasons: (1) duration is not only associated with event word but also the context. for example, "watch a movie" takes around 2 hours, while "watch a bird fly" only takes about 10 seconds; (2) the compositional nature of events makes it difficult to train a learning-based system only based on hand annotated data (since it's hard to cover all the possible events). thus, external knowledge and commonsense are needed to make further progress on the task. however, most current approaches @xcite @xcite focus on developing features and cannot utilize external textual knowledge. the only exception is the web count based method proposed by @xcite , which queries search engine with event word (e.g., "watch") and temporal units, and make predictions based on hitting times. however, this method achieves better performance when query only with the event word in the sentence, which means it does not enable contextualized understanding. to benefit from the generalizability of learningbased methods and utilizing external temporal knowledge, we introduce a framework, which includes (1) a procedure for collecting durationrelated news sentences, and automatic labeling the duration unit in it (section 2.1); foot_0 (2) two effective end-to-end models that leverage external temporal knowledge via pre-training (section 2.2). specifically, our first model (r-pred) predicts the most likely temporal unit/range for the event, with a classification output layer; and the other model (e-pred) predicts the exact duration value, with a regression output layer. our best model (e-pred) achieves state-of-the-art performance on the time-bank dataset and the mctaco duration prediction task. in addition, in the unsupervised setting, our model (e-pred) trained with only collected web data outperforms the supervised bert baseline by 10.24 f1 score and 9.68 exact match score on mc-taco duration prediction task. we also provide detailed comparisons and analysis between the regression objective (e-pred) and the classification objective (r-pred). ## additional related work for supervised duration prediction, @xcite annotates duration length of a subset of events in timebank @xcite . new features and learning based models are proposed for timebank @xcite @xcite . in particular, aspectual @xcite features have been proved to be useful. concurrent to our work, zhou et al. ( 2020 ) also utilize unlabeled data. different from our work, they focus on temporal commonsense acquisition in a more general setting (for frequency, typical time, duration, etc.) and the models predict the discrete temporal unit, while we propose two models (classification and regression-based). in addition, they focus on providing better representation instead of directly generating duration prediction. for the unsupervised setting, @xcite ; @xcite use rule-based method on web data and generate collections of mapping from verb/event pattern to numeric duration value. @xcite develop queries for search engines and utilize the returned snippets / hitting times to make prediction. ## conclusion we propose a framework for leveraging free-form textual knowledge into neural models for duration prediction. our best model (e-pred) achieves state-of-the-art performance in various tasks. in addition, our model trained only with externallyobtained weakly supervised news data outperforms supervised bert_qa baseline by a large margin. we also find that model trained with exact duration value seems to better capture duration nuance of event, and has more tunable range that is more flexible to make prediction for quantitative attributes such as duration.