Temporal Common Sense Acquisition with Minimal Supervision (2020)

Abstract: Temporal common sense (e.g., duration and frequency of events) is crucial for understanding natural language. However, its acquisition is challenging, partly because such information is often not expressed explicitly in text, and human annotation on such concepts is costly. This work proposes a novel sequence modeling approach that exploits explicit and implicit mentions of temporal common sense, extracted from a large corpus, to build TacoLM, a temporal common sense language model. Our method is shown to give quality predictions of various dimensions of temporal common sense (on UDST and a newly collected dataset from RealNews). It also produces representations of events for relevant tasks such as duration comparison, parent-child relations, event coreference and temporal QA (on TimeBank, HiEVE and MCTACO) that are better than using the standard BERT. Thus, it will be an important component of temporal NLP.

## introduction time is crucial when describing the evolving world. it is thus important to understand time as expressed in natural language text. indeed, many natural language understanding (nlu) applications, including information retrieval, summarization, causal inference, and qa @xcite @xcite @xcite , rely on understanding time. however, understanding time in natural language text heavily relies on common sense inference. such inference is challenging since commonsense information is rarely made explicit in text (e.g., how long does it take to open a door?) even when such information is mentioned, it is often affected by another type of reporting bias: people rarely say the obvious, in order to communicate more efficiently, but sometimes highlight rarities @xcite @xcite @xcite . this is an even more pronounced phenomenon when it comes to temporal common sense (tcs) @xcite . in example 1, human readers know that a typical vacation is likely to last at least a few days, and they would choose "will not" to fill in the blank for the first sentence; instead, with a slight change of context "vacation" → "walk outside," people typically prefer "will" for the second one. similarly, any system which correctly answers this example for the right reason would need to incorporate tcs in its reasoning. example 1: choosing from "will" or "will not" dr. porter is now (e1:taking) a vacation and be able to see you soon. dr. porter is now (e2:taking) a walk outside and be able to see you soon. acquiring the multiple dimensions of tcs (e.g., duration and frequency) is challenging. as shown in example 1, the duration of "taking a vacation" and "taking a walk" are not expressed explicitly, so that systems are required to read between the lines to support the inference. a pre-trained language model may not handle this issue well, as it cannot identify the tcs dimensions in temporal mentions and effectively learn from them. as a result, it cannot generalize well to similar events without explicit temporal mentions. to handle this problem, we design syntactic rules that can collect a vast amount of explicit mentions of tcs from unannotated corpus such as gigaword @xcite ( §3.3). we use this data to pre-train our model so that it distinguishes different dimensions. a second challenge occurs when the text is highlighting rare and special cases. as a result, temporal mentions in natural text may follow a distorted distribution in which certain kinds of "common" events are under-represented. for instance, we may rarely see mentions of "i opened the door in 3 seconds," but we may see "it took me an hour to open this door" in text. to overcome this challenge, we exploit the joint relationship among temporal dimensions. although we rarely observe the true duration of "opening the door" in free-form text, we may see phrases like "i opened my door during the fire alarm," providing an upper-bound to the duration of the event (i.e., "opening the door" does not take longer than the alarm.) we believe that we are the first to exploit such phenomena among temporal dimensions. this paper studies several important dimensions of tcs inference: duration (how long an event takes), frequency (how often an event occurs) and typical time (when an event typically happens). 2 as a highlight, fig. 1 shows the distributions (over time units) we predict for the duration and frequency of three events. we can see that "taking a vacation" lasts from days to months while "taking a walk" lasts from minutes to hours. as shown, our model is able to produce different and sensible distributions for the "take" event, depending on the context in which "take" occurs. our work builds upon pre-trained contextual language models @xcite @xcite . however, a standard language modeling objective does not lead to a model that handles the two challenges mentioned above; in addition, other systematic issues limit its ability to handle tcs. in particular, language models do not directly utilize the ordinal relationships among temporal units. for example, "hours" is longer than "minutes," and "minutes" are longer than "seconds." foot_1 fig. 2 shows that bert does not produce a meaningful duration distribution for a set of events with a gold duration of "day" (extracted in §3.3). our proposed system, on the other hand, is able to utilize the ordinal relationships and produce unimodal distributions around the correct labels in both fig. 1 and fig. 2 . contributions. this work proposes an augmented pre-training for language models to improve their understanding of several important temporal phenomena. we address two kinds of reporting biases by effectively acquiring weak supervision from free-form text and utilizing it to learn multiple temporal dimensions jointly. our model incorporates other desirable properties of time in its objective (ordinal relations between temporal phrases, the circularity of certain dimensions, etc.) to improve temporal modeling. our experiments show 19% relative improvement over bert in intrinsic evaluations, and 5-10% improvements in most extrinsic evaluations done on three timerelated datasets. furthermore, the ablation study shows the value of each proposed component of our construction. overall, this is the first work to incorporate a wide range of temporal phenomena within a contextual language model. the rest of this paper is organized as follows. we distinguish our work with the prior work in §2. the core of our construction, including extraction of cheap supervision from raw data and augmenting a language model objective function with temporal signals, is in §3 . we conclude by showing intrinsic and extrinsic experiments in §4. on the other hand, time has long been an important research area in nlp. prior works have focused on the extraction and normalization of temporal expressions @xcite @xcite , temporal relation extraction @xcite @xcite , and timeline construction @xcite . recently, mctaco @xcite summarizes five types of tcs and the three temporal dimensions studied here are all in their proposal. 4 mctaco shows that modern nlu techniques are still a long way behind humans on tcs understanding, suggesting that further research on this topic is needed. there have been works on temporal common sense, such as event duration @xcite @xcite @xcite , typical temporal ordering @xcite , and script learning (i.e., what happens next after certain events) @xcite @xcite . those on duration are highly relevant to this work. @xcite annotates a subset of documents from timebank @xcite with "less-than-one-day" and "more-than-one-day" annotations and provides the first baseline system for this dataset. @xcite significantly improve earlier work by using additional aspectual features for this task. @xcite annotate the uds-t dataset with event duration annotations and propose a joint method that extracts both temporal relations and event durations. our approach has two notable differences from this line of work. first, we work on duration, frequency, and typical time-jointly on three dimensions of tcs, while the works above only focused on duration. second, we focus more on obtaining cheap supervision signals from unlabeled data, while these other works all have access to human annotations. with respect to harnessing cheap supervision, @xcite ; @xcite propose to mine web data using a collection of hand-designed query patterns. in contrast to our approach, they are based on counting instead of machine learning and cannot handle the contextualization of events. ## temporally focused joint learning with minimal supervision in this section, we elaborate our approach to designing and pre-training tacolm, a time-aware language model.