Generic Temporal Reasoning with Differential Analysis and Explanation (2023)

Abstract: Temporal reasoning is the task of predicting temporal relations of event pairs. While temporal reasoning models can perform reasonably well on in-domain benchmarks, we have little idea of these systems’ generalizability due to existing datasets’ limitations. In this work, we introduce a novel task named TODAY that bridges this gap with temporal differential analysis, which as the name suggests, evaluates whether systems can correctly understand the effect of incremental changes. Specifically, TODAY introduces slight contextual changes for given event pairs, and systems are asked to tell how this subtle contextual change would affect relevant temporal relation distributions. To facilitate learning, TODAY also annotates human explanations. We show that existing models, including GPT-3.5, drop to random guessing on TODAY, suggesting that they heavily rely on spurious information rather than proper reasoning for temporal predictions. On the other hand, we show that TODAY’s supervision style and explanation annotations can be used in joint learning, encouraging models to use more appropriate signals during training and thus outperform across several benchmarks. TODAY can also be used to train models to solicit incidental supervision from noisy sources such as GPT-3.5, thus moving us more toward the goal of generic temporal reasoning systems.

## introduction temporal relation extraction @xcite is traditionally viewed as an information extraction task, where a model uses explicit temporal signals such as "happened before" to identify the temporal order of events. while these models have contributed to many downstream pipelines, they are not enough for more complicated tasks such as timeline generation, where most event pairs do not come with explicit signals. these implicit temporal relation extractions @xcite thus require temporal reasoning, which relies on both common sense and semantic understanding of the context. in recent works, a popular approach to address these predictions is to finetune pre-trained language models (plms) with annotated supervision data. unfortunately, existing temporal benchmarks @xcite @xcite only annotate hard labels and ignore the fact that temporal labels can often be soft and nondeterministic. this approach allows models to exploit spurious signals and annotation artifacts easily for performance. for example, a model may learn to predict "lunch" before "dinner" regardless of the surrounding context, yet most existing benchmarks will not challenge such beliefs because most "lunch" annotations will happen to be before "dinner." this is not always the case though, e.g. if the "lunch" and "dinner" were today's lunch and yesterday's dinner, and we know that yesterday's dinner must happen before today's lunch. this means that the current high performances of existing models may be misleading, and the community may actually possess an inaccurate perception of models' capacity to generalize. in this work foot_0 , we bridge this evaluation gap with a novel benchmark that evaluates whether a temporal reasoning model is making the correct predictions for the right reasons by properly identifying potential alternatives (e.g., "dinner" can be before "lunch" under certain contexts). our intuition is that a model with good temporal generalizability should be able to understand the effect of subtle context changes and explain how the change will shift the temporal relation distribution of an event pair. to evaluate this, we propose the framework called temporal differential analysis. under this setting, we select event pairs where the temporal relation is not 100% deterministic based on the context, meaning that both before/after relations are possible if additional information in regard to the context is given. then, we annotate a hypothetical change in the form of an additional sentence added to the beginning of the context. as fig. 1 shows, this context change will shift the event pair's temporal relation distribution, making it either "more before" or "more after". each hypothetical change is also annotated with human explanations of why the change affects the temporal relation. we collect 2,241 such instances with a rigorous human annotation pipeline and call the resulting dataset today (temporal differential analysis). we find that models that achieve relatively high in-domain test performances are brittle and demonstrate minimal capabilities for differentiating subtle context changes that affect temporal relations. for example, the patterntime model @xcite that achieves 77% binary accuracy on tracie @xcite drops dramatically to 54% on today, which is barely above random guessing. to mitigate this gap, we propose a general jointlearning technique that uses temporal explanations that today annotates. specifically, we argue that explanations of temporal relations are an excellent proxy for understanding temporal reasoning. we show models trained with today's task formulation and explanation annotation are better at perceiving cross-dataset supervision and achieve superior performances on multiple datasets with a single model. we also find that while large language models (llms) are not good enough for temporal differ-ential analysis, they do sometimes produce reasonable explanations for a given temporal relation. we design a pipeline that automatically collects supervision signals based on this finding. the pipeline starts with giving gpt-3.5 @xcite both an instance from today and a hypothetical temporal relation, and then uses gpt-3.5 to generate several explanations. finally, we train an explanation verifier based on today's human annotations, which selects the generated explanations that are more likely to be plausible. we show that adding such explanations from gpt-3.5 further boosts the performance across our benchmarks. our contributions are threefold: 1) we design a novel evaluation framework and collect a new dataset today that uses differential analysis to test whether systems can perform temporal reasoning with the right reasons; 2) we show that today's supervision, especially the use of explanations, contributes toward a generic temporal reasoning model; 3) we use llms to generate pseudo explanations and filter these with a novel explanation verification system to show that such incidental supervision signals are helpful. ## dataset in this section, we introduce the evaluation framework and collection process of today. ## modeling in this section, we show how to fully use today's supervision signals (especially the explanations) to build a more generic temporal reasoning model. joint learning. today annotates temporal distribution shifts instead of absolute relations. this means that an instance may have a gold label "before" (i.e., the additional sentence as makes the relation more "before" compared to the original context), yet the likelihood of "after" can still be higher, and the argmax label will be "after". as a result, a model cannot sufficiently learn to predict absolute labels with only supervision signals from today. to mitigate this issue, we propose a joint learning model that requires joint supervision from a dataset that annotates hard labels for temporal relations, such as matres or tracie. modeling. we adopt tracie's formulation @xcite to format temporal reasoning into textual entailment and use a seq-to-seq pre-trained language model as the base model. specifically, the input sequence consists of the premise, which is as +c +exp foot_2 in our case, as well as the hypothesis, which is e 1 starts [r] e 2 . here, r is a hypothetical relation we plug into the hypothesis since systems are unaware of the gold label from the input sequence. the output sequence contains an entailment label, which is either answer: positive for entail or answer: negative for contradiction. hard label instances. as we note above, a system does not know the gold label when plugging in the hypothetical relation in the hypothesis. as a result, at learning time, we construct two entailment instances for a temporal relation instance with an absolute hard label. the first instance uses a hypothesis that is e 1 starts before e 2 . we want the model to learn to output answer: positive for entail if the gold label is also "before", or answer: negative for contradiction if the gold label is "after". the second instance uses e 1 starts after e 2 as the hypothesis, where the output sequences are reversed compared to the first one. we use the regular cross-entropy loss for optimization and denote the loss as ℓ ce . at test time, we similarly construct two entailment instances for each event pair and conduct a simple probability-based vote to infer a final "before/after" relation. ## llm incidental supervision as we hypothesize and later show in §6, humanannotated explanations greatly benefit generic temporal reasoning models, as they encourage models to learn to use the correct signals. however, it is extremely difficult and expensive to crowdsource such explanations for training purposes since collecting one instance costs $1 on average. on the other hand, large language models (llms) can produce a large amount of generated explanations at a much cheaper cost. unfortunately, these generated explanations are mostly unusable as they are simply model guesses based on textual correlations. in this section, we introduce a knowledge distillation method that combines the benefits of both human annotations and llm generations by training verification models based on our seed annotation, which is then used to select generations more likely to be plausible. compared to previous work @xcite , we propose a verification system composed of multiple models that individually verify different aspects of automatically-generated explanations. we detail our pipeline below. ## experiment in this section, we conduct a series of experiments to show that 1) existing systems do not truly understand temporal relations, 2) today and incidental supervision signals partially address this issue, and 3) today motivates future work towards generic temporal reasoning. ## conclusion we introduce a novel differential analysis framework and dataset called today that interprets and evaluates if a temporal model can make correct predictions without using spurious information and biases. we show that existing temporal models' performances drop to random guessing on today due to model limitations and supervision biases. to address this issue, we propose to jointly train with today and its explanation annotations, resulting in improved performances on multiple temporal reasoning benchmarks, namely tracie (+7%), matres (+3%), and today (+10%). we also demonstrate that today can be used to distill gpt-3.5 and automatically generate and filter incidental supervision instances with high-quality explanations, which further improves performances. despite these advances, the gap in performance on today still motivates future work toward generic temporal reasoning.