TIMEDIAL : Temporal Commonsense Reasoning in Dialog (2021)

Abstract: Everyday conversations require understanding everyday events, which in turn, requires understanding temporal commonsense concepts interwoven with those events. Despite recent progress with massive pre-trained language models (LMs) such as T5 and GPT-3, their capability of temporal reasoning in dialogs remains largely under-explored. In this paper, we present the first study to investigate pre-trained LMs for their temporal reasoning capabilities in dialogs by introducing a new task and a crowd-sourced English challenge set, TimeDial. We formulate TimeDial as a multiple choice cloze task with over 1.1K carefully curated dialogs. Empirical results demonstrate that even the best performing models struggle on this task compared to humans, with 23 absolute points of gap in accuracy. Furthermore, our analysis reveals that the models fail to reason about dialog context correctly; instead, they rely on shallow cues based on existing temporal patterns in context, motivating future research for modeling temporal concepts in text and robust contextual reasoning about them. The dataset is publicly available at https://github.com/google-research-datasets/timedial .

## introduction humans can effortlessly reason about temporal concepts of everyday events such as their duration, frequency, or relative ordering @xcite based on rich commonsense knowledge about how the world works, especially in relation to time. however, reasoning about such concepts has been challenging for machines @xcite since it requires both understanding the local temporal expressions and reasoning about their global contexts such as their relative ordering and relations ( @xcite @xcite . the problem becomes even more challenging in dialogs, where explicit and implicit inter-dependencies among temporal concepts can appear across conversation turns. for instance, for the first dialog in although previous works have studied temporal reasoning in natural language, they have either focused on specific time-related concepts in isolation, such as temporal ordering and relation extraction @xcite , and/or dealt with limited context, such as single-sentence-based question answering @xcite and natural language inference @xcite . in this work, we make the first systematic study of temporal commonsense reasoning in a multi-turn dialog setting. the task involves complex reasoning that requires operations like comparison and arithmetic reasoning over temporal expressions and the need for commonsense and world knowledge. we design a new task for dialog-based temporal reasoning and present a new challenge set in english, called timedial, to evaluate language understanding models on the task. we formulate the problem as a crowd-sourced cloze task with multiple choices based on dialogs in the dailydialog dataset @xcite . given a dialog with one temporal span masked out, the model is asked to find all correct answers from a list of four options to fill in the blank ( the challenge set requires the models to demonstrate understanding of the context and use temporal commonsense to make right choices. our final challenge set consists of 1.1k carefully curated dialog instances. we then study the performance of several stateof-the-art pre-trained language models on time-dial along several dimensions including modeling paradigms (classification, mask filling, and generation), the scope of dialog contexts, in-domain vs. out-of-domain training, dependence on shallow text matching for reasoning, and the types of reasoning required. our experiments demonstrate that offthe-shelf, pre-trained language models cannot effectively reason about temporal aspects in a dialog, even with domain-specific finetuning. our findings indicate that large-scale pre-trained models even after fine-tuning may not be sufficient for robust temporal reasoning in dialogs, and motivate future research toward modeling temporal concepts over diverse everyday events, and contextual reasoning about them. ## task: temporal reasoning in dialog we formulate the dialog-based temporal commonsense reasoning problem as a cloze task @xcite . formally, given a multi-turn dialog context of n conversational turns between two speakers a and b, where a temporal words span within the context is masked out, the task is to predict the suitable temporal expression(s) for the masked-out span from a list of options. that is, we want the conversation model to select all the correct answers from the options based on the dialog context. following similar cloze-style challenge datasets, we use accuracy as the evaluation metric @xcite @xcite . having a non-trivial set of options is crucial to build a challenge set and to avoid accidental spurious biases @xcite @xcite . we ensure this via the following filtering process. (1) for each masked span, there is more than one correct answer in the options. this makes the task more challenging for models since more comprehensive understanding of the context is required to recognize all the correct choices. in our dataset ( §3) we guarantee two correct answers for each masked span. (2) some incorrect options are selected to be spuriously correlated with the dialog context. for example, we include temporal spans in the dialog context as negative options, which will challenge models that rely primarily only on shallow pattern matching without correct temporal reasoning. we present more information in §3 about how the negative options were created by human annotators. ## dataset: timedial the timedial dataset is derived from dailydialog data @xcite , which is a multi-turn dialog corpus containing over 13k english dialogs. dialogs in this dataset consist of turn-taking between two people on topics over 10 broad categories, ranging from daily lives to financial topics. ## modeling we consider a broad set of methods and evaluate their performance on our challenge timedial dataset. these methods vary in terms of the modeling paradigms, the scope of the dialog contexts, and training settings. in particular, they encompass the major ways pre-trained lms are currently used in downstream tasks ( §4.1) which often outperform earlier specialized non-pretrained models. we also consider different lengths of context used in reasoning, varying by their vicinity to the masked span ( §4.2). finally, we study different training settings, including zero-shot, in-domain, and out-of-domain training ( §4.3). ## experiments and analyses using the proposed timedial challenge set, we next conduct extensive experiments and analyses on the different model variants and context settings. we use either 4x4 or 8x8 cloud tpus v3 pod slices foot_4 for fine-tuning and one v100 gpu for inference. we provide more details of the experiment configurations in the appendix. ## conclusions we introduced timedial, a challenge set consistting of 1.1k multiple-choice cloze questions for temporal commonsense reasoning in dialog. the dataset is carefully curated to evaluate a models' ability to do temporal commonsense/numerical reasoning over dialog context. in order to establish strong baselines and provide information on future model development, we conducted extensive experiments with state-of-the-art language models with different settings: the scope of context, weak supervision strategies, and learning objectives. while humans can easily answer these questions (97.8% accuracy), even our best model variant (t5-large with in-domain training) struggles on this challenge set (73%). moreover, our qualitative error analyses show that these large language models often rely on shallow, spurious features (particularly text matching) when answering these questions, instead of truly doing reasoning over the context.