Towards a Language Model for Temporal Commonsense Reasoning (2021)

Abstract: Temporal commonsense reasoning is a challenging task as it requires temporal knowledge usually not explicit in text. In this work, we propose an ensemble model for temporal commonsense reasoning. Our model relies on pre-trained contextual representations from transformer-based language models (i.e., BERT), and on a variety of training methods for enhancing model generalization: 1) multi-step fine-tuning using carefully selected auxiliary tasks and datasets, and 2) a specifically designed temporal masked language model task aimed to capture temporal commonsense knowledge. Our model greatly outperforms the standard fine-tuning approach and strong baselines on the MC-TACO dataset.

## introduction although recent pre-trained language models such as bert @xcite have achieved great success in a wide range of natural language processing (nlp) tasks, these models may still perform poorly on temporal reasoning scenarios. @xcite has shown that such models often fail to make even simple temporal distinctions, for example, to distinguish the words before and after, resulting in degraded performance. an especially challenging task is temporal commonsense reasoning. for instance, given two events "going on a vacation" and "going for a walk", while most humans would know that a vacation is typically longer and occurs less often than a walk, computers have difficulty understanding and reasoning about temporal commonsense @xcite . in this paper, we focus on developing a model for temporal commonsense reasoning. following best practices from recent work on enhancing model generalization, we propose a model that enriches pre-trained contextual representations with temporal knowledge and general commonsense knowledge by leveraging carefully selected auxiliary datasets in a multi-step fine-tuning setting. moreover, we specifically designed a temporal taskadaptive pre-training task aimed to capture temporal commonsense knowledge by masking temporal indicators in text. we evaluate our model on the challenging multiple choice temporal common-sense (mc-taco) dataset @xcite . in our experiments, our model substantially outperforms the standard finetuning approach, as well as other strong baselines. ## temporal commonsense reasoning task this task entirely focuses on a specific reasoning capability: temporal commonsense. @xcite used crowdsourcing to create the multiple choice temporal common-sense (mc-taco) dataset, which collects the temporal knowledge of five temporal properties: (1) duration (how long an event takes), (2) temporal ordering (typical order of events), (3) typical time (when an event occurs), ( 4 ) frequency (how often an event occurs), and (5) stationarity (whether a state is maintained for a very long time or indefinitely). it contains 13k tuples, each consisting of a sentence, a question, and a candidate answer, that should be judged as plausible or not. the sentences are taken from different sources such as news, wikipedia and textbooks. an example from the dataset is below. the correct answer is in bold. paragraph: growing up on a farm near st. paul, l. mark bailey didn't dream of becoming a judge. question: how many years did it take for mark to become a judge? a) 63 years b) 7 weeks c) 7 years d) 7 seconds e) 7 hours reasoning type: duration we use the mc-taco dataset for evaluating the performance of our model. ## model our work uses bert @xcite as the text encoder. it has obtained high performance on several natural language understanding (nlu) benchmarks and it is relatively simple to adapt its architecture to downstream tasks. we focus on exploring different training techniques using bert @xcite , given its superior performance on a wide range of nlp tasks. the text encoder and the training methods used in our model are detailed below. ## conclusion in this paper, we proposed a model for temporal commonsense reasoning. we specifically designed a temporal masked language model task aimed to capture temporal commonsense knowledge by masking temporal indicators in text and used it in a multi-step fine-tuning setting. moreover, we found out that an ensemble of this model with other models achieves the best results, outperforming other state-of-the-art models. in order to improve our model, we plan to conduct attention and saliency analysis.