Toward Building a Language Model for Understanding Temporal Commonsense (2022)

Abstract: The ability to capture temporal commonsense relationships for time-related events expressed in text is a very important task in natural language understanding. On the other hand, pre-trained language models such as BERT, which have recently achieved great success in a wide range of natural language processing tasks, are still considered to have poor performance in temporal reasoning. In this paper, we focus on the development of language models for temporal commonsense inference over several pre-trained language models. Our model relies on multi-step fine-tuning using multiple corpora, and masked language modeling to predict masked temporal indicators that are crucial for temporal commonsense reasoning. We also experimented with multi-task learning and build a language model that can improve performance on multiple time-related tasks. In our experiments, multi-step fine-tuning using the general commonsense reading task as auxiliary task produced the best results. This result showed a significant improvement in accuracy over standard fine-tuning in the temporal commonsense inference task.

## introduction commonsense reasoning is crucial for natural language processing (nlp). commonsense is the basic level of practical knowledge that is commonly shared among most people foot_0 . a specific type of commonsense is temporal commonsense. temporal commonsense refers to the common knowledge about various temporal aspects of events, such as duration, frequency, and temporal order. capturing temporal commonsense relations for time-related events expressed in sentences is a very important task in natural language understanding. however, pre-trained language models such as bert @xcite , which have recently achieved significant results in a wide range of nlp tasks, are still said to perform poorly in temporal reasoning @xcite . for example, given two events, "going on a vacation" and "going for a walk," most humans know that "vacation is longer and occurs less frequently than walks," or that "going on a walk is more frequent than going on a vacation. however, it is difficult for computers to make inferences based on such commonsense knowledge. in this paper, we focus on the development of a language model for understanding temporal commonsense. in a prior study @xcite , bert was used, and in this study, we also use roberta @xcite and albert @xcite , which are improved models of bert. we use them for multi-step fine-tuning using multiple corpora and continual pre-training by performing the masked language modeling (mlm) task @xcite on the target dataset. mlm task is a fill-in-the-blank task that has been employed as a pre-training task for various language models. for multi-step fine-tuning, we thought an additional stage of fine-tuning on an intermediate related supervised task might help improve performance because temporal datasets usually have only a small amount of training data available. for continual pre-training on the target dataset, we aimed to resolve the domain mismatch between the pretrained models and the target task, and make the model better weight temporal indicators and event triggers for our downstream tasks. in addition, we apply multi-task learning to further improve our model's generalization performance. our contributions are summarized as follows: ## related work although research on temporal inference has been conducted for a long time, in recent years, many studies have been proposed on temporal expression extraction @xcite , temporal relation extraction @xcite , and the construction of timelines @xcite . as for temporal commonsense, there are studies focusing on the duration of events @xcite , the temporal order of events @xcite , and so forth. @xcite proposed methods for constructing language models that produce representations of events for relevant tasks such as duration comparison, parent-child relations, event coreference and temporal questionanswering tasks. in particular, some recent works have focused on the construction of challenging benchmarks for temporal commonsense inference. the story cloze test @xcite dataset focuses on the typical temporal and causal relationships between events. torque @xcite ) is a machine reading comprehension dataset that focuses on the temporal ordering of events. mc-taco @xcite is a challenging multiple choice temporal commonsense reasoning task that focuses on temporal properties such as duration and ordering of events. timedial @xcite is a dataset consisting of dialogues containing temporal information and is a complex temporal commonsense inference task using multi-turn dialogues. in addition, pre-trained language models such as bert have succeeded on broad-coverage probing benchmarks. however, in the case of domain mismatch between the pre-trained model and the target task, these models may still suffer catastrophic accuracy degradation. in this study, we focus on temporal commonsense reasoning and attempt to improve the performance of the pre-trained language model for understanding temporal commonsense. our model effectively leverages continual pre-training, multi-step fine-tuning, and multi-task learning. it substantially outperforms the standard fine-tuning approach. the sentences are taken from different sources such as news, wikipedia and textbooks. an example from this dataset is below. the correct answers are in bold. paragraph: he layed down on the chair and pawed at her as she ran in a circle under it. question: how long did he paw at her? a) 2 minutes b) 2 days c) 90 minutes e) 7 seconds reasoning type: duration we mainly use the mc-taco dataset for evaluating the performance of our model. in the later sections, we also show evaluation on additional temporal-related tasks. ## methods we focus on exploring different training techniques, i.e., multi-step fine-tuning, continual pre-training, and multi-task learning, for building our language model for understanding temporal commonsense. each technique is detailed below. ## conclusion in this paper, we focused on the development of a language model for temporal commonsense reasoning, and tried to develop a language model for understanding temporal commonsense. we conducted multi-step fine-tuning, continual pre-training, and multi-task learning on bert, roberta, and albert, using several datasets. we confirmed that the multi-step fine-tuning model that uses the general commonsense knowledge task as an auxiliary task was often better than that obtained by ordinary fine-tuning and we were able to construct a language model that understands temporal commonsense. comparing bert, roberta, and albert, albert produced the best results overall. for future work, we plan to further investigate multi-task learning. in multi-task learning, we would like to visualize attention scores, for example, and pursue what setting can improve generalization performance. also, we plan to construct a new general-purpose language model that performs well in a variety of time-related tasks.