Temporal Reasoning in Natural Language Inference (2020)

Abstract: We introduce five new natural language inference (NLI) datasets focused on temporal reasoning. We recast four existing datasets annotated for event duration—how long an event lasts—and event ordering—how events are temporally arranged—into more than one million NLI examples. We use these datasets to investigate how well neural models trained on a popular NLI corpus capture these forms of temporal reasoning.

## introduction the ability to reason about how events unfold in time is core to how humans structure their knowledge about the world @xcite @xcite , and modeling such temporal reasoning has been central to many classical ai approaches @xcite @xcite @xcite . natural language supports various forms of temporal reasoning, including reasoning about the chronology and duration of events, and many natural language understanding (nlu) tasks and models have been employed for understanding and capturing different aspects of temporal reasoning @xcite @xcite @xcite @xcite @xcite @xcite @xcite . more broadly, the ability to perform temporal reasoning is important for understanding narratives @xcite @xcite , answering questions @xcite @xcite , and summarizing events @xcite . ## dataset creation we construct five new nli datasets recast from four existing datasets that focus on two key aspects of temporal reasoning: (a) temporal ordering and (b) event duration. across these datasets, we have more than a million nli examples and we retain the training, development, and test splits from the original (for datasets in which such splits exist). ## dataset validation to assess whether the recast nli pairs are correct, we conduct a validation experiment by randomly sampling 100 nli pairs from the train split of each dataset. for each nli pair, we ask the annotators to answer the question -how likely is it that the second sentence is true if the first sentence is true? we provide 5 options to choose from -extremely likely, very likely, even chance, very unlikely, extremely unlikely. we recruited 48 annotators from amazon mechanical turk to validate the sampled nli pairs for each of our 5 recasted datasets. we selected only those annotators who passed an american nativespeaker test with 90% or above accuracy. each item in our validation task listed 10 nli pairs. if our recasting produces valid nli pairs, we should see that entailed pairs receive higher likelihood judgments than not-entailed pairs, even when adjusting for the dataset the pair comes from, the annotator, the pair, and the list of pairs the annotator saw the pair in. to test this, we fit an ordinal mixed effects model to the likelihood responses given by annotators, with a fixed effect for the source of the nli pair as well as random intercepts for annotator, pair, and list. we compare this model to a model that additionally includes a fixed effect for the entailment label associated with the pair by our recasting. we find a reliable positive effect of the label being entailed (χ 2 (1) = 226.1, @xmath0.001), indicating our recasting method produces valid nli pairs. ## experimental setup we use our recast datasets to explore how well different common classes of nli models capture temporal reasoning. specifically, we use three types of models: (i) neural bag of words (nbow; @xcite (ii) infersent @xcite , and (iii) roberta @xcite . 4 our nbow model represents contexts and hypotheses as an average of glove embeddings @xcite . the concatenation of these representations is fed to a mlp with one hidden layer. the infersent model encodes contexts and hypotheses independently with a bilstm and sentence representations are extracted using max-pooling. the concatenation of these sentences, their difference, and their element-wise product @xcite are then fed to a mlp. for roberta, we use a classification head on top of the pooled output of roberta-large to predict the labels. 5 in our experiments, we train and test these models on each recast temporal dataset. for each model, we include a hypothesis-only baseline to evaluate how much the datasets test nli as opposed to just the likely duration and order of events in general. additionally, we train each model on multi-genre nli @xcite and test the model on our datasets to see if the model learns temporal reasoning from a generic nli dataset that does not necessarily focus on temporal reasoning. model uds-duration uds-order tempeval3 timebank-dense red majority 50.00 54.52 54.57 50.54 52.51 mnli baseline nbow 43.37 33.83 34.13 35.36 34.02 infersent 49.04 49.58 47.57 47.84 47.64 roberta 50.28 55.01 54.87 51.01 52.97 hypothesis-only nbow 84.96 54.52 54.57 50.54 52.51 infersent 91.18 71.96 63.22 68.29 63.47 roberta 91.52 71.22 63.25 68.83 52.51 context and hypothesis nbow 82.45 54.52 54.57 50.54 52.51 infersent 92.65 73.22 62.2 68.29 63.47 roberta 94.51 80.17 54.57 94.60 80.59 ## results & discussion ## conclusion to better capture temporal reasoning inference capabilities, we create a million nli pairs recast from existing corpora in the literature that focus on two aspects of temporal reasoning -temporal duration and temporal order. we test existing models trained on mnli on our datasets and find that a generic nli model is not able to capture temporal reasoning. we show that training on our datasets can improve the performance of models in capturing temporal reasoning, and some aspects of temporal reasoning, specifically how long an event lasts, might be learned from lexical entities alone. we hope that our recast datasets push the research community to further explore how learning temporal reasoning could benefit other tasks.