TORQUE : A Reading Comprehension Dataset of Temporal Ordering Questions (2020)

Abstract: A critical part of reading is being able to understand the temporal relationships between events described in a passage of text, even when those relationships are not explicitly stated. However, current machine reading comprehension benchmarks have practically no questions that test temporal phenomena, so systems trained on these benchmarks have no capacity to answer questions such as “what happened before/after [some event]?” We introduce TORQUE, a new English reading comprehension benchmark built on 3.2k news snippets with 21k human-generated questions querying temporal relationships. Results show that RoBERTa-large achieves an exact-match score of 51% on the test set of TORQUE, about 30% behind human performance.

## introduction time is important for understanding events and stories described in natural language text such as news articles, social media, financial reports, and electronic health records @xcite @xcite @xcite @xcite . for instance, "he won the championship yesterday" is different from "he will win the championship tomorrow": he may be celebrating if he has already won it, while if he has not, he is probably still preparing for the game tomorrow. the exact time of an event is often implicit in text. for instance, if we read that a woman is "expecting the birth of her first child", we know that the birth is in the future, while if she is "mourning the death of her mother", the death is in the past. these relationships between an event and a time point (e.g., "won the championship yesterday") or between two events (e.g., "expecting" is before "birth" and "mourning" is after "death") are called temporal relations @xcite . this work studies reading comprehension for temporal relations, i.e., given a piece of text, a computer needs to answer temporal relation questions (fig. 1 ). reading comprehension is a natural format for studying temporal phenomena, as the flexibility of natural language annotations allows for capturing relationships that were not possible in previous formalism-based works. however, temporal phenomena are studied very little in reading comprehension @xcite @xcite @xcite , and existing systems are hence brittle when handling questions in torque ( reading comprehension for temporal relationships has the following challenges. first, reading ## natural language annotation of temporal relations motivated by recent works @xcite @xcite , we propose using natural language question answering as an annotation for- ## data collection we used amazon mechanical turk to build torque. following prior work, we focus on passages that consist of two contiguous sentences, as this is sufficient to capture the vast majority of non-trivial temporal relations @xcite . we took all the articles used in the tempeval3 (te3) workshop (2.8k articles) @xcite and created a pool of 26k two-sentence passages. given a random passage from this pool, the annotation process for crowd workers was: the annotation guidelines 4 and interface 5 are public. in the following sections, we further discuss issues of quality control and crowdsourcing cost. ## torque statistics torque has 3.2k passage annotations (∼50 tokens/passage), foot_3 24.9k events (7.9 events/passage), and 21.2k user-provided questions (half of them were labeled by crowd workers as modifications of existing ones). every passage comes with 3 hardcoded warm-up questions asking which events in the passage had already happened, were ongoing, or were still in the future. in a random sample of 200 questions in the test set of torque, we found 94 questions querying about relations that cannot be directly represented by the previous single-interval-based labels. ## experiment we split torque into train (80% of all the questions), dev (5%), and test (15%) and these three parts do not have the same articles. to solve torque in an end-to-end fashion, the model here takes as input a passage and a question, then looks at every token in the passage and makes a binary classification of whether this token is an answer to the question or not. specifically, the model has a one-layered perceptron on top of bert @xcite or roberta @xcite , and the input to the perceptron layer is the transformers' output corresponding to the token we're looking at. we fine-tuned bert/roberta (both "base" and "large") on the training set of torque. we fixed batch @xmath0, one question, and all its answers) with gradient accumulation @xmath1. we selected the learning rate (from (1e -5 , 2e -5 )), the training epoch (within 10), and the random seed (from 3 arbitrary ones) based on performance on the dev set of torque. 7 to compute an estimate of human performance, one author answered 100 questions from the test set and compared with crowd workers' annotations. both the human performance and system performances are shown in ## related work the study of time is to understand when, how long, and how often things happen. while how long and how often usually require temporal common sense knowledge @xcite @xcite , the problem of when often boils down to extracting temporal relations. ## conclusion understanding temporal ordering of events is critical in reading comprehension, but existing works have studied very little about it. this paper presents torque, a new english machine reading comprehension (mrc) dataset of temporal ordering questions. torque has 3.2k news snippets, 9.5k hard-coded questions asking which events had happened, were ongoing, or were still in the future, and 21.2k human-generated questions querying more complex phenomena. we argue that an mrc setting allows for more convenient representation of these temporal phenomena than conventional formalisms. results show that even a state-of-the-art language model, roberta-large, falls behind human performance by a large margin, necessitating more investigation on improving mrc on temporal relationships in the future. figure 13 further shows the 50 most common events in torque. unsurprisingly, the most common events are reporting verbs (e.g., "say", "tell", "report", and "announce") and copular verbs. other common events such as "meeting", "killed", "visit", and "war" are also expected given that the passages of torque were taken from news articles.