F orecast QA : A Question Answering Challenge for Event Forecasting with Temporal Text Data (2021)

Abstract: Event forecasting is a challenging, yet important task, as humans seek to constantly plan for the future. Existing automated forecasting studies rely mostly on structured data, such as time-series or event-based knowledge graphs, to help predict future events. In this work, we aim to formulate a task, construct a dataset, and provide benchmarks for developing methods for event forecasting with large volumes of unstructured text data. To simulate the forecasting scenario on temporal news documents, we formulate the problem as a restricted-domain, multiple-choice, question-answering (QA) task. Unlike existing QA tasks, our task limits accessible information, and thus a model has to make a forecasting judgement. To showcase the usefulness of this task formulation, we introduce ForecastQA, a question-answering dataset consisting of 10,392 event forecasting questions, which have been collected and verified via crowdsourcing efforts. We present our experiments on ForecastQA using BERTbased models and find that our best model achieves 61.0% accuracy on the dataset, which still lags behind human performance by about 19%. We hope ForecastQA will support future research efforts in bridging this gap.

## introduction forecasting globally significant events, such as outcomes of policy decisions, civil unrest, or the economic ramifications of global pandemics, is a consequential but arduous problem. in recent years there have been significant advances in applying machine learning (e.g., time-series prediction methods) to generate forecasts for various types of events including conflict zones @xcite , duration of insurgency @xcite , civil unrest @xcite and terrorist events @xcite . a) south korea [0.41] b) syria [0.28] c) south africa [0.15] d) portugal [0.16] yes [0.38] / no [0.62] q: who will drop japan as a trading partner in august 2019? q: will primary schools in europe admit non-vaccinated children around september 2019? current automated forecasting methods perform well on problems for which there are sufficient structured data (e.g., knowledge graphs), but are not well suited for events for which such data may not exist. humans, though, can often accurately forecast outcomes by leveraging their judgement, domain knowledge, and prior experience @xcite , along with the vast amounts of unstructured text data available to us (e.g., news articles). we are able to identify and retrieve salient facts from the near-endless pool of unstructured information, synthesize those facts into coherent beliefs, and generate probabilistic forecasts. unfortunately, the process does not scale well in terms of the amount of information that must be processed and the number of events one has to forecast. here we address the above problem by formalizing a forecasting task, creating a dataset, and providing benchmarks to develop methods for the task. specifically, we formulate the forecasting problem as a multiple-choice question answering (qa) task, where the input is a news corpus, questions, choices and timestamps associated with each question, and the output is one of the given choices per question. our approach is rooted in the observation that both forecasting and qa follow a similar process: digesting massive amounts of textual data, identifying supporting pieces of evidence from text, and chaining different pieces to generate answers/forecasts. forecast question answering (forecastqa) introduces a novel timestamp constraint per question that prohibits the model from accessing new articles published after the timestamp. by doing so, forecastqa simulates a forecasting scenario; each question's timestamp is chosen to ensure that the question is about the outcome of a future event. to illustrate this, consider the question, "will primary schools in europe admit non-vaccinated children around september 2019?" in figure 1 , and the fact that models only have access to articles before "2019-09-01." with the addition of this timestamp constraint, our query becomes a question about a future event in "september, 2019" based on articles from the "past"; the model is now being tested for its forecasting ability 2 . to answer the question, the model must find pertinent events from "past" information, resolve the temporal and causal relations between them, and finally make a forecasting judgement based on its interpretation of past information to answer the question. our task differs from that of other works that require an understanding of temporal relationships @xcite and temporal commonsense reasoning @xcite , as our task forces a model to make a forecasting judgement. in support of the proposed forecastqa formulation, we construct a dataset of 10,392 yes-no and multiple-choice questions. this data is collected via crowdsourcing based on news articles, where workers are shown articles and asked to come up with yes-no and multiple-choice questions. we also crowdsourced appropriate timestamps for each question. finally, we design a method based on pre-trained language models to deal with retrieved articles for our task. in our experiments, the methods using retrieved articles slightly outper2 the ability to predict the outcome of future events based on unstructured text describing past events, without access to an extracted sequence of historical event triples, nor provided a fixed set of possible relations between events; as is the case with human forecasters. q: who will drop japan as a trading partner in august 2019? choices: south korea (answer), south africa, syria, portugal. article: why japan and south korea just can't get along. (1/1/19) apart from the fact of being one another's closest neighbours, the people of south korea and japan have a remarkable amount in common. economically, they are among one another's biggest trading partners. and yet, time and again, relations between seoul and tokyo are marked, not by mutual support and co-operation but by anger, reproach and exasperation. reasoning process: seoul is in south korea, tokyo is in japan (commonsense -world knowledge). seoul and tokyo are big trading partners (language understanding -lexical variations). the relations between seoul and tokyo are marked by anger, reproach and exasperation and these relations might cause trading relations to cease (forecasting skills -causal relationwe can infer the answer from this part). ## related work event forecasting. there are several types of approaches exist to do event forecasting. one approach could learn from highly structured eventcoded data such as icews @xcite and gdelt @xcite . when these datasets are used for forecasting, they are often represented as a time series @xcite , in which each data point is associated with a timestamp. another approach is script-learning, in which a model is provided with a chain of events and a subsequent event and is asked to predict the relation between the chain and the "future" event @xcite @xcite . they require to convert text data into event triples and translate the questions and answer choices into their format, which limits the expressiveness of natural text. however, unlike these datasets and approaches, forecastqa does not provide any structured data to a model. the model must learn how to extract, keep track of, and link pertinent events from unstructured text to solve forecasting questions. ## the forecastqa task forecastqa is a question answering task whose goal is to test a machine's forecasting ability. we consider forecasting as the process of anticipating the outcome of future events based on past and present data @xcite . we focus on forecasting outcomes of news-based events coming from topics such as politics, sports, economics, etc. training a machine to make forecasting decisions is inherently difficult, as the ground-truth label of event outcome (e.g., whether an event will occur) -so often required for model training -is only obtainable "in the future". to make progress in our goal, we devise a way to simulate the forecasting scenario by introducing a novel time constraint, allowing us to validate the machine predic- will the will there what will what is what kind what type what does why will when will is the does the are the will there be electricity in canada despite hurricane dorian in september 2019? who will be german chancellor by november 2019? who will be wanted to execute by saudi prosecutors in july 2019? who will visit pittsburgh for first 2020 campaign rally in april 2019? who will be the fifa president in september 2019? what will lyft return to its san francisco area fleet in june 2019? what will be the budget of terminator dark fate in october 2019? what will belinda carlisle want to be by september 2019? what will be difficult for boeing to get approval for by may 2019? where will the glasgow derby be played in september 2019? how many instagram followers will noor charchafchi have by september 2019? what country how long will the global stock market fall in may 2019? will the james bond actor arrive italy in september 2019? will the public charge rule impact us taxpayers by august 2019? will the mona lisa be missing in the louvre by october 2019? will the wright family blame boris johnson for its failure in september 2019? will the duke of sussex refuse to tour africa in september 2019? statistic train dev test all questions 8,210 1,090 1,092 10,392 yes-no questions 4,737 582 584 5,903 multi-choice questions 3,473 508 508 4,489 ## dataset construction and analysis in this section, we describe how we construct our forecastqa dataset and analyze it. ## methods to evaluate the forecasting capabilities of recent multi-choice/binary qa model architectures on forecastqa, we provide a comprehensive benchmarking analysis in this work. we run the experiments in two settings: (1) closed-book and (2) constrained open-domain setup. in the closed-book scenario only q (question) and c (answer choices) are provided to the model (q, c), while a (news articles) is provided for setting (2), (q, c, a) foot_4 . we run these settings to understand the difficulty of both the closed-book and open-domain challenges presented by the questions in forecastqa. bert bert aggregator [pooling/rnn] [cls] [cls] 𝐴 ! c 𝑄 𝐴 " 𝑄 c yes / no 𝑄 c summarized articles bert [cls] yes / no 𝑄 c bert [cls] yes / no for both settings, we explore several baseline models, but all follows a general architecture of a text encoder f and an optional context aggregation module g to aggregate information from a set of retrieved articles. fig. 5 shows the architectures used. we model both yes-no and multiple-choice questions as a binary classification task; a model's prediction is the class with the largest probability. below we introduce the details of our baselines. text encoder. we use pre-trained language model, bert @xcite , as a text encoder (f from above) foot_5 . f is designed to deal with (q, c) and (q, c, a) inputs, where a is a set of timestamped articles that are retrieved from a to answer q. each input of f is transformed into [[cls]q [sep]c[sep]a i ] (for each a @xmath0, @xmath1, or [[cls]q[sep]c] (for each @xmath2. the [cls] token is the same as the one commonly used for fine-tuning ptlms for a classification task, and [sep] is the special separator token. the embedding of [cls] is then used for predictions with an mlp layer (the leftmost model architecture in fig. 5 ), or as input into a context aggregation module (the middle architecture in fig. 5 ) subsequently introduced. ## experiments 6.1 experimental setup we adopt two types of settings: the closed-book setting (q, c) and the constrained open-domain setting (q, c, a). in the constrained open-domain setting, we use bm25 @xcite as our ir method foot_6 to obtain a, 10 retrieved articles. we also explore other ir methods in the later section. note that we retrieve articles that do not violate the time constraints. we feed the question q as a query and limit our access to articles in a by t q . additionally, we validate the answerability of our questions by providing gold articles instead of retrieved articles (sec. 6.3). ## conclusion forecasting is a difficult task that requires every possible advantage to do well. it would be wise to harness this pool of unstructured data for training automatic event forecasting agents. to utilize this form of data for forecasting, we proposed a question-answering task that requires forecasting skills to solve forecastqa, and provided the accompanying dataset. various baseline methods did not perform well, but this is not surprising given the inherent difficulty of forecasting. our benchmark dataset can benefit future research beyond natural language understanding and hope forecasting performance will be significantly improved.