Written Justifications are Key to Aggregate Crowdsourced Forecasts (2021)

Abstract: This paper demonstrates that aggregating crowdsourced forecasts benefits from modeling the written justifications provided by forecasters. Our experiments show that the majority and weighted vote baselines are competitive, and that the written justifications are beneficial to call a question throughout its life except in the last quarter. We also conduct an error analysis shedding light into the characteristics that make a justification unreliable.

## introduction the wisdom of the crowd refers to the idea that aggregating information collected from many nonexperts often yields good answers to questions-as close to the truth or even better than asking an expert. perhaps the best known example is by @xcite , who observed that the median estimate of the weight of an ox (out of 800 country fair attendees) was within 1% of the truth. there is a lot of support for the idea, although it is well know that it is not foolproof @xcite . mackay (1841) presents historical examples where crowds behaved irrationally, and more recently, world chess champion gary kasparov beat the crowd playing chess @xcite . in this day and age, the benefits of the crowd are commonplace. wikipedia is written by volunteers, and community question answering has received the attention of researchers @xcite . when aggregating information collected from crowds, it may be important to know whether judgments were collected independently of each other. if they were not, crowd psychology @xcite and the power of persuasion (o'keefe, 2015) can bias individual judgments and eliminate the wisdom of the crowd. in this paper, we work with forecasts about questions across the political, economic, and social question: will there be a new prime minister of italy before 1 september 2021? start date: 1/28/2021, closing date: 2/13/2021 forecast 1: 100% yes, 0% no justification: actually the media talk about potential candidates @xcite the crowd is 98% yes forecast 2: 99% yes, 1% no justification: with a substantial majority now backing draghi (who in turns seems to be an obvious eu favourite which brings better prospects for bail out funding) this seems to be a virtual certainty at this stage. we move to a discussion of important terminology. we define a question as a sentence that elicits information (e.g., 'will legislation raising the us federal minimum wage become law before 20 august 2021?'). questions have an opening and closing day, and the days in between are the life of the question. forecasters are people who submit a forecast. a forecast consists of a prediction and a justification. the prediction is a number indicating the chances that something will happen. following with the question above, a prediction could be '70% chance' (of the legislation becoming law before 20 august 2021). a justification is the text forecasters submit in support of their predictions (see examples in figure 1 and section 5). we use the phrase call a question to refer to the problem we work with: make a final prediction after aggregating individual forecasts. we call questions each day throughout their life using two strategies: forecasts submitted in the given day (daily) and the last forecast submitted by each forecaster (active). note that in this paper we use prediction to refer to the submission by a forecaster, not the output of a machine learning model. inspired by previous work on identifying and cultivating better forecasters @xcite , and analyzing written justifications to estimate the quality of a single forecast @xcite or all forecasts by a forecaster (zong et al., 2020), we experiment with the problem of automatically calling a question through its life based on the available forecasts in each day. the main contributions of this paper are empirical results answering the following research questions: ## previous work the language people use is indicative of several attributes. previous work includes both predictive models (input: language samples, output: some attribute about the author) and models that yield useful insights (input: language samples and attributes of the authors, output: differentiating language features depending on the attributes). among many others, previous research has studied gender and age @xcite @xcite , political ideology @xcite , health outcomes @xcite , and personality traits @xcite . in this paper, we do not profile forecasters. instead, we build models to call questions based on forecasts by the crowd without knowledge of who submitted what. previous research has also studied the language people use to communicate depending on the relationship between the parties. for example, the language people use when they are in positions of power (e.g., more seniority) has been studied in social networks @xcite , online communities @xcite , and corporate emails @xcite . similarly, @xcite study how language provides clues about the interactions and relationships between people. regarding language form and functions, prior research has analyzed politeness @xcite , empathy @xcite , advice @xcite , condolences (zhou and jurgens, 2020) usefulness @xcite , and deception @xcite . more related to the problem we work with, @xcite analyze the influence of wikipedia editors, and @xcite study influence levels in online communities. persuasion has also been studied from a computational perspective @xcite , including dialogue systems @xcite . the work presented here complements these works. we are interested in identifying credible justifications in order to aggregate crowdsourced forecasts, and we do so without explicitly targeting any of the above characteristics. within computational linguistics, the previous task that is perhaps the closest to our goal is argumentation: a good justification for a forecast is arguably a good supporting argument. previous work includes identifying argument components such as claims, premises, backings, rebuttals, and refutations @xcite , and mining supporting and opposing arguments for a claim @xcite . notwithstanding these works, we found that crowdsourced justifications rarely fall into these argumentation frameworks despite the former are useful to aggregate forecasts. finally, there are a few works on forecasting that use the same or very similar corpora than we do. from a psychology perspective, @xcite present strategies to improve forecasting accuracy (using top forecasters, i.e., superforecasters) and analyze the characteristics of superforecaster performance, which can be used for cultivating better forecasters. @xcite discuss explanations of what makes forecasters better. these works aim at identifying superforecasters and do not take into account the written justifications. unlike them, we build models to call questions without using any information about forecasters. within computational linguistics, @xcite assess the language of quality justifications (rating, benefit, and influence). zong et al. ( 2020 ) is perhaps the closest experiment to ours. they build models to predict forecaster skill using the text justifications of forecasts from good judgment open data, and they also use another dataset, company earnings reports, to individually predict which forecasts are more likely to be correct predictions. unlike us, none of these works aim at calling the question throughout its life. ## dataset we work with data from the good judgment open, 1 a website where questions are posted and people submit forecasts. questions are about geopolitics and include topics such as domestic and international politics, the economy, and social issues. we collected all binary questions along with all their forecasts including a prediction and a justification. in total, the dataset we work with contains 441 questions and 96,664 forecasts submitted in 32,708 days. this is almost twice the amount of forecasts considered by @xcite . since our goal is to call questions throughout their life, we work with all forecasts with written justifications regardless of length, how many forecasts have been submitted by the same forecaster, etc. additionally, our framework preserves privacy as we do not use any information about the forecaster. the bottom plot in figure 2 shows the average number of daily and active forecasts over the life of all questions. there is roughly a uniform number of forecasts submitted each day, thus the amount of active forecasts increases linearly over the life of the question. the majority baseline with both daily and active forecasts submitted in the previous 10 days is quite accurate, especially towards the closing date of questions. the experiments presented in this paper aim at calling questions throughout their life. as we shall see, models to automatically call questions benefit from taking into account justifications during the first three quarters of the life of a question. analyzing the questions analyzing the justifications to make predictions about the future (questions do not have negations). we also look at the sentiment polarity of justifications using textblob (loria, 2020). the majority of justifications are neutral (polarity close to 0). in terms of readability, we compute the flesch @xcite and dale-chall @xcite scores. both scores indicate that around a quarter of justifications require a college education to be understood. in terms of verbs and nouns, we analyze them using the wordnet lexical files @xcite . the most common verb classes are change (26% of justifications, e.g., happen, remain, increase) social (24%, e.g., vote, support, help) cognition (22%, e.g., think, believe, know) and motion (19%, e.g., go, come, leave). the most common noun classes are act (71%, e.g., election, support, deal), communication (57%, e.g., questions, forecast, news), cognition (38%, e.g., point, issue, possibility), and group (38%, e.g., government, people, party). ## experiments and results we experiment with the problem of calling a question throughout its life. the input to the problem is the question itself and forecasts (predictions and justifications), and the output is an answer to the question aggregating all the forecasts. the number of instances is the number of days all questions were open (recall our dataset contains 441 questions and 96,664 forecasts submitted in 32,708 days). we experiment both with simple baselines and a neural network taking into account (a) daily forecasts and (b) active forecasts submitted up to ten days prior. experimental results showed that considering earlier active forecasts is not beneficial. we divide the questions into training, validation, and test subsets. then, we assign to each subset all the forecasts submitted throughout the life of the questions. note that randomly splitting forecasts would be unsound, as forecasts for the same questions submitted on different days would end in the training, validation, and test subsets. baselines we consider two unsupervised baselines. the majority vote baseline calls a question based on the majority prediction in the forecasts. the weighted vote baseline calls a question after weighting the chances assigned to the predictions in the forecasts. consider these three forecasts: 99%, 45%, and 45% chance the answer is yes (thus 1%, 55%, and 55% chance the answer is no). the majority vote baseline would output no (2 out of 3 believe fc (256) bert forecast 1 -prediction: 0.6 -current/past: past -justification: i think that the chances are low because in the past ... fc (256) bert forecast 2 -prediction: 0.6 -current/past: past -justification: ... turmoil in the are leads me to believe that it won't ... fc (256) bert forecast n -prediction: 0.6 -current/past: current -justification: this is just impossible, there is no chance at all. lstm ... fc (256) bert question: before 1 september 2021, will justice stephen breyer announce his retirement from the us supreme court? figure 4 : neural network architecture to call a question on a given day based on crowdsourced forecasts. the network consists of three main components: one for the question, one for each forecast (prediction + flag indicating current day or past + justification), and an lstm to process the sequence of forecasts. we experiment with two scenarios: feeding the network the forecasts submitted on a given day (daily) or the last forecast by each forecaster within the ten previous days of a given day (active). ## qualitative analysis in this section, we present insights into (a) what makes questions harder to forecast and (b) characteristics of justification submitted with wrong and correct predictions ( questions we looked at three characteristics of the 88 questions in the test set depending on whether the best model (bottom row in ## conclusions forecasting is the process of predicting future events. government and industry alike are interested in forecasting because it affords them the capability to anticipate and address potential challenges to come. in this paper, we work with questions across the political, economic, and social spectrum published in the good judgment open website, and forecasts submitted by the crowd without special training. each forecast consists of a prediction and a justification in natural language. we have shown that aggregating the weighted predictions of forecasters is a robust baseline to call a question through its life. models that take into account both the question and justifications, however, obtain significantly better results when calling a question in the first three quartiles of its life. crucially, our models do not profile forecasters or use any information about who submitted which forecast. the work presented here opens the door to assessing the credibility of anonymous forecasts in order to come up with aggregation strategies that are robust without tracking forecasters.