Measuring Forecasting Skill from Text (2020)

Abstract: People vary in their ability to make accurate predictions about the future. Prior studies have shown that some individuals can predict the outcome of future events with consistently better accuracy. This leads to a natural question: what makes some forecasters better than others? In this paper we explore connections between the language people use to describe their predictions and their forecasting skill. Datasets from two different forecasting domains are explored: (1) geopolitical forecasts from Good Judgment Open, an online prediction forum and (2) a corpus of company earnings forecasts made by financial analysts. We present a number of linguistic metrics which are computed over text associated with people’s predictions about the future including: uncertainty, readability, and emotion. By studying linguistic factors associated with predictions, we are able to shed some light on the approach taken by skilled forecasters. Furthermore, we demonstrate that it is possible to accurately predict forecasting skill using a model that is based solely on language. This could potentially be useful for identifying accurate predictions or potentially skilled forecasters earlier.

## introduction people often make predictions about the future, for example meteorologists tell us what the weather might look like tomorrow, financial analysts predict which companies will report favorable earnings and intelligence analysts evaluate the likelihood of future geopolitical events. an interesting question is why some individuals are significantly better forecasters @xcite ? previous work has analyzed to what degree various factors (intelligence, thinking style, knowledge of a specific topic, etc.) contribute to a person's skill. these studies have used surveys or psychological tests to measure dispositional, situational and behavioral variables @xcite . another source of information has been largely overlooked, however: the language forecasters use to justify their predictions. recent research has demonstrated that it is possible to accurately forecast the outcome of future events by aggregating social media users' predictions and analyzing their veridicality @xcite , but to our knowledge, no prior work has investigated whether it might be possible to measure a forecaster's ability by analyzing their language. in this paper, we present the first systematic study of the connection between language and forecasting ability. to do so, we analyze texts written by top forecasters (ranked by accuracy against ground truth) in two domains: geopolitical forecasts from an online prediction forum, and company earnings forecasts made by financial analysts. to shed light on the differences in approach employed by skilled and unskilled forecasters, we investigate a variety of linguistic metrics. these metrics are computed using natural language processing methods to analyze sentiment @xcite , uncertainty @xcite , readability, etc. in addition we make use of word lists taken from the linguistic inquiry and word count (liwc) software @xcite , which is widely used in psychological research. by analyzing forecasters' texts, we are able to provide evidence to support or refute hypotheses about factors that may influence forecasting skill. for example, we show forecasters whose justifications contain a higher proportion of uncertain statements tend to make more accurate predictions. this supports the hypothesis that more open-minded thinkers, who have a higher tolerance for ambiguity tend to make better predictions @xcite . beyond analyzing linguistic factors associated with forecasting ability, we further demonstrate that it is possible to identify skilled forecasters and accurate predictions based only on relevant text. estimating the quality of a prediction using the forecaster's language could potentially be very beneficial. for example, this does not require access to historical predictions to evaluate past performance, so it could help to identify potentially skilled individuals sooner. also, forecasters do not always provide an explicit estimate of their confidence, so a confidence measure derived directly from text could be very useful. ## linguistic cues of accurate forecasting in this section, we are interested in uncovering linguistic cues in people's writing that are predictive of forecasting skill. we start by analyzing texts written by forecasters to justify their predictions in a geopolitical forecasting forum. linguistic differences between forecasters are explored by aggregating metrics across each forecaster's predictions. in §3, we analyze the accuracy of individual predictions using a dataset of financial analysts' forecasts towards companies' (continuous) earnings per share. by controlling for differences between analysts and companies, we are able to analyze intra-analyst differences between accurate and inaccurate forecasts. ## companies' earnings forecasts in §2, we showed that linguistic differences exist between good and bad forecasters, and furthermore, these differences can be used to predict which forecasters will perform better. we now turn to the question of whether it is possible to identify which individual forecasts, made by the same person, are more likely to be correct. the good judgment open data is not suitable to answer this question, because forecasts are discrete, and thus do not provide a way to rank individual predictions by accuracy beyond whether they are correct or not. therefore, in this section, we consider numerical forecasts in the financial domain, which can be ranked by their accuracy as measured against ground truth. in this paper, we analyze forecasts of companies' earnings per share (eps). earnings per share is defined as the portion of a company's profit allocated to each share of common stock. it is an important indicator of a company's ability to make profits. for our purposes, eps also supports a cleaner experimental design as compared to stock prices, which constantly change in real time. ## comparison of findings across domains in §2 and §3, we analyze the language people use when they make forecasts in geopolitical and financial domains. specifically, these two sections reveal how language is associated with accuracy both within and across forecasters. in this section, we compare our findings from these domains. our studies reveal several shared characteristics of accurate forecasts from a linguistic perspective over geopolitical and financial domains (in we also note one main difference between two domains is uncertainty metric: in good judgment open dataset, we observe that more skilled forecast-ers employ a higher level of uncertainty; while for individual forecasts, less uncertainty seems to be better. it makes us consider the following hypothesis: within each forecaster, people are more likely to be correct when they are more certain about their judgments, while in general skilled forecasters exhibit a higher level of uncertainty. to test this hypothesis, we calculate the spearman's ρ between the financial analysts' mean forecasting errors and their average portion of uncertain statements. results show that these two variables are negative correlated with ρ=-0.24, which provides some support for our hypothesis, however the sample size is very small (there are only 19 analysts in the financial dataset). also, these mean forecasting errors are not standardized by the difficulty of companies analysts are forecasting. ## related work many recent studies have analyzed connections between users' language and human attributes @xcite @xcite @xcite . @xcite developed a tool for discourse analysis in social media and found that older individuals and females tend to use more causal explanations. another example is work by @xcite , who developed automatic classifiers for temporal orientation and found important differences relating to age, gender in addition to big five personality traits. @xcite showed that language expressed on twitter can be predictive of community-level psychological correlates, in addition to rates of heart disease. @xcite analyzed political polarization in social media and @xcite examined the connections between police officers' politeness and race by analyzing language. a number of studies @xcite @xcite have examined the connection between users' language on social media and depression and alcohol use @xcite . other work has analyzed users' language to study the effect of attributes, such as gender, in online communication @xcite @xcite . in this work we study the relationship between people's language and their forecasting skill. to the best of our knowledge, this is the first work that presents a computational way of exploring this direction. our work is also closely related to prior research on predicting various phenomenon from users' language. for example @xcite study the effect of wording on message propagation, @xcite examine the connection between language used by politicians in campaign speeches and applause and pérez-rosas and mihalcea (2015) explored linguistic differences between truthful and deceptive statements. ganjigunte @xcite show linguistic cues drawn from authors' language are strong indicators of the success of their books and @xcite presented an unsupervised model to analyze the helpfulness of book reviews by analyzing their text. there have been several studies using data from good judgment open or good judgment project @xcite . one recent study examining the language side of this data is @xcite . their main goal is to suggest objective metrics as alternatives for subjective ratings when evaluating the quality of recommendations. to achieve this, justifications written by one group are provided as tips to another group. these justifications are then evaluated on their ability to persuade people to update their predictions, leading to real benefits that can be measured by objective metrics. prior work has also studied persuasive language on crowdfunding platforms @xcite . in contrast, our work focuses on directly measuring forecasting skill based on text justifications. finally we note that there is a long history of research on financial analysts' forecasting ability @xcite @xcite . most work relies on regression models to test if pre-identified factors are correlated with forecasting skill (e.g., @xcite ; @xcite ). some work has also explored the use of textual information in financial domain. for example, @xcite present a study of predicting companies' risk by using financial reports. we also note a recent paper on studying financial analysts' decision making process by using text-based features from earning calls @xcite . as far as we aware, our work is the first to evaluate analysts' forecasting skill based on their language. ## limitations and future work our experiments demonstrated it is possible to analyze language to estimate people's skill at making predictions about the future. in this section we highlight several limitations of our study and ethical issues that should be considered before applying our predictive models in a real-world application. in our study, we only considered questions with binary answers; future work might explore questions with multiple-choice outcomes. prior studies have found that people's forecasting skills can be improved through experience and training @xcite . our study does not take this into account as we do not have detailed information on the forecasters' prior experience. finally, we have not investigated the differences in our model's outputs on different demographic groups (e.g., men versus women), so our models may contain unknown biases and should not be used to make decisions that might affect people's careers. ## conclusion in this work, we presented the first study of connections between people's forecasting skill and language used to justify their predictions. we analyzed people's forecasts in two domains: geopolitical forecasts from an online prediction forum and a corpus of company earning forecasts made by financial analysts. we investigated a number of linguistic metrics that are related to people's cognitive processes while making predictions, including: uncertainty, readability and emotion. our experimental results support several findings from the psychology literature. for example, we observe that skilled forecasters are more open-minded and exhibit a higher level of uncertainty about future events. we further demonstrated that it is possible to identify skilled forecasters and accurate predictions based solely on language.