Uncertainty over Uncertainty: Investigating the Assumptions, Annotations, and Text Measurements of Economic Policy Uncertainty (2020)

Abstract: Methods and applications are inextricably linked in science, and in particular in the domain of text-as-data. In this paper, we examine one such text-as-data application, an established economic index that measures economic policy uncertainty from keyword occurrences in news. This index, which is shown to correlate with firm investment, employment, and excess market returns, has had substantive impact in both the private sector and academia. Yet, as we revisit and extend the original authors’ annotations and text measurements we find interesting text-as-data methodological research questions: (1) Are annotator disagreements a reflection of ambiguity in language? (2) Do alternative text measurements correlate with one another and with measures of external predictive validity? We find for this application (1) some annotator disagreements of economic policy uncertainty can be attributed to ambiguity in language, and (2) switching measurements from keyword-matching to supervised machine learning classifiers results in low correlation, a concerning implication for the validity of the index.

## introduction the relatively novel research domain of text-asdata, which uses computational methods to automatically analyze large collections of text, is a rapidly growing subfield of computational social science with applications in political science @xcite , sociology (evans and aceves, 2016), and economics @xcite . in economics, textual data such as news editorials @xcite , central bank communications @xcite , financial earnings calls @xcite , company disclosures @xcite , and newspa- * this work was done during an internship at bloomberg. pers (thorsrud, 2020) have recently been used as new, alternative data sources. in one such economic text-as-data application, @xcite aim to construct an economic policy uncertainty (epu) index whereby they quantify the aggregate level that policy is influencing economic uncertainty (see the index has had impact both on the private sector and academia. 1 in the private sector, financial companies such as bloomberg, haver, fred, and reuters carry the index and sell financial professionals access to it. academics show economic policy uncertainty has strong relationships with other economic indicators: @xcite find a negative relationship between the index and firmlevel capital investment, and @xcite find that the index can positively forecast excess market returns. the epu index of baker et al. has substantive impact and is a real-world demonstration of finding economic signal in textual data. yet, as the subfield of text-as-data grows, so too does the need for rigorous methodological analysis of how well the chosen natural language processing methods operationalize the social science construct at hand. thus, in this paper we seek to re-examine baker et al.'s linguistic, annotation, and measurement assumptions. regarding measurement, although keyword look-ups yield high-precision results and are interpretable, they can also be brittle and may suffer from low recall. baker et al. did not explore alternative text measurements based on, for example, word embeddings or supervised machine learning classifiers. no. example 1 demand for new clothing is uncertain because several states may implement large hikes in their sales tax rates. 2 the outlook for the h1b visa program remains highly uncertain. as a result, some high-tech firms fear that shortages of qualified workers will cramp their expansion plans. 3 the looming political fight over whether to extend the bush-era tax cuts makes it extremely difficult to forecast federal income tax collections in 2011. 4 uncertainty about prospects for war in iraq has encouraged a build-up of petroleum inventories and pushed oil prices higher. 5 some economists claim that uncertainties due to government industrial policy in the 1930s prolonged and deepened the great depression. 6 it remains unclear whether the government will implement new incentives for small business hiring. in exploring baker et al.'s construction of epu, we identify and disentangle multiple sources of uncertainty. first, there is the real underlying uncertainty about economic outcomes due to government policy that the index attempts to measure. second, there is semantic uncertainty that can be expressed in the language of newspaper articles. third, there is annotator uncertainty about whether a document should be labeled as epu or not. finally, there is modeling uncertainty in which text classifiers are uncertain about the decision boundary between positive and negative classes. in this paper, we revisit and extend baker et al.'s human annotation process ( §3) and computational pipeline that obtains epu measurement from text ( §4). in doing so, we draw on concepts from quantitative social science's measurement modeling, mapping observable data to theoretical constructs, which emphasizes the importance of validity (is it right?) and reliability (can it be repeated?) @xcite @xcite . overall, this paper contributes the following: we demonstrate that a measure of external predictive validity, i.e., correlations with a stock-market volatility index (vix), is particularly sensitive to these decisions ( §4). ## assumptions of measuring economic policy uncertainty from news the goal of @xcite is to measure the theoretical construct of policy-related economic uncertainty (epu) for particular times and geographic regions. baker et al. assume they can use information from newspaper articles as a proxy for epu, an assumption we explore in great detail in section 2.2, and they define epu very broadly in their coding guidelines: "is the article about policyrelated aspects of economic uncertainty, even if only to a limited extent?" foot_2 for an article to be annotated as positive, there must be a stated causal link between policy and economic consequences and either the former or the latter must be uncertain. ## annotator uncertainty reliable human annotation is essential for both building supervised classifiers and assessing the internal validity of text-as-data methods. in order to validate their epu index, baker et al. sample documents from each month, obtain binary labels on the documents from annotators, and then construct a "human-generated" index which they report has a 0.86 correlation with their keyword-based index (aggregated quarterly). yet, in our analysis of baker et al.'s annotations (denoted below as bbd), we find only 16% of documents have more than one annotator and of these, the agreement rates are moderate: 0.80 pairwise agreement and 0.60 krippendorff's α chance-adjusted agreement @xcite . see line 2 of the original authors did not address whether this disagreement is a result of annotator bias, error in annotations, or true ambiguity in the text. in contrast to the popular paradigm that one should aim for high inner-annotator agreement rates @xcite , recent research has shown "disagreement between annotators provides a useful signal for phenomena such as ambiguity in the text" @xcite . additionally, recent research in natural language processing man perception is important: in the epu index released to the public, one of three underlying components is a disagreement of economic forecasters as a proxy for uncertainty. see http: //policyuncertainty.com/methodology.html . the following evidence lends to our hypothesis that there is inherent ambiguity in whether documents encode epu: (1) the original coding guide of baker et al. had 17 pages of "hard calls" that describe difficult or ambiguous documents, (2) there was a moderate amount of annotator disagreement in bbd ( ## measurement for text-as-data applications, substantive results are contingent on how researchers operationalize measurement of the (latent) theoretical construct of interest via observed text data. using baker et al.'s original causal assumptions (section 2.2), we formally define the measurement of interest as: rq2: what happens when we change g to equally or more valid measurement functions? in particular, we are interested in sensitivity: for two measurements, g 1 and g 2 , does u 1 correlate well with u 2 ; and external predictive validity: for each measurement, g i , does u i correlate well with the vix, a stock-market volatility index based on s&p 500 options prices? baker et al. also use the vix as a measure of external validity, and like baker et al. we note that the vix is a good proxy for economic uncertainty, but does not necessarily capture policy uncertainty. as baker et al. mention, "differences in the topical scope between the vix and the epu index are an important source of distinct variation in the two measures." in the future, we could compare our measures to the other two external validity measures of baker et al.: mentions of uncertain in the federal reserve's beige books and large daily moves in the s&p stock index. data and pre-processing. although baker et al. use 10 newspapers to construct their us-based index, we instead use the new york times annotated corpus (nyt-ac) @xcite because the text data is cleaned, easily accessible, and results on the corpus are reproducible. this collection includes over 1.8 million articles written and published by the new york times between january 1, 1987 and june 19, 2007. baker et al. assume that using newspapers based in the united states is sufficient to find a signal of us-based epu. to test this assumption, we apply a simple heuristic to the dateline of nyt-ac articles and remove articles that mention non-us cities. however, we find relatively little variation in results via this heuristic (see appendix, figure 7 ). ## related work there have been only a few other attempts to construct alternative, non-keyword measurements of economic policy uncertainty. azqueta-gavaldón (2017) apply topic models and manually map the topics to baker et al.'s epu categories and find their method tightly correlates (0.94) with the original index. in an unpublished manuscript, nyman and ormerod (2020) expand the uncertainty keywords of baker et al. via nearest neighbor embeddings and find granger causality between their expanded keyword list and the original epu index. in contrast, we are the first to take a fully supervised learning approach to measuring epu and analyze the original annotations of baker et al.. measurement of economic variables from text. other work has examined measuring economic variables from text data (see @xcite for a survey). for example, topic models have been applied to central bank communications @xcite and newspaper articles @xcite while other work identifies negated uncertainty markers (e.g. "there is no uncertainty") in the federal reserve's beige books @xcite and extracts sentiment from central bank communications @xcite . @xcite use off-the-shelf supervised document classifiers to demonstrate that the information in news can predict stock prices. text-as-data methods. traditional ways of analyzing textual data include content analysis where human annotators read and hand-code documents for particular phenomena @xcite . in the last decade, many researchers have adapted machine learning and nlp methods to the needs of social scientists @xcite . nlp technologies such as lexicons, topic models @xcite , supervised classifiers, word embeddings @xcite , and largescale pre-trained language model representations @xcite have been applied to textual data to extract relevant signals. more recent work attempts to extend text-as-data methods to incorporate principles from causal inference @xcite @xcite @xcite . ## future directions in the future, estimating the sensitivity of causal estimates to the different measurement approaches presented in this paper could potentially have substantive impact. using a bayesian modeling approach to annotator uncertainty @xcite , investigating better calibration, which has been shown to improve prevalence estimation @xcite , or estimating model uncertainty could improve measurement. one could also shift from document-level predictions of epu to paragraph, sentence, or span-level predictions. annotating discourse structure and selecting discourse fragments, e.g. @xcite , could potentially increase annotator agreement. these subdocument extraction models could also potentially provide human-interpretable contextualization of movements in an epu index. ## conclusion there is great promise for text-as-data methods and applications; however, we echo the cautionary advice of @xcite that automatic methods require extensive "problem-specific validation." our paper's investigation of baker et al. provides a number of general insights for text-as-data practioners along these lines. first, content validity: when dealing with text data, one needs to think carefully about the kinds of linguistic information one is trying to measure. for instance, mapping economic policy uncertainty to a document-level binary label collapses all types of semantic uncertainty, many of which cannot be identified via keywords alone. second, one needs to examine social perception assumptions. is one trying to prescribe an annotation schema, or, as we argue in this paper, are people's perceptions about the concept as important as the concept itself, especially in the face of ambiguity in language? third, sensitivity of measurements: text-as-data practitioners can strengthen their substantive conclusions if multiple measurement approaches give similar results. for economic policy uncertainty, this paper demonstrates that switching from keywords to aggregating the outputs of a document classifier are not tightly correlated, a concerning implication for the validity of this index.