The Misestimation of Uncertainty in Affective Disorders
Authors: Erdem Pulcu, Michael Browning
Journal: Trends in Cognitive Sciences (2019)

Maintaining estimates of why unexpected events happen, of why we are uncertain, allows humans to adjust their inference and learn as efficiently as possible.Humans adjust their learning in response to the levels of different types of uncertainty, including expected uncertainty, caused by probabilistic associations between observed events and what we are learning, and unexpected uncertainty, caused by changes in the thing we are learning.People who experience anxiety are less sensitive to changes in levels of unexpected uncertainty.Misestimation of the levels of uncertainty may lead to a tendency to be more influenced by negative events, which is believed to cause depression and anxiety.Understanding how the estimation of uncertainty is related to symptoms of anxiety and depression promises to enhance our understanding of these illnesses and may suggest novel therapeutic interventions. Our knowledge about the state of the world is often incomplete, making it difficult to select the best course of action. One strategy that can be used to improve our ability to make decisions is to identify the causes of our ignorance (i.e., why an unexpected event might have occurred) and use estimates of the uncertainty induced by these causes to guide our learning. Here, we explain the logic behind this process and describe the evidence that human learners use estimates of uncertainty to sculpt their learning. Finally, we describe recent work suggesting that misestimation of uncertainty is involved in the development of anxiety and depression and describe how these ideas may be advanced. Our knowledge about the state of the world is often incomplete, making it difficult to select the best course of action. One strategy that can be used to improve our ability to make decisions is to identify the causes of our ignorance (i.e., why an unexpected event might have occurred) and use estimates of the uncertainty induced by these causes to guide our learning. Here, we explain the logic behind this process and describe the evidence that human learners use estimates of uncertainty to sculpt their learning. Finally, we describe recent work suggesting that misestimation of uncertainty is involved in the development of anxiety and depression and describe how these ideas may be advanced. When we observe our environment, we observe it incompletely and intermittently. Our senses report aspects of the particular location we find ourselves in, but many of the most important processes we want to know about cannot be directly perceived. For example, if I stroke my cat, will it scratch me or not (Figure 1A , Key Figure)? If we knew whether the cat was feeling angry or friendly, then we would know whether it was likely to scratch us, but the internal state of the cat is hidden and so, if we want to know it, it must be inferred from our previous experience of how it has behaved. To complicate matters further, the hidden processes (see Glossary) we estimate will often change over time, so even if the cat was friendly the last time we stroked it, it may now be angry. Therefore, the challenge is how to make reasonable inferences about hidden, changeable processes when faced with such fundamental uncertainty about these processes. Failure to meet this challenge reduces our ability to accurately gauge the opportunities and costs afforded by the environment and, ultimately, to select the best course of action. One approach to mitigating the impact of uncertainty on our decisions is to identify its potential causes and use estimates of the extent of these causes to sculpt how we learn [1.Yu A.J. Dayan P. Uncertainty, neuromodulation, and attention.Neuron. 2005; 46: 681-692Abstract Full Text Full Text PDF PubMed Scopus (1078) Google Scholar]. In other words, we need to learn why we do not know what we are interested in. In this opinion article, we focus on how estimates of two key types of uncertainty, expected uncertainty and unexpected uncertainty (see Box 1 for detailed descriptions of these and other forms of uncertainty), may be used to improve learning, summarise the evidence that human learners utilise such estimates, and finally argue that difficulties with uncertainty estimation may be one mechanism underlying symptoms of psychiatric difficulties, such as anxiety and depression.Box 1Forms of UncertaintyIn this paper, we focus specifically on expected and unexpected uncertainty. Here, we describe these and other forms of uncertainty [36.de Berker A.O. et al.Computations of uncertainty mediate acute stress responses in humans.Nat. Commun. 2016; 710996Crossref PubMed Scopus (145) Google Scholar, 38.Mathys C.D. et al.Uncertainty in perception and the Hierarchical Gaussian Filter.Front. Hum. Neurosci. 2014; 8: 825Crossref PubMed Scopus (183) Google Scholar, 39.Marshall L. et al.Pharmacological fingerprints of contextual uncertainty.PLoS Biol. 2016; 14e1002575Crossref PubMed Scopus (59) Google Scholar].Estimation UncertaintyThe first time we meet a new cat, we do not know whether it is likely to scratch us. This lack of knowledge is called 'estimation uncertainty' and, in some respects is similar to expected uncertainty. The important distinction is that estimation uncertainty occurs because of lack of experience; thus, we can reduce it by gathering more evidence (e.g., stroking the cat and seeing what happens), whereas, with expected uncertainty, the lack of knowledge is a fundamental aspect of the system that is not reduced by gathering more evidence.Expected UncertaintyExpected uncertainty arises when events occur probabilistically, rather than deterministically (Figure 1 in the main text) and is linked to 'risk'. Generally, the term 'risk' is used when a probability is explicit and expected uncertainty when it is estimated. Gathering more experience does not reduce expected uncertainty, which has resulted in it sometimes being called 'irreducible uncertainty'.Unexpected UncertaintyUnexpected uncertainty arises when the thing we are learning about changes (Figure 1 in the main text). Unexpected uncertainty may be conceptualised in several ways. In the example illustrated in Figure 1 in the main text, being scratched by the cat is ambiguous, so we can never be sure whether the mood of the cat has changed. In these cases, unexpected uncertainty is often described as 'volatility', roughly an estimate of the average amount that an association will change between observations. However, if we are estimating how much food our cat eats and, after eating 100–120 g a day for 3 months, it suddenly eats 300 g, we can clearly say that a change has just occurred (sometimes called a 'change point'). In fact, the two descriptions are complimentary: the volatility of an association is the probability that a change point has occurred over a specific time period.Higher Order UncertaintyUncertainty may change over time. This allows the possibility of higher order uncertainties, uncertainty about the current level of uncertainty. While the number of higher order uncertainties may theoretically be infinite, their calculation becomes progressively more demanding, and their impact on estimates of the environment more subtle, meaning there is likely to be a limit beyond which they are not accounted for in human learning. This highlights a more general question when investigating how humans learn: what are the hidden processes that are (or should be) estimated? In this paper, we focus specifically on expected and unexpected uncertainty. Here, we describe these and other forms of uncertainty [36.de Berker A.O. et al.Computations of uncertainty mediate acute stress responses in humans.Nat. Commun. 2016; 710996Crossref PubMed Scopus (145) Google Scholar, 38.Mathys C.D. et al.Uncertainty in perception and the Hierarchical Gaussian Filter.Front. Hum. Neurosci. 2014; 8: 825Crossref PubMed Scopus (183) Google Scholar, 39.Marshall L. et al.Pharmacological fingerprints of contextual uncertainty.PLoS Biol. 2016; 14e1002575Crossref PubMed Scopus (59) Google Scholar]. Estimation Uncertainty The first time we meet a new cat, we do not know whether it is likely to scratch us. This lack of knowledge is called 'estimation uncertainty' and, in some respects is similar to expected uncertainty. The important distinction is that estimation uncertainty occurs because of lack of experience; thus, we can reduce it by gathering more evidence (e.g., stroking the cat and seeing what happens), whereas, with expected uncertainty, the lack of knowledge is a fundamental aspect of the system that is not reduced by gathering more evidence. Expected Uncertainty Expected uncertainty arises when events occur probabilistically, rather than deterministically (Figure 1 in the main text) and is linked to 'risk'. Generally, the term 'risk' is used when a probability is explicit and expected uncertainty when it is estimated. Gathering more experience does not reduce expected uncertainty, which has resulted in it sometimes being called 'irreducible uncertainty'. Unexpected Uncertainty Unexpected uncertainty arises when the thing we are learning about changes (Figure 1 in the main text). Unexpected uncertainty may be conceptualised in several ways. In the example illustrated in Figure 1 in the main text, being scratched by the cat is ambiguous, so we can never be sure whether the mood of the cat has changed. In these cases, unexpected uncertainty is often described as 'volatility', roughly an estimate of the average amount that an association will change between observations. However, if we are estimating how much food our cat eats and, after eating 100–120 g a day for 3 months, it suddenly eats 300 g, we can clearly say that a change has just occurred (sometimes called a 'change point'). In fact, the two descriptions are complimentary: the volatility of an association is the probability that a change point has occurred over a specific time period. Higher Order Uncertainty Uncertainty may change over time. This allows the possibility of higher order uncertainties, uncertainty about the current level of uncertainty. While the number of higher order uncertainties may theoretically be infinite, their calculation becomes progressively more demanding, and their impact on estimates of the environment more subtle, meaning there is likely to be a limit beyond which they are not accounted for in human learning. This highlights a more general question when investigating how humans learn: what are the hidden processes that are (or should be) estimated? 'I thought that the cat was in a good mood, but it scratched me when I stroked it.' Surprising events occur when our understanding of the environment is less than perfect [2.MacKay D.J.C. Information Theory Inference and Learning Algorithms. Cambridge University Press, 2003Google Scholar]. As introduced earlier, the potential causes of a surprising event may be useful to know. In the example of the cat that scratches you when you stroked it, it may be that the cat is in fact feeling friendly, but decided to scratch on this occasion, or that the cat is no longer feeling friendly and is now angry. In the first of these cases, the surprising event is caused by a probabilistic relationship between the cat scratching and its mood. In other words, the cat may occasionally scratch even if it is feeling friendly and/or not scratch if it is feeling angry. This means that, even after experiencing whether the cat has scratched us or not, we cannot be sure of its mood. This form of uncertainty is called 'irreducible' or 'expected' uncertainty [1.Yu A.J. Dayan P. Uncertainty, neuromodulation, and attention.Neuron. 2005; 46: 681-692Abstract Full Text Full Text PDF PubMed Scopus (1078) Google Scholar] and increases as the association between an observation and a hidden process becomes less deterministic. For example, expected uncertainty would be relatively higher if the cat scratches on 30%:75% of the times that it is in a good:bad mood and relatively lower if it scratches on 10%:88% of occasions. When considering how to respond to this source of surprising events, the higher the expected uncertainty, the less a specific observation tells us about the hidden process we are interested in, and the more observations we need to collect to confidently estimate the state of the hidden process (see Box 2 for a description of the effect of different sources of uncertainty on the information content of events). More concretely, having high expected uncertainty about the behaviour of the cat means that we should consider its average behaviour across multiple previous time points to be confident of whether it is in a good or bad mood now, whereas a low expected uncertainty would allow us to make a similar judgement based on fewer events.Box 2The Effect of Uncertainty on the Information Content of EventsAs described in the main text, different forms of uncertainty influence how useful recent relative to more distant previous events are when learning about a hidden process. Expected uncertainty, arising from a probabilistic relationship between the hidden process being learned and observations, reduces the amount of information individual events provide, requiring learners to rely on more distant previous events during learning (i.e., use a lower learning rate, see Equations [1], [2] in main text). By contrast, unexpected uncertainty erodes the information content of past observations by making it more likely that the underlying generative process has changed since they were observed (i.e., prompting the use of a higher learning rate). Here, the term 'information content' can be understood as the amount of information observing an event provides to a learner. The information content is challenging to quantify because it depends on the nature of the event as well as the learner's prior belief(s) about the state and structure of the world. One approach that has been used to measure the information content of events is to construct a Bayesian Ideal Observer model of a task and use the Kullback–Leibler divergence (KLDiv) of the belief of the model from before to after a particular event as a measure of the information content of the event [5.O'Reilly J.X. et al.Dissociable effects of surprise and model update in parietal and anterior cingulate cortex.Proc. Natl. Acad. Sci. U. S. A. 2013; 110: E3660-E3669Crossref PubMed Scopus (203) Google Scholar].Here, the Bayesian Observer (see [7.Behrens T.E.J. et al.Learning the value of information in an uncertain world.Nat. Neurosci. 2007; 10: 1214-1221Crossref PubMed Scopus (1193) Google Scholar] for a description of an ideal observer model) maintains its belief about the state of relevant aspects of the task as dimensions of a multivariate probability distribution, with updates applied to this distribution that reflect the assumptions of the model about the structure of the task (e.g., that the process it is learning about may change over time; Figure I). The KLDiv is a distance measure, in this case of the change in the ideal observer's belief from before to after the event, and can be conceptualised as the degree to which the event changes the learner's belief about the generative process that it uses to predict future outcomes. An example of this approach is shown in Figure I. Data from this model allow quantification of the effects of unexpected and expected uncertainty ('volatility' and 'noise' in Figure I) on the information content ('KLdiv' in Figure I) of events: KLDiv is significantly higher when volatility is higher [F(1,831)=4.3, P=0.038] and when noise is lower [F(1,831)=142, P<0.001]. In other words, the learning rate adaptation in response to uncertainty described in the main text reflects the relative information content of events: a higher learning rate is used when events have relatively higher information content. As described in the main text, different forms of uncertainty influence how useful recent relative to more distant previous events are when learning about a hidden process. Expected uncertainty, arising from a probabilistic relationship between the hidden process being learned and observations, reduces the amount of information individual events provide, requiring learners to rely on more distant previous events during learning (i.e., use a lower learning rate, see Equations [1], [2] in main text). By contrast, unexpected uncertainty erodes the information content of past observations by making it more likely that the underlying generative process has changed since they were observed (i.e., prompting the use of a higher learning rate). Here, the term 'information content' can be understood as the amount of information observing an event provides to a learner. The information content is challenging to quantify because it depends on the nature of the event as well as the learner's prior belief(s) about the state and structure of the world. One approach that has been used to measure the information content of events is to construct a Bayesian Ideal Observer model of a task and use the Kullback–Leibler divergence (KLDiv) of the belief of the model from before to after a particular event as a measure of the information content of the event [5.O'Reilly J.X. et al.Dissociable effects of surprise and model update in parietal and anterior cingulate cortex.Proc. Natl. Acad. Sci. U. S. A. 2013; 110: E3660-E3669Crossref PubMed Scopus (203) Google Scholar]. Here, the Bayesian Observer (see [7.Behrens T.E.J. et al.Learning the value of information in an uncertain world.Nat. Neurosci. 2007; 10: 1214-1221Crossref PubMed Scopus (1193) Google Scholar] for a description of an ideal observer model) maintains its belief about the state of relevant aspects of the task as dimensions of a multivariate probability distribution, with updates applied to this distribution that reflect the assumptions of the model about the structure of the task (e.g., that the process it is learning about may change over time; Figure I). The KLDiv is a distance measure, in this case of the change in the ideal observer's belief from before to after the event, and can be conceptualised as the degree to which the event changes the learner's belief about the generative process that it uses to predict future outcomes. An example of this approach is shown in Figure I. Data from this model allow quantification of the effects of unexpected and expected uncertainty ('volatility' and 'noise' in Figure I) on the information content ('KLdiv' in Figure I) of events: KLDiv is significantly higher when volatility is higher [F(1,831)=4.3, P=0.038] and when noise is lower [F(1,831)=142, P<0.001]. In other words, the learning rate adaptation in response to uncertainty described in the main text reflects the relative information content of events: a higher learning rate is used when events have relatively higher information content. A second reason why the cat may scratch is because its mood has changed and, even though it was previously feeling friendly, it is now angry. This form of uncertainty, called 'unexpected uncertainty', is produced when the hidden process we are estimating changes [1.Yu A.J. Dayan P. Uncertainty, neuromodulation, and attention.Neuron. 2005; 46: 681-692Abstract Full Text Full Text PDF PubMed Scopus (1078) Google Scholar]. It is high if the mood of the cat changes frequently (is volatile) relative to if it is stable. Unexpected uncertainty requires the opposite response to expected uncertainty. Specifically, when learning about a volatile process, we cannot rely so much on previous observations because it is more likely that the hidden process will have changed since they occurred; rather, we have to base our current beliefs on more recent observations. Thus, if the behaviour of the cat has high unexpected uncertainty, we should base our belief about its mood mainly on how it responded the last few times we stroked it, whereas, if it has low unexpected uncertainty, we can incorporate a greater number of prior events in our estimate. Reinforcement learning models [3.Sutton R. Barto A.G. Reinforcement Learning. MIT Press, 1998Google Scholar] offer a relatively straightforward framework for describing how agents update their beliefs in response to experience and, thus, provide a useful tool for thinking about and measuring adaptation to uncertainty. One of the simplest models is the Rescorla–Wagner learning rule [4.Rescorla R.A. Wagner A.R. A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement.in: Black A.H. Prokasy W.F. Classical Conditioning II: Current Research and Theory. Appleton-Centuary-Crofts, 1972: 64-99Google Scholar] (Equation 1), in which a belief at time t, r(t) is updated by a prediction error, the difference between an event which has just occurred o(t) and what the agent expected to happen (its current belief about the event), multiplied by a learning rate, α:rt+1=rt+αot−rt[1] The learning rate in Equation 1 is simply some number between 0 and 1. The role of the learning rate becomes more obvious if we rearrange Equation 1 to collect the terms on the right (Equation 2):rt+1=1−αrt+αot[2] As can be seen, the Rescorla–Wagner learning rule defines the updated belief as a weighted mean of new information from the recently observed outcome, o(t), and old information in the form of the previous belief, r(t), with the learning rate acting as the weight. When the learning rate is high, the learner places more weight on the recent observation; when it is low, the learner places more weight on prior belief. As described earlier, if an agent believes that surprising events are being generated by noise in the relationship between observations and the hidden process (expected uncertainty), then it should be more influenced by prior events, and the agent should use a lower learning rate (Figure 1C). By contrast, if an agent thinks the surprising events are caused by changes in the hidden process being estimated (unexpected uncertainty) then it should be more influenced by recent events and use a higher learning rate (Figure 1B). Here, it is important to note that the Rescorla–Wagner learning rule itself does not estimate uncertainty: it assumes that learning rate does not change. It is presented here as a simple framework to illustrate how learning rates should adapt to changes in expected and unexpected uncertainty. A range of more complex models, which are able to estimate uncertainty, are described in Box 3. These more complex models use their estimates of uncertainty to adapt their learning rate in the manner described earlier.Box 3Models of Uncertainty EstimationIn this paper, we have framed the logic behind using estimates of uncertainty to guide learning. We have also described studies in which the simple Rescorla–Wagner learning rule was used to obtain separate estimates of learning rate when uncertainty was high versus low [7.Behrens T.E.J. et al.Learning the value of information in an uncertain world.Nat. Neurosci. 2007; 10: 1214-1221Crossref PubMed Scopus (1193) Google Scholar, 8.Behrens T.E.J. et al.Associative learning of social value.Nature. 2008; 456: 245-249Crossref PubMed Scopus (648) Google Scholar, 10.Browning M. et al.Anxious individuals have difficulty learning the causal statistics of aversive environments.Nat. Neurosci. 2015; 18: 590-596Crossref PubMed Scopus (191) Google Scholar, 11.Pulcu E. Browning M. Affective bias as a rational response to the statistics of rewards and punishments.Elife. 2017; 6e27879Crossref PubMed Scopus (35) Google Scholar]. In these studies, the Rescorla–Wagner rule was used as a method of measuring learning rate; it does not describe how learners might calculate whether learning rate should be changed (the model is simply fit separately to blocks in which uncertainty is high or low). Several more complex models, which are able to adapt their effective learning rate in response to uncertainty, have been described, which we briefly summarise here.A relatively simple adaptation of the Rescorla–Wagner learning rule was suggested by Pearce and Hall [40.Pearce J.M. Hall G. A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli.Psychol. Rev. 1980; 87: 532-552Crossref PubMed Scopus (2174) Google Scholar] and developed by others [41.Li J. et al.Differential roles of human striatum and amygdala in associative learning.Nat. Neurosci. 2011; 14: 1250-1252Crossref PubMed Scopus (232) Google Scholar]. The broad class of 'Pearce–Hall type' models are able to adapt to changes in unexpected uncertainty by estimating the surprise associated with a stimulus (i.e., learning the absolute magnitude of the prediction error) and using this estimate to adjust the main learning rate of the model. More elaborate than this are models in which various sources of uncertainty are estimated separately before impacting learning [9.Nassar M.R. et al.Rational regulation of learning dynamics by pupil-linked arousal systems.Nat. Neurosci. 2012; 15: 1040-1046Crossref PubMed Scopus (364) Google Scholar, 42.Nassar M.R. et al.An approximately Bayesian delta-rule model explains the dynamics of belief updating in a changing environment.J. Neurosci. 2010; 30: 12366-12378Crossref PubMed Scopus (259) Google Scholar]. Finally, Bayesian learners, which invert generative models of the environment, including estimates of various sources of uncertainty, have been described. The most prominent of these is the Kalman Filter [43.Kalman R.E. A new approach to linear filtering and prediction problem.J. Basic Eng. 1960; 82: 35-45Crossref Scopus (20176) Google Scholar], which can efficiently estimate and adapt to changes in unexpected uncertainty. More recent Bayesian 'filters' have also been described which provide estimates of higher order and/or expected uncertainty [7.Behrens T.E.J. et al.Learning the value of information in an uncertain world.Nat. Neurosci. 2007; 10: 1214-1221Crossref PubMed Scopus (1193) Google Scholar, 38.Mathys C.D. et al.Uncertainty in perception and the Hierarchical Gaussian Filter.Front. Hum. Neurosci. 2014; 8: 825Crossref PubMed Scopus (183) Google Scholar] (see also Box 2 in the main text). These different models solve similar problems in different ways, with the complexity of the model increasing from the Pearce–Hall to the Bayesian examples. To date, it is not clear which model provides a more veridical description of how humans solve the problem of estimation of uncertainty. In this paper, we have framed the logic behind using estimates of uncertainty to guide learning. We have also described studies in which the simple Rescorla–Wagner learning rule was used to obtain separate estimates of learning rate when uncertainty was high versus low [7.Behrens T.E.J. et al.Learning the value of information in an uncertain world.Nat. Neurosci. 2007; 10: 1214-1221Crossref PubMed Scopus (1193) Google Scholar, 8.Behrens T.E.J. et al.Associative learning of social value.Nature. 2008; 456: 245-249Crossref PubMed Scopus (648) Google Scholar, 10.Browning M. et al.Anxious individuals have difficulty learning the causal statistics of aversive environments.Nat. Neurosci. 2015; 18: 590-596Crossref PubMed Scopus (191) Google Scholar, 11.Pulcu E. Browning M. Affective bias as a rational response to the statistics of rewards and punishments.Elife. 2017; 6e27879Crossref PubMed Scopus (35) Google Scholar]. In these studies, the Rescorla–Wagner rule was used as a method of measuring learning rate; it does not describe how learners might calculate whether learning rate should be changed (the model is simply fit separately to blocks in which uncertainty is high or low). Several more complex models, which are able to adapt their effective learning rate in response to uncertainty, have been described, which we briefly summarise here. A relatively simple adaptation of the Rescorla–Wagner learning rule was suggested by Pearce and Hall [40.Pearce J.M. Hall G. A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli.Psychol. Rev. 1980; 87: 532-552Crossref PubMed Scopus (2174) Google Scholar] and developed by others [41.Li J. et al.Differential roles of human striatum and amygdala in associative learning.Nat. Neurosci. 2011; 14: 1250-1252Crossref PubMed Scopus (232) Google Scholar]. The broad class of 'Pearce–Hall type' models are able to adapt to changes in unexpected uncertainty by estimating the surprise associated with a stimulus (i.e., learning the absolute magnitude of the prediction error) and using this estimate to adjust the main learning rate of the model. More elaborate than this are models in which various sources of uncertainty are estimated separately before impacting learning [9.Nassar M.R. et al.Rational regulation of learning dynamics by pupil-linked arousal systems.Nat. Neurosci. 2012; 15: 1040-1046Crossref PubMed Scopus (364) Google Scholar, 42.Nassar M.R. et al.An approximately Bayesian delta-rule model explains the dynamics of belief updating in a changing environment.J. Neurosci. 2010; 30: 12366-12378Crossref PubMed Scopus (259) Google Scholar]. Finally, Bayesian learners, which invert generative models of the environment, including estimates of various sources of uncertainty, have been described. The most prominent of these is the Kalman Filter [43.Kalman R.E. A new approach to linear filtering and prediction problem.J. Basic Eng. 1960; 82: 35-45Crossref Scopus (20176) Google Scholar], which can efficiently estimate and adapt to changes in unexpected uncertainty. More recent Bayesian 'filters' have also been described which provide estimates of higher order and/or expected uncertainty [7.Behrens T.E.J. et al.Learning the value of information in an uncertain world.Nat. Neurosci. 2007; 10: 1214-1221Crossref PubMed Scopus (1193) Google Scholar, 38.Mathys C.D. et al.Uncertainty in perception an