metadata: authors: 0: author: Brown,Casey
1: author: Fozzard,Adrian
2: author: Gill,Stuart Peter Dunkley
3: author: Hallegatte,Stephane
4: author: Kalra,Nidhi Rajiv
5: author: Lempert,Robert J.
6: author: Shah,Ankur Ravi
count: World
docna: 0: docna: Agreeing on robust decisions : new processes
            for decision making under deep uncertainty
docty: Policy Research Working Paper
subtopic: Adaptation to Climate Change,Climate Change Mitigation and Green House Gases,Science of Climate Change,Climate Change and Health,Climate Change and Environment,Natural Disasters
teratopic: Science and Technology Development,Transport,Macroeconomics and Economic Growth,Environment,Finance and Financial Sector Development
abstracts: cdata!: Investment decision making is already
            difficult for any diverse group of actors with different
            priorities and views. But the presence of deep uncertainties
            linked to climate change and other future conditions further
            challenges decision making by questioning the robustness of
            all purportedly optimal solutions. While decision makers can
            continue to use the decision metrics they have used in the
            past (such as net present value), alternative methodologies
            can improve decision processes, especially those that lead
            with analysis and end in agreement on decisions. Such
            "Agree-on-Decision" methods start by
            stress-testing options under a wide range of plausible
            conditions, without requiring us to agree ex ante on which
            conditions are more or less likely, and against a set of
            objectives or success metrics, without requiring us to agree
            ex ante on how to aggregate or weight them. As a result,
            these methods are easier to apply to contexts of large
            uncertainty or disagreement on values and objectives. This
            inverted process promotes consensus around better decisions
            and can help in managing uncertainty. Analyses performed in
            this way let decision makers make the decision and inform
            them on (1) the conditions under which an option or project
            is vulnerable; (2) the tradeoffs between robustness and
            cost, or between various objectives; and (3) the flexibility
            of various options to respond to changes in the future. In
            doing so, they put decision makers back in the driver's
            seat. A growing set of case studies shows that these methods
            can be applied in real-world contexts and do not need to be
            more costly or complicated than traditional approaches.
            Finally, while this paper focuses on climate change, a
            better treatment of uncertainties and disagreement would in
            general improve decision making and development outcomes.
repnme: repnme: Agreeing on robust decisions : new
            processes for decision making under deep uncertainty
display_title: Agreeing on robust decisions : new
            processes for decision making under deep uncertainty
listing_relative_url: /research/2014/06/19616379/agreeing-robust-decisions-new-processes-decision-making-under-deep-uncertainty
topicv3: Climate Change m1326829 567,Climate Change Mitigation m1327079 579,Climate Change Adaptation m1327767 574,Hurricane m1503314 3179,Flood m1503315 3216
publishtoextweb_dt: 2014-06-01T00:00:00Z
historic_topic: Science and Technology Development,Transport,Macroeconomics and Economic Growth,Environment,Finance and Financial Sector Development
no_of_pages: 37
author_keys: 0: author_key: 901348
1: author_key: 489520
2: author_key: 995890
3: author_key: 1103816
4: author_key: 1245653
5: author_key: 1137201
6: author_key: 1153913
model_extractions: WPS6906 Policy Research Working Paper 6906 Agreeing on Robust Decisions New Processes for Decision Making Under Deep Uncertainty _Nidhi Kalra Stéphane Hallegatte Robert Lempert Casey Brown Adrian Fozzard Stuart Gill Ankur Shah_ The World Bank Climate Change Group Office of the Chief Economist June 2014
Policy Research Working Paper 6906

# **Abstract**

Investment decision making is already difficult for any diverse group of actors with different priorities and views. But the presence of deep uncertainties linked to climate change and other future conditions further challenges decision making by questioning the robustness of all purportedly optimal solutions. While decision makers can continue to use the decision metrics they have used in the past (such as net present value), alternative methodologies can improve decision processes, especially those that lead with analysis and end in agreement on decisions. Such “Agree-on-Decision” methods start by stress-testing options under a wide range of plausible conditions, without requiring us to agree ex ante on which conditions are more or less likely, and against a set of objectives or success metrics, without requiring us to agree ex ante on how to aggregate or weight them. As a result, these methods are easier to apply to contexts of large uncertainty or disagreement on values and objectives. This inverted process promotes consensus around better decisions and can help in managing uncertainty. Analyses performed in this way let decision makers make the decision and inform them on (1) the conditions under which an option or project is vulnerable; (2) the tradeoffs between robustness and cost, or between various objectives; and (3) the flexibility of various options to respond to changes in the future. In doing so, they put decision makers back in the driver’s seat. A growing set of case studies shows that these methods can be applied in real-world contexts and do not need to be more costly or complicated than traditional approaches. Finally, while this paper focuses on climate change, a better treatment of uncertainties and disagreement would in general improve decision making and development outcomes.

This paper is a product of the Office of the Chief Economist, Climate Change Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at shallegatte@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Agreeing on Robust Decisions: New Processes for Decision Making Under Deep Uncertainty**

Nidhi Kalra, Stéphane Hallegatte, Robert Lempert, Casey Brown, Adrian Fozzard, Stuart Gill, Ankur Shah

**Keyword:** decision-making under uncertainty, investment, climate change, adaptation

**JEL:** D81, H54, O22, O18, Q54

Nidhi Kalra<sup>1,2</sup> , Stéphane Hallegatte<sup>2</sup> , Robert Lempert<sup>1</sup> , Casey Brown<sup>3</sup> , Adrian Fozzard<sup>4</sup> , Stuart Gill<sup>5</sup> , Ankur Shah<sup>6</sup> 1 RAND Corporation 2 The World Bank, Climate Change Group, Office of the Chief Economist 3 University of Massachusetts 4 The World Bank, Poverty reduction and economic management network 5 The Global Facility for Disaster Risk Reduction, The World Bank 6 Centre International de Recherche sur l’Environnement et le Développement The authors would like to thank Roberto Aiello, Arturo Ardila, Ken Chomitz, Leo Dobes, Patrice Dumas, Marianne Fay, Erick Fernandez, Nagaraja Harshadeep, Abas Jha, Todd Johnson, Marcelino Madrigal, Shomik Mehndiratta, John Nash, Niels Holm-Nielsen, Apurva Sanghi, Pascale Scandizzo, Tomas Serebrinsky, and Xiaoping Wang for their time, insights, and contributions. Parts of this article are based on Hallegatte et al. (2012).
# **TABLE OF CONTENTS**

|A. Introduction ....................................................................................................................................... 3|
|---|
|B. Decision and Metrics and decision processes ................................................................................... 7|
|C. Agree-on-Assumptions Processes ..................................................................................................... 7|
|Cost-Benefit Analysis and Flood Risk Management ......................................................................... 8|
|Real Option Analysis ....................................................................................................................... 10|
|Aggregation and Consensus building in an agree-on-assumption framework ................................. 13|
|D. Agree-on-Decisions processes ........................................................................................................ 15|
|Example: Forest Management using an Agree-on-decision Process ............................................... 17|
|Features of Robust Decisions ........................................................................................................... 20|
|Agree-on-Decisions Applications .................................................................................................... 21|
|Robust Decision Making for Water Planning in Southern California ............................................. 21|
|Climate Informed Decision Analysis For Managing the Great Lakes Basin ................................... 23|
|Info-Gap for Seismic Resilience in Water Systems ......................................................................... 25|
|E. Conclusions ..................................................................................................................................... 26|
|Appendix 1. Understanding Climate Uncertainty .................................................................................... 27|
|References ................................................................................................................................................ 30|

# **TABLE OF FIGURES**

|Figure 1. Three policies in the Paris agglomeration ................................................................................... 17|
|---|
|Figure 2. Steps in an RDM Analysis........................................................................................................... 22|
|Figure 3. Performance of IEUA’s Urban Water Management Plan in 200 Futures ................................... 23|
|Figure 4. Decision scaling steps compared to a traditional analytical process. .......................................... 25|
|Figure A1. Change in annual rainfall in 2080-2100 (with respect to the 1980-2000 period) in Africa<br>according to two climate models. ............................................................................................................... 27|
|Figure A2. Source of uncertainty over time globally (left) and in the British Isles (right). ........................ 28|

# **TABLE OF TABLES**

Table 1. Assessment of the performance of the three forest and flood management plans across four scenarios. ..................................................................................................................................................... 19

# **TABLE OF BOXES**

|Box 1. The difficulty of predicting a deeply uncertain future ....................................................................... 5|
|---|
|Box 2. Uncertainty in climate change projections ........................................................................................ 6|
|Box 3. Real options for planning in the Thames estuary ............................................................................ 11|
|Box 4. Setting the Social Cost of Carbon ................................................................................................... 13|

2
# <mark>A. INTRODUCTION</mark>

1. **Many investment and policy decisions have long-term consequences.** Infrastructure like power plants, roads, and dams often last for decades and need to be useful throughout their lifetimes. They will shape and be shaped by the future. These investments can shape development well beyond their lifetimes, sometimes for centuries, because the long-term socioeconomic system reorganizes itself around those changes. The effect of transport infrastructure on urban forms and economic activities can for example be observed over very long timeframes, sometimes even after the infrastructure has become obsolete (Gusdorf, Hallegatte, and Lahellec 2008; Bleakley and Lin 2010). Policies such as urbanization plans, risk management strategies, and building codes and norms can influence development for equally long. Therefore, to make sound plans we must consider the performance of our investments and decisions in the near and long term.

2. **Yet deep uncertainty about the future exacerbates the challenge of sound decision making.** Developing countries in particular have experienced unprecedented changes in their political economy, land use, demographics, and natural environment. Yet, as Box 1 illustrates, past evidence and current research suggests that our ability to predict the future is limited at best (Silver 2012; Kahneman 2011; Taleb 2007). Compounding the problem, parties to a decision often have competing priorities, beliefs, and preferences. These conditions lead to _deep uncertainty_ . Deep uncertainty occurs when parties to a decision do not know or cannot agree on (1) models that relate the key forces that shape the future, (2) probability distributions of key variables and parameters in these models, and/or (3) the value of alternative outcomes (R. J. Lempert, Popper, and Bankes 2003). In his seminal 1921 paper, Knight offered a similar definition, distinguishing between two kinds of ignorance about our uncertain future – that which we can reliably quantify (called Knightian _risk_ ) and that which we cannot (Knightian _uncertainty_ , which corresponds to deep uncertainty). For example, the likelihood of experiencing a car crash is easily calculable from ample historical data and is an example of Knightian risk. In contrast, likelihood estimates of long-term land use patterns or global economic growth would be neither reliable nor verifiable. They are Knightian uncertainties or _deep_ uncertainties.

3. **The impact of climate change looms large as a deep uncertainty with global consequences** (World Bank 2009; IPCC 2014) **.** There is scientific consensus that the planet is warming due to greenhouse gas emissions. This may bring radical changes in climate, with tremendous implications for the long-term success of near-term decisions in nearly all sectors. Yet, as Box 2 shows, there is deep uncertainty and disagreement among both scientists and policy makers about what the future climate will be in a particular region and the specific effects that climate change will have on different sectors and groups. Continued efforts by climate scientists and others to increase knowledge about the climate and future climate scenarios are valuable. However, uncertainties about climate change and its impacts may increase as scientific inquiry diversifies and deepens (Hallegatte 2009). Therefore, decision makers should accept the irreducible uncertainty about the future climate and formulate adaptation and mitigation policies to manage it.

4. **Failing to manage climate and other uncertainties will have serious consequences for adaptation.** At the very least it can result in inefficient and ineffective investments. At its worst, it can be maladaptive, hindering effective interventions and leading to social and economic outcomes that are worse than if the interventions had never taken place. Inefficiency may arise from underadaptation to climate change, i.e., if an intervention fails to consider future climate conditions or does not consider the full range of possible conditions, exposing the population and economy to adverse impacts. Inefficiency may also arise from over-adaptation, i.e. if an intervention turns out to be unnecessary. Construction of a dike provides a simple example of each, since the dike could prove to be either too low (under-adaptation) or too high (over-adaptation). Interventions can also lead to 3
maladaptation, where the intervention subsequently leads to responses that increase vulnerability to climate threats. For instance, building dikes may encourage settlement and investment in the protected area, potentially exposing people to high flood risk if sea level rise is higher than expected and increases in the dike height are not or cannot be undertaken.

5. **Failure to manage deep uncertainties and competing beliefs can also hinder much-needed mitigation efforts** . Deep uncertainties exist about how much we should mitigate CO2 emissions and at what cost, the effectiveness of different policies, and the availability of future technologies. Indeed, many who are reluctant to address climate change have cited uncertainty as a reason to postpone action. There is an urgent need for methods to manage deep uncertainty and reach sound adaptation and mitigation decisions.

6. **In this paper, we seek to help decision makers better manage uncertainty and disagreement, particularly around climate change, by guiding them to the right decision making processes** . Traditional decision processes ask us to first reduce uncertainty by agreeing on assumptions about current and future conditions, and _then_ analyze our decision options. When faced with disagreement and uncertainty, these traditional “Agree-on-Assumptions” processes lack transparency, are vulnerable to bias and gridlock, and lead us to brittle decisions – those that perform poorly when the future diverges from our projections. We can instead invert these steps, deferring on agreement until we have analyzed the options under many different assumptions. This inverted “Agree-on-Decisions” process promotes consensus around robust decisions and can help manage uncertainty and disagreement around climate change and other conditions. While some decision-making methodologies assume that the analysts or experts will provide the “best” solution to the decisionmaker, agree-on-decisions methodologies usually build on participatory processes and close interactions between experts and decision-makers. As such, they aim to “ _assist the decision maker to evaluate options, to develop strategies, and to evoke and evolve preferences in light of the analysis of uncertainties_ ” (Ben-Haim 2006). Such an approach is deemed more appropriate in the presence of deep uncertainty, and ensure that decisions are legitimate (Renn 2008; World Bank 2013).

7. **By making decisions with “inverted” processes that analyze options first and seek agreement second, we can**

- Generate buy-in by including diverse beliefs and making analyses more transparent;

- Identify strategies that are robust, performing well no matter what the future brings;

- Focus decision makers’ attention on the tradeoffs between decision options;

- Seek agreement where it matters most: on actions, rather than on assumptions;

- Inform the decision while making sure that decision-makers – not experts and analysts – make the decision.

8. **The rest of this paper is divided into four sections, B through E.** In Section B, we briefly distinguish between decision metrics and decision processes, and we posit that they should be considered separately given that any decision process can be used with any decision metric. Section C then focuses “Agree-on-Assumptions” processes while Section D presents “Agree-on-Decisions” processes. In the latter, we review features of robust decisions around which decision makers can reach consensus and offer three real-world applications. We conclude in Section E. The paper also offers an introduction to climate uncertainty in Appendix 1.

4
<!-- Start of picture text --> 22<br>ac} 20 2000 Actual 1975 Scenarios<br>Sa 18 !<br>Bie i<br>Es 1.6 }<br>ao 14 1990 5 Z<br>co U<br>28 12 ; Historical trend<br>atfF’ 10 1980wore continued<br>8 §a 87 4890 1973pote__,} 1970<br>O= 6 S00 wer 1960<br>oe 4 1950<br>> ees |)<br>2 agi 4999<br>a<br>o Le *1910<br>0 2 40 60 80 100 120 140 160 180<br>Energy use (10'> Btu per year}<br><!-- End of picture text -->

<!-- Start of picture text --> +6%<br>+5% %< x<br>x<br>+4%<br>+3%<br>a= +2%<br>3<br>°=<br>OG +1%<br>6 x<br>° 0%<br>-1%<br>-2%<br>-3% x<br>-4%<br>owrnwoneanodaoaaoaaoonoaoanon®mdd9sdrds=AanarnaoaaqodQaqgdadroeqnodod0dddéd=ds0dAnti nti dt oH TH TH NNNvonN oo@mNONrOOCNRTCCNTCUCNTCCUNECUNOUNNbs mwonreamnooogooo00Oo a4&©<br>Year<br><!-- End of picture text -->
<!-- Start of picture text --> ys as<br>== SSS SSE SSS"SS<br>En Ds (%)<br>-50 -40 -30 -20 -10 0 10 20 30 40 ~=©«6©50<br><!-- End of picture text -->
# <mark>B. DECISION AND METRICS AND DECISION PROCESSES</mark>

9. **We begin by making a distinction between decision metrics and decision processes.** Often, when we speak of approaches to decision making we are simultaneously referring to two distinct components of approaches:

- a. The _performance metric_ we use to differentiate between different decision options, and

- b. The _process_ we use to evaluate each option according to that performance metric and to come to a decision.

Consider, for example, a choice between investing in mobile clinics or centralized clinics to provide vaccines to children as cheaply as possible. We might use a metric of _cost per vaccination delivered_ to measure how well each option meets our objective. We can use different processes to assess each option according to this metric. We might first make assumptions about the geographic distribution of people in the region, their costs of traveling to different locations, the safety of different routes, and other conditions relevant to our choice. Based on these assumptions, we could calculate how many vaccinations each approach could deliver and the cost of each option, from which we could calculate our cost per vaccine. We would choose the least-cost approach, and use a sensitivity analysis to identify the main assumptions that drive our conclusion. We could instead use a different process while still using the metric of costs-per-vaccine. For instance, we might start by evaluating each option against a wide range of potential conditions and then choose the one that has the lower cost per vaccine across the widest range of conditions.

10. **Treating metrics and processes separately allows us to choose the right tools for a decision.** Metrics and processes have distinct strengths and weaknesses. Cost effectiveness metrics like the cost per vaccine can be valuable when it is difficult or impossible to monetize the benefit (such as lives saved) or when our options have the same main result (such as delivering vaccines). This is a characteristic of the _metric_ and not of the process. On the other hand, one might criticize the above steps as too sensitive to the assumptions about geographic distribution and travel costs, particularly when there are large data gaps. This is a characteristic of the _process_ , and not of the metric. In this paper, we focus the discussion on the processes used to evaluate and prioritize investments and policies and we investigate two types of processes: Agree-on-Assumptions processes and Agree-onDecision processes.

# <mark>C. AGREE-ON-ASSUMPTIONS PROCESSES</mark>

11. **One approach, which is consistent with most traditional decision processes, is to first reduce uncertainty by agreeing on assumptions about the current and future conditions and** **_second_ analyze our decision options.** The first step in many analyses, regardless of the metric, is to define the assumptions – about the present and the future – under which our investments must perform. This can be a single estimate of the future, as in our vaccine example, or a probability distribution. We then analyze our options, choosing the one that performs best under our assumptions. As one example, urban planners making flood risk investments would first characterize the future urban form, sea level rise, and other factors. They would second evaluate the risk reduction benefits (the metric) of their investment options (e.g. constructing dikes) under these assumptions. We might perform a sensitivity analysis to assess how much influence each assumption has on the outcome. Such approaches have also been termed “predict-then-act” (R. J. Lempert, Popper, et al. 2013) or “science first” (Dessai and Hulme 2007).

12. **When faced with disagreement and deep uncertainty, these traditional “Agree-onAssumptions” processes are vulnerable to bias and gridlock.** First, many important assumptions are buried in models, rather than in front of decision makers. This makes it difficult for decision makers to understand and assess potentially critical assumptions on which their investment decisions 7
hinge. Second, many factors are difficult, if not impossible, to predict. Stakeholders also know that the choice of assumptions drives the choice of investment option. They may press for assumptions that will lead to the options they already favor, making consensus difficult (R. J. Lempert, Popper, and Bankes 2003). We risk losing stakeholders’ buy-in early if the foundations of the decision process lack transparency, appear arbitrary, or do not include their beliefs.

13. **Agree-on-Assumptions approaches are vulnerable to reaching brittle decisions – ones that are optimal for a particular set of assumptions, but which perform poorly or even disastrously under other assumptions.** Sensitivity analyses are often not sufficient for exploring the full range of plausible assumptions and future conditions (Bonzanigo and Kalra 2014), and Agree-on-Assumptions create little opportunity for exploring the performance of our decision options under unexpected conditions. They yield no information about how our “optimal” solution performs if the future surprises us, and they do not guide us to solutions that might work well if the predicted future does not come to pass. Yet, there is a great need for understanding the effect of surprises and unexpected conditions: repeated studies have shown, human beings have a widespread tendency towards overconfidence, believing strongly in our ability to predict the future when we cannot (Kahneman 2011). The following case example of cost-benefit analysis illustrates these shortcomings.

# COST-BENEFIT ANALYSIS AND FLOOD RISK MANAGEMENT

14. **Cost-benefit analysis is one of the most widely used methodologies to inform investment decision-making.** In this methodology, all the costs and benefits of an investment are aggregated into a monetary value in present terms. That is, the methodology (1) calculates the value of each monetary consequence (e.g., a flux of revenues from the project) and non-monetary consequence (e.g., lives that are saved by the investment, the impact of ecosystems and biodiversity, loss of cultural heritage) over time and (2) translates the future costs and benefits into a “present value” using a discount rate. Costs and benefits can be transformed into a single metric such as the _benefit-cost ratio_ (the ratio of benefits to costs which, if larger than one, indicates the investment is desirable) or the _net present value_ of the project (difference between the benefits and the costs which, if larger than 0, indicates the investment is desirable). Investments can be ranked according to these metrics to guide the selection of those that are expected to generate the greatest benefit overall. Cost benefit analysis has a long pedigree and has been honed and refined over many years. It is widely used in the economic analysis for World Bank projects (Tan et al. 2001). For more details, see Boardman et al. (2011).

15. Cost-benefit analysis, as traditionally practiced, is an example of an Agree-on- Assumptions process since it can only be applied if stakeholders agree on how to quantify various impacts (i.e., agree on how to attribute a monetary value to non-monetary consequences such as lives saved) and how to aggregate impacts at different points in time (i.e., agree on the discount rate). If all stakeholders agree on these assumptions, then the methodology provides an unambiguous answer regarding the desirability of an investment and a clear ranking of investment alternatives.

16. **Consider the decision of whether to build infrastructure to protect New Orleans against category 5 hurricanes based on a cost-benefit metric.** We might ask, “ _Is the net present value of such an investment positive?”_ The answer depends on many assumptions about the cost of construction, the probability of a category 5 hurricane losses that would occur in the event of a hurricane, and discount rates. In a traditional approach, we might start by estimating these values. Engineers estimate that it will cost $20 billion to build and operate the protection system. The benefits can be can be estimated as a discounted sum of the benefits (i.e. the avoided losses), over the lifetime of the protection (Hallegatte 2006): 8
where

- T is the lifetime of the protection;

- pn is the annual probability that a Category 5 hurricane hits New Orleans, which is estimated at about 1/500;

- δ is the a discount rate, for which US regulations require assuming two values -- 3 and 7 percent – in two different calculations.

- dn is an estimate of the annual direct cost of the New Orleans flooding, estimated at approximately $20 billion in damages and $5 billion in human losses;

Under these assumptions, the expected present benefit of a category 5 flood protection system in New Orleans can be calculated at $1.3 billion with a 7 percent discount rate and $6 billion with a 3 percent discount rate. This results in a net present value of -$18.3 billion or -$14 billion, respectively. This rough estimate would rule out the development of a system that protects against category 5 storms.

17. **There is no scientific agreement or political consensus on the assumptions upon which this conclusion rests.** In general, nearly all parameters in a cost benefit analysis of long-term investments are deeply uncertain. In the above example, this includes:

- <u>Discount rate. As illustrated by our comparison of 3 and 7 percent discount rates, the influence of</u> this political choice is large. Since this choice is very controversial and depends on ethical judgment on which it is difficult to reach consensus, the assessment of the system’s benefit will remain very uncertain and controversial.

- <u>Probability of occurrence. Suppose climate change and subsidence result in a five-fold increase in</u> the probability of the floods currently caused by category 5 hurricanes over the 21st century, an increase that is within the bounds of current estimates. Then expected benefits from protection against category 5 hurricanes would increase from $1.3 to $2.4 billion or from $6 to $23 billion, using a discount rate of 7 percent and 3 percent, respectively. In the former case, the system would have a negative NPV and remain inadvisable, while in the latter it would have a positive NPV and become justifiable.

- <u>Flood damages. Flood damages evaluated by insurance companies are poor proxies of welfare</u> costs, especially for large-scale events (Hallegatte 2014). A conservative estimate of the actual overall cost of the New Orleans floods is at least $60 billion, three times the insurers’ approximation based on direct losses only. Using the new values of event probability and potential damages, the expected benefit of an upgraded protection system would be $4.8 billion with a 7 percent discount rate and $46 billion with a 3 percent discount rate.

- <u>Countervailing risks and side effects. The implementation of a large-scale protection system can</u> attract more people in at-risk location and increase exposure to floods if the defense fails. But it can also attract more activities, improve infrastructure, and create jobs and income, thus improving welfare more than an analysis of direct costs only would suggest. The above equation assumes countervailing risks and side effects do not exist, but in reality they can have significant though uncertain implications. These effects are very difficult to estimate and can easily double (or halve) expected benefits, making benefits in the above example range from $0.6 billion (roughly half of 1.3 billion) to $92 billion (twice $46 billion).

- <u>Risk aversion. The above equation also assumes that society is risk-neutral. A risk-neutral agent</u> weighs positive and negative risk equally and is not affected by the degree of risk. For example, a risk neutral agent would not see any difference between losing $1 with certainty and having a 10

9
percent chance of losing $10, because the expected loss is the same in both cases. Yet people are often not risk neutral, e.g. preferring to pay disproportionately more to avoid large negative outcomes (Kahneman and Tversky 1979). Including an aversion to risk increases the benefit from protection.

18. **In such a situation of deep uncertainty, it is sometimes possible to assign subjective probabilities** , i.e. beliefs on the likelihood of different possible future conditions. We could then evaluate the expected benefits as the probability-weighted average of the benefits in the different possible future conditions. For example, we may believe that by 2100 there is a P=1/3 chance that the category-5 hurricane remains a 1 in 500 year event, and a (1-P)=2/3 chance that it increases to 1-in100-year event. Then the expected benefit can be written: Where

- pn stays constant at 1/500, reflecting a state of the world in which climate change has no influence on hurricanes;

- p’n increases from 1/500 to 1/100 by 2100, reflecting a state of the world in which climate change causes a five-fold increase in the likelihood of a category 5 hurricane landfall by the end of the century;

- d0 is the present-day damages from a category 5 hurricane landfall;

- g is the annual economic growth, estimated at 3 percent.

19. **The problem is that – for climate change and many other deep uncertainties – we do not have a strong methodology to assign these subjective probabilities.** The probability of a category 5 hurricane under future climate change, for example, cannot be fully based on historical data, because climate change is a new process for which we have no past equivalent. Models of climate change share common flaws and their dispersion cannot be used to assess the real uncertainty. Moreover, when using probability distribution function for different outcomes, the Agree-on-Assumptions approach is extremely sensitive to tails of distribution function. As suggested by Weitzman (2009), for instance, the "optimal" policy regarding climate change mitigation is highly dependent on the lowprobability high-impact possible futures, on which knowledge is very limited and uncertainty is particularly large. At the extreme, Weitzman shows that if the probability distribution function has a "heavy tail" (i.e. decreases less than exponentially for increasing damage values), then a cost-benefit analysis suggests that all GHG emissions should be stopped immediately.

REAL OPTION ANALYSIS 20. **A decision is often not strictly between “investing” and “not investing” but between “investing now” and “investing later.”** To help make this type of decision, some have proposed to mobilize the “real option,” which was initially developed for financial markets (Arrow and Fisher 1974; Henry 1974; Ha-Duong 1998; Pindyck 2001; Gollier C. and Treich N. 2003).

21. **Real-options (RO) analysis generalizes cost-benefit analysis by including the possibility of delaying decisions, revising them over time, or making a series of sequential decisions over time as more information becomes available.** A real option is the ability, but not the obligation, to undertake a project of uncertain future benefits at a known cost. In finance, an option is the right to purchase an asset at a future point in time for a specific price acquired through the payment of a fee. The option may be realized against the payment of the balance of the agreed price, allowing the 10
option holder to make a profit if market prices for the asset rise above the price of the option. The option holder is not required to exercise this option and so limits losses if the market price falls below the price of the option to the amount of the fee paid to acquire the option.

22. **Applied to decision-making, real option values the options created and destroyed by a project, alongside its expected net present value.** The analysis itself does not differ from a classical costbenefit analysis, except that the NPV includes additional consideration, namely the options created and destroyed by the project. The project's Extended NPV is calculated as follows: ENPV= Expected Net Present Value + ( Value of Options created – Value of Options Destroyed ) Thus an investment that has positive net benefits (exploiting existing capabilities) but fails to create new options may be less desirable than an investment with fewer direct benefits but which results in increased options (the ability to explore different opportunities later). In other words, there is a value in implementing a project that does not provide any benefit per se, but makes it possible to implement another project at a later point in time.

23. **Applications of real-option approaches are difficult because of the technical complexity of the analysis; however, there are a few applications to real cases.** Scandizzo (2011) applied real options to the development of sea defenses in Campeche, Mexico. A comparison of the Extended NPV for building a sea wall and for restoring a mangrove forest revealed that construction of a high sea wall would be uneconomical for several decades. In contrast, the cost of planting and maintaining a bioshield represents an option premium that creates the opportunity to delay the building of an expensive seawall until it is required. Box 3 describes how real options are embedded into a strategic planning exercise for the Thames Estuary. Real options analysis also has value for understanding climate change mitigation options (Ha-Duong 1998). Investing in research and development on renewable energy may have a negative NPV if considered in isolation from other policies. But such investments may create new technologies that could provide large benefits in 20 years when very ambitious climate policies are implemented. So, an investment in renewable energy research and development may be desirable because it will create the option of rapid decarbonization in the future, not because of its own return.

24. **Real options approaches allow for a better inclusion of the time dimension in planning, including the option to delay and the introduction of more flexible solutions. Despite these strengths, traditional real-options analysis is an extension of the traditional Benefit-Cost analysis and remains an Agree-on-Assumptions process.** Similarly to the cost-benefit analysis, it requires agreement on the value of future options, which may be deeply uncertain, and an agreement on the probability of various outcomes **.** When the assessment of outcome probabilities is impossible – at least in a consensual way – or when there is strong disagreement on values, real-option approaches are thus as difficult to implement as traditional cost-benefit analysis. In particular, they are vulnerable to the same shortcomings, including bias, gridlock, and brittle decisions.

# **Box 3. Real options for planning in the Thames estuary**

The UK Treasury’s Green Book (guidance for policy, program, and project appraisal) cites the planning for the Thames Estuary 2100 as an application of Real Options approach to climate change. The Thames estuary floodplain contains 1.25 million people, about £200bn of property, and key transport and infrastructure assets, 16 hospitals and eight power stations. The planning process followed a five step approach:

**Stage 1: Assess Climate Risks** . A series of sea level rise and storm surge scenarios were identified, including a central ‘most likely’ scenario.

11
<!-- Start of picture text --> Max water level rise: Defra and upper part of Top of new Previous<br>new TE2100 likely range H++ range extreme<br>[om tf tm [2m : [3m am |<br>i 1 1<br>Improve Thames Barrier & raise d/s defences : 1<br>HLOZ] ! '<br>, T {<br>Barrier and restore Flood storage, improve 1 I<br>interim defences Barrier, raise u/s : ;<br>' t<br>1 F 1 1<br>Existing system ' Fo ' '<br>i ) , i] 1<br>1!<br>Raise Flood stonge, Thames 1' H<br>' }<br>Flood L] 1<br>interim restore ' t<br>' HLO 3a It<br>i]: | ':'<br>New barrier, retain Thames Barrier, raise defences ‘to4<br>' Hip 3b ;<br>New barrier, raise defences<br>1 i]' 1!<br>1 i] 1<br>Key: ===> Predicted max water level under each scenario<br>Measures for managing flood ritk indicating effective range against water level<br><!-- End of picture text -->
AGGREGATION AND CONSENSUS BUILDING IN AN AGREE-ON-ASSUMPTION FRAMEWORK 25. **As the above examples show, because Agree-on-Assumptions approaches rely on our being able to come to consensus on the parameters that affect a decision, they break down when there are ethical judgments and differing worldviews.** This is particularly the case of cost-benefit analysis, which seeks to aggregate all categories of costs and benefits into a single metric. This encourages analysts to use models and relationships for which we have or can potentially produce consensus. This may mean ignoring or assuming away many large and hidden uncertainties for which there is little consensus or knowledge. Additionally, choosing the costs and benefits to include can be highly subjective and reflect the priorities of analysts, rather than the priorities of stakeholders. For instance, should GHG emissions be a factor in prioritizing new energy generation projects? Those principally concerned with providing lowcost energy may argue that it should be given little weight or ignored, while those concerned with climate change may argue that it should be a key consideration.

26. **Decision makers must also ask themselves, “What evidence should we use to evaluate the options?”** Cost benefit metrics typically use market values to set the relative importance of different criteria. However, analysts may judge that no market value exists or the market undervalues a good. This is another subjective choice around which there is much disagreement. Do the market prices for developing shoreline adequately reflect the value of ecosystem services that coastal habitats provide? Does the market price of carbon emissions adequately reflect the future impact of climate change? If not, analysts may use shadow prices instead of market prices. But, as Box 4 shows, this too can be contentious, and different preferences for discount rate and other factors may lead to different costs.

# **Box 4. Setting the Social Cost of Carbon**

The United States has developed estimates of the Social Cost of Carbon (SCC) to analyze regulatory policies. The SCC was constructed using integrated assessment models that value discounted global damages arising from climate change scenarios through to 2050. The selection of the discount rate has a significant impact, given that costs are estimated over fifty years and that costs will increase as more severe impacts are felt over time. The SCC is presented as a range of values with four discount rates, 5 percent, 3 percent, 2.5 percent and 1 percent, with 3 percent as the central estimate. In 2010 the SCC at the 3 percent discount rate is estimated at $21 per ton of CO2 (tCO2) (at 2007 prices) rising 2 to 3 percent a year in real terms.

The United Kingdom uses a different approach and value for the SCC. The UK set an official shadow price for Carbon at £25 per ton of CO2 equivalent (tCO2e) in 2007 for the purposes of cost-benefit analysis of government policies, programs, and projects. This was based on the SCC advised by Stern Review, equivalent to £19/tCO2e. This is higher than the values in the US because the UK uses a lower discount rate and equity weighting. The shadow price was set higher than the underlying SCC in order to adequately reflect abatement costs that would be incurred to meet the government’s abatement goal (and thereby incentivize action) and in recognition of the government’s desire to be seen as a leader in climate change action. The SCC would increase by 2 percent a year to capture the rising incremental damage of each unit of carbon as temperatures rise. It would be subject to periodic review to assess progress towards the government’s abatement objectives and target emissions reductions.

This policy was substantially revised in 2009, when the UK shifted to a target-consistent approach, based on estimates of the abatement costs that would be incurred to meet specific emissions reduction 13
targets laid out in the Government’s Carbon Budget. SCC would continue to be monitored but would no longer provide the basis for setting the carbon price because of the uncertainty surrounding SCC estimates (Pindyck 2013). Two prices are now in use. For appraising policies in sectors covered by the EU Emissions Trading System (ETS), estimates of future traded carbon price are used, giving a carbon price of £25 in 2020, with a range of £14 - £31. For appraising policies in sectors not covered by the EU ETS, a “non-traded price of carbon‟ will be used, based on estimates of the marginal abatement cost required to meet a specific emission reduction target, with a price of £60 tCO2e in 2020, and a range of £30 - £90. These estimates are periodically revised.

_Sources:_ (Wolverton et al. 2012; Price, Thornton, and Nelson 2007; UK Department of Energy & Climate Change 2009) 27. **In general, the value of costs and benefits can be deeply uncertain in the absence of readily available market prices or if market prices do not adequately capture value** . This includes valuations of environmental costs and benefits and intangibles such as the well-being of persons. The benefits of building a sea wall, for example, are likely to be higher than the damage costs alone because of the suffering and inconvenience from flooding. In these cases cost-benefit analysis requires the construction of ‘shadow’ prices that better reflect utility. There are many different approaches to establishing shadow pricing, each of which may reveal a different shadow price and each of which has different strengths and weaknesses nearly all of which are data intensive. These include surveys, choice modeling (Bennett, Van Bueren, and Whitten 2004), and hedonmic price modeling (Mahan, Polasky, and Adams 2000). The choice of method can be controversial, particularly if the resulting shadow price has a significant impact on the performance of the policies or investments being considered.

28. **Choice of discount rate is particularly contentious in the context of climate change** . The discount rate determines the viability of climate change mitigation and adaptation projects, given that costs are incurred in the short-term and benefits may only accrue in the distant future. High discount rates will discourage these investments. As with market prices, there is little agreement among economists as to the theoretically ‘correct’ method of setting discount rates, particularly for programs and projects that will generate costs and benefits over several generations. In practice, discount rates are set by the central finance agency as a ‘plug-in value’ for use in policy, program or program appraisal. Most developed countries apply much lower discount rates, mostly in the range of 3 to7 per cent, with many reducing their rates in recent years. Developing countries use a much higher discount rate, reflecting the higher opportunity cost of capital, in the range of 8 to 15 percent. Multilateral development banks follow similar practices with high discount rates (Zhuang et al. 2007).

29. **CBA or RO approaches often ignore the distribution of costs and benefits between different social groups, since those who benefit can theoretically compensate those who bear the costs.** In practice, however, it will rarely be the case that the beneficiaries of a government project will actually compensate the losers. There are multiple ways of taking this into account, including through distributional weights (Fleurbaey and Hammond 2004; Harberger 1984; Harberger 1978). In many instances, it is extremely difficult to build a consensus on which distributional weights should be applied; and the large impact of these weights on decisions makes it even more difficult to create a consensus. Given the many potential perspectives on the issue of equity, it is unlikely that any particular evaluation tool can provide a satisfactory resolution to the problem: it is typically recommended in CBA that an unweighted analysis be carried out in addition to any weighting exercise.

30. **These problems suggest that we should seek to encompass the whole set of possible assumptions to check the robustness of our assessment of options according to whatever metric we have chosen.** The example of New Orleans and hurricane protection shows that we can arrive at very 14
different net present values for an investment, even with reasonable parameter values. Here benefits from protection range from $0.6 billion to $140 billion depending on scientific assumptions (e.g., the impact of climate change on hurricanes) and ethical judgments and views (e.g., the aversion to risk and inequality). This situation is common and the Agree-on-Assumptions approach can rarely be used to make a decision in an objective way when uncertainty and disagreement are large. Thus, it is interesting and useful to explore alternative ways to approach such decisions.

# <mark>D. AGREE-ON-DECISIONS PROCESSES</mark>

31. **We can manage deep uncertainty by seeking a robust decision -- one that performs well across a wide range of futures, preferences, and worldviews, though it may not be optimal in any particular one.** Consider two crops: Crop A provides steady yield in drought or excessive rain, while Crop B provides still greater yield under specific conditions consistent with historical precipitation, but fails otherwise. If we could control precipitation or could reliably predict that this year’s precipitation would look like the past, we would do well to plant Crop B and maximize yield. But this decision is likely to be brittle – we can rarely predict precipitation, and we may instead prefer to hedge our bets and plant Crop A if Crop B appears too vulnerable. Robustness becomes important when the consequences of making a wrong decision are high. If crop insurance is available to help protect against potentially poor yields, or if sufficient savings are available, optimizing (and coping with bad years) may be the best strategy. If these tools and resources are not available and the consequences of a few years of low yields are disastrous, then robustness becomes a priority.

32. **We can identify robust strategies by inverting the traditional steps, i.e. using Agree-onDecisions processes.** They are sometimes also called “context-first” methods (Ranger et al. 2010). They start by “stress-testing” our options under a wide range of plausible conditions, without requiring us to decide or agree upon which conditions are more or less likely. They evaluate our decision options repeatedly, under many different sets of assumptions. We can evaluate our options under low-likelihood, but high-consequence events. We can treat as uncertain the assumptions buried in models. We can use every stakeholder’s beliefs about the future – we don’t need to agree first on these assumptions. This reveals which of our options are robust – meeting our needs under a wide range of conditions, rather than performing well in only a few. Analytical tools can then help us identify the specific conditions in which each option no longer meets our goals. For example, in stress testing a flood risk management strategy involving dikes, we might find that it results in high risk if two conditions occur: (a) sea level rise proves higher than expected and (b) rapid development results in a larger than forecast population living behind the dikes. This focuses our attention on conditions that matter to the decision and on reaching agreement upon a decision option, without necessarily agreeing on the assumptions that might lead us to prefer that option.

33. **This inverted process promotes consensus around decisions and can help manage deep uncertainty around climate change.** Analyses performed in this way do not make the decisions for decision makers. Instead, they help decision makers debate important questions:

- Are the conditions under which our option performs poorly sufficiently likely that we should choose a different option?

- What are tradeoffs do we wish to make between robustness and, for example, cost?

- Which options leave us with most flexibility to respond to changes in the future?

Decision makers may conclude, for example, that the threat of unexpectedly high sea level rise is sufficiently great to warrant modifying the dike plan or augmenting it with other policies. Knowing that dikes may fail to reduce risk in a future with high sea level rise and extensive urban development, decision makers might modify the current plan to increase dike height or, alternatively, augment the 15
original dike design with policies to shift development away from the dikes. These can be difficult debates. But, they are much more useful than debates about the unknowable future.

34. **Importantly, these Agree-on-Decisions methodologies can use the very same metrics as Agreeon-Assumptions methods.** For instance, the net present value of a project or the benefit-cost ratio (the metrics often used in cost-benefit analysis) will play an important role in economic analyses that use Agree-on-Decisions methods (e.g., Bonzanigo and Kalra 2014; Lempert, Sriver, and Keller 2012). In other words, the metrics for evaluating and comparing decision options is the same, but the decision-making process is different.

35. **One of the potential strengths of Agree-on-Assumptions methods is that they can rank decision options and identify the one that optimize decision makers’ metrics.** However, this is only possible if all priorities and concerns are aggregated into a single metric along which the decision options can be monotonically ranked.

36. **However, it is often useful to use multiple metrics to evaluate decision options** (e.g., the net present value can be complemented with a metrics for distributional impacts), which avoids the difficult problem of having to aggregate all costs and benefits into a unique metric. While Agree-onAssumptions can use multiple metrics, they often aggregate those metrics into a single weighted utility function in order to achieve a ranking of projects. Agree-on-Decisions are better suited to using multiple disaggregated and distinct metrics since the aim is often to find decisions that work well across a range of assumptions and a range of goals or objectives.

37. **Using multiple distinct metrics has at least three major benefits** :

- **First it helps stakeholders with different values reach consensus,** since it does not require starting the analysis with an ex ante agreement on valuation techniques and relative prices and does not provide a unique ranking of projects. For instance, evaluating a risk management project through a cost-benefit analysis requires an agreement on the value of life, which is controversial and can result in gridlock. This can be avoided if we instead count the number of lives saved as a separate metric.

- **Second, by tracking diverse impacts along the analysis, multiple metrics help identify the major trade-offs implied by the decision.** Consider for instance the choice between two options to mitigate risks: building a dike or restoring a mangrove forest. While a cost-benefit analysis would produce a ranking as the final outcome, there may be an important tradeoff between these choices. For instance, it may be that the dike has a lower cost but leads to environmental losses while the more expensive ecosystem-based approach creates co-benefits from recreation and biodiversity. A constructive discussion among stakeholders with divergent views is more likely if this trade-offs is identified and explicit.

- **Third, identifying tradeoffs helps design policy mixes in which complementary policies smooth or mitigate adverse effects for some stakeholders or in some sectors.** For instance, Figure 1 shows the impact of various climate-urban policies in the Paris agglomeration –  a green belt policy, a public transport subsidy, and flood zoning – as measured by five metrics. Each of these policies has benefits along certain metrics and costs along others. The net benefit depends on how one values these different metrics, and agreeing on an aggregation techniques would be difficult. An analysis that tracks multiple metrics simultaneously makes it possible to add specific actions to correct for the negative outcomes. For example, a green belt policy (in which construction of new buildings is restricted) has a negative impact on housing affordability, and may become socially acceptable only if accompanied by complementary actions to reduce housing and building construction prices (e.g., with a reduction in land acquisition taxes). These options may not be evident when using a single, aggregated metric.

16
<!-- Start of picture text --> Climate change<br>mitigation<br>— Adaptation and<br>oe « ~ natural risk - - — Do-nothing scenario<br>affordability ~ rT ducti<br>;<br>— Green belt policy<br>|<br>/ © Public transport subsidy<br>Ned Zoning policy to reduce the risk of<br>Natural area flooding<br>Policy neutrality’ ‘and biodiversity<br>protection<br><!-- End of picture text -->
to attribute subjective probabilities to the different scenarios and calculate the expected value of each option. However, as noted earlier, we cannot develop reliable probabilities for future climate or other conditions, and so this approach could also lead decision makers astray.

40. **As an alternative, we could identify the plan that is robust, working well across all the scenarios.** Table 1 shows the performance of each of three candidate plans in each of the four scenarios developed by experts, rated from a very high positive NPV (+4) to a very negative NPV (- 4). Looking at the performance across all scenarios helps understand the vulnerability and robustness of each of the plans. First, this analysis suggests that no plan is the strongest in all cases. While Plan 1 has the highest NPV in Scenario A, it also has the lowest NPV in Scenario B, if heavy erosion, siltation, and flooding occur. This suggests that Plan 1 is particularly vulnerable to heavy rainfall. It would also have excess flood mitigation capacity in scenarios C and D and might not be able to meet timber demands either. Those concerned about worst-case plausible scenarios might eliminate Plan 1 as an option. Plans 2 and 3, on the other hand, are more robust, performing reasonably well in nearly all futures.

41. **One way to quantify the performance is to calculate each plan’s regret.** Consistent with our use of the term in everyday life, mathematical regret is the difference between the utility of a decision in a particular scenario, and utility of the best decision that could have been made in that scenario. For example, Plan 1 performs very poorly in Scenario B — resulting in a utility of -2. The best option in Scenario B would have been Plan 2, with a utility of 3. This gives Plan 1 a very high potential regret of 5 (3 – (-2)) in Scenario B. The table calculates regret for each plan in each scenario, where 0 is the lowest possible regret.

42. **Plans that have the lowest possible regret across all scenarios are often attractive, as they suggest that the plan will work well no matter what the future brings.** Plans 2 and 3 have the same maximum regret of 1. The average regret is also a useful metric. If we assumed that all scenarios are equally probable, a defensible assumption if we have no additional information on the scenarios, Plan 3 has a slightly lower average regret of 0.50, in addition to having a low worst-case regret. Experts could choose between Plan 2 and Plan 3 based on considerations of cost, feasibility, and other factors, or based on later judgments of which scenarios may be more or less likely.

43. **In addition, the analysis shows the potential of developing flexible strategies. It is perhaps easier to build a smaller dam first, with expansionary capability, than to build a larger dam that is excessive or inappropriate in various scenarios.** We might use a real options metric to calculate the present value of this future flexibility. Other approaches that are more flexible, such as the forest management policy, can be adapted as new information on climate and demand emerges.

18
||Plan|1|Plan|2|Plan|3||||
|---|---|---|---|---|---|---|---|---|---|
||1Mediumda<br>Someirrigatio<br>Noforestmgm<br>Performance|m<br>n canals__<br>t<br>(Regret)|2Small Dams<br>| Large irrigati<br>Large forest m<br>|Performance|on ponds<br>gmt<br>(Regret)|1Small Dam<br>Someirrigati<br>Small forestm<br>|Performance|on canals<br>gmt<br>(Regret)|Best<br>(Performance<br>In Scenario|Planw/Best<br>Performance<br>in Scenario||
|ScenarioA<br>||||||||||
|SameRainfall|4|oO|3|i|3|i|4|Plan 1||
|SameTimber||||||||||
|Scenario B<br>More Rainfall|-2|5|3|oO|2|i|3|Plan 2||
|MoreTimber||||||||||
|ScenarioC||||||||||
|LessTimber||||||||||
|ScenarioD<br>LessRainfall|1|1|1|1|2|fy)|2|Plan 3||
|LessTimber||||||||||
|Best<br>Performance<br>of Plan|4||||||4|Plan1|Planwithhighestperformancein<br>anysinglescenario|
|Performance<br>of Plan|0.75||||||25|Plan3|Plan with highestaverage<br>“<br>ae<br>performanceacross allscenarios|
|Maximum<br>Regret <sup>of Plan</sup>||5||1||1|1|Plan2Plan3|Plan withlowestmaximumregret<br>.<br>across allscenarios|
|Average Regret<br>gems<br>ofPlan||2.25||0.75||05|0.5|Plan3|Plan with lowest average<br>regret<br>es<br>acrossallscenarios|
FEATURES OF ROBUST DECISIONS 44. **The above example highlights several features that can make decisions robust in the face of deep uncertainties** These include but are not limited to (Hallegatte 2009):

- a. No- and low-regret decisions,

- b. Reversible and flexible decision,

- c. Safety-margin decisions, and

- d. Decisions with reduced time horizons.

45. **“No-regret” or “low-regret” decisions have high utility no matter what the future brings.** Thus, they can be robust even to deep uncertainties. For example, reducing leaks in water distribution systems is almost always a good investment, regardless of how the climate, future demand, and other factors change. Land-use policies that seek to limit urbanization and development in low-lying, floodprone areas would reduce disaster losses in the present climate. Climate change, which may increase the frequency or intensity of storms, may make such a policy even more desirable. These examples suggest that finding a system’s existing shortcomings may reveal no-regret or low-regret strategies: such strategies are beneficial over the short term (and thus easier to implement from a sociopolitical point of view) and may offer benefits under a wide range of future conditions. This approach needs to be used with caution, however, to avoid solving near-term problems with decisions that are sensitive to future uncertainties. Building a diversion to transfer water across basins may help reduce near-term water scarcity. However, such investments can take decades and cost billions, and climate change may alter precipitation patterns, reducing or negating the value of these investments by the time they are completed.

46. **Reversible and flexible decisions are typically more robust than irreversible ones because they enable us to adjust our decisions as new information becomes available.** In this way, reversible and flexible decisions can help us reduce our regret. For example, insurance and early warning systems can be adjusted every year in response to the new information on emerging risks. As another example, when deciding whether to allow the urbanization of an area potentially at risk of flooding due to climate change-induced increases in river runoff, the decision-maker must be aware that one option is reversible while the other is not. Restricting urbanization has a short-term cost, but if new information shows that the area is safe, urbanization can be allowed immediately. This option, therefore, is highly reversible, even though it is not without cost since it may prevent profitable investments from being realized in the near term. Allowing urbanization now, on the other hand, is irreversible or very expensive to reverse. If the area becomes high risk in the future, the choice will be to abandon the area or to protect it, both of which may be difficult and expensive. This does not imply that urbanization should be rejected. Rather, in the decision-making process, the value of the reversibility of a strategy (the option value) should be taken into account.

47. **Flexibility can be built into engineering solutions.** The Stormwater Management and Road Tunnel (SMART) that runs under the financial district in Kuala Lumpur to relieve traffic congestion offers an example (“What Is SMART?” 2013). The tunnel has three levels: two for road traffic and a lower level for carrying flash floods from the Klang River under the city and out to the Kerayong River. During major storms, cars are excluded from the two traffic lanes and gates are opened to allow stormwater to flow through the upper levels of the tunnel. Traffic can enter again within about 48 hours of closure. The additional construction cost of the multi-purpose tunnel and surface congestion costs during closure represent an option premium. But the cost of the multipurpose tunnel is less than a traffic-only tunnel combined with a duplicate tunnel dedicated solely to channelling intermittent floodwaters.

20
48. **Many “safety margin” strategies can reduce the risk of bad options at negative, zero, or negligible cost.** For instance, to calibrate drainage infrastructure, in 2008 water managers in Copenhagen used runoff estimates that were 70 percent larger than their current level. This margin helps manage increases in drainage needs due to future population growth and due to potential climate change, which may increase heavy drainage demands in Denmark. This 70 percent increase has not been precisely calibrated because such a calibration is made challenging by climate change uncertainty. But this increase is thought to be large enough to cope with almost any possible climate change during this century, considering the information provided by all climate models. This move is justified because, in the design phase, it is inexpensive to implement a drainage system able to cope with increased precipitation. On the other hand, modifying the system after it has been built is difficult and expensive. It is wise, therefore, to seek to be over-pessimistic in the design phase where possible (Hallegatte 2009).

49. **Cheap safety margins are especially important for adaptation measures that are not reversible or flexible.** Irreversible decisions (e.g., development in coastal areas) without inexpensive safety margins are particularly brittle to deep climate uncertainties – they can neither be undone nor protected. Adding safety margins to irreversible decisions (e.g., adding coastal defenses or improving urban water-management infrastructure) can improve robustness, but only if we carefully consider the range of potential future climate change scenarios. At the same time, we must recognize that resources are extremely constrained in developing countries, and safety margins with modest or even low-costs may be beyond reach.

50. **Reducing the lifetime of investments is one way of reducing uncertainty around a decision.** The uncertainty regarding future climate conditions increases rapidly with time. This strategy has already been implemented in the forestry sector with plants species that have a shorter rotation time. Since species choice cannot be made reversible and no safety margins are available in this sector, this option is interesting in spite of its cost. In other sectors, it is also often possible to avoid long-term commitment and choose shorter-lived decisions. For example, if houses are to be built in an area that may become at risk of flooding if precipitation increases, it may be useful to build cheaper houses with shorter lifetimes instead of high-quality houses meant to last one hundred years under historical conditions **.** AGREE-ON-DECISIONS APPLICATIONS 51. **There are many practical applications of Agree-on-Decision processes.** The specific methodologies share a common underpinning: they manage uncertainty by seeking decisions that are robust, performing well across a wide range of potential futures. This section summarizes three applications of these methodologies to investment decisions: Robust Decision Making applied to water resource planning in Southern California; Climate Informed Decision Analysis applied to management of the Great Lakes; and Info-Gap for seismic resilience in water systems. These examples are not exhaustive – there are many other Agree-on-Decision processes and many other applications of these processes. Rather, we seek to illustrate how these methods can be successfully applied to complex real-world problems that face deep uncertainty.

ROBUST DECISION MAKING FOR WATER PLANNING IN SOUTHERN CALIFORNIA 52. **Robust Decision Making (RDM) is a decision framework developed specifically for decisions with long-term consequences and deep uncertainty** (Schwartz 1996; R. J. Lempert and Schlesinger 2000; R. J. Lempert et al. 2006; Groves and Lempert 2007; R. J. Lempert and Collins 2007; Stéphane Hallegatte 2009; R. J. Lempert, Popper, et al. 2013). RDM is iterative and follows a “deliberation 21
with analysis” process recommended by the U.S. National Research Council for managing climate uncertainty (2009). It has been applied to water resource management (Groves, Fischbach, et al. 2013; Groves, Bloom, et al. 2013; R. J. Lempert and Groves 2010) flood risk management (Fischbach 2010; R. J. Lempert, Kalra, et al. 2013), energy investments (Popper et al. 2009; Bonzanigo and Kalra 2014), and terrorism insurance (Dixon et al. 2007).

53. **RDM involves four main steps as shown in Figure 2.** First, stakeholders and analysts engage in a decision structuring exercise to identify the goals, uncertainties, and decision options to be analyzed. Second, analysts use computer simulation models to evaluate the performance of the decisions under hundreds or thousands of combinations of uncertainties, which produces a database of performance results. Third, visualizations and analysis of this database help decision makers and analysts understand the vulnerabilities of their potential decisions, i.e. conditions under which their choices would fail to meet their goals. They may suggest new decision options to mitigate those vulnerabilities, resulting in an iteration back to step 1. Alternatively, they may fourth consider tradeoffs between the options based on their vulnerabilities and other considerations. They iterate upon this process until decision makers agree upon a robust strategy. Also, RDM normally involves a detailed quantitative analysis, but the underlying framework can inform more “heuristic” – less resource-intensive – evaluations (R. J. Lempert and Kalra 2011). RDM is especially useful when project plans can be framed as a series of decisions over time, to take advantage of learning explicitly and adjust to new information as it comes along.

**Figure 2. Steps in an RDM Analysis**

_Source: (R. J. Lempert, Popper, et al. 2013)_ 54. **Robust Decision Making (RDM) was used to help the Inland Empire Utilities Agency (IEUA) of Southern California develop its long-term water management plan (R. J. Lempert, Popper, et al. 2013; R. J. Lempert and Groves 2010).** The utility sought a plan that would provide reliable, low-cost water to its customers for the long-term future. It sought to incorporate the impacts of climate change into its planning but was hampered by the deep uncertainty in future climate forecasts. The results of 21 atmosphere-ocean general circulation models scaled down to the Southern California region indicated that the climate could range from 0.1-2.1°C warmer and there could be a - 19 to +8% change in wintertime precipitation.

55. **The robustness of the utility’s Urban Water Management Plan (UWMP) to climate and other uncertainties was explored in the range of possible future climates and other socio-economic conditions.** The utility sought a plan that would have a low total cost, which included the cost to implement the plan and the potentially additional cost of meeting future water shortages with imports. The RDM analysis evaluated the cost of the UWMP in 200 futures with different assumptions regarding the extent of climate change, future socioeconomic conditions, the agency’s ability to 22
<!-- Start of picture text --> 4.0<br>a*.<br>g—_— 38 sok *.o a .eeos «#<br>=gs5 Hae : free* om 6 *. a<br>= of<br>Fn 3.0 fs pate ar . e<br>3 we Fa 8 “8 oe se<br>c . * . : * .<br>2.5<br>0 1.0 2.0 3.0 4.0<br>Present value shortage cost (5 billions)<br><!-- End of picture text -->
scenario was rejected in favor of a robust decision making process designed to incorporate multiple and at times conflicting sources of climate information. The analysis of the Great Lakes Basin plan comprised three phases.

59. **First, stakeholders defined performance benchmarks and identified vulnerabilities that would lead to unacceptable performance.** To prioritize concerns for the regulation of Lake Superior, stakeholder experts were convened from state, provincial, federal, and local government agencies, as well as special interest groups (boating, hydroelectricity, navigation), and environmental groups (e.g. The Nature Conservancy). The experts were tasked with identifying the metrics by which they would like the candidate regulation plans to be evaluated. Based on these goals, the stakeholder groups then defined what combination of lake level and duration led to acceptable outcomes, and what levels led to impacts that were either less favorable or unacceptable. In this way, thresholds designating of acceptable performance for candidate regulations plans were established. This allowed the evaluation of regulation plans using a common numeraire for each of the different interests, namely, the occurrence of unacceptable lake levels as defined by each interest.

60. **Second, stakeholders quantified the climate sensitivity for each proposed plan.** Analysts created a climate response function, which estimated the consequences (lake levels and associated performance metrics) of a given decision (regulation plan) for a set of mean climate conditions that were varied to evaluate a broadly defined plausible range of climate change. The function thus related climate effects to the performance metrics influencing the decision in a way that was independent of any assumptions about the relative likelihood of future climate conditions. Using a parametrically varied set of stochastic time series (each representing a given mean climate), analysts identified those climate conditions that presented risks (unacceptable lake levels) to each regulation plan. Note that climate model projections had not been used in the analysis to this point, yet the vulnerability of proposed plans to potential climate futures was revealed.

61. **Third, the plausibility (relative probability) of those conditions was estimated through tailored climate information.** Given the uncertainty associated with the probability estimates even after maximizing credibility, the term ‘‘plausibility’’ is used in place of probability. The decision maker, in this case the Study Board, was presented with the risk of each candidate regulation plan, where risk was defined as the plausibility of problematic climate conditions (the climate changes that would cause unacceptable performance). The plausibility estimates were represented as imprecise probabilities, where a range of probabilities was shown for each climate state based on the difference sources of climate information (e.g., climate projections, stochastic analysis, paleo-climate data, and expert opinion). Thus, plans could be evaluated in terms of the range of climate change over which they maintained acceptable lake levels, i.e., their robustness to climate change. The result was a climate-informed robustness index for each candidate plan. The plausibility estimates could be adjusted based on different comfort levels of the Board members with different sources of climate information. For example, some preferred considering only the probabilities generated from climate model projections, while others preferred only probabilities from historical observations.

62. **In sum, recommendation for a final regulation plan focused on two plans that performed best over a very wide range of future climates.** The analysis also revealed that even these “best” plans were not robust to climate changes that were quite plausible, leading the Study Board to also recommend that the regulation plan be coupled with an adaptive management program to address future uncertainty. The Board expressed satisfaction in the use of climate information to make this decision.

24
<!-- Start of picture text --> Traditional Approach Decision Scaling<br>oF 1. Downscale<br>Jb multipleprojections model plausibilitycaaaie3, Determine of<br>conditions/<br>2. Generate a few vulnerability<br>serieswater supply {\<br>3. Find whether<br>system is<br>vulnerable for 2. Link to climate<br>these series. conditions<br>Vulnerability domain<br>1. Determine the vulnerability domain<br><!-- End of picture text -->
recovering from an outage at reasonable levels” (Irias and Cicala 2013). In an initial study, EBMUD used Info-Gap to weigh two options: construct new water pipeline or tunnel crossings to the island or improve the seismic resilience of an existing pipeline. Their goal was to ensure that at any time, one crossing would survive a seismic event, and a second crossing could be returned to service quickly. The key uncertainties were the probabilities of failures and the length of time needed to conduct repairs, which is a sum of the time of individual repair steps: discovery, setup, investigation, and repair. The Info-Gap analysis is currently under way and is helping EBMUD answer two important questions (Irias and Cicala 2013):

- a. _How wrong can we be about the probabilities of various types of failures, and still have an acceptably large overall probability that a crossing survives?_

- b. _For any given design and construction alternative, how wrong can we be about the duration for return to service and still be within the maximum allowable down time?_

# <mark>E. CONCLUSIONS</mark>

67. **Investment decision making is already difficult for any diverse group of actors with different priorities and views.** But the presence of deep uncertainties linked to climate change further challenges decision-making by questioning the robustness of all purportedly optimal solutions. We suggest that decision makers can continue to use the decision metrics they have used in the past. Cost benefit, cost effectiveness, and multi-criteria metrics are still valid for measuring decision options. They differ in their strengths and drawbacks, but these characteristics are distinct from the challenges of applying these metrics in a deeply uncertain world.

68. We suggest that decision makers consider **alternative** decision processes **to traditional ones, especially** those that lead with analysis and end in agreement on decisions. Agree-on-Decision methods start by “stress-testing” options under a wide range of plausible conditions, without requiring us to agree ex ante on which conditions are more or less likely, and against a set of objectives or success metrics, without requiring us to agree ex ante on how to aggregate or weight them. As a result, these methods are easier to apply in contexts of large uncertainty or disagreement on values and objectives.

69. **This inverted process promotes consensus around better decisions and can help manage uncertainty** . Analyses performed in this way do not make the decisions for decision makers or provide a single “best” solution, _deus ex machina_ . Instead, they help decision-makers debate important questions: “Are the conditions under which an option is vulnerable sufficiently likely that we should choose a different option? What tradeoffs do we wish to make between robustness and cost, or between various objectives and policy goals? Which options leave us with most flexibility to respond to changes in the future?” These can be difficult debates. But, they are much more useful than debates about the unknowable future.

70. **A growing set of case studies shows that these methods can be applied in real-world contexts and do not need to be more costly or complicated than traditional approaches.** While this paper focuses on climate change, a better treatment of uncertainties and disagreement would in general improve decision-making and development outcomes.

26
<!-- Start of picture text --> ccsmMs<br>ee<br><!-- End of picture text --> <!-- Start of picture text --> GFDL-CM2.6<br>a<br><!-- End of picture text -->
<!-- Start of picture text --> 100 ‘Global, decadal mean surface air temperature 100 British isles, decadal mean surface air temperature<br>90 a0<br>= 80 a<br>70 = 70<br>i ie<br>> ><br>z* 3”<br>: 0 : 40<br>E> :*<br>20 a)<br>10 10<br>o0<br>0rt 4) Li Et 100 0 rl] 40 Li] a) 100<br>Lead time [years from 200s) Lead time [years from 2000]<br><!-- End of picture text -->
moderate role, and climate model uncertainty remains large (A2, right). This suggests that it is much more difficult to predict future climates when looking at one country or one region than globally, regardless of future progress in our understanding of climate change. Natural variability means that the climate signal is more difficult to extract (and – as already mentioned – forecasts of future climate remain out of reach).

8. **There is also large uncertainty from differences between global climate models.** The IPCC (2013) provides results from 42 global climate models. The models agree on the very big picture (more warming in high latitude than in low latitude; more precipitations in high latitudes; less precipitation around the tropics; more precipitation around the equator). However, the differences can be huge in some regions (e.g., half of the models predict an increase in precipitation over India; half of the models predict the opposite; and – as a consequence – the “average model” predicts no change, showing the risk of averaging projections).

9. **When looking locally, we usually do not use global climate models.** Instead, we use downscaling techniques which can be done with statistical tools or with regional climate models (RCM). Statistical methods use statistical relationships, calibrated on historical data, to relate large-scale drivers – which climate models can reproduce – to local phenomena – which climate models cannot reproduce (Elsner and Jagger 2006; Mestre and Hallegatte 2009; Nuissier et al. 2012). Even though our knowledge of the laws of physics helps select potential predictors, this method is not directly based on physical laws. Such statistical methods are computationally efficient and reproduce the current climate well. Statistical models, however, have two main drawbacks: first, they need long series of reliable data; second, even with a sufficiently large data range, it is difficult to know whether a statistical relationship will remain valid in a future climate.

10. **To avoid the problem of validity of historical relationships, one may use physical models, which are based on physical laws.** Physical models, which model and simulate biophysical and atmospheric processes, are of particular interest when investigating extreme patterns and variability changes. Of course, physical models often require calibration and bias correction, so the distinction between physical models and statistical models is sometimes fuzzy. Examples are Regional Climate Models (RCMs); see (Knutson et al. 2010). RCMs are difficult to develop and may only have resolution of about 25 km, but they are less data-dependent. The confidence in RCMs is generally higher because they do not have to assume that the relationship between large-scale and small-scale climate variables remains constant in a future – possibly very different – climate. Such an assumption is questionable. Thus, statistical analyses are more reliable over the short to medium term, while RCMs are necessary to understand large warming over the long term. Nonetheless, in the long term RCMs remain driven by the input from GCMs, and so they do not resolve uncertainty related to climate variability, for example, that is produced by the GCMs.

11. **In almost all cases, downscaling improves our ability to reproduce the current climate, but it does not reduce the uncertainty on future changes.** Downscaling makes it possible to represent additional mechanisms and is likely to improve both the ability of represent the current and future climate (for instance, downscaling technique is required to look at precipitation in coastal or mountainous areas, or to look at small-scale phenomenon like heavy precipitations). But downscaling cannot help manage uncertainty if global climate models disagree. As in West Africa and the monsoon, downscaling can increase the resolution of scenario with an increase in precipitation scenario and a scenario with a decrease in precipitation, but the difference between both will not be resolved.

29