metadata: authors: 0: author: Bonzanigo,Laura
1: author: Kalra,Nidhi Rajiv
count: World
docna: 0: docna: Making informed investment decisions in an
            uncertain world : a short demonstration
docty: Policy Research Working Paper
theme: Economic statistics, modeling and forecasting,Climate change
majtheme: Economic management,Environment and natural resource management
subtopic: Climate Change Mitigation and Green House Gases,Science of Climate Change,Climate Change and Health,Climate Change and Environment,Energy Policies & Economics,Hydrology,Energy Demand,Energy and Environment,Energy and Mining
teratopic: Energy,Macroeconomics and Economic Growth,Environment,Finance and Financial Sector Development
abstracts: cdata!: Governments invest billions of dollars
            annually in long-term projects. Yet deep uncertainties pose
            formidable challenges to making near-term decisions that
            make long-term sense. Methods that identify robust decisions
            have been recommended for investment lending but are not
            widely used. This paper seeks to help bridge this gap and,
            with a demonstration, motivate and equip analysts better to
            manage uncertainty in investment decisions. The paper first
            reviews the economic analysis of ten World Bank projects. It
            finds that analysts seek to manage uncertainty but use
            traditional approaches that do not evaluate options over the
            full range of possible futures. Second, the paper applies a
            different approach, Robust Decision Making, to the economic
            analysis of a 2006 World Bank project, the Electricity
            Generation Rehabilitation and Restructuring Project, which
            sought to improve Turkey's energy security. The
            analysis shows that Robust Decision Making can help decision
            makers answer specific and useful questions: How do options
            perform across a wide range of potential future conditions?
            Under what specific conditions does the leading option fail
            to meet decision makers' goals? Are those conditions
            sufficiently likely that decision makers should choose a
            different option? Such knowledge informs rather than
            replaces decision makers' deliberations. It can help
            them systematically, rigorously, and transparently compare
            their options and select one that is robust. Moreover, the
            paper demonstrates that analysts can use the same data and
            models for Robust Decision Making as are typically used in
            economic analyses. Finally, the paper discusses the
            challenges in applying such methods and how they can be overcome.
repnme: repnme: Making informed investment decisions in an
            uncertain world : a short demonstration
display_title: Making informed investment decisions
            in an uncertain world : a short demonstration
listing_relative_url: /research/2014/02/18895127/making-informed-investment-decisions-uncertain-world-short-demonstration
topicv3: Water m1326522 137,Energy and Extractives m1326595 507,Climate Change m1326829 567,Climate Change Mitigation m1327079 579,Power and Electricity Sector m1327094 515
publishtoextweb_dt: 2014-02-01T00:00:00Z
historic_topic: Energy,Macroeconomics and Economic Growth,Environment,Finance and Financial Sector Development
no_of_pages: 22
author_keys: 0: author_key: 1255963
1: author_key: 1245653
model_extractions: WPS6765 Policy Research Working Paper 6765 Making Informed Investment Decisions in an Uncertain World A Short Demonstration _Laura Bonzanigo Nidhi Kalra_ The World Bank Sustainable Development Network Office of the Chief Economist February 2014
Policy Research Working Paper 6765

# **Abstract**

Governments invest billions of dollars annually in longterm projects. Yet deep uncertainties pose formidable challenges to making near-term decisions that make longterm sense. Methods that identify robust decisions have been recommended for investment lending but are not widely used. This paper seeks to help bridge this gap and, with a demonstration, motivate and equip analysts better to manage uncertainty in investment decisions. The paper first reviews the economic analysis of ten World Bank projects. It finds that analysts seek to manage uncertainty but use traditional approaches that do not evaluate options over the full range of possible futures. Second, the paper applies a different approach, Robust Decision Making, to the economic analysis of a 2006 World Bank project, the Electricity Generation Rehabilitation and Restructuring Project, which sought to improve Turkey’s energy security. The analysis shows that Robust Decision Making can help decision makers answer specific and useful questions: _How do options perform across a wide range of potential future conditions? Under what specific conditions does the leading option fail to meet decision makers’ goals?_ Are those conditions sufficiently likely that decision makers should choose a different option? Such knowledge informs rather than replaces decision makers’ deliberations. It can help them systematically, rigorously, and transparently compare their options and select one that is robust. Moreover, the paper demonstrates that analysts can use the same data and models for Robust Decision Making as are typically used in economic analyses. Finally, the paper discusses the challenges in applying such methods and how they can be overcome.

This paper is a product of the Office of the Chief Economist, Sustainable Development Network.. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at lbonzanigo@worldbank.org .

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Making Informed Investment Decisions in an Uncertain World: A Short Demonstration**

Laura Bonzanigo<sup>1</sup> and Nidhi Kalra<sup>2</sup>

**JEL** : C44, D81, H54, O21, O22

**Keywords** : decision making under uncertainty, long-lived infrastructure investments, economic analysis, robust decision making

> 1 Sustainable Development Network, Office of the Chief Economist, The World Bank; Department of Economics, Ca’ Foscari University, Venice

> 2 Sustainable Development Network, Office of the Chief Economist, The World Bank
# **1 Introduction**

Each year governments invest billions of dollars towards development. Many of these investments – in energy, land use, transportation, and other sectors – are long-term and will shape the course of development in their countries. Yet we live in an unpredictable world governed by competing beliefs and preferences. Our decisions are engulfed in _deep uncertainty_ about the long-term cost of energy inputs, the impact of climate change, and a host of other factors that shape our decisions. Deep uncertainty occurs when the parties to a decision do not know—or do not agree on— the likelihood of future events, the best model for relating actions to outcomes, or the value of potential outcomes (Lempert et al., 2003).

Deep uncertainties pose formidable challenges to making near-term decisions that make long-term sense. Governments grapple with deep uncertainties daily. Traditionally, they have asked, _“Which investment option best meets our goals given our beliefs about the future?”_ Such approaches, sometimes called “Predict-then-Act”, hinge on our accurately predicting and then reaching consensus on what the future will bring (Hallegatte et al., 2012; Lempert and Kalra, 2011). But disagreements about the future can lead to gridlock. Worse, investments tailored to one set of assumptions about a deeply uncertain future often prove inadequate or even harmful if another future comes to pass. Governments and international development organizations increasingly recognize that current approaches to decision making struggle to meet these ubiquitous challenges (Rennkamp, 2012; Ranger, 2013; IEG, 2010, 2012; The World Bank, 2013).

For instance, throughout the World Bank, there is a growing recognition that deep uncertainties need to be better managed in decision making (Hallegatte et al., 2012; IEG, 2010; Independent Evaluation Group (IEG), 2012). In 2010, The World Bank’s Independent Evaluation Group released a report pressing World Bank staff to conduct more rigorous economic analyses of investment options (IEG, 2010) and to use these analyses to inform decisions. This was endorsed by the World Bank’s Board of Directors in a Guidance Note to project managers (The World Bank, 2013). More recently, a more explicit consideration of risk and uncertainty is also one of the key messages of the World Development Report 2014 (The World Bank, 2014).

Many methods have been developed over the last half-century to help decision makers manage deep uncertainties and make investments that are robust to the unpredictable future. These approaches ask, _“Which investment option best meets our goals given that we cannot know what the future will bring?”_ (Hallegatte et al., 2012). These methods seek to identify _robust_ decisions – those that satisfy decision makers’ objectives in many plausible futures, rather than being optimal in any single best estimate of the future (Lempert et al., 2013). They have been recommended for investment lending but are not widely used in practice, leaving many decisions vulnerable to surprise. One reason may be that we do not understand well the extent to which deep uncertainties affect lending decisions, or know the usefulness and practicality of these methods for investment lending.

We seek to help bridge this gap. In this study, first we review the economic analysis of ten World Bank projects approved between 2002 and 2011 in order to understand how they managed risk and uncertainty. Then, we test and demonstrate the practicality and valueadded of new methods for long-term infrastructural investment lending decisions. Research suggests that having practical tools to solve a problem can increase one’s awareness of the 2
problem and motivation to solve it (Coombes and Devine, 2010; Kolb, 1984). Our hope is that a straightforward demonstration of these methods may motivate and equip analysts to better manage uncertainty in lending decisions.

In particular, we apply Robust Decision Making (RDM, Lempert et al., 2003) to the economic analysis of a prior World Bank project, the Electricity Generation Rehabilitation and Restructuring Project, which in 2006 sought to improve Turkey’s energy security in part by increasing near-term energy supply. We use the same data and models utilized in the original analysis, but in a different way. Rather than seeking to inform electricity investments in Turkey with predictions of the future, we seek to inform them with assessments of their robustness to an unpredictable future.

The original decision was to rehabilitate an existing coal plant. Other options included building new coal-fired, gas-fired, or other power plants. Decision makers were concerned with two key metrics: a) whether the investment passed a cost minimization metric – i.e., produced electricity at lower cost than any other option, and b) whether the investment passed a cost-benefit test, in this case had a rate of return of at least 12%. We evaluated each option according to these metrics in 500 plausible future states of the world that varied under seven different sources of uncertainty.

We used the results to answer a series of specific and useful questions:

- How do decision makers’ options perform across a wide range of potential future conditions?

- Under what specific conditions does the leading option fail to meet decision makers’ goals?

- Are those conditions sufficiently likely that decision makers should choose a different option that is more robust?

Our analysis suggests that methods like RDM can provide decision makers with much more salient information about the merits and vulnerabilities of different options. This can focus decision makers’ attention on the uncertainties that matter most to a decision. It can make them aware of the important trade-offs and of the actions they could take to reduce their vulnerability. Ultimately, it puts the decision back in the hands of decision makers by helping them take measured risks and be less vulnerable to surprise.

Importantly, the purpose of this exercise is not to prove the original project decision right or wrong or to re-create the analysis under today’s conditions. At the time of this project, for instance, climate change mitigation and the possibility of a price on carbon were not widespread concerns, and this analysis reflects the original priorities of decision makers related to energy security. The same analysis today would almost certainly include these considerations. Rather, our aim is to demonstrate that an analysis of robustness can be incorporated into projects’ standard economic analyses using the same data and models, and to examine the different kinds of information that emerge from a Predict-then-Act versus a robustness analysis. Moreover, while our analysis focuses on a World Bank case study, we believe this methodology holds value for the broader international lending community.

We believe methods like RDM can be readily incorporated into cost-benefit and other economic analyses that analysts perform every day, with few additional resources. We hope 3
this analysis will be a useful template by which projects can better manage uncertainty in their investment decisions.

# **2 Review of World Bank Projects**

The typical approach to economic analysis is to conduct a cost-benefit analysis of the preferred investment option under best estimate predictions of the relevant future conditions. Consistent with the World Bank’s manual on economic analyses, analysts sometimes conduct a sensitivity analysis around the best estimate projection to address uncertainty (Belli, 2001). Occasionally they may replace a single projection with a probabilistic projection, assigning likelihoods to a variety of future outcomes.

These approaches have a common underpinning. Often termed Predict-then-Act (Dessai and Wilby, 2010), they seek to characterize the future (i.e. make predictions) and then measure policy options against the characterizations to determine the best near-term course of action. Predict-then-Act forms the basis of several analytic methods commonly used to evaluate investments (Lempert and Kalra, 2011; Hallegatte et al., 2012) – including traditional risk analysis, cost-benefit analysis (Arrow and Fisher, 1974), and real options analysis (Henry, 1974).

Yet the World Bank increasingly recognizes that such approaches may not help manage deep uncertainties that arise in many lending decisions (Hallegatte et al., 2012; IEG, 2010; Independent Evaluation Group (IEG), 2012). In this study, we sought to understand the World Bank’s economic analyses in greater detail by reviewing the Project Appraisal Documents (PADs)<sup>3</sup> of ten projects that were approved between 2002 and 2011. PADs are compulsory documents that projects must to submit to the World Bank’s Executive Board for lending approval. They contain basic information on the proposed projects, such as Country Ministries involved, project length and its objectives, a section on key risks and plausible mitigation measures, and the economic and financial analyses. The economic analysis should quantify the benefits and the costs of the proposed project and of the other options examined.

In our review of ten PADs we asked several questions:

- How many and which alternative options did the project originally evaluate?

- What uncertainties did analysts identify in the project narrative?

- Which of these were carried into the economic analysis of projects?

- How did the projects manage uncertainty?

While our small sample size lends no statistical weight to the answers, it enriches the observations made by the Independent Evaluation Group and others at the World Bank. We randomly selected investment projects in two sectors – energy and transportation – that were characterized by the World Bank’s project database as addressing climate change, a deep uncertainty.

> 3 PADs summarize the project team’s assessment of a lending project and are the basis for the World Bank's review and approval of lending projects. PADs include a project rationale, key components of the loan, potential areas of risk, and an assessment of the economic merits of the project.

4
We found that project documents often identify multiple investment options but present detailed evaluations only for the already-preferred investment. The alternative options are rarely analyzed quantitatively or compared meaningfully. It is possible that analysts conducted detailed analyses prior to the project approval stage to whittle the options. Our conversations with project leaders suggest that the preferred investment is often chosen through an informal process and dialogue, not through a quantified analysis of costs and benefits. Sometimes, the results of a Technical Assistance phase inform investment lending, but this does not substitute for the lack of a clear quantification of risks. This confirms the Independent Evaluation Group’s observation that economic analyses are in practice used to justify rather than inform investment decisions, because they are done once the project is already well advanced (IEG, 2010).

Projects often describe many deep uncertainties in the project narrative and rationale. For instance, of the ten projects in our sample, eight note that population growth and nine note that future socio-economic developments may shape the success of the proposed project. Seven cite the uncertainty of energy supply and demand, four cite the uncertainty of urban development, and three cite the worsening of environmental conditions as relevant to future project outcomes.

However, in most projects (six out of ten) these external drivers are not included in the economic analysis. Only two projects include considerations of future water availability in their cost-benefit analysis. Another two projects include socio-economic developments in their estimation of energy and transport investments, respectively. However, even when economic analyses include these external conditions, they are not treated as deep uncertainties but rather as well understood parameters, using single best estimate projections.

There is also a disconnect between the risk section and the economic analysis in the PAD: most of the potential risks mentioned in the risk section are not examined in the economic analysis for the evaluation of the available options. Analysts address risks qualitatively, often by proposing monitoring and capacity building. In the sample projects analyzed, none of the risks mentioned in the risk analysis are carried through to the economic analysis.

To address uncertainty, projects assess the sensitivity of only the preferred option and with respect to only a few of the many relevant uncertainties. Nine out of ten projects assess the sensitivity of the chosen option to investment costs, yet only one tests its sensitivity to the discount rate. Moreover, projects varied the uncertainties by a small amount around their best estimate projection – often only 20%. This provides a very limited understanding of the vulnerabilities of the project.

Moreover, many deep uncertainties that could clearly affect the success of a project are never mentioned. For instance, the timely completion of a project may significantly affect the project outcomes. All of the projects we reviewed mention this as a key risk but only two included project delays in the economic analysis even though delays would affect the projects’ rates of return. Similarly, the success of many investment projects depends on effective future operations and maintenance. But, particularly in developing countries, operations and maintenance of large infrastructure often falls short of needs (Foster et al., 2010; Ostrom et al., 1993). Yet many projects assume operations and maintenance will continue as planned – or that inefficiencies will be manageable - and the economic analysis 5
does not consider how the investment would perform with degraded operations and maintenance.

Despite these shortcomings, two of ten projects do recognize that deep uncertainties affect their choices. Instead of Predict-Then-Act approaches, they seek to manage uncertainty with a handful of diverse scenarios that challenge prediction-based thinking. Scenarios can usefully encourage decision makers to assess their investment options under unexpected conditions.

However, scenario planning also struggles to manage deep uncertainties. Analysts usually only consider a small number of scenarios. Analysts also typically handcraft the scenarios, so they are subject to the same biases as analysts’ predictions about an expected future. Making decisions based on diverse scenarios may also lead us again to a problem of consensus around predictions – to which scenario should we tailor our investment decision?

Collectively, these shortcomings suggest that there is room for improvement to meet the Independent Evaluation Group’s recommendations of rigorous economic analysis.

# **3 Robust Decision Making**

Robust Decision Making is a decision support methodology designed to help manage deep uncertainty by helping develop plans that are robust and flexible (Lempert et al., 2003). It has been applied to water resource management, (Groves et al., 2008) flood risk management (Fischbach, 2010), terrorism risk insurance (Willis et al., 2005), and energy investments (Popper et al., 2009), primarily in the U.S. Lempert et al. (2013) provide a concise summary of RDM and several case studies.

RDM is one of several methods like InfoGap (Ben-Haim, 2006) and Climate-Informed Decision Analysis (Brown, 2010), that seek to better understand how decision makers’ options perform under a wide range of conditions, rather than under a single or handful of predicted conditions. The World Bank recommends RDM and similar approaches for managing uncertainty and making better-informed decisions (The World Bank, 2014). In one of the first applications in developing countries, a recent World Bank study demonstrates how RDM could help make robust flood risk management plans in Ho Chi Minh City (Lempert et al., 2013).

RDM involves four basic steps that are embedded in a process of stakeholder engagement:

1. Decision makers structure the decision problem to identify their potential options, the metrics and performance thresholds they will use to evaluate whether their options meet or fail to meet their goals, and the uncertainties that could affect the performance of the policy option according to the metrics. Analysts use models to relate these factors.

2. Analysts statistically generate hundreds of futures and evaluate the performance of their options in each of those futures. This generates a large table of inputs (the uncertain future conditions) and outputs (the metrics) for each option.

3. Analysts identify which input conditions best explain when each option meets or fails to meet decision makers’ performance thresholds. These conditions describe scenarios to which each option is vulnerable.

6
4. Analysts and decision makers together compare the scenarios with available evidence to determine if they are sufficiently plausible to hedge against. They compare trade-offs between robustness, feasibility, cost, and other factors and select those options that best balance their needs.

Analysts and decision makers iterate upon earlier steps to examine more options or modify features of options, explore a wider range of uncertainties, and consider additional metrics. As Lempert et al. (2013) describe, this approach can be used to time investments and develop flexible plans – ones designed to evolve as new information becomes available. RDM can be used to design individual projects, portfolios of projects, or to compare different exclusive alternatives, as we do in this study.

# **4 Making Robust Energy Investments in Turkey**

In 2006, the Electricity Generation Rehabilitation and Restructuring Project sought to improve Turkey’s energy security in the near and mid-term. Turkey’s electricity demand had grown rapidly between 2002 and 2005 – 6% on average -- and it was expected to continue to grow, potentially outstripping its supply (The World Bank, 2006). The country had issued licenses for about 6,000MW of new capacity, yet in 2006 little new construction had begun. The Electricity Generation Rehabilitation and Restructuring Project focused on solving the near term solutions as well as more systemic solutions for the medium term. In the near term, it sought to increase electricity production through supply investments; for the medium term, it sought to provide support for restructuring the sector.

The project examined eight different supply investment options for the Afsin Elbistan electricity production area: seven types of power plants and the import of energy from Bulgaria. They evaluated these projects based on two economic metrics. First, it should pass a cost-minimization criterion, producing electricity at lower cost than any of the other options. Second, the investment should pass a cost-benefit test; in this case, it should have at least a 12% internal rate of return (IRR), a benchmark commonly used at the World Bank to demonstrate economic value of investments.

At the time, decision makers originally preferred rehabilitating the existing local lignite-fired plant at Afsin Elbistan because they believed it would produce electricity the soonest. The project’s economic analysis further suggested that it would meet both their cost-benefit and cost-minimization targets. The project was approved in 2006. However, it was never implemented due to legal dispute with a private company, which had signed a concession agreement with the government in 1999 for the operation of the plant and the eventual construction of a new energy production plant.<sup>4</sup> We use this project to demonstrate RDM for several reasons. The energy sector is fraught with deep uncertainties that challenge decision making. It is also one of the most important sectors for long-term economic growth, and simultaneously it can shape the development

> 4 The litigation was on-going at the time of project design, but both parties seemed willing to find a solution. Hence, the risk of delays (but not of failures) was considered as moderate in the Project Appraisal Document (The World Bank, 2006, p.11), though such delays were not included in the economic analysis.

7
path of a nation. Additionally, this particular project has detailed data and a model that enabled us to repeat the earlier economic analysis using RDM.

# **_4.1 Summary of the Original Economic Analysis_**

The project’s original economic model evaluated seven power plants’ options for the Afsin Elbistan area for increasing Turkey’s electricity supply, but five were addressed in detail<sup>5</sup> :

1. Rehabilitating the existing local lignite-fired power plant;

2. Building a new local lignite-fired power plant; 3. Building a new imported coal-fired power plant; 4. Building a new gas-fired power plant; and 5. Building a new lignite fluidized bed power plant, which would be fired by a higher quality coal than local lignite.

We replicated the analysis for these five investment options. Table 1 summarizes the uncertainties examined in the project’s original analysis, and in our application of RDM to the project. These relate to future energy conditions, future power plant investment characteristics, and economic parameters.

The original analysis evaluated each option under a best estimate of several deep uncertainties, listed in column A of Table 1. Analysts’ projected or assumed values are listed in column B and were based on the data available at the time. For instance, the assumed price of local lignite (6.05 US$/ton) is the levelized economic cost over a 20 year period from the expanded and upgraded local lignite supply mine (The World Bank, 2006).

The original analysis prioritized options according to their cost-minimization, specifically the discounted cost per unit of production. It showed that rehabilitating the existing plant would be the most cost effective at 3.88 cents US$/kWh, followed by constructing a new gas-fired combined cycle plant at 4.41 cents US$/kWh.

The project also sought to ensure that the selected investment would pass a cost-benefit test, specifically having an IRR of at least 12%. It showed that rehabilitating the existing plant would have an IRR of 25% under the projection in column B, comfortably passing the 12% target. To address uncertainty, the project conducted a sensitivity analysis of the IRR to some of the projected values (column C). These values reflect 10%, 20%, or 50% deviations from the projected value. The IRR was at least 14% across all the sensitivity tests.<sup>6</sup> This analysis suggested that the rehabilitation project passes decision makers’ economic tests of being the most cost effective option while also having high returns in the analysts’ best

> 5 A wind and a nuclear plant were also considered. The wind plant was eliminated as a candidate because its cost effectiveness, which was derived from the literature, was not competitive with other options. The nuclear plant was eliminated because it would take much longer to construct.

> 6 These results are reported directly from the project appraisal document. We reconstructed the economic model using the same formulae and data as the original model, but with slightly different time series of the price of inputs, which were difficult to replicate from the original analysis. Our model results showed the same relative performance of options. However, running the original sensitivity analysis for rehabilitating the plant in our model revealed that the IRR fell below12% when electricity prices were 4 cents US$/kWh, or when the cost of lignite is 9 US$/ton. This does not suggest that the original sensitivity analysis is incorrect, but rather that the results can be highly sensitive to slight variations in model parameters. A small sensitivity analysis may not reveal the full behaviour of the interventions under analysis.

8
estimate of future conditions. The project’s economic analysis concludes that “[...] the rehabilitated plant will be the least cost option for Turkey under any scenario [...]” (The World Bank, 2006:58).

**Table 1. Uncertainties evaluated in the original analysis and in this study’s RDM analysis.**

||**Original**|**Analysis**|**RDM**|**Analysis**|
|---|---|---|---|---|
|**A. Deep Uncertainties**|**B. Projected**<br>**Value**|**C. Sensitivity**<br>**Test Values**<sup>**b**</sup>|**D. LH**<br>min|**S Range**<br>max|
|Wholesaleprice of electricity (US$/kWh)<sup>a</sup>|0.05|0.04,0.045|0.04|0.10|
|Discount Rate<sup>a</sup>|0.1|—|0.01|0.20|
|Estimated Life of the Plant(years)|||||
|Rehabilitation; New local lignite plant;<br>New gas-fired plant;|20|(10, 16)|5|25|
|New imported coal plant; New lignite<br>fluidized bed plant|40|—|5|45|
|Capacity Utilization|||||
|Rehabilitation; New gas-fired plant; New<br>lignite fluidized bed plants|0.76|(0.6, 0.68)|0.5|0.9|
|New local lignite plant; New imported<br>coalplant<br>|0.90|—|0.6|0.95|
|Capital Costs(US$,in millions) <sup>7</sup>|||||
|Rehabilitation|683|(819,1024.5)|600|1030|
|New local ligniteplant|1998|—|1800|3000|
|New imported coalplant|586.7|—|500|880|
|Newgas-firedplant|420|—|350|630|
|New lignite fluidized bedplant|451.84|—|400|670|
|Length of Construction Time (years)|||||
|Rehabilitation|4|—|2|6|
|New local lignite plant|6|—|3|9|
|New imported coal plant; New gas-fired<br>plant;New lignite fluidized bedplant|3|—|1.5|4.5|
|Cost of energy inputs|||||
|Local lignite (US$/ton, for the<br>rehabilitated and the new plant|6.05|(7.06, 9.08)|3|12|
|Imported coal (US$/ton)|60|—|30|120|
|Gas (US$/tcm, for the new gas-fired<br>plant)<br>|220|—|110|440|
|Improved coal<sup>8</sup>for the new lignite<br>fluidized bed plant (US$/ton)|21|—|10|42|

> _~~a~~ Only the wholesale price of electricity and the discount rate have the same ranges across all options’ cost effectiveness models, both in the original and in the RDM analyses. b The original analysis conducted sensitivity tests only for the IRR of rehabilitating the existing plant._

> 7 Project capital costs included two key technological improvements. Given the poor quality of local lignite, both the rehabilitated and the new local lignite-fired power plants would have flue-gas desulfurization (FGD) to remove sulfur dioxide from exhaust flue gases. During project preparation, dust reduction technology was also made compulsory in all options with low quality coal.

8 The three types of coal – local lignite, coal for the fluidized bed plant, and the imported coal -- have a different energy values. Local lignite has the least and imported coal has the most potential energy.

9
# **_4.2 Demonstration of Robust Decision Making_**

Robust Decision Making can help us more fully understand the strengths and weaknesses of our options and make a sound decision, without relying on accurate predictions of the unpredictable future. It helps us answer several useful questions:

1. How do our options perform across a wide range of potential future conditions?

2. Under what specific conditions does the _rehabilitating the existing plant_ fail to meet our goals?

3. Are those conditions sufficiently likely that we should choose a different _energy investment_ option?

In practice, these questions are addressed in a close collaboration between decision makers, stakeholders, and analysts. A series of structured workshops help participants frame the analysis, deliberate over interim findings and direct subsequent analyses, and reach decisions. Because we are re-creating an earlier project analysis, we focus in this section on the analytical steps and draw on the outcomes of stakeholder engagement in the original project.

## **Structuring the Problem**

To conduct the RDM analysis, we first structure our analysis to more fully identify the uncertainties we face. In this study, we used the same uncertainties as in the original analysis, but in nearly all cases, we have expanded the range of possible values as shown in column D of Table 1. Figure 1 shows this visually for the option of rehabilitating the existing plant. The values’ ranges were drawn from the literature and historical data, and from consultations with energy experts. For example, literature on investment planning biases suggests that on average, capital costs are underestimated by 28% (Siemiatycki, 2010). World Bank energy experts confirmed that for a comprehensive thermal rehabilitation project, cost overruns of 50% are possible. Hence, we considered a range of capital costs from a lower bound of just below each option’s stated cost, to an upper bound of 150% of the stated cost.

We also added two new uncertainties: the length of construction time and discount rate. The length of construction time is an uncertain and important parameter. Most projects experience delays in implementation (Ahsan and Gunawan, 2010). Indeed, the original analysis identified possible delays in rehabilitation and construction as plausible risks. Moreover, decision makers were concerned with how quickly each option could be implemented and would begin producing electricity.

<mark>The discount rate is also deeply uncertain. It is a political choice and often highlycontested (Arrow et al., 2013). It shapes how we allocate resources between the present and the future (Gollier, 2011). A higher discount rate signifies an urgency to satisfy present needs, whereas a lower discount rate expresses concerns for the long-term impacts of an investment. Although the World Bank typically uses discount rates of 10% to 12%, no single discount rate is appropriate for all projects and it may be difficult for stakeholders to come to consensus (Hoekstra, 1985; Oxera, 2011). After consultation with World Bank experts, we use a range from 1 to 20% to explore both the longer-term considerations of contributing to the country's growth and the short-term objectives of avoiding an energy crisis.</mark> 10
<!-- Start of picture text --> Ekctricity Prices TODO SCOIIOBR Mo occ ccc cccccc<br>0.04 US$/kWh 0.10 US$/kWh<br>Discount Rate TIIDID ISIS I SLCC ISI III II oL WR CoCo l llc ccs llslslcca<br>1% 20%<br>Estimated Lie TID ISOS ICICI CISC IIS IS ICI SCE LISS Lio MEIC Iolo scl clsa<br>5 Years 25 Years<br>Capacity Utilization CHIC o coool Ics ccc lc cco llc lla cca<br>50% 90%<br>Capital Cost Clio loolololiec we Dee eee eee ee ee ee!<br>US$ 600M US$ 1030M<br>Length of CLIC loll llc ccc ce Eccl<br>Construction Time 2 Years 6 Years<br>Cost of Local Lignite CIID IDI ITI ICI I IIo MBC ICI IC OIL III IOLIIIlllsllia<br>3 US$/ton 12 US$/ton<br>__._._<br>Values explored O Values explored by e Values explored in<br>by RDM sensitivity tests projected scenario<br><!-- End of picture text -->
**Table 2. Sample from the database with the 500 futures**

|Future ID<br>Option|Wholesale Price of<br>Electricity|**I**<br>Capital Cost|**nputs**<br>Length of<br>Construction time|Other four<br>Uncertainties*|**Out**<br>Discounted Cost<br>per kWh|**puts**<br>IRR|
|---|---|---|---|---|---|---|
|1<br>Rehabilitation|5.15|824.89|2.85|[...]|6.39|0.05|
|2<br>Rehabilitation|8.79|945.29|4.14|[...]|7.11|0.35|
|1<br>Gas-fired plant|5.15|526.68|4.01|[...]|4.21|0.27|
|2<br>Gas-firedplant|8.79|514.92|4.38|[...]|8.91|0.15|

* Other uncertainties are the discount rates, cost of inputs, lifetime of the plants, and capacity utilization rates.

# **How do our options perform across a wide range of potential future conditions?**

Figure 2 shows the performance of the investment options in 500 plausible futures. Rehabilitating the existing local-lignite plant meets the cost-minimization and cost-benefit targets in most futures. However, there are many in which constructing a new gas-fired plant would be more cost effective. There are also some futures in which the rehabilitation option has an IRR of less than12%. The other three options meet performance goals in far fewer futures. This indicates that rehabilitation meets our goals under a wider range of assumptions about the future than do other options.

Up to this point, our analysis resembles the steps in traditional Monte Carlo analyses used for project evaluation:<sup>11</sup> both approaches run many simulations over randomly generated cases. However, the approaches diverge significantly in how they use such simulations. A Monte Carlo analysis uses the sampled cases to represent the likelihood of future conditions (e.g. probabilities of future energy prices, construction time, etc.). It further uses the results of the simulations to make inferences about the likely performance of the project, e.g. that rehabilitation is more likely to meet decision makers’ goals than other interventions because it outperforms other interventions in most cases. This approach works well and the inferences are credible when we have reliable probability distributions.

However, in many investment decisions such as this one, we do not have defensible probability distributions of our deep uncertainties and therefore we cannot infer the relative success or failure of our options from these results. For example, although rehabilitation produces energy at lowest cost in 269 of 500 futures, this does _not_ mean that the probability of doing so is 54% (269/500). Similarly, although rehabilitation falls short of a 12% IRR in 99

> 11 Monte Carlo analysis describes a useful approach to numerically generating a probability distribution of an outcome by repeatedly and randomly sampling input parameters from probability distributions of their values (Belli, 2001). For project evaluation, analysts first assign probability distributions to inputs such as future energy prices, length of project implementation, climate change impacts. The resulting distribution of outcomes (e.g., IRR) is interpreted as a statement of the project’s likelihood of success or failure.

12
<!-- Start of picture text --> Most Cost-Effective Passes Cost-Benefit Test<br>450<br>400<br>350<br>300<br>o<br>2<br>ze 250<br>=]<br>wu<br>‘o 200<br>‘=<br>2<br>= 150<br>2 100<br>50<br>0 = — =<br>S c ro r= < 5 r= = S <<br>2 & & G & = & & z &<br>= . al a a & a a ao =<br>= a 3 = 2 © ®<br>s 2 3 ¢& 8 5 2 Bg ¢@ &<br>« 8 6 o ke « 8 S fo) 3S<br>a a Es I S = F3 =<br>= £ 2 t = ~ z S<br>® = = ® > =<br>Zz o () z o o<br>Zz 2 z z<br><!-- End of picture text -->
Scenario discovery reveals that a gas-fired plant is more cost effective than rehabilitating the existing plant if:

1. The cost of local lignite (US$/ton) is more than 4.5% of the cost of gas (US$/tcm).<sup>13</sup>

Scenario discovery also reveals that rehabilitating the existing plant fails our cost-benefit test, i.e. has an IRR below 12%, if two conditions hold simultaneously:

1. The wholesale price of electricity is below 0.059 US$/kWh

2. Local lignite costs more than 6.3 US$/ton<sup>14</sup> The discount rate, estimated life of the plant, capacity utilization, length of construction time, and capital costs – though highly uncertain and potentially a source of disagreement among stakeholders -- are less important in determining whether or not investing in rehabilitating the plant is economically sound. We could invest significant time discussing what these values should be, when indeed they are not key drivers of the economic performance of our investment.

# **Are those conditions sufficiently likely that we should choose a different energy investment option?**

So far we have sought to better understand the merits and drawbacks of our options under a wide range of conditions, and to identify the conditions that may lead them to fall short of our goals. We have said little about what the future may actually hold. We now turn to the evidence to assess whether those conditions are sufficiently likely that we should choose a different energy investment option. We begin by examining the vulnerabilities to costminimization and then to cost-benefit goals.

Recall that rehabilitating the existing power plant is less cost effective than building a new gas-fired plant if the cost of local lignite (US$/ton) is more than 4.5% of the cost of gas (US$/tcm). The original cost-benefit analysis assumed that with an upgrade, the mine would sell local lignite at an average of 6.05 US$/ton, 2.8% of the 2006 gas price of 220 US$/tcm and well below the 4.5% threshold.<sup>15</sup> However current and historical trends do not guarantee or bound future outcomes, and we should not use them to predict what will be. Instead, we should use them to answer a more useful question, _“Is there evidence that the conditions that threaten the success of our_

> 13 This condition is a statistically strong predictor of when building a new gas-fired plant is more costeffective than rehabilitating the existing plant. Of the 97 futures with this condition, building a new gas-fired plant is more cost effective in 92 of them (95%). However, it is not a complete predictor: this condition exists in 92 of the 153 futures (60%) in which the new gas-fired plant is more cost effective. Further scenario discovery analysis, beyond the scope of this demonstration, would reveal additional sets of conditions that explain the remaining 40%, which policy makers could weigh against additional evidence. Nevertheless, this single condition offers useful information for a policy dialogue on the potential vulnerabilities of the rehabilitation option.

> 14 This condition is also a statistically strong predictor of when rehabilitating the existing plant fails the cost-benefit test. Of the 92 futures with these conditions, rehabilitation fails the cost-benefit test in 73 (79%). Of the 99 cases in which rehabilitation fails the cost-benefit test, this condition occurs in 73 (74%).

> 15 In the time series utilized by the project, the ratio maintains an average of 3.1% and never exceeds the 3.5% thresholds (The World Bank, 2006).

14
# _investment could occur?”_

The answer is yes. The cost of local lignite depends substantially on the efficiency and output of the mine. Let us turn briefly to the main drivers of local lignite costs. The cost of local lignite is at least partially under decision makers’ control. It depends on the efficiency of the mine that supplies lignite to the plant, i.e. how many millions of tons of lignite are extracted annually. Most of the mine’s costs, including labor, electricity, and materials, are fixed and when production is down, prices are high. At low production in 2004, just before the project’s preparation phase, the mine produced only 6.7M tons, about 30% of the mine’s annual capacity of 18.6M tons, which were sold at 11.46 US$/ton (The World Bank, 2006). As part of this project, however, this mine would have been expanded and upgraded to ensure a cheaper and more stable lignite supply. Hence, policy makers have options to reduce the cost of lignite, for instance by improving the mine’s maintenance and operation.

From 2002 until 2008, gas prices increased sharply in Turkey, reaching over 300 US$/tcm (Ozen, 2012) but have since declined and stabilized at around 220 US$/tmc (DG Energy, 2012; Lomsadze, 2013). Hence, the highest ratio based on recent historical trends, when lignite cost is high (11.46 US$/ton) and gas price is low (220 US$/tmc), would be 5.2%. This exceeds the threshold of 4.5%. Indeed, if current trends continue to hold and the price of gas remains low, the cost of local lignite would have to be below 9.9 US$/ton (i.e. produce at least at about half of its capacity) to be under the 4.5% threshold (9.90/220).

It thus appears that rehabilitating the existing plant may not be the most cost-effective option under all plausible future conditions, though policy makers have options to influence those conditions.

We can more fully understand the relative merits of the two investments by also comparing their cost-benefit performance. Recall that rehabilitating the existing plant fails our costbenefit test in 99 of 500 futures, i.e. has an IRR below 12%, when two conditions hold:

1. The wholesale price of electricity is below 0.059 US$/kWh

2. Local lignite costs more than 6.3 US$/ton<sup>16</sup> Building a new gas-fired plant fails our cost-benefit test in 128 of our 500 scenarios, which occurs when:

1. The wholesale price of electricity is below 0.073 US$/kWh

2. Gas costs more than 220 US$/tcm<sup>17</sup> Rehabilitating the existing plant would be more robust to decreases in the price of electricity than would the gas-fired plant. In 2009, the Government of Turkey guaranteed a minimum price of 0.068 US$/kWh for the following 10 years (U.S. Commercial Service Turkey, 2009). This price is above the vulnerable threshold for rehabilitating the existing plant, but below

> 16 This condition is also a statistically strong predictor of when rehabilitating the existing plant fails the cost-benefit test. Of the 92 futures with these conditions, rehabilitation fails the cost-benefit test in 73 (79%). Of the 99 cases in which rehabilitation fails the cost-benefit test, this condition occurs in 73 (74%).

> 17 This condition is also a statistically strong predictor of when building a new gas-fired power plant fails the cost-benefit test. Of the 125 futures with these conditions, rehabilitation fails the cost-benefit test in 100 (80%). Of the 128 cases in which rehabilitation fails the cost-benefit test, this condition occurs in 100 (78%).

15
the threshold for the gas-fired plant. Nevertheless, in 2009, Turkish electricity’s wholesale prices were approximately 0.10-0.11 US$/kWh and were not likely to decrease for years to come until excess capacity was in the marketplace.

As noted above, policy makers have options to reduce the cost of lignite. In particular, operating the mine at about 13 million tons per year -- roughly two-thirds capacity -- would keep the lignite cost below 6 US$/ton (The World Bank, 2006). Simultaneously, the cost of gas has remained above 220 US$/tcm since 2006. In combination, the conditions that would lead the gas-fired plant to fail the cost effectiveness test seem more plausible than the conditions that would lead rehabilitation to fail the cost-benefit test.

In sum, this analysis leads to several important insights about the merits of our options:

1. Rehabilitating the existing lignite plant fails to meet decision makers’ costminimization goals under plausible future conditions. If gas costs remain low while local lignite costs increase, a new gas-fired plant would be more cost effective.

2. Yet, the cost of local lignite is at least partially under decision makers’ control, suggesting that decision makers can take action to avoid those threatening conditions.

3. Moreover, building a new gas-fired plant fails to pass the cost-benefit test under conditions that are less constraining and more plausible than the conditions in which the rehabilitation fails a cost-benefit test.

These observations could reasonably lead decision makers to conclude that rehabilitating the existing plant is an economically sound choice: it performs well under a wide range of potential future conditions and is less vulnerable than other options. However, other factors may also influence their decision. The relative importance of cost-benefit versus costeffectiveness metrics, decision makers’ risk aversion, and other concerns may shape their choice. Nevertheless, a more complete and more transparent understanding of the merits of decision makers’ options helps inform these deliberations and keeps the decision in the hands of decision makers.

# **5 Key Findings and Recommendations**

Faced with pervasive deep uncertainties about the future, decision makers struggle to choose near-term decisions that make long-term sense. In this paper, we sought to answer three key questions:

1. How are deep uncertainties currently managed in World Bank projects? 2. Can new approaches help projects better manage those uncertainties?

3. Are they practical and applicable, and what challenges do they pose?

To help answer these questions, we reviewed prior project analyses and then applied RDM to the economic analysis of a long-term energy investment in Turkey. Our goal is not to reevaluate the earlier decision, which today would be very different and for example, include climate change concerns. Rather, we show the type of questions decision makers and analysts should ask about the robustness of their investments, and how they can 16
answer them. Through this demonstration, we also hope to motivate and equip analysts to better manage uncertainty in investment lending decisions.

# **Current World Bank Project Analyses Struggle to Manage Deep Uncertainty**

Our review of prior project analyses suggests that analysts typically use traditional Predictthen-Act methods to assess project performance, and do not explore the full range of deep uncertainty. In particular:

1. Project documents contain efforts to manage uncertainties.

2. However, even though project narratives may identify several deep uncertainties, the economic analyses address only a few.

3. There is a disconnect between risk and economic analysis. For instance, analyses typically assume that project costs and implementation will occur as planned, even though implementation issues are often cited as potential risks to the project.

4. Projects often carry out sensitivity analyses only for the already-preferred investment option or for the option shown to perform best under a single prediction of future conditions.

5. Most project managers we contacted during this study are aware of the potential for improving economic analysis.

In sum, project analyses can be improved to help make projects more robust.

# **Methods Like RDM Can Help Make Better-Informed Investment Decisions**

The findings from the Turkey project suggest that methods like RDM can improve our understanding of the vulnerabilities and strengths of our investment options, despite deep uncertainty and a potentially surprising future. In particular, this analysis demonstrates how RDM can help analysts and decision makers

- Identify and make explicit the many deep uncertainties that may affect the performance of investment options;

- Analyze the performance of investment options across a full range of plausible futures, without needing to assign controversial likelihoods to those futures;

- Identify and focus attention on the specific combinations of conditions that determine whether an investment meets or fails to meet decision makers’ goals; and

- Use historical and scientific evidence credibly, to assess whether threatening conditions are plausible, rather than to make predictions about what will be.

Such knowledge informs rather than replaces decision makers’ deliberations. It helps them systematically, rigorously, and transparently compare their options and select one that is robust – meeting their needs in the widest range of possible futures. Decision makers can have confidence in a robust decision, even if they cannot have confidence about the future.

17
# **These Methods Can Be Readily Applied, Though They Pose Some Challenges**

In this study, we used the same economic models and data that analysts used in the original analysis; we just used them differently. Rather than using the data and models to assess performance in a single best estimate of the future, we used them to stress test our options in hundreds of possible futures. This approach was more forgiving of data gaps: where a quantity was unknown, we could use a wide range of plausible values, rather than tenuously choosing a single value. This also enabled us to include uncertainties that may not have been feasible in the original analysis.

Nevertheless, methods like RDM may have a steep learning curve – from understanding how to structure a robustness analysis, to learning software that aids in scenario discovery, to interpreting the results of scenario discovery, to communicating the findings to stakeholders. However, prior applications of RDM with other organizations and agencies suggest that these challenges can be readily overcome with time and training.

# **Managing Uncertainty Requires Rethinking “Good” Decision Making**

Analysts and decision makers routinely face pressures to demonstrate that a decision is risk-free. Political and cultural expediency press them to ignore rather than acknowledge uncertainty and present their decision as advantageous and certain. Such thinking keeps us in the dark about the real threats to our decision, and may lead us to brittle decisions that fail when the future surprises us.

To manage uncertainty, we may need to revisit our beliefs about what makes a decision “good.” Instead of ignoring uncertainty, we should seek more fully to understand the threats it may pose to our choices. This will enable us to make decisions that are robust to an unpredictable future.

Such a change requires a cultural shift as much as it requires an analytical shift. Yet methodological innovations like RDM can help. By motivating and equipping analysts to manage uncertainty, they can shape how we think, discuss, and ultimately make decisions.

18