metadata: authors: 0: author: Wilkinson,Thomas David
1: author: Wang,Mengxiao
2: author: Friedman,Jed
3: author: Prestidge,Marelize
count: World
docna: 0: docna: A Framework for the Economic Evaluation of
            Digital Health Interventions
docty: Policy Research Working Paper
abstracts: cdata!: Numerous digital health interventions
            have been piloted in response to the health care challenges
            low- and middle-income countries face. Because the
            opportunity cost of investing in digital health
            interventions can be large, countries must make choices
            about which interventions to scale up. To make good
            investment decisions about digital health interventions,
            there is a need to define and establish their value, and to
            use economic evaluation to make informed decisions, however
            DHIs present methodological challenges for economic
            evaluation. To address these challenges, this paper first
            creates a &#x2018;gap map&#x2019; of digital health
            intervention evidence which reveals a dearth of economic
            evaluation evidence about digital health interventions; this
            lack can limit decisions on policy, programming, and
            appropriate scale-up of digital health interventions. To
            advance work in this field, this paper then develops an
            economic evaluation framework  that can be used when
            determining the economic value of digital health
            interventions. Such a standardized approach, alongside
            guidance to assist the conduct and use of economic evidence,
            can improve decision making and investments in DHI under
            constrained health budgets. The resulting digital health
            intervention economic evaluation framework consists of 5
            steps: (1) determine the context, (2) determine the
            intervention type, (3) establish the level of complexity,
            (4) set the analytic principles, and (5) represent the value
            proposition. Users of the framework should attempt to adhere
            to its steps and principles, but where this is not feasible
            or appropriate, they should provide justification for the
            methodological choice. The framework should facilitate
            methodological transparency, thereby improving the overall
            usefulness of economic evaluations of digital health interventions.
display_title: A Framework for the Economic
            Evaluation of Digital Health Interventions
listing_relative_url: /research/2023/04/34040153/framework-economic-evaluation-digital-health-interventions
topicv3: Digital Government m1325964 678,Health Technologies m1327278 1331,Digitization m1505678 3078,Health Systems m1583385 4581
publishtoextweb_dt: 2023-04-12T00:00:00Z
dois: 10.1596/1813-9450-10407
origu: Off of Sr VP Dev Econ/Chief Econ (DECVP)
author_keys: 0: author_key: 1646931
1: author_key: 1654076
2: author_key: 669320
3: author_key: 664365
model_extractions: Policy Research Working Paper 10407 A Framework for the Economic Evaluation of Digital Health Interventions _Thomas Wilkinson Mengxiao Wang Jed Friedman Marelize Görgens_ Health, Nutrition and Population Global Practice & Development Research Group April 2023
Policy Research Working Paper 10407

# **Abstract**

Numerous digital health interventions have been piloted in response to the health care challenges low- and middle-income countries face. Because the opportunity cost of investing in digital health interventions can be large, countries must make choices about which interventions to scale up. To make good investment decisions about digital health interventions, there is a need to define and establish their value, and to use economic evaluation to make informed decisions, however DHIs present methodological challenges for economic evaluation. To address these challenges, this paper first creates a ‘gap map’ of digital health intervention evidence which reveals a dearth of economic evaluation evidence about digital health interventions; this lack can limit decisions on policy, programming, and appropriate scale-up of digital health interventions. To advance work in this field, this paper then develops an economic evaluation framework that can be used when determining the economic value of digital health interventions. Such a standardized approach, alongside guidance to assist the conduct and use of economic evidence, can improve decision making and investments in DHI under constrained health budgets. The resulting digital health intervention economic evaluation framework consists of 5 steps: (1) determine the context, (2) determine the intervention type, (3) establish the level of complexity, (4) set the analytic principles, and (5) represent the value proposition. Users of the framework should attempt to adhere to its steps and principles, but where this is not feasible or appropriate, they should provide justification for the methodological choice. The framework should facilitate methodological transparency, thereby improving the overall usefulness of economic evaluations of digital health interventions.

This paper is a product of the Health, Nutrition and Population Global Practice and the Development Research Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at mwang8@worldbank.org, jfriedman@ worldbank.org and mgorgens@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# A Framework for the Economic Evaluation of Digital Health Interventions

_Thomas Wilkinson Mengxiao Wang Jed Friedman Marelize Görgens_ Keywords: digital health interventions; economic evaluation; low- and middle- income countries.

JEL classifications: I10, I15, O33 This paper is a product of the Digital Health Flagship Program at the Health, Nutrition and Population Global Practice and the Development Research Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at mwang8@worldbank.org, jfriedman@worldbank.org and mgorgens@worldbank.org.

The writing team thanks the insightful contributions of those who attended the Technical Working Group meetings in July 2019 and October 2020 (in person and virtually) and/or who participated in subsequent consultation and discussion on development of the framework. We also acknowledge the strategic advice of David Wilson, Program Director, Health Nutrition and Population Global Practice, World Bank.
# 1. Introduction

## 1.1 Background of the Digital Health Intervention Economic Evaluation Framework

The number of digital health interventions (DHIs) is growing rapidly, with the promise of enormous benefits to health systems and the people who use them. While DHIs have the potential to fundamentally alter the practice of health care and wellness, the opportunity cost of investments is also large and often sunk. This is particularly relevant for developing country contexts where the potential gains from harnessing digital technology can be transformative, bypassing existing infrastructure and process bottlenecks to address substantial unmet health needs. However, the opportunity costs of investing in DHIs that are simply ineffective or impractical, or whose costs outweigh the benefits that could have been achieved had the money been invested elsewhere, is particularly acute in resource-constrained contexts.

One of the most effective tools to generate information to inform health policy decision making is economic evaluation – the comparative analysis of one or more interventions in terms of costs and consequences [1]. Economic evaluation can be used to reflect the local value of DHI investments, facilitating adoption and efficient use of limited resources, realizing the promise of DHIs to rapidly improve global health. However, economic evaluation applied inconsistently or only partially can hinder the decision-making process and even lead to suboptimal decisions. An economic evaluation framework will provide guidance and structure to facilitate the practice of economic evaluation to optimize its use in good decision making.

## 1.2 Objective of the DHI Economic Evaluation Framework

The framework aims to provide a consistent mechanism for representing the value of DHIs in context while adhering to the central concept that an economic evaluation is a comparative analysis of alternative courses of action in terms of both their costs and consequences.

It achieves this by recommending a standardized and coherent methodological approach that can be used prospectively or retrospectively by a range of users in the planning, conduct and reporting of economic evaluations of DHIs. The framework builds substantially on existing methodological improvement initiatives in the economic evaluation field [2] [3] [4] and draws on methodological advancements in evidence generation, synthesis and evaluation in digital health [5] [6] [7]. It is not constrained to a singular analytic approach or decision-making construct but utilizes elements of established methodology to accurately represent costs and consequences in context, reflect the nature of the intervention, and complexity of the analytic question. In this way it finds a pragmatic balance between consistency, validity and comprehensiveness and seeks to improve the evidence available for decision making while accommodating the unique attributes of DHIs.

Because cost-effectiveness analysis (CEA) is a predominant method of economic evaluation used to inform investments in health technologies, this framework focuses on adapting it for use in evaluating DHIs. The framework also discusses benefit-cost analysis (BCA), which is more frequently used when evaluating policies implemented in other sectors, including public health interventions implemented outside the health care system.

The framework does not attempt to provide methods to advocate for DHIs over other potential health system interventions. It seeks objective assessment and reflection of all relevant costs and consequences and their distribution, including the opportunity cost of spending of limited health resources. The framework aims to be used to assist in generating
economic evidence to improve health in a digital world [8] rather than viewing DHIs as isolated health system investments.

The framework is targeted towards DHIs as defined by the World Health Organization’s (WHO) classification of DHIs [9], with additional consideration for economic evaluation of DHIs that are enabled by predictive analytics, and non-digital health interventions that are enabled by predictive analytics as part of a wholistic intervention. The framework is also intended to be used within the larger guidance and technical support ecosystem for investments in digital health, represented in the Digital Implementation Investment Guide (DIIG), a global initiative coordinated by the WHO to outline the steps for country investment in digital health.

# 1.3 Problems That the DHI Economic Evaluation Framework Addresses

While the development and implementation of DHIs is rapidly expanding, the generation of economic evidence for DHIs is currently disparate, inconsistent and of variable quality, particularly for DHIs containing AI components. The framework will signal to those generating and synthesizing evidence the nature and type of evidence that will be useful for informing purchasing and investment decisions for DHI.

Some DHIs in use or being developed may have demonstrated and measurable outputs but limited meaningful impact to the wider health system or people. Conversely, DHIs may offer extensive indirect or wider health or societal benefits but do not deliver immediate outcomes in terms of health or process improvements that may be expected in a traditional economic evaluation. The framework will provide a signal of the components of value that are important for those investing as well as the intended beneficiaries from DHI implementation.

There is a critical need for DHI development to meaningfully involve the patients and communities they impact, particularly related to applications of AI [10]. A standardized framework has the potential to facilitate the democratization of DHIs by enabling a comprehensive understanding of value and the methods required for representation.

DHIs are frequently complex and have the potential to disrupt existing processes and pathways of care. This creates challenges in terms of common evaluation principles such as identification of comparators and identifying intended and unintended consequences. The framework provides a mechanism for identifying complexity and appropriately tailoring method choice, enabling context specific analysis that seeks to appropriately represent the uncertainty that complexity creates.
# 2. Scope: Digital Health Interventions and Artificial Intelligence

## 2.1 What Are Digital Health Interventions?

DHIs are a broad and diverse grouping. The World Health Organization adopts a user-focused categorization of DHIs<sup>1</sup> into interventions [9] for:

- Clients, e.g., text message targeted alerts or medication reminders to patients

- Health care providers, e.g., clinical decision support or diagnostic assistive interventions

- Health system managers, e.g., digital interventions to monitor or organize health care providers

- Data services, e.g., digital interventions for data synthesis and visualization or to parse unstructured data into structured data.

These categories incorporate interventions whose intermediate and final outcomes are likely to be health user experience related (such as improved medication adherence leading to better treatment outcomes) and outcomes that relate to systems efficiency (such as an improved clinical coding data management system leading to reduced administration costs for a provider).

To meet the range of available DHIs, the scope of this framework must also incorporate technologies enabled by artificial Intelligence (AI) processes and systems. As an emerging technology and branch of computer science, AI has the potential to exploit data, extract useful information, provide predictive capabilities and support evidence-based decision making to optimize health system performance [11]. Although no common definition of AI has been universally accepted, AI holds a natural character of being able to make machines utilize information to act intelligently. This report adopts the understanding as proposed by others [10] [12] that something “acts intelligently” when 1) what it does is appropriate for its circumstances and its goals; 2) is flexible to a changing environment and changing goals; and 3) learns from experience and makes appropriate choices given its perceptual and computational limitations. The current applications of AI in health are largely limited to “narrow intelligence”, which is the use of data for a specific task, rather than generalized intelligence, which is human-like behavior across a range of activities. Agrawal, Gans, Goldfarb [13] describe an AI application in terms of the anatomy of a task (figure 1), where the existing human approach to a task involves intake of information, with prediction and judgement acting together to result in an action and outcome and an associated information feedback loop where a human learns from each task undertaken. Conceptualization of narrow AI involves separating the prediction element of a specific task while maintaining human judgment and action, ideally resulting in an improved outcome. The use of AI to enhance prediction within a human-operated task is underpinned by the availability and flow of data creating unique considerations for the treatment and role of evidence, with a distinction made between input and feedback data and the initial training data used to develop the AI prediction functionality.

> 1 The WHO Classification of Digital Health Interventions is currently being updated (in 2023). Once the revised version is available, this paragraph will be updated with the revised classification.
<!-- Start of picture text --> we" | Judgement<br>:<br>Current Al “<br>TM applications © SI |. | V<br>Action /<br>‘ a on Intervention<br>alin Predicti eur 7 Outcome<br>| DATA | sosoa DATA —<br><!-- End of picture text -->
Digital Health Systems Digital Health Interventions ; SS =
## 2.3 Usefulness to Whom? The Intended Audience

A fundamental aim of the framework is that it should be “useful” to those planning, conducting, and using economic evaluations of DHIs. There would be limited merit in attempting to develop methodology for only some stakeholders related to economic evaluation of DHIs as there needs to be a consistent understanding and representation through the chain of evidence generation and use.

As a World Bank product, the framework has an immediate intended use for country governments supported by World Bank country and global teams. The broader target audience however can be defined into three major groupings:

- Investor: aiming to invest in the highest value interventions

- Product developer: aiming to develop high-value interventions

- Researcher: aiming to analytically assess and represent value.

Investors are those who are determining whether to fund or reimburse a digital health intervention in a health system. This includes country governments, development partners, and private providers. This category also includes those supporting purchasing decisions in digital health, such as development partners providing technical assistance.

Product developers are those who are producing or creating DHIs. This includes developers in private and public research institutions and start-ups, reflecting the multiple different DHI types and use cases. This also incorporates those who are directly funding development of DHIs, noting that it is particularly important that the representation of value at the product development stage should align closely to the representation of value at implementation.

Researchers and analysts are those who conduct economic evaluations of DHIs within research institutes or other public and private organizations including consultancies. This category also includes research funding organizations that may be providing funding for economic evaluations of DHIs, and researchers who may be using economic evaluations in associated research in digital health such as policy analysis or systematic reviews of the literature. This is a particularly important group as it will undertake the application and production of economic evidence and develop methodological improvements and innovation, building on the research agenda promoted by the framework.

# 3. A Review of the Literature: Evaluating Digital Health Interventions

To determine the utility of an Economic Evaluation framework for DHIs, it is necessary to have a good understanding of the existing evidence base. The World Bank and 3ie developed an evidence gap map (EGM) to investigate the available published literature on impact evaluations of DHIs [15]. The broad goal of this EGM was to facilitate decision-making regarding investment in research to assess the effects and costs of DHIs and AI globally. A more specific objective of the EGM was to support the development of the economic evaluation framework for DHIs, and key findings are summarized below.

## 3.1 Main Findings and Gaps

For the EGM, 63,014 records were retrieved, and 17,142 duplicates were excluded. Most of the records (n=40,374) were excluded during the title and abstract screening. After the fulltext screening, the EGM included 632 Impact Evaluations (IEs) (436 completed and 196 ongoing) and 97 Systematic Reviews (SRs) (89 completed and 8 ongoing).
The EGM reveals two major types of evidence gaps in the literature: ‘absolute’ gaps, where no or few primary studies have been conducted, and ‘synthesis’ gaps, where no SR exists despite a cluster of IEs or the quality of the SRs is substandard or the existing SRs are dated. The major gaps are presented below.

# 3.1.1 The geographic and economic context

The evidence base is skewed towards high-income and Western countries with more limited evidence from low- and middle-income countries (LMICs). In terms of location, there is limited evidence from the WHO regions of Africa, South-East Asia, and Eastern Mediterranean as most of the studies (66%) were conducted in the two regions of America and Europe. Similarly, very few studies were conducted in LMICs and there exists a huge gap in the volume of studies conducted in rich and poor countries. For instance, over three-quarters of the studies were conducted in high-income countries, with the United States alone contributing one-third of the included IE studies.

# 3.1.2 The intervention coverage

The intervention framework comprises four DHI categories divided further into 28 specific types of DHIs. Across the four intervention categories, almost all studies evaluated interventions for clients and health care providers. Significant evidence gaps were found for the other two categories, which were together covered by five studies only. See Figure 3 for a survey of coverage.

Notwithstanding the large volume of IEs covering clients and providers’ DHIs, the evidence base within these intervention categories is unevenly distributed. For example, there is a limited literature on DHIs that empower clients to take the lead on health issues affecting them through, for example, citizen reporting or client-client communication. The focus for most studies is largely on top-down or externally led interventions such as targeted communication interventions. For the health care providers, there is a heavy focus on DHIs concerned with service delivery (e.g., telemedicine or decision support). Interventions that support providers in planning or coordination of health services (e.g., referral or activity scheduling) have received relatively low attention. There is limited evidence of DHIs for health system or resource managers and data services.

# 3.1.3 Frequency of AI use

Despite the potential role of AI in strengthening health systems and improving the quality of health care, a shallow evidence base was found in the published literature for AI. The number of studies that evaluated AI-powered interventions is extremely low. In total, only 13% (n=83) of the studies covered interventions that incorporated AI. Possible reasons include slow adoption of AI in the health sector, limited use of peer-reviewed publications as a mechanism of measuring impact, or a long and rigorous vetting processes that may be required in the medical field. This could also be due to the inclusion criteria of the EGM, which was focused on IEs that utilize counterfactual analysis. This may not be a common approach in AI-related evaluations.

# 3.1.4 Health domains

Not many studies covered critical global health domains such as maternal health, nutrition, HIV/AIDS, and family planning/reproductive health. No more than 45 studies covered each of these domains compared to the 343 studies that covered NCDs. While the majority of IEs were concerned with NCDs, there is also a substantial focus on mental health, infectious
diseases, and child health. The plausible explanation for this evidence gap is that most of the infrequently covered domains predominantly concern LMICs, where very few DHI studies have been conducted.

# 3.1.5 Health outcomes

The impact of DHIs on summary or ultimate health outcomes such as mortality, quality of life and health care has been under-explored. The focus of most studies concerned short-term outputs and directly measurable health outcomes (e.g., number of clinic visits). In fact, there is a huge gap in the number of studies that either report intermediate or summary health outcomes.

The health outcomes reported by the studies are organized along the DHI theory of change (TOC) casual pathway from outputs through impact. For this analysis, outputs comprised process outcome (non-therapeutic) as well as knowledge and beliefs. Intermediate outcomes include behavior change, client/provider satisfaction, health status (natural units), process outcome (therapeutic) and health care utilization. The final outcomes consist of health status (aggregated/summary units) and quality of care. Based on this classification, most outcomes reported by the included studies are intermediate (n=2,306 outcomes), followed by impact (n=342) and outputs (n=296).

<!-- Start of picture text --> SRs<br>IEs<br>SRs<br>IEs<br>SRs<br>IEs<br>SRs<br>IEs<br>SRs<br>IEs<br>SRs<br>IEs<br>SRs<br>IEs<br>SRs<br>IEs<br>SRs<br>IEs<br>SRs<br>IEs<br>SRs<br>IEs<br>0 50 100 150 200 250 300<br>Clients Healthcare providers Health system managers Data services<br>Others<br>Knowled ge and beliefs<br>Health status (aggreg ated/su mmary units)<br>Health status (natural units)<br>Quality of care<br>are n<br>Healthc utilisatio<br>utic)<br>Process outcome (therape<br>(non utic)<br>Process outcome therape<br>on<br>Client or provider satisfacti<br>ur<br>Behavio change<br>c s<br>Economi outcome<br><!-- End of picture text -->

**Figure 3 Outcomes reporting from the DHI Evidence base by intervention type**

# 3.1.6 Economic outcomes

Cost data for the interventions are infrequently reported. About 80% (n=632) of the studies did not report any cost data and out of those that reported more than half simply provided cost data without performing further analysis such cost-effectiveness analysis. A majority of
the studies that did include CEA covered DHIs for clients (n=119 IEs and n=11 SRs) and health care providers (n=45 IEs and n=8 SRs). Only three IEs and no SR covering DHIs for health system managers reported economic outcomes. Moreover, none of the studies on DHIs for data services included any cost data (See Figure 4).

The economic outcomes reported by the studies are further organized into simple, intermediate and summary/impact based on the DHI’s TOC causal pathway. The simple economic outcomes refer to reporting of cost data without any reference to health or wellbeing outcomes (e.g., cost incurred/saved). However, some refer to outputs such as cost per user or persons reached. Studies reporting simple economic outcomes are generally variations on forms of costing analyses. Intermediate and summary economic outcomes link intervention cost data to select health outcomes. The difference between the two is the level of aggregation and finality of the health outcome. Intermediate economic outcomes focus on immediate or natural outcomes such as cost per infection averted whereas the summary outcomes invoke ultimate health outcomes such as cost per life saved or cost per quality adjusted life-year gained. Highly aggregated outcomes, such as net benefits calculated under a cost-benefit analysis framework, would also be classified as a summary outcome, however no cost-benefit analyses were identified in the included studies.

<!-- Start of picture text --> Cost only/Cost analysis 85 8<br>Cost minimization analysis 1<br>Cost consequence analysis 2<br>Cost effectiveness analysis 55 8<br>Cost utility analysis 26 2<br>Cost benefit analysis 0<br>0 10 20 30 40 50 60 70 80 90 100<br>Impact evaluation Systematic review<br><!-- End of picture text -->

**Figure 4 Economic Outcomes reporting from the DHI Evidence base**

# 3.1.7 Evaluation methods

Most studies (n=583, 91%) evaluated the interventions through an RCT framework. The remainder of the studies used quasi-experimental designs. Among the studies with an explicit economic evaluation framework, the majority did not adopt any specific economic evaluation method but instead simply reported the cost incurred or saved. Cost-effectiveness analysis was the most frequently used economic evaluation method (n=63), followed by cost utility analysis (n=28). Cost minimization and cost consequence analyses were reported in one and two studies, respectively. None of the included studies used cost-benefit analysis.
## 3.1.8 Synthesis evidence gaps

The EGM found significant gaps in evidence syntheses for multiple DHI types. Most of the SRs covered process outcome (therapeutic), economic outcomes, health care utilization, health status (natural units), client/provider satisfaction and behavior change outcomes.

There are opportunities for possible syntheses relating to the effects of client health records and health care decision support system interventions. A substantial number of studies have explored the effects of client health records on health care utilization, quality of care and health status (natural units), but no high-quality SR exists. Another area for potential synthesis is the link between practitioner decision support systems and each of the following outcomes: health care utilization, process outcomes (either therapeutic or non-therapeutic), and quality of care.

The effects of telemedicine on process outcome (therapeutic), health care utilization, health status (aggregated/summary units), health status (natural units), knowledge and beliefs and process outcome (non-therapeutic) present additional synthesis gaps for future reviews. Telemedicine is one of the most common interventions with a large body of evidence across all the outcomes of interest in this EGM. Most of the studies have measured outcomes of health status (natural units), process outcome (therapeutic) and health care utilization. A cluster of completed and ongoing reviews on telemedicine were identified, but only two of them (under the outcome categories of client/provider satisfaction, economic outcomes, and behavior change) are rated as high confidence.

Synthesis gaps also exist among targeted digital health communication interventions as most of the included SR are of substandard quality. This was the most covered type of DHI in the included literature as multiple studies have explored the link between targeted digital health communication and health care utilization (n=56), process outcome (non-therapeutic) (n=47), health status (natural units) (n=134) and client/provider satisfaction (n=67).

# 3.2 Conclusions and Implications Related to Existing Evidence

This comprehensive assessment of the digital health intervention impact evaluation literature identified substantive gaps for key types of interventions and in the type of outcomes reported.

Based on intervention coverage analysis, there is a need to explore the availability and possible barriers to assessing the health system management and data service interventions. EGM found few IEs that assessed DHI containing an artificial intelligence component. This is one of the areas with substantive evidence gaps that could be addressed by future studies. In terms of health domains analysis, it is evident from the findings that most DHIs were used to address NCDs. Evidence of the effects in other priority global health issues such as HIV, maternal health and nutrition is very limited. These health domains should be considered in future studies to improve the availability of evidence in this field.

The EGM proposed that future studies on DHIs should consider measuring long-term health and economic outcomes, which are relatively neglected in the current literature. These outcomes include those relating to the quality of care, quality of life, survival rate and summary economic outcomes. There is a limited economic evaluation evidence base for DHIs, which can hinder decisions on policy, programming and scale-up.
# 4. The Case for a DHI Economic Evaluation Framework

## 4.1 Methods Frameworks for Economic Evaluation in Health

A methods framework guides the planning, conduct and reporting of economic evaluation and assists in maintaining a consistent approach allowing comparison of analytical results over time, and context. An economic evaluation framework builds on the concept of a reference case but is less prescriptive and accommodates a wider range of methodological options.

The World Bank has published substantively in applied benefit-cost analysis with methodological recommendations developed as early as the 1970s with subsequent methods initiatives in 1992 and 2010 [16]. The concept of a reference case was first used by the 1st US panel of cost and cost effectiveness in health care in 1996 [17]. The overarching aim was to establish a standardized methodology for the conduct of economic evaluation (in the case of the US panel, cost effectiveness analysis) as a mechanism to improve the usefulness of economic evaluation in decision making. The US panel members recognized that improving the quality of economic evaluation through methodological advancements and comprehensive guidance was insufficient, and that for economic evaluation to find a place in policy, there needed to be a degree of consistency in the analytic approach for decision makers to be able to have confidence in evidence produced and be able to compare and weigh the evidence for competing investments. The 2000s saw a proliferation of national priority setting agencies (also termed Health Technology Assessment (HTA) Agencies) that built on the reference case concept for the specification of economic evaluation that would inform investment decisions for local national health systems. A high-profile national methodology developed by the National Institute for Health and Care Excellence in England and Wales defined a series of methodological specifications for economic evaluation, including how analytical comparators were to be chosen, instruments for health-related quality of life measurement, and the management of equity and time horizon and discount rates. This enabled a level of consistency in the analytical approach used to inform investment decisions. The establishment of the Health Intervention and Technology Assessment Program (HITAP) in Thailand that was tasked with conducting economic evaluation to inform investments in the country’s health benefit package and essential drugs list highlighted the importance of rigorous economic evaluation methodology and processes as a component of a sustainable Universal Health Coverage (UHC) system. HITAP also demonstrated the applicability of this approach for a middle-income country context. The World Health Assembly Resolution 67.23 in 2014 specified the integral role of health technology assessment for countries journeying towards UHC and established a direct link between consistent methods and process when considering evidence of economic evaluation and achievement of UHC. A growing number of countries now have country-specific guidance, reference cases or frameworks for economic evaluation to inform national decision making. While these initiatives are a significant stimulus for economic evaluation methods innovation, the direct global application of these frameworks is limited, as by definition, these are developed from the perspective of a local decision maker concerned with national budgets and national population health in a particular country.

## 4.2 Existing Economic Evaluation Frameworks for Generalized Interventions

To build on the progress of national-level priority setting and economic evaluation methodology to provide a more globally focused guidance, a Bill and Melinda Gates Foundation funded workstream led by the International Decision Support Initiative (iDSI) created a generalized principle-based reference case that would have applicability to multiple
contexts, intervention types and decision makers. The iDSI reference case identified common or “universal principles” of economic evaluation that should be considered for generating economic evidence intended to inform decision making in a UHC context. Further initiatives extended the iDSI reference case, including the Global Health Costing Collaborative [18] that expanded substantially on the “costs” principle, and the Benefit-Cost Analysis Reference Case, that refined multiple principles including the valuation of mortality and morbidity.

The WHO guidance on Generalized Cost Effectiveness Analysis detailed an approach to conducting cost effectiveness analysis that would have applicability at a global, regional, and national level and could be utilized to inform local decision making [19]. Components of the GCEA methodology is used to inform practice of economic evaluation at country and global levels.

Table 2 details four initiatives with an international scope that developed methodological frameworks for the conduct of cost effectiveness analysis and benefit-cost analysis.

A further initiative targeted specifically towards reporting of economic evaluation was the development of the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement in 2013 [20]. The CHEERS statement built on the CONSORT statement designed for trial reporting and outlined the key elements of an economic evaluation to be reported and was aimed primarily at researchers and journal editors.
**Table2  Methods Framework for Economic Evaluation (International scope)**

|**TITLE**|**WHO GUIDE TO COST-**<br>**EFFECTIVENESS**<br>**ANALYSIS**|**REFERENCE CASE GUIDELINES**<br>**FOR BENEFIT-COST ANALYSIS IN**<br>**GLOBAL HEALTH AND**<br>**DEVELOPMENT**|**THE INTERNATIONAL DECISION**<br>**SUPPORT INITIATIVE REFERENCE CASE**<br>**FOR ECONOMIC EVALUATION: AN AID**<br>**TO THOUGHT**|<br>**RECOMMENDATIONS FOR CONDUCT,**<br>**METHODOLOGICAL PRACTICES, AND**<br>**REPORTING OF COST-EFFECTIVENESS**<br>**ANALYSES**|
|---|---|---|---|---|
|**Objectives**|To provide policymakers and<br>researchers with a clear<br>understanding of the<br>concepts and benefits of<br>Generalized CEA (GCEA)|<br>To clarify important concepts, aid in<br>implementation, and provide default<br>values for key parameters including<br>options for standardized sensitivity<br>analysis|To support decisions aimed at improving<br>population health from within available<br>funding while acknowledging the relevance<br>and trade-offs associated with the<br>incorporation of social values into those<br>decisions|To improve quality and comparability of<br>CEAs|
|**Intended**<br>**audience**|Analysts with some<br>background in CEA|Practitioners with some training and<br>experience in conducting economic<br>evaluations|Policymakers|Multiple stakeholders|
|14<br>**Methodological**<br>**approach**|GCEA|BCA|Standardized principle-based methodology|CEA (with cost-consequences analysis<br>(CCA) components)|
|**Structure**|A standard set of<br>methodological choices on<br>how to perform GCEA|Stepwise analytical approach, specific<br>guidance in key areas with focus on<br>approaches implementable in LMIC<br>settings|<br>11 principles with associated methodological<br>specifications and reporting standards|<br>Central topics in economic evaluation in<br>book format|
|**Key**<br>**features**<br>**(non-exhaustive)**|<br>Maximize generalizability of<br>results across settings by<br>comparing to starting point<br>of doing none of the current<br>interventions|<br>Updated and coordinated guidance on<br>key BCA concepts including<br>valuation of morbidity and mortality<br>risk reduction, standardized VSL<br>benefits transfer approach|<br>Each principle accompanied by<br>corresponding methodological specification<br>and reporting standard|Dual perspectives – health system and<br>societal, introduction of the Impact<br>Inventory concept (cross-sectoral CCA)|

<!-- Start of picture text --> 14<br><!-- End of picture text --> Source: Authors, World Bank.
<!-- Start of picture text -->
| _ | Health outeaenes eg disease free<br>INVESTOR (TM.. VALUEsurvival, INCORPORATESDALYssurvival, INCORPORATESDALYs INCORPORATESDALYsDALYs averted<br>; (govermen i! Differingcons undertitutecons undertitute s tandingvaluevalue limts of. of.. what . timesProcessand quicomeshealthProcessand quicomeshealthand quicomeshealth quicomeshealthhealth systennegreduced efficiancy; efficiancy; reduced efficiancy; efficiancy; wait<br>_ development Paine Paine investment, development, and development, and and | | | jee experience oute “9<br>aiming to invest in highest value : ' ONES Gg.<br>infarvantions research into DHIs DHIs satisfaction<br>: | Gross-sectorial outcomes eg inmpacts<br>| on education sector<br>PRODUCTT * Data and evidence generation i.e the the<br>aiming.. pevetoresfafa deverap f wale;; i ’ ] value of deta deta in own own right<br>infarvantions Defining valueas valueasas * Disruption potential eg value of<br>what matters for spportunity for systenvmarket<br>: i 7<br>RESEARCHER people / patients / patients patients ! population population reconfiguraII aration<br>provides a Common a Common Common cam Prediction value 6g value to decision decision<br>coseomenomen ae understanding, anchored in making<br>UHC and and human capital * Equily oulcomes eg improvernents in<br>| | BCCRSS<br>' Costs — bosses and gains, including<br>opportunity eosts<br><!-- End of picture text -->

| | _ | Health outeaenes eg disease free INVESTOR (TM.. VALUEsurvival, INCORPORATESDALYssurvival, INCORPORATESDALYs INCORPORATESDALYsDALYs averted i! undertitutetitute **s** tandingvaluevalue what . eg ; wait ; (govermen Differingcons undertitutecons undertitute limts of. of.. timesProcessand quicomeshealthProcessand quicomeshealthand quicomeshealth quicomeshealthhealth systennegreduced efficiancy; efficiancy; _ development Paine Paine investment, development, and development, and and | | | jee experience oute “9 aiming to invest in highest value : ' ONES Gg. infarvantions research into DHIs DHIs satisfaction . : | Gross-sectorial outcomes eg inmpacts f | on education sector PRODUCTT * Data and evidence generation i.e the the aiming.. pevetoresfafa deverap f wale;; i ’ ] value of deta deta in own own right infarvantions Defining valueas valueasas * Disruption potential eg value of what matters for spportunity for systenvmarket : : i 7 RESEARCHER people / patients / patients patients ! population population reconfiguraII aration provides a Common a Common Common cam Prediction value 6g value to decision decision coseomenomen ae understanding, anchored in making . UHC and and human capital * Equily oulcomes eg improvernents in | | BCCRSS ' Costs — bosses and gains, including opportunity eosts
The review summarized assessment frameworks across telemedicine and mobile health and generic conceptualizations for DHI assessment. Using the structure of the EUnetHTA core model (a methods framework developed by European national HTA agencies [22]), a list of categories was elicited to screen against and map the scope of included publications. The list included 12 categories: health problem and current use of technology; safety; clinical effectiveness; patient and social aspects; economic; legal; ethical; organizational; usability; data security; and technical aspects and stability. In total, 11 value assessment frameworks were identified. The most often discussed assessment domains were safety, clinical effectiveness, usability, economic aspects, and interoperability which were discussed in seven of the frameworks, while organizational, usability, data security, and technical considerations were discussed in six of the frameworks. The least attention was devoted to the understanding of ethical (two out of 11 frameworks) and legal aspects (one out of 11 frameworks).

The review proposed several recommendations regarding the criteria to be considered in the measurement of value of DHIs, as detailed in Table 3.

**Table 3 Summary Findings from Review of Existing DHI Analytical Frameworks**

|**FIVE RECOMMEN**|**DATIONS**|
|---|---|
|**Choice of**<br>**comparator**|The value of a DHI should be determined by the incremental advantage<br>compared to the current standard of care.|
|**Multi-stakeholder**<br>**perspective**|The value of a DHI should quantify incremental differences it delivers to all<br>beneficiaries.|
|**Organizational**<br>**impact**|The value of DHIs should be conditional on the health care system<br>preparedness to consume efficiency gains and assurance that data generated by<br>DHI will be accessible by health care professionals.|
|**Multidimensional**<br>**outcomes**|The value delivered by digital health solutions should consider multiple<br>dimensions such as clinical, organizational, behavioral, and technical.|
|**Interoperability**|Connectivity to other data sources must be considered in the evaluation of a<br>digital health solution.|
|**ONE METHODOL**|**OGICAL SUGGESTION**|
|**Value aggregation**<br>**function**|Given the multiple criteria to be considered in the pricing and reimbursement<br>process, each value attribute should be weighted in any calculations based on<br>the preferences of chosen stakeholder groups, and the aggregate value score for<br>each DHI should be estimated.|

_Source: Adapted from Kolasa et al (2020)_ .[6]
# 4.5 Digital Health Interventions and the Role of Evidence

Digital Health Interventions are unique among other health interventions in the way that evidence is generated and synthesized. An economic evaluation typically views the range of evidence for a health intervention at a single point in time in a particular context and many methodological approaches are used to reflect how health intervention parameters may change over time, context and scale. However, there are unique attributes to the evidence base for DHIs, including the way that the implementation of a digital intervention will in itself commonly generate further evidence and that the specific attributes of a digital intervention can evolve over time to adapt to context.

The distinction between efficacy (the effect observed in a controlled and ideal environment) and effectiveness (the effect in routine conditions in a specific context) is a key component of any type of economic evaluation intended to inform decision making. Evaluation of noncomplex health interventions, such as pharmaceuticals, will typically adopt a primary effect estimate from clinical trial data and then attempt to reflect any variation from the observed effect that is expected in the context of the economic evaluation. This information is also supplemented by observed data and real-world evidence (RWE) but a common objective is to utilize efficacy information to estimate effectiveness.

The regulatory environment for a particular type of health intervention is a major driver of the approach to evidence generation. The strong traditional focus of health intervention regulation on patient safety and efficacy means that evidence generation is commonly focused on whether potential benefits outweigh potential harms rather than cost effectiveness or value for money. When conducting economic evaluation for an intervention that already has regulatory approval (such as pharmaceutical), evidence required at the point of regulation will have demonstrated efficacy and safety for a particular clinical indication. In contrast, economic evaluations of non-regulated products (or where the regulatory environment does not routinely require demonstrated efficacy through clinical trial) may not have baseline evidence of impact to parameterize an analysis.

DHIs, in common with many other complex interventions, are intricately integrated into the context within which they are used. This diminishes the importance of the concept of efficacy within an economic evaluation of a DHIs and limits the applicability of the traditional evidence hierarchy. The randomized controlled trial (RCT) remains a gold standard for isolating the treatment effect of a digital or non-digital interventions, however RCTs are unlikely to be able to routinely provide the intervention effectiveness estimate required in an economic evaluation of a DHI.

However, published peer-reviewed reporting trials remains an essential element of evidence generation for DHIs and should be used where applicable. The SPIRIT-AI and CONSORT-AI statements provide clear specification for the design and reporting of protocols and trials of interventions involving Artificial Intelligence components [23] [24]. They expand on the established CONSORT and SPIRIT statements with specific components to enable the generation of evidence related to AI technologies. The expanded use of the statements will enable improved trial conduct and reporting and is likely to have a positive impact on the evidence available for use in economic evaluation.

A DHI, by its nature, generates data through its use. The data may be utilized in different ways depending on the information system in which the DHI operates, but the routine generation of data has implications for the management of evidence in an economic evaluation [7]. The data gathered will inform basic parameters such as utilization, but also enables linkages with other sources to impact on health and processes dynamically. As many DHIs include features
of adaptive design, the generation of evidence can lead directly to changes in the effect size of the intervention. There is also substantive scope for DHIs to dynamically improve the effectiveness of other digital and non-digital interventions as a result of improved evidence. For example, a basic digital clinical decision support tool to aid prescribing may have direct process efficiency impacts on the time taken for clinicians to complete a patient consultation. Its efficiency impact may improve as it expands to more therapeutic areas or facilitates a greater granularity of decision support, and over time may also improve the effectiveness of the medical interventions being prescribed through improved targeting of indications and dosing and monitoring of side effects. Considerations for the assessment of digital predictive capabilities can build on existing economic evaluation of non-digital risk prediction models. A systematic review assessed the range and quality of economic evaluations of 39 risk prediction models in clinical practice, evaluating models such as the Framingham Risk Score (for risk of Coronary Health Disease) and Nottingham Prognostic Index (for prognosis following breast cancer surgery) [25]. The review found wide variation in methodological approaches particularly in areas such as comparator choice and inclusion of costs, with the majority adopting cost utility or cost effectiveness analysis (life/life year saved). Although limited, this existing evidence base can be used to inform approaches to evaluating digital prediction models that may incorporate predictive analytics functionality.

Existing initiatives have consolidated approaches to the consideration of the evidence base for DHIs. Murray et al developed a series of research questions that assist in designing research that is tailored to generating evidence for decision making related to DHIs [26]. The UK’s National Institute for Health and Care Excellence (NICE) have developed an Evidence Standards Framework that details the approach to use of evidence when an evaluation is conducted to inform investments in the National Health Service in England and Wales [7]. While this initiative is targeted towards a particular high-income country context, it demonstrates an approach to the organization of evidence that can be applied more generally.

The dynamics of evidence generation and use for DHIs are clearly different from traditional health interventions, but any investment in DHIs, particularly by country governments, will immediately displace actual or potential spending on other interventions, be they digital or non-digital. Therefore, DHIs should not be seen as exceptional or warranting a lower standard of evidence to inform economic evaluation or decision making [7]. The framework concept attempts to ensure that the principles supporting evidence production and use are applied consistently across digital and non-digital interventions, while accommodating operational and pragmatic differences.

# 4.6 The Risks and Benefits of a DHI Economic Evaluation Framework

Any proposed methodological framework is intended to be widely adopted and incentivized to improve analysis and use of resulting information. The benefits of using a framework is linked to its capability to enhance both the quality of individual decisions through robust standards for the planning, conduct, and reporting of economic evaluations and the consistency in decision-making over time [27].

However, there are risks associated with the use of a prescriptive framework. Economic evaluation is a rapidly evolving field, and particularly in the case of DHIs, there are several areas where consensus does not yet exist or methods have not been developed to appropriately represent the full range of costs and benefits, their inherent uncertainty, and to accurately understand the nature of the opportunity cost of investment. Therefore, a framework carries the risk that certain practices and ways of representing value in economic evaluation of DHIs become “set in stone”.
<!-- Start of picture text --> 1 DETERMINE THE CONTEXT<br>2 DETERMINE THE INTERVENTION TYPE<br>2E\J= DIGITALWITHOUT HEALTH PREDICTIVE INTERVENTION ANALYTICS WITHDIGITAL PREDICTIVE HEALTH INTERVENTION ANALYTICS | <®ayeCD NON-DIGITAL‘INTERVENTION HEALTHWITH<br>¢ 4 efs Yi r. " afs. ®,PREDICTIVE5 Axe ANALYTICSioiz<br>3 ESTABLISH LEVEL OF COMPLEXITY<br>INTERVENTION CAUSAL Gh -¢ om fin<br>PATHWAY AADay,ese> i x= -<br>O es<br>4 ANALYTICAL PRINCIPLES<br>COMPARATOR TIME FRAME COST BENEFITS | PREDICTIVE ANALYTICS DYNAMICS | UNCERTAINTY<br>5 = nd<br>DE a th= Os a<br>5 REPRESENT THE VALUE PROPOSITION<br>=]<br>IMPACT INVENTORY Fg] VALUE AGGREGATION<br>ooo<br>Ve Distributiona oe Distribution a"<br>—<br><!-- End of picture text -->
an essential element and includes digital architecture and equipment, information systems, and software where applicable. Analysts should use existing context analysis tools as appropriate to the intervention, such as the Digital Health Investment Review Tool developed under the Maternal and Child Survival Program [28].

The intended decision maker is the organization, body, or group of people who will make use of the information produced in the economic evaluation. It is an important component of the economic evaluation to specify as it informs interpretation and application of all other methodological concepts, including the perspective of the analysis. It is important to note that there is no generalized “correct” intended decision maker for an economic evaluation, but the production of an economic evaluation with an undefined decision maker can potentially lead to inappropriate or incorrect use of the evidence, and sub-optimal decisions. For example, an economic evaluation of a digital health technology intended to inform an investment decision by a secondary care provider (such as a private hospital group) is likely to approach the definition of the comparator, scoping of costs, time horizon and representation of clinical effects and summary impact measures very differently to a country government department of health considering rolling out the same intervention nationally.

Intrinsically linked to the intended decision maker is the context in which the DHI will be implemented. Given the major influence that the context will have on all aspects of a DHI functionality and impact, there is little utility in concepts of efficacy for the economic evaluation of DHIs. The context of the DHI should therefore be established from the outset, with clear explanation of key components of the context, including the country context, health system environment, and digital enabling environment.

Determining the intended decision maker and context does not preclude applicability of the outputs of an economic evaluation to other contexts. In practice, the clear definition of the decision maker, with explanation of how this perspective affects methodological decisions and parameterization will facilitate transferability of economic evaluation to other contexts as users will be able to assess which aspects of an analysis require modification to apply it to another context.

# STEP 2: Determine the intervention type

Specification of the intervention, including its use-case and intended population is an essential element of the evaluation. This framework requires not only a general description of the intervention, but also alignment to established classification systems. In the first instance, the intervention should be assigned to a WHO DHI type, with specification as to whether there is a predictive analytics component. This creates three major categories of DHI, with further granularity for DHIs under the WHO classification system.

1. DHIs defined under the WHO DHI classification system without AI enabling functionality – e.g., simple text message to patients as a reminder for regular followup.

2. DHIs defined under the WHO DHI classification system with AI enabling functionality – e.g., machine learning radiology diagnostics.

3. Non-digital health interventions enabled by the used of AI technology – e.g., use of demographic and clinical information for identification of at-risk patients for health professional consultation.
<!-- Start of picture text -->
1. DH! not utilizing 2. DH! utilizing 3. Non digital interventions<br>Al funetionality Al functionality and processes using Al<br>functionality<br>Digital health interventions Al functionality<br><!-- End of picture text -->
**Table 4 Digital Health Interventions and Complexity**

|**TYPE**|**DESCRIPTION**|
|---|---|
|**Intervention**<br>**complexity**|Many DHIs are multi-attribute, with inter-dependent parts and delivery<br>mechanisms, and often adapt over time.|
|**Causal pathway**<br>**complexity**|The causal pathway linking observed outputs to outcome and impact can have<br>substantial complexity, compounded by range of identified outcomes and<br>complex attribution and feedback loops.|
|**Outcome complexity**|DHIs commonly have range of outcomes, including health and health system<br>effects and involve spillovers and externalities, impact on patients, users, and<br>the wider population.|

_Source: Petticrew et al 2013, McNamee 2016_

# STEP 4: Apply analytical principles

The fourth step of the framework is to apply economic evaluation principles as detailed in table 5. This builds on the extensive methods literature in economic evaluation [4][33]. It is beyond the scope of this framework document to replicate existing methodological guidance, but to identify the principles that should ideally be followed when making context and intervention-specific methods choices. An important consideration is that while there may be some variation to the particular approach of any economic evaluation, these principles are expected to apply for both digital and non-digital interventions (with the exception of predictive analytics that is expected to be unique to DHIs with a predictive analytics component).

**Table 5 Analytical Principles**

|**ITEM**|**PRINCIPLE**|
|---|---|
|**Comparator**|The comparator(s) against which costs and benefits are measured should accurately<br>reflect the decision problem|
|**Timeframe**|The time horizon of the analysis should be of sufficient length to capture all<br>significant differences between the intervention and comparator|
|**Costs**|All differences between the intervention and the comparator in expected resource<br>and costs of delivery to the target population(s) should be incorporated into the<br>evaluation|
|**Benefits**|All differences between the intervention and the comparator in expected benefits to<br>the target population(s), should be incorporated into the evaluation|
|**Predictive**<br>**analytics**|The dynamics of predictive analytics on evidence generation, costs and benefits<br>should be reflected in the analysis|
|**Uncertainty**|The uncertainty associated with the evaluation estimates should be appropriately<br>represented|
Establishing the comparator against which the intervention will be evaluated is fundamental to the approach and outputs of an analysis. Economic evaluation seeks to represent costs and consequences in context, which requires understanding of what the intervention would replace, including if the current practice is doing nothing or minimal action. However, it is often difficult to determine exactly what would be replaced by an intervention, and so this framework recommends the development of two comparator scenarios, drawing on the approach of the iDSI reference case. Analyst judgement is required to determine the validity of each scenario depending on the context and needs of the analysis. The first scenario is standard of care or existing process, and the second scenario is minimal supportive care. For example, an AI-enabled diagnostic tool that assisted interpretation of radiology scans to identify disease and referral would be compared to a scenario that consisted of a radiologist manually interpreting scans and a scenario where patients wouldn’t routinely have access to radiology services. The costs and benefits of the new diagnostic tool could then be represented relative to these two scenarios to gain a full understanding of the impact that the new intervention represents.

The timeframe adopted in an analysis is important to ensure that any bias based on the timing of costs and benefits is minimized. The general principle is that the timeframe should be of sufficient length to capture all relevant costs and benefits between the intervention and comparator. Where there are observed mortality differences between the intervention and comparator, this will usually require that a lifetime time horizon is adopted even if mortality estimates are largely projected to a longer horizon. This is particularly relevant for DHIs where a significant proportion of costs are frequently incurred upfront at the stage of development or implementation, with benefits extending into the future. An important consideration for timeframe choice is the discount applied to future costs and effects. This framework does not prescribe a particular annual discount rate but requires that the discount rate chosen is clearly specified and incorporates local contextual evidence.

The framework requires a comprehensive approach to costing, with the principle specifying that “all differences between the intervention and the comparator in expected resource and costs of delivery to the target population(s) should be incorporated into the evaluation”. The approach to collection, synthesis, and inclusion of costs in an economic evaluation is detailed extensively in the literature. The Global Health Costing Collaborative (GHCC) is an initiative dedicated to establishing best practice in costing and has made a significant contribution to methodological understanding and practice of costing health interventions and aligned to the costing principle of this framework. However, the nature of costing for DHIs is likely to require specific approaches and it is expected that dedicated research relating to DHI costing is required. In particular, assessment of a DHI is likely to require consideration of how fixed costs are to be attributed in relation to costs of development and future cost structures and at different levels of scale. A “legacy” intervention may pass costs or cost savings onto the health system over a long time period and the marginal cost per user may be substantially reduced with many concurrent DHIs.

In common with costing, estimation of the benefits associated with a DHI should take a comprehensive approach, with the principle that all differences between the intervention and the comparator in expected benefits to the target population(s) should be incorporated into the evaluation. The principle requires in the first instance that benefits are identified in natural units, with approaches to aggregation and synthesis considered in Step 5. The impact inventory details specific areas of value that will guide the organization of the benefits into meaningful categories, aiding interpretation by decision makers. A particular consideration
for representing benefits of DHIs is the potential for intervention modification after initial implementation (see earlier section on 3.5 on role of evidence) [34].

The Predictive Analytics principle requires that the impact associated with the ability of a DHI to assist in prediction within the anatomy of a task is explicitly detailed and quantified where possible. This category is unique to DHIs with predictive functionality and enables the impact of the prediction component of an intervention to be isolated from the wider impact of intervention. For example, a radiology diagnostic intervention with predictive analytics may be used for assisting diagnosis of tuberculosis disease from radiology scans. The complete diagnostic intervention may have many components, such as digital information capture, automated messaging and referral functionalities, but the prediction value would be represented by the improved diagnostic accuracy in terms of sensitivity and specificity of the intervention compared to either standard of care or combinations of the intervention in conjunction with radiologist prediction and judgement. Representing the prediction value in isolation enables comparison of differences in prediction value between interventions and at different time points and contexts.

This principle interacts with other principles (particularly costs and benefits); however, the unique predictive impacts should be explicitly established in the analysis. The approach to representing predictive analytics dynamics will need to be tailored to the nature of the intervention, with specific consideration given to the identification of the impacts relating to the nature of evidence generation and the disruptive and process changes that may be enabled by the intervention.

# STEP 5: Represent the value proposition

The fifth step in the framework is to represent the value proposition. The framework takes a dual approach of disaggregation using an impact inventory (see Table 6), and aggregation in a form required by the intended decision maker. Throughout the representation of the value proposition, it is important that the uncertainty and distribution of impacts are clearly represented. Methods for representation of uncertainty in economic evaluation are well established, and analysis should reflect major aspects of uncertainty including parameter, structural, and methodological uncertainty [33], incorporating the unique attributes of DHI in relation to evidence generation and use (Section 2). The distribution of the costs and consequences of any health intervention is an essential input to an investment or implementation decision and should be represented as clearly as possible in both the impact inventory and in any aggregation. Representing the distribution of impacts across populations enables consideration of factors beyond efficiency or health maximization and includes equity and priority to disadvantaged populations. While the methodological approaches to representing distribution of impacts is likely to be common across digital and non-digital interventions, [35] [36] DHIs have important and unique distributional considerations. Firstly, in many instances DHIs have the potential to mitigate existing access barriers to traditional health care provision particularly for marginalized or rural populations. However, there is also potential for DHIs to exacerbate existing inequalities where access to a DHI is limited to those that already have access to an enabling digital health infrastructure, such as a smartphone or a local clinic with an internet connection. While an economic evaluation may not be able to quantify the full distributional impacts of a DHI investment, the Value Proposition step should attempt to represent impacts using established economic evaluation methods.
# 5.2 Disaggregation: Impact Inventory

An impact inventory is a series of health and non-health outcomes and costs presented in a tabulated form and is a common approach in the cost consequence analysis form of economic evaluation. The US Panel on Cost-Effectiveness in Health and Medicine proposed the use of an impact inventory when conducting CEAs, noting that many outcomes and costs were important to a decision maker, but could not reasonably be incorporated within an Incremental Cost Effectiveness Ratio (ICER). In addition, the US Panel Impact Inventory enabled the representation of both a health services perspective and a wider societal perspective to the analysis, enabling a versatile approach that could address the economic evidence requirements of a range of potential decision makers. This framework builds on this approach with specification of particular impacts that should be represented in an economic evaluation of DHIs.

The Impact Inventory specifies five broad inventory areas, and it is expected that depending on the nature of the DHI being evaluated, additional sub-categories will be required. Application of the impact inventory will facilitate further specification and detail on the approach for comprehensive representation of appropriate sub-categories, building on existing methodological guidance on outcomes measurement of general health interventions. Identification of the “type” of DHI in Step 2 will guide a generalized approach to representing components of the impact inventory.

An essential element of the impact inventory is that outcomes are represented in units that are applicable to the particular category. It is likely that the different categories may not be completely mutually exclusive, and some categories will have downstream attribution effects on summary metrics and costs. However, the main aim of the impact inventory step is to facilitate transparent representation of results in disaggregated form and to provide a common framework for value aggregation. Considerations about the appropriate level of aggregation and development of summary metrics are addressed in the value aggregation step.

The Health impact component requires representation of the observed health effects relative to the identified comparator(s), with a requirement for both positive and negative health effects to be represented. The Health System impact requires representation of the changes in processes that are expected to be facilitated by the introduction of the DHI, and is likely to incorporate intermediate outcomes such as user experience and acceptability, where these effects lead to meaningful process change.

The User-experience impact will commonly be reported in a qualitative synthesis or through use of established tools for measurement of the user experience. It is beyond the scope of this framework to recommend specific user experience measurement tools; however, the analyst should ensure that any tools are validated and appropriate for use in the context in which the intervention is to be used. The User-experience impact may be a fundamental contributor to the observed health impact or health system impact. For example, a recent DHI that utilized a personalized digital counseling application to enable consumer choice in contraceptive use found a positive effect on increasing adoption [37]. As a joint patient-andprovider facing intervention, a substantive proportion of the impact was contingent upon the provider/user being able to effectively interact and use the application. Economic evaluation that specifically identifies user experience (whether that user is a patient, the general population, a health provider or a health services manager) will be essential to accurate representation of the overall intervention impact.
The Cross-sectoral impact enables a wider societal impact of a DHI to be incorporated, including impacts on other social sectors such as education. While the “Health Impact”, “Health System Impact” and “Cross-sectoral impact” components of value representation are methodologically similar whether the intervention is non digital or digital, the Value of Data and “User experience impact” value categories are more specifically tailored towards the unique impact of DHIs It is likely that there may not be quantitative information for some aspects of the Impact Inventory, in which case a qualitative description of the impacts should be provided.

**Table 6 Components of Impact Inventory**

|**COMPONENT**|**DESCRIPTION**|
|---|---|
|Health impact|Health impacts: natural units, generalized units, and net health<br>benefits that incorporates the opportunity cost of lost health|
|Health system impact|The impact on the health system as a result of intervention<br>implementation, includes direct measurable impacts and longer<br>term predicted process changes or disruption potential.|
|User experience impact|Impacts on patients, population and health workforce not<br>represented within health or health system impact such as trust,<br>privacy, choice, satisfaction and knowledge.|
|Value of data|The value represented by the availability of data gathered<br>through the use of the DHI that is not represented in direct<br>impacts on health, health system or user experience|
|Cross-sectoral<br>impact|<br>Impacts on non-health sector e.g., education and wider societal|
|(beyond health system)|benefits e.g., human capital|

Value of data requires explicit identification of what evidence is being generated through the use of the DHI. This could range from simple utilization data to comprehensive datasets on clinical pathways, processes, and patient outcomes. This value is likely to be contingent on the context of the DHI and in particular the existing information systems and digital infrastructure, further highlighting the need for explicit specification of context in Step 1.

# 5.3 Value Aggregation

A central consideration in an economic evaluation is what, if any, aggregation of the costs and consequences is required and is appropriate. The ability for an economic evaluation to coherently summarize costs and benefits into easily understood metrics is often a fundamental component of communication, comparability and usefulness in decision making. However, the approach to aggregation often involves a series of assumptions, simplifications and underlying conceptual frameworks that if not represented explicitly can lead to misunderstanding or misrepresentation of results.

Representing an incremental cost effectiveness ratio (ICER) with a generalized measure of health such as the Quality-Adjusted Life Year (QALY) is commonly used for marginal health budget decisions under an extra-welfarist framework where improvement in population health is a major, but not only, consideration of decision makers. Use of the ICER with a generalized measure enables representation of the local opportunity cost of the decision with fixed or known budget where the marginal productivity of the health system can be estimated. This approach is commonly used by national priority setting institutions such as Health Technology Assessment agencies within a broader decision making framework for investment decisions about individual interventions or groups of interventions and in development of clinical guidelines and treatment protocols.
Estimating a return on investment (ROI), Benefit-Cost Ratio (BCR) or net benefits utilizing a benefit-cost analysis (BCA) analytical approach is commonly applied to major policy or regulatory changes or large-scale investments where there are multiple impacts across a range of constraints and sectors and the objective is to represent the impact and distribution on societal welfare. Health impacts of interventions can be monetized utilizing concepts such as the value of statistical life (VSL) informed by valuation of mortality or morbidity risk reduction. Converting all positive and negative impacts into monetary form enables aggregation using a common metric which enables clear and simple communication of results. Applying a BCA framework to DHI investments will require consideration of how the value categories introduced in the impact inventory can be monetized. Methodology for monetization of Health Value, Processes Value, and Cross-sector Value impacts are largely in common with established BCA methods. If a total monetized summary metric is required in the analysis, the bespoke approaches would need to be adopted for monetization of Evidence generation value, Prediction Value, and Disruption Potential, with careful consideration required to ensure correct attribution and avoidance of double counting. Some emerging approaches that may be applicable for monetization of DHI-specific impacts are detailed in table 7, and it is expected that ongoing methods research will generate more tailored methods.

**Table 7  Approaches for monetization of DHI-specific impacts**

|**COMPONENT**|**ESTABLISHED/EMERGING APPROACHES TO**<br>**MONETIZATION**|
|---|---|
|Health value|Value of Statistical Life (Robinson et al 2019) [38]|
|Process value|Efficiency analysis (Durrett et al) [39]<br>Time preference (Whittington & Cook 2019) [40]<br>Contingent Valuation(Bayoumi 2004) [41]|
|Cross-sectorial value|Economy-wide effects (Strzepek et al 2018) [42]|
|Evidence generation value|Data Shapley (Ghorbani & Zou 2019) [43]<br>Value of DataAssets (First San Francisco) [44]|
|Prediction value|Scenario analysis – preliminary development (Agrawal et al<br>2019) [45]|
|Disruption potential value|Analysis of potential future scenarios with valuation|

A social return on investment (SROI) analysis is a form of BCA. Building on the approach for estimating ROI it explicitly attempts to identify and isolate wider social benefits from immediate financial returns of an investment and is useful where there are particular benefits that are considered socially desirable by a decision maker and that should be explicitly represented.

The major types of value aggregation are detailed in table 8.

**Table 8 key features of different types of value aggregation**

|**TYPE OF VALUE**<br>**AGGREGATION**|**REPRESENTATION**|**INCORPORATES**|**INTERPRETATION**<br>**REQUIRES**|**FURTHER**<br>**METHODOLOGICAL**<br>**GUIDANCE**|
|---|---|---|---|---|
|**Cost-benefit**<br>**analysis**|Benefit-cost ratio: net<br>benefits, return on<br>investment|Full costs and<br>benefits converted<br>to monetized|<br>Understanding of<br>valuation technique,<br>particularly health|Benefit-Cost Analysis<br>Reference Case [4]|
|||form to extent<br>possible|benefits||
|**Cost utility**<br>**analysis (a subset**<br>**of CEA)**|Incremental cost<br>effectiveness ratio<br>(effect in generalized<br>units) Net<br>health/monetary<br>benefits|Costs and effects<br>from a defined<br>perspective|Marginal<br>productivity of<br>health system (k);<br>ICERs of competing<br>investments|iDSI Reference Case;<br>[2] Drummond 2015<br>[46]|
|---|---|---|---|---|
|**Cost effectiveness**<br>**analysis**|Incremental cost<br>effectiveness ratio|Costs and effects<br>from a defined<br>perspective<br>Effects in natural<br>units|ICERs of competing<br>investments (in same<br>effect units)|<br>Drummond 2015 [46]|
|**Social return on**<br>**investment**|SROI|All costs and<br>benefits converted<br>to monetized<br>form, plus wider<br>social impact|<br>Understanding of<br>relative weight or<br>importance of<br>additional social<br>benefits|Olsen 2003 [47]|

The complexity and variety of DHIs and wide range of potential decision makers necessitates that the concept of “Return on Digital Investment” utilizing the DHI economic evaluation framework does not prescribe a particular form of aggregation, however it does require that researchers and decision makers are cognizant of the options available, the additional information required for their interpretation, and the risk of creating false certainty through obfuscation of important gaps in evidence, attribution, and valuation.

The use of the Impact Inventory in addition to value aggregation will assist in transparent representation of results, and it is expected that a comprehensive assessment of the Return on Digital Investment will involve a series of quantitative and descriptive outputs.

# 6. Using the DHI Economic Evaluation Framework

## 6.1 Case Example ONE

The overarching concept when applying the framework is to “comply or justify” – that is, users should attempt to adhere to the steps and principles of the framework, but where this is not feasible or appropriate, should provide justification for the methodological choice. In this way, the framework will facilitate methodological transparency over conformity, improving the overall usefulness of economic evaluations of DHIs.

This section details the initial planning steps for application of the framework to the evaluation of an information system and predictive analytics intervention to serve as an example of how the framework can be used to plan the methodological approach. As more evaluations utilize the framework, further examples will enable refinement of methods specifications.

The example analytic plan utilizing the framework is an economic evaluation of the Orbit data capture system and the Zenysis data analytics platform compared to a pre-implementation scenario in a targeted population HIV prevention program in South Africa. Orbit is a webbased data collection and information management system for capturing programmatic performance information. Data entry is completed by health workers and enables the generation of multiple monitoring and evaluation indicators. The pre-implementation scenario against which Orbit is compared is an Excel-based information management system, where health workers would electronically send completed Excel templates monthly, entered
from paper-based point of care data collection forms. Data collected through the Orbit system is utilized by Zenysis, an analytics interoperability platform that can integrate multiple sources of data, enabling data triangulation, processing, and visualization of fragmented data to improve planning and decision making. The economic evaluation would aim to assess Orbit and Zenysis as a combined intervention to inform potential scale-up or use within other country contexts.
# **Table 9 Framework Example 1- Data Analytics Intervention**

|**FRAMEWORK**<br>**COMPONENT**<br>**SPECIFICATION**<br>**CONSIDERATIONS**|
|---|
|**STEP 1: IDENTIFY THE DECISION MAKER AND CONTEXT**|
|Global Fund and<br>subsidiaries.<br>South African HIV<br>program targeting specific<br>groups<br>Although the direct intention is to inform<br>Global Fund programs, there is a secondary<br>objective to provide general information to<br>inform other potential funders or country<br>governments<br>**STEP 2. DETERMINE THE INTERVENTION TYPE**<br> <br>|
|Data collection and<br>organization and predictive<br>analytics<br>Type: Data Services. (WHO Class 4.1:<br>“data collection, management and use”)<br>with predictive analytics component.|
|<br> <br>**STEP 3. ESTABLISH THE LEVEL OF COMPLEXITY**|
|**Intervention**<br>Low complexity<br>Relatively simple data capture and analytics<br>intervention|
|**Outcomes**<br>High complexity<br>A range of process and health outcomes|
|**Causal pathway**<br>High complexity<br>The pathway establishing links from<br>immediate outcome measures to summary<br>metrics of health are highly complex and<br>uncertain with limited evidence of<br>attribution. This complexity likely to affect<br>the approach to value aggregation|
|**STEP 4: ANALYTICAL PRINCIPLES**|
|**Standing**<br>Patients; health workers; program implementers and sub-recipients;<br>Global Fund|
|**Comparator**<br>Routine data collection (paper based with manual entering into MS<br>Excel)|
|**Timeframe**<br>3-year time horizon; disaggregated (determined based on decision<br>maker’s requirements)|
|**Costs**<br>Implementation and recurring costs; cost offsets for reduced health<br>system utilization|
|**Benefits**<br>Process efficiencies in data gathering, Health worker user experience,<br>Speed and precision of decision making, Health impact of improved<br>care|
|**Predictive analytics**<br>Identification of “high need” recipients and targeted package of<br>interventions based on characteristics<br>**Table continued...**|
# **Table 9 Framework Example 1- Data Analytics Intervention (continued)**

|**FRAMEWORK**<br>**COMPONENT**|**SPECIFICATION**|**CONSIDERATIONS**|
|---|---|---|
|**STEP 5: VALUE PRO**|**POSITION**||
|**Value aggregation**|Limited to financial return<br>on investment|With available evidence and complexity of<br>causal pathway it is not possible to<br>represent aggregated health impacts<br>required for net benefits or an incremental<br>cost effectiveness ratio. Decision maker can<br>observe financial return on investment with<br>clear identification of impacts beyond the<br>value aggregation|
|**Impact inventory**|_Health value_|No directly identifiable health impact|
||_Process value_|Valuation of process improvements and<br>program efficiencies|
||_Value of data_|Estimate the unique value that the<br>generated information represents to<br>identified process efficiencies, and potential<br>value to other similar programs*|
||_Prediction value_|Qualitative representation of predictive<br>analytics functionality*|
||_Cross-sectoral value_|No directly identifiable cross-sector value|
|**Uncertainty**|One-way parameter<br>sensitivity analysis;<br>scenario analysis||
|**Distribution**||Limited sub-population analysis as this is<br>targeted program; represent the costs and<br>consequences annual over analytical period|

S _ource:_ Authors, World Bank. _Note: *Current methods likely limited to descriptive statistics, further methodological development on quantification of value required_
# 6.2 Case Example TWO

The second case example is a potential approach for economic evaluation of an intervention to utilize deep learning neural networks to interpret chest radiography (CXR) to screen and triage patients with pulmonary tuberculosis (TB). The example draws on an existing assessment of the diagnostic accuracy of three systems “CAD4TB”, “qXR” and “Lunit” for a hypothetical decision for scale up of the interventions in the Cameroon country context, funded by domestic health resources. The existing retrospective assessment investigates the diagnostic accuracy of the systems using CXR images collected from Nepal and Cameroon as part of different studies. Included populations were adults (aged 15 years or older) with symptoms suggestive of TB (cough more than 2 weeks, fever, night sweats, weight loss). While the existing evaluation provides useful insights into the accuracy of the different systems compared to existing methods of radiologist interpretation of CXR, an economic evaluation would increase the understanding and representation of the different components of value for the interventions. In this example, it is assumed the government seeks information to guide allocatively efficient use of the limited health budget.

**Table10  Framework Example 2 – AI-enabled diagnostic:**

|**FRAMEWORK**<br>**COMPONENT**|**SPECIFICATION**|**CONSIDERATIONS**|
|---|---|---|
|**STEP 1: IDENTIFY T**|**HE DECISION MAKER AND**<br>|**CONTEXT**<br>|
|**STEP 2. DETERMIN**|Country government,<br>Cameroon<br>**E THE INTERVENTION TYP**<br>|Existing evidence is from Nepal and<br>Cameroon, analysis would isolate<br>Cameroon data only and/or consider the<br>transferability of data from Nepal to<br>Cameroon setting<br>**E**<br>|
||Deep Learning system to<br>analyze digital chest<br>radiographs<br>for TB-related<br>abnormalities|Type: Health Service User targeted. (WHO<br>Class 2.10: “laboratory and diagnostics<br>imaging management”) with predictive<br>analytics component.|
|**STEP 3. ESTABLISH**<br>|**THE LEVEL OF COMPLEXI**<br>|**TY**<br>|
|**Intervention**|High complexity|Depending on operation and interface the<br>intervention may represent low complexity<br>to the user|
|**Outcomes**|High complexity|For immediate diagnostic accuracy there is<br>low complexity. Improved TB management<br>and increased health is high complexity|
|**Causal pathway**|High complexity|**Table continued...**|
**Table 10 Framework Example 2 – AI-enabled diagnostic (continued)**

|**FRAMEWORK**<br>**COMPONENT**|**SPECIFICATION**<br>**CONSIDERATIONS**|
|---|---|
|**STEP 4: ANALYTICAL**|**PRINCIPLES**|
|**Comparator**|Human radiologist with Xpert as<br>confirmatory|
|**Time horizon**|Lifetime<br>Additional, shorter time horizon<br>depending requirements of decision<br>maker|
|**Standing**|Patients, Radiologists, other<br>hospital staff, government;<br>general population|
|**Costs**|Equipment and introduction<br>costs of an DL system; cost<br>offsets of improved targeting<br>and reduction in TB care|
|**Benefits**|Increase capacity to aid TB<br>diagnosis; Reduce the number of<br>follow-on tests|
|**Predictive analytics**|The predictive analytics<br>component is a key feature of<br>the intervention<br>ROC curves from initial evaluation<br>provides baseline accuracy estimates<br>compared to the human radiologist +<br>Xpert (the established comparator)|
|**COMPONENT**|**APPROACH**|
|**STEP 5: VALUE PROP**<br>|**OSITION**<br>|
|**_Health value_**|Intermediate impacts on improved TB treatment outcomes; identify any<br>impact on reduction in mortality and morbidity to develop utility measure|
|**_Process value_**|Improved diagnostic accuracy, and patient flow through diagnostic and<br>treatment pathway|
|**_Value of data_**|Generate data on localized TB prevalence|
|**_Prediction value_**|The unique prediction value can be assessed using ROC curve approach|
|**_Cross-sectorial value_**|Limited immediate cross sectorial impacts including labor supply and<br>worker productivity, however any impact on social sector through<br>improved TB outcomes should be considered|
|**Uncertainty**|One-way parameter sensitivity analysis; scenario analysis, probabilistic<br>sensitivity analysis depending on information/analytical resources|
|**Distribution**|Represent impact distribution by income and socioeconomic status|
# 7. Implications for Research and Policy

Any methodological guidance is only as good as the extent to which it improves the information available to decision makers. As a global public good, it is envisioned that investors, product developers and researchers will adopt and utilize this framework as a thought aid when planning, conducting and interpreting economic evaluations for DHIs, building on the principle-based concepts of the International Decision Support Initiative Reference Case for Economic Evaluation [2].

The framework is intended to integrate with existing guidance, and the ongoing application of the framework into specific evaluations of digital health interventions will provide methodological consistency and further opportunity to refine the framework specifications. The WHO Guide to Monitoring and Evaluating DHIs [21] notes the role of economic evaluation within the stages in the maturity life-cycle of an intervention. Within the life cycle, an intervention can be at pre-prototype/protype; pilot; demonstration; scale-up; or integration and sustainability. It is expected that this framework will be used in conducting economic evaluation predominantly at the demonstration and scale-up phases and following pilots. However, economic evaluation can be a useful tool for informing ongoing investment and modifications to established DHIs within the integrations/sustainability phase.

The framework is not developed as a “rule book” to constrain economic evaluation practice or create an unachievable standard to which economic evaluations of DHIs should be conducted. Nor is it a “textbook”, seeking to replicate the extensive field of methodological practice in economic evaluation. It is intended that the framework can be applied to evaluations conducted across a range of resource, time, and evidence constraints. As the DHI landscape and evidence base is rapidly evolving, methodological innovation in the economic assessment of DHIs will be necessary and it is expected that the framework will enable the development of a prioritized methodological research agenda. The overarching concept when applying the framework is to “comply or justify” – that is, users should attempt to adhere to the steps and principles of the framework, but where this is not feasible or appropriate, should provide justification for the methodological choice. In this way, the framework will facilitate methodological transparency and improve the overall usefulness of economic evaluations of DHIs.