Deriving a practical framework for the evaluation of health apps
Authors: Philip Henson, Gary David, Karen Albright, John Torous
Journal: The Lancet Digital Health (2019)

The rise of digital health, fuelled by access to smartphones and connectivity to apps, has resulted in a plethora of mobile technology-based innovations. The US Food and Drug Administration (FDA) estimates that in 2018, 1·7 billion people downloaded mobile health apps.1Mobile Medical ApplicationsUS Food and Drug Administration.https://www.fda.gov/medicaldevices/digitalhealth/mobilemedicalapplications/default.htmDate accessed: December 11, 2018Google Scholar With over 325 000 to choose from across all health domains2Bates DW Landman A Levine DM Health apps and health policy: what is needed?.JAMA. 2018; 320: 1975-1976Crossref PubMed Scopus (60) Google Scholar and many updating as frequently as every week, finding and keeping up with useful mental health apps is a challenge. For patients and clinicians, picking the most suitable apps is difficult because of evolving evidence, emerging privacy risks, usability concerns, and the fact that apps constantly update and change. To help guide selection, people typically rely on the use of star rating systems and user reviews in app stores, despite strong evidence that such evaluation methods are misleading.3Singh K Drouin K Newmark LP et al.Many mobile health apps target high-need, high-cost populations, but gaps remain.Health Affairs. 2016; 35: 2310-2318Crossref Scopus (127) Google Scholar The FDA has realised the diminished value of traditional evaluation approaches and the challenge of regulating apps. To solve this problem, the FDA has begun piloting a novel certification programme, Pre-Cert, that shifts the scrutiny from the apps themselves to the developers.4Digital Health Software Precertification (Pre-Cert) ProgramUS Food and Drug Administration.https://www.fda.gov/MedicalDevices/DigitalHealth/UCM567265Date accessed: December 11, 2018Google Scholar Despite this effort, the programme is nascent, with just nine companies partaking. The UK National Health Service has released a second version of its App Library as another solution. In this void, numerous app evaluation frameworks have emerged. Although some frameworks have been published in peer-reviewed journals, many live in the grey literature. A systematic review5Moshi MR Tooher R Merlin T Suitability of current evaluation frameworks for use in the health technology assessment of mobile medical applications: a systematic review.Int J Technol Assess Health Care. 2018; 34: 464-475Crossref PubMed Scopus (31) Google Scholar examined existing evaluation frameworks for mobile medical apps, identifying 45 unique frameworks. Not surprisingly, the study found gaps in each framework.5Moshi MR Tooher R Merlin T Suitability of current evaluation frameworks for use in the health technology assessment of mobile medical applications: a systematic review.Int J Technol Assess Health Care. 2018; 34: 464-475Crossref PubMed Scopus (31) Google Scholar Given the obvious need for a more comprehensive evaluation framework, a novel approach is necessary. We have reframed the app evaluation process away from picking the best-rated app and towards making an informed decision on the basis of clinically-relevant criteria. Just as no single best antidepressant or therapy exists, no single best app exists to treat all patients or mental illnesses. Clinical presentation, patient preferences, technology literacy, accessibility, and treatment goals are all important factors that determine the best course of care for any patient. Such factors must be considered when deciding what the most suitable app might be for a patient in any given situation. This challenge has stimulated interest and new efforts from organisations like the FDA and the European National Institute for Health and Care Excellence. In late 2018, a group of international authors representing diverse stakeholders (including patients, clinicians, researchers, insurance organisations, technology companies, and US National Institute of Mental Health programme officers) crafted a consensus statement around standards for mental health apps.6Torous J Andersson G Bertagnoli A et al.Towards a consensus around standards for smartphone apps and digital mental health.World Psychiatry. 2019; 18: 1-2Crossref PubMed Scopus (159) Google Scholar Similar to the American Psychiatric Association (APA) App Evaluation Framework,7Torous JB Chan SR Gipson SY-MT et al.A hierarchical framework for evaluation and informed decision making regarding smartphone apps for clinical care.Psychiatr Serv. 2018; 69: 498-500Crossref PubMed Scopus (97) Google Scholar the consensus statement includes four levels, which are data safety and privacy, app effectiveness, user experience and adherence, and data integration. The goal of the consensus statement is not to evaluate any specific app, but rather to offer guidance for informed app decision making. We harmonised the 961 app evaluation questions included in the 45 frameworks that were identified in the systematic review around this consensus statement. We adopted a broad stakeholder analysis approach,8Brugha R Varvasovszky Z Stakeholder analysis: a review.Health Policy Plan. 2000; 15: 239-246Crossref PubMed Google Scholar involving integration of both clinical and patient perspectives to create an app evaluation structure. We followed a six-step process to yield a harmonised evaluation framework. This process included gathering data on all existing frameworks for mobile medical applications; breaking down each framework into discrete evaluation questions (eg, a framework assessing credibility might ask whether clinicians were involved in designing the tool); removing duplicate, redundant, and non-relevant questions; mapping remaining questions to APA priority levels and subcategories; selecting representative questions that could be used in discussions between clinicians and patients; and sharing the final document with members of our patient advisory panel for feedback and verification. Because frameworks are constantly changing, emerging, and being phased out no list can be complete. However, we did assemble a list of frameworks for evaluating mobile medical apps from the systematic review5Moshi MR Tooher R Merlin T Suitability of current evaluation frameworks for use in the health technology assessment of mobile medical applications: a systematic review.Int J Technol Assess Health Care. 2018; 34: 464-475Crossref PubMed Scopus (31) Google Scholar and supplemented them with additional frameworks we identified. The final list of frameworks assembled is presented in the appendix. After removal of duplicate, redundant, and non-relevant questions (n=604), the 357 remaining questions were each analysed for face and construct validity9Holden RR Face validity.in: Weiner IB Craighead WE The Corsini encyclopedia of psychology. Wiley, Hoboken, NJ2010: 637-638Crossref Google Scholar and mapped onto five priority levels, reflected in the APA app evaluation framework and the consensus statement. Within priority levels, subcategories were identified by further clustering questions. The process was modelled as a kind of qualitative factor analysis, in which all four authors, including one clinician, examined and reached consensus regarding how the questions loaded into different factors, or categories.10Mirkin BG Additive clustering and qualitative factor analysis: methods for similarity matrices.J Classif. 1987; 4: 7-31Crossref Scopus (48) Google Scholar For example, four subcategories emerged that reflected the themes of the 135 questions within the level ease of use. The final list of mapped 357 questions was then sent to two members of our patient advisory panel for feedback and verification, a process known as member checking.11Creswell JW Research design: qualitative, quantitative, and mixed methods approaches. SAGE Publications, Thousand Oaks, CA2014Google Scholar Two patients then reviewed the entire data set and agreed on the derived subcategories and in their feedback suggested the selection of new questions, often similar to initially selected ones, from the existing 357, to be incorporated into the derived framework as representative of each subcategory. The final framework (figure) represents the five priority levels, subcategories within them, and corresponding representative questions for each subcategory shown below the pyramid. Of the 357 questions, usability accounted for 135 (37%) questions. By contrast, data integration was the least represented category, with only 25 (7%) questions. Questions about data privacy and security were also less prevalent, with 49 questions (14%). The most prevalent subcategory was short-term usability, with 93 (69%) of 135 questions (appendix). As mental health apps have proliferated, choosing from among them has become increasingly challenging for patients and clinicians alike. Further, the sheer number of unique app evaluation frameworks can be overwhelming and confusing when trying to select the best evaluation method. Most frameworks emphasise ease of use and its subcategory of short-term usability, but do not devote enough attention to privacy, evidence, and clinical integration. We have shown that, despite the existence of many health app rating frameworks, proposing a more coherent framework is possible by reducing, simplifying, and presenting the frameworks' core elements. The benefit of our framework is that it draws upon the insights of previous efforts to offer a new, synergistic product that can be useful across diverse health conditions and stakeholder groups. A crucial feature of our framework is that no scoring is associated with any single question. The pyramid shape is to remind users that they should begin at the bottom and do not need to proceed if any single level does not meet their needs. For example, there is no need to assess evidence if privacy and security is not adequate. Questions (summarised in the appendix) are intended to encourage and facilitate dialogue between patients and clinicians. The answers to each question should be considered in relation to the patient's unique clinical situation, patient and clinician preferences, and treatment goals. For example, a patient using this framework might realise that an app is collecting or selling their geolocation data as part of the terms of use. One patient might accept those terms, while another might not. The goal is not to pass judgement or offer reductionistic scoring, but rather to ensure that users are aware of what the app is doing to be able to make an informed decision. As the clinical evidence for apps continues to evolve, we hope this framework will continue to offer future benefit by focusing on informed decision making of the state of app safety, evidence, usability, and integration, rather than anchoring to any single fact that will soon be past and dated. With no universal standard for evaluating health apps, many competing review systems have emerged that do not fully capture the range of important aspects to consider. We plan to use these results to undertake more formal evaluation efforts, develop self-assessment tools, and inform regulatory processes. JT reports grants from Otsuka, outside of the submitted work. All other authors declare no competing interests. Download .pdf (.18 MB) Help with pdf files Supplementary appendix