Limitations of the new ISO standard for health and wellness apps
Authors: David Neal, Thomas Engelsma, Josephine Tan, Michael P Craven, Romaric Marcilly, Linda Peute, Tom Dening, Monique Jaspers, Rose-Marie Dröes
Journal: The Lancet Digital Health (2022)

Software apps for health and wellness are proliferating rapidly.1Levine DM Co Z Newmark LP et al.Design and testing of a mobile health application rating tool.NPJ Digital Medicine. 2020; 3: 74Crossref PubMed Scopus (20) Google Scholar Policy makers, health-care providers, and consumers can benefit from assessment and standardisation of these apps, to support decision making in a rapidly developing field. Recognising this unmet need, the International Organization for Standardization (ISO) published a standard in July, 2021, with the purpose of defining a framework for quality assessment and labelling of health apps.2International Organization for StandardizationHealth software—part 2: health and wellness apps—quality and reliability (ISO/TS 82304-2).https://www.iso.org/standard/78182.htmlDate: July, 2021Date accessed: September 15, 2021Google Scholar The framework defined by the standard comprises four subscales: healthy and safe, easy to use, secure data, and robust build, with corresponding questions and proposed evidence standards for those seeking to evaluate the apps in question. However, we fear that, in its current form, the standard could stigmatise some app users and worsen inequalities in access to digital health technologies. In particular, the proposed quality assessment method is insufficiently nuanced to be reliably applicable, taking into account the diverse characteristics of app users. ISO harmonises international standardisation in 165 countries. In the field of health and medicine, more than 1600 ISO standards have been published. The impact of ISO standards on health policy, clinical research, and practice is well illustrated by standard ISO13485 on medical devices, given that this is the basis of the US Food and Drug Administration's regulations on medical devices.3International Organization for StandardizationFDA plans to use ISO 13485 for medical devices regulation.https://www.iso.org/news/ref2318.htmlDate accessed: September 15, 2021Google Scholar The new standard for health and wellness apps might more directly affect clinical practice because it mandates quality labelling of these apps which are specifically targeted at the general public (analogous to energy certificates in the EU and UK). This labelling will affect all patients and clinicians working with such apps. We recognise that this new standard could help to improve the quality of health apps; however, we do not feel that the standard sufficiently incorporates the evaluation of user outcomes as opposed to design process, which is particularly problematic with respect to the proposed easy-to-use label. Focusing primarily on the design process rather than empirically demonstrated outcomes is an important limitation. A user-centred design process should increase the probability that an app is easy to use, but is no guarantee.4Diaz Baquero AA Dröes RM Perea Bartolomé MV et al.Methodological designs applied in the development of computer-based training programs for the cognitive rehabilitation in people with mild cognitive impairment (MCI) and mild dementia. Systematic review.J Clin Med. 2021; 101222Crossref PubMed Scopus (4) Google Scholar Evidence provided by app developers of good design process should be backed up by empirical evidence of user friendliness—ideally from independent expert-led evaluations—to justify an easy-to-use label. It is well established that ease of use is dependent on the context of use, as well as the goals and characteristics of the user.5International Organization for StandardizationErgonomics of human-system interaction—part 11: usability: definitions and concepts (ISO 9241-11:2018).https://www.iso.org/standard/63500.htmlDate: March, 2018Date accessed: September 15, 2021Google Scholar The standard attempts to account for the influence of user characteristics and context on ease of use by mandating assessment of the evidence that apps have been tested with intended users from the target population. This approach is insufficient to ensure reliable labelling because it assumes that users involved in testing are representative of all intended users. According to the standard, testing should include users with a particular condition, if the app is "geared towards…people with a specific health condition".2International Organization for StandardizationHealth software—part 2: health and wellness apps—quality and reliability (ISO/TS 82304-2).https://www.iso.org/standard/78182.htmlDate: July, 2021Date accessed: September 15, 2021Google Scholar An app designed to support individuals with type 2 diabetes in monitoring their diet should, therefore, be tested among people with type 2 diabetes. However, this group is highly heterogenous. Furthermore, a quarter of adults in the USA, and many more people globally, live with multiple comorbidities.6Boersma P Black LI Ward BW Prevalence of multiple chronic conditions among US adults, 2018.Prev Chronic Dis. 2020; 17200130Crossref Google Scholar A person with type 2 diabetes might be an older person (eg, aged ≥65 years) living with mild cognitive impairment, both of which are characteristics associated with specific requirements when using apps.7Wildenbos G Peute L Jaspers M Aging barriers influencing mobile health usability for older adults: a literature-based framework (MOLD-US).Int J Med Inform. 2018; 114: 66-75Crossref PubMed Scopus (128) Google Scholar If younger people with type 2 diabetes but without cognitive impairment were the only patient group included in design and testing of the app, then results will fail to represent a very large number of intended users. The lack of nuance, to account for the diversity of app users and user groups, is also evident with respect to the healthy and safe subscale of the framework. For example, an app designed to support weight loss by monitoring calorie consumption might promote healthy and safe eating for many people. However, weight loss is an issue that intersects mental health and wellbeing. The same app might be considered to pose serious health risks to users with eating disorders.8Honary M Bell BT Clinch S et al.Understanding the role of healthy eating and fitness mobile apps in the formation of maladaptive eating and exercise behaviors in young people.JMIR Mhealth Uhealth. 2019; 7e14239Crossref PubMed Scopus (19) Google Scholar Given the high morbidity associated with eating disorders, this occurrence would not be a trivial health and safety matter. Comorbidity, culture, gender, sexuality, language, and many other factors can be expected to modulate the extent to which apps are healthy and safe or easy to use for different users or user groups. Nevertheless, it is unclear how the representativeness of users involved in testing should be assessed using this standard, or how contextual information and limitations of testing should be communicated by the resulting quality label. Without such reports, labels could be unreliable or even misleading for users, particularly individuals living with multiple conditions or impairments, who have specific needs and might be under-represented in user testing. These people have the greatest need for robustly assessed and clearly labelled health technology; however, unreliable easy-to-use labels might increase stigma, reduce self-efficacy (ie, belief in their own ability to use technology), and result in slow adoption of technology. A common stigmatising misconception among older adults, as well as those around them, is that, as a group or as individuals, they are unable to use technology. There is a real risk that people encountering problems when using supposedly easy technologies will perpetuate existing stigma when, in reality, the label is not justified for that user or user group. Given the limitations of the ISO's proposed quality labels for health and wellness apps, we make three urgent recommendations. First, we strongly encourage our colleagues in research to investigate the validity and limitations of the proposed quality labels for diverse groups of users. Second, we recommend that the ISO revises the standard without delay. The most immediate solution to this problem would be for the ISO to add questions and evidence requirements around diversity and representativeness in user testing to the assessment framework, and corresponding information on the public-facing quality labels. This action would increase transparency, make the quality labels more nuanced and meaningful, and be a first step towards reducing the likelihood of stigmatising misconceptions arising. We also suggest that the ISO take into account the in-development ISO standard regarding cognitive accessibility, besides alternative approaches to assessing and labelling apps, such as the NHS Digital Technology Assessment Criteria.9Steel EJ Janeslätt G Drafting standards on cognitive accessibility: a global collaboration.Disab Rehabil Assist Technol. 2017; 12: 385-389Crossref PubMed Scopus (7) Google Scholar, 10National Health Service XDigital Technology Assessment Criteria for Health and Social Care (DTAC).https://www.nhsx.nhs.uk/key-tools-and-info/digital-technology-assessment-criteria-dtac/Date accessed: September 15, 2021Google Scholar In particular, the approach to the development of the in-development standard for cognitive accessibility seems to focus more on the nuanced needs of individuals and groups, to ensure that "products, systems, services, environments and facilities can be used by people from a population with the widest range of cognitive characteristics and abilities to achieve a specified goal in a specified context of use."9Steel EJ Janeslätt G Drafting standards on cognitive accessibility: a global collaboration.Disab Rehabil Assist Technol. 2017; 12: 385-389Crossref PubMed Scopus (7) Google Scholar Third, we recommend that policy makers, health-care providers, and consumers remain critical with respect to the proposed app quality labels, particularly in the context of users with multiple comorbidities or impairments. If all parties follow these recommendations, we hope that the result will be an appropriately nuanced standard, allowing the reliable assessment of the quality of health and wellness apps. MPC's role as Principal Research Fellow at the University of Nottingham is supported by the National Institute for Health Research (NIHR MindTech MedTech Co-operative), which is funded by the Department of Health and Social Care of the UK Government. DN's role is funded via the Dementia: Intersectoral Strategy for Training and Innovation Network for Current Technology Innovative Training Network, which is funded by the European Union's Horizon 2020 research and innovation program under the Marie Sklodowska-Curie actions (grant agreement number 813196). All other authors declare no competing interests. The following international experts in digital health technology and dementia are acknowledged as sharing the concerns raised in this Comment: Louise Nygård, Sophie Gaber, Ann-Helen Patomella, Sarah Wallcook, and Elin Jakobsson (Karolinska Institutet, Solna, Sweden); Frans Verhey, Marjolein de Vugt, Fania Dassen, Golnaz Atefi, and Pascale Heins (Maastricht University, Maastricht, Netherlands); Martin Orrell, Lesley Garcia, and Esther Gerritzen (University of Nottingham, Nottingham, UK); Lieve Van den Block and Fanny Monnet (Vrije Universiteit Brussel, Ixelles, Belgium); Manuel Franco Martin and Mauricio Molinari Ulate (University of Salamanca, Salamanca, Spain); Karin Dijkstra and Yvonne Kerkhof (Saxion University of Applied Sciences, Enschede, Netherlands); Beliz Budak and Simone Anna Felding (German Center for Neurodegenerative Disease, Bonn, Germany); Steven H Zarit (Pennsylvania State University, State College, PA, USA); Evelyn Finnema (University Medical Centre Groningen, Groningen, Netherlands); Maurice Mulvenna (Ulster University, Coleraine, UK); Teake Ettema (Amsterdam University Medical Centre, Amsterdam, Netherlands); Phil Joddrell (University of Sheffield, Sheffield, UK); Gianna Kohl (University College London, London, UK); Viktoria Hoel (University of Bremen, Bremen, Germany); Wei Qi Koh (National University of Ireland Galway, Galway, Ireland); Channah Osinga (University of Amsterdam, Amsterdam, Netherlands); Floor van den Berg (University of Groningen, Groningen, Netherlands); and Jean Georges (Alzheimer Europe Luxembourg, Luxembourg).