Why is measuring the effects of information technology on medication errors so difficult?
Authors: Johanna I Westbrook, Valentina Lichtner
Journal: The Lancet Digital Health (2019)

“Boeing had underestimated the effect that a malfunction of new automated software in the aircraft could have on the environment in the cockpit” reported The New York Times on Sept 26, 2019. In October, 2018, and March, 2019, two Boeing 737 MAX airplanes crashed, killing everyone on board. Worldwide, all MAX airplanes have been grounded while investigations continue on what caused the crashes.1Johnston P Harris R The Boeing 737 MAX saga: lessons for software organizations.Software Quality Professional. 2019; 21: 4-12Google Scholar The US National Transport Safety Board, highlighted the effects that a large number of concurrent alerts firing in response to the system malfunction had on pilots' response, and opined that “[…] the company had not considered the chaos that ensued inside the cockpit”. Identifying how a technology contributes to errors is a complex endeavour. It is easy to establish the role of the system when the fault can be isolated to single components, such as an improperly manufactured part or a faulty software update. It is much more difficult when the problem lies in the interaction between humans and technology or in the wider sociotechnical system. This difficulty is perhaps a reason why Google has removed human intervention in its autonomous cars. Automation and information technology (IT) have been integrated in health care to improve patient safety. Technology targeted to the problem of medication errors in hospitals is potentially transformative in reducing adverse drug events, increasing appropriate and effective prescribing and administration of medications, and reducing health-care costs.2Westbrook JI Gospodarevskaya E Li L et al.Cost-effectiveness analysis of a hospital electronic medication management system.J Am Med Inform Assoc. 2015; 22: 784-793Crossref PubMed Scopus (20) Google Scholar, 3Westbrook JI Reckmann M Li L et al.Effects of two commercial electronic prescribing systems on prescribing error rates in hospital in-patients: a before and after study.PLoS Med. 2012; 9e1001164Crossref PubMed Scopus (148) Google Scholar Electronic prescribing systems (also known as computerised provider order entry systems) have been associated with substantial reductions in medication errors, but important questions remain regarding their effects and the mechanisms that contribute to these effects over time. These are adaptive sociotechnical implementations, often involving numerous people. In health care, not only technology but also sociotechnical systems need to be optimised. Establishing the outcomes of an IT implementation and its role in causing these outcomes presents serious challenges. Does the introduction of an electronic prescribing systems in a hospital reduce medication errors? Which errors are facilitated by the system? What are the mechanisms for these effects? As hospitals continue to refine these systems, add electronic decision support, and improve interfaces, the effects on safety are expected to increase. Yet, to date, long-term evaluations of this optimisation process are rare. Sarah Slight and colleagues4Slight SP Tolley CL Bates DW et al.Medication errors and adverse drug events in a UK hospital during the optimisation of electronic prescriptions: a prospective observational study.Lancet Digital Health. 2019; (published online Oct 31)https://doi.org/10.1016/S2589-7500(19)30158-XGoogle Scholar present a longitudinal investigation in a UK hospital to examine the association between an electronic prescribing system and the rate and type of medication errors over 22 months as serial changes were made to the system. The electronic prescribing system had been in place for several years and the researchers examined changes in error rates at four time points, across four wards. System changes included the introduction of new decision support, links to allow pharmacists and medical staff access to summary general-practice information, an enhanced interface, insulin prescribing, and access to outpatient information. Overall, no significant change in the primary medication error rates over the four periods was found, although certain types of errors appeared to reduce over time. The study illustrates the complexity of studying technology longitudinally, considering a range of contextual and work system-related changes, and trying to assess their relative effects. The methodological challenges of assessing the effect of electronic prescribing systems on errors are considerable. Challenges lie both in the definition of the IT intervention that is being evaluated (since the technology evolves over time) and in the research design and tools used for evaluation. A rigorous foundational research infrastructure—involving, for example, validated definitions of errors and harm associated with errors5Gates PJ Baysari MT Mumford V Raban MZ Westbrook JI Standardising the classification of harm associated with medication errors: the Harm Associated with Medication Error Classification (HAMEC).Drug Saf. 2019; 42: 931-939Crossref PubMed Scopus (18) Google Scholar—and an established audit processes to accurately and consistently identify and assess errors, including system-related errors, are needed.3Westbrook JI Reckmann M Li L et al.Effects of two commercial electronic prescribing systems on prescribing error rates in hospital in-patients: a before and after study.PLoS Med. 2012; 9e1001164Crossref PubMed Scopus (148) Google Scholar Assessing when an error ends and the next begins is also an important question, in the effort to better understand the dynamics of clusters of errors in the medication process during a patient's hospitalisation.6Franklin BD Medication errors: do they occur in isolation?.BMJ Qual Saf. 2014; 23: e1Crossref PubMed Scopus (8) Google Scholar Slight and colleague's study contributes new information about clusters of errors and hypotheses about how these might be associated with specific IT design features. Studies need a baseline against which to assess change. To have confidence in attributing causation, there must be sufficient sample size and appropriate controls—which are difficult in natural settings. To account for the development (and perhaps optimisation) of the IT system over time and to account for users' adjustments to workflows and workarounds,7Baysari MT Hardie R-A Lake R et al.Longitudinal study of user experiences of a CPOE system in a pediatric hospital.Int J Med Inform. 2018; 109: 5-14Crossref PubMed Scopus (24) Google Scholar these studies must also be longitudinal and multisite. Ultimately, the aim is to inform improvement and system redesign. At a local level, studies must be able to generate a high level of detail about the effects of electronic prescribing systems on different error types and identify potential contributing factors to allow action to be taken. At the policy level, comparative evaluations between systems8Holmgren AJ Adler-Milstein J McCullough J Are all certified EHRs created equal? Assessing the relationship between EHR vendor and hospital meaningful use performance.J Am Med Inform Assoc. 2018; 25: 654-660Crossref PubMed Scopus (30) Google Scholar and information about how contextual factors affect system effectiveness and safety are needed.9Westbrook JI Li L Shah S et al.A cross-country time and motion study to measure the impact of electronic medication management systems on the work of hospital pharmacists in Australia and England.Int J Med Inform. 2019; 129: 253-259Crossref PubMed Scopus (11) Google Scholar A strength of Slight and colleagues' study4Slight SP Tolley CL Bates DW et al.Medication errors and adverse drug events in a UK hospital during the optimisation of electronic prescriptions: a prospective observational study.Lancet Digital Health. 2019; (published online Oct 31)https://doi.org/10.1016/S2589-7500(19)30158-XGoogle Scholar was their attempt to investigate the longitudinal effects of the electronic prescribing system on medication error rates, which is rarely undertaken. However, the changes in their IT system varied considerably in their potential scope to affect decision making and overall error rates in each of the four periods examined. Substantial system and work-practice changes occurred in some periods, whereas such changes were minimal in other periods. Attributing resultant error rates to specific optimisation strategies adopted or—as the authors point out—to “the cumulative effect of system optimisation”4Slight SP Tolley CL Bates DW et al.Medication errors and adverse drug events in a UK hospital during the optimisation of electronic prescriptions: a prospective observational study.Lancet Digital Health. 2019; (published online Oct 31)https://doi.org/10.1016/S2589-7500(19)30158-XGoogle Scholar is thus difficult, which limits the lessons that can be drawn from the findings for other settings. Linking specific optimisation strategies with their target outcome (eg, whether strategies targeting the prescription of clozapine or oxycodone affected error rates associated with these medications) would have been beneficial in understanding their effect. Future studies should first clearly identify the mechanisms by which optimisation strategies are expected to produce changes in prescribing behaviours and errors and then measure these outcomes. 2017 saw the launch of the WHO's Global Safety Challenge, Medication Without Harm, in recognition of the personal and financial costs of preventable harm and death as a result of medication errors, consuming over US$42 billions annually worldwide.10Donaldson LJ Kelley ET Dhingra-Kumar N Kieny M-P Sheikh A Medication Without Harm: WHO's Third Global Patient Safety Challenge.Lancet. 2017; 389: 1680-1681Summary Full Text Full Text PDF PubMed Scopus (181) Google Scholar Policy makers, researchers, and clinicians have shared responsibility to establish improved evidence of the effects of clinical information technologies on safety and how and where to intervene to make their use more effective in reducing errors and harm. We declare no competing interests. Medication errors and adverse drug events in a UK hospital during the optimisation of electronic prescriptions: a prospective observational studyAlthough the overall rates of primary medication errors per admission did not change, we found a reduction in some error types and a significant decrease in the rates of potential adverse drug events over a 2-year period, during which system optimisation occurred. Targeting some error types could have the added benefit of reducing others, which suggests that system optimisation could ultimately help improve patient safety and outcomes. Full-Text PDF Open Access