Evaluating the Impact of Organizational Changes in Health Care Delivery: Challenges in Study Design
Authors: Deborah Schrag
Journal: Journal of Clinical Oncology (2009)

The tenets of evidence-based medicine are by now so canonic that even organizational interventions to improve health outcomes require systematic evaluation to demonstrate their value. Of course, the most compelling strategy for establishing benefit is the randomized trial. In most cases, this is a welcome advance, one made in recognition of the reality that even interventions posing minimal health risks involve the use of scarce resources, the depletion of which can be measured in lost time, dollars, or opportunities. Behavioral and organizational interventions—such as installation of an exercise program or employment of patient navigators by a practice—typically target units larger than the individual. In this context, researchers are increasingly employing a design strategy known as the group randomized trial, also called the cluster randomized trial. Cluster randomized trials make sense when random assignment of individual patients is compromised because shared exposure may lead to contamination and thereby potential for resulting bias. Consider the example of an educational intervention carried out by infusion room chemotherapy nurses to foster adherence to antiemetic regimens. Randomization at the individual patient level might prompt providers with knowledge of the study intervention to provide their patients with additional education. Similarly, patients not exposed directly to the intervention might learn about it second hand from other study participants in the waiting room. Randomization at the level of the clinic mitigates these potential biases, but it creates other design challenges. In this issue of Journal of Clinical Oncology (JCO), Passalacqua et al describe the results of a cluster randomized trial designed to minimize anxiety and satisfy information needs of more than 3,000 patients with cancer treated at 38 Italian hospitals. Passalacqua et al developed an intervention to address the information needs of oncology inpatients in Italian hospitals. This intervention, known as the Point of Information and Support (PIS) system, was motivated by survey research evidence suggesting that information deficits were a source of considerable anxiety, which often led patients to adopt alternative unproven medical treatments. The PIS system involved creating a library with information materials and training hospital nurses to staff a resource room adjacent to the hospital ward. Each hospital was required to meet minimum staffing, opening hours, and publicity requirements but otherwise had considerable leeway in designing and implementing its own PIS system. This flexibility in design is pragmatic in that it allows for local variation in patterns of care delivery; however, the lack of standardization also creates a vulnerability that proved severe in this case. Six months after the PIS systems were integrated into the hospitals, more than 3,000 patients rated their satisfaction with information provided during their hospitalization according to the Hospital Anxiety and Depression Scale. Passalacqua et al found no differences in levels of anxiety, depression, or satisfaction between patients treated at the 21 hospitals that did and the 17 hospitals that did not implement a PIS system in the context of this study. JCO readers are likely to empathize with the frustration that these investigators must have felt when their intervention failed to improve outcomes. The main reason the study results were negative was that only 10 of the 21 hospitals assigned to develop a PIS system actually did so properly. This study failed to improve outcomes because the intervention could not be implemented or standardized across sites. The enormous amount of effort that went into the design of this study prompts us to question how this trial could have been designed differently to avoid such a disappointing result. What alternative strategies might have been pursued to optimize the value of the information gained? Passalacqua et al are to be congratulated for applying the same critical and systematic approach to evaluation of an organizational intervention that we apply to treatment trials. However, their study and its pitfalls highlight some important challenges confronted when evaluating organizational interventions. These issues merit consideration before the launching of a cluster randomized trial. Interventions vary enormously in the degrees of risk and cost that they create for patients and health care systems. In contrast to administering chemotherapy, which is high cost and high risk, placing a box of tissues in each examination room is a low-risk, low-cost intervention. The tissues are likely to be harmless to most patients, and even if rarely used, their relatively low cost means that the resources expended to keep the examination rooms stocked are unlikely to detract significantly from the ability of clinics to pursue other interventions of priority. No additional resources are required to maintain tissue boxes in good condition, and there are negligible adverse consequences if they go unused. This absurd example illustrates that some interventions are so innocuous that they do not merit investment of resources requisite for evaluation in the context of a trial. A logical first question to ask is whether the intervention under consideration is so similar to a box of tissues that it is reasonable to deploy it immediately and evaluate patterns of use downstream. In the case of the study by Passalacqua et al, if training clinical staff to attend to unmet patient psychological and information needs could have been accomplished without substantial investment of additional clinical resources, it is conceivable that the cost and risk involved could have JOURNAL OF CLINICAL ONCOLOGY E D I T O R I A L VOLUME 27 NUMBER 11 APRIL 1