Addressing the disconnect between trial intention and design
Authors: Alison Howie, Monica Taljaard, Paxton Montgomery Moon, Natalie Nightingale, Merrick Zwarenstein
Journal: The Lancet Global Health (2021)

In 1967, Daniel Schwartz and Joseph Lellouch proposed that a randomised controlled trial can have either of two main intentions: to provide evidence to directly inform decision makers on choosing between alternative interventions in usual care settings (a so-called pragmatic intention), or to test a hypothesis about the mechanism of action of an intervention under laboratory-like conditions (a so-called explanatory intention).1Schwartz D Lellouch J Explanatory and pragmatic attitudes in therapeutical trials.J Chronic Dis. 1967; 8: 637-648Summary Full Text PDF Scopus (995) Google Scholar These authors proposed that trials can, and should, be designed to align broadly with their chosen intention; however, they reported that, often, the design decisions contradict this intention.1Schwartz D Lellouch J Explanatory and pragmatic attitudes in therapeutical trials.J Chronic Dis. 1967; 8: 637-648Summary Full Text PDF Scopus (995) Google Scholar Designing trials that support decision making requires a good match between a pragmatic intention and pragmatic design features if the trial is to yield results that are applicable. This match is particularly important in global health trials carried out in low-income and middle-income countries, where resources are scarce and evidence to support local decision making is in short supply. We carried out an analysis of recently published (between 2018 and 2019) primary trial reports in The Lancet Global Health (appendix pp 1–3) to characterise their apparent intention, their design features, and the alignment between intention and design. We identified key phrases in each report that were indicative of the trial's intention, which were categorised as pragmatic (decision-making support), explanatory (mechanism of action), or uncertain. Intention was classified as pragmatic if the report indicated an aim to provide information of importance to a decision maker or indicated that the findings are relevant to a decision maker. Trials that had no mention of decision making and that appeared to test a mechanism of action were classified as having an explanatory intention. Trials were labelled uncertain if they did not meet either set of criteria. We used the well-tested pragmatic explanatory continuum indicator summary (PRECIS-2) tool2Loudon K Treweek S Sullivan F Donnan P Thorpe KE Zwarenstein M The PRECIS-2 tool: designing trials that are fit for purpose.BMJ. 2015; 350h2147Crossref PubMed Scopus (674) Google Scholar to characterise the design features of each trial. PRECIS-2 includes nine design characteristics, each scored from one (very explanatory) to five (very pragmatic): eligibility, recruitment, setting, organisation, flexibility (delivery), flexibility (adherence), follow-up, primary outcome, and primary analysis.2Loudon K Treweek S Sullivan F Donnan P Thorpe KE Zwarenstein M The PRECIS-2 tool: designing trials that are fit for purpose.BMJ. 2015; 350h2147Crossref PubMed Scopus (674) Google Scholar For training purposes, six trials were selected at random and reviewed, discussed, and agreed on by all five authors together; thereafter, each trial was assessed, discussed, and agreed on by two raters who independently assessed each trial and participated in discussions to reach agreement. When insufficient information was provided to score a design characteristic, the rating was left blank. Trials were classified as having a mostly pragmatic design if the majority of their assessable design characteristics scored four or five. We then compared each trial's overall design with its intention and calculated the number of trials whose designs aligned with their intention. 38 eligible primary reports of randomised controlled trials were identified through hand searching The Lancet Global Health (regular editions) and were included in the analysis. 19 (50%) were judged as having a pragmatic intention, six (16%) explanatory, and 13 (34%) uncertain. 19 (50%) of the 38 trials had mostly pragmatic design characteristics, and 19 (50%) had mostly explanatory characteristics. Of the 25 trials in which the intention was clear, 13 (52%) had design features that aligned with their intention and 12 (48%) had design features that did not align. Disagreement occurred in both directions (eg, trials with a pragmatic intention had mostly explanatory designs, and trials with an explanatory intention had mostly pragmatic designs). Although 19 (50%) of the 38 trials in The Lancet Global Health had a pragmatic intention, ten (53%) of these 19 trials chose design characteristics that failed to align with this intention. Two (33%) of the six trials with an explanatory intention also failed to design their trial in accordance with this intention. This misalignment accords with Schwartz and Lellouch's original assertion, and causes substantial problems in decision making and contributes to research waste. Trials with a pragmatic intention but that are designed with mostly explanatory characteristics might mislead decision makers to interpret their findings as being ready to implement into routine practice in a real-world setting, with potential for wasted resources. Although less common, the findings from trials that had an apparent explanatory intention but mostly pragmatic design choices might be ignored by decision makers who are misled by the absence of explicit mention of the applicability of the trial results. We suspect that the failure to consciously consider the intention of a trial (either decision-making support or mechanism testing) and poor reporting of this intention, even when it is considered, is the likely cause of these misalignments. A reviewer suggested that intentions expressed upfront are likely to differ from those expressed through recommendations. We agree with this suggestion, and in future retrospective analyses, the classification of intention (as declared in the introduction) should be separate from that of implications (as declared in the discussion). Although we attempted to classify intention based solely on information provided in the introduction, the absence of explicit statements about trial intent forced us, in many instances, to rely on information provided in the discussion. Moreover, our decision to dichotomise trial design as mostly pragmatic or mostly explanatory might raise questions; however, this decision was purely for analytic purposes. We strongly support Schwartz and Lellouch's message that each design characteristic lies on a continuum between explanatory and pragmatic,1Schwartz D Lellouch J Explanatory and pragmatic attitudes in therapeutical trials.J Chronic Dis. 1967; 8: 637-648Summary Full Text PDF Scopus (995) Google Scholar and should be mindfully selected to match the intention of the trial and to fit the limitations and particulars of the context in which the trial is carried out. We recommend that authors include an explicit statement describing their intention and a reference to a PRECIS-2 table or wheel in manuscripts to improve the match between intention and design. Stating a trial's intention upfront (in the abstract and introduction) ensures that readers understand the exact purpose for which the trial was designed, and guides the use and wider application of the findings. The term pragmatic should be reserved for trials that are both carried out in real-world contexts and that make explicit the implications of their findings to relevant decision makers. Conversely, the term explanatory should be reserved for trials that are carried out under tightly controlled, laboratory-like settings and that aim to test a hypothesis about a mechanism of action. With the increasing appreciation for trials with real-world applicability, editors might wish to favour trials with both pragmatic intentions and pragmatic designs, and guide their reviewers and authors to avoid over-reach in making real-world recommendations from trials that are mostly explanatory in design. We declare no competing interests. The study had no funding source. Download .pdf (.19 MB) Help with pdf files Supplementary appendix