Avoidable research waste in dermatology: what is the problem?
Authors: H.C. Williams
Journal: British Journal of Dermatology (2022)

Is there such a thing as research waste? Surely all research is potentially useful? Potentially yes, but the reality is that much clinical dermatological research is done without adequate prioritization or context to other research; it is often poorly designed, is open to publication bias and is poorly reported. This editorial documents some examples of avoidable research waste in dermatology and a view on why it occurs. In 2009, Chalmers and Glasziou wrote a powerful viewpoint in The Lancet entitled ‘Avoidable waste in the production and reporting of research evidence’.1 Although concepts such as under-reporting of ‘negative’ studies or incomplete reporting had been known for many years, it was Chalmers and Glasziou who systematically put it all together into a conceptual framework that followed the typical stages of a research study (Figure 1). Importantly, they pointed out that research waste is cumulative – if for example around 50% is wasted at stages 2, 3 and 4, then the total research waste loss would be around 85%, representing loss of benefit from tens of billions of dollars of investment in research because of correctable problems. In a subsequent essay in 2018, the same authors documented key milestones of medical research waste.2 They refer to the 1994 wake-up call by Altman, who suggested that ‘huge sums of money are spent annually on research that is seriously flawed through the use of inappropriate designs, unrepresentative samples, small samples, incorrect methods of analysis, and faulty interpretation’.3 Glasziou and Chalmers go on to describe key initiatives such as the EQUATOR reporting guidelines collection, the AllTrials campaign (all trials registered, all trials reported), conferences dedicated to understanding and reducing research waste, and how some funders such as the National Institute for Health Research (NIHR) have responded to the challenge. Despite such laudable efforts, they conclude that research waste is still a scandal. Although little has been published on the extent and nature of research waste in dermatology, it is clear that the problem is rife. Working through the stages suggested in Figure 1, it is clear that many clinical studies have not been prioritized in any way by consulting with clinicians, patients and other stakeholders. Some curiosity-based research is fine, but it may not address the needs of patients in a system with finite funding and human resources. For example, outcome measures for many dermatological diseases are a mess, with a profusion of unvalidated measures of unclear clinical interpretability. In 2019, Schmitt et al. assessed concordance between efficacy outcomes in a random sample of 10 Cochrane Skin systematic reviews and the 220 included trials.4 Of the 60 outcomes the reviews sought, 17 (28%) were not reported in any trial. For 11 of 23 (48%) primary review outcomes, meta-analysis was impossible, because trial outcomes were absent or unclear – a clear call for core outcome sets to be used in all studies so that they can be combined in a meaningful way. Moving onto stage 2, many dermatology studies fail to be designed with reference to existing evidence. A glimpse at most Cochrane Skin reviews will show how investigators across the world seem to repeat the same old studies over and over again instead of pausing and taking stock and either doing one large well-designed definitive study or doing something else. Industry-led studies are not immune – an analysis of pimecrolimus trials in 2016 showed that 25 were placebo controlled with just one comparison against a mild topical corticosteroid – the obvious active comparator to inform clinical practice.5 In a related commentary, Flohr and Weidinger suggested that such studies are just the tip of the iceberg.6 Risk-of-bias tables in Cochrane reviews for skin conditions often reveal serious concerns about basic design issues such as not concealing the randomization sequence from investigators or failure to analyse all those who were randomized in an intention-to-treat analysis. Yet the trials keep coming in – often so poorly reported that it is impossible to make a judgement on study quality. Despite my enthusiasm for systematic reviews, it is with regret that I now think they have spun completely out of control and have become a ‘sausage machine’.7 Around 90% of dermatology systematic reviews are of poor quality.8 Redundant duplication is also rife – there are now over 20 network meta-analyses in the field of systemic treatments for psoriasis – is that a good use of resources? For stage 3 in the figure, it has been estimated that 50% of studies are never published in full – regardless of funding source or country. Take the example of topical imiquimod for viral warts. Two large pivotal trials of 702 patients completed in 2006 showed no benefit for imiquimod for warts. Yet these studies were missed in three subsequent key reviews. Why? Because they were never published and remain unpublished to this day.9 With respect to stage 4 in Figure 1, it is clear that many important clinical outcomes that have been collected in dermatological clinical studies never see the light of day. Nankervis et al.10 found that among 190 identifiable randomized controlled trials of eczema, only 18 out of 109 trials (17%) had been registered ‘properly’ in terms of submitting the registration before the trial end date and nominating a primary outcome, and only five out of 109 provided enough information to be confident that the outcomes reported in the published trial were consistent with the original registration. A notable example of such selective reporting outcome bias is the Multicenter Selective Lymphadenectomy Trial – one of the largest and longest trials of melanoma treatment in dermatology – which, despite publication in the New England Journal of Medicine, failed to publish its registered primary outcome of overall mortality. Subsequent analysis showed no survival benefit.11 Other forms of publication waste also occur in clinical dermatology research, including covert duplicate publications and the use of spin that distorts the interpretation of results.12 These include lack of researcher training; failure of funders to identify, prioritize and commission research; university rewards and pressure to publish resulting in publication addiction;13 journal editors and publishers failing to ensure adequate reporting; vested financial interests and a general lack of awareness – especially with research users. The late Alessandro Liberati wrote about the ‘the butterfly behaviour of researchers, moving onto the next flower well before the previous one has been fully exploited’.14 One thing is clear – it is not just a case of one rotten apple but a system failure that belongs to all stakeholders including researchers, funders, research users and patients. The losers in research waste are ultimately always the patients and the public. The purpose of this editorial is not to condemn well-intentioned efforts but to raise awareness that there is such a thing as research waste in dermatology, that it is quite common and that it will continue and even accelerate unless stakeholders do something about it. Dermatological research waste is everyone’s business. In the next editorial, I will offer some suggestions of what can be done, by whom and how. Some of the content of these editorials has been presented at various previous meetings including the British Society for Investigative Dermatology Annual Meeting in Dundee on 4 April 2016; the Sven Helleström memorial lecture, Karolinska, Sweden, 2017; the Arthur Rook Oration – British Association of Dermatologists, July 2018 and the Royal College of Physicians Advanced Medicine Croonian Lecture, April 2021. Hywel Williams: Conceptualization (equal); Data curation (equal); Investigation (equal); Methodology (equal); Project administration (equal).