The Quality of Reporting of Kidney Research: A Challenge to JASN
Authors: Josephine P. Briggs
Journal: Journal of the American Society of Nephrology (2018)

More than 5 years ago, a "crisis" of confidence in the published literature erupted at the National Institutes of Health (NIH) and across the scientific community. It was precipitated, at least in part, by a publication from pharmaceutical industry investigators reporting major problems replicating preclinical research findings.1 The provocative article, "Believe it or not: How much can we rely on published data on potential drug targets?," reported that pharmaceutical company teams could not validate one half or more of published studies on potential therapeutic targets. Initially, there was no consensus about the validity of these claims, but slowly, more examples surfaced, and confirmatory analyses appeared.2,3 Soon, there was general acknowledgment of a problem.4 Clinical research did not come out fault free, but the main concern was reproducibility of preclinical animal and cell culture studies. The initial starting point was oncology, but concerns were soon identified in other fields, including cardiovascular disease, women's health, and neurotherapeutics. Nephrology largely escaped attention during that reproducibility "crisis," protected, ironically perhaps, by the sparse pharmaceutical investment in kidney disease and the rarity of formal replication studies. The NIH responded to this crisis with a series of meetings with stakeholder groups. A variety of problems were identified, including problems in the review of grants and academic incentives, and the NIH made changes in the grant review process. Attention quickly centered on the quality of reporting of research methods. Many replicable attempts foundered, because the original reports lacked the detail necessary for replication. Journals and journal editors were recognized as an important part of the solution, and meetings of journal editors convened. Out of this process grew the development of a number of guidelines and checklists, most prominently the ARRIVE checklist.5 The ARRIVE checklist is a key tool used in a report published in this issue of the Journal of the American Society of Nephrology (JASN), a systematic review assessing the quantity and reporting quality of research in nephrology.6 The study was performed by a group of researchers at the University of Munich. The corresponding author, Hans-Joachim Anders, is one of JASN's Associate Editors. Like all submitted manuscripts authored by one of our Editors, review of this manuscript was overseen by a Guest Editor and read critically by tough external peer reviewers. After a decision was made to publish, the entire current editorial team looked hard at the study and recognized that the analysis represents a challenge to us. There is good news and bad news in this report. The study undertakes assessment of both the quantity of nephrology research and the quality of reporting, the latter being, of course, where JASN must focus. The authors examined studies published in leading journals: they assessed randomized clinical trials in three kidney and two general medicine journals and preclinical studies in three kidney journals. It is important to emphasize that the differences between kidney journals were minimal. JASN does not look better (or worse) than other kidney journals. To other kidney journal editors, I say, we are all in this together. First, the good news. The authors found clear and steady improvement in the quality of reporting of clinical trials. Their analysis used the CONSORT criteria to examine trials from three time points: 1996, 2006, and 2016. Virtually all parameters showed improved reporting. The CONSORT checklist and the underlying rationale are now familiar to clinical trialists, and there seems to be clear buy in to these standards. Nevertheless, clinical trial reporting still shows a number of worrisome shortcomings. Although preregistration requirements were generally followed, prespecified outcome measures and analytic plans were often disregarded. Details on randomization and blinding were found in less than one half of the papers, studies were underpowered, and adverse event reporting was incomplete. Some of the missing details may have been included in separate trial design papers or supplemental material not examined by these investigators, but these elements generally belong in the primary report of a trial. It is noteworthy that the authors found that reporting standards were generally higher in the two leading general medical journals examined, New England Journal of Medicine and Lancet, than in kidney journals. Clearly, there is work that we must do. Now to the other set of worries revealed by this study. The authors used the ARRIVE guidelines to examine reporting standards in 135 preclinical studies published in three major kidney journals (JASN, Kidney International, and Nephrology, Dialysis and Transplantation) over the same timeframe; differences between journals were again minimal. Some aspects of reporting were satisfactory: animal ethical standards were generally understood and reported. The statements of objectives were considered satisfactory. Nevertheless, on the core issues identified in the "reproducibility" discussion, these studies collectively earn a failing grade. Critics of preclinical research methods generally concur that a recurrent problem is unthinking reliance on a P value of <0.05 and sample sizes that are too small and are not carefully calibrated to the actual data variability.7 The authors found that a method for sample size estimation was not described in any of the preclinical trials that they examined; the total number of animals used in each experimental group was not apparent in more than one half of the studies. Blinding, although more frequent in 2016 than in previous years, was reported in less than one third. Description of the overall statistical approach was generally graded as satisfactory, but details of statistical analysis, such as the unit of analysis for each dataset and criteria for data exclusion, were generally missing. We are not alone in finding implementation of ARRIVE guidelines challenging.8,9 A recurrent problem is the right approach to exploratory studies. It is accepted that the initial hypothesis-generating phase of research should enjoy greater flexibility than the hypothesis-testing phase as required, for example, for a preclinical validation study. Some work falls on a spectrum, and the boundary is not always clear. Making this distinction is challenging for authors, reviewers, and editors alike, and it is not a distinction that is embedded in the culture of basic research in nephrology. Clearly, a new paradigm is needed. At a practical level, the ARRIVE checklist is 20-items long. Preclinical studies published in JASN typically include a number of separate experimental series: six to eight is not unusual. Because a number of the questions in the ARRIVE checklist will have different answers for each experimental series, conscientious application of the ARRIVE becomes burdensome. The need for a simplified and prioritized list is recognized, and a number of journals are establishing modified checklists.10 Developing an approach that will work for our community will be a priority for JASN in the coming year. For some of the complex manuscripts that we publish, the best approach may be a simple table organizing reporting of the key features of statistical analysis and experimental design for each experimental series. Some of the steps taken last year by the new JASN editorial team also should help. We now require a structured abstract. We have eliminated word limits on the methods section and moved this section from the end of articles to a more prominent position after the introduction. Our information for authors emphasizes that the methods section is a critical element in the manuscript, complete enough to give the reader a fundamental understanding of the experiments performed and the core principles of the methodology. We are also paying much more attention to statistical analysis. We have expanded our Statistical Editorial Board to insure the range of statistical expertise needed for modern biomedical research and now obtain advice from a statistical reviewer on many reviewed manuscripts. In addition, we are pushing hard for data display methods that illustrate the variability of the data, eliminating bar graphs with error bars (so-called dynamite plots) and encouraging use of scatter plots and other types of graph that show individual data points. Display of individual data helps the reader understand the strength of an effect and creates transparency about replication units and sample size. The standards for reporting biomedical research evolve. We know that our readers and our authors want JASN to adhere to peer and editorial standards that will ensure the replicability and respect of the research that we publish. Patients with kidney disease deserve this. I urge our readers and our authors to study the analysis reported here6 and help us make wise decisions to address the shortcomings identified in this tough-minded and thorough analysis. Disclosures J.P.B. is a member of the Scientific Advisory Board at Verily, for which she receives consulting fees, and a member of the Board of the People Centered Research Foundation, an unpaid position.