Preprints versus peer-reviewed articles – Author's reply
Authors: B Ian Hutchins
Journal: The Lancet Global Health (2023)

Accustomed to seeing scientific discoveries presented in peer-reviewed articles, stakeholders have wondered how to weigh evidence reported in preprints. One way to measure the robustness of preprinted evidence is to compare data reported in a preprint with its published version. Accordingly, we matched preprints to keywords related to four epidemiological measurements: COVID-19 incidence, infection fatality rate, case fatality rate, and R0.1Nelson L Ye H Schwenn A Lee S Arabi S Hutchins BI Robustness of evidence reported in preprints during peer review.Lancet Glob Health. 2022; 10: e1684-e1687Summary Full Text Full Text PDF PubMed Scopus (7) Google Scholar Preprinted data were highly stable compared with their published versions. We concluded that our findings, alongside others', support the use of evidence reported in preprints in decision making, as one component of the biomedical literature. Kent Anderson conducts a follow-up analysis, poses an alternative interpretation, and raises a concern about data quality. The concern noted that while we reported 100 matched preprints, there are only 96 unique article identifiers. As we stated, we analysed article–keyword matches to ensure that data were central enough to be mentioned in the title or abstract. It is possible for articles to match more than one keyword, and this did occur at a low frequency (<5%). As we noted in the Methods, these multiple matches were analysed separately. This design choice facilitated comparisons of different types of epidemiological estimate, but entailed that the number of unique article identifiers is slightly lower than the number of article–keyword matches. Anderson posits that some of the stability that we noted might be due to authors posting manuscripts after acceptance, potentially minimising differences between preprint and published versions by definition. Based on this comparison of preprint posting and article acceptance dates, up to around 20% of preprints could in principle be accepted versions. As we noted in our Discussion,1Nelson L Ye H Schwenn A Lee S Arabi S Hutchins BI Robustness of evidence reported in preprints during peer review.Lancet Glob Health. 2022; 10: e1684-e1687Summary Full Text Full Text PDF PubMed Scopus (7) Google Scholar our study was not a measurement of the marginal effects of peer review on manuscripts, but rather a measurement of the expected change from preprint to publication version, because it is not knowable how much peer review (either for gatekeeping at journals or for improvement from informal peer feedback) has occurred before preprint posting. Nor is it knowable how much observed change was driven by reviewer feedback versus being author-initiated. This follow-up analysis reinforces our findings of high stability of data in preprints through to the published version, and proposes one small but notable potential contributor to that observation. Taken together, these results reinforce our interpretation that the use of evidence presented in preprints in decision making is warranted, as one valuable component of the biomedical literature. I declare no competing interests. Robustness of evidence reported in preprints during peer reviewScientists have expressed concern that the risk of flawed decision making is increased through the use of preprint data that might change after undergoing peer review. This Health Policy paper assesses how COVID-19 evidence presented in preprints changes after review. We quantified attrition dynamics of more than 1000 epidemiological estimates first reported in 100 preprints matched to their subsequent peer-reviewed journal publication. Point estimate values changed an average of 6% during review; the correlation between estimate values before and after review was high (0·99) and there was no systematic trend. Full-Text PDF Open Access