Whose Conclusion is it Anyway?
Authors: Patrick C Waller
Journal: Journal of the Royal Society of Medicine (2006)

A former editor of the BMJ is reputed to have said something along the lines of ‘we know that half the papers we publish are rubbish—the problem is we don't know which half before we publish them’. Anyone who has some experience of peer review from both sides of the fence will know of the problems with this process. Authors are irritated by reviewers who clearly have not understood their work and have made inappropriate or impractical suggestions. Reviewers are frustrated when they point out important correctable flaws that slip through the editorial net into the finished version. And yet most of us accept that peer review is necessary, principally because there is no better alternative. In this article, I propose, for debate, a potential major change to the peer review process.

Central to my argument is that what matters most about any scientific paper are the conclusions drawn from it. Ultimately it is this part of the paper which will be cited and may influence practice, future research and, through the media, public perceptions. Therefore, the main aim of the review process should be to ensure that the published conclusions are reasonable and supported by what goes before them. Over the years, many papers have been published with conclusions that are unsustainable on the basis of the evidence presented. In some cases these have been pointed out during peer review or in the correspondence columns. However, the latter does not undo the damage because it is the paper rather than the rebuttal that generally receives preference. A critical editorial or commentary published simultaneously is better but not the ultimate solution to this problem.

Given these problems, it is reasonable to wonder whether authors are the best people to draw such influential conclusions from their study. Are they likely to be objective, unbiased and able to take a broad view of their own work in context or do they have a vested interest in making the most of it? If not the authors, who else could perform this task? Perhaps if the peer reviewers had to write the conclusions they would, at least, have to make a serious attempt to understand the work. At the moment, authors write the conclusions, peer reviewers may challenge them, appropriately or not and successfully or not. Why could this order not be turned on its head? And if it was, might not the whole process be better because the conclusions would be more objective and because reviewers would be forced to do their job more carefully?

The principal sections of the paper that would need to be different are the abstract and the discussion. Assuming that the conclusions are normally contained within the latter, they would be shifted into a separate section at the end headed ‘Conclusions of the peer reviewers’. If there are to be boxes with key messages or ‘what this study adds’, these could also be drawn up by the reviewers. In the abstract and discussion authors would be barred from drawing conclusions and it might be useful to define, through standard headings, what the latter could legitimately contain—e.g. strengths, limitations, comparisons with previous studies. Ideally, the conclusions would be joint work of at least two to three people (perhaps two reviewers and an editor). Before publication the authors would have a right to comment and propose amendments (which may or may not be accepted by the reviewers). If, ultimately, they disagreed with the conclusions drawn, they would have a right of reply (or could, of course, withdraw the paper from the journal).

The biggest potential drawback could be that such a process might make it more difficult to find willing reviewers than at present (and for specialist journals that is already a frequent problem). But this would not necessarily be the case—perhaps the process would be more attractive to reviewers who are disillusioned because their comments are sometimes ignored. A further potential difficulty is that authors would no longer ‘own’ the whole paper but it would be clearly defined what was not their responsibility.

Editors like public debate about conclusions, and this change might reduce that by moving some of the debate forwards prior to publication. I would argue that editors should be prepared to regard this as an acceptable price for an increase in the quality of their publication. Debate is essential but would ultimately be better from a more solid starting point.

I am not arguing that such a major change should be introduced forthwith. It could turn out to be impractical, lacking in the suggested benefits or have major drawbacks I have not identified. However, perhaps it is worth a pilot study?