D e S pin: a prototype system for detecting spin in biomedical publications (2020)

Abstract: Improving the quality of medical research reporting is crucial to reduce avoidable waste in research and to improve the quality of health care. Despite various initiatives aiming at improving research reporting – guidelines, checklists, authoring aids, peer review procedures, etc. – overinterpretation of research results, also known as spin, is still a serious issue in research reporting. In this paper, we propose a Natural Language Processing (NLP) system for detecting several types of spin in biomedical articles reporting randomized controlled trials (RCTs). We use a combination of rule-based and machine learning approaches to extract important information on trial design and to detect potential spin. The proposed spin detection system includes algorithms for text structure analysis, sentence classification, entity and relation extraction, semantic similarity assessment. Our algorithms achieved operational performance for the these tasks, F-measure ranging from 79,42 to 97.86% for different tasks. The most difficult task is extracting reported outcomes. Our tool is intended to be used as a semi-automated aid tool for assisting both authors and peer reviewers to detect potential spin. The tool incorporates a simple interface that allows to run the algorithms and visualize their output. It can also be used for manual annotation and correction of the errors in the outputs. The proposed tool is the first tool for spin detection. The tool and the annotated dataset are freely available.

## related work although there has been no attempt to automate spin detection in biomedical articles, a number of works addressed developing automated aid tools to assist authors and readers of scientific articles in performing various other tasks. some of these tools were tested and were shown to reduce the workload and improve the performance of human experts on the corresponding task. @xcite assessed the impact of a writing aid tool based on the consort statement @xcite on the completeness of reporting of rcts. the tools was developed for six domains of the methods section (trial design, randomization, blinding, participants, interventions, and outcomes) and consisted of reminders of the corresponding consort item(s), bullet points enumerating the key elements to report, and good reporting examples. the tool was assessed in an rct in which the participants were asked to write a methods section of an article based on a trial protocol, either using the aid tool ('intervention' group) or without using the tool ('control' group). the results of 41 participants showed that the mean global score for reporting completeness was higher with the use of the tool than without it. ## spin: definition and types we adopt the definition and classification of spin introduced by @xcite and @xcite , who divided instances of spin into several types and subtypes. we addressed the following types of spin: the primary end point of this trial was overall survival. <...> this trial showed a significantly increased r0 resection rate although it failed to demonstrate a survival benefit. in this example, the primary outcome ("overall survival"), the results for which were not favourable, is mentioned in the conclusion, but it is not reported in the first place and occurs within a concessive clause (starting by "although"). this way of reporting puts the focus on the other, favourable, outcome ("r0 resection rate"). the median pfs was 10.3 months in the xeliri and 9.3 months in the folfiri arm (@xmath0.78). conclusion: the xeliri regimen showed similar pfs compared to the folfiri regimen. the results for the outcome "median pfs" are not significant, which is often erroneously interpreted as a proof of similarity of the treatments. however, a non-significant result means that the null hypothesis of a difference could not be rejected, which is not equivalent to a demonstration of similarity of the treatments. this would require the rejection of the null hypothesis of a difference, or a substantial difference, in outcomes between treatments. both groups showed robust improvement in both symptoms and functioning. the goal of randomized controlled trials is to compare two treatments with regard to some outcomes. if the superiority of the experimental treatment over the control treatment was not shown, within-group comparisons (reporting the changes within a group of patients receiving a treatment, instead of comparing patients receiving different treatments) can be used to persuade the reader of beneficial effects of the experimental treatment. two concepts are vital for spin detection and play a key role in our algorithms: the primary end point was a difference @xmath1. trial results are commonly reported as an effect on the (primary) outcome foot_0 , along with the p-value. microcirculatory flow indices of small and medium vessels were significantly higher in the levosimendan group as compared to the control group (@xmath2.05). statistical significance levels of trial outcomes are vital for spin detection, as spin is commonly related to non-significant results for the primary outcome, or to selective reporting of significant outcomes only. ## algorithms spin is a complex notion and thus detecting spin cannot be seen as a binary classification problem. we believe that the most viable approach to spin detection is to assess each (sub)type of spin separately. we aimed at developing algorithms to extract and analyse pieces of information relevant to the addressed types of spin. the extracted information and its analysis, provided by our tool, can help human experts in making the conclusion on presence or absence of spin of the given (sub)type. detection of spin and related information is a complex task which cannot by fully automated. our system is designed as a semi-automated tool that finds potential instances of the addressed types of spin and extracts the supporting information that can help the user to make the final decision on the presence of spin. in this section, we present the algorithms currently included in the system, according to the types of spin that they are used to detect. as we aim at detecting spin in the results and conclusions sections of articles' abstracts, we first need an algorithm analyzing the given article to detect its abstract and the results and conclusions sections within the abstract. we will not mention this algorithm in the list of algorithms for each spin type to avoid repetition. if we talk about extracting some information from the abstract, it implies that the text structure analysis algorithm was applied. ## methods in this section, we briefly outline the methods used in our algorithms, the datasets used for evaluation, and the current performance of the algorithms. our approach is based on some previous works for the related tasks. as the details on development of the algorithms, annotating the data and testing different approaches are described in detail in the corresponding articles, we limit ourselves here to only a brief description of the best-performing method that we selected for each task. the methods we employ can be divided into two groups: machine learning, including deep learning, used for the core tasks for which we have sufficient training data, and rule-based methods, used for the simpler tasks or for tasks where we do not have enough data for machine learning. ## interface our prototype system allows the user to load a text (with or without annotations), run algorithms, visualize their output, correct, add or remove annotations. the expected input is an article reporting an rct in the text format, including the abstract. figure 1 shows the interface with an example of a processed text. the main items of the drop-down menu on the top of the page are annotations, allowing to visualize and manage the annotations, and algorithms, allowing to run the described algorithms to detect potential spin and the related information. the text fragments identified by the algorithms can be highlighted in the text. when running the algorithms, a report is generated that contains the extracted information and its analysis by the tool (e.g. a mismatch between the outcomes in the text and in the trial registry; absence of the declared primary outcome among the reported outcomes in the abstract). the report is saved into the metadata section of annotations menu, which can be accessed through the interface, and can be exported to a file via the generate report item of the algorithms menu. human experts can use this report to check the extracted information and the analysis performed by the tool, and to make a final decision on the presence/absence of a given type of spin. ## results and conclusions the current functionality, methods in use, annotated datasets and the best achieved results are outlined in in this paper, we presented a first prototype tool for assisting authors and reviewers to detect spin and related information in abstracts of articles reporting rcts. the employed algorithms show operational performance in complex semantic tasks, even with relatively low volume of available annotated data. we envisage two possible applications of our system: as an authoring aid or as peerreviewing tool. the authoring aid version can be further developed into an educational tool, explaining the notion of spin and its types to the user. possible directions for future work include: improving the implementation and interface (adding prompts for interaction with the user; facilitating installation process), algorithms (improving current performance, adding detection of new spin figure 1 : example of a processed text types), application (promoting the tool among the target audience; encouraging users to submit their manually annotated data, to be used to improve the algorithms), and optimization (parallel processing of multiple input text files). our system can be easily incorporated into other text processing tools. another interesting yet challenging direction for the future work is detecting spin/distorted reporting in texts belonging to scientific domains other than biomedicine. first of all, a qualitative study of spin is needed to define and classify spin in each scientific domain (similar to the work of @xcite and @xcite for clinical trials). to our best knowledge, there have been no attempts to conduct such a study for non-biomedical texts. it is therefore difficult to hypothesise whether spindetection algorithms developed for texts reporting clinical trials could be applicable for other domains. it appears that the definition and the types of spin are domain-specific (e.g. outcome-related types of spin, prevalent in the biomedical domain, would not be relevant in domains that do not use the notion of outcome). hence, we suppose that spindetection algorithms are domain-specific as well and cannot be applied to other domains.