Detecting Health Advice in Medical Research Literature (2021)

Abstract: Health and medical researchers often give clinical and policy recommendations to inform health practice and public health policy. However, no current health information system supports the direct retrieval of health advice. This study fills the gap by developing and validating an NLP-based prediction model for identifying health advice in research publications. We annotated a corpus of 6,000 sentences extracted from structured abstracts in PubMed publications as ‘“strong advice”, “weak advice”, or “no advice”, and developed a BERT-based model that can predict, with a macro-averaged F1-score of 0.93, whether a sentence gives strong advice, weak advice, or not. The prediction model generalized well to sentences in both unstructured abstracts and discussion sections, where health advice normally appears. We also conducted a case study that applied this prediction model to retrieve specific health advice on COVID-19 treatments from LitCovid, a large COVID research literature portal, demonstrating the usefulness of retrieving health advice sentences as an advanced research literature navigation function for health researchers and the general public.

## introduction clinical practice and public health policies need to be guided by evidence presented in peer-reviewed medical literature, where researchers present their findings and discuss implications @xcite . sometimes, researchers would even give clinical and policy recommendations, defined as "health advice" in this study. however, whether to give health advice in medical research papers is a controversial issue. the opponents are concerned about the quality of health advice given in individual research papers in that a single paper lacks sufficient information for all evidence in real practice and there is limited manuscript space for full review of alternative choices @xcite . for example, some medical experts warned that a large proportion of health advice inferred from over-interpreted observational results was not fully supported by the study presented @xcite , although such health advice is frequently found in medical publications, even in highly influential journals @xcite . on the other hand, proponents of "actionable research" would like to encourage more efficient and effective transmission of science evidence into practice @xcite . giving health advice may also benefit the general public. some journal editors suggested "keeping the needs of less scientifically inclined readers in mind" @xcite . after all, if researchers themselves do not discuss the practical value of their findings, press officers and journalists might misinterpret the results and give exaggerated health advice in press releases and news articles @xcite . both arguments for and against giving health advice in individual studies indicate a strong need for identifying and accessing health advice, for either practical use or quality evaluation purpose. however, navigating the large volume of medical literature is a daunting task @xcite , and outdated information system has been a barrier to accessing health advice @xcite . the fast growth of medical and health literature further exacerbates the challenge @xcite . for example, the most recent covid-19 outbreak has brought an explosion of scientific papers about the disease. the strong need for understanding the fastgrowing scientific evidence has led to the creation of specialized data hubs and search platforms (e.g. @xcite @xcite . nevertheless, they have not been able to provide functions to support the direct retrieval of health advice. a new information service that allows for direct access to health advice would be able to reduce the information barrier. the core function would be a prediction model that can automatically identify health advice in science literature. however, past studies of health advice were limited to small-scale manual analyses (e.g. @xcite @xcite , although they were instrumental in defining health advice semantically and establishing snapshot views based on small data sets. due to the significant time and labor cost, machine learning and nlp-based methods are needed for fast and iterative analyses of health advice in a large volume of research publications, news articles, and online posts, not to mention tracking the diffusion of health advice across domains and over time. recent advance in natural language processing has resulted in a body of research on information extraction from medical literature (e.g. @xcite @xcite , however, automated detection of health advice has not been well explored. this study aims to develop a computational approach to automatically detect and categorize health advice. we developed and validated an nlpbased prediction model to automatically find health advice in pubmed literature. a total of 6,000 sentences were extracted from research papers, manually annotated, and then used to fine-tune a bertbased prediction model to predict whether a sentence contains "strong advice", "weak advice", or "no advice". to demonstrate the potential use scenario of the prediction model, this model was then applied to retrieving health advice regarding the use of hydroxychloroquine (hcq) as a treatment option from litcovid, a large covid research literature database curated by nih. hcq was considered a promising treatment option at the beginning of covid but was found to be ineffective in later clinical trials. more specifically, we seek empirical answers to the following research questions: 1) to what extent can nlp models identify health advice in medical research literature? 2) what health advice has been made regarding the use of hcq for covid-19 treatment? 2 related work ## corpus construction we chose pubmed as the data source, since pubmed is the largest health literature database. besides abstracts, it provides rich metadata that can help select research papers with different types of study designs. pubmed central, an open-access subset of pubmed, provides full text content of a portion of pubmed-indexed publications. the evidence-based medicine (ebm) pyramid specifies that different study designs lead to different levels of evidence toward medical decision making @xcite . to ensure our health advice prediction model is effective in identifying health advice across study designs, we sampled research papers from both randomized controlled trials (rcts) and observational studies by the mesh terms in pubmed. an rct would randomly assign individuals into experiment and control groups in order to compare the effect of treatments or interventions @xcite . in an observational study, individuals are observed or certain outcomes are measured; however, no interventions and treatments are carried out by researchers to affect the outcome @xcite . observational studies are widely applied in fields of epidemiology, social sciences, and psychology, when rcts are not always possible or ethical to conduct @xcite . within the observational studies, we further sampled from four common subtypes, namely cross-sectional, case-control, retrospective, and prospective studies, in increasing order of evidence strength. health advice normally appears in either abstracts or conclusion/discussion sections. since sentences containing health advice account for a very small portion of all sentences, to avoid annotating a large number of non-advice sentences, we annotated a sample of sentences from the conclusion subsections in structured abstracts. note that the abstracts in pubmed are either "unstructured" or "structured". an "imrad" structured abstract consists of several subsections: introduction/background, method, result, and conclusion/discussion. in fact structured abstracts have now become the predominant mode of abstracts in major medical journals @xcite . the conclusion/discussion subsection is usually a few sentences long, and thus is the most balanced source for both health advice and non-advice sentences. in the end, a total of 6,000 sentences were sampled from conclusion/discussion subsections, including 3,000 from observational studies and 3,000 from rcts. based on the three-category coding schema, each sentence was assigned to one of the three category labels "no advice", "weak advice", or "strong advice". later in this paper we will demonstrate that a prediction model based on this annotated data set is generalizable to distinguishing advice and non-advice sentences in unstructured abstracts and discussion sections. for this evaluation we further annotated all sentences in 100 unstructured abstracts and 100 discussion sections, which will be described in the model evaluation section. to test the validity of the proposed schema, a sample of 100 sentences were randomly selected for inter-coder agreement evaluation. two annotators each labelled the 100 sentences and highlighted the linguistic cues for health advice. the overall cohen's kappa agreement @xcite was 0.86, indicating a near-perfect inter-coder agreement @xcite . disagreed cases were later resolved by the two annotators through discussion. three annotators with academic backgrounds in clinical psychology, linguistics, and information science were then trained to annotate the entire training corpus. all ambiguous cases were highlighted during the annotation and brought to all team members for group discussion to reach an agreement on the annotations. in most cases, an advice sentence has one advice type only. occasionally, a sentence includes both "weak advice" and "strong advice". we treated such cases as mixed examples and excluded them from the training corpus. the final corpus includes 5,982 sentences. since the majority of conclusion sentences do not contain advice, the category distribution in ## prediction model development and evaluation similar to tasks in suggestion mining, we framed automated detection of health advice as a sentencelevel text classification task. we trained and evaluated three machine learning approaches: linear svm, bert @xcite , and biobert @xcite to identify sentences containing health advice. ## identifying health advice on covid-19 treatment: a case study on hydroxychloroquine (hcq) in this section we apply the health advice prediction model to research articles in the litcovid corpus, in order to find health advice on specific treatment options for covid-19. this is a case study to demonstrate the model's usefulness for retrieving health advice for a specific medical topic, especially when used in combination with existing health information services like litcovid. litcovid is a corpus curated by nih that includes all covid-19 related research publications in pubmed. litcovid has organized the research papers by topics, such as "transmission", "diagnosis", "prevention", and "treatment". litcovid further tagged all chemicals that were studied, and assigned normalized terms for chemicals with multiple names, such as the mesh unique id d006886 for "hydroxychloroquine" and its 40+ alternative names like "hcq" and "(hydroxy)chloroquine sulfate". we downloaded the litcovid corpus on 04/30/2021 that includes 126,000 research papers. using the mesh id of hcq, we retrieved 3,400 hcq-related papers with 10,000 sentences tagged with hcq in abstracts and discussion sections. these sentences were then sent to our prediction model to identify hcq-related health advice. in the prediction result we found 605 strong advice sentences and 815 weak ones. the advice ranges from recommendation to use (see example 1 below), advice on doses and usage (example 2), cautions and warnings on treating patients with certain conditions (example 3), and objection to use (example 4). examples of health advice regarding hcq: ## conclusion in this study, we developed a high-performing nlp model that can detect weak and strong health advice from abstracts and discussion sections in medical research publications. we further developed a case study, in which we applied this model to retrieve health advice regarding the use of hcq for covid-19 treatment from the litcovid data hub. the case study demonstrated that this health advice prediction model can be combined with existing health information service systems to provide more convenient navigation of a large volume of health literature. if further combined with other nlp tools, such as claim and stance classification, the health advice service would be able to compare and summarize the evidence strength of recommendations for or against certain policies or treatments. the prediction model may be further extended to detect exaggerated health advice in science communication by comparing advice given in research papers against its counterparts in press releases, news articles, and social media posts @xcite . in future work, we will extend health advice identification to news and social media. our annotated corpus and code are available at https://github.com/junwang4/ detecting-health-advice .