M ason NLP + at S em E val-2023 Task 8: Extracting Medical Questions, Experiences and Claims from Social Media using Knowledge-Augmented Pre-trained Language Models (2023)

Abstract: In online forums like Reddit, users share their experiences with medical conditions and treatments, including making claims, asking questions, and discussing the effects of treatments on their health. Building systems to understand this information can effectively monitor the spread of misinformation and verify user claims. The Task-8 of the 2023 International Workshop on Semantic Evaluation focused on medical applications, specifically extracting patient experience- and medical condition-related entities from user posts on social media. The Reddit Health Online Talk (RedHot) corpus contains posts from medical condition-related subreddits with annotations characterizing the patient experience and medical conditions. In Subtask-1, patient experience is characterized by personal experience, questions, and claims. In Subtask-2, medical conditions are characterized by population, intervention, and outcome. For the automatic extraction of patient experiences and medical condition information, as a part of the challenge, we proposed language-model-based extraction systems that ranked $3ˆ{rd}$ on both subtasks’ leaderboards. In this work, we describe our approach and, in addition, explore the automatic extraction of this information using domain-specific language models and the inclusion of external knowledge.

## introduction social media platforms, like reddit, allow users to share their experiences pseudonymously. on these platforms, users tend to share different kinds of information, including questions, their personal experiences using certain prescribed drugs, or how they deal with the different symptoms caused by their medical condition. users may also make claims about certain clinical and non-clinical interventions (e.g. home remedies) and the associated outcomes. while this information can provide insight into * equal contribution medical conditions and how the condition presentation among users, their personal experience may be inconsistent with current medical evidence. thus, effective monitoring of these medical condition-related forums is needed to prevent the spread of misinformation and corroborate various user claims against scientific evidence. information from social media could also augment current scientific understanding of conditions and treatments. natural language processing (nlp) information extraction (ie) techniques can extract information from user posts, which could be incorporated into systems for monitoring and preventing user misconception. the redhot @xcite corpus consists of over 22,000 user posts from 24 medical condition-related subreddits annotated for entities related to patient experiences, claims, interventions, and outcomes. the posts are annotated in two stages: (1) subtask-1 annotations include user claims, personal experiences, questions, and claim-based personal experiences and (2) subtask-2 annotations contain more granular information through entities related to patient population, intervention, and outcome (pio). in this work, we explore the automatic extraction of these entities from a subset of the redhot corpus as released through task-8 foot_0 @xcite of the 17th workshop on semantic evaluation. the extraction of the patient experience and pio entities was pursued in subtask-1 and subtask-2, respectively. we propose bidirectional encoder representations from transformer (bert)-based systems that extract the patient-experience and medical-condition entities. in this work, we compare bert-based systems pre-trained on the general domain with domain-adapted bert models and explore augmentation techniques incorporating external clinical knowledge priors. we extract the patient experience entities from subtask-1 and medical condition entities from subtask-2 at an over-all f1 of 68.59 (sentence-level) and 32.65 (tokenlevel), respectively, on the challenge test set. our bert-based systems ranked 3 rd in both subtasks of the challenge. ## related work extracting medical information from free text is a well-explored research domain. this medical ie work can broadly be viewed as belonging to two major categories: (1) work attempting to extract information from biomedical literature and clinical text from electronic health records and (2) work that focuses on extracting relevant medical information from public forums, user platforms, and social media. medical ie work includes the development of annotated data sets and data-driven extraction architectures. we discuss relevant datasets first and then briefly review extraction approaches. in biomedical literature, commonly extracted information includes medical problems and their related characteristics @xcite . the evidence-based medicine (ebm)-nlp (ebm-nlp) @xcite corpus has over 5,000 abstracts of medical articles describing clinical trials annotated for population, intervention comparators, and outcome entities. this dataset is widely used for benchmarking and evaluating model performance in the biomedical domain @xcite . the ebm-nlp corpus utilized medical experts and amazon mechanical turk (mturk) workers. the biocreative-v chemical disease relation (bc5cdr) corpus @xcite contains pubmed articles annotated for disease and chemical entities. other corpora include annotations for biomedical phenomena like drugdrug interaction @xcite and gene-disease associations @xcite . prior work describes the benefits of extracting information from online forums for providing better treatments to patients @xcite @xcite . this information can be extracted from medical condition-specific forums (e.g. breastcancer.org, diabetesinfo etc.) or social media platforms (e.g. reddit, twitter). for e.g. @xcite extract medical condition-specific information from patient forums. other works use social media platforms like reddit to identify suicide risk @xcite , detect anxiety(shen and rudzicz, 2017), and depression @xcite , and identify schizophrenia by using linguistic analysis @xcite of the posts from online discussion forums. social media and online forum corpora have been annotated with a range of medical information. the self-reported mental health diagnoses (smhd) dataset consists of reddit posts of users who claimed to have been diagnosed with mental health conditions @xcite and build machine-learning-based systems to automatically classify diagnosed users from control with reddit posts. the early risk prediction on the internet (erisk) @xcite corpus focuses on detecting various text-related indicators of mental health conditions in reddit posts. medical ie tasks have been pursued using rulebased and data-driven approaches. early medical ie work used rule-based systems with semantic and lexical information @xcite or discrete machine learning approaches like logistic regression, support vector machine(svm) @xcite with engineered features. more recent medical ie work leverages neural networks. contemporary state-of-the-art medical ie work utilizes pre-trained language models, like bert @xcite , which have gained prominence across various nlp tasks and biomedical applications. there are many domain-specific bert models, including clinical and biomedical variants with mixed domain and in-domain pre-training techniques @xcite @xcite . initial work using redhot focused on building a machine-learning-based system to verify user claims. the redhot @xcite corpus contains reddit posts annotated for patient experiences, claims, claim-based personal experiences, questions, and granular annotations describing patient populations, interventions, and outcomes across 24 health conditions. the system extracts relevant user claims and pio entities from user posts. the system then uses this extracted information to inform the retrieval of trustworthy evidence relevant to the claim from a scientific knowledge base. ## conclusions we explore a novel medical information extraction task in which reddit posts are characterized by entities related to patient experiences and medical conditions. we perform this ie task from a sequence-tagging approach using domain-specific and knowledge-augmented systems. domainspecific pre-trained models are better at identifying the medical condition-related entities while general-domain pre-trained models are as good as domain-specific models in understanding patient experience-related descriptions. the inclusion of external knowledge, specifically disease information, helped improve population entity detection, and including chemical information helped identify intervention. with thorough error analysis, we identify where knowledge-augmented systems overcome errors faced by general-domain systems. in future work, it may be useful to utilize crosssentence information and perform entity normalization to extract the experience-related and medical condition-related entities with higher performance.