S em E val-2023 Task 8: Causal Medical Claim Identification and Related PIO Frame Extraction from Social Media Posts (2023)

Abstract: Identification of medical claims from user-generated text data is an onerous but essential step for various tasks including content moderation, and hypothesis generation. SemEval-2023 Task 8 is an effort towards building those capabilities and motivating further research in this direction. This paper summarizes the details and results of shared task 8 at SemEval-2023 which involved identifying causal medical claims and extracting related Populations, Interventions, and Outcomes (“PIO”) frames from social media (Reddit) text. This shared task comprised two subtasks: (1) Causal claim identification; and (2) PIO frame extraction. In total, seven teams participated in the task. Of the seven, six provided system descriptions which we summarize here. For the first subtask, the best approach yielded a macro-averaged F-1 score of 78.40, and for the second subtask, the best approach achieved token-level F-1 scores of 40.55 for Populations, 49.71 for Interventions, and 30.08 for Outcome frames.

## introduction social media allows individuals to discuss, potentially rare, medical conditions with others and to find "long-tail" information about condition trajectories and treatment strategies @xcite . while users may feel empowered, the unvetted nature of such online discourse makes it vulnerable to misinformation (factually incorrect statements, though not necessarily motivated by the aim to mislead) and disinformation (falsehoods crafted explicitly to shape public opinion) @xcite . the problem of medical misinformation was particularly visible at the height of the covid-19 pandemic, during which many unsubstantiated claims about treatments for the disease circulated. for example, ivermectin might be an effective treatment for covid-19. here, we propose a task which enlists nlp to detect health related conversations on social media. this is a crucial first step for countering health misinformation (among other potential applications). more specifically, we propose a multi-stage task. the first objective is to identify causal medical claims @xcite made within a given text snippet from a reddit post, consisting of either a single or multiple sentences, i.e., to identify the text spans that contains claims. the second objective is to extract key clinical elements from identified causal claims, in particular: the population (i.e., the condition), the intervention, and the outcome i.e., the pio elements @xcite . for example, if we identify a claim such as "drinking bleach can cure covid-19!", both the population and outcome would be "covid-19", while the intervention would be "bleach". 2 in an identified claim, the extracted interventions and the outcomes have a causal relationship communicated between them, either explicitly or implicitly. this task generated a lot of interest from researchers in the nlp community both because of the timely application for the pressing issue of medical misinformation @xcite , and the challenging nature of the task, which entails identifying claims on social media @xcite and then extracting @xcite ) key elements from these identified claims. additionally, the current advancements and widespread availability of large language model (llm) based systems for text generation in medical and clinical applications @xcite @xcite also underscore the need for research in this area. ## data and resources data sources the data we release as part of this shared task stems from a larger corpus of over 22k richly annotated social media posts from reddit spanning 24 health conditions; this is described at length in @xcite . for this task, we released 5,695 annotated reddit posts of this data focused on 10 of those health conditions. these include 597 posts containing pio annotations for claims. for the full dataset in @xcite , we identified a set of 24 condition-focused health communities ("subreddits") on reddit, ranging from common health conditions such as diabetes (generating very high online activity) to relatively rare chronic diseases like multiple sclerosis (ms). for each subreddit, we extracted the most recent posts (up to 1000) to include in the dataset. to annotate for gold labels, we relied on amazon mechanical turk (mturk). annotations were collected in two stages. in the first stage (stage-1), we obtain sentence-level annotations by asking workers to identify sentences (text spans) that correspond to claims, personal experiences, claims based on personal experiences, and questions. for the second stage (stage-2), we considered only instances where we find pure-claims, i.e., broad claims that are not related to a personal experience (∼7.7% of posts). here, we collected annotations to identify the relevant pio elements that correspond to those claims. the annotations from the first stage have an average span length of ∼23.0 tokens while the the second stage covers entities with an average span length of ∼2.0 tokens. for the first stage, we use the following definitions to identify sentences belonging to each category: obtaining high quality annotations was one of our top priorities and challenges. to that end, we ran three pilot experiments: the first, was an internal experiment with a very small sample of about 100 reddit posts which were annotated for stage-1 by two people with expertise in analysing biomedical data. we evaluated the quality of annotations through token-wise label agreement between our internal annotators. then, we conducted two pilot experiments on mturk with ∼6000 samples to identify and recruit workers. recruited workers were paid periodic bonuses based on the quality of a random subset of their annotated samples. in total, it took us a little over six weeks to accumulate stage-1 annotations and approximately two weeks to accumulate stage-2 annotations. this includes the time it took for us to evaluate the annotated data in batches and provide feedback to the workers. we provide additional details on quality validation of our data in @xcite . to account for user consent, we sent a short message to every reddit user whose public post we scraped to inform them about the potential inclusion in this corpus, the intended purpose of the data and to provide them the option to opt-out by responding within a period of 30 days. every user who responded within the 30 day period had their data completely removed from the broader dataset. ## results and discussion during the evaluation phase of our task, a total of seven teams participated with 48 valid submissions for subtask 1 and six teams participated with 43 valid submissions for subtask 2. the teams were given the opportunity to make an unlimited number of submissions for each subtask. the results presented in ## conclusion and future work we presented semeval-2023 shared task 8, a novel task to address the important and timely problem of identifying medical causal claims on social media posts. this was formulated as a multi-step process involving two subtasks: 1) causal claim identification, which consisted in classifying sentences as containing medical claims, personal experiences, claims based on personal experiences, or questions; and 2) pio frame extraction, aiming to extract spans corresponding to populations, interventions and outcomes associated to the identified claims. semeval participants were asked to build and evaluate systems to either or both subtasks given a dataset of reddit posts discussing 10 different health conditions. in total, seven teams participated in our shared task and six submitted a paper describing their systems. while there was a clear variation across different submitted systems, all of them used transformer-based models. overall, the final results show that both our subtasks are difficult and there is considerable room for improvement. the current advancements and widespread availability of llms capable of generating text that is indistinguishable from human-written text exacerbated the risk of mass production and dissemination of medical mis-and disinformation. therefore there is a growing imperative to conduct further research into methods that can help to detect and combat the spread of such misleading and potentially harmful content. a promising approach in this direction is to combine systems for the proposed subtasks and integrate them with a module to retrieve trustworthy evidence from the scientific literature that can help to validate or refute medical claims on social media.