R ed HOT : A Corpus of Annotated Medical Questions, Experiences, and Claims on Social Media (2023)

Abstract: We present Reddit Health Online Talk (RedHOT), a corpus of 22,000 richly annotated social media posts from Reddit spanning 24 health conditions. Annotations include demarcations of spans corresponding to medical claims, personal experiences, and questions. We collect additional granular annotations on identified claims. Specifically, we mark snippets that describe patient Populations, Interventions, and Outcomes (PIO elements) within these. Using this corpus, we introduce the task of retrieving trustworthy evidence relevant to a given claim made on social media. We propose a new method to automatically derive (noisy) supervision for this task which we use to train a dense retrieval model; this outperforms baseline models. Manual evaluation of retrieval results performed by medical doctors indicate that while our system performance is promising, there is considerable room for improvement. We release all annotations collected (and scripts to assemble the dataset), and all code necessary to reproduce the results in this paper at: https://sominw.com/redhot .

## introduction social media platforms such as reddit provide individuals places to discuss (potentially rare) medical conditions that affect them. this allows people to communicate with others who share in their condition, exchanging information about symptom trajectories, personal experiences, and treatment options. such communities can provide support @xcite and access to information about rare conditions which may otherwise be difficult to find @xcite . however, the largely unvetted nature of social media platforms make them vulnerable to mis and disinformation @xcite ). an illustrative and timely example is the idea that consuming bleach might be a viable treatment for covid-19, foot_0 which quickly gained traction on social media. all misinformation can be dangerous, but medical misinformation poses unique risks to public health, especially as individuals increasingly turn to social media to inform personal health decisions @xcite . in this paper, we introduce redhot: an annotated dataset of health-related claims, questions, and personal experiences posted to reddit. this dataset can support development of a wide range of models for processing health-related posts from social media. unlike existing health-related social media corpora, redhot: (a) covers a broad range of health topics (e.g., not just covid-19), and, (b) comprises "natural" claims collected from real health-related fora (along with annotated questions and personal experiences). furthermore, we have collected granular annotations on claims, demarcating descriptions of the population (e.g., diabetics), interventions, and outcomes, i.e., the pio elements @xcite . such annotations may permit useful downstream processing: for exam-ple, in this work we use them to facilitate retrieval of evidence relevant to a claim. specifically, we develop and evaluate a pipeline to automatically identify and contextualize healthrelated claims on social media, as we anticipate that such a tool might be useful for moderators keen to keep their communities free of potentially harmful misinformation. with this use-case in mind, we propose methods for automatically retrieving trustworthy published scientific evidence relevant to a given claim made on social media, which may in aggregate support or debunk a particular claim. the contributions of this work are summarized as follows. first, we introduce redhot: a new dataset comprising 22, 000 health-related reddit posts across 24 medical conditions annotated for claims, questions, and personal experiences. claims are additionally annotated with pio elements. second, we introduce the task of identifying health-related claims on social media, extracting the associated pio elements, and then retrieving relevant and trustworthy evidence to support or refute such claims. third, we propose redhot-der, a dense evidence retriever trained with heuristically derived supervision to retrieve medical literature relevant to health-related claims made on social media. we evaluate baseline models for the first two steps on the redhot dataset and assess the retrieval step with relevance judgments collected from domain experts (medical doctors). the reddit posts we have collected are public and typically made under anonymous pseudonyms, but nonetheless these are health-related comments and so inherently sensitive. to respect this, we (a) notified all users in the dataset of their (potential) inclusion in this corpus, and provided opportunity to opt-out, and, (b) we do not release the data directly, but rather a script to download annotated comments, so that individuals may choose to remove their comments in the future. furthermore, we consulted with our institutional review board (irb) and confirmed that the initial collection and annotation of such data does not constitute human subjects research. however, eacl reviewers rightly pointed out that certain uses of this data may be sensitive. therefore, to access the collected dataset we require researchers to self-attest that they have obtained prior approval from their own irb regarding their intended use of the corpus. ## the redhot dataset we have collected and manually annotated health related posts from reddit to support development of language technologies which might, e.g., flag potentially problematic claims for moderation. reddit is a social media platform that allows users to create their own communities (subreddits) focused on specific topics. subreddits are often about niche topics, and this permits in-depth discussion catering to a long tail of interests and experiences. notably, subreddits exist for most common (and many rare) medical conditions; we can therefore sample posts from such communities for annotation. ## tasks and evaluation redhot may support a range of tasks related to processing health-related social media posts. here we focus on an important, timely task: identifying medical claims on social media, and then retrieving relevant and trustworthy evidence that may support or refute them. methods for this task could aid content moderation on health-related forums, by providing an efficient means to (in)validate claims. more generally, such methods may permit meaningful "fact checking" of health-related claims by providing relevant contextualizing evidence. we outline a three-step approach for this task. (1) identify spans/sentences corresponding to pure claims. (2) extract from these specific pico elements. (3) retrieve clinical literature -specifically, reports of rcts -relevant to the claim, i.e., the extracted pio elements. we limit our focus to the problem of evidence retrieval here; future work might consider the subsequent step of automated claim validation on the basis on this. below we assess components for each of these steps. for the span and pio extraction steps (1 and 2), we evaluate models retrospectively under standard classification metrics (i.e. precision, recall, and f1 scores) using fixed train, development, and test sets which we will distribute with redhot. the final step (3) requires relevance judgments to evaluate model performance; for this we enlisted medical doctors (section 3.4). ## conclusions we presented redhot: a new, publicly available dataset comprising of about 22,000 richly annotated reddit posts extracted from 24 medical condition-based communities ("subreddits"). this dataset meets a need for corpora that can facilitate development of language technologies for processing health-related social media posts. we evaluated baseline models for categorizing posts as containing claims, personal experiences, and/or questions. focusing on claims, we then proposed and evaluated models for extracting descriptions of populations, interventions, and outcomes, and then using such snippets to inform retrieval of trustworthy (published) evidence relevant to a given claim. to this end, we introduced a heuristic supervision strategy, and found that this outperformed pre-trained retrieval models.