Leveraging Social Media as a Source for Clinical Guidelines: A Demarcation of Experiential Knowledge (2022)

Abstract: In this paper we present a procedure to extract posts that contain experiential knowledge from Facebook discussions in Dutch, using automated filtering, manual annotations and machine learning. We define guidelines to annotate experiential knowledge and test them on a subset of the data. After several rounds of (re-)annotations, we come to an inter-annotator agreement of K=0.69, which reflects the difficulty of the task. We subsequently discuss inclusion and exclusion criteria to cope with the diversity of manifestations of experiential knowledge relevant to guideline development.

## introduction messages shared on social media platforms are widely acknowledged as a source to gain insights into current events and the related vox populi. recently, this source of information has been extensively leveraged for identifying topics of interest @xcite , emotions @xcite , and stances @xcite during the covid-19 pandemic. a lesser scrutinized type of information is experiential knowledge in the medical domain: messages that describe practical contact with diseases or treatments (e.g.: having been contaminated, having received a vaccination), which may include contextual information that is relevant to the experience (e.g.: being a doctor, having certain allergies). such information is valuable for clinical guideline development, where social media may provide insights into latent experiences that are relevant to sharpen guidelines on, for example, exemptions and communication to patients. in this paper we discuss the task of automatically identifying experiences in facebook messages in dutch. as an input for clinical guidelines, experiences on social media are most valuable when they are provided with context and accompanied with value considerations. discussions following news reports on dedicated facebook pages are a relevant source in that respect, since these messages tend to be more extended than, for example, twitter messages, and may express personal disclosures as well. there are, however, two prominent challenges to identifying experiences from these discussions. first, only a small part of these messages share an experience, making it challenging to locate examples to train and test a machine learning classifier on. second, experiences take a diversity of manifestations, posing difficulties to merely reach a sufficient agreement among humans. this is also reported in several studies that aim to identify experiences in twitter @xcite and patient fora @xcite . in this paper we present our approach to these challenges, contributing an iterative procedure to zoom in on experiences within dutch facebook discussions on the topic of covid-19 vaccinations, as well as an annotation guideline catered for annotating experiences as input for vaccination guidelines. ## related work the most common aim for targeting experiential information from social media messages is to inquire into unforeseen effects of medicine intake @xcite @xcite @xcite @xcite . other aims are to gain insight into the experiences of particular patient groups @xcite and to inquire into topics of discussions related to hpv vaccinations, where the sharing of experiences is identified as one of these topics @xcite . studies that aim at identifying experiences mostly focus on twitter as a data source, while user fora are leveraged to a lesser extent @xcite . in contrast to these studies, we set out to identify experiences as input for rapid medical guideline development, and make use of facebook discussions as a data source. while ex-periences shared on social media have not been used before as evidence for clinical guidelines, automated approaches have been applied to swiftly identify scientific papers on a particular medical context to facilitate rapid clinical guideline development @xcite . in alignment with most of the studies that target experiences, we deployed manual annotations to come to a ground truth and encountered difficulties to reach a sufficient inter-annotator agreement on this task. @xcite ascribe these difficulties to the wide range of linguistic patterns by which experiences are manifested. we follow their solution to develop an annotation guideline that includes the different variants. in line with @xcite , we discuss the particular dilemma's that pose difficulties to decide on an annotation. a hampering factor to our endeavour to gain insight into experiences related to covid-19 vaccinations, is that the topic of vaccination is highly debated and therefore dominated by opinionated posts rather than experiences. participants in the debate express strong attitudes in favor and against vaccines, discussing issues related to the safety of vaccinations, the side effects, and the moral aspects of enforcing mandatory vaccination @xcite . nowadays, with collaborative media, anyone can join in a discussion and share information and opinions. this makes it difficult to attest the reliability of on-line content @xcite . doubts about the reliability of vaccines can easily grow among the population under the influence of negative information about vaccines or unbalanced reports of vaccine risk @xcite . knowing what people think about vaccines and why people decide to vaccinate or not has always been interesting for health practitioners who need to be adequately informed on public perception of the safety and necessity of vaccines. there have been efforts aimed at annotating data about the vaccination debate @xcite and at automatically finding opinions in favour or against vaccines @xcite @xcite . ## identifying experiences from facebook discussions our study sets off from a set of facebook posts discussing news articles related to covid-19 vaccinations, without a clear notion of the frequency and characteristics of messages that share an experience in these discussions. next we describe the iterative procedure that we conducted to identify experiential messages and collect enough examples to train a machine learning classifier on. an overview of the stages of this procedure and samples used is presented in ## annotation guideline and development the annotation guideline including different manifestations of experiences and examples was written after the annotations of sample 2 and refined after the annotations of sample 3. 4 for reasons of space, we focus here on the cases that posed most difficulties to decide on an annotation: in some cases, obtaining information from a doctor or medical specialist is mentioned, which could be seen as an experience. we chose to exclude this as it relates too much to sources of information. ## conclusion we set out to identify experiential knowledge from facebook discussions in dutch in relation to covid-19 vaccinations, using filtering criteria, manual annotations and machine learning in order to broaden the filter and draw more samples for manual annotation. after several annotation rounds, we defined annotation guidelines for experiential knowledge. annotators reached an agreement of 0.69 cohen's kappa, which indicates that the task is rather subjective with diverse manifestations of the targeted type of posts. messages on social media provide information that is difficult to obtain in other ways for rapid guideline development, with access to a diversity of backgrounds and experiences. before being able to integrate this type of information, insight is needed into what separates utterances baring experiential information from other types of messages in facebook discussions. the current endeavour to filter for such messages and discuss their characteristics has been a crucial step towards this point, but needs to be further refined to come to a machine learning classifier that yields a sufficient performance to be integrated in a hybrid workflow where a clinical guideline expert can further inspect the collected messages. future work to come to this point includes experimenting with different algorithms, feature representations and distant supervision filters, and further refining the annotation guideline.