Narrative Detection and Feature Analysis in Online Health Communities (2022)

Abstract: Narratives have been shown to be an effective way to communicate health risks and promote health behavior change, and given the growing amount of health information being shared on social media, it is crucial to study health-related narratives in social media. However, expert identification of a large number of narrative texts is a time consuming process, and larger scale studies on the use of narratives may be enabled through automatic text classification approaches. Prior work has demonstrated that automatic narrative detection is possible, but modern deep learning approaches have not been used for this task in the domain of online health communities. Therefore, in this paper, we explore the use of deep learning methods to automatically classify the presence of narratives in social media posts, finding that they outperform previously proposed approaches. We also find that in many cases, these models generalize well across posts from different health organizations. Finally, in order to better understand the increase in performance achieved by deep learning models, we use feature analysis techniques to explore the features that most contribute to narrative detection for posts in online health communities.

## introduction narrative forms of communication are widely used for conveying information and building connections. broadly defined as a representation of someone's experience of a series of events @xcite , narratives take on different formats, ranging from short anecdotes and testimonials to lengthy entertainment tv shows and movies @xcite . in the health context, extensive research has found that narratives are more effective than nonnarratives (e.g., statistics, didactic arguments) in communicating health risks @xcite and promoting health behavior change @xcite . moreover, telling personal illness narratives helps patients to better cope with the illness @xcite and for health care professionals to better understand the illness @xcite . given that social media has become a widely used platform for cancer patients and their caregivers to share stories and connect with others @xcite , it is critical to understand what cancer narratives are told on social media and how they engage social media users. however, in order to understand the impact of narratives in online communication, narratives must first be identified in social media datasets. doing this often requires annotations from subject matter experts, which can be a costly process and difficult to scale up to massive datasets. in this work, we seek to understand the extent to whether natural language processing methods, specifically, fine-tuned large language models, can be used to automatically detect narratives within social media posts in the health domain using only a relatively small number of expert annotations. additionally, analyzing models that are able to successfully detect narratives can provide insights into the types of textual features that are most related to narrative text within a corpus. toward these aims, we collect and annotate a dataset of social media posts created by breast cancer organizations and address the following research questions: rq1 which text classification models provide the best performance for automatic narrative detection for social media texts posted by breast cancer organizations? rq2 how does the ability to detect narratives generalize across posts written by different organizations? rq3 which features are most important for automatic narrative detection in this context? to answer rq1, we compare a range of text classification methods and find that transformer-based deep-learning based methods outperform classical approaches like support vector machines, as well as the previous state-of-the-art method for detecting narratives within health-related social media posts @xcite ). to answer rq2, we split our dataset so that the same organizations' accounts are not used for both train and test data, finding that in most cases, it is possible for our best models to generalize well across organizations. finally, to answer rq3, we use machine learning analysis tools to identify which features contribute most to the prediction of narratives, finding that references to people, such as pronouns and names, as well as state-of-being verbs like "is", contributed strongly to cases where models predicted that texts contained narratives. our results suggest that automatic detection of narratives in social media posts is a promising application of text classification, and can help ease the burden of manual annotation for researchers seeking to study the relationship between narrative and other variables of interest at scale. foot_0 2 related work online health communities have been computationally studied before in order to understand how users show social support for one another @xcite , to automatically extract information needs of patients @xcite , and to identify linguistic patterns associated with anxiety @xcite . additionally, @xcite analyzed birth stories from an online forum and demonstrated the utility of these stories for computational work. machine learning models have been trained using textual health forum data to predict attributes such as the sentiment @xcite or cancer stage of the patients posting to forums @xcite . yet, most work in the area of computational analysis of online medical forums has not considered the importance of narrative. at the same time, computational approaches incorporating and extracting narratives have led to advances in the study of corporate finance @xcite , environmental issues @xcite , the analysis of clinical records @xcite , and emotion classification within stories @xcite . as nlp datasets, narratives are often directly collected by sampling data from sources that are already known to use narrative based on the genre of the corpus, such as literary works @xcite , doctors' notes @xcite , or fan fiction @xcite . in the social media domain, data is often sampled in a way to ensure the presence of narratives, e.g., by collecting posts from specific subreddits which typically contain narrative style posts @xcite . in other cases, the presence or location of narrative content is unknown beforehand and needs to be to detected or extracted. this might be done using filtering criteria like the length of the post or the presence of predefined linguistic patterns @xcite . however, some datasets contain a balanced mixture of both narrative and non-narrative content, and quick rulebased filtering is not adequate. in the domain of online health communities specifically, prior work has relied on expert annotations to determine what should or should not be considered a narrative @xcite . in each of these works, text classification models were trained to automatically determine whether or not a given post contained narratives, and support vector machines (svm) using bag-of-words or character n-gram features were found to be the best approach. we build upon this existing work by applying deep learning text classification models to the task of narrative detection in social media posts from breast cancer organizations as an example use case that includes personal narratives, texts for which narrative presence is unknown a priori, and provide the potential for enabling larger scale studies of the importance of narratives in health communication. we find that these approaches outperform svmbased models similar to those used by dirkson et al. (2019) 2 and verberne et al. (2019 ) and explore their effectiveness on our dataset throughout the rest of this paper. ## data collection and annotation a list of breast cancer non-profit organizations was identified from the canadian cancer survivor netthe top 10% posts in terms of total interactions were sampled for annotation. following standard procedures in content analysis @xcite , two expert coders annotated the presence of narratives (48.83%). all disagreements were resolved by discussion, and the consensus results were used for further analyses (i.e., the highest standard of intercoder reliability) @xcite . the overall agreement rate was above 0.9. for this study, we omit 9 posts which did not contain any text and only videos or images. the breakdown of the annotated dataset by non-profit organization account is presented in ## detecting narrative style next, we set out to determine how well various text classification models could detect the presence of narratives given the expert annotations as training data. for this experiment, we appended data from all five non-profit organizations into a single dataset. all the data points were then shuffled and split using 80% of the data for training, and each 10% for validation and test sets. the metrics that were used for model evaluation are the f1 scores, precision, and recall of the narrative class. we consider two categories of models: classical machine learning models using bag-of-words features, and transformer-based deep learning models. for the classical models, we experiment with various preprocessing schemes in terms of low-ercasing, lemmatization, and stopword removal, and choose the approach that gave the best performance on our validation set. that process included: lowercasing, removing urls, lemmatization using nltk's wordnet @xcite lemmatizer, and stopword removal using the nltk @xcite . however, given the importance of pronouns in narrative detection as evidenced in prior work @xcite , we do not remove pronouns as part of our stopword removal step. the models that we consider are naive bayes, logistic regression, and svm-classification, using each model's scikit-learn @xcite python implementation. model-specific hyperparameters were also tuned using the validation set as described in appendix a. additionally, we consider the best reported approach from @xcite , which is the previous best reported narrative detection model for online health forum data. we use the code provided by the authors to both preprocess the data and train the predictive model. the authors used an svm classifier with a linear kernel and characterlevel trigram features as input, and so we refer to this model as svm-trigram in our results. for the deep learning models, we use distil-bert @xcite , bert @xcite , and roberta @xcite ) models based on the distilbert-base-uncased, bert-base-uncased, and roberta-base checkpoints available from huggingface @xcite . the tokenizer for each model was automatically determined using the autotokenizer() class. we use the output representation of the [cls] token as input to the classification layer (the default approach when using the huggingface trainer class). hyperparameters are described in appendix a. the results of running each of these models are presented in ## analysis of narrative detection models we have established that deep learning models are very effective at detecting narratives from social media data, substantially outperforming classical machine learning approaches. however, it is not immediately apparent why these models are able to achieve better f1 scores. therefore, in this section, we use model interpretability tools to further examine which features contributed to the ability of our models to detect narratives. we chose the best performing models in each category, i.e., bert for the deep learning approaches, and svm for the classical models, and use the explainable ai tool for local interpretable model agnostic explanation (lime; ribeiro et al. ( 2016 )) to understand the significance of text-based features to each model. in both cases, we use the lime explainer function foot_5 to learn which features best explain the narrative class and non-narrative class. we chose 5000 samples and 25 features as parameters for the function, based on the suggested default values and our desire to include a reasonable number of features per example. each instance in the test dataset is examined using lime, which generates an importance score for each feature (token) in the input based on how much it contributes to predictions for the positive class (narrative) or negative class (non-narrative). for a given feature j in a given text i, a higher positive score w ij denotes greater importance of that feature in the overall narrative class and a lower positive score denotes a weaker importance of that feature for the same class. likewise, a greater negative value w ij for a feature indicates a stronger association with predictions of the non-narrative class. several examples of lime explanations are presented in figure 1 . we can see that for posts where both models made the correct prediction, the set of important features is approximately the same. however, when bert made the correct prediction and svm did not, we notice that bert places a greater emphasis on first names in the case of narratives, and features like "fatigue" and "common", which refer side effects of breast cancer, are correctly identified as important indicators that the post does not contain a narrative. while these qualitative results are highly useful, lime only provides the w ij score for a specific text, i, yet we sought to quantitatively understand which features were important across the entire test set. therefore, we use global aggregations of local explanations (gale; @xcite ) to aggregate the lime scores. for the purposes of aggregation, we set a cut-off of ϵ = 0.001 and consider any w @xmath0. a feature importance score of zero indicates that the feature does not explain much of either the narrative or the non-narrative class while making predictions. gale suggests several different methods for aggregating scores, but we use the global average importance i av g as it was found to correlate well with external measures of feature importance for model classification. the global average importance i av g j for a given feature j is defined as: considering the tokens with negative values, indicating that they were more relevant when predicting the "non-narrative class", we found words related to scientific studies, sharing songs, and describing clinical procedures. hashtags such as "myreserachstory" and "mondaymotivation" were also present, indicating posts that may have been trying to seek engagement through means other than the use of narrative. while our bert model was successful in detecting narratives by learning associations between features like pronouns and first names, the svm model failed to consistently learn these associations as indicated by the placement of several first names in the non-narrative (negative valued) end of the list. bert -svm word score celeste 0.29 latasha 0.24 barbe 0.14 mother 0.12 hall 0.11 beautiful 0.11 she 0.10 found 0.09 is 0.09 i 0.08 strong -0.02 diagnosis -0.02 bell -0.03 reality -0.03 be -0.05 journey -0.06 her -0.07 it -0.08 his -0.09 my -0.12 while these results illustrate which features were important to each model, they do not directly quantify the difference between the bert and svm. to investigate that further, we checked the extent to which the degree of importance i av g j for each feature differed between bert and the svm model ( ## conclusion in this paper, we show that deep learning models like bert, distilbert and roberta are effective at detecting narratives from social media data. previous research focused on the use of classical machine learning models to understand narratives in online health discussion forums, but we demon-strate that deep learning models outperform these when detecting the presence of narratives. we studied generalizability of the deep learning models across organizations, finding that overall, models are able to generalize well across accounts, suggesting that deep learning models provided with sufficient data can perform well on an unseen dataset with similar distributions. we also analyze the performance of deep learning models with explainable ai methods, uncovering important features that contribute to narratives in a particular context. however, there are certain limitations and challenges associated with these models. although they are quite successful at understanding narratives, performance of deep learning models is directly proportional to the quality of the dataset and they are highly susceptible to annotator and dataset bias. with the growing amount of health information being shared on social media, understanding narratives becomes extremely important to study public health behavior and estimate health risks. the work described in this paper is a step towards helping researchers automatically annotate narratives in social media posts, thus enabling larger scale studies of the impact of narratives on health conversations.