Identifying Self-Disclosures of Use, Misuse and Addiction in Community-based Social Media Posts (2024)

Abstract: In the last decade, the United States has lost more than 500,000 people from an overdose involving prescription and illicit opioids making it a national public health emergency (USDHHS, 2017). Medical practitioners require robust and timely tools that can effectively identify at-risk patients. Community-based social media platforms such as Reddit allow self-disclosure for users to discuss otherwise sensitive drug-related behaviors. We present a moderate size corpus of 2500 opioid-related posts from various subreddits labeled with six different phases of opioid use: Medical Use, Misuse, Addiction, Recovery, Relapse, Not Using. For every post, we annotate span-level extractive explanations and crucially study their role both in annotation quality and model development. We evaluate several state-of-the-art models in a supervised, few-shot, or zero-shot setting. Experimental results and error analysis show that identifying the phases of opioid use disorder is highly contextual and challenging. However, we find that using explanations during modeling leads to a significant boost in classification accuracy demonstrating their beneficial role in a high-stakes domain such as studying the opioid use disorder continuum.

## introduction extensive ongoing overuse of opioid medications, both from medical prescriptions and from illegal sources has led to a major public health crisis @xcite . there have been a total of 103,664 drug overdose deaths in the us in the 12-month period ending april i'm 18m and i've been taking norcos since i was 16 but just on and off. starting this year i've been taking it every day basically and now i'm tired of it. i still get high so ig my addiction isn't that bad as others but i don't want to get to that point. i'm tired of chasing the high. i've spent at least 3k on norcos this year and i can't control myself. i try to go a day sober but my mind is telling me i need and then withdrawals starts [...] due to their anonymous and real-time participation, community-based social media platforms such as reddit, have been used by researchers to understand issues around mental health selfdisclosure (choudhury and @xcite , suicide among youth @xcite , marijuana regulations @xcite , drug community analysis @xcite and covid-19 impact on people who use opioids @xcite . we choose reddit for our research, specifically the popular opioid-related subreddits r/opiates, r/opiatesrecovery as well as r/drugs to collect our data ( § 2.1). our research focuses on predicting the presence of self-disclosures related to oud phases in users' reddit posts (refer to our experiments demonstrate that: 1) the model performance improves significantly when trained/prompted with explanations. a further ablation study on human-annotated explanations versus machine-generated explanations confirms that the quality of explanations is key to such improvement; 2) smaller models fine-tuned on our novice-annotated data with explana-tions works best, surpassing zero-shot and few-shot large models, including gpt-4, by a large margin ( § 4); 3) an ablation study taking into account label uncertainty sheds light on model errors for cases where humans agree or disagree on the label; 4) our error analysis shows preliminary insights in understanding users' self-disclosure ( § 6). ## experiments we show results on both the expert-annotated test set and the novice-annotated test set. as the oud category distribution in our dataset is unbalanced, we report both accuracy and macro f1 scores in we highlight several takeaways. first, adding explanations helps the models both on expert and novice-annotated data (except for t0pp and gpt3 on expert data), particularly in few-shot and fully supervised settings. in § 5, we will show additional experiments to study the role of explanations and their quality for model predictions. second, supervised learning with small models outperform few-shot methods with larger models including gpt-4 by a large margin, on both expert and novice evaluation datasets, even if the training data is novice-annotated. t5-11b is the best overall model. while our training data is not annotated by experts, the quality of the data is still high. the accuracy on the expert evaluation set for a random baseline would be 17%, while a majority baseline would be 35%, which is significantly lower than 71.4% or 76.6% for the t5-11b model performance w/o explanation and w/ explanation, ## the role of explanations to test the quality and helpfulness of the annotated explanations on model prediction, we conduct three different experiments using our two best-performing models trained w/ explanation (t5-11b and deberta). all these experiments are conducted at inference time on top of a model fine-tuned @xmath0, text, e gold >. for convenience, from here on, we will refer to this model as m1. gold explanations at inference. in the first experiment, we use the gold explanations from our test sets (expert and novice). in particular, during inference, we prompt the two best-performing models (t5-11b and deberta) with an input that consists @xmath1, text, e gold >. silver explanations at inference. in a realworld setting, it is not possible to expect gold explanations at inference time. thus, in this setting, we investigate whether model-generated explanations can still be helpful for final label prediction. prior works in explainability have trained two types of models: 1) pipeline model, which maps an input to an explanation (@xmath2, and then an explanation to an output (@xmath3; and 2) joint self explaining models that map an input to an output and explanation (@xmath4. the latter has been shown to be more reliable @xcite . thus, we first train a t5-11b model (m2) that can jointly @xmath5, ex-@xmath6, text>. at inference time, we first generate a silver explanation e silver i by prompting m2 with a @xmath7, text @xmath8. we then prompt m1 @xmath9, text i , e silver @xmath10. while these explanations are not as high quality as gold explanations, they still outperform random explanations. it should be noted that the goal of this paper is not to build models that facilitate extracting accurate explanations. however, such models might improve the silver quality explanations and thereby improve overall classification results. we leave this for future work. random explanations at inference. as a baseline, we use a randomly selected sentence from the post as the explanation. we repeat the random selection for five random seeds and report the mean and standard deviation of these five runs ( ## error analysis to understand the challenges and limitations of our best models, we perform an error analysis. ## related research machine learning for substance use machine learning methods' application to substance use research is growing @xcite . several studies have attempted to predict substance use treatment completion among individuals with substance use disorders @xcite @xcite . this study takes advantage of anonymous data to identify treatment needs among individuals who may not currently be in formal substance use treatment. researchers have also used natural language processing to identify substance misuse in electronic health records @xcite and to classify substances involved in drug overdose deaths @xcite . @xcite collect user-level data on a social platform, forum 77, to build a crf model predicting three phases of drug use: using, withdrawing, and recovering. our work is different in several aspects: 1) we propose an annotation scheme grounded in research on addiction and substance use that defines behaviors such as medical use, misuse, addiction, recovery, relapse (and not using), that enable us to code selfdisclosures of such behaviors using both expert and novice annotators; 2) we develop explanationinfused accurate models to identify self-disclosure at the post level. these two innovations will enable future research on using these models for a reliable global, user-level analysis across time. learning from explanations there have been works focusing on learning from human-annotated explanations. @xcite investigates how free-form explanations and predicted labels are associated and use it as a property to evaluate the faithfulness of explanations. different from that, our work focuses more on the utility of extractive span-level explanations as an additional source of supervision in a high-stakes domain and further shows how the quality of explanations impacts inference time results @xcite . similar to our work, carton et al. ( 2022 ) leverages extractive explanations and shows a consistent trend that using explanations can improve model performance in reading comprehension. our work is most similar to @xcite , who noticed that the quality of explanations could have a huge impact on model performance and explore the utility of extractive explanations, and to @xcite , who perform similar studies using free-form explanations. understanding the oud continuum scientists have explained how opioids produce changes in brain structure and function that promote and sustain addiction and contribute to relapse (koob and @xcite . now recognized as a chronic but treatable disease of the brain, oud is characterized by clinically significant impairments in health and social function and influenced by genetic, developmental, behavioral, social, and environmental factors @xcite . the healing communities study implemented the opioid-overdose reduction continuum of care approach (orcca) to reduce opioid-overdose deaths across the oud continuum @xcite . taking advantage of self-disclosures on community-based social media, as this study aims to do, could lead to the development of interventions that better address risks associated with oud. ## conclusions we presented a novel task aimed to deepen our understanding of how people move across the oud continuum: given a user's post in an opioidrelated reddit, predict whether it contains a selfdisclosure of various phases of oud. we provided an annotation scheme grounded in research on addiction and substance use, which enables us to code self-disclosures of such behaviors using both expert and novice annotators. following the annotation scheme, we created a high-quality dataset annotated with class labels and text explanations. we presented several state-of-the-art explanationinfused models, showing they can achieve accurate results in identifying self-disclosures of use, misuse, addiction, recovery, and relapse. accurate models will enable further research in this space by considering a global user-level analysis across time. our error analysis showed that explanations could provide insights both into annotator disagreement and errors in model predictions. in addition, our findings shed light on how annotation uncertainty impacts model performance. this study's results are not without limitations. the anonymity of reddit users does not allow us to characterize the demographics or geographic extent of the study population. moreover, the current study looks at identifying self-disclosures at the message level without taking a global (userlevel) and temporal view. in our future work, we plan to apply our models to study users' posts in opioid-related reddits and observe their behavior over time. in addition, we will work on improving our models to both predict a label and provide a textual explanation for the prediction.