ODD : A Benchmark Dataset for the Natural Language Processing Based Opioid Related Aberrant Behavior Detection (2024)

Abstract: Opioid related aberrant behaviors (ORABs) present novel risk factors for opioid overdose. This paper introduces a novel biomedical natural language processing benchmark dataset named ODD, for ORAB Detection Dataset. ODD is an expert-annotated dataset designed to identify ORABs from patients’ EHR notes and classify them into nine categories; 1) Confirmed Aberrant Behavior, 2) Suggested Aberrant Behavior, 3) Opioids, 4) Indication, 5) Diagnosed opioid dependency, 6) Benzodiazepines, 7) Medication Changes, 8) Central Nervous System-related, and 9) Social Determinants of Health. We explored two state-of-the-art natural language processing models (fine-tuning and prompt-tuning approaches) to identify ORAB. Experimental results show that the prompt-tuning models outperformed the fine-tuning models in most categories and the gains were especially higher among uncommon categories (Suggested Aberrant Behavior, Confirmed Aberrant Behaviors, Diagnosed Opioid Dependence, and Medication Change). Although the best model achieved the highest 88.17% on macro average area under precision recall curve, uncommon classes still have a large room for performance improvement. ODD is publicly available.

## introduction the opioid overdose (ood) crisis has had a striking impact on the united states, not only threatening citizens' health @xcite but also bringing about a substantial financial burden @xcite . according to a report by the centers for disease control and prevention (2023), ood accounted for 110,236 deaths in a single year in 2022. in addition, fatal ood and opioid use disorder (oud) cost the united states @xmath01.5 trillion in 2021 @xcite . identifying patients at risk of ood could help prevent serious consequences @xcite . the opioid crisis is multifaceted, with factors like inadequate health insurance coverage @xcite , regulatory lapses @xcite , and profit-motivated campaigns by pharmaceutical firms (haffajee and mello, 2017) contributing to its complexity. countermeasures include deploying prescription drug monitoring programs (pdmps) (center for disease control and prevention , 2023), enhancing addiction drug education for healthcare providers @xcite , and developing less addictive drugs @xcite . notably, pdmps are data-driven systems tailored to detect patients at risk of oud. by leveraging data analytics, these systems have successfully shielded many from critical ood outcomes @xcite . opioid-related aberrant behaviors (orabs) or aberrant drug related behaviors (adrbs) are patient behaviors that may indicate prescription medication abuse @xcite . orabs can be categorized into confirmed aberrant behavior and suggested aberrant behavior @xcite ; national institute on drug abuse, 2023). herein, confirmed aberrant behaviors have a clear evidence of medication abuse and addiction while suggested aberrant behaviors do not have a clear evidence @xcite . orabs are not only clinically significant due to their strong association with ood @xcite and drug misuse @xcite ), but they also pose intriguing and challenging problems in terms of natural language processing (nlp). this is for two primary reasons. firstly, unlike other bionlp tasks where reliance is primarily on medical terms or jargon @xcite , orabs orab type example confirmed aberrant behavior misuse of legal substances (e.g. alcohol) falsification of prescription-forgery or alteration injecting medications meant for oral use suggested aberrant behavior asking for or even demanding, more medication asking for specific medications reluctance to decrease opioid dosing once stable previously, orabs have been detected by monitoring opioid administration (e.g., frequency and dosage) @xcite or self-reported questionnaires @xcite . however such measurements do not include the full spectrum of orabs (e.g., medication sharing, denying medication changing). in addition, patients can obtain opioids from multiple resources (e.g. illegal purchase and medication sharing), which are not captured in the structured data. it has been known that orabs are widely described in ehr notes and nlp techniques can be used to identify orabs @xcite . nonetheless, the previous study relied on a small amount of annotated notes, which were not publicly available. moreover, the previous work only considered orabs as a binary classification (present or not) and only explored traditional machine learning models (e.g., support vector machine (svm)). this paper proposes orab detection that is a novel biomedical nlp (bionlp) task. we also introduce an orab detection dataset (odd) which is large-size, expert-annotated, and multilabel classification benchmark dataset corresponding to the task. for this, we first designed a robust and comprehensive annotation guideline that labels text into nine categories which encompass two types of orabs (confirmed aberrant behavior and suggested aberrant behavior) and seven types of auxiliary opioid-related information (opioids, indication, diagnosed opioid dependency, benzodiazepines, medication change, central nervous system related, social determinant of health). following the guideline, domain experts annotated 750 ehr notes from 500 opioid-treated patients in the mimic-iv database @xcite , finding 399 notes with opioid prescriptions. in total, 3,718 instances were annotated across 2,940 sentences, including 162 instances of orabs ( 115 confirmed and 47 suggested aberrant behavior instances). we conducted experiments on two opioid-related aberrant behavior (orab) detection models, employing state-of-the-art (sota) nlp models. these experiments utilized two distinct approaches: traditional fine-tuning, as described by @xcite , and prompt-based finetuning, following the methodology outlined by @xcite . the experimental results on mimic showed that prompt-based tuning models surpass fine-tuning models in almost all categories. particularly noteworthy is the performance improvement in less common categories with fewer than 150 instances (referred to as uncommon categories such as suggest aberrant behavior, confirmed aberrant behaviors, diagnosed opioid dependency, and medication change). in these categories, the performance improvements were notably substantial, with the diagnosed opioid dependency, medication change, and suggested aberrant behavior classes each showing an increase of over 20 points. the main contributions of this paper can be organized as follows: ## task definition and evaluation criteria task definition the orab detection is an information extraction task that identifies whether an input text contains orabs (confirmed, and suggested aberrant behaviors) and additional concepts relevant to ood and oud. in addition, since all labels can be co-occurred together in a sentence, we formulate the multi-label classification. evaluation criteria previous study on nlpbased orab detection @xcite utilizes accuracy as an evaluation criterion. however, since the labels in the dataset are highly imbalanced (in ## orab detection models this section demonstrates pretrained language model (lm) based orab detections models; traditional fine-tuning model ( zahera et al., 2019) and prompt-tuning model. the prompt-based finetuning model has shown advantages in rare category classification (e.g. zero-shot or few-shot classification) @xcite . figure 1 demonstrates the baseline orab detection models. ## experiment 6.1 experimental environment experimental models to verify the generalizability of experimental results, we utilized two different lms pretrained on biomedical literature; biobert @xcite and bioclinicalbert @xcite . ## conclusion this paper introduces a novel bionlp task called orab detection, which aims to identify two orab categories and seven categories relevant to opioid usage from ehr notes. we also present the associated benchmark dataset, odd. the paper provides baseline models and their performances on odd. to this end, we trained two sota pretrained lms using a fine-tuning approach and prompt-based fine-tuning. experimental results demonstrate that the performance in three uncommon categories was notably lower compared to the other categories. however, we also discovered that prompt-based fine-tuning can help mitigate this issue. additionally, we provide various error analysis results to guide future studies.