What Clued the AI Doctor In? On the Influence of Data Source and Quality for Transformer-Based Medical Self-Disclosure Detection (2023)

Abstract: Recognizing medical self-disclosure is important in many healthcare contexts, but it has been under-explored by the NLP community. We conduct a three-pronged investigation of this task. We (1) manually expand and refine the only existing medical self-disclosure corpus, resulting in a new, publicly available dataset of 3,919 social media posts with clinically validated labels and high compatibility with the existing task-specific protocol. We also (2) study the merits of pretraining task domain and text style by comparing Transformer-based models for this task, pretrained from general, medical, and social media sources. Our BERTweet condition outperforms the existing state of the art for this task by a relative F1 score increase of 16.73%. Finally, we (3) compare data augmentation techniques for this task, to assess the extent to which medical self-disclosure data may be further synthetically expanded. We discover that this task poses many challenges for data augmentation techniques, and we provide an in-depth analysis of identified trends.

## introduction self-disclosure is a complex communicative process @xcite ) that involves sharing one's personal thoughts, feelings, or memories with another individual @xcite . reciprocal self-disclosure between conversation partners may strengthen relationships @xcite and improve the communicative experience @xcite . selfdisclosure may take nuanced forms, such as medical self-disclosure (see figure 1 ), or the act of disclosing symptoms, diagnoses, or other information related to mental or physical health problems @xcite . some medical self-disclosures may be explicit (a), whereas others may be less direct (b and c). medical self-disclosure is helpful from a clinical perspective and reinforces therapeutic relationships during medical interactions @xcite . it also may enable earlier detection and treatment of medical issues @xcite @xcite . detecting medical self-disclosure automatically could support clinicians and other medical practitioners in productively identifying helpful patient information from untapped sources or during review of medical conversations @xcite @xcite . however, research towards automated medical self-disclosure detection has been limited and may require different techniques from those used to detect self-disclosure in the general domain @xcite . in this paper, we comprehensively investigate automated medical self-disclosure detection. we manually expand the only existing dataset (medsd) in this domain to include nearly 4,000 additional instances collected from publicly available forums, strengthening our understanding of this task and its relationship between performance and dataset size. in parallel, we experiment with data augmentation to study the feasibility of automated dataset expansion for this task. finally, we perform a comparative analysis of transfer learning models, to disentangle the subtle distinctions and shared charac-teristics between medical self-disclosure detection and other language tasks. our primary contributions are as follows. first (1), we create a publicly-available 3,919instance expansion to medsd, sourced from health-related social media posts. the dataset is triple-annotated with continuous (0-5) and graded (no self-disclosure, possible self-disclosure, and clear self-disclosure) labels. next (2), we conduct transfer learning experiments using transformer-based models to determine the utility of different source datasets in the context of detecting medical self-disclosure. we find that our highest-performing model outperforms the state of the art @xcite by relative percentage increases of 14.19% and 16.73% for accuracy and f 1 , respectively. finally (3), we conduct the first study of data augmentation in the context of medical self-disclosure. we find that this task poses unique challenges for data augmentation, and we explore these challenges in a detailed analysis. it is our hope that our findings add to the burgeoning knowledge base surrounding the detection and processing of medical selfdisclosure, and can be used as a guide for others working within this domain. we make our data and source code publicly available to facilitate replication and rapid follow-up. ## related work clinical literature has extensively examined the effects of patient self-disclosure with medical professionals, finding both positive benefits to the patient's health and overall wellbeing @xcite , as well as negative experiences if the patient feels that professional or personal boundaries have been crossed @xcite . often self-disclosure in these studies is broadly construed, with less frequent exploration of medical self-disclosure specifically @xcite . for instance, @xcite examine the correlation between information-seeking behaviors and self-disclosure, and @xcite investigate disclosure in online and in-person settings through statistical meta-analyses of standardized interviews, questionnaires, tests, and scales reported in existing studies. they conclude that disclosure of sensitive and private information occurs more frequently in online settings, highlighting the urgency of analyzing these settings more closely. @xcite developed a semi-supervised algorithm to classify self-disclosure levels from twitter conversations, and @xcite studied self-disclosure in a large dataset of twitter conversations about the coronavirus pandemic, proposing an unsupervised approach to detect voluntary disclosure of personal information. @xcite detected positive and negative self-disclosures based on linguistic features, such as liwc and word embeddings, in a supervised model. recently, @xcite created a multi-task roberta model to measure self-disclosure across varying domains, including both general and medical selfdisclosure. they found poor generalization across datasets, with in-domain self-disclosure detection models performing much better than across-domain models due to widespread differences in the topics and targets of self-disclosure. only two existing works have focused on detecting medical self-disclosure specifically. balani and de choudhury (2015) predicted levels of self-disclosure of mental wellness in reddit posts, and in our prior work we introduced the medsd dataset and established a transformer-based performance benchmark for the task of medical selfdisclosure detection @xcite . since the latter is the current state of the art and offers the only publicly available dataset and corresponding benchmark in this domain, we primarily build upon that work. we investigate medical self-disclosure specifically, and also study the manual and automated expansion of the medsd dataset. ## methods we compared fine-tuned transformer models from varying source domains on the task of medical selfdisclosure detection ( §4.2), and also studied the utility of data augmentation for automated dataset expansion within this domain ( §4.3). we describe these studies in the following subsections. ## evaluation we compare model performance using accuracy, precision, recall, and macro-averaged f 1 , following prior work on self-disclosure detection @xcite . for each experiment, we randomly split the specified data into training (80%), validation (10%), and test (10%) subsets. we measure the efficacy of our manual dataset expansion ( §5.1), empirically compare the performance of the proposed transformer models ( §5.2), and evaluate the performance of da techniques for medical self-disclosure ( §5.3). ## conclusion in this work, we study data source, quantity, and quality as it pertains to detecting medical selfdisclosure. first, we manually expanded medsd with 3,919 additional instances and clinically validated its labels in collaboration with a healthcare expert. next, we compared transformer models pretrained on varying source datasets for predicting medical self-disclosure, finding that our bestperforming model outperforms the state of the art by relative percentage increases of 14.19% and 16.73% for accuracy and f 1 , respectively. our findings also suggest that stylistic patterns prove more revealing than task-specific trends. finally, we study data augmentation in the context of this task, finding that it poses many da challenges. we document these in our analysis, opening the door to intriguing follow-up studies. we make our dataset and models available to the research community upon request, and we hope that our work can be used as a roadmap for future experiments in selfdisclosure detection.