Navigating Data Scarcity: Pretraining for Medical Utterance Classification (2023)

Abstract: Pretrained language models leverage self-supervised learning to use large amounts of unlabeled text for learning contextual representations of sequences. However, in the domain of medical conversations, the availability of large, public datasets is limited due to issues of privacy and data management. In this paper, we study the effectiveness of dialog-aware pretraining objectives and multiphase training in using unlabeled data to improve LMs training for medical utterance classification. The objectives of pretraining for dialog awareness involve tasks that take into account the structure of conversations, including features such as turn-taking and the roles of speakers. The multiphase training process uses unannotated data in a sequence that prioritizes similarities and connections between different domains. We empirically evaluate these methods on conversational dialog classification tasks in the medical and counseling domains, and find that multiphase training can help achieve higher performance than standard pretraining or finetuning.

## introduction current language technologies have enabled the analysis of large amounts of medical conversations to gain insights into important aspects of providerpatient interactions such as patient experience, response to treatment, time allocation for health issues, or quality assurance @xcite . however, many challenges remain open for this growing field of research on natural language processing (nlp) for healthcare. among them, there is a need for efficient training frameworks that address the lack of large-scale, publicly available medical dialog datasets. the recent success of large transformer-based models @xcite in many natural language processing (nlp) tasks related to dialog has motivated their application in the healthcare domain, mainly because of their adaptability ca-pabilities. work in this area has shown that large pretrained language models (plms) are effective for tasks such as assessing and analyzing the quality of counseling conversations or building chatbots for mental health care @xcite . previous work on dialog-oriented pretraining approaches has used discourse-aware (intersentential) learning tasks to learn "rich and robust context representations and interactive relationships of dialog utterances" @xcite . in addition, the composition and order of pretraining corpora have also been studied as crucial factors for downstream task performance @xcite , with a multiphase pretraining regimen consisting of general, domain-adaptive, and task-adaptive shown to be effective. while these methods have been found useful for general-purpose dialog, it is still unclear how they perform in medical dialog. different from other types of conversational dialog, medical conversations are domain-specific, and classification models require not only capturing the discourse relations between dialog utterances and turns but also being aware of speaker dynamics and medical terminology. in this work, we seek to empirically study the effectiveness of dialog-aware pretraining and multiphase pretraining in medical utterance classification tasks. we focus on pretraining tasks that allow the model to leverage conversational properties, such as turn-shift, speaker role, and intersentential dependencies. we evaluate these methods on datasets that are limited in size, especially compared to large corpora that are typically used to pretrain language models. thus, our goal is to confirm if the pretraining approaches lead to improvements over just finetuning with a small dataset. the contributions of this work are threefold. (1) we design and implement simple dialog-aware pretraining tasks for medical conversations. (2) we evaluate dialog-aware pretraining and multiphase pretraining and show that while the former does not fare better than dialog-agnostic approaches, the latter can effectively leverage unannotated corpora of varying task relevance. (3) we draw lessons for practitioners from our experiments. pretraining methods for conversations. recent work in this area has used intersentential learning objectives to infer properties associated with the relationship of sentences in the input. for instance, @xcite used masked-utterance retrieval to guess the replaced (masked) utterance based on the context utterance, and zhang and zhao (2021) explored intersentential coherence through utterance order restoration and contrastive loss. other approaches have explicitly incorporated the dialog structure and information as part of the pretraining task. for example, mpc-bert @xcite focused on multi-party conversations and used dialog-unique information such as utterance speakers and receivers. ## pretraining approaches we explore dialog-aware pretraining and multiphase training to leverage unlabeled data in medical conversations. ## datasets we evaluate our pretraining approach using two datasets portraying clinical interactions between patients and their care-providers @xcite and also a generalpurpose chit-chat dataset. ## experiments we focus on two utterance classification tasks: categorization and forecasting, formulated as shown in utterance classification tasks ## conclusion & lessons learned in this work, we studied the performance of pretraining strategies on utterance classification in the medical field, a domain that often suffers from a lack of large, publicly available datasets. we evaluated existing and novel dialog-aware and inter pretraining with unlabeled non-target corpora is a useful strategy when the availability of finetuning data is limited. our experimental results showed that pretraining with similar non-target data can boost performance. this is in line with previous findings by @xcite , showing that after general-domain pretraining on large corpora, additional, domain or target-related training can lead to performance gains. moreover, we recommend using a multiphase pretraining schedule that uses pretraining corpus on increasing order of task similarity. however, one caveat we observed during our low-resource experiments is that in settings where the amount of fine-tuning data is below a certain threshold, the advantage of pretraining can be limited. pretraining with domain-specific data does not result in better performance when compared to domain-agnostic objectives. we implemented several dialog-aware objectives and adapted mlm so that the masking procedure can utilize speaker information assigned to each utterance in conversation. however, we did not see conclusive evidence that these task-specific adaptations led to a significant improvement. furthermore, in some cases, pretraining with dialog-aware objectives led to a degradation in performance.