DS 4 DH at MEDIQA -Chat 2023: Leveraging SVM and GPT -3 Prompt Engineering for Medical Dialogue Classification and Summarization (2023)

Abstract: This paper presents the results of the Data Science for Digital Health (DS4DH) group in the MEDIQA-Chat Tasks at ACL-ClinicalNLP 2023. Our study combines the power of a classical machine learning method, Support Vector Machine, for classifying medical dialogues, along with the implementation of one-shot prompts using GPT-3.5. We employ dialogues and summaries from the same category as prompts to generate summaries for novel dialogues. Our findings exceed the average benchmark score, offering a robust reference for assessing performance in this field.

## introduction the unprecedented size of textual data in electronic health records has led to the information overload phenomenon @xcite , which interferes with healthcare workers' information processing capabilities, diminishes their productivity, and prevents them from acquiring timely knowledge. records of complex patients, such as those chronically ill, are particularly difficult to organize and to present concisely @xcite , requiring physicians to read many clinical notes during a regu lar medical visit, which is often unfeasible. studies have shown that information overload can increase task demand and mental effort, which potentially impairs healthcare worker's understanding of patients' medical conditions and hinders optimal medical decisions, leading sometimes to fatal consequences @xcite @xcite . to tackle information overload phenomena, clinical text summarization methods have been proposed to support healthcare workers' textual data workflow interaction @xcite @xcite . clinical text summarization generates concise representations of documents using nlp methods @xcite . by doing so, it helps healthcare workers focus on the relevant information, which enhances medical decision-making and thus healthcare quality. indeed, usability studies conducted with physicians for ehr summarization indicated the effectiveness of reading automatically generated summaries as compared to raw records @xcite . to support efficient doctor decision-making, in this paper we investigate a novel approach that combines a traditional machine learning method, support vector machines (svm) @xcite , with a cutting-edge language model, gpt-3.5 @xcite , to effectively extract valuable information for the creation of doctorpatient dialogue summaries. we implemented a svm model for short medical dialogue classification, exploring its potential on a new task to distinguish between different categories of doctor-patient encounters. advanced generative language models have shown remarkable capabilities in text generation and reasoning. we incorporated gpt-3.5 with one-shot prompts, using dialogues and summaries from the same category as prompts to generate summaries for new dialogues. foot_0 ## related work we discuss two key aspects of the current state of the art: (1) text classification, particularly in medical dialogue classification, and (2) summarization, with a special focus on abstractive summarization. text classification text classification is a wellstudied problem in natural language processing, with various algorithms and techniques proposed for different domains. traditional machine learning methods, such as naive bayes @xcite , decision trees @xcite , k-nearest neighbors (k-nn) @xcite and svm @xcite , have been extensively used for text classification tasks @xcite . in the medical domain, these techniques have been employed to categorize clinical notes, medical dialogues, and other types of health-related text @xcite . deep learning approaches like convolutional neural networks (cnn) @xcite , recurrent neural networks (rnn) @xcite ), long short-term memory networks (lstm) @xcite , and transformer-based architectures @xcite , including pretrained language models such as bert @xcite , roberta @xcite , and xl-net @xcite , have demonstrated stateof-the-art efficacy in a diverse range of domains @xcite . leveraging the hierarchical structure of documents, graph neural networks (gnns) have also been effectively proposed to assign categories to biomedical documents @xcite @xcite . compared to deep learning models, svm requires lower computational resources and training time and is a more efficient choice for certain applications @xcite . abstractive summarization automatic text summarization includes extractive and abstractive summarization. extractive summarization identifies and selects important phrases or sentences from the original text. abstractive summarization generates summaries by creating novel sentences that capture the core information @xcite . abstractive summarization helps in generating concise representations of clinical notes, medical dialogues, and scientific articles @xcite . sequence-to-sequence (seq2seq) models utilizing rnns @xcite and transformer architectures @xcite @xcite are utilized in the abstractive summarization. the development of pre-trained language models, such as bidirectional encoder representations from transformers (bert) @xcite , generative pre-trained transformer (gpt) @xcite , and text-to-text transfer transformer (t5) @xcite , has further advanced the state-of-the-art of this field @xcite @xcite . recent studies have explored the use of fine-tuned versions of gpt-based models for medical text summarization, showing promising results @xcite . our work extends this line of research by employing gpt-3.5 with one-shot prompts for medical dialogue summarization, aiming to enhance performance and practicality. medical dialogue summarization more recently, the summarization of medical dialogues has started to gain momentum. @xcite use a knowledge-intensive approach, combining ontologies, guidelines and knowledge graphs to create a dialogue summarization system. the extracted triples are used to create a subjective-objectiveassessment-plan (soap)-like report. the model achieves relatively high precision but low recall for relevant summary items. @xcite attempted the generation of complete soap notes from doctor-patient conversations by first extracting and clustering noteworthy utterances and then leveraging lstm and transformer models to generate a single sentence summary from each cluster. @xcite showed that the quality of generated summaries can be improved by encouraging copying in the pointer-generator network. lastly, @xcite describe an abstractive approach based on bart, in which a two-stage summary model is created. the resulting models greatly surpass the performance of an average human annotator and the quality of previously published work for the task. ## methods we address task a of mediqa-chat 2023 @xcite , which focuses on dia-logue2note summarization in short dialogue classification and summarization. the objective of task a is to accurately predict the summarization and section header (as shown in ## experimental results in the following, we present the official results of our experiments on the mediqa-chat 2023 task a. ## conclusion our study demonstrates the effectiveness of combining traditional machine learning techniques, such as svm, with advanced language models, like gpt-3.5, for medical dialogue summarization. this hybrid methodology has the potential to improve documentation procedures during patient care and facilitate informed decision-making for healthcare professionals by classifying medical dialogues and generating concise summaries. for future work, we plan to address the limitations identified in this study. for classification, we will experiment with model configurations and explore alternative machine learning algorithms. for summarization, we will refine prompt strategies, incorporate domain-specific knowledge, and investigate various fine-tuning techniques. lastly, conducting user studies with medical professionals will provide valuable feedback to assess the utility and accuracy of our generated summaries in real-world clinical settings and further refine our approach.