clulab at MEDIQA -Chat 2023: Summarization and classification of medical dialogues (2023)

Abstract: Clinical Natural Language Processing has been an increasingly popular research area in the NLP community. With the rise of large language models (LLMs) and their impressive abilities in NLP tasks, it is crucial to pay attention to their clinical applications. Sequence to sequence generative approaches with LLMs have been widely used in recent years. To be a part of the research in clinical NLP with recent advances in the field, we participated in task A of MEDIQA-Chat at ACL-ClinicalNLP Workshop 2023. In this paper, we explain our methods and findings as well as our comments on our results and limitations.

## introduction with the increase in accumulated digital medical records in the healthcare field, it is vital to recognize the need of automation in processing medical documents. the automation of medical document processing has been enhancing the efficiency of clinical documentation while enabling healthcare professionals to increase their quality of service. the advancements of medical imaging with machine learning has been integrated into medical decision making systems for the last decades @xcite @xcite , whereas nlp techniques have recently been proven useful for the field @xcite . the interest in clinical nlp applications has been growing each year. especially with the emergence of large language models, there has been an increasing number of research work in exploring their potential applications in the clinical domain. use of transformer based large language models has been proven to give impressive performance increases on variety of benchmarks in nlp @xcite . we have seen dramatic growth on llm applications across many nlp tasks @xcite . llms have also shown significant potential on clinical nlp tasks @xcite . prompt/instruct based language models @xcite have recently gained attention and already shown promising results in the clinical domain @xcite . these large language models hold promise especially for generative tasks like summarization @xcite . mediqa-chat tasks (ben abacha et al., 2023) at acl-clinicalnlp workshop is a shared task that focuses on summarization and generation of patient-doctor conversations. the shared task has 3 subtasks. in the task a, participants aim to generate an artificial section summary from a short patient-doctor dialogue and its associated section header out of 20 possible headers. in the task b, participants aim to generate an artificial clinical note from a full patient-doctor dialogue. in the task c, participants aim to generate an artificial doctorpatient dialogue from a clinical note. we officially participated in task a and reporting results for both task a and task b in this paper. the submission scripts can be found here foot_0 . ## dataset in our experiments we only used the official dataset of the shared task. ## methods in this section we explain the methods we applied to approach task a and task b. in all our experiments we used transformer @xcite task training set validation set test set a (ben abacha et al., 2023) 1201 100 200 b and c @xcite 67 20 40 ## experiments and results in all our experiments, we used 4 32gb nvidia v100 gpus. we used huggingface's transformers library @xcite as the basis of our experiment scripts. for our efforts to obtain the best models based on validation sets, we explored a hyperparameter space that can be seen in ## ethics statement certain ethical considerations should be taken into account while creating automated systems for processing doctor-patient conversations. the common faults of the proposed systems should be disclosed to system users. users should be trained to properly use and identify common mistakes of the systems. since the data to be processed is medical records, it is essential that both data and background models should be stored within strong security measures. lastly, patients and doctors should be informed that their conversations are recorded and may be used by the automated systems. ## discussion and future scope in this paper, we explored the capabilities of llms on summarization and classification of doctorpatient dialogues. we experimented for task a and task b but managed to have an official submission on task a. we documented our thought processes and approaches and stated our results. we obtained results that both supported and contradicted our hypothesis. due to hardware and budget limitations we did not have the chance to explore latest large models. the obvious future work would be on applying public instruct based models if the hardware capacity is enough or private instruct based models if the budget allows. more future work could be on preprocessing of the dialogues. intuitive postprocessing approaches could also be explored. ## limitations in this shared task, our main limitation has been lack of access to advanced gpus that can fit massive language models. given the limited time, we explored a small range of models and hyperparameter space. considering their proven generative capabilities, these models would be better starting point for producing summaries and dialogues which would allow researchers to focus more on pre/post processing and error analysis. another limitation has been lack of free access to massive language models that offer paid api. however, using private/commercial models for research purposes is open to debate in nlp community and isn't in the scope of this work.