Multi-Task Training with In-Domain Language Models for Diagnostic Reasoning (2023)

Abstract: Generative artificial intelligence (AI) is a promising direction for augmenting clinical diagnostic decision support and reducing diagnostic errors, a leading contributor to medical errors. To further the development of clinical AI systems, the Diagnostic Reasoning Benchmark (DR.BENCH) was introduced as a comprehensive generative AI framework, comprised of six tasks representing key components in clinical reasoning. We present a comparative analysis of in-domain versus out-of-domain language models as well as multi-task versus single task training with a focus on the problem summarization task in DR.BENCH. We demonstrate that a multi-task, clinically-trained language model outperforms its general domain counterpart by a large margin, establishing a new state-of-the-art performance, with a ROUGE-L score of 28.55. This research underscores the value of domain-specific training for optimizing clinical diagnostic reasoning tasks.

## introduction the electronic health record (ehr) contains daily progress notes authored by healthcare providers to represent the daily changes in care plans for their patients, including an updated list of active diagnoses. the daily progress note is one of the most important note types in the ehr and contains the daily subjective and objective details in the patient's care, which is summarized into an assessment of the overall leading diagnoses with a treatment plan section @xcite . however, note bloat is a common phenomenon in medical documentation intermixed with billing requirements, non-diagnostic information, and copy and paste from prior notes @xcite . these additional documentation practices contribute to provider burnout and cognitive overload @xcite . problem-based charting is important figure 1 : training t5 with multi-task setup with six tasks from dr.bench @xcite to improve care throughput and help reduce diagnostic errors @xcite . the medical reasoning process is complex and incorporates medical knowledge representation with analytical and experiential knowledge @xcite . patel and groen developed a theory from the ai literature that experts use "forward-reasoning" from data to diagnosis 1986. the recently released benchmark dr.bench (diagnostic reasoning benchmark) is intended to assess the ability of ai models to perform such reasoning, with multiple component tasks including diagnostic reasoning with ehr data for experiential knowledge, medical exams for knowledge representation, progress note structure prediction, and problem summarization tasks that included both extractive and abstractive medical diagnoses @xcite . in this work, we focus primarily on the problem summarization task from the dr.bench suite, but with the hypothesis that using all tasks in dr.bench would improve the problem summarization task over the problem summarization task being fine-tuned alone. we make use of the t5 fam-ily of sequence-to-sequence language models, (raffel et al., 2020), which are first pretrained on a large unlabeled dataset and then finetuned on specific multiple downstream tasks. the text-to-text approach in our experiment makes it possible to perform multi-task training. hence, the t5 models were ideal for experimenting with single and multi-task techniques. further, we experimented with a recently developed clinically-trained t5 model to quantify the value of in-domain pretraining data @xcite . we make our software publically available at https://git.doit.wisc.edu/smph-public/dom/uw-icudata-science-lab-public/drbench . ## related work in the clinical domain, biomedical text summarization is a growing field. common approaches to text summarization include feature-based methods @xcite , fine-tuning large language models @xcite , and domain adaptation with fine-tuning methods @xcite . researchers developed clinical methods for summarization from progress notes but these methods were restricted to specific diseases such as diabetes and hypertension @xcite . moreover, these methods for summarization were more extractive than abstractive, using a combination of heuristics rules and deep learning techniques, and did not use large language models @xcite . in another work, an extractive-abstractive approach was used where meaningful sentences were extracted from the clinical notes first; these sentences were then fed into the transformer model for abstractive summarization @xcite . unfortunately, the transformer model frequently produced hallucinated outputs, and was not coherent when compared to the ground truth @xcite . in a similar extractive-abstractive approach, researchers used a pointer generator network to generate a note summary cluster and a language model such as t5 to generate an abstractive summary @xcite . none of these approaches used multi-task training or focused on clinically trained encoder-decoder since clinical t5 ## results and discussion the results of our experiments are summarized in clinical-t5 770m trained in the multi-task setting demonstrated the best performance (28.55) for the summarization task, establishing a new stateof-the art for this task. the single-task setting for the same t5 variant was a close second (28.28). t5 variants trained on in-domain data (scifive and clinical-t5) performed better than their general domain counterpart t5 models of the same size. all models, except clinical-t5 experienced a drop in performance when trained in a multi-task approach. we hypothesize that the models pretrained on non-clinical data were overwhelmed with outof-domain (i.e. clinical) data when trained in a multi-task way and failed to generalize as a result. predictably, larger models performed at least as well as the smaller models and outperformed the smaller models in most scenarios. admittedly, our work leaves open the question of whether the state-of-the-art performance obtained by clinical-t5 770m has to do with the fact that it was pretrained on mimic notes, which were also annotated in the problem summarization task. at the same time, the performance of other t5 variants, such as scifive 770m, was close, without it pretraining on mimic. this suggests that another t5 variant trained on a corpus of clinical notes that was different from mimic would perform at least as well or better depending on the size of the pretraining corpus. it should be noted that the model of this size, 770m parameters, can very likely absorb significantly larger amounts of clinical notes than what was available in mimic @xcite . we leave verifying this hypothesis for future work. ## conclusion in this work we experiment with the dr.bench suite of tasks and established a new state-of-the-art result on the problem list generation task, a task critical for ai-assisted diagnostic reasoning. our other contribution indicates that multi-task learning does not work well, unless in-domain data was used for pretraining and that included (unlabeled) task data during pretraining (a scenario known as transductive learning) leads to the best performance. finally, our work provides evidence that generative models benefit from pretraining on in-domain data. in future work, we plan to explore the utility of decoder-only llms for clinical diagnostic reasoning.