Overview of the Problem List Summarization ( P rob S um) 2023 Shared Task on Summarizing Patients’ Active Diagnoses and Problems from Electronic Health Record Progress Notes (2023)

Abstract: The BioNLP Workshop 2023 initiated the launch of a shared task on Problem List Summarization (ProbSum) in January 2023. The aim of this shared task is to attract future research efforts in building NLP models for real-world diagnostic decision support applications, where a system generating relevant and accurate diagnoses will augment the healthcare providers’ decision-making process and improve the quality of care for patients. The goal for participants is to develop models that generated a list of diagnoses and problems using input from the daily care notes collected from the hospitalization of critically ill patients. Eight teams submitted their final systems to the shared task leaderboard. In this paper, we describe the tasks, datasets, evaluation metrics, and baseline systems. Additionally, the techniques and results of the evaluation of the different approaches tried by the participating teams are summarized.

## introduction electronic health records (ehr) document patients' daily progress and, therefore, are an essential component of hospital care @xcite . however, note-taking in the ehr, while necessary, suffers from note bloat and information overload, causing challenges for end-users @xcite . the cognitive burden on providers using ehr documents remains high and only increases in our digital era @xcite . summarization tasks in clinical natural language processing (nlp) are a promising approach to overcome note bloat and help extract relevant, active diagnoses to help mitigate diagnostic errors. augmented intelligence with computerized diagnostic decision support is a potential solution to improve care throughput and quality at the bedside @xcite . over the past two decades, many tasks in the field of clinical nlp (cnlp) have focused on infor-mation extraction, document classification (e.g., named entity recognition), and relation extraction, as noted in a recent scoping review of cnlp tasks @xcite . as generative nlp systems have advanced significantly in recent years, the field has a growing opportunity on taking advantage of the latest advancements in generative large language models such as t5 @xcite , galactica @xcite , and gpt @xcite . tasks of clinical text generation include radiology report generation @xcite , clinical note generation @xcite @xcite , and patients' diagnoses and problems summarization @xcite . for the bionlp workshop 2023, the three shared tasks for the bionlp workshop 2023 all focused on biomedical text generation, in line with the current research trend on generative nlp models. this paper presents the overview of the shared task for the bionlp workshop 2023, problem list summarization. the problem list summarization (probsum) task, initially proposed by @xcite , aimed to facilitate the development of nlp models for downstream applications in diagnostic decision support systems. the goal of this shared task was to attract future research efforts in developing nlp systems for real-world decision support applications, where a system generating relevant and accurate diagnoses can augment the healthcare providers' decision-making process. the probsum task utilized the daily progress notes, a documentation approach that is ubiquitous in the ehr with a specific format that is taught in medical schools. the progress note contains the following four major sections: subjective, objective, assessment, and plan (soap) @xcite . participants developed models to create a list of relevant diagnoses or problems based on the information present in the subjective, objective, and assessment sections of the note. the plan contained the diagnoses and problems that served as the ground truth labels. summarizing the list of problems and diagnoses from daily progress notes is a challenging task for several reasons. first, progress notes may contain old diagnoses that are no longer relevant or list diagnoses that are not described or mentioned elsewhere in the note. second, the progress note may contain extra information needed for billing purposes or quality improvement metrics that do not contain diagnoses. further, the data collected in the progress note may be embedded from other parts of the ehr, such as laboratory results or medications. the structured text and tabular data alongside unstructured text present a multimodal problem. lastly, progress notes may contain medical jargon and abbreviations (e.g. "gi", "egd" in figure 1 ) that can be challenging for nlp models to understand and summarize. to accurately summarize patient problems, models must possess a deep understanding of the medical concepts and terminology involved, as well as the broader context of the patient's medical history and current condition. ## task description healthcare providers, including physicians, write daily progress notes to record the patient's medical status, treatment, and progress while they are hospitalized. these notes contain essential information, such as vital signs, medications, procedures, and any significant developments in the patient's condition. daily progress notes are crucial for monitoring the patient's overall health and ensuring that the healthcare team is informed of any changes or updates in their treatment. daily progress notes are formatted using the soap format. the subjective section of a soap format daily progress note comprises the patient's self-reported symptoms, concerns, and medical history. the objective section consists of objective data collected by healthcare providers during observation or examination, such as vital signs (e.g., blood pressure, heart rate), laboratory results, or physical exam findings. the assessment section summarizes the patient's overall condition with a focus on the most active problems/diagnoses for that day. finally, the plan section contains multiple subsections, each outlining a diagnosis/problem and its treatment plan. the aim of the task was to predict the list of problems and diagnoses outlined (subjective) chief complaint: cc: diarrhea admission: gi bleed 24 hour events: -gi performed egd: diffuse erythema/ ulceration, in esophagus, stomach, duodenum.. . . (objective) 07:39 am vital signs hemodynamic monitoring fluid balance 24 hours since am tmax: 38.1 c (100.6 tcurrent: 37.7 c (99.8 hr: 122 (96 -128) bpm . . . ## data description the task contained 768 hospital daily progress notes and 2783 diagnoses in the training set, which was previously described in more detail @xcite . a critical care physician and clinical informatics expert annotated a test set of 237 daily progress notes with only one progress note per patient to avoid redundancy. ## evaluation as the evaluation metric, we employed rouge-l, a measurement that captures the longest common subsequence (lcs) between a generated summary and its corresponding reference summary @xcite . in what follows, we report rouge-l precision (rl-p), rouge-l recall (rl-r), and rouge-l f-score (rl-f). although we acknowledge the limitations of using rouge as an evaluation approach, as it does not assess semantics, the field of automated evaluation for clinical note summarization is still in its early stages, with most research relying on human evaluation besides conventional metrics like rouge @xcite . therefore, for the purpose of the shared task, which requires ranking systems, we have chosen to use rouge as our evaluation metric. ## participation we released the training data and test data through physionet foot_1 , which serve as the data owners of the mimic-iii data and the executors of the data use agreement with participants. during system evaluation phase, we created a codalab competition to manage the system submission and leaderboard @xcite . to be specific, the codalab was configured to receive system prediction outputs in a text file format, and subsequently ran the rouge-l evaluation script to produce scores. each user was permitted to submit a maximum of 30 runs. at the time of writing this paper, eight teams from the united kingdom, india, spain, south korea, ireland, australia, and mexico submitted a total of 164 runs. along with the system output text file, we asked participants to provide a brief description of their methodology; however, some users submitted their results without providing us with any system details. ## discussion the 2023 probsumm bionlp workshop revealed that transformer-based language models trained on medical resources are capable of generating summaries from medical texts. while this finding was consistent with previous research that showed the effectiveness of these models in various nlp tasks, including text summarization, the probsumm task proved to be very difficult and is far from solved with a best f1 score of 32.77. none of the teams involved in the task included a medical provider or sought consultation from medical experts while developing their models. this could be attributed to practical limitations such as time and resource constraints. however, incorporating the expertise of medical professionals during the development process may provide valuable insights into the clinical implications of the generated summaries, resulting in the creation of more clinically relevant and useful models. given the effectiveness of incorporating human feedback in various nlp tasks @xcite , we recommend future research to explore the performance of involving medical experts in the development and evaluation through a human-in-the-loop approach. ## conclusion the probsum task, part of the bionlp shared task 1a, focused on summarizing patients' diagnoses from daily progress notes. eight teams from 7 countries submitted the final systems for evaluation, and the top-performing system has achieved substantial gains from the baseline with a new state-of-the-art result of 32.77. the experience gained by participants can inform the development of more effective strategies for medical text summarization, which could have significant implications for augmenting computerized diagnostic decision support systems at the bedside and potentially translate into better patient care.