Overview of the MEDIQA 2021 Shared Task on Summarization in the Medical Domain (2021)

Abstract: The MEDIQA 2021 shared tasks at the BioNLP 2021 workshop addressed three tasks on summarization for medical text: (i) a question summarization task aimed at exploring new approaches to understanding complex real-world consumer health queries, (ii) a multi-answer summarization task that targeted aggregation of multiple relevant answers to a biomedical question into one concise and relevant answer, and (iii) a radiology report summarization task addressing the development of clinically relevant impressions from radiology report findings. Thirty-five teams participated in these shared tasks with sixteen working notes submitted (fifteen accepted) describing a wide variety of models developed and tested on the shared and external datasets. In this paper, we describe the tasks, the datasets, the models and techniques developed by various teams, the results of the evaluation, and a study of correlations among various summarization evaluation measures. We hope that these shared tasks will bring new research and insights in biomedical text summarization and evaluation.

## introduction text summarization aims to create natural language summaries that represent the most important information in a given text. extractive summarization approaches tackle the task by selecting content from the original text without any modification @xcite @xcite , while abstractive approaches extend the summaries' vocabulary to out-of-text words @xcite @xcite . several past challenges and shared tasks have focused on summarization. the document understanding conference 1 (duc) organized seven 1 www-nlpir.nist.gov/projects/duc challenges from 2000 to 2007 and the text analysis conference foot_0 (tac) ran four shared tasks @xcite @xcite on news summarization. the last tac 2014 summarization task tackled biomedical article summarization with referring sentences from external citations. recent efforts in summarization have focused on neural methods @xcite using benchmark datasets compiled from news articles, such as the cnn-dailymail dataset (cnn-dm) @xcite . however, despite its importance, fewer efforts have tackled text summarization in the biomedical domain for both consumer and clinical text and its applications in question answering (qa) @xcite @xcite . while the 2019 bionlp-mediqa foot_1 edition focused on question entailment and textual inference and their applications in medical question answering @xcite , mediqa 2021 foot_2 addresses the gap in medical text summarization by promoting research on summarization for consumer health qa and clinical text. three shared tasks are proposed for the summarization of (i) consumer health questions, (ii) multiple answers extracted from reliable medical sources to create one answer for each question, and (iii) textual clinical findings in radiology reports to generate radiology impression statements. for the first two tasks, we created new test sets for the official evaluation using consumer health questions received by the u.s. national library of medicine (nlm) and answers retrieved from reliable sources using the consumer health question answering system chiqa 5 . for the third task, we created a new test set by combining public radiology reports in the indiana univer-sity dataset @xcite and newly released chest x-ray reports from the stanford health care. through these tasks, we focus on studying: ## official results we published three aicrowd projects (one for each task) to release the datasets and manage team registration, submission, and leaderboard ranking foot_8 . ## correlations among the evaluation measures in this section, we discuss correlations between the different evaluation metrics that we used in the challenge. in contrast, bertscore is substantially closer than holms in its correlation with the rouge metrics for both the mas task (cf. both language model-based measures had positive correlations with rouge for the qs task, but the level of correlation was substantially lower when compared to the mas and rrs tasks, going from a pearson coefficient range between 0.663 and 0.958 to a range between 0.193 and 0.372. as all submitted qs runs were described as abstractive or hybrid approaches, this discrepancy might be due to a stronger disagreement on summary assessment due to semantically-close but lexically distant summaries. it is also likely that the lexical distance between paraphrases was more pronounced due to the lengths of the question summaries, which are shorter than the summaries in the mas task. ## conclusion we presented an overview of the mediqa 2021 shared tasks on summarization in the medical domain. we presented the results for the three tasks on question summarization, multi-answer summarization and radiology reports summarization, and discussed the impact of summarization approaches and automatic evaluation methods. we find that pre-trained transformer models, fine-tuning on the carefully selected domainspecific text and ensemble methods worked well for all three summarization tasks. the results encourage future research to include in-depth exploration of ensemble methods, systematic approaches to selection of datasets for pre-training and fine-tuning, as well as a thorough assessment of the quality and relevance of different evaluation measures for summarization. we hope that the mediqa 2021 shared tasks will encourage further research efforts in medical text summarization and evaluation.