C hic H ealth @ MEDIQA 2021: Exploring the limits of pre-trained seq2seq models for medical summarization (2021)

Abstract: In this article, we will describe our system for MEDIQA2021 shared tasks. First, we will describe the method of the second task, multiple answer summary (MAS). For extracting abstracts, we follow the rules of (CITATION). First, the candidate sentences are roughly estimated by using the Roberta model. Then the Markov chain model is used to evaluate the sentences in a fine-grained manner. Our team won the first place in overall performance, with the fourth place in MAS task, the seventh place in RRS task and the eleventh place in QS task. For the QS and RRS tasks, we investigate the performanceS of the end-to-end pre-trained seq2seq model. Experiments show that the methods of adversarial training and reverse translation are beneficial to improve the fine tuning performance.

## introduction the mediqa 2021 shared tasks aim to investigate the most advanced summary models, especially their performance in the medical field. there are three tasks. the first is question summary (qs), which classifies long and complex consumer health problems into simple ones, which has been proved to be helpful to answer questions automatically @xcite . the second task is multiple answer summary (mas) @xcite . different answers can bring complementary views, which may benefit the users of qa system. the goal of this task is to develop a system that can aggregate and summarize answers scattered across multiple documents. the third task is radiology report summary (rrs) @xcite , which generates radiology impression statements by summarizing the text results written by radiologists. automatic summarization is an important task in the field of medicine. when users use google, medline and other search engines, they need to read a large number of medical documents about a certain topic and get a list of possible answers, which is very time-consuming. first, the content may be too specialized for laymen to understand. second, one document may not be able to fully answer queries, and users may need to summarize conclusions across multiple documents, which may lead to a waste of time or misunderstanding. in order to improve the user experience when using medical applications, automatic summarization technology is needed. in the mas task, we improve upon (xu and lapata, 2020) via three methods. first, during the coarse ranking of a sentence in one of the given documents, we also add the surrounding sentences as input and use two special tokens marking the positions of the sentence. this modification improves the coarse ranking with a large margin. second, due to the low resource settings of this task, we find that applying a roberta @xcite model which is already fine-tuned on the glue benchmark @xcite ) can be beneficial. in the mas task, we use two methods to improve (?). first, when we rank a sentence coarsely in a given document, we add the surrounding sentences as input. this modification greatly improves the efficiency of coarse ranking. secondly, due to the low resource setting of this task, we find that it is beneficial to apply the roberta (liu2019robertaar) model, which has been fine tuned on the glue benchmark (wang2018glueam). for the other two tasks, we mainly discuss how the pre trained seq2seq model, such as bart @xcite , pegasus @xcite , can be implemented in these tasks. you can make two takeout. first, for tasks with smaller datasets, freezing part of the parameters is beneficial. second, backtranslation is beneficial for generalization. our team chichealth participated in all three tasks and won the first place for the overall per-formances. experiments show that our methods are beneficial for pre-trained models' downstream performances. ## extractive mds let q denote a query, and @xmath0, d 2 , ..., d m } a set of documents. we have implemented multi granularity mds following the implementation of @xcite . we first break down the document into paragraphs, which are sentences. then, a trained roberta model quantifies the semantic similarity between the selected sentence and the query, and estimates the importance of the sentence (evidence estimator) according to the sentence itself or the local context of the sentence. thirdly, in order to give the global estimation of the importance of each part in the summary, we use the centrality estimator based on the markov chain. ## abstractive summarization pre-trained models. in this section, we investigate the pretrained seq2seq models to obtain abstractive summarizations, after finetuning their on our datasets. we mainly investigate two types of models, bart ( @xcite ) and pegasus (( zhang et al., 2020a ## conclusion in this work, we elaborate on the methods we employed for the three tasks in the mediqa 2021 shared tasks. for the extractive summarization of mas task, we build upon xu and lapata (2020), and achieve improvements by adding contexts and sentence position markers. for generating abstractive summaries, we leverage the pre-trained seq2seq models. to improve the fine-tuning performances on the downstream tasks, we implement a few techniques, like freezing part of the models, adversarial training and back-translation. our team achieves the 1st place for the overall performances. in this work, we elaborate the methods used in the three shared tasks of mediqa 2021. for mas task, we employ the methods that are similar to @xcite . in order to generate abstract abstracts, we take advantages of the pre-trained seq2seq model. in order to improve the fine-tuning performance of downstream tasks, we use freezing part of the model, adversarial training. our team ranks first in the overall performances of the three task.