paht_nlp @ MEDIQA 2021: Multi-grained Query Focused Multi-Answer Summarization (2021)

Abstract: In this article, we describe our systems for the MEDIQA 2021 Shared Tasks. First, we will describe our method for the second task, Multi-Answer Summarization (MAS). For extractive summarization, two series of methods are applied. The first one follows (CITATION). First a RoBERTa model is first applied to give a local ranking of the candidate sentences. Then a Markov Chain model is applied to evaluate the sentences globally. The second method applies cross-sentence contextualization to improve the local ranking and discard the global ranking step. Our methods achieve the 1st Place in the MAS task. For the question summarization (QS) and radiology report summarization (RRS) tasks, we explore how end-to-end pre-trained seq2seq model perform. A series of tricks for improving the fine-tuning performances are validated.

## introduction automatic summarization is an essential task in the medical domain. it is time consuming for users to read a lot of medical documents when they use a search engine like google, medline, etc, about some topic and obtain a list of documents which are potential answers. first, the contents might be too specialized for layman to understand. second, one document may not answer the query completely, and the users might have to summarize the conclusions across multiple documents, which may lead to waste of time or misunderstanding. in order to improve the users' experiences when using medical applications, automatic summarization techniques are required. the mediqa 2021 shared tasks are held to investigate the current state of the art summarization models, especially how they perform in the medical domains. three tasks are held. the first one is question summarization (qs), which summarizes long and potentially complex consumer health * contact: 52205901018@stu.ecnu.edu.cn. in the mas task, we improve upon (xu and lapata, 2020) via three methods. first, during the coarse ranking of a sentence in one of the given documents, we also add the surrounding sentences as input and use two special tokens marking the positions of the sentence. this modification improves the coarse ranking with a large margin. second, during fine-grained re-ranking, instead of incorporating a inverse sentence frequency (ifs) score based similarity matrix between sentences in the markov chain model, we find that directly using semantic similarity scores to form the similarity matrix performs better. third, due to the low resource settings of this task, we find that applying a roberta @xcite model which is already fine-tuned on the ms-macro task @xcite can be beneficial. for the other two tasks, we mainly explore how the pre-trained seq2seq model like bart @xcite , pegasus @xcite , etc, can perform in these tasks. two take-aways can be made. first, for tasks with small dataset size, freezing a part of the transformer blocks can be beneficial. second, for the rrs task, we find that controlling the maximum output sequence length can improve the rouge score on the test set. our team paht_nlp participate in all the three tasks, and won the 1st place in the mas task. experiments will show that our modifications are beneficial for both stage of the mas task. we also report extensive experiments for task 1 and task 3. ## contextualized evidence estimation the previous section describe a two-step method for extractive mds. however, it does not fully exploit the advantages of pretrained sentence encoders, since it only compares the query to single sentences which suffers from losing the contexts. in this section, we provide a simple method to conduct extractive mds in one step, and promote the performances. let q denote a query sequence and {s 1 , s 2 , ..., s n } the set of candidate answers. and we put each sentence s i back into its contexts by concatenating the sentences surrounding it. denote the s i with its contexts as c @xmath0; s i ; n r i ]. for implementation, we limit the sequence length of n i by l max , which is 512 for roberta. for formulating the input of roberta, we concatenate c i following its sequential order, so that its contexts is not corrupted. thus the sequence input should be @xmath1, q, < /s >, < s >, n l i , < /s >, < s >, s i , < /s >, < s >, n r i , < /s >. the above operation adds the contextual information of s i , but the position of s i is not emphasized, and the model might focus on n r i or n l i instead of s i . thus, we add a pair of special tokens before and after s i to address the position of the sentence we are concerning. thus, the input sequence @xmath2, q, < /s >, < s >, n l i , < /s >, < s >, < t1 >, s i , < t2 >, < /s >, < s >, n r i , < /s >. the roberta will encode the above sequence and outputs the semantic relevance score, which we will use as the final semantic score of the sentence regarding summarization. ## end-to-end abstractive summarization pre-trained models. in this section, we experiment on applying pretrained seq2seq models to obtain abstractive summarizations, after finetuning their on our datasets. we mainly investigate two types of models, bart @xcite ) and pegasus ( @xcite ). in terms of architecture, bart adopts a standard transformer seq2seq architecture ( @xcite ) with some small changes. it uses gelu (xxx, ) rather than relu (xxx, ) as activation function and initiates paramaters with normal distribution. for pre-training tasks, bart allows arbitrary noising transformations of input texts and learns a model to rebuild original text. bart achieves the state-of-the-art (sota) results on a wide range of tasks, including summarization and machine translation. pegasus uses pre-training objectives tailored for abstractive text summarization. during pretraining, the text inputs are documents with several important missing sentences and the output is the predicted missing sentence sequences. pegasus can perform quite well on summarization tasks with low resources, e.g., when the training sets only contains only hundreds of samples. finetuning techniques. for finetuning the pretrained seq2seq models, we experiment a few methods/techniques which can improve the downstream task performances: ## experiments on mas in task 2, we used two methods to deal with the problem of low resource data. the first method is to add muti-ext-summary and single-ext-summary as targets to the training data. since some sentences in the summary are not exactly the same as the sentences in the article, the jaccard similarity is used to align the sentences in article to the sentences in the extractive summary. because the final target is multi-text-summary, in order to increase its weight, we repeatedly sampled sentences in multiext-summary and added it to the training set. the second method, public dataset ms-marco is used to pre-train the roberta model. finally, the top 20 sentences based on the model score are selected and we restore their relative positions by recording the position of each sentence in the article in advance as the target. the result is shown in our model is optimized with adam on one tesla v100 gpu using the following parameters: learning @xmath4, maximum @xmath5. the learning rate is warmed up over the first 1 epoch. early stopping strategy for 5 epoch is used to select the optimal model in the end, we submitted the results of ensemble model and achieved the first place, as shown in over chqa datasets, we train a span prediction model based on the pointer networks and a question type classification model to predict the question focus and question type, respectively. the span prediction model obtains the performance of 83% exact match f1, and the question type classification model achieves 78% f1. based on those two models, we process train, valid and test set to the same pattern as the input: "subject:{question_focus};{question_type} message:{message}". the questions are highly like to be predicted as two sentence patterns: "what the treatments for . . . " and "where can i find information on . . . ". we find these patterns appear more than 300 of 1000 train data, so we do the re-sampling for train data according to the frequency of the first four word of target questions. we train model on this re-sampled train set and get the result on valid set ( although the score on valid set has decreased but the final score in the test set has increased. we conclude that the improvement are due to the higher diversity of the sentence patterns. ## experiments on rrs according to the results of table1, we choose pegasus as our best model. pegasus models stand out from other popular models due to their specially designed pretrain tasks. we test how different optimizers influence performances. by analysing data carefully, we find that almost half of the summaries start with pattern like "no acute ..." or "no evidence of ...". a simple idea is that we can separate the data according to the pattern into two kinds, one with pattern of starting from "no", one with other patterns, and train models separately. when predicting, we also need a classifier to classify samples and send samples into according models. we label samples of which summaries start with "no ..." as label 1, and label other samples as label 0. we than train pegasuslarge models to generate summaries and bertbase model to classify. the results are shown on considering our classifier does make mistakes when predicting, we set a threshold of 0.75. only when the classifier give samples probabilities higher than this, will we use the separately trained models. otherwise, we will use the wholly trained model to predict.