SB _ NITK at MEDIQA 2021: Leveraging Transfer Learning for Question Summarization in Medical Domain (2021)

Abstract: Recent strides in the healthcare domain, have resulted in vast quantities of streaming data available for use for building intelligent knowledge-based applications. However, the challenges introduced to the huge volume, velocity of generation, variety and variability of this medical data have to be adequately addressed. In this paper, we describe the model and results for our submission at MEDIQA 2021 Question Summarization shared task. In order to improve the performance of summarization of consumer health questions, our method explores the use of transfer learning to utilize the knowledge of NLP transformers like BART, T5 and PEGASUS. The proposed models utilize the knowledge of pre-trained NLP transformers to achieve improved results when compared to conventional deep learning models such as LSTM, RNN etc. Our team SB_NITK ranked 12th among the total 22 submissions in the official final rankings. Our BART based model achieved a ROUGE-2 F1 score of 0.139.

## introduction the question summarization (qs) task aims to promote the development of new summarization models that are able to summarize lengthy and complex consumer health questions. the consumer health questions can have a variety of subjects like medications, diseases, effects, medical treatments and procedures. the medical questions can also contain a lot of irrelevant information that makes automated question summarization a difficult and challenging task @xcite . it is also often cumbersome to go through lengthy questions during the question answering process and then formulate relevant answers @xcite . the automated summarization approaches for consumer health questions thus have many medical applications. an effective automated summarization approach for obtaining simplified medical health questions can be crucial to improving medical question answering systems. the mediqa 2021 @xcite ) proposes three different shared tasks to promote the development, performance improvement and evaluation of text summarization models in the medical domain: the role of question summarization or simplification in answering consumer health questions is not explored extensively when compared to the summarization of documents and news articles @xcite . @xcite explored various extractive and abstractive methods for summarization of questions that are posted on a community question answering site. the results showed that abstractive methods with copying mechanism performed better than extractive methods. @xcite proposed a closed-domain question answering technique that uses bi-directional lstms trained on the squad dataset to determine relevant ranks of answers for a given question. @xcite proposed sequence-to-sequence attention models with pointer generator network for summarization of consumer health questions collected from meqsum, quora question pairs dataset and other sources. the addition of pointer generator and cov-erage mechanisms on the sequence-to-sequence has improved the rouge scores considerably. in this paper, we describe the different models and experiments that we designed and evaluated for the consumer health question summarization (qs) task. the proposed models utilize the knowledge of pre-trained nlp transformers to achieve improved results when compared to conventional deep learning models such as lstm, rnn etc. the proposed models are based on transfer learning and fine tuning the dataset on different versions of nlp transformers like bart @xcite , t5 @xcite and pegasus @xcite . we have also benchmarked all the proposed models against traditional seq2seq lstm encoderdecoder networks with attention. the rest of this article is organized as follows. in section 2, we provide information about the data used such as description of datasets, dataset augmentation and pre-processing. section 3 gives an overview of transformer architecture and transfer learning. in section 4, we describe and compare results obtained from fine-tuning various transformer models on our augmented dataset. in section 5, we compare the performance of our proposed models with different transformer models in detail, followed by conclusion and directions for future work. ## models and results during the system development phase, we experimented with various models for the task of question summarization. the ranking for the task is based on the rouge2-f1 score. rouge-2 (recall-oriented understudy for gisting evaluation), is a metric which measures the overlap of bigrams between the model-generated and reference summaries in a summarization task. in the following sections, we discuss the various versions of the models that we fine-tuned for the question summarization task. ## comparative evaluation during the testing phase, we experimented with various models based on the transformer architec-ture, such as bart, t5 and pegasus as mentioned previously. we were allowed to submit a maximum of 10 runs per task. therefore, we submitted two runs each for t5 and pegasus models, and six runs for various approaches of the bart model. the test set provided for the question summarization task comprises of 100 nlm questions with their associated question ids. the test set was pre-processed in a similar fashion as the augmented dataset we had used for training. additionally, certain tokens such as "[name]", "[loca-tion]", "[contact]", "[date]", "subject: " and "message: " were removed from the test dataset to avoid their appearance in the generated summaries. the holms @xcite and bertscores @xcite for the different models used are referenced in ## conclusion and future work in this paper, we presented models that explore the use of transfer learning to utilize the knowledge of nlp transformers like bart, t5 and pegasus for the task of question summarization. the observed scores and the sample summaries generated by different transformer architecture based models clearly delineated the best performing model among the ones proposed. the summaries produced by the bart-large-xsum achieved the best score, followed by the pegasus-wikihow model. this can be largely attributed to the transfer learning technique that was adapted, by utilizing models which are pre-trained on massive datasets. as part of future work for the question summarization task, we plan to exploit question type feature, in addition to the currently used question focus feature for further enhancing the performance.