WBI at MEDIQA 2021: Summarizing Consumer Health Questions with Generative Transformers (2021)

Abstract: This paper describes our contribution for the MEDIQA-2021 Task 1 question summarization competition. We model the task as conditional generation problem. Our concrete pipeline performs a finetuning of the large pretrained generative transformers PEGASUS (Zhang et al.,2020a) and BART (Lewis et al.,2020). We used the resulting models as strong baselines and experimented with (i) integrating structured knowledge via entity embeddings, (ii) ensembling multiple generative models with the generator-discriminator framework and (iii) disentangling summarization and interrogative prediction to achieve further improvements. Our best performing model, a fine-tuned vanilla PEGASUS, reached the second place in the competition with an ROUGE-2-F1 score of 15.99. We observed that all of our additional measures hurt performance (up to 5.2 pp) on the official test set. In course of a post-hoc experimental analysis which uses a larger validation set results indicate slight performance improvements through the proposed extensions. However, further analysis is need to provide stronger evidence.

## introduction the internet provides a wealth of information on health topics through specialised websites, forums, blogs and social networks. increasingly, consumers are using these information sources to answer their medical and health-related questions. in the course of this development, also the consumers' expectations regarding search engine functionalities have become much more demanding. instead of reading through a list of relevant articles returned by a classical search engine, short and precise passages are now expected to answer questions. this transformation also has an impact on the technologies used to fulfill the user's information needs. in particular, approaches for automatic questions answering as well as automatic summarization and simplification of (long) articles has received a lot of attention by researchers in recent years @xcite @xcite @xcite . this trend is also addressed by task 1 of the mediqa 2021 shared task @xcite through investigating consumer health-questions asked on the (experimental) medical question answering system chiqa foot_0 . as we participated only in this task, we refer to it as shared task (st) in the following. the goal of task 1 was to foster the development of new summarization approaches, specifically designed for the challenges of long and potentially complex consumer health questions. one major challenge of chiqa is the extraction of the user's main concern from the question text. the given questions are often lengthy and contain a lot of peripheral information, which makes automatic processing and answering (much more) difficult. recent studies highlight that expert-based summarizations of such questions can lead to significant enhancements of the overall qa process @xcite . effective automatic summarization methods could therefore play a key role for improving medical question answering. we contribute to this task by first building a baseline using the general conditional generation framework and then investigating three modifications to summarize the consumer health questions. our baseline relies on finetuning the large pretrained generative transformers pegasus @xcite and bart @xcite . we explore three different strategies to improve the performance of these baseline models, i.e. (i) integrating structured knowledge via entity embeddings, (ii) ensembling multiple generative models with the generator-discriminator framework and (iii) dis-entangling summarization and question word prediction. our best performing model, a fine-tuned vanilla pegasus, reached the second place in the competition. we observed that all measures hurt performance (up to 5.2 pp) on the evaluation set. however, a post-hoc experimental analysis (see section 3), using a larger validation set, indicates slight improvements through the model extensions. the remainder of the paper is organized as follows: the next section introduces our baseline and the three extension strategies in detail. section 3 highlights and discusses the experiments and results we obtained in our own evaluation as well as in the official assessment. the paper concludes which a summary of the main findings. ## conclusion in this work we investigate the large-scale pretrained generative transformers pegasus and bart for the task of health-related consumer question summarization in the context of the mediqa 2021 shared task (task 1). we propose and evaluate three different strategies, i.e. integrating structured knowledge via entity embeddings, utilizing a generator-discriminator framework and applying interrogative prediction, to extend these strong baseline models. our best performing model, a fine-tuned pegasus-large transformer, reaches an rouge-2-f1 score of 15.99 and is ranked second place in the competition. experimental results for our proposed extensions show a mixed picture and further analysis is needed to assess the quality of these extensions.