MNLP at MEDIQA 2021: Fine-Tuning PEGASUS for Consumer Health Question Summarization (2021)

Abstract: This paper details a Consumer Health Question (CHQ) summarization model submitted to MEDIQA 2021 for shared task 1: Question Summarization. Many CHQs are composed of multiple sentences with typos or unnecessary information, which can interfere with automated question answering systems. Question summarization mitigates this issue by removing this unnecessary information, aiding automated systems in generating a more accurate summary. Our summarization approach focuses on applying multiple pre-processing techniques, including question focus identification on the input and the development of an ensemble method to combine question focus with an abstractive summarization method. We use the state-of-art abstractive summarization model, PEGASUS (Pre-training with Extracted Gap-sentences for Abstractive Summarization), to generate abstractive summaries. Our experiments show that using our ensemble method, which combines abstractive summarization with question focus identification, improves performance over using summarization alone. Our model shows a ROUGE-2 F-measure of 11.14% against the official test dataset.

## introduction the mediqa 2021 shared task consists of several independent tasks: task 1 is question summarization, task 2 is multi-answer summarization, and task 3 is radiology report summarization. we participated in task 1, question summarization. we approached the task by developing an ensemble learning method that combines information from automatic question focus identification with information from a state-of-the-art summarization model. we also studied the effects of different preprocessing techniques for this challenge. the * these authors contributed equally to this work descriptions of the dataset are shown in the task guidelines @xcite . the training datasets are from ben abacha and demner-fushman (2019b) along with the focus of each question. the test dataset contains consumer health questions only. ## related works the goal of consumer health question answering (chqa) is to construct an automated question answering system aimed toward answering questions from individuals who are unlikely to possess professional medical knowledge. typical consumer health questions include requests for information regarding symptoms of particular diseases, queries regarding possible diseases from individuals experiencing symptoms, and whether an individual would be safe to mix specific medications and so forth. in this field, there are circumstances in which individuals submit straightforward questions, but there are many cases where people list extra background and other unnecessary information which are not required to answer their question. in fact, this additional information can essentially serve as a source of noise which can reduce the effectiveness of the qa system as a whole. recent chqa systems employ pipeline architectures that utilize question understanding, information retrieval and answer generation components sequentially @xcite . this architecture facilitates modular optimization. furthermore, it allows individual components to be swapped, either for need or to provide special features. this allows the entire qa system to adapt to the specific nature of the problem at hand. as previously mentioned, many chqs possess extraneous information in addition to the primary question. therefore, the question understanding component of such an architecture is especially important, and improvements to it can be particularly beneficial to the overall chqa system. facilitating question understanding through summarizing the consumer health questions has demonstrated significant improvement as shown by ben abacha and demner-fushman (2019b) and ben abacha and demner-fushman (2019a). thus, given the benefits to the overall qa system, improving upon existing summarization methods was selected as task 1 in mediaqa 2021. in this component, the extraneous information can be removed via preprocessing prior to inputting into further stages of the qa system. different preprocessing methods have also been explored to perform this task, and a performance improvement on deep learning models has been shown (camacho-collados and pilehvar 2017; husain et al. 2020). ## methodology our model follows a traditional language generation pipeline: pre-processing, abstractive summarization, and post-processing. we experimented with several different combinations of preprocessing to generate multiple summaries from a single given question. we selected the best summary in the post-processing stage from these numerous generated summaries, which use ensemble learning. ## results all experiments are done on google colab pro with tesla v100 gpu, ram 25.51 gb0, cpu of intel(r) xeon(r) (2.20ghz). pointer generator networks training took up to 1 hour for both focus extraction and abstractive summarization. to finetune pegasus took 1.5 hours for 20k training steps, 12 hours for 80k steps, and 23 hours for 150k steps. ## discussion and future work we found many incorrect summaries with the wrong focus during the experiment of different combinations of pre-processing. for example, given input question "i have chronic renal disease and worry that magnesium silicofluoride treat-ment of moth infestation of a large living room rug will be harmful to my health. if the rug is treated in house how long before any toxic fumes or skin contact would be a hazard .", pegasus generated output (a) and ( b in the given question, we see that the person is concerned with the effect of magnesium silicofluoride on individuals with chronic renal disease. thus, the focus of the given question would be chronic renal disease and magnesium silicofluoride. in contrast, both generated output (a) and (b) summaries are built on incorrect focus. we believe extracting the correct focus and studying how to incorporate the focus would improve accuracy. in this paper, we applied focus in the post-processing step. we ranked the output using the equation 1, and then select the output with the highest score. the limitation of equation 1 is that the extracted focus may not be accurate. if the extracted focus is not correct, the ensemble model may choose a non-relevant output. for example, input "hydroxychloroquine for rheumatoid arthritis. can you tell me if this medication that my doctor put me on could make me sweat profusely at the slightest little strenuous activity i'm also methotrexate 6 2.5 mg once a week ." gives the following answers: the question asks if the hydroxychloroquine for rheumatoid arthritis and methotrexate be taken together. the generated summary (a) shows a reasonably accurate answer. in contrast, the focus extraction model assumed rheumatoid arthritis to be a focus, which leads the model to choose summary (b) over (a). despite this limitation, our experiments showed performance improvements after applying focus detection and the ensemble method in postprocessing. the post-processing effect is limited to the performance of the summarization model, accuracy of focus for each question, and the number of outputs from the summarization models. due to time limitations, we use two outputs in the ensemble process, while typical ensemble learning ## conclusion in this paper, we present our question summarization system for consumer health questions(chq). we explored effect of multiple pre-processing methods (de-identification, sentence exclusion and focus extraction) and on state-of-art abstractive summarization. our results show the best f-measure score of 11.14% through applying ensemble learning to different combinations of the pre-processing outputs. in our analysis, we identified future directions, including investigating the use of extracted focus, ensemble learning for the generative model.