damo_nlp at MEDIQA 2021: Knowledge-based Preprocessing and Coverage-oriented Reranking for Medical Question Summarization (2021)

Abstract: Medical question summarization is an important but difficult task, where the input is often complex and erroneous while annotated data is expensive to acquire. We report our participation in the MEDIQA 2021 question summarization task in which we are required to address these challenges. We start from pre-trained conditional generative language models, use knowledge bases to help correct input errors, and rerank single system outputs to boost coverage. Experimental results show significant improvement in string-based metrics.

## introduction question summarization for medical forum is important for medical knowledge discovery and retrieval and facilitates downstream tasks such as biomedical question answering @xcite . medical questions are often complex, scattered with non-medical information, and can sometimes be erroneous because forum users are not domain experts @xcite . in addition, annotation in the medical domain is harder to acquire than in the general domain. these challenges make medical question summarization an important and difficult task where annotation is often scarce. the mediqa 2021 shared task 1 @xcite , medical question summarization, requires participants to build summarization systems for noisy medical forum texts with limited annotation data. the official training set of the task is the meqsum dataset @xcite , which is composed of 1,000 medical questions and their corresponding summaries. the validation and test sets consist of 50 and 100 questions respectively and topic words are sometimes misspelled. scarcity of data, noisy input, and complexity and redundancy of text all pose challenges for ques-tion summarization systems. we try to address these challenges using a combination of knowledgebased error correction, pre-trained generative language models, and output reranking. knowledge-based error correction leverages multiple levels of lexical resources and a high coverage knowledge base to correct errors in input. our experiments show that knowledge-based error correction helps downstream summarization performance according to the rouge metric. pre-trained generative language models are transformer-based language models trained with loss functions that facilitate sequence to sequence generation. models such as pegasus @xcite , bart @xcite , and t5 @xcite achieve state-of-the-art performance on various text generation tasks and are shown to perform well on few-shot generation scenarios @xcite . we finetune pre-trained language models to obtain baseline systems with limited amount of training data. output reranking picks the best output among multiple systems. the availability of different language models offers a diverse set of summaries to choose from. we observe difference in summarization styles between the training and the validation set and devise a simple heuristic to pick the best output based on this observation. in the rest of the paper, we describe these components and report evaluation results on the validation and the test set. ## task and architecture overview the mediqa question summarization task requires participants to summarize user generated medical queries into shorter, more focused questions. we present an example from the mediqa 2021 task 1 validation set in figure 1 @xcite . we note that the name of the disease "folliculitis" is spelled incorrectly in the input question and the question contains a lot of irrelevant information. we attempt to correct misspellings with a dedicated module in our system. as useful information is often scattered in different sentences in the input, abstractive summarization suits this task better than extractive summarization. we perform abstractive summarization with pre-trained language models. we illustrate the architecture of our submission in figure 1 (b): we first try to correct spell errors in the input; then summarize each question with three generative lms: pegasus, bart, and t5; finally, for each question, we pick the best output with a feature-based reranker and the best output is chosen as the summarization of the question. ## knowledge-based error correction misspellings are prevalent in medical forums, where non-expert users discuss highly specialized medical topics. uncorrected misspellings can lead to mismatch between the source text and the summary during training and cause errors if copied verbatim during prediction. these errors are penalized heavily by string matching-based metrics like rouge as they break n-grams. in this shared task, we conservatively correct misspelled words in input by reusing a cascade of candidate generation modules from an entity linking system. entity linking is the task to link entity mentions in text to entities in a knowledge base (kb). candidate generation is an intermediate step in entity linking to generate candidate kb entities from potentially abbreviated, misspelled, or alias text mentions (see e.g. @xcite ). our method is also comparable to previous work on levenshtein distance-based (levenshtein, 1966) medical query correction @xcite , but we augment that approach with cascaded knowledge sources and an alias table. error correction can be implemented easier and with possibly higher quality if search suggestions from online search engines @xcite are utilized. we use in-house error correction to keep the submission offline. ## summarization with pre-trained conditional generative language models pre-trained conditional generative language models have become the dominating paradigm for text generation and especially summarization, with recent models such as pegasus @xcite , bart @xcite , t5 @xcite , and palm @xcite achieving stateof-the-art results on standard benchmarks cnn-dailymail @xcite and xsum @xcite . recent work has also shown that these models achieve good performance in few-shot medical summarization settings @xcite . following @xcite , we use pegasus, bart, and t5 single systems as our baselines. we use pegasus-large, bart-large, and t5-base respectively in our experiments. ## output reranking following previous work on reranking generative lm outputs @xcite , we pick the best summary for each question using the following linear model from outputs of three heterogeneous generative lms, our approach. we use a simple and coverageoriented approach for reranking, based on the size and characteristics of the validation data. we notice that the writing style of the validation set is different from the meqsum data set which we use for training: in meqsum 18.5% sentences start with "what are the treatments for", 14.6% start with "where can i find", and 2.5% start with "what are the causes of ". a model trained on meqsum tends to generate these topic-based boilerplates that are not mentioned in the source text. but in the validation set, summaries do not have these boilerplate texts and resemble the content of the source text more closely, which inspires us to pick the output with high coverage of the source. we consider the validation set (50 sentences) too small for automatic tuning, so we design a minimal set of features and set the weights w i manually. features. we use fidelity, length, consensus and wellformedness features: for our experiments on the validation set and rouge-2 experiments on the test set, we set w @xmath0, w @xmath1.01, w @xmath2, w @xmath3. the idea is to select the summary that has highest coverage of the source that is a one sentence question, with at least one bi-gram in common with other summaries. the choice to favor high coverage summary is based on this particular pair of training and validation data, rather than general ensemble principles for text generation. we switch the weights for w f and w l for length reranking experiments on the test set. impact of reranking is evaluated in section 6.2.2. ## experiments 6.1 experimental settings our systems are based on the transformers @xcite package. we finetune baseline models on the meqsum @xcite rouge-l on the validation set in preliminary experiments. for experiments on the test set, models for ensemble are further finetuned for 50 epochs on the validation set. models for error-corrected input are finetuned on an automatically corrected version of the validation set. ## conclusion we reported our experiments in mediqa 2021 shared task 1. we used knowledge-based error correction and coverage-oriented reranking improve summarization. our system performed well on string-based rouge metrics, but less so on bertbased semantic metrics. we plan to investigate methods that improve summarization according to human judgement.