Reinforcement Learning for Abstractive Question Summarization with Question-aware Semantic Rewards (2021)

Abstract: The growth of online consumer health questions has led to the necessity for reliable and accurate question answering systems. A recent study showed that manual summarization of consumer health questions brings significant improvement in retrieving relevant answers. However, the automatic summarization of long questions is a challenging task due to the lack of training data and the complexity of the related subtasks, such as the question focus and type recognition. In this paper, we introduce a reinforcement learning-based framework for abstractive question summarization. We propose two novel rewards obtained from the downstream tasks of (i) question-type identification and (ii) question-focus recognition to regularize the question generation model. These rewards ensure the generation of semantically valid questions and encourage the inclusion of key medical entities/foci in the question summary. We evaluated our proposed method on two benchmark datasets and achieved higher performance over state-of-the-art models. The manual evaluation of the summaries reveals that the generated questions are more diverse and have fewer factual inconsistencies than the baseline summaries. The source code is available here: https://github.com/shwetanlp/CHQ-Summ .

## introduction the growing trend in online web forums is to attract more and more consumers to use the internet for their health information needs. an instinctive way for consumers to query for their health-related content is in the form of natural language questions. these questions are often excessively descriptive and contain more than required peripheral information. however, most of the textual content is not particularly relevant in answering the question @xcite . a recent study showed that manual summarization of consumer health questions (chq) has significant improvement (58%) in retrieving relevant answers @xcite . however, three major limitations impede higher success in obtaining semantically and factually correct summaries: (1) the complexity of identifying the correct question type/intent, (2) the difficulty of identifying salient medical entities and focus/topic of the question, and (3) the lack of large-scale chq summarization datasets. to address these limitations, this work presents a new reinforcement learning based framework for abstractive question summarization. we also propose two novel question-aware semantic reward functions: question-type identification reward (qtr) and question-focus recognition reward (qfr). the qtr measures correctly identified question-type(s) of the summarized question. similarly, qfr measures correctly recognized key medical concept(s) or focus/foci of the summary. we use the reinforce-based policy gradient approach, which maximizes the non-differentiable qtr and qfr rewards by learning the optimal policy defined by the transformer model parameters. our experiments show that these two rewards can significantly improve the question summarization quality, separately or jointly, achieving the new state-of-the-art performance on the meqsum and matinf benchmark datasets. the main contributions of this paper are as follows: ## related work in recent years, reinforcement learning (rl) based models have been explored for the abstractive summarization task. @xcite introduced rl in neural summarization models by optimizing the rouge score as a reward that led to more readable and concise summaries. subsequently, several studies @xcite @xcite @xcite have proposed methods to optimize the model losses via rl that enables the model to generate the sentences with the higher rouge score. while these methods are primarily supervised, @xcite proposed an unsupervised method that accounts for fluency, brevity, and coverage in generated summaries using multiple rl-based rewards. the majority of these works are focused on document summarization with conventional non-semantics rewards (rouge, bleu). in contrast, we focus on formulating the semantic rewards that bring a high-level semantic regularization. in particular, we investigate the question's main characteristics, i.e., question focus and type, to define the rewards. recently, ben abacha and demner-fushman (2019) defined the chq summarization task and introduced a new benchmark (meqsum) and a pointer-generator model. @xcite organized the mediqa-21 shared task challenge on chq, multi-document answers, and radiology report summarization. most of the participating team @xcite @xcite utilized transfer learning, knowledgebased, and ensemble methods to solve the question summarization task. yadav et al. (2021a) proposed question-aware transformer models for question summarization. xu et al. (2020) automatically created a chinese dataset (matinf) for medical question answering, summarization, and classification tasks focusing on maternity and infant categories. some of the other prominent works in the abstractive summarization of long and short documents include cohan et al. (2018); zhang et al. (2019a); macavaney et al. (2019); sotudeh et al. (2020). ## proposed method given a question, the goal of the task is to generate a summarized question that contains the salient information of the original question. we propose a rl-based question summarizer model over the transformer @xcite encoderdecoder architecture. we describe below the proposed reward functions. ## conclusion in this work, we present an rl-based framework by introducing novel question-aware semantic rewards to enhance the semantics and factual consistency of the summarized questions. the automatic and human evaluations demonstrated the efficiency of these rewards when integrated with a strong encoder-decoder based prophetnet transformer model. the proposed methods achieve state-of-the-art results on two-question summarization benchmarks. in the future, we will explore other types of semantic rewards and efficient multirewards optimization algorithms for rl.