Reinforcement Replaces Supervision: Query focused Summarization using Deep Reinforcement Learning (2023)

Abstract: Query-focused Summarization (QfS) deals with systems that generate summaries from document(s) based on a query. Motivated by the insight that Reinforcement Learning (RL) provides a generalization to Supervised Learning (SL) for Natural Language Generation, and thereby performs better (empirically) than SL, we use an RL-based approach for this task of QfS. Additionally, we also resolve the conflict of employing RL in Transformers with Teacher Forcing. We develop multiple Policy Gradient networks, trained on various reward signals: ROUGE, BLEU, and Semantic Similarity, which lead to a \mathit{10} -point improvement over the State-of-the-Art approach on the ROUGE-L metric for a benchmark dataset (ELI5). We also show performance of our approach in zero-shot setting for another benchmark dataset (DebatePedia) – our approach leads to results comparable to baselines, which were specifically trained on DebatePedia. To aid the RL training, we propose a better semantic similarity reward, enabled by a novel Passage Embedding scheme developed using Cluster Hypothesis. Lastly, we contribute a gold-standard test dataset to further research in QfS and Long-form Question Answering (LfQA).

## introduction query-focused summarization (qfs) @xcite @xcite advances text summarization by letting the user provide a query, pertaining to which the summary must be generated from the document(s). specifically, we target qfs for questions on a single document (see problem statement: we develop a system which abstractively generates summaries @xcite @xcite ) from a single document given the query. specifically, input: query and document, and output: summary. we design a novel reinforcement learning algorithm for training and propose a novel passage embedding-based reward function. motivation: qfs lets the user drive the summarization. this improves the user experience by letting the user extract the needed information quickly. additionally, present-day qa systems (calijorne soares and parreiras, 2020) produce only short answers to factual questions. what they lack is the ability to consume a large piece of information and present a coherent, compressed summary based on the needs of the user. our work aims to further research in lfqa by successfully building a qfs system for questions. we utilize reinforcement learning (rl) for training the model, as rl provides a generalized loss to train our model. while cross-entropy loss utilizes token-by-token match, rl helps us generalize this further to phrase/sentence/semantic match. such a generalization gives the model more freedom on what to generate at each time step, as long as the final generation suffices. inspired by this, we employ rl for qfs. we detail on how rl is a generalization to supervised learning (sl) using cross-entropy loss in section 3.1. however, employing rl for text generation using transformers is non-trivial. rl helps train agents that perform a sequence of actions, each deciding the next state and the available set of actions. in contemporary text generation through transformers, teacher forcing foot_1 (williams and @xcite is used, where the generation (action) at timestep t -1 has no influence on the generation at time-step t. effective utilization of rl for text generation needs to model this influence, thus necessitating the omission of teacher forcing. however, this increases the training time and memory footprint significantly. we propose a way to employ rl for text generation without teacher forcing, based on scheduled sampling @xcite , section 3.1. to the best of our knowledge, we are the first work to resolve this conflict of employing rl in transformers with teacher forcing. our contributions are: eli5 is an automatically curated dataset with a few shortcomings, which make it unsuitable for testing models @xcite . motivated by these shortcomings, we propose a novel, humancurated test set for qfs on well-formed questions consisting of high-quality 250 instances. rl for summarization/qfs: the usage of rl in qfs has been limited to extractive summarization only @xcite @xcite . @xcite ; mollá et al. (2020); @xcite use rl to train sentence selector models, which select sentences to be incorporated into the summary. @xcite present a hybrid summarization, where an extractive module selects text from the document, which is then used by the abstractive module to generate an abstractive summary. however, they use rl for the extractive module only. to the best of our knowledge, we are the first to utilize rl for abstractive qfs. we find an abundance of precedence of rl for abstractive summarization. @xcite is the first work in employing rl to train lstm models, without teacher forcing, for abstractive summarization. @xcite provide follow-up works with better reward functions. both works train lstm models without teacher forcing. @xcite were the first to employ rl to a transformer architecture for abstractive summarization. they fine-tune the gpt2-small @xcite model without teacher forcing. however, the omission of teacher forcing led to a training time of 10 days for a generation length of just 10 tokens. this highlights the severity of the problem which arises by omitting teacher forcing and using sequential generation during training, and motivates the need for our approach to effectively incorporate rl. ## modeling in this section we provide the formulation for qfs and passage embedding. section 3.1 discusses the application of policy gradient to qfs. section 3.2 discusses our approach to obtain passage embeddings. we present architectures in appendix g. ## datasets we use three datasets in this work: (a) eli5 @xcite , (b) reliable qfs tester, rqft (our contribution), and (c) reddit passage embedding dataset, rpedt (our contribution). in the following discussion, we provide a brief overview of the eli5 dataset, and move on to discuss rqft in section 4.1, and rpedt in section 4.2. @xcite proposed eli5 for longform question answering (lfqa). it consists of a triplets in the format: <query, document, answer>, where the query is a well-formed, open-ended question, and the answer is not a span from the document. given the nature of the query and the answer, it fits perfectly in our problem statement. the dataset contains 234420, 9930 and 24820 samples for training, validation and testing respectively. ## experiments and results in this section we provide details on our experiments. section 5.1 presents the results obtained on all the experiments. finally, section 5.2 presents the results obtained from human evaluation on rqft. sl represents our implementation of of the model trained by @xcite for eli5. ## analysis our initial analyses on the eli5 test set revealed a recurring problem: our models (all of the 7 models) would copy the query into the generated summary. on further probing, we discovered that this correlated highly with the absence of content in the document relevant to the given query. frequent occurrences motivated us to create a manually curated test set for reliable analyses. in this section we present a detailed analysis of our models. in section 6.1 we present a comparative analysis of all the models, in section 6.2 we analyse the models when multiple queries are posed on the same document and in section 6.3 we analyse our models when the document is replaced with a random document. our analysis reveals that the rl models actually learn qfs as compared to the bart-sl model, which has the tendency to ignore the provided document. this justifies the significant boost (∼ 10 points) in performance. ## conclusion and future work in this work we present reinforcement learning (rl) based models for the task of qfs. we observe that these models perform significantly better than a model, with the same architecture, trained using supervised learning (sl). we also observe that the rl-based models generate summaries with much lesser hallucination than the supervised learning model. additionally, we also resolve the conflict of employing rl in transformers with teacher forcing, by utilizing scheduled sampling. we present a novel reward derived using the cluster hypothesis. through the work, we also contribute two datasets for the community: rpedt, to train passage embedding models, and rqft, a gold standard test dataset for analyses of qfs models. the takeaway from our work is that rl is a better framework to tackle qfs. we observe that rl helps the model learn qfs much better than sl, even with straightforward lexical rewards, such as rouge. additionally, we also conclude, from results, that cluster hypothesis leads to much better semantic feedback than competing passage embedders. training rl models for long horizon (higher generation length) poses challenges in terms of exploration of action space and temporal credit/blame assignment @xcite . in our future work, we would focus on tackling this challenge. our motivation is that it would reduce the convergence time, and provide even better results.