Inverse Reinforcement Learning for Text Summarization (2023)

Abstract: We introduce inverse reinforcement learning (IRL) as an effective paradigm for training abstractive summarization models, imitating human summarization behaviors. Our IRL model estimates the reward function using a suite of important sub-rewards for summarization and concurrently optimizes the policy network. Experimental results across datasets in different domains (CNN/DailyMail and WikiHow) and various model sizes (BART-base and BART-large) demonstrate the superiority of our proposed IRL model for summarization over MLE and RL baselines. The resulting summaries exhibit greater similarity to human-crafted gold references, outperforming MLE and RL baselines on metrics such as ROUGE, coverage, novelty, compression ratio, factuality, and human evaluations.

## introduction most fine-tuned abstractive summarization systems @xcite are trained using maximum likelihood estimation (mle) and the negative log-likelihood (nll) loss. previous research has demonstrated that mle training possesses certain disadvantages: (1) object mismatch @xcite , where the nll loss concentrates on word-level matches, neglecting token rearrangement and paraphrases; (2) exposure bias @xcite , the discrepancy between training and inference regarding reference tokens. to address these issues, reinforcement learning (rl), which optimizes policy networks by directly maximizing the discrete reward, has emerged as an alternative training paradigm for summarization @xcite . typically, rl-trained summarization models require a predefined reward function and a common practice @xcite is to use rouge @xcite . rouge-base reward does not, however, consider * corresponding author. in contrast to rl, we argue that inverse reinforcement learning (irl) may be more suitable for text summarization. irl focuses on estimating an agent's reward function based on their observed behavior, rather than predefining it @xcite . consequently, irl can be advantageous in situations where the reward function is not explicitly known @xcite or challenging to define through interactions @xcite . our experimental results suggest that by using irl to automatically learn weights over combined summarization subrewards and imitate human generations/expert demonstration, we can jointly optimize the reward function and policy network, yielding superior summaries as measured by both automatic and human evaluations. more specifically, inspired by @xcite ; @xcite , we integrate irl into text summarization, which estimates the reward function for summarization and optimizes the model accordingly. by employing irl, we gain the ability to dynamically learn the weights of various sub-reward components crucial to the summarization task based on the training data. once the subrewards are defined, the training process with irl consists of two alternating phases: the reward update phase, which focuses on learning the reward function, and the policy update phase, which aims to optimize the model to maximize the reward. figure 1 presents an overview of our approach. compared to the models trained with mle or rl, our empirical results on multiple summarization datasets and different model sizes suggest that the irl-trained agent produces summaries that are significantly more similar to the human-written reference summaries across multiple evaluation metrics, including rouge, coverage, novelty, and compression ratio. in addition, although we did not explicitly incorporate faithfulness as a sub-reward during training, our empirical results indicate that our models exhibit a higher level of abstraction, with a notably lower decline rate in hallucination. this particular characteristic closely aligns with the behavior observed in human-generated summaries. our contributions can be summarized as follows: ## related work summarization extensive research has been conducted on fine-tuned deep learning approaches for both abstractive summarization @xcite @xcite and extractive summarization @xcite . in spite of the differences in generation paradigms, these models trained with the nll loss have similar limitations, including exposure bias and objective mismatch @xcite . to address these challenges, reinforcement learning has emerged as a popular alternative for training abstractive @xcite @xcite and extractive summarization systems @xcite @xcite . rl-based approaches are designed to optimize a discrete reward, often chosen heuristically, such as rouge. for instance, @xcite propose a reward function specifically tailored for consumer health question (chq) datasets, while @xcite construct a reward function based on textual entailment and coverage of summaries in semantic spaces. however, manually designing reward functions can limit their generalizability beyond specific domains. pa notable work similar to ours is the model introduced by @xcite , which focuses on learning reward functions rather than manually selecting them for policy optimization. in contrast to our approach, they train neural networks to learn directly from human ratings instead of expert demonstrations, which may result in lower interpretability. ## method this section presents an overview and formulation of summarization, along with a discussion of reinforcement learning (rl) and our approach to training summarization models using inverse reinforcement learning (irl). problem formulation the task of abstractive summarization can be viewed as a conditional language generation task. given a source document @xmath0, x 2 , . . . , x n } with n tokens, the summarization task involves learning a conditional probabilistic model p θ (y|x) that produces a summary @xmath1, ..., y |y| ), where y i is chosen from a predefined vocabulary v and θ denotes the parameters of the summarization model. p θ (y|x) can be further decomposed into the product of conditional probabilities for each token based on the previously generated context: generally, p θ in eqn. (1) is trained using the maximum likelihood estimation (mle) objective with teacher forcing @xcite , which aims to maximize the likelihood of the human-written reference summary y * = (y * 1 , ..., y * |y| ): reinforcement learning due to exposure bias @xcite in mle with teacher forcing, rl has emerged as an alternative for training neural summarization models @xcite . it offers the advantage of directly optimizing discrete metrics such as rouge @xcite , which considers a certain degree of flexibility for token rearrangement and paraphrasing. in rl training, models typically require predefining the reward r. the reward function r(y) is established by comparing the output sequence y with the ground truth sequence y * using evaluation metrics such as the average of rouge -1, -2, -l, and f1 scores with respect to the gold references @xcite . the objective of rl is to learn a policy that maximizes the predefined discrete metric: ## analysis this section provides in-depth analyses of the proposed framework from different perspectives, including faithfulness, coverage, entity-level analysis and weight analysis. ## conclusion we introduce inverse reinforcement learning into text summarization and demonstrate the efficiency of this method. using irl, we can train a policy to better match human behaviors by learning from expert demonstrations. our experimental results in-dicate that irl can improve the summary quality in a variety of measures, including rouge, novelty, coverage, compression ratios, and factuality. thus, our empirical results suggest that irl can better fit into the goal of summarization, in addition to providing more interpretable training.