Make The Most of Prior Data: A Solution for Interactive Text Summarization with Preference Feedback (2022)

Abstract: For summarization, human preferences is critical to tame outputs of the summarizer in favor of human interests, as ground-truth summaries are scarce and ambiguous. Practical settings require dynamic exchanges between humans and AI agents wherein feedback is provided in an online manner, a few at a time. In this paper, we introduce a new framework to train summarization models with preference feedback interactively. By properly leveraging offline data and a novel reward model, we improve the performance regarding ROUGE scores and sample-efficiency. Our experiments on three various datasets confirm the benefit of the proposed framework in active, few-shot and online settings of preference learning.

## introduction the advent of ai has changed business practices, though the human involvement is still important. the human roles in interaction with ai-powered machines have been evolving under the concept of human-in-the-loop (hitl) @xcite . hitl allows humans to actively participate in supervising ai systems by approving, rejecting, or re-labeling current outputs, and providing expert-guided advices to the system. it will also act as the unique source of external knowledge from humans. by observing the outputs of ai systems, humans can hand-pick some potential outcomes and then feedback to the models for better performance. in nlp, document summarization is considered as a subjective task @xcite . they argued that it would be hard to quantify what makes a "good summary" without the human judgment input. collecting human feedback and evaluating the crafted summaries from documents for building the training datasets is time-consuming (wu et al., * these authors contributed equally to this work. 2021) and labor-expensive. it is true particularly where the domain knowledge is required. moreover, @xcite argued the importance of the user's intentions for modeling natural language understanding with high performance. given these attributions, we are more interested in deploying an interactive hitl-based text summarization framework, which continuously collects the user-feedback to consequently improve model prediction robustness. here, the user's intention is implicitly acknowledged as a factor influencing the extraction of important sentences from the source documents. upon this formulation, the ai model will be trained with human-produced summaries and adapted as more human-feedback is fed in. previous studies used human feedback to rank the label of objects @xcite , which employed reinforcement learning to minimize the user's effort to provide feedback for training a ranker. however, these approaches rank the entire solution space including relevant and irrelevant pairs, which are a waste of computing power. to tackle this shortcoming, @xcite employed bayesian optimization, which substitutes the standard uncertainty-based acquisition functions for active learning. however, the model still consumes a lot of computing power for a larger number of iterations, and is vulnerable to the curse of dimensionality of input data. recently, researchers proposed to learn a reward model simulating human preferences @xcite . the reward model was then used to translate real-time human feedback to the reward score for fine-tuning the model under rl training. the method was designed for online learning of language models and required numerous interactions to achieve good performance. however, how to achieve sample-efficiency is still an open question. in this paper, we propose a novel interactive pref-erence learning for the summarization task. to do that, we fine-tune a pretrained extractive summarizer as the backbone with reinforcement learning by using a reward model that enforces the distancebased order of preferences. the reward model is trained to differentiate two summaries regarding the topic, length, and quality (human preferences). to enable sample-efficiency, we propose to utilize offline data, which was previously used to pretrain the model. we show that naively using the offline data is harmful for preference learning. instead, we introduce two mechanisms to selectively sample offline data in favor of human feedback learning. our sampling strategies focus on low-rewarded samples or documents which are similar to fine-tuning data. we demonstrate that our method can be used in various settings: active, few-shot and online learning. tested on three summarization datasets, our method consistently achieves significantly better results compared to competitive baselines in each setting. in summary, our contribution is three-fold: ## conclusion this work proposes a novel approach for learning reward functions in preference learning. by utilizing offline data with a reward model that focuses on the distance between the summary and the document, our method improves the rouge-scores by 2 -5% in comparison to the april framework, while beating the random baseline by a large margin on three different datasets. our experimental results also suggest that by applying low-reward sampling or document-similarity sampling, we can achieve efficiency in terms of both running time and the number of human interactions. regarding limitations, our method is not tested with large online stream data, which may cause catastrophic forgetting. future work will confirm our model's effectiveness for abstractive summarization.