Improving Summarization with Human Edits (2023)

Abstract: Recent work has shown the promise of learning with human feedback paradigms to produce human-determined high-quality text. Existing works use human feedback to train large language models (LLMs) in general domain abstractive summarization and have obtained summary quality exceeding traditional likelihood training. In this paper, we focus on a less explored form of human feedback – Human Edits. We propose Sequence Alignment (un)Likelihood Training (SALT), a novel technique to use both the human-edited and model-generated data together in the training loop. In addition, we demonstrate simulating Human Edits with ground truth summaries coming from existing training data – Imitation edits, along with the model-generated summaries obtained after the training, to reduce the need for expensive human-edit data. In our experiments, we extend human feedback exploration from general domain summarization to medical domain summarization. Our results demonstrate the effectiveness of SALT in improving the summary quality with Human and Imitation Edits. Through additional experiments, we show that SALT outperforms the conventional RLHF method (designed for human preferences) – DPO, when applied to human-edit data. We hope the evidence in our paper prompts researchers to explore, collect, and better use different human feedback approaches scalably.

## introduction large-scale language model pretraining has become increasingly prevalent to achieve high performance on various natural language processing (nlp) tasks @xcite @xcite @xcite . when applying these models to a specific task, they are usually finetuned to maximize the likelihood of human-written text. while this strategy has led to markedly improved performance in many metrics, models still cannot consistently produce human-determined high-quality output. the nlp community has pointed out some key drawbacks of traditional finetuning. first, important errors (e.g. hallucinations) and unimportant errors (e.g. minor grammar errors) equally contribute to the final loss. second, the model weighs the loss equally on all labeled data of different types, qualities, and difficulties. third, distribution shifts in new data degrade performance (catastrophic forgetting) @xcite . ## related work most directly related to our work is research on automatic clinical note generation from doctorpatient conversations @xcite @xcite @xcite @xcite , and the difference is that those works focus on training a summarization model with pre-labeled data, while we focus on using hf further to improve the summary quality of the trained models. previous work used hf to train summarization models with reinforcement learning (rl) @xcite @xcite and used gpt-2 and gpt-3 to optimize hf across various summarization tasks. these rlbased methods focus on training a reward function through hf data and use such rewards as training objectives by comparing different summaries (rlhf). recently, some rlhf variants collect or use rewards more flexibly and stably @xcite @xcite . we introduce unlikelihood training as an additional learning objective in supervised learning. our technique aims to decrease the probability of unlikely sequences, defined as those which appear in the s ai but not in s e , and increase the probability of verified sequences, which are in s ai and reinforced by s e , as well as novel sequences which do not appear in s ai but do appear in s e . unlikelihood training @xcite involves adding unlikelihood loss to lower the probability of negative candidates. previous work has explored many scenarios with various negative candidates for unlikelihood training, including: style transfer @xcite , repetition, copying, and contradictions @xcite , factuality @xcite ), text degeneration @xcite , and clinical summarization @xcite . in this work, we align the s e with s ai to identify negative candidates and train different tokens with unlikelihood and likelihood loss. we also show that our experiments on human edits can be extended to imitation edits to reduce the need for hf data which can be expensive to get. ## methods the user edits this summary from s ai to s e , where s @xmath0, y 2 , y 3 , ...y lens e ]. we aim to update parameters in m based on both s ai and s e . let, lenu , lens ai , and lens e be the number of tokens in u , s ai , and s e respectively. ## experiments we use the following symbols: 1. m refers to models that are trained and al ## conclusion in this work, we explore improving language models with human edits feedback, which can be collected scalably than others. specifically, we propose the salt training objective based on sequence alignment and unlikelihood training and show how to design imitation edits to reduce the need for expensive hf. we further show on human edits data, salt performs better than a straightforward rlhf (dpo) approach.