Human-in-the-loop Abstractive Dialogue Summarization (2023)

Abstract: Abstractive dialogue summarization has received increasing attention recently. Despite the fact that most of the current dialogue summarization systems are trained to maximize the likelihood of human-written summaries and have achieved significant results, there is still a huge gap in generating high-quality summaries as determined by humans, such as coherence and faithfulness, partly due to the misalignment in maximizing a single human-written summary. To this end, we propose to incorporate different levels of human feedback into the training process. This will enable us to guide the models to capture the behaviors humans care about for summaries. Specifically, we ask humans to highlight the salient information to be included in summaries to provide the local feedback, and to make overall comparisons among summaries in terms of coherence, accuracy, coverage, concise and overall quality, as the global feedback. We then combine both local and global feedback to fine-tune the dialog summarization policy with Reinforcement Learning. Experiments conducted on multiple datasets demonstrate the effectiveness and generalization of our methods over the state-of-the-art supervised baselines, especially in terms of human judgments.

## introduction abstractive conversation summarization, which aims at processing, organizing, and distilling human interaction activities into natural, concise, and informative text @xcite , is one of the most challenging and interesting tasks in text summarization. growing attention has been paid to neural abstractive conversation summarization through a variety of designs including transferring document summarization models @xcite @xcite , utilizing conversational structures @xcite @xcite @xcite @xcite , introducing conversational data augmentation @xcite , incorporating controllable signals @xcite and pre-training conversation models @xcite . most of them are trained with supervised learning, which maximizes the log probability of human written summaries. while they have gained impressive performances, there are still huge gaps in generating high-quality summaries as determined by humans such as coherence or faithfulness @xcite , largely due to a misalignment between the fine-tuning objective (maximizing the likelihood of single human-written summary) and the actual needs (generating more human-favored summaries) @xcite . to train the summarization models on objectives that can more closely capture the behaviors humans care about, reinforcement learning (rl) has been used to directly optimize the rewards learned and constructed from human feedback @xcite @xcite . different kinds of feedback have been explored to construct the reward functions such as human ratings over cnn/dm summaries @xcite , overall preferences among pairs of summaries @xcite , and the similarity-redundancy matrix @xcite . while achieving promising performances, they are mainly designed for document summarization with a single reward function learned from overall assessments on summaries @xcite . as a result, they might not be directly adapted to dialogue summarization because of the intrinsic differences between documents and conversations. compared to documents, conversations are generally less structured and more complex @xcite . there are diverse interactions between multiple speakers and complex structures such as interruptions, discourse relations, and speaker roles in dialogues @xcite . therefore, more subtle levels of human feedback with the consideration of conversation structural information is needed to provide more comprehensive, consistent, and generalizable rewards, which may lead to better performances for dialogue summarization. to fill in this gap, we introduce human-in-the-loop (hitl) abstractive dialogue summarization with different levels of human feedback to leverage various conversation structures. specifically, we incorporate two levels of human feedback: (1) local feedback, which consists of highlighted words or phrases in dialogues to capture the salient structural information, including but not limited to speaker's intents, identifiable events/topics, and discourse relations (e.g., causal relationships and important emotions), and (2) global feedback, which includes dimensions like coherence, accuracy, coverage, concise and the overall quality, to provide more comprehensive human preferences on the given summary. we hire and train human annotators to provide the introduced two levels of human feedback on 1,000 randomly sampled conversations from the dialogsum dataset @xcite . with the collected human feedback, we construct the local reward (r l ) based on the similarities between the generated summaries and the annotated highlights and learn the global reward (r g ) models via supervised learning which predict the human preferences. finally, we train the summarization policy via rl to maximize the rewards predicted by r l and r g . specifically, the policy generates a token of text at each time step and is updated using the ppo algorithm @xcite based on the reward given to the entire generated summary. we conducted extensive experiments and ablation studies in different settings on the recent conversation summarization dataset, di-alogsum @xcite and samsum @xcite , to demonstrate the superiority of our methods compared to the state-of-the-art supervised learning baselines, especially in terms of human judgments and generalization abilities. to summarize, our contributions are: (1) we introduced and collected the local and global feedback tailored for abstractive conversation summarization; (2) we designed the hitl to learn better conversation summarization policies via rein-forcement learning where different levels of human feedback are directly optimized; (3) we performed extensive experiments to study the effectiveness and generation abilities of our hitl methods on dialogsum and samsum datasets. ## methods in this section, we introduce our human-in-the-loop conversation summarization (hitl) pipeline (in figure 1 ) where we incorporate two levels of human feedback, the local and global feedback, into the learning process. inspired by @xcite , our pipeline for abstractive conversation summarization includes 3 stages: (1) collecting two levels of human feedback from conversationsummary pairs where summaries are generated with baseline models; (2) learning and designing reward models from two levels of human feedback;