An Exploratory Study on Long Dialogue Summarization: What Works and What’s Next (2021)

Abstract: Dialogue summarization helps readers capture salient information from long conversations in meetings, interviews, and TV series. However, real-world dialogues pose a great challenge to current summarization models, as the dialogue length typically exceeds the input limits imposed by recent transformer-based pre-trained models, and the interactive nature of dialogues makes relevant information more context-dependent and sparsely distributed than news articles. In this work, we perform a comprehensive study on long dialogue summarization by investigating three strategies to deal with the lengthy input problem and locate relevant information: (1) extended transformer models such as Longformer, (2) retrieve-then-summarize pipeline models with several dialogue utterance retrieval methods, and (3) hierarchical dialogue encoding models such as HMNet. Our experimental results on three long dialogue datasets (QMSum, MediaSum, SummScreen) show that the retrieve-then-summarize pipeline models yield the best performance. We also demonstrate that the summary quality can be further improved with a stronger retrieval model and pretraining on proper external summarization datasets.

## introduction large amount of dialogue data have been produced in meetings, tv series, and interviews @xcite @xcite . dialogue summarization aims to generate a short summary for long dialogues to help the readers capture important information more efficiently. a number of existing works on dialogue summarization focus on extracting the main events of a short conversation @xcite . however, unlike the short dialogues which contains less than 20 utterances, some tasks for summarizing much longer dialogues have been proposed recently @xcite . these datasets are usually derived from meetings and interviews, with hundreds of turns in each dialogue. the length of such dialogues typically exceeds the input limits imposed by recent transformer-based models @xcite , making it difficult to train an end-to-end summarization model for such tasks. this poses the challenge: how can we effectively use the current neural summarization models on dialogues that greatly exceed their length limits? additionally, compared with document summarization, dialogues are interactive in nature, makes it more context-dependent and the information in dialogues is more sparsely distributed. besides, the informal language used in dialogues leads to difficulties in modeling relevance and salience. to solve these issues, hierarchical methods are proposed to model the dialogues at turn level @xcite . however, generating a short summary that contains all the salient information remains challenging. in this paper, we systematically investigate these issues on dialog summarization: we first explore the various solutions to the lengthy input problem. then, we analyze and compare the methods to improve generic summarization models on challenging dialogue datasets. to address the long input issue, we investigate extended transformer models such as longformer @xcite , and several dialogue utterance retrieval methods for a retrieve-then-summarize pipeline model, as well as hierarchical dialogue encoding models. for the specific challenges in dialogues, we explore different datasets for pretraining to test the transferability between similar summarization tasks. we evaluate these models on three recent long dialogue summarization datasets: qmsum for meetings @xcite , mediasum for inter-views @xcite , summscreen for tv series transcripts @xcite . in our experiments, we find that the pipeline method with a dialogue utterance retrieval model yields the best performance, and it can be further improved with a stronger retrieval model. our experiment results also suggest that pretraining on proper external summarization datasets can effectively improve the performance of dialogue summarization models. ## related work long sequence summarization recent summarization models are based on transformer @xcite that has a quadratic time and memory complexity with respect to the input length, preventing it from being used for longer sequences. to address this issue, @xcite used the sliding window and global attention, while @xcite used a combination of random, sliding window and global attention mechanism to reduce the quadratic complexity to close-linear. previous benchmarks for long sequence summarization mostly focus on documents instead of dialogues: pubmed and arxiv @xcite consists of scientific papers which are typically very long; billsum @xcite is a corpus of u.s. congressional bills and their summaries; bigpatent @xcite ) contains 1.3 million u.s. patent files and human-written summaries. dialogue summarization dialogue summarization aims to generate concise summaries for dialogues, such as meetings @xcite @xcite @xcite , tv series @xcite , interviews @xcite , and chitchat @xcite @xcite . some summarization datasets (not limited to dialogues) contain queries asking for summarizing specific parts of dialogues @xcite , while others only need to summarize whole dialogues @xcite @xcite . as for dialogue summarization models, @xcite described a hierarchical model for both inner-and cross-utterance attention, while chen and yang (2020) proposed a multi-view decoder to leverage different extracted views of dialogues, such as topic view and stage view. ## methodology in this section, we will introduce the dataset used to evaluate and pretrain the model, two types of summary models, and the details of the experiment setup. ## result and analysis here we demonstrate our findings in four corresponding subsections. we also show some concrete examples and perform qualitative analysis in § 4.5 ## conclusion and future work we first explore the lengthy input problem of dialogue summarization through experiments on transformers and retrieval models. we conclude that the retrieve-summarize pipeline results in the best performance. then, the experiments demonstrate the important role of queries and robustness to input length for different types of models. we found that adding a single query sentence in the input greatly improves rouge scores on qmsum. additionally, bart performs worse when the input is beyond 512 tokens, even with extended positional embeddings; on the contrary, the hierarchical model performs better for longer inputs. we also test the transferability of summarization datasets by pretraining the language model on similar tasks. we conclude that the bart-large model pretrained on cnn/dm yields the best performance on both qmsum and summscreen-fd. for future work on solving the long input problem, we found that using an utterance retrieval model for summarization is a promising direction, yet modeling relevance between query and dialogue utterances remains a challenging task. and for the summarization models, it is worth exploring methods to 1) pretrain on valuable datasets for dialogue summarization, 2) better fuse the queries into the neural models, and 3) make the model robust to the input length (like hmnet) and maintain the high performance in the meantime (like bart).