Narrate Dialogues for Better Summarization (2022)

Abstract: Dialogue summarization models aim to generate a concise and accurate summary for multi-party dialogue. The complexity of dialogue, including coreference, dialogue acts, and inter-speaker interactions bring unique challenges to dialogue summarization. Most recent neural models achieve state-of-art performance following the pretrain-then-finetune recipe, where the large-scale language model (LLM) is pretrained on large-scale single-speaker written text, but later finetuned on multi-speaker dialogue text. To mitigate the gap between pretraining and finetuning, we propose several approaches to convert the dialogue into a third-person narrative style and show that the narration serves as a valuable annotation for LLMs. Empirical results on three benchmark datasets show our simple approach achieves higher scores on the ROUGE and a factual correctness metric.

## introduction online dialogues are increasingly important in the modern working environment, emphasizing the need for an automatic system to generate concise and accurate summaries. neural dialogue summarization has become an emerging research direction in recent years @xcite with the creation of several benchmarks @xcite @xcite @xcite . most works utilized large-scale language models (llm) and finetune it on downstream dialogue summarization datasets. despite the strong generalization power of llm on summarization tasks @xcite , dialogue summarization holds some unique challenges @xcite . first, dialogues and their summaries are in different language styles. this discrepancy requires the summarization model to complete the tasks of both style transfer and summarization. the domain discrepancy also exists between the pretraining and finetuning stages because llms are often pretrained on the web corpus where the majority of text is in written language. the dialogues, on the other hand, are in spoken language. second, the amount of training data is generally smaller than news summarization. for instance, the widely used samsum dataset @xcite for dialogue summarization contains about 16k annotations, while the cnn/dailymail dataset @xcite is a magnitude larger at size of 300k. third, dialogues contain complex dialogue acts with frequent topic changes and event occurrences. news or academic articles follow certain patterns in writing and could be exploited for summarization @xcite . on the other hand, key information is scattered in the dialogues. to generate comprehensive summaries, a model needs to identify salient information across the dialogue and rephrase the terms. to address these challenges, existing works incorporate external models or tools to help with the dialogue summarization @xcite @xcite . however, they either annotate the dialogues on the token or utterance level, such as coreference resolution @xcite , personal named entity @xcite , utterance intent @xcite , or redundant utterance identification @xcite . in this work, we propose a dialoguelevel annotation describing "what happened in the dialogue?" in natural language. the narrated dialogues effectively close the domain gap between pretrain and finetune, as well as between dialogues and summaries. it also helps with data sparsity since the model could effectively transfer knowledge from out-of-domain annotations such as news summarization datasets. the narrative description of dialogues serves as an effective annotator to label coreference, dialogue acts, events, and user intents. since the narration are in natural language just as the dialogues themselves, we simply replace original dialog instruction bootstrap @xmath0. nickola: have you found it? 2. sophie: no! still looking :( 3. nickola: check pockets and handbags. 4. sophie: checked them all twice already... convert the dialog to narrative and formal style for each turn: 1. nickola asked if sophie had found it. 2. sophie said that she had not found it yet and was still looking. 3. nickola told sophie to check her pockets and handbags. 4. sophie said that she had already checked them all twice. <same dialogue transcript as @xmath1: this is a conversation between two people who are looking for something. nickola suggests that sophie check her pockets and handbags again, and sophie says that she has already checked them twice. <same dialogue transcript as @xmath2: example of the prompt for dialog-to-narrative conversion with instructgpt model. the narrative types are rephrase, plot and event from the top to bottom. or concatenate the narration with the dialogue text. as as result, our frame can be plug-and-play with most llms without any modification on the model architecture. we empirically verified the effectiveness of different narrating methods on three benchmark datasets in both supervised and zero-shot settings. our best narrating method, despite being very simple, outperforming complex and strong baselines on both rouge @xcite and factcc (kryscinski et al., 2020) scores. ## methods we define a dialogue d to be a sequence of turns, where the i-th turn consists of a speaker s i and a utterance u i : a turn in a dialogue is usually represented in the textual format as follows: "s i : u i ", where a colon is used to separate the speaker from the utterance. the format has been treated as the default for some widely used datasets. to better format the turns in a dialogue so that they will be closer to a narrative sentence, we convert the style of a turn using two categories of methods: rule-based transfer and model-based transfer. after conversion, we follow the standard finetune strategy of large-scale language models on the converted dialogue and the original summarization. ## experiment we conduct through experiment on three dialogue summarization datasets: samsum @xcite , dialogsum @xcite , and adsc @xcite under both supervised and zeroshot settings. the properties and statistics of the datasets used in our experiment is shown in ## related work to alleviate the domain mismatch and label scarcity, pretraining on dialogue or news domains @xcite @xcite , multi-tasking @xcite , and data argumentation @xcite have shown to be effective for summarization. other works focus on the structured information of dialogues and model the auxiliary input via graph attention network @xcite or the manipulation of transformer attention @xcite . the internal structured information includes speaker-utterance relationship @xcite , semantic slot @xcite , topic @xcite , and coreference @xcite . and the external structured information includes commonsense graph from knowledge bases @xcite . another line of work addresses the challenges of dialogue summarization by directly injecting knowledge into model input. for instance, @xcite uses personal named entities to control the occurrence of speakers in the summary generation. @xcite make use of dialog-gpt @xcite to annotate keywords, redundant utterance, and topic changes in the dialogue transcript. @xcite use external tools to annotate speaker intent and key phrases as a "sketch" of the dialogue. our work falls in this category but differs from previous approaches as we attempt to incorporate external knowledge by directly narrating the dialogues in natural language. ## conclusion we propose a general framework to narrate a dialogue into a third-person description for dialogue summarization. we empirically compare different ways of narration and found that the proposed framework improves the performance, especially the factual correctness of the generated summary, for both supervised and zero-shot settings in three benchmark datasets. the improvement is most consistent when including the salient events in the dialogue as narrative. the resulting summarization model surpasses existing strong baselines on samsum and dialogsum datasets.