Post-Training Dialogue Summarization using Pseudo-Paraphrasing (2022)

Abstract: Previous dialogue summarization techniques adapt large language models pretrained on the narrative text by injecting dialogue-specific features into the models. These features either require additional knowledge to recognize or make the resulting models harder to tune. To bridge the format gap between dialogues and narrative summaries in dialogue summarization tasks, we propose to post-train pretrained language models (PLMs) to rephrase from dialogue to narratives. After that, the model is fine-tuned for dialogue summarization as usual. Comprehensive experiments show that our approach significantly improves vanilla PLMs on dialogue summarization and outperforms other SOTA models by the summary quality and implementation costs.

## introduction dialogue summarization is a specialized summarization task that takes a series of utterances from multiple speakers in the first person as input, and outputs fluent and concise summaries in third persons as shown in figure 1 . different from previous monologue inputs such as news @xcite and scientific publications @xcite , dialogues are always less well-organized. they usually contain complicated reference relations, inconsecutive inter-utterance dependencies, informal expressions, and so on, making dialogue summarization a more challenging task. the most obvious characteristic of this task is the difference in the format and language styles between dialogue and its narrative summary. @xcite mentioned that coreference resolution models trained on general narrative text underperforms by about 10% on dialogue corpus, demonstrating the inherent gap between dialogue and narrative text. as a result, popular plms such as bart @xcite and pega-sus @xcite which excel on news summarization perform mediocrely on dialogue summarization. to narrow this gap, previous work on dialogue summarization mainly resort to injecting dialogue features into plms to enhance dialogue understanding. these features include dialogue acts @xcite , topic transitions @xcite , coreference relations @xcite , discourse graphs @xcite , etc, leading to the rule-based conversion from dialogues to plain text @xcite . however, they suffer from three weaknesses. first, collecting or extracting these features becomes an additional step in the summarization pipeline, complicating the inference procedure at runtime. second, oracle feature labels are hard to collect and errors can propagate from wrong labels to poor summaries. third, additional layers or more encoders are required to incorporate features into plms, increasing the gpu memory footprint both during training and inference. a more natural way to bridge this gap is to give the model more dialogue-narrative pairs to train on. due to the scarcity of dialogue summarization data, one approach @xcite is to convert other text summarization pairs into dialogue to summary pairs via some template, but such work requires additional data 2 . in this paper, we propose an alternative approach that doesn't use any more data than the original dialogue summarization dataset. we convert each existing data pair into many "pseudo-paraphrase" pairs between a dialogue and a narrative sentence. then we post-train a pre-trained seq2seq language model using a prefix-guided generation (pgg) task on the augmented paraphrase dataset. after that, the post-trained model is further fine-tuned as usual for dialogue summarization. to this end, no human efforts on crafting complicated rules or hyper-parameter tuning, or additional memory costs, as well as additional training data, is required. in sum, our contributions are: ## approach the training of a dialogue summarization model is divided two stages: post-training and fine-tuning. the model can be any seq-to-seq plms and it remains unchanged except for the parameters which are updated stage by stage. we will elaborate on the post-training stage in the rest of this section. ## evaluation we first present the experimental setups, then conduct an ablation study to determine the proper prefix in pgg training, before our main results. more implementation details are in appendix b. ## conclusion we propose to post-train dialogue summarization models to enhance their cross-format rephrase ability by prefix-guided generation training on dialogue-sentence pseudo-paraphrases, and get promising results. creating self-supervised tasks for cross-format post-training and incorporating compatible features for downstream fine-tuning are plausible future directions.