Topic-Informed Dialogue Summarization using Topic Distribution and Prompt-based Modeling (2023)

Abstract: Dealing with multiple topics should be considered an important issue in dialogue summarization, because dialogues, unlike documents, are prone to topic drift. Thus, we propose a new dialogue summarization model that reflects dialogue topic distribution to consider all topics present in the dialogue. First, the distribution of dialogue topics is estimated by an effective topic discovery model. Then topic-informed prompt transfers estimated topic distribution information to the output of encoder and decoder vectors. Finally, the topic extractor estimates the summary topic distribution from the output context vector of decoder to distinguish its difference from the dialogue topic distribution. To consider the proportion of each topic distribution appeared in the dialogue, the extractor is trained to reduce the difference between the distributions of the dialogue and the summary. The experimental results on SAMSum and DialogSum show that our model outperforms state-of-the-art methods on ROUGE scores. The human evaluation results also show that our framework well generates comprehensive summaries.

## introduction in general, text summarization aims to generate a summary by capturing the core meaning from an original document consistently written by one participant on a single topic, such as news, scientific publications, etc @xcite . on the other hand, since a dialogue consists of multi-speakers, the topic of the dialogue may be changed as a topic drift according to the speaker's intentions @xcite . therefore, the dialogue summarization should take into account the distribution of multiple topics in a dialogue and reflect this distribution in generating the summary @xcite ). * corresponding author figure 1 shows an example of dialogue summary generation results from bart @xcite and tidsum models in the samsum dataset @xcite ; bart is a baseline model that has been widely used due to high performance in summary tasks. the content of the example dialogue in figure 1 can be divided into three parts: 1 tina and steve are having pasta for dinner, 2 they will do the shopping together, and 3 they make an appointment to meet in the car park after steve finishes work. that is, we can think that the dialogue contains three topics. however, bart did not capture the most important purpose of tina and steve's appointment to have pasta for dinner. they made an appointment to go shopping for ingredients for pasta in dinner, so this should be included in the summary. therefore, we focus on generating a more comprehensive summary that captures all the topics in the dialogue, without missing an important topic. in this paper, we propose a novel model, topic-informed dialogue summarizer (tidsum), that generates a summary by considering the distribution of topics existing in the dialogue. to estimate the distribution of multiple topics in a dialogue, we exploit the topclus model to obtain the distribution for the input dialogue, which was developed for automatic topic discovery from text corpora @xcite . moreover, a task-specific soft-prompt, namely the topic-informed prompt, is added to make the encoder context vector foot_0 well capture the dialogue topic information in the decoding phase as well as the encoding phase. the topic-informed prompt is created by the latent embedding of the auto-encoder in the topclus model. since the latent embedding of dialogue from the topclus model sufficiently contains the topic information of the dialogue, the topic-informed prompt can influence the context vector of other tokens in the encoder and decoder of summarizer through the encoder self-attention and decoder cross-attention processes. the output context vectors from the decoder are used for the topic extractor to estimate the topic distribution of the generated summary with it. then the topic extractor provides an auxiliary loss function to reduce the difference between the dialogue topic distribution and the summary topic distribution in the training phase. this learning approach generates a better summary that well reflects the topic of the dialogue. in the experiments, two daily dialogue summarization datasets, samsum and dialogsum, were used to evaluate our model. compared to the previous model, the proposed model improved the sota performance by 1.19%p and 1.94%p in rouge-1 for samsum and dialogsum, respectively. ## related work recently, there has been increasing attention on neural summarization for dialogues. current studies mainly apply transformer-based models (e.g., bart @xcite ) to abstractly summarize dialogues. however, these models are pre-trained on generic text corpora and it is essential to finetune them in a specific way for dialog data. many studies have investigated how to find topics in dialogues. @xcite modeled the dialogue as an interactive graph according to the topic word information extracted from lda @xcite . @xcite , which used dialogpt @xcite as the annotator, performed three dia-logue annotation tasks, including keywords extraction, redundancy detection, and topic segmentation. @xcite proposed two topic-aware contrastive learning objectives. this method implicitly modeled the topic change and handled information scattering challenges for the dialogue summarization. since summarizing dialogues is essential in customer services, @xcite proposed a topic-augmented two-stage summarizer with a multi-role-specific topic modeling mechanism. @xcite presented a novel curriculum-based prompt learning and applied a topic-aware prompt, from which we got the idea for a topic-informed prompt. ## topic-informed summary generation framework to perform the dialogue summarization effectively, it is necessary to identify the distribution of topics in the dialogue scattered across multiple utterances. therefore, we propose a model to generate a topic-informed summary by reflecting the dialogue topic distribution to the summary topic distribution. the base architecture of tidsum is a transformerbased auto-regressive language model, bart.