TCS _ WITM _2022 @ D ialog S um : Topic oriented Summarization using Transformer based Encoder Decoder Model (2022)

Abstract: In this paper, we present our approach to the DialogSum challenge, which was proposed as a shared task aimed to summarize dialogues from real-life scenarios. The challenge was to design a system that can generate fluent and salient summaries of a multi-turn dialogue text. Dialogue summarization has many commercial applications as it can be used to summarize conversations between customers and service agents, meeting notes, conference proceedings etc. Appropriate dialogue summarization can enhance the experience of conversing with chat- bots or personal digital assistants. We have pro- posed a topic-based abstractive summarization method, which is generated by fine-tuning PE- GASUS1, which is the state of the art abstrac- tive summary generation model. We have com- pared different types of fine-tuning approaches that can lead to different types of summaries. We found that since conversations usually veer around a topic, using topics along with the di- aloagues, helps to generate more human-like summaries. The topics in this case resemble user perspective, around which summaries are usually sought. The generated summary has been evaluated with ground truth summaries provided by the challenge owners. We use the py-rouge score and BERT-Score metrics to compare the results.

## introduction automatic text summarization is an important task in natural language processing, and it has been studied for decades. while extractive summarization focused on picking up the most important sentences from the text and create a summary, abstractive summarization generates new concise sentences with the important concepts. the task of abstractive summarization thus has two sub-tasks -identifying the important concepts within content and generating new sentences that are grammatically correct and can cover all important concepts sufficiently without repetition or redundancy. both the summarization techniques have received attention from researchers of natural language processing. some of the most cited works in the area of extractive summarization are @xcite , and for abstractive summarization one may refer to @xcite @xcite . however, most of the above-mentioned works has focused on single-speaker documents such as news @xcite , scientific publications @xcite etc. the documents considered also were short and assumed to contain a limited number of concepts around which summaries were to be generated. on demand summarization based on user queries, summarization of multi-section large reports are some of the problems that are currently being explored in the above area. content generated through interaction between two or more speakers is known as a dialogue. dialogues are important forms of communication, which contain lot of information about ideas exchanged and nature of the participants. dialogue summarization aims to condense a piece of content generated by multiple participants into a short passage. dialogues are difficult to summarize since the underlying data contains diverse interactive patterns between speakers as well as inherent topic drifts @xcite . human summarization sometimes focuses only on the content. sometimes gives more attention to the nature of interaction, while at others may be considering both. for example, while summarizing an argument it may be needed to capture the key points made by both the speakers separately and highlight it in the summary. for other scenarios like a customer communication it may be more important to detect dissents, agreements and the topics around which they occur. the difficulty of dialogue summarization stems from the heterogeneity of the model summary #person1# and #person2# talk about who stands out in their mind as a man or woman of sound character. human summary #person1# and #person2# are talking about abraham lincoln. they think he was a noble man. model summary #person1# and #person2# are talking about famous people. they admire abraham lincoln for his great vision, courage, and humility. human summary #person2# admires abraham lincoln for his perseverance, courage and humility. model summary #person1# and #person2# talk about abraham lincoln as a man or woman of sound character. human summary #person1# and #person2# talk about abraham lincoln and his glorious history. they both admire him. dialogue text #person1#: who stands out in your mind as a man or woman of sound character? #person2#: if i think of famous people, i think of abraham lincoln. #person1#: he's the us president, who walked five miles just to give a lady her change, isn't he? #person2#: that's the one. he also was famous for never giving up on his goals. #person1#: that's right. he ran for office quite a few times before he was finally elected. #person2#: and i also admire him for his courage in fighting for equal rights. #person1#: he had great vision, didn't he? #person2#: and humility. i would have liked to meet him personally. topic -sound character topic -famous people topic -discuss abraham lincoln figure 1 : an example of topic focused summarization underlying content. dialogue summarization is an important problem that can be further classified into various sub-areas depending on the nature of input considered such as speech summarization, meeting summarization, chat summarization, email thread summarization and so on. a detailed survey on abstractive summarization is presented in @xcite . in recent times, masked language models using transformers that are basically multi-headed attention-based encoder-decoder models, have created a remarkable impact in the area of of text generation @xcite , and consequently tasks like abstractive summarization which are heavily dependent on it. pegasus is a transformer based model trained on large c4 corpora introduced in @xcite containing 350m web-pages and hugenews dataset which consists of 1.5b articles from news-like website @xcite @xcite @xcite @xcite . its pre-training objectives were set as gap sentence generation (gsg), which was more aligned to the downstream task of summarization, and the model thereby is found to achieve much better and faster performance for abstractive summarization tasks, after some fine-tuning. in gsg, top m principal sentences, which are found to be most similar to the other sentences in the document according to rouge-f1 score, are initially masked while feeding the document to the model. these sentences are concatenated into a psuedo-summary, and the model is trained to generate these using a sequence to sequence generation task. this pre-training objective has pushed forward state of the art model on 12 diverse summarization datasets. it is found to perform exceptionally well on summarization tasks even when very few training samples are available for fine-tuning @xcite in this challenge, a dialogue is found to contain information related to multiple topics. for example, "#person2# arrives late because of traffic jam. #person1# suggests #person2# quitting driving and taking public transport" contains two topics-'reason for being late' and 'benefit of public transport.' since the pre-trained pegasus model includes the salient information from the input text irrespective of user perspective, it can't generate a topicdriven or user-perspective driven summary. the novelty of the proposed approach lies in proposing a new fine-tuning task in which a topic is passed as an input along with the dialogue text, to reformulate the task of dialogue summarization. the incorporation of the topic along with the input and a target summary during training allows for additional training of the model to generate topic-focused summaries. this enhances the quality of summary generated by pegasus in two ways -it learns to focus on different text segments that are centered around a given topic, and then use those portions to pick up the principal sentences. in the current context, the model learnt to focus on text segments that contained the parts of the conversation that were more relevant to the user-perspectives and thereby generated a topical summary. the significance of the proposed model is that the same text can be summarized differently based on the topics given, by focusing on different portions of the text. fig 1 shows a sample dialogue from the test set, humangenerated summaries around different topics and the outputs generated by our system for each of the given topics. this paper is organized as follows: section 2 gives the details of the shared task and the dataset provided. section 3 provides a detailed description of the proposed methodology. section 4 gives the details of baseline models and training parameters. results are discussed in the section 5 which is followed by the conclusion in section 6. ## shared task details and dataset the dialogsum challenge @xcite ) is focused on summarizing real-life dialogues. the task is to generate a fluent, concise, and coherent summary of the multi-turn dialogue text. the di-alogsum dataset @xcite consists of 13, 460 dialogue conversations collected from three datasets viz dailydialog @xcite , dream @xcite , mutual @xcite , and a few dialogues from english-speaking practice websites. this aggregated dataset foot_1 consists of a training set of 12460 dialogues, development set of 500 dialogues, and test set of 500 dialogues, where each dialogue was of average length 120 words. both the training set, and the development set included a topic which usually spans over one to three words, and a human summary whose average length is 19 words. each dialogue in the test set however had three topics and corresponding topic-focused human-generated summaries, which could be used for evaluating the model. a hidden test set with 100 dialogues and one topic each was provided as the actual challenge task. ## the proposed method for a given dialogue text @xmath0, d 2 , ...d n of n words and the topic t of the conversation where @xmath1, t 2 , ..., t k consists of k words, the task is to generate a dialogue summary @xmath2, y 2 , ...y m containing m words. the end goal is to find the summary of a dialogue y * that maximizes the probability p(y|t, d). in order to achieve this objective, we adopt the state of the art pre-trained pegasus model 3 , which was further fine-tuned on the downstream summarization task using the cnn/daily news dataset @xcite . the target fine-tuning task was designed to generate the news highlights from the text. the proposed framework used by us is shown in figure 2 (b) , while the standard one is shown in (a). we have fed the topic along with the dialogue text, where the two are separated by a special character. the target summary was a human input that came as a part of the data-set. it was observed that the topics represented human conceptualization of the content succinctly without borrowing key-words from the dialogue itself, unless necessary. the motivation to use the topic to fine-tune the model was derived from the fact that the test dataset came with three different human summaries, formed around different topics for each dialogue. one such example dialogue with three target summaries are shown in figure1. this clearly indicated that the same conversation could be viewed from different perspectives and hence summarized differently. though humans inherently tend to map any piece of text to topics, a human summarization tends to occur around these topics. in this dataset, the human annotation contained both the topic and the summary, which we could use to train our model in order to obtain better summaries than default pegasus. the idea was that using the topics as input for fine-tuning will be able to generate more topic-oriented summaries, by guiding the model towards sentences that are important for the topic and not by default rouge f1 similarity. since the final hidden dataset also had a topic given, the task could clearly be modeled as one of topic-oriented summarization. however, not all possible summarization scenarios may come with the topics explicitly mentioned, though the need may still be to do topic-focused summarization. the model in that case may be enhanced to identify the key topics first and then use them for summarization. the present dataset may serve as a good source for training a model to identify topics. ## experiments this section describes the different baselines we used for comparison, followed by the training parameters used in these experiments. ## results and discussion when we compared our proposed approach with the baselines, we found that our model outperformed the baselines with significant improvement. rouge-l has increased by 4.62% compared to ft_pegasus. the improvements indicate that fine-tuning of the pegasus model on the dialog-sum dataset and topic relevance helped the model in extracting the essential information from the dialogues. we also computed the average length difference between our outputs and ground-truth summaries as recall depends on the length of generated summary. the average length of our modelgenerated summaries is 22.28 words, which is comparable to the ground-truth summaries, whose average length was 19.99 words. ## conclusion as part of the dialogsum shared task on learning to generate a concise, fluent and topic-oriented summary of dialogues picked up from real-life scenarious, we have enhanced the performance of a pretrained abstractive summarizer model by incorporating the topic along with the input text, to generate a topic-oriented summary. we have shown that the sota pre-trained transformer-based encoderdecoder model pegasus can be fine-tuned using the proposed methodology, to generate more human-like summaries of dialogues. our model performed better in comparison to the baselines. in future, we plan to improve the method further by incorporating nuances of dialogue, speech act