Dialogue Summarization using BART (2022)

Abstract: This paper introduces the model and settings submitted to the INLG 2022 DialogSum Chal- lenge, a shared task to generate summaries of real-life scenario dialogues between two peo- ple. In this paper, we explored using interme- diate task transfer learning, reported speech, and the use of a supplementary dataset in addi- tion to our base fine-tuned BART model. How- ever, we did not use such a method in our final model, as none improved our results. Our final model for this dialogue task achieved scores only slightly below the top submission, with hidden test set scores of 49.62, 24.98, 46.25 and 91.54 for ROUGE-1, ROUGE-2, ROUGE-L and BERTSCORE respectively. The top submitted models will also receive human evaluation.

## introduction dialogue summarization is a variation of text summarization which aims to generate concise, coherent summaries of conversations. dialogue summarization requires far deeper insight than summarizing a news article or similar documents, as is done in text summarization. when handling a dialogue, a model must address semantic roles, resolve definite pronouns and coreference, and handle various other complexities @xcite . we investigate the best methods for summarizing a dialogue while retaining these difficult relations that do not present a problem when summarizing a simple text. the inlg 2022 dialogsum challenge is a shared task with the goal of generating summaries of real-life scenario dialogues between two people. in this paper, we will describe our approach to this task using a fine-tuned bart model. additionally, we explore the effects of using intermediate task transfer learning, reported speech for this task. however, we did not use such a method in our final model, as none improved our results. ## background the field of text summarization has been in focus for decades. research into automatic text summarization began as early as 1958 with the summarization of magazine articles and technical papers @xcite . text summarization proves challenging for many reasons. the model must be able to identify important topics and condense them in a way that is not redundant, but yet remains readable and cohesive @xcite . primarily, there are two approaches to text summarization: extractive and abstractive. extractive summarization seeks to extract the most important information and present it as is. abstractive summarization, in contrast, may use novel words to create a linguistically correct condensed representation @xcite . originally, research on extractive summarization was in the foreground @xcite but the field is now moving towards abstractive summarization based on neural sequenceto-sequence encoder-decoder models @xcite . top performing models to create summaries have also been based on transformers @xcite . pointer-generator models @xcite are another state-of-the-art summarization technique, combining extractive and abstractive methods. dialogue summarization is now emerging as a new interest in the field of natural language processing. as early as 2010, higashinaka et al. were exploring methods of extractive summarization to summarize contact center dialogues using a hidden markov model called class speaker hmm. since then, more unique and effective methods have emerged. @xcite proposed a scaffold pointer network (spnet), which incorporated three types of semantic scaffolds found in dialogue: speaker role, semantic slot, and dialog domain. chen and yang (2020) introduced a multiview sequence-to-sequence model, which utilized conversational structures and topic segmentation to assist in better dialogue summarization. bart is a sequence-to-sequence model that pre-trains by combining bidirectional and auto-regressive transformers, and achieves good results on a range of abstractive dialogue and summarization tasks @xcite . @xcite found bart to be a viable base model for dialogue summarization and showed additional methods could improve results. for this reason, we selected bart as our base model. ## system overview and methods in this section, we discuss the setup and hyperparameters of our final model, as well as attempts to improve our results, which included using intermediate task transfer learning, reported speech, and an additional dataset. ## results many of the generated summaries produced were close matches to the target summaries. sometimes generated summaries seemed as though they were a good summarization of the dialogue, but nonetheless had low rouge scores. in some cases, this was due to length discrepancies. in other cases, our model generated novel word choices which varied from the gold standard. examples of a high scoring target #person1# tells kate that masha and hero get divorced. kate is surprised because she thought they are perfect couple. generated #person1# tells kate masha and hero are getting divorced. kate is surprised because she thought they are the perfect couple. target #person1# and mike are discussing what kind of emotion should be expressed by mike in this play. they have different understandings. generated #person1# thinks mike is acting hurt and sad because that's not how his character would act in this situation, but #person2# thinks jason and laura had been together for 3 years so his reaction would be one of both anger and sadness. the results were evaluated on rouge-1, rouge-2, rouge-l and bertscore. rouge scores measure the n-grams shared between the generated and target summaries. rouge-l measures the longest shared n-gram. bertscore looks at contextual embeddings instead of exact matches to give a similarity score @xcite . our model performed comparable to current leaderboard results on the public test set, and also shows what seem to be respectable results on the hidden test set. our scores can be found in our attempts utilizing intermediate task transfer learning, reported speech, and additional datasets all proved unsuccessful. we hypothesize this is a result of insufficient hyperparameter tuning or training. when more complexity is introduced in a model, it often requires specific hyperparameter tuning to result in success, and we suspect this may be one reason our attempts failed. ## conclusion in this paper, we have described our attempt at the inlg 2022 dialogsum challenge shared task, aimed at generating summaries of real-life scenario dialogues. we utilized a fine-tuned bart model trained on the dialogsum dataset provided to us to achieve our best results. we explored utilizing intermediate task transfer learning to improve our model, however we speculate that this failed due to a domain mismatch in the r1 r2 rl bertscore public 47.29 21.65 45.92 92.26 hidden 49.75 25.15 46.50 91.76 our results show that it is possible to achieve relatively successful dialogue summarization results using only a basic bart model and fine-tuning on this dataset. in the future, we would further explore the methods we described above.