A Multi-Task Learning Approach for Summarization of Dialogues (2022)

Abstract: We describe our multi-task learning based ap- proach for summarization of real-life dialogues as part of the DialogSum Challenge shared task at INLG 2022. Our approach intends to im- prove the main task of abstractive summariza- tion of dialogues through the auxiliary tasks of extractive summarization, novelty detection and language modeling. We conduct extensive experimentation with different combinations of tasks and compare the results. In addition, we also incorporate the topic information provided with the dataset to perform topic-aware sum- marization. We report the results of automatic evaluation of the generated summaries in terms of ROUGE and BERTScore.

## introduction much of the early works on summarization devoted attention to monologues such as news articles @xcite , patents @xcite , wikipedia articles @xcite , scientific research papers @xcite , government reports @xcite and even court judgements @xcite . but more recently, the focus of the summarization community has started shifting from monologues to dialogues largely owing to the rising popularity of chatbots, personal assistants, instant messaging platforms and online meetings. while monologues are characterised by the fact that they are authored by a single person, a dialogue involves the utterances of more than one participant (which alone can make them inherently more difficult to summarize). however, the available dialogue summarization datasets @xcite @xcite are fewer in number, limited in scale, domain-specific and sometimes even extremely noisy and semi-structured @xcite as compared to the datasets available for monologue texts. to mitigate these issues a high-quality largescale dialogue summarization dataset named di-alogsum was released by @xcite . the dataset consists of a wide variety of task-oriented dialogues from daily-life conversations. one sample dialogue and its corresponding summary from dialogsum's training set is presented in figure 1 , which is a conversation between a doctor and his patient on the topic of getting a check-up. to further encourage research in dialogue summarization, the authors proposed a shared task named dialogsum challenge @xcite as part of inlg 2022, and in this article, we describe our submission to the shared task as team iitp-cuni. specifically, we attempt to tackle the problem of abstractive dialogue summarization through the use of a mutli-task learning model @xcite @xcite based on transformers @xcite . we intend to improve the main task of abstractive summarization of the dialogues through the auxiliary tasks of extractive summarization, novelty detection and language modeling. additionally, we also explore the usefulness of topic-aware summarization, as in the dialogsum dataset, topics are provided along with the summaries (see figures 1 and 2 ). the rest of the paper is organised as follows. related work is presented in section 2. the dialog-sum challenge is described in details in section 3. section 4 presents our system. results and discussion are in section 5. finally, the conclusion is drawn in section 6. ## related work in this section, we discuss some of the most recent works on dialogue summarization and multitask learning strategies for abstractive summarization. for long dialogue summarization, @xcite proposed a window-based pre-training strategy using five different types of dialogue-related noise -speaker mask, turn splitting, turn merging, text infilling and turn permutation. at first, the window is corrupted with noise, and then the model is tasked with de-noising and reconstructing the window. on the other hand, @xcite utilize a multi-stage approach for dealing with long dialogues. in the preliminary stages, they segment the input and produce coarse summaries, while in the final stage, the coarse summaries are used to generate the final fine-grained summary. @xcite studied the effectiveness of different strategies to deal with long dialogues and concluded that a retrieve-then-summarize pipeline model works better in comparison to longformer @xcite or hmnet @xcite . however, in the case of dialogsum, as the input data is well within the limit of the popular pre-trained transformer models such as bart @xcite , we are not faced with any such issues. moreover, @xcite al. (2021a) a slightly different but closely related task that deserves mention is that of automatic minuting of meeting transcripts. the first shared task on automatic minuting (automin) @xcite at interspeech 2021 and the sigdial 2021 special session on summarization of dialogues and multi-party meetings (summdial) @xcite brought out a plethora of interesting works targeting the task such as the attempt to use bart for generation of readable minutes @xcite , 2021). @xcite present an empirical analysis of the state-of-the-art summarization models for the task of generating meeting minutes and arrive at the conclusion that they are far from being satisfactory. a novel dataset of meetings in english and czech @xcite is also being released to further encourage the research community to take up the challenging task. @xcite claim to be the first ones to have applied multi-task learning to dialogue summarization task. leveraging part-of-speech (pos) information, they constructed a syntax-aware dialogue summarization model on samsum corpus @xcite . the main intuition behind their approach is that different speaker roles are characterised by different syntactic structures (voiceprints), which could be captured via pos information. more recently, for low-resource datasets @xcite experimented with several combinations of auxiliary tasks for abstractive summarization in a multi-task setting. they concluded that a certain combination of tasks indeed improved the abstractive summarization results across different datasets and models. prior to these, in the multitask setting, the primary task of abstractive summarization has been combined and experimented with several other auxiliary tasks such as entailment generation @xcite ; question generation and entailment generation @xcite ; extractive summarization @xcite ; text categorization and syntax labeling @xcite ; dialogue act classification and extractive summarization @xcite ; keyword extraction and key-sentence extraction @xcite . very recently, @xcite formulated the five different tasks of dialogue understanding (du) as a unified generation task. these tasks include dialogue summarization, dialogue completion, dialogue state tracking, slot filling and intent detection. then they experimented with eight different multi-task training strategies and concluded that their proposed method achieves superior performance on both few-shot as well as zero-shot settings. these encouraging results of the multi-task learning strategies on abstractive summarization motivated us to apply the same to the dialogsum challenge. ## dialogsum challenge in this section, we give a brief overview of the di-alogsum challenge by first describing the dataset and then going through the task description. ## our system we employ a multi-task learning approach for the dialogsum challenge. in multi-task learning, a machine learning model is trained simultaneously on more than one related task @xcite . usually, there is a main task and one or more auxiliary tasks. in our case, the main task is abstractive summarization and the auxiliary tasks are extractive summarization, novelty detection and language modeling. there are many variants of multi-task learning. in this work, we employ a hard parameter sharing @xcite transformers-based architecture in which all tasks share the same encoder layers but have task-specific decoder and/or lm head(s). the multi-task model architecture is depicted in figure 4 . it consists of a single bart encoder which is shared amongst all the tasks. the bart decoder is used for the main task of abstractive summarization, while task-specific heads are used for each of the respective auxiliary tasks. we now describe each of the tasks of our model oneby-one: abstractive summarization (as): for the main task of abstractive summarization, the transcripts are given as input to the bart encoder and the abstractive summaries are obtained as output from the bart decoder. this is a sequence-to-sequence task accomplished with the encoder-decoder architecture. in cases where we want to run only the single task for establishing the baseline, only this task is undertaken while keeping all other auxiliary tasks inactive through the training parameters. ## results and discussion in this section, we first describe the experimental setup used and then present the results. finally, we analyse the summaries generated by our bestperforming model. ## conclusion in this paper, we describe our submission to the shared task on dialogue summarization named di-alogsum challenge at inlg 2022. dialogsum consists of 13,460 real-life scenario dialogues. we employ a multi-task learning approach for the task and achieve considerable improvement over the single-task baseline. our best performing model is the multi-task combination of abstractive summarization as the main task and extractive summarization as the auxiliary task. we also incorporate the topic information supplied alongside the summaries to gain marginal improvement in performance over the baseline. in future work, we would like to experiment with other tasks to find the optimal combination. we would also like to explore methods other than multi-task learning for improving the abstractive summarization of dialogues.