TAN et: T hread- A ware P retraining for A bstractive C onversational S ummarization (2022)

Abstract: Although pre-trained language models (PLMs) have achieved great success and become a milestone in NLP, abstractive conversational summarization remains a challenging but less studied task. The difficulty lies in two aspects. One is the lack of large-scale conversational summary data. Another is that applying the existing pre-trained models to this task is tricky because of the structural dependence within the conversation and its informal expression, etc. In this work, we first build a large-scale (11M) pretraining dataset called RCSum, based on the multi-person discussions in the Reddit community. We then present TANet, a thread-aware Transformer-based network. Unlike the existing pre-trained models that treat a conversation as a sequence of sentences, we argue that the inherent contextual dependency among the utterances plays an essential role in understanding the entire conversation and thus propose two new techniques to incorporate the structural information into our model. The first is thread-aware attention which is computed by taking into account the contextual dependency within utterances. Second, we apply thread prediction loss to predict the relations between utterances. We evaluate our model on four datasets of real conversations, covering types of meeting transcripts, customer-service records, and forum threads. Experimental results demonstrate that TANet achieves a new state-of-the-art in terms of both automatic evaluation and human judgment.

## introduction text summarization is a long-standing challenging task in artificial intelligence, aiming to condense a piece of text to a shorter version, retaining the critical information. there are various promising applications of conversational summarization in the real world, emphasizing the need to build auto summarization systems. for example, online * ## problem formalization in general, the abstractive conversational summarization task could be formalized as follows. denote the dataset as @xmath0, s i )} n @xmath1, where ∀i, (c i , s i ) is a conversation-summary pair and n is the size of d. the conversation c @xmath2,j , a i,j )} n i @xmath3,j } n i @xmath4,j } n i @xmath5. for example, each meeting transcript in the ami dataset comprises multiple turns, where each turn is an utterance of a participant who has a specific role in the project, such as manager or designer. our goal is to learn a generation probability p (s|c), so that given a new conversation input c, we can generate a summary s . since there is always limited availability of d to support accurately learning for p (s|c), we propose to build a large-scale summarization-like corpus d @xmath6, s k )} m @xmath7. s k represents the pseudo summary of the conversation c k . in this way, we first pretrain our model on d p and then finetune it on the respective dataset d of each downstream task. ## reddit for conversational summarization existing conversational summarization corpora @xcite @xcite @xcite have a low number of conversations, which prevents research community from engaging into this problem. different from @xcite ) that using news documents to simulate multi-person conversations for pretraining, it is reasonable to hypothesize that leveraging real conversation data could lead to better downstream performance. in this work, to benefit from the large-scale conversation corpus, we mined and processed a largescale dataset from reddit foot_0 for conversational summarization called rcsum. figure 1 shows an example in the dataset. to our best knowledge, rc-sum is the first large-scale pretraining corpus with real conversations for abstractive conversational summarization. we collected the posts on the reddit site from 2019 to 2020. a post is composed of a title and its corresponding discussions which usually consist of multiple threads. the comments in a thread can naturally expand into a tree structure. remarkably, each comment has rich attributes, including the user information, creation timestamp and the accumulated score foot_1 , etc. with the large-scale real multi-person conversation data, the key is how to construct a summary-like instance for a thread. we consider two strategies to select sentences that appear to dominate the thread: (1) title. the discussions of each post are all developed upon the topic of the title, so we select the title as a part of the pseudo summary; (2) lead comment. despite the topic given by the title, the lead comment (i.e., the first comment of a thread) also well influences the future direction of what is discussed in this thread. we concatenate them as the pseudo summary of the discussions in a thread. lead comment's original position is replaced by a special token @xcite . to clean up rcsum, we adopted a series of heuristics including: (1) we removed any threads where the number of comments less than 10; (2) we discarded any not-safe-for-work posts, such as posts containing adult or violent content; (3) we replaced all urls with a special token [url]; (4) we removed all markup and any other non-text content such as "*, ∼, [, ]"; (5) we removed any threads whose title or lead comment scored less than 0; (6) we removed any posts which contain quarantine, picture, or video, etc. after then, the dataset has 11, 200, 981 instances. ## methodology in this section, we present tanet, a thread-aware pretrained model which incorporates the inherent dependencies between utterances to enable improved conversation's representations for summary generation. below, we first introduce the model architecture, thread-aware attention, and then introduce our pretraining objectives. finally, we move on to the application of downstream tasks. ## pretraining objectives in this section, we describe the pretraining objectives used for pretraining tanet. in addition to the causal language modeling, we newly introduce another thread-aware pretraining task to predict the contextual relation between utterances. ## related work with the recent success of seq2seq models, the research focus of conversational summarization has been transferred from the extractive methods to abstractive models. various semantic patterns have been applied to these abstractive approaches, such as dialogue acts (goo and chen, 2018), auxiliary key point sequences @xcite , topic segments @xcite , conversational stages and dialogue overview (chen and yang, 2020), discourse relations @xcite @xcite . at the same time, some work is devoted to providing highquality datasets to promote the development of this research direction @xcite @xcite @xcite . however, these corpora have a low number of conversations, which hinders the progress of abstractive summarization @xcite . recently, large neural models pre-trained on huge corpora have led to strong improvements on numerous natural language understanding and generation tasks @xcite @xcite . encouraging by the promising progress of pretraining, @xcite first introduce a hierarchical structure and propose pretraining on cross-domain data for meeting summarization. the pretraining data is collected from the news domain. regarding one document as the utterances from one participant, multiple documents are combined and reshuffled to simulate a multi-person meeting. however, there are two disadvantages -the first is the style inconsistency between conversation and news, and the second is that there is no contextual relationship between the two documents, so the participants have no communication actually. in this work, we build a large-scale corpus based on real conversations. besides, we further incorporate the structure information of thread in the model. ## conclusions in this work, we introduce tanet, a thread-aware pre-trained model for abstractive conversational summarization. tanet employ the thread-aware attention and a new pretraining objective to fully leverage the structure information of conversation. furthermore, we build a large-scale pretraining corpus based on the discussions on reddit. experiments on four downstream tasks demonstrate the effectiveness of tanet.