Improving Abstractive Dialogue Summarization with Hierarchical Pretraining and Topic Segment (2021)

Abstract: With the increasing abundance of meeting transcripts, meeting summary has attracted more and more attention from researchers. The unsupervised pre-training method based on transformer structure combined with fine-tuning of downstream tasks has achieved great success in the field of text summarization. However, the semantic structure and style of meeting transcripts are quite different from that of articles. In this work, we propose a hierarchical transformer encoder-decoder network with multi-task pre-training. Specifically, we mask key sentences at the word-level encoder and generate them at the decoder. Besides, we randomly mask some of the role alignments in the input text and force the model to recover the original role tags to complete the alignments. In addition, we introduce a topic segmentation mechanism to further improve the quality of the generated summaries. The experimental results show that our model is superior to the previous methods in meeting summary datasets AMI and ICSI.

## introduction meeting is a common activity for people to discuss, exchange views and obtain information around specific topics. the widespread application of speech transcription technology has brought about the rapid expansion of meeting corpus. therefore, automatic meeting summaries are valuable to people and society by providing quick access to important content of the information. the recently successful sequence-to-sequence @xcite based architecture has greatly inspired the existing meeting summary methods. specifically, earlier studies use rnns @xcite structures such as lstm @xcite to capture the local composition of documents and learn the semantic representation of documents. unfortunately, rnns lack global modeling capability and is difficult to deal with long-term dependency. to overcome this limitation, more and more researchers introduce convolution or transformer @xcite model. these methods are easy to modify to capture more global information. however, recent studies indicate that they may not be sufficient to build long-term dependency models, which makes them significantly less effective in the context of long-term multi-human dialogue. therefore, constructing hierarchical encoding structure @xcite to capture the content information of each speaker and the high-level semantic information hidden among utterances has become the mainstream method in the field of meeting summary. different from news texts, utterances are often turned from different interlocutors, which leads to the topic drifts, and lower information density. these problems need to be overcome by introducing external high-level semantic information, such as conversation behavior, topic mining and so on. @xcite proposed to use the dialogue act signals in a neural summarization model. @xcite introduced visual focus of attention (vfoa),which represents the common concerns of all conference participants in each time stamp, to keep the keep meeting summaries on topic. @xcite improved abstractive dialogue summarization with graph structures and topic words. these studies have proved that the introduction of external high-level semantic information has positive feedback on the results of meeting summary. meanwhile, the use of carefully designed unsupervised pre-training tasks and large scale pretraining corpus has achieved great success in the field of document summary and dialogue understanding. bart @xcite ) corrupted text with an arbitrary noising function and learned to reconstruct the original text. pegasus @xcite ) masks the key sentences in the original text and requires the model to generate those designated sentences at the decoder. besides, di-alogbert @xcite applied next utterance generation, masked utterance regression and distributed utterance order ranking tasks to capture discourse-level coherence among utterances. combined with different downstream tasks, only using a small number of labeled training sets in the field for supervised fine tuning can get quite good results. unlike the context of many people's short conversations or texts in the form of news or papers, each participant's speech contains not only a complete fragment of their own views, but also a common discussion and exchange of views with other speakers. we think that the meeting summary task can combine the document summary and dialogue understanding to get a better result. therefore, we propose a hierarchical transformer encoder-decoder network with auxiliary multi-task learning. we mainly follow the model structure of hmnet @xcite and construct new pretraining tasks on different levels of encoder. our contributions are as follows: to evaluate our model, we employ the widely used ami @xcite and icsi @xcite meeting corpus. results show that our model significantly outperforms previous meeting summarization methods. we then conduct ablation studies to verify the effectiveness of different components in our model. ## related work meeting summarization. the early works of meeting summary often focused on the use of unsupervised extraction algorithm to obtain the key information in the conversation. @xcite propose a multimodal fusion model, which combines audio, video, motion and language. the model is trained by convolutional neural network method, and can identify important words that should be included in the summary of group discussion. furthermore, many researchers have focused on improving the abstractive meeting summarization model. @xcite used the pointer generation network, which can sense the topic transfer of conversation, integrates the external topic information to improve the quality of summary generation. in the work of hmnet, a hierarchical conference summary network is proposed, which is pre-trained with news datasets, and obtained good results in ami and icsi. pre-trained language models. bert @xcite introduces masked language modelling and next sentence prediction, which leads to the upsurge of pre-training research in nlp field. however, bert does not perform well in the field of text generation due to the feature of autoencoding model. the pre-training task for text generation task is designed based on mass @xcite and bart. in mass, an input sequence with a masked span of tokens is mapped to a sequence consisting of the missing tokens, while bart is trained to reconstruct the original text from corrupted input with some masked tokens. furthermore, pegasus build gsg task for text summary scenario, it masks the key sentences in the original text and requires the model to generate those designated sentences at the decoder. in order to make the model fully learn the high-level semantic information hidden between dialogues, @xcite proposed a transformer based hierarchical model and various unsupervised goals for the pre-training of the context semantics of dialogue discourse. dialogbert @xcite applied next utterance generation, masked utter-ance regression and distributed utterance order ranking tasks to capture discourse-level coherence among utterances. ## models we mainly follow the hierarchical meeting summarization network model structure and make some improvements on the utterance-level encoder, which its fusion blocks come from ernie @xcite . the problem of meeting summarization can be formalized as follows. the input meeting transcripts x contain some of meeting participants and the corresponding speech content. each meeting transcript consists of multiple utterances u, where each utterance belongs to a specific topic t. the input meeting transcripts @xmath0, r 1 , t 1 ), (u 2 , r 2 , t 1 ), . . . , (u m , p n , t l )}, where u j , 1 ≤ @xmath1, t j , 1 ≤ @xmath2, 1 ≤ @xmath3. the golden summary y written by human beings is a sequence of tokens. an utterance is made up with w 1 , w 2 , . . . , w n , where w i means the word in an utterance. so the goal of the model is to generate meeting summary @xmath4, . . . , y s ) from the meeting transcripts word-level encoder. the word-level encoder (w-encoder) is designed to extract the semantic information of a single utterance in meeting transcripts. we encode each token in one utterance using glove embeddings from spacy library. since the parallelization mechanism of transformer can not obtain the position information of the sequence, the positional encodings are added to the input vector.there are standard transformer encoder modules on the embedded layer, which is stacked by the same block with a multi-head attention layer and a feed-forward layer. to incorporate syntactic and semantic information, we also train two embedding matrices to represent the part-of-speech (pos) and entity (ent) tags. we directly take the output of the last hidden layer and do the average pooling operation to get the semantic representation of the turn. so we denote the output of the word-level transformer: utterance-level encoder. the utterance-level encoder (u-encoder) processes the word-level outputs of all utterances in a meeting and gets the high-level semantic information hidden among utterances. each meeting participant has a different role, such as project manager and industrial designer. the speaker's information should be considered when generating the summary of the model. to be specific, we represent speaker identities with fixed-length vector called role vector. then, both role embedding and utterance embedding are fed into utterance-level encoder for fusing heterogeneous information and computing final output embeddings. the utterance-level encoder consists of stacked fusions, which are designed for encoding both tokens and entities as well as fusing their heterogeneous features. in each fuser block, the input utterance embeddings and role embeddings from the preceding aggregator are fed into two multi-head self-attentions (mh-atts) respectively. next, the fusion block adopts an information fusion layer for the mutual integration of the utterance embedding and role embedding, and computes the output embedding for each utterance and role. for an utterance vector u j and its aligned role vector r k , the process of information fusion is as follows: the utterance output embedding will be input to the decoder to participate in the summary generation, and the role output embedding will be used in dra pre-training task. decoder. the decoder receives the output of wordlevel encoder and utterance-level encoder, and generates the corresponding summary according to the semantic information of the meeting transcripts. based on the structure of transformer decoder, the transformer block includes two cross-attention layers. the decoder input embedding first passes through masked self-attention layer, and then performs the attention operation with word-level output and utterance-level output successively. this makes the model pay attention to the hierarchical semantic information in each inference step. the output of the decoder transformer is denoted as: we illustrate the whole model network in fig. 1 . ## pretraining we expect that our model can fully extract the semantic information of different levels in the hierarchical encoder-decoder structure through carefully designed pre-training tasks. the following three sections describe our tasks built on a hierarchical network. gap sentences generation (gsg). this pretraining task is proposed in pegasus for the first time. it is based on the assumption that the model can achieve better and faster fine-tuning performance when the pre-training target is very similar to the downstream task. the principle is to mask the whole key sentences from the document and concatenate the gap-sentences into a pseudosummary. in order to obtain the key sentences in the original text unsupervised, the researchers select top-m scored sentences according to importance. as a proxy for importance they compute rouge @xcite between the sentence and the rest of the document. different from pegasus, we try the graph based sorting algorithms textrank and maximum margin relevance(mmr) to get the key sentences in the original text. textrank regards each sentence in the text as a node v. if two sentences are similar, it is considered that there is a non-directional weighted edge between the corresponding nodes of the two sentences. according to the similarity calculation formula, the algorithm circularly calculates the similarity between any two nodes, sets a threshold to remove the edge connection between the two nodes with low similarity, constructs a node connection graph g(v,e), and then iteratively calculates the textrank value of each node. after sorting, the sentences corresponding to the nodes with the highest textrank value are selected as the key sentences. the iterative formula of textrank is as follows: where w ji is the weight of the edge from v i to v j . d is the damping coefficient, representing the probability of pointing from one node to any other node in the graph, generally 0.85. in(v i ) and out(v i ) are the node set pointing to node v i and the node set pointing from the edge of node v i . at the beginning of the design, mmr is used to calculate the similarity between the query text and the searched document for ranking the documents. the algorithm formula is as follows: where q and c represent the whole document, r is an initial set which has been obtained based on the correlation, d i represents a sentence in the document. the physical meaning of the first term in the formula refers to the similarity between the sentences to be extracted and the whole document, while the latter term refers to the similarity between the sentences to be extracted and the key sentences obtained. the key sentences extracted by mmr algorithm can not only express the meaning of the whole document, but also have diversity. word-level encoder s 1 . [mask] s 3 . s 4 . s 5 . s 6 . s 7 . [mask] s 9 . utterance-level encoder decoder pm: gm: me: pm: s 1 . s 2 . s 3 . gm: s 4 . s 5 . s 6 . me: s 7 . s 8 . s 9 . input text: <@xmath5. s 8 . target text [shifted right]: s 2 . s 8 . <@xmath6: s i : the i-th sentence model to predict all corresponding role embedding based on aligned utterance embedding. as our task is similar to training a denoising auto-encoder @xcite , we refer to this procedure as denoising role auto-encoder (dra). given the utterance sequence {u 1 , . . . , u m } and its corresponding role embedding {r 1 , . . . , r k }, we define the aligned role distribution for the utterance embedding u i as follows: p (r j |u i ) will be used to compute the cross-entropy loss function for dra. figure 3 shows the implementation process of the dra task. similar to bert, we perform the following operations for dra: (1) in 5% of the time, we replace the role embedding with another random role embedding, which aims to train our model to correct the errors that the turn is aligned with a wrong role; (2) in 15% of the time, we mask turn-role alignments and predict the role embedding; (3) in the rest of the time, we keep turn-role alignments unchanged, which aims to encourage our model to integrate the role information into turn embedding for better language understanding. topic segmentation. we add a special symbol @xcite of topic segmentation to further improve the summary effect. specifically, in addition to the position embedding, we also add the topic segmentation embedding in the utterance lever encoder input. it is a kind of interval segment embedding to distinguish multiple topics in meeting transcripts. for example, for utterance (u 1 , u 2 , [t sep ], u 3 , u 4 , [t sep ], u 5 ...), where every two utterances belong to the same topic. we would assign the topic segmentation embedding (t a , t a , t a , t b , t b , t b , t a ...). besides, we restrict the scope of attention computation to different topics, it can alleviate the noise impact caused by long-distance dependencies. in each transformer block, multiple self-attention heads are used to aggregate the output embeddings of the previous layer. the attention score is calculated as follows: we introduce the mask matrix o determines whether a pair of tokens can be attended to each other. each utterance can only focus on the other utterances under the same topic, as illustrated in figure 4 . ## experiment datasets. we tested our model on the widely used ami and icsi meeting summary datasets. both the ami and icsi meeting corpus are multi-modal datasets which consist hundreds of hours of meeting recordings. each meeting participant has a role tag, such as marketing manager, industrial designer, professor and so on. moreover, the datasets also have a wide range of annotations, including conversation behavior, topic segmentation, extractive and abstract summaries. in ami, there are an average of 4757 words in the transcripts, 289 circles, 4 speakers and 322 words in the abstract. in icsi, the average conference record was 10189 words, 464 laps, 6.2 speakers, and 534 words in the abstract. we roughly divide datasets into training, valid and test part according to the ratio of 8:1:1. except using some news domain pre-training data in hmnet, we also introduce dialog domain datasets mediasum @xcite and tv4dialog (leilan zhang, 2019) as the pretraining datasets. mediasum is a large scale modular media interview dataset composed of 463.6k texts and abstracts. the dataset is mainly from the interview records of npr and cnn, with an average of 30 rounds per conversation, six to seven participants, and a total of 1554 words. the dataset has the characteristics of large data scale, multiparty dialogue in multiple fields and clear theme. tv4dialog is a multi round dialogue corpus extracted from the subtitles of american tv series. it contains about 260000 utterances with speaker tags. in order to make the dataset suitable for our pre-training task, we do the following three aspects of preprocessing. firstly, the original dataset is cleaned, and with the help of spacy library, the pos and ent tags are added. secondly, we also randomly stitch several different dialogues on the cleaned data to simulate the change of conversation topic in real conference scene. finally, aiming at the problem that the label quality of speakers in pre-training datasets are uneven, we combine some of the similar role tags. metrics. we evaluated performance with the widely used rouge-1, rouge-2 and rouge-su4 metrics in automatic summarization. more specifically, it focuses on measuring the number of overlapping units such as n-gram between the generated summary and the reference summary. when matching the reference summary and the summary to be evaluated, rouge-su4 does not require that the gram must be continuous, and can "skip" some words. through the above three metrics, we can measure the generated summary quality from many aspects. ## results ablation study. we conduct ablation experiments to analyze the impact of each part of the model on the final results. in we also compare the effect of different unsupervised key sentence extraction methods in gsg task on the final summary. corerank @xcite construct undirected weighted graph and calculate the corerank value of nodes, it is state-of-the-art extractive summarization method. pgn method is the pointergenerator network @xcite , which focuses on addressing the reproducing and repeating problem in general abstractive text summarization task. hasmr @xcite proposes a hierarchical neural encoder based on adaptive recurrent network to learn the semantic representation of conference session based on adaptive session segmentation. bertsum @xcite is a pretraining model with good performance in the field of text generation. mm is a multimodal model, hmnet(2020) 53.02 18.57 24.85 46.28 10.60 19.12 our model 51.90 21.16 25.12 44.41 11.86 19.95 ## conclusion in this paper, we apply two kinds of pre-training tasks to hierarchical transformer network to improve the effect of meeting summary generation. we adjust the structure of utterance level encoder to better integrate the role vector of each participant. in addition, we introduce additional topic segmentation information to constrain the attention range, which is further improved the model's performance. experimental results show that our model performs well on ami and icsi datasets. in the future, we plan to add some new pretraining tasks to obtain the semantic information of discourse coherence in conference texts. we plan to utilize knowledge graph and dialog act, which can better capture salient information from the transcript.