Few-Shot Learning of an Interleaved Text Summarization Model by Pretraining with Synthetic Data (2021)

Abstract: Interleaved texts, where posts belonging to different threads occur in a sequence, commonly occur in online chat posts, so that it can be time-consuming to quickly obtain an overview of the discussions. Existing systems first disentangle the posts by threads and then extract summaries from those threads. A major issue with such systems is error propagation from the disentanglement component. While end-to-end trainable summarization system could obviate explicit disentanglement, such systems require a large amount of labeled data. To address this, we propose to pretrain an end-to-end trainable hierarchical encoder-decoder system using synthetic interleaved texts. We show that by fine-tuning on a real-world meeting dataset (AMI), such a system out-performs a traditional two-step system by 22%. We also compare against transformer models and observed that pretraining with synthetic data both the encoder and decoder outperforms the BertSumExtAbs transformer model which pretrains only the encoder on a large dataset.

## introduction interleaved texts are increasingly common, occurring in social media conversations such as slack and stack exchange, where posts belonging to different threads may be intermixed in the post sequence; see a meeting transcript from the ami corpus @xcite in in conversation disentanglement, interleaved posts are grouped by the thread. however, a reader still has to read all posts in each cluster of threads to get the gist. @xcite proposed a two-step system that takes an interleaved text as input and first disentangles the posts thread-wise by clustering, and then compresses the thread-wise posts to single-sentence summaries. however, disentanglement e.g., @xcite , propagates error to the downstream summarization task. an end-to-end supervised summarization system that implicitly identifies the conversations would eliminate error propagation. however, labeling of interleaved texts is a difficult and expensive task @xcite @xcite . ami utteracnes . . . who is gonna do a powerpoint presentation ? think we all huh. you will . . . . . . ξ and uh the sender will send to the telly itself an infrared signal to tell it to switch on or switch. . . . . . ζ so y so it's so it's so you got so that's something we should have a look into then i when desi when designing the ergonomics of see have a look . . . . . . ψ ,the little tiny weeny batteries, all like special longlasting batteries. . . . . . summary 1) the project manager had the team members re-introduce . . . 2) the industrial designer discussed the interior workings of a remote and the team discussed options for batteries and infra-red signals. we propose a pretraining approach to tackle these issues. we synthesized a corpus of interleaved text-summary pairs out of a corpus of regular document-summary pairs and train an end-toend trainable encoder-decoder system. to generate the summary the model learns to infer (disentangle) the major topics in several threads. we show on synthetic and real-world data that the encoderdecoder system not only obviates a disentanglement component but also enhances performance. thus, the summarization task acts as an auxiliary task for the disentanglement. additionally, we show that fine-tuning of the encoder-decode system with the learned disentanglement representations on a real-world ami dataset achieves a substantial increment in evaluation metrics despite a small number of labels. we also propose using a hierarchical attention in the encoder-decoder system with three levels of information from the interleaved text; posts, phrases, and words, rather than traditional two levels; post and word @xcite @xcite . the remaining paper is structured as follows. in section 2, we discuss related work. in section 3, we provide a detailed description of our hierarchical seq2seq model. in section 4, we provide a detailed description on the synthetic data creation algorithm. in section 5, we describe and discuss the experiments. and in section 6, we present our conclusions. there are two kinds of summarization: abstractive and extractive. in abstractive summarization, the model utilizes a corpus level vocabulary and generates novel sentences as the summary, while extractive models extract or rearrange the source words as the summary. abstractive models based on neural sequence-to-sequence (seq2seq) @xcite proved to generate summaries with higher rouge scores than the feature-based abstractive models. @xcite proposed an encoder-decoder (auto-encoder) model that utilizes a hierarchy of networks: word-to-word followed by sentence-tosentence. their model is better at capturing the underlying structure than a vanilla sequential encoderdecoder model (seq2seq). @xcite and @xcite showed multi-sentence captioning of an image through a hierarchical recurrent neural network (rnn), topic-to-topic followed by word-to-word, is better than seq2seq. these works suggest a hierarchical decoder, thread-tothread followed by word-to-word, may intrinsically disentangle the posts, and therefore, generate more appropriate summaries. integration of attention into a seq2seq model @xcite led to further advancement of abstractive summarization @xcite . @xcite devised a hierarchical attention mechanism for a seq2seq model, where two levels of attention distributions over the source, i.e., sentence and word, are computed at every step of the word decoding. based on the sentence attentions, the word attentions are rescaled. our hierarchical attention is more intuitive, computes post(sentence)-level and phraselevel attentions for every new summary sentence, and is trained end-to-end. semi-supervised learning has recently gained popularity as it helps training parameters of large models without any training data. researchers have pre-trained masked language models, in which only an encoder is used to reconstruct the text, e.g., bert @xcite . @xcite used bert as seq2seq encoder and showed improved performance on several abstractive summarization tasks. similarly, researchers have published pre-trained seq2seq models using a different semi-supervised learning technique, where a seq2seq model is learned to reconstruct the original text, e.g., bart @xcite and mass @xcite . in this work, we rely on transfer learning and demonstrate that by pretraining with appropriate interleaved text data, a seq2seq model readily transfers to a new domain with just a few examples. ## model at step k of thread decoder (d t2t ), we compute elements of post-level attention as γ k @xmath0, p i ) @xmath1, . . . , n}, where attn γ aligns the current thread decoder state vector h d t2t k-1 to vectors of matrix p i . a phrase is a short sequences of words in a sentence/post. phrases in interleaved texts are equivalent to visual patterns in images, and therefore, attending phrases are more relevant for thread recognition than attending posts. thus, we have phrase-level attentions focusing on words in a channel and with a responsibility of disentangling threads. at step k of thread decoder, we also compute a sequence of attention weights, the motivation is to provide information about the previous sentence. the current state h d t2t k is passed through a single layer feedforward network and a distribution over @xmath2: at step l of word-to-word decoding of summary of thread k, we compute elements of word level attention, i.e., α k,l i,• ; we refer to @xcite for further details on it. however, we use phrase-level word attentions for rescaling the word level attention as αk,l i,@xmath3,j × α k,l ij ), where norm (softmax) renormalizes the values. thus, contrary to popular two-level hierarchical attention @xcite @xcite , we have three levels of hierarchical attention and each with its responsibility and is coordinated through the rescaling operation. we train our model end-to-end to minimize the following objec-notations c a sequence of pairs of single-thread texts and singlesentence summaries a a minimum number of threads b a maximum number of threads m a minimum number of posts n a maximum number of posts window returns an iterator object that traverse a given sequence elements in windowed manner w window size t step size o an iterator object that returns a window size sequence of pairs of single-thread texts and single-sentence summaries e a window size sequence of pairs of single-thread texts and single-sentence summaries i a sequence of sentences m a sequence of single-sentence summaries u a uniform sampling function a a single thread text as a sequence of sentences t a single-sentence summary as a sequence of words î a multi-thread interleaved text m a multi-sentence summary of an interleaved text {a×b} a set in which variable a is repeated b times s a sequence of positive integers reverse reverses an array pop removes the last element from an array p a sentence/post as a sequence of words [•] array indexing operation x \ y a set of elements that belong to x but not to y z a sequence of pairs of multi-thread interleaved texts and multi-sentence summaries and y k,0 , . . . , y k,q are words in a ground-truth summary and d w2w generation respectively. ## synthetic dataset obtaining labeled training data for interleaved conversation summarization is challenging. the available ones are either extractive @xcite or too small @xcite to train a neural model and thoroughly verify the architecture. to get around this issue, we synthesized a dataset by utilizing a corpus of conventional texts for which summaries are available. we created a corpus of interleaved texts from the abstracts and titles of articles from the pubmed corpus @xcite . we chose pubmed abstracts as it has, in contrast to other corpora such as news articles or stackoverflow posts, a single-sentence summary that can only be comprehended out of a whole abstract. further, the number of sentences more closely resembles that of a conversationalist in a conversation. algorithm 1 interleaving algorithm 1: procedure interleave(c, a, b, m, n) 2: o, z ← window(c, w, t), array() 3: while @xmath4: e, i , m , s ← o.next(), array(), array(), {} 5: r ∼ u(a, b) 6: a, t ← e[j] 8: i .add(a[1:q]) 10: m .add(t) 11: s ← s ∪ {j×q} 12: î, m, l ← array(), array(), |s| 13: î.add(p) 17: t ← m [k] 18: m.add(t) 20: s ← s \ k 21: z.add( î; m) 22: random interleaving of the sentences from a small number of pubmed abstracts roughly resembles interleaved texts, and, correspondingly, interleaving of titles resembles its multi-sentence summary. we devised an algorithm for creating synthetic interleaved texts based on this idea. interleave algorithm: the interleave algorithm generates interleaved texts, each containing randomly interleaved sentences from a small number of abstracts, where the number is a random value within a specified range. the number of sentences used per abstract is also a random value within a specified range. abstracts to be included in an interleaved text are first selected, then the selected abstracts are interleaved, and finally the interleaved texts together with a concatenation of the titles of the selected abstracts are returned. we first refer to window helps to enlarge the interleaved corpus without redundancy as abstracts are randomly sampled out of an iterator element, e, and also new abstracts are always included in the next element through sliding. similarly, sets of sentences are randomly sampled out of the selected abstracts. thus, interleaved text-summary pairs in the corpus are different. the two parts of the interleave algorithm, selection and interleaving, will be described next. selection: u in step 5 determines the number of threads, r. then, thread candidates for an interleaved text are chosen out of an iterator element e, a window size sequence of pairs of single-thread texts and single-sentence summaries. next, post candidates for each selected thread, a, are chosen. u in step 8 determines the number of posts, q. thread indices are repeated as many times as its posts and stored in a set, s. interleaving: in every step in a loop of a size equivalent to the length of indices s, u randomly selects a thread index. reverse and pop in step 15 help in selecting a post, p , in the selected thread in a fifo manner. the single-sentence summary, t, of the thread is added to the multi-sentence summary sequence, m, if it didn't exist previously. as an interleaved text-summary pair in the corpus has a thread size between a and b and post size per thread between m and n, the larger the difference between a and b and m and n in a corpus, the harder the disentangling and summarization task. so, we vary these parameters and create different synthetic corpora of varying difficulty for the experiments. ## conclusion we investigated the use of an end-to-end hierarchical encoder-decoder model, hier2hier, with three levels of hierarchical attention for jointly summarizing and implicitly disentangling interleaved text. to train this model, we examined the use of pretraining using synthesized data and fine-tuning for adaptation to a new domain with limited labeled real-world data. on real-world ami data, our fine-tuned end-to-end system outperforms a twostep system by 22% (rouge-1). experiments were also conducted against the transformer-based bert-sumextabs and bart systems, which indicate that these transformer models can also summa-rize interleaved texts. specifically our hier2hier model also outperformed the transformer-based bertsumextabs but not bart, which suggests that use of pretraining of both the decoder as well as the encoder is important, and also indicates the utility of our synthetic data.