Weakly Supervised Extractive Summarization with Attention (2021)

Abstract: Automatic summarization aims to extract important information from large amounts of textual data in order to create a shorter version of the original texts while preserving its information. Training traditional extractive summarization models relies heavily on human-engineered labels such as sentence-level annotations of summary-worthiness. However, in many use cases, such human-engineered labels do not exist and manually annotating thousands of documents for the purpose of training models may not be feasible. On the other hand, indirect signals for summarization are often available, such as agent actions for customer service dialogues, headlines for news articles, diagnosis for Electronic Health Records, etc. In this paper, we develop a general framework that generates extractive summarization as a byproduct of supervised learning tasks for indirect signals via the help of attention mechanism. We test our models on customer service dialogues and experimental results demonstrated that our models can reliably select informative sentences and words for automatic summarization.

## introduction automatic summarization systems are useful in many applications where they aim to create a concise version of large amounts of textual data. much effort has been devoted to developing automatic summarization systems in recent years by using deep learning, such as sentence compression with lstms @xcite , sentence summarization with neural attention networks @xcite , text summarization using sequence-to-sequence rnns @xcite , end-to-end dialogue description generation @xcite , and summarization with deep reinforced models @xcite . these approaches fall into one of two broad categories: extractive and abstractive. extractive summarization directly chooses and assembles sentences and words from the original texts as the summary. abstractive summarization collects high quality information and a summary is written in a concise manner. central to both approaches is the availability of labeled data for training. for extractive summarization, training requires sentences and words being labeled as summary-worthy or not. for abstractive summarization, training requires document-summary pairs where each document has a summary available to supervise the training of a model that can produce such summaries to capture the highlights of the document. however, such labeled data may not be available in many applications. on the other hand, indirect signals for summarization are often accessible. for example, for dialogues, the resulting actions contain valuable signals for summarization. for a news article, its category (such as politics, sports, technology, weather, etc.) and its title could provide guidance on summary key points. for an electronic health record (ehr), the concluding diagnosis can be a very important piece of information. in this paper, we develop a general framework for automatic extractive summarization for scenarios where there are no pre-labeled sentences/words for summary-worthiness but other indirect signals are available. imagine how a human annotator reads texts and produces summaries. instead of reading through the entire texts, memorizing all information, and then writing up a summary based on memories, humans often read the texts word by word, sentence by sentence, and highlight those containing key information such as the resulting actions for a dialogue, the category for a news article, the diagnosis for an ehr, etc. our approach mimics this human behavior on picking out important content by using an attention mechanism @xcite @xcite . the model structure composes a hierarchical attention network @xcite as the reader, a downstream ancillary prediction task of the indirect signal, and an extractor for identifying important words and sentences for automatic extractive summarization. we use a dataset for the ancillary task in the learning process to prediction the indirect signals. during the learning process, the reader first composes a sequence of word vectors into a sentence vector for each sentence, and then composes the sequence of sentence vectors into a document vector. it has an attention layer on both word level and sentence level to score each word and each sentence in order to locate the region of focus during prediction of the indirect signals. these attention scores are then used to extract informative sentences and words for summarization, which is a byproduct of the supervised learning process for the indirect signals. the most distinguishing feature of our approach from other extractive summarization approaches is that it does not require a large training corpus of documents with labels indicating which sentences or words should be in the summary. we test our models on customer service dialogues. the results show that the trained attention scores reflect a linguistically plausible representation of the importance for each sentence and word. therefore, it provides an intuitive way to extract summarization in the absence of pre-labeled sentences or words for supervised learning. the main contributions of this work are: we first formally define the task in section 2 and then introduce the general framework in section 3. we describe our experiment settings in section 4 and present our results in section 5. finally, we discuss related work in section 6 and conclude in section 7. ## task definition assume that the input texts consist of a sequence of l sentences. sentence i contains a sequence of t i words (w i1 , ..., w it i ). the task is to extract the l most informative sentences and the k j most informative words for each of the selected sentence j. we first rank each sentence in the document based on its informativeness using attention scores, and then select a subset of the l most informative sentences (where @xmath0. we then rank the words in each of the selected l sentences and highlight the most informative k j words for sentence j (where k @xmath1. ## attention based extractive summarization in this section, we propose a novel architecture that generates extractive summarization as a byproduct of the supervised learning tasks for indirect signals via the help of attention mechanism. the sentences and words that have provided strong signals to the supervised learning tasks will naturally have high attention weights and become good candidates for the summary. we call this process weakly supervised extractive summarization with attention. the architecture consists of a hierarchical attention network (han) @xcite reader that composes the source texts into a continuous-space vector representation, a downstream ancillary prediction task that takes the representation and generates the output for the indirect signal, and an extractor for identifying important words and sentences for automatic extractive summarization. ## experimental setup in this paper, we conduct experiments to evaluate the plausibility of the attention scores to extract informative words and sentence for the use case of summarizing amazon customer service dialogues. in a customer service context, dialogue summaries are especially useful in terms of providing contexts and highlights for contact transfers, escalations, and offline analysis. our proposed approach addresses the issue of absence of labeled data and solves the problem for automatic extractive summarization. ## results after each of the two annotators finish annotating 500 contacts, the gold annotator has validated their results and verified that the standards and qualities are consistent across all 1,000 contacts. the word extraction models are less promising. this is somewhat expected given that our models select a pre-determined number (proportional to sentence length) of words for each sentence while the true number of key words in a sentence could vary largely for sentences with the same length but in different contexts. this suggests that an alternative to our network would be to employ a word extractor that can learn the optimal number of words to extract given the context in the sentence and in the entire dialogue. we leave this to future work. one of the motivations to use an attention mechanism in the seq2seq + att models was to overcome the bottleneck of a fixed-length context vector in the basic encoder-decoder seq2seq approach. in figure 3 , we compare model performance for varying length of dialogues. we observe that the performance of all models except for seq2seq + att (base) and seq2seq + att (ensemble) dramatically decreases as the length of the dialogue increases. for shorter dialogues, seq2seq + att (base) and seq2seq + att (ensemble) are slightly better than the other models while for longer dialogues they significantly outperform the others. they show no significant performance deterioration even with dialogues of 50 or more sentences, which is critical for customer service as the need for a good summary increases as the length of a conversation grows. ## related work much effort has been devoted to automatic summarization in recent years due to an increasing need to access and digest large amounts of textual data. an ideal summarization system would understand each document and generate an appropriate summary directly from the results of that under- other sentence extraction methods heavily relied on human-engineered features such as sentence position and length @xcite , the words in the title, the presence of proper nouns, word frequency @xcite , and event features such as action nouns @xcite . @xcite and @xcite developed a sentence extraction approach based on pretrained sentence embeddings. @xcite proposed a neural attention model for abstractive summarization for individual sentences which was trained on a corpus of pairs of headlines and first sentences in news articles. @xcite extended this approach and developed a general framework for document summarization. to address the lack of sentence extraction word extraction model rouge-1 rouge-2 rouge-l rouge-1 rouge-2 rouge-l classification (base) 62.7 53.1 59.9 33.9 27.8 30.9 classification(ensemble) 50.7 24.5 41.8 24.4 13.5 29.2 seq2seq (base) 69.1 57.6 70.2 36.2 29.6 33.8 seq2seq (ensemble) 64.4 48.4 64.5 30.3 22.9 35.9 seq2seq + att (base) 74.5 51.2 69.6 43.7 26.6 36.4 seq2seq + att(ensemble) 88.2 52.3 76.7 54.6 32.0 39.3 a key point sequence acts as an auxiliary label to help the model learn the logic of the summary. the model predicts the key point sequence first and then uses it to guide the prediction of the summary. didi requires its customer service agents to write summaries about dialogues with users, therefore, lack of labeled data is not an issue in their use case. our work can be considered as a continuous form of the hierarchical attention network implemented in @xcite . @xcite which was developed for document classification and the prediction had to be a categorical variable, we presented a few different types of decoders that can make predictions on either categorical outcomes (such as customer sentiment) or sequential outcomes (such as customer issues and agent actions). in this paper we explore the application of hierarchical attention mechanism in dialogue summarization in the absence of labeled data. to the best of our knowledge this is the first such instance. ## conclusion and future work the conventional approach to summarize documents/texts does not apply to cases with lack of existing summaries to supervise a training process. in this paper, we propose a novel approach based on ancillary labels and attention mechanism to address this issue. we show that this approach generates intuitive summaries and the good performance does not deteriorate as the length of dialogue increases. we test the proposed models on amazon customer service contacts and reveal that the atten-tion mechanism can correctly locate and retrieve relevant sentences and words which are then used to form the summaries. we leave several summarization challenges as open questions. for example, in our approach, we set the summary length threshold of selected sentences and words to 20% . further evaluation can be performed to observe the summarization performance with respect to different summary lengths. furthermore, an alternative model that can jointly learn the optimal number of sentences and words to extract during training would be worthy of interest. in our work, we rank the sentences/words with their attention scores and use the sentences/words with the highest scores as the summary. in other words, we are more interested in the relative ranking of each sentence/word rather than its exact scores. therefore, another future work direction is to incorporate a ranking algorithm in attention retrieval. lastly, machine-generated extractive summaries may contain multiple sentences which are similar in meaning, hence not a desirable factor. it is also worthwhile to explore a redundancy elimination approach that takes a machine generated summary as a rough summary, identifies the semantic similarity between sentences in the summary, and further refines the summary by removing redundant segments.