Energy-based Self-attentive Learning of Abstractive Communities for Spoken Language Understanding (2020)

Abstract: Abstractive community detection is an important spoken language understanding task, whose goal is to group utterances in a conversation according to whether they can be jointly summarized by a common abstractive sentence. This paper provides a novel approach to this task. We first introduce a neural contextual utterance encoder featuring three types of self-attention mechanisms. We then train it using the siamese and triplet energy-based meta-architectures. Experiments on the AMI corpus show that our system outperforms multiple energy-based and non-energy based baselines from the state-of-the-art. Code and data are publicly available.

## introduction today, large amounts of digital text are generated by spoken or written conversations, let them be human-human (customer service, multi-party meetings) or human-machine @xcite . such text comes in the form of transcriptions. a transcription is a list of time-ordered text fragments called utterances. abstractive summarization of conversations is an open problem in nlp. previous work @xcite @xcite decomposes this task into two subtasks a and b as shown in fig. 1 . subtask a, or abstractive community detection (acd), is the focus of this paper. it consists in grouping utterances according to whether they can be jointly summarized by a common abstractive sentence @xcite . such groups of utterances are called abstractive communities. once they are obtained, an abstractive sentence is generated for each group (subtask b), thus forming the final summary. acd includes, but is a more general problem than, topic clustering. indeed, as shown in fig. 2 , communities should capture more complex relationship than simple semantic similarity. also, two utterances may be part of the same community even if they are not close to each other in the transcription. finally, a given utterance may belong to more than one community, which results in overlapping groupings (e.g., a and d in fig. 2 ), or be a singleton community (b in fig. 2 ). in this paper, we depart from previous work and argue that the acd subtask should be broken down into two steps, a1 and a2 in fig. 1 . that is, summary-worthy utterances should first be extracted from the transcription (a1), and then, grouped into abstractive communities (a2). this process is more consistent with the way humans treat the summarization task. e.g., during the creation of the ami corpus @xcite , annotators were first asked to extract summaryworthy utterances from the transcription, and then to link the selected utterances to the sentences in the abstractive summary (links in fig. 2 ), i.e., create communities. abstractive summaries comprise four sections: abstract, actions, problems, and decisions. step a1 plays an important filtering role, since in practice, only a small part of the original utterances are used to construct the abstractive communities (17% on average for ami). however, this step is closely related to extractive summarization, which has been extensively studied in the conversational domain @xcite ## energy framework energy-based modeling (ebm) @xcite is a unified framework that can be applied to many machine learning problems. in ebm, an energy function assigns a scalar called energy to each pair of random variables (x, y ). the energy can be interpreted as the incompatibility between the values of x and y . training consists in finding the parameters w * of the energy function e w that, for all (x i , y i ) in the training set s of size p , assign low energy to compatible (correct) combinations and high energy to all other incompatible (incorrect) ones. this is done by minimizing a loss functional 2 l: for a given x, prediction consists in finding the value of y that minimizes the energy. ## proposed utterance encoder notation. the time t (as superscript) denotes the position of a given utterance in the conversation of length t , and the position i (as subscript) denotes the position of a token within a given utterance of length n . e.g., u t 1 is the representation of the first token of u t , the t th utterance in the transcription. word encoder. as shown in the upper right corner of fig. 4 , we obtain u t i by concatenating the pre-trained vector of the corresponding token with the discourse features of u t (role, position and dialogue act), and passing the resulting vector to a dense layer. utterance encoder. as shown in the center of fig. 4 , we represent u t as a sequence of n ddimensional token representations u t 1 , . . . , u t n . in addition, because there is a strong time dependence between utterances (see fig. 2 ), we inform the model about the preceding and following utterances when encoding u t . to accomplish this, we prepend (resp. append) to u t a context vector containing information about the previous (resp. next) utterances, finally obtaining u @xmath0, u t 1 , . . . , u t n , u t @xmath1. we then use a non-stacked bidirectional recurrent neural network (rnn) with gated recurrent units (gru) @xcite to transform u t into a sequence of annotations h @xmath2. in practice, the pre and post-context vectors indirectly initialize the left-to-right and right-to-left rnns with information about the utterances preceding and following u t . this is similar in spirit to the warm-start method of @xcite , that directly initializes the hidden states of the rnns with the context vectors. self-attention. we then pass the annotations h t to a self-attention mechanism @xcite . more precisely, h t go through a dense layer and the output is compared via dot product with a trainable vector u γ , initialized randomly. then, a probability distribution over the n + 2 tokens in u t is obtained via a softmax: the attentional vector for u t is finally computed as a weighted sum of its annotations, and, as shown in fig. 4 , is finally passed to a dense layer to obtain the utterance embedding u @xmath3: content-aware self-attention. at level 1, we apply the same attention mechanism to each utterance in this mechanism is the same as in eq. 9, except for two differences. first, we operate directly on the matrix of token vectors of the previous utterance u t-1 rather than on rnn annotations. second, there is an extra input that consists of the elementwise sum of the token vectors of the current utterance u t . the latter modification is inspired by the coverage vectors used in translation and summarization to address under(over)-translation and repetition, e.g., @xcite . in our case, we hope that by letting the model know about the tokens in the current utterance u t , it will be able to extract complementary -rather than redundant-information from its context, and thus produce a richer embedding. to recapitulate, the content-aware attention mechanism transforms the sequence of utterance matrices u t-c , . . . , u t-1 ∈ r c×n ×d into a sequence of vectors u t-c , . . . , u t-1 ∈ r c×d . these vectors are then aggregated into a single precontext vector u t @xmath4. note that since there is no inherent difference between preceding and following utterances 3 , we use the same content-aware self-attention mechanism for the pre and post contexts. this also gives us a more parsimonious and faster model. one should note, however, that the pre and post-context encoders still differ in terms of their time-aware attention mechanisms at level 2, described next. ## community detection once the utterance encoder g w presented in section 4 has been trained within the siamese or triplet meta-architecture presented in section 3, it is used to project the summary-worthy utterances from a given test transcription to a compact embedding space. we assume that if training was successful, the distance in that space encodes community structure, so that a basic clustering algorithm such as k-means (macqueen, 1967) is enough to capture it. however, since we need to detect overlapping communities, we use a probabilistic version of k-means, the fuzzy c-means (fcm) algorithm @xcite . fcm returns a probability distribution over all communities for each utterance. more details are provided in app. e. ## experimental setup 6.1 dataset we experiment on the ami corpus @xcite , with the manual annotations v1.6.2. the corpus contains data for more than 100 meetings, in which participants play 4 roles within a design team whose task is to develop a prototype of tv remote control. each meeting is associated with the annotations described in the introduction and shown in fig. 2 . there are 2368 unique abstractive communities in total, whose statistics are shown in ## results context sizes. larger contexts bring richer information, but increase the risk of considering unrelated utterances. using our proposed encoder within the triplet meta-architecture, we tried different values of c on the validation set, under two settings: (pre, post) = (c, 0), and (pre, post) = (c, c). results are shown in fig. 5 . we can observe that increasing c always brings improvement, with diminishing returns. results also clearly show that considering the following utterances in addition to the preceding ones is useful. note that the curves look similar for f 1@@xmath5. in the end, we selected (11,11) as our best context sizes. quantitative results. final test set results are shown in one of the reasons for the superiority of our utterance encoder is probably that it considers contextual information while encoding the current utterance, while han and ld take as input the context utterances together with the current utterance, without distinguishing between them. moreover, we use an attention mechanism dedicated to temporality, whereas han is only able to capture an implicit notion of time through the use of recurrence (rnn), and ld, with its dense layers, completely ignores it. also, all variants of our model using best context sizes (11,11) outperform the ones using reduced (3,0) or no (0,0) context, regardless of the metaarchitecture. this confirms the value added by our context encoder. for siamese, our model outperforms its two variants (ca-s and s-s) for all metrics, indicating that both the content-aware and the time-aware selfattention mechanisms are useful. however, it is interesting to note that when training under the triplet configuration, the ca-s variant of our model is better, suggesting that in that case, the content-aware mechanism is beneficial, but the time-aware one is not. lcseg (row m) and tf-idf (11,11) (row n3) are the best of all (un)supervised baseline systems, but both perform significantly worse than all energybased approaches, highlighting that training with the energy framework is beneficial. in terms of omega index, supervised baseline systems are logically better than unsupervised ones. qualitative results. we visualize in app. a that the three self-attention mechanisms behave in a cooperative manner to produce a meaningful utterance representation. we also inspect the closest utterances to a given query utterance in app. d. we also visualize the time-aware self-attention coefficients in fig. 6 , and find that interestingly, the distributions over the pre and post-context are not symmetric. indeed, only the utterances immediately following u t (t + 1 → t + 5) seem to matter, while the attention weights are much more uniform across the utterances preceding u t . this suggests that in dialogues, considering a long history of preceding utterances helps understanding the current one. overall, the three terms (see eq. 14) altogether do produce a universal function that decreases as time distance increases, which is in accordance with intuition. simplified task. finally, we also experimented on a much simpler task, where only the communities of type abstract were considered. this makes acd much simpler, because most of the overlapping communities are of the other types (see ## conclusion this paper proposes one of the first applications of energy-based learning to acd. using the siamese and triplet meta-architectures, we showed that our novel contextual utterance encoder learns better distance and communities than state-of-the-art competitors.