What Helps Transformers Recognize Conversational Structure? Importance of Context, Punctuation, and Labels in Dialog Act Recognition (2021)

Abstract: Dialog acts can be interpreted as the atomic units of a conversation, more fine-grained than utterances, characterized by a specific communicative function. The ability to structure a conversational transcript as a sequence of dialog acts—dialog act recognition, including the segmentation—is critical for understanding dialog. We apply two pre-trained transformer models, XLNet and Longformer, to this task in English and achieve strong results on Switchboard Dialog Act and Meeting Recorder Dialog Act corpora with dialog act segmentation error rates (DSER) of 8.4% and 14.2%. To understand the key factors affecting dialog act recognition, we perform a comparative analysis of models trained under different conditions. We find that the inclusion of a broader conversational context helps disambiguate many dialog act classes, especially those infrequent in the training data. The presence of punctuation in the transcripts has a massive effect on the models’ performance, and a detailed analysis reveals specific segmentation patterns observed in its absence. Finally, we find that the label set specificity does not affect dialog act segmentation performance. These findings have significant practical implications for spoken language understanding applications that depend heavily on a good-quality segmentation being available.

## introduction human dialog is a never-ending source of diversity, abundant with exceptions and surprising ways to express one's thoughts. as a community, we have spent a massive effort in the past few decades to help the machine achieve even the slightest level of understanding of our means of communication. remarkably, to some extent, we have succeeded. a consequence of this fact is the widespread presence of so-called voice assistants, that is, conversational agents of limited capabilities, which have gained much popularity in recent years. while the main focus of modern dialog research is placed on these human-machine interactions, it is the conversation between humans that poses the greatest challenges to spoken language understanding. consider the task of intent recognitionin a goal-oriented dialog, where the human expects their machine interlocutor to have only limited understanding capabilities, one can reasonably expect there to be a single, self-contained and straightforward utterance expressing the person's request. @xcite show in a subjective evaluation of alexa users that they consider such a conversation ''more difficult'' than talking to a human. with a simpler dialog structure, it is natural to approach intent recognition as a multiclass classification task, by classifying each utterance's underlying intent. the same task of intent recognition becomes much more complex when the dialog involves two or more humans. their conversations are riddled with various disfluencies, such as discourse markers, filled pauses, or back-channeling @xcite . @xcite propose multitask training for a disfluency detection model capable of spotting hesitations, prepositional phrase restarts, clausal restarts, and corrections. spontaneous dialogs are also characterized by much more dynamic structure than written text data. @xcite show that dialog may be viewed as a sequence of incremental contributions-called split utterancesrather than complete sentences, and propose the dynamic syntax paradigm, claiming that standard syntactic models are insufficient to capture dialog. another study @xcite finds that up to 20% of utterances in the british national corpus @xcite dialogs fit the definition of split utterances, with about 3% of them being cross-speaker utterance completions. eshghi et al. (2015) propose to view backchannels and other discourse markers as feedback in conversation that is a core component of its semantic structure, rather than a nuisance in downstream processing. this point is further argued by @xcite , who propose incremental models for detecting miscommunication phenomena in human-human conversations. clearly, an attempt to determine a person's intent grows beyond a turn-level classification task in such scenarios. dialog acts are vital to understanding the structure of the dialog. one of their modern definitions states that they are atomic units of conversation, which are more fine-grained than utterances and more specific in their function @xcite . the part of utterance that forms a dialog act is also known as a functional segment. recently, the definition, taxonomy, and annotation process of dialog acts has been standardized through an iso norm @xcite @xcite . earlier studies on this topic typically used custom-tailored dialog act sets-notably, this category includes the dialog act markup in several layers (damsl) scheme (core and allen, 1997), which was later adopted and modified to annotate the switchboard corpus @xcite , illustrated in figure 1 . interestingly, dialog acts are related to the philosophy of language speech acts theory introduced initially by @xcite , in the sense that they view utterances as actions performed by the speakers. dialog act recognition typically entails two tasks: dialog act segmentation (das) and dialog act classification (dac). in this work, we address both of them jointly and refer to their combination further as dialog act recognition. at the time of the conception of the first widely studied corpus for this task, the switchboard dialog act (swda), das was considered a problem too difficult to address, and the pioneering research focused solely on the classification of dialog acts given the oracle segmentation @xcite . more recent work attempts to retrieve the segmentation through conditional random fields (crfs) or recurrent neural networks (rnns). however, these models still suffer from a significant margin of error, as shown by @xcite and later in section 5.1. it is worth noting that in some downstream applications, the availability of high-quality segmentation is valuable regardless of any classification errors: some examples include intent classification @xcite , semantic clustering @xcite , or temporal sentiment analysis @xcite , all of which heavily depend on the segmentation. to the best of our knowledge, the das performance of transformer models @xcite has not yet been investigated. transformers recently demonstrated state-of-the-art performance across a range of natural language processing (nlp) tasks when combined with language model pre-training @xcite @xcite . a major obstacle in applying transformer models to das is their o(n 2 ) computational complexity with respect to the input sequence length, making it infeasible to process conversations longer than a couple of hundred tokens. thus, there are few transformer applications to segmentation tasksfor example, glavas and somasundaran (2020) employed transformers for topic segmentation, but they assume that text had already been segmented and use the sentence representations instead of word representations as input to transformers. to address the transformers' limitations, we investigate two approaches. in the first one, we use xlnet @xcite , a model based on the transformerxl architecture @xcite , which is capable of processing the input sequence in windows while propagating the activations of the intermediate layers across as additional inputs in the following window. in the second approach, we use longformer @xcite , which processes the whole sequence in a single pass, but for each token attends only to neighboring n other tokens, reducing the complexity to o(mn), which is linear with respect to the input length. furthermore, we ask several questions to better understand the factors affecting dialog act recognition and design the experiments accordingly: ## results in this section, we present the results of our experimental evaluation. each result table is first split into lower and nolower sections, which, respectively, stand for a lowercase transcript with no punctuation, and an original case transcript with punctuation symbols. for both scenarios, we always show the results on both mrda and swda datasets. ## discussion this section presents a detailed analysis of various factors affecting dialog act segmentation and recognition performance. in particular, we look into the effects of label set specificity, punctuation, and context. ## conclusions we investigated how two transformer models capable of dealing with long sequences, xlnet and longformer, can be applied to dialog act recognition. we used the well-studied swda and mrda corpora and compared the performance with an established bigru baseline. first, we showed that the pre-trained transformers offer a substantial improvement with respect to to bigru when processing individual speaker turns, without any additional context. then, we proposed adapting the transformers to consider a broader dialog context through turn concatenation with the turn token, the use of joint coding, and local attention patterns or windowed processing. with this improvement, we achieved strong segmentation results on swda and mrda dialog act recognition with dser of 8.4% and 14.2% on the original transcripts and competitive results on lowercase transcripts with no punctuation (17.5% and 32.9%). we found that xlnet was able to get the most out of the additional dialog context. we observed that the additional context is the most beneficial for segmentation while also improving the classification performance. on a practical note, longformer allowed for approximately five times quicker inference on a modern cpu. across all of our experiments, it was evident that punctuation and original character cases were crucial for both segmentation and classification performance. no other factor influences the results as much-the best lowercase-transcript model (broad context xlnet+prop) still lags behind the simplest unmodified-transcript model (turn-context bigru). we analyzed the effect of punctuation and found that it is often correlated with some dialog act classes. the model leverages punctuation as a cue, especially to insert segment boundaries, but to a lesser extent also to classify dialog acts (e.g., question marks in questions). by considering different dialog act label sets available in mrda and a pure segmentation task, we found that xlnet's segmentation performance does not depend on the dialog act labels, further with segmentation experiments on swda. regardless of the label set size (or whether the task is pure segmentation), the model performs just as well. finally, we found that the addition of broader context is beneficial for the model to learn rare dialog act classes-without it, more than 50% of dialog act classes were never correctly recognized even once in swda. with the inclusion of context, that number decreased to less than 10%. our findings have significant practical implications for applications that depend on text segmentation, such as the automatic discovery of intents and processes in a given domain or building graphs describing conversational flow from unstructured transcripts. we have shown that the dialog act labels do not have to be specific in order to be able to retrieve good segmentation automatically. this can significantly ease the annotation efforts, removing the need to memorize large label sets for the annotators. furthermore, we show that the current pre-trained transformer models suffer from limitations when punctuation is not available. they tend to under-segment the text, often merging disfluencies with neighboring dialog acts. while these phenomena would likely affect, for example, systems trying to measure the semantic similarity of two segments, we expect that even the segmentation predicted on lower-case text would be useful in practical applications. it is interesting to see whether automatically retrieved punctuation can mitigate the gap between manual annotation and no punctuation; we consider this a promising candidate for future work. to foster further research in this direction, we make our code available under the apache 2.0 license. 5