Speaker-change Aware CRF for Dialogue Act Classification (2020)

Abstract: Recent work in Dialogue Act (DA) classification approaches the task as a sequence labeling problem, using neural network models coupled with a Conditional Random Field (CRF) as the last layer. CRF models the conditional probability of the target DA label sequence given the input utterance sequence. However, the task involves another important input sequence, that of speakers, which is ignored by previous work. To address this limitation, this paper proposes a simple modification of the CRF layer that takes speaker-change into account. Experiments on the SwDA corpus show that our modified CRF layer outperforms the original one, with very wide margins for some DA labels. Further, visualizations demonstrate that our CRF layer can learn meaningful, sophisticated transition patterns between DA label pairs conditioned on speaker-change in an end-to-end way. Code is publicly available.

## introduction a conversation can be seen as a sequence of utterances. the task of dialogue act (da) classification aims at assigning to each utterance a da label to represent its communicative intention. dialogue acts originate from the notion of illocutionary force (speaker's intention in delivering an utterance) introduced back in the theory of speech act @xcite . das are assigned based on a combination of syntactic, semantic, and pragmatic criteria @xcite . change speaker utterance da -b of course i use, sd true @xmath0. x true b credit cards. + false b i have a couple of credit cards sd true a yeah. b true b and, uh, use them. + true a uh-huh, b false a do you use them a lot? qy true b oh, we try not to. ng as shown in it is difficult to predict the da of a single utterance without having access to the other utterances in the context. for instance, for an utterance such as "yeah", it is hard to tell whether the associated da should be 'agreement', 'yes answer' or 'backchannel'. plus, different labels have different transition probabilities to other labels. e.g., an initial greeting da is very likely to be followed by another greeting da. likewise, a question da is more likely to be followed by an answer da. to summarize, it is necessary for a da classification model to capture dependencies both at the utterance level and at the label level. recent works @xcite @xcite @xcite @xcite treat da classification as a sequence labeling problem. the bilstm-crf model @xcite , originally introduced for the tasks of pos tagging, chunking and named entity recognition, is the most widely used architecture. in it, a bidirectional recurrent neural network with lstm cells is first applied to capture the dependencies among consecutive utterances, and then, a conditional random field (crf) layer is used to capture the dependencies among consecutive da labels. crf is a discriminative probabilistic graphical framework @xcite used to label sequences @xcite . it models the conditional probability of a target label sequence given an input sequence. general crf can essentially model any kind of graphical structure to capture arbitrary dependencies among output variables. for nlp sequence labeling tasks, linear chain crf is the most common variant. the labels are arranged in a linear chain, i.e., only neighboring labels are dependent (first-order markov assumption). the bilstm-crf architecture employs a linear chain crf. hence, for brevity, in the rest of this paper, the term crf is short for linear chain crf. recently, neural versions of the crf have been developed mainly for nlp sequence labeling tasks @xcite @xcite . while traditional crf requires defining a potentially large set of handcrafted feature functions (each weighted with a parameter to be trained), neural crf has only two parameterized feature functions (emission and transition) that are trained with the other parameters of the network in an end-to-end fashion. ## motivation most sequence labeling tasks in nlp, such as pos tagging, chunking, and named entity recognition, involve only two sequences: input and target. in da classification however, we have access to an additional input sequence, that of speaker-identifiers. this extra input could, in principle, greatly improve da prediction. indeed, research on turn management @xcite has shown that dialogue participants do not start or stop speaking arbitrarily, but follow an underlying turn-taking system to occupy or release the speaker role @xcite . for instance, the last two utterances in however, state-of-the-art da classification models ignore the sequence of speaker-identifiers @xcite @xcite . this is a clear limitation. to address this limitation, we propose in this paper a simple modification of the crf layer where the label transition matrix is conditioned on speaker-change. we evaluate our modified crf layer within the bilstm-crf architecture, and find that on the swda corpus, it improves performance compared to the original crf. furthermore, visualizations demonstrate that sophisticated transition patterns between da label pairs, conditioned on speaker-change, can be learned in an end-to-end way. ## related work in this section, we first introduce the two major da classification approaches, and then focus on previous work involving the use of bilstm-crf and speaker information. multi-class classification. in this first approach, consecutive da labels are considered to be independent. the da label of each utterance is predicted in isolation by a classifier such as, e.g., naive bayes @xcite , maxent @xcite , or svm @xcite . since the first application of neural networks to da classification by @xcite , deep learning has shown promising results even with some simple architectures @xcite . more recent work developed more advanced models, and started taking into account the dependencies among consecutive utterances @xcite @xcite . for example, in @xcite , the representations of the current utterance and the three preceding utterances are fed into a rnn, and the last annotation is used to predict the da label of the current utterance. sequence labeling. in the second approach, the da labels for all the utterances in the conversation are classified together. traditional work uses statistical approaches such as hidden markov models (hmm) @xcite @xcite and crfs @xcite @xcite with handcrafted features. in hmm based approaches, the da labels are hidden states and utterances are observations emanating from these states. the hidden states are evolving according to a discourse grammar, which essentially is an n-gram language model trained on da label sequences. following advances in deep learning, neural sequence labeling architectures @xcite @xcite have set new state-of-the-art performance. two major architectures have been tested: bilstm-softmax @xcite @xcite and bilstm-crf. this study focuses on the bilstm-crf architecture. bilstm-crf. @xcite were the first to introduce the bilstm-crf architecture for da classification. their model is hierarchical and consists of two levels, where at level 1, the text of each utterance is separately encoded by a shared bidirectional lstm (bilstm) with last-pooling, resulting in a sequence of vectors. at level 2, that sequence is passed through another bilstm topped by a crf layer. at test time, the optimal output label sequence is retrieved from the trained model by viterbi algorithm @xcite . @xcite and @xcite improved on the previous model by adding different attention mechanisms. @xcite discovered that performing topic classification as an auxiliary task, can assist in predicting da labels. the topic of each utterance is automatically determined using latent dirichlet allocation @xcite . their model consists of two bilstm-crf architectures for predicting simultaneously the target da label sequence and the target topic label sequence. this model represents the state-of-the-art in da classification. ## model here, we describe the general bilstm-crf model for da classification, shown in fig. 1 . then, in the next section, we present our modification of the crf layer that takes speaker-change into account. notation. let us denote by {(x t , y t )} t @xmath1. @xmath2, where each utterance ## our contribution given the sequence of speaker-identifiers @xmath3, we can instantly derive the sequence of speakerchanges @xmath4,t+1 } t -1 @xmath5. e.g., z 2,3 = 0 means the speaker does not change from time @xmath6. we extend the original crf so that it considers as additional input, the sequence z. that is, crf now models p (y |x, z) instead of just p (y |x). in other words, the prediction of the da label sequence is now conditioned both on the utterance sequence and the speaker-change sequence. specifically, transition scores in our modified crf layer are computed as follows: ## experimental setup dataset. we experiment on the widely-used swda foot_0 (switchboard dialogue act) dataset @xcite . this corpus contains telephone conversations recorded between two randomly selected speakers talking about one of various general topics (air pollution, music, football, etc.). in this dataset, utterances are annotated with 42 mutually exclusive da labels, based on the swbd-damsl annotation scheme @xcite . inter-annotator agreement is 84%. the frequency of the 10 most represented da labels are illustrated in fig. 2 . we can see that labels are highly imbalanced and follow a long-tailed distribution. detailed statistics for all 42 labels are provided in appendix b. in this paper, we followed the approach of @xcite @xcite , and used the '+' tag to reconnect, bottom-up, all the parts of an interrupted utterance together. e.g., in models were trained with the adam optimizer @xcite . early stopping was used on the validation set with a patience of 5 epochs and a maximum number of epochs of 100. the best epoch was selected as the one associated with the highest validation accuracy. usually, the best epoch was within the first 10. we set our batch-size to be 1, i.e, one conversation for one training step. batch sizes of 1, 2, 4, 8, and 16 were also tried, without observing significant differences. performance comparison. results show that, in terms of overall accuracy on the test set of 42 da labels, our model a) outperforms the base model b) by 1%. moreover, the small standard deviations highlight the consistency of this improvement over the 10 runs. note that this performance gain is solely caused by our modified crf layer capturing speaker-change, and is greater than the gains of 0.26% @xcite and 0.09% @xcite reported by previous attempts at leveraging speaker information. to interpret the results in more detail, we show in fig. 3 the confusion matrices of our model and the base model, for the 10 most frequent da labels, representing close to 91% of all annotations. the rows correspond to true classes, and the columns to predicted classes. 5 by looking at the diagonals, we can see that our model (on the left) better predicts 6 labels out of 10 with absolute accuracy gains of up to 5.9% (for statement-opinion, sv) foot_3 and is on par with the baseline model for one label (non-verbal, x), at 98.9%. by looking at off-diagonal values, miss rates are decreased up to 5.8% (for sv misclassified as sd) by our model. also, our model provides a large boost for the (acknowledge/backchannel, agree/accept), or (b, aa) pair. it increases the respective accuracies by 1.6% (89.4%→91.0%) and 4.1% (63.8%→67.9%). the respective miss rates are decreased by 0.9% (4.9%→4.0%) and 3.9% (25.3%→21.4%), respectively for b misclassified as aa and aa misclassified as b. this is to be noted, as these two labels are among the most frequently confused pairs @xcite . although our model achieves significant gains on a majority of the most frequent labels, it decreases performance for the most frequent label, sd, which accounts for 36.9% of all labels, as shown by fig. 2 . this explains why, in terms of overall accuracy, our improvements are modest. in addition, the performance drop regarding sd can be interpreted as a consequence of the trade-off between sd and sv, since the distinction between them was very hard to make even by annotators @xcite . this can be demonstrated in terms of precision, recall, and f1 score, as shown in we can observe in confusion matrices. the benefits of considering speaker information vary across da labels. our model and the base model performed very closely on 4 labels: non-verbal (x), conventional-closing (fc), appreciation (ba), and yes-no-question (qy). we found that the utterances of these labels contain clear lexical cues that can be mapped to corresponding da labels in a non-ambiguous way. some examples include "<@xmath9, "bye-bye." → fc, "that's great." → ba, and "do you ...?" → qy. in other words, predicting well these four das does not require having access to speaker information. it can be done solely from the text of the current utterance. having access to context is not even required. this explain why our speaker-aware crf is not helpful here. this interpretation is supported by the fact that, as explained in appendix a, our model is most useful for the das that require speaker-change awareness. ensembling and joint training. since the model using our crf and the model using the vanilla crf appear to have their own strengths and weaknesses, we tried combining them to improve performance. more precisely, we experimented with two approaches. first, an ensembling approach that combines the predictions of the two trained models by averaging their emission and transition scores (respectively). second, a joint training approach that combines the two models into a new one and trains it from scratch. in that second model, our crf and the vanilla crf are combined, and transition scores are computed as: results in for the sake of completeness, we repeated these experiments with our model. results, available in rows a1 and a2 of results in these results are not very surprising, since, on other tasks than da classification, multiple recent works have reported that bilstm-crf does not always outperform @xcite @xcite . for example, in @xcite , crf brought improvement for named entity recognition and chunking, but not for pos tagging. one of the reasons might be that the simple markov label transition model of crf does not give much information gain over strong neural encoding @xcite . that is, bilstm may be expressive enough to implicitly capture the obvious dependencies among labels. in any case, the model equipped with our crf layer (row a) outperforms all variants of bilstm-softmax and bilstm-vanilla crf. this suggests that our crf layer can capture richer and not obvious label dependencies given speaker information, which, in the end, makes the use of a crf layer valuable in assisting da classification. as far as the transition matrix g of the vanilla crf layer (right of fig. 4 ), we can observe that it tries to capture, at the same time, the transition patterns of both the "speaker changed" and "speaker unchanged" cases. for example, sv equally tends to transfer to sv, aa and ar in g, while the transitions towards sv/aa,ar are only probable if the speaker stays the same/changes, as clearly illustrated by g 0 /g 1 . obviously, using two matrices as in our approach gives much more expressiveness to the model in capturing da label transition patterns. to summarize, visualizations show that the transition matrices g 0 and g 1 in our modified crf layer are able to encode speaker-change-aware, sophisticated da transition patterns. ## discussion note that, for our utterance encoder, we also experimented with a bidirectional lstm (also with last pooling), as in @xcite , and with a bidirectional lstm with self-attention mechanism @xcite . however, since they were not giving better results, we opted for the simplest option. one possible explanation for the self-attention mechanism not being helpful could be the very short size of the utterances in the swda dataset (68.7% of utterances are shorter than 10 tokens). on such short sequences, a rnn with a 300-dimensional hidden layer is very likely able to keep the full sequence into memory. as far as why a forward rnn suffices, it should be noted that with last pooling, the last time step corresponds to the first annotation of the backward rnn. this is not adding much information to the last annotation of the forward rnn, which represents the entire sequence. our goal was not to exceed the state-of-the-art accuracy reported in @xcite and @xcite , this is why we used simple models in all of our experiments. however, our improved crf layer can be directly plugged into more advanced architectures, such as att-bilstm-crf @xcite or transformer-crf @xcite @xcite , and should in principle be able to boost performance regardless of the model used. future research should be devoted to address the limitation of the markov property of crf layer, by developing a model that is capable of capturing longer-range dependencies within and among the three sequences: that of speakers, utterances, and da labels.