Dialogue Coherence Assessment Without Explicit Dialogue Act Labels (2020)

Abstract: Recent dialogue coherence models use the coherence features designed for monologue texts, e.g. nominal entities, to represent utterances and then explicitly augment them with dialogue-relevant features, e.g., dialogue act labels. It indicates two drawbacks, (a) semantics of utterances are limited to entity mentions, and (b) the performance of coherence models strongly relies on the quality of the input dialogue act labels. We address these issues by introducing a novel approach to dialogue coherence assessment. We use dialogue act prediction as an auxiliary task in a multi-task learning scenario to obtain informative utterance representations for coherence assessment. Our approach alleviates the need for explicit dialogue act labels during evaluation. The results of our experiments show that our model substantially (more than 20 accuracy points) outperforms its strong competitors on the DailyDialogue corpus, and performs on par with them on the SwitchBoard corpus for ranking dialogues concerning their coherence. We release our source code.

## introduction considering rapid progresses in developing open-domain dialogue agents @xcite @xcite , the need for models that compare these agents in various dialogue aspects becomes extremely important @xcite . most available methods for dialogue evaluation rely on word-overlap metrics, e.g. bleu, and manually collected human feedback. the former does not strongly correlate with human judgments @xcite , and the latter is time-consuming and subjective. a fundamental aspect of dialogue is coherence -what discriminates a high-quality dialogue from a random sequence of dialogue utterances @xcite @xcite . dialogue coherence deals with semantic relations between utterances considering their dialogue acts @xcite . a dialogue act (henceforth da) gives a meaning to an utterance in a dialogue at the level of "illocutionary force", and therefore, constitutes the basic unit of communication @xcite . a da captures what a speaker's intention is of saying an utterance without regard to the actual content of the utterance. for example, a da may indicate whether the intention of stating an utterance is to ask a question or to state a piece of information. recent approaches to dialogue coherence modeling use the coherence features designed for monologue texts, e.g. entity transitions @xcite , and augment them with dialoguerelevant features, e.g., da labels @xcite . these da labels are provided by human annotators or da prediction models. such coherence models suffer from the following drawbacks: (a) they curb semantic representations of utterances to entities, which are sparse in dialogue because of short utterance lengths, and (b) their performance relies on the quality of their input da labels. we propose a novel approach to dialogue coherence assessment by utilizing dialogue act prediction as an auxiliary task for training our coherence model in a multi-task learning (mtl) scenario (figure 1 ). our approach consists of three high-level components: an utterance encoder, a dialogue coherence model (dicoh), and a dialogue act prediction (dap) model. the layers of the utterance encoder are shared between the dap and the di-coh model. this idea enables our dicoh model to learn to focus on salient information presented in utterances considering their das and to alleviate the need for explicit da labels during coherence assessment. we evaluate our mtl-based approach on the dailydialog @xcite and switchboard (jurafsky and shriberg, 1997) english dialogue corpora in several discriminating experiments, where our coherence model, dicoh, is examined to discriminate a dialogue from its perturbations (see our core contributions are: (1) proposing an mtl-based approach for dialogue coherence assessment using dap as an auxiliary task, yielding more informative utterance representations for coherence assessment; (2) alleviating the need for da labels for dialogue coherence assessment during evaluations; (3) an empirical evaluation on two benchmark dialogue corpora, showing that our model substantially outperforms the state-of-theart coherence model on dailydialog, and performs on par with it on switchboard. ## related work early approaches to dialogue coherence modeling are built upon available models for monologue, such as the entitygrid model @xcite . entitygrid and its extensions @xcite @xcite @xcite rely on entity transitions, as proxies of semantic connectivity, between utterances. these approaches are agnostic to discourse properties of dialogues @xcite @xcite . inspired by entitygrid, @xcite define transition patterns among da labels associated with utterances to measure coherence. @xcite combine the above ideas by augmenting entity grids with utterance da labels. this model restricts utterance vectors only to entity mentions, and needs gold da labels as its inputs for training as well as evaluation. however, obtaining da labels from human annotators is expensive and using dap models makes the performance of coherence model dependent on the performance of dap models. recent approaches to dialogue coherence modeling benefit from distributional representations of utterances. @xcite quantify the coherence of dialogue using the semantic similarity between each utterance and its preceding utterances. this similarity is estimated, for example, by the cosine similarity between an utterance vector and a context vector where those vectors are the average of their pre-trained word embeddings. @xcite measure dialogue coherence based on the consistency of new concepts introduced in a dialogue with background knowledge. similarly, @xcite utilize a natural language inference model to assess the content consistency among utterances as an indicator for dialogue coherence. however, these approaches lack dialogue-relevant information to measure coherence. our mtl-based approach solves these issues: (i) it benefits from das and semantics of utterances to measure dialogue coherence by optimizing utterance vectors for both dap and coherence assessment, and (ii) it uses da labels to define an auxiliary task for training the dicoh model using mtl, instead of utilizing them in a pipeline. therefore, it efficiently mitigates the need for explicit da labels as inputs during coherence assessment. ## method we represent a dialogue between two speakers as a sequence of utterances, @xmath0, ..., utt m ]. we address the problem of designing a coherence model, dicoh, which assigns a coherence score to dial, s @xmath1. given a pair of dialogues φ = (dial i , dial j ), our dicoh model ideally assigns s dial @xmath2. instead of using gold da labels as inputs to dicoh, we use them to define an auxiliary task and model, dap, to enrich utterance vectors for dicoh in an mtl scenario. figure 2 shows a low-level illustration of our mtl-based approach. utterance encoder we use a word embedding layer, emb, to transform the words in utterance @xmath3, ..., w n ] to a sequence of embedding vectors @xmath4, ..., e n ], where n is the number of words in utt. the embedding layer can be initialized by any pre-trained embeddings to capture lexical relations. we use a bidirectional recurrent neural network with long short-term memory cells, bilstm, to map embeddings e to encode words in their utterance-level context: @xmath5, we apply a self-attention mechanism, atten, to the hidden state vectors in h u to obtain the vector representation, u, of utterance utt: generally, the attention layer, atten, for an input vector x is defined as follows: ## conclusions we propose a novel dialogue coherence model whose utterance encoder layers are shared with a dialogue act prediction model. unlike previous approaches that utilize these two models in a pipeline, we use them in a multi-task learning scenario where dialogue act prediction is an auxiliary task. our coherence method outperforms its counterparts for discriminating dialogues from their various perturbations on dailydialog, and (mostly) performs on par with them on switchboard. our model (a) benefits from dialogue act prediction task during training to obtain informative utterance vectors, and (b) alleviates the need for gold dialogue act labels during evaluations. these properties holistically make our model suitable for comparing different dialogue agents in terms of coherence and naturalness. for future work, we would like to deeply study the impacts of our perturbations on the coherence of the examined dialogues. we will also investigate to what extent the rankings of dialogues obtained by our model correlate with human-provided rankings.