Is this Dialogue Coherent? Learning from Dialogue Acts and Entities (2020)

Abstract: In this work, we investigate the human perception of coherence in open-domain dialogues. In particular, we address the problem of annotating and modeling the coherence of next-turn candidates while considering the entire history of the dialogue. First, we create the Switchboard Coherence (SWBD-Coh) corpus, a dataset of human-human spoken dialogues annotated with turn coherence ratings, where next-turn candidate utterances ratings are provided considering the full dialogue context. Our statistical analysis of the corpus indicates how turn coherence perception is affected by patterns of distribution of entities previously introduced and the Dialogue Acts used. Second, we experiment with different architectures to model entities, Dialogue Acts and their combination and evaluate their performance in predicting human coherence ratings on SWBD-Coh. We find that models combining both DA and entity information yield the best performances both for response selection and turn coherence rating.

## introduction dialogue evaluation is an unsolved challenge in current human-machine interaction research. this is particularly true for open-domain conversation, where compared to task-oriented dialogue (i.e., restaurant reservations), we do not have a finite set of entities and intents, and speakers' goals are not defined a priori. in this work, we address the problem of dialogue evaluation from the perspective of dialogue coherence and how this concept can be formalized and evaluated. our approach could be applied to both task-oriented and non-task-oriented dialogue. coherence in language, i.e., the property which determines that a given text is a logical and consistent whole rather than a random collection of sentences, is a complex multifaced concept which has been defined in different ways and to which several factors contribute @xcite , e.g., rhetorical structure @xcite , topics discussed, and grounding @xcite . while much recent work has focused on coherence for response generation @xcite @xcite , we argue that there is still much to be understood regarding the mechanisms and substructures that affect human perception of dialogue coherence. in our approach, in particular, we are interested in studying the patterns of distribution of entities and dialogue acts (das), in regards to dialogue coherence. approaches to coherence based on entities have been studied extensively by the natural language processing literature @xcite , especially in text (e.g., news, summaries). coherence evaluation tasks proposed by this literature @xcite have the advantage of using weakly supervised training methodologies, but mainly considering documents as-a-whole, rather than evaluating coherence at the utterance level. the dialogue literature @xcite , on the other hand, has focused mainly on coherence in connection to das, a generalized version of intents in dialogue (e.g., yes-no-question, acknowledgement). recent work @xcite , in particular, showed the importance of both das and entities information for coherence modeling in dialogue. however, even in this case dialogue coherence was rated for entire dialogues rather than studying turn coherence structures. in this work, we investigate underlying conversation turn substructures in terms of da and entity transitions to predict turn-by-turn coherence in dialogue. we start by annotating a corpus of spoken open-domain conversations with turn coherence ratings, the switchboard coherence corpus (swbd-coh) foot_0 , and perform an analysis of the human perception of coherence in regards to das and entities. a multiple regression analysis shows the importance of both types of information for human rating of coherence. secondly, we present novel neural models for turn coherence rating that combine das and entities and propose to train them using response selection, a weakly supervised methodology. while previous work on response selection @xcite is mainly based on using the entire text as input, we deliberately choose to use only entities and das as input to our models, in order to investigate entities and das as a signal for turn coherence. finally, we test our models on the swbd-coh dataset to evaluate their ability to predict turn coherence scores foot_1 . the main contributions of this work are: ## related work coherence evaluation in text coherence models trained with weakly supervised methodologies were first proposed for text with applications to the news domain and summarization @xcite . these models rely on the entity grid, a model that converts the entities (noun phrases) mentioned in the text to a sentence-by-sentence document representation in the form of a grid. the tasks on which coherence models in this line of research are usually evaluated are sentence ordering @xcite , i.e., ranking original documents as more coherent than the same documents with the order of all sentences randomly permuted, and insertion, i.e., ranking original documents as more coherent than documents with only one sentence randomly misplaced. these tasks are still considered standard to this day and found wide applications, especially for text @xcite . recent models proposed for these tasks are based on convolutional neural networks (nguyen and @xcite , also applied to thread reconstruction @xcite , while the current state-of-the-art is based on a combination of bidirectional long short-term memory encoders and convolution-pooling layers @xcite . these tasks, however, consider documents as-a-whole and rely mainly on entities information. coherence evaluation in dialogue models for dialogue coherence evaluation have mainly been explored using supervised approaches, i.e., training on corpora with human annotations for coherence, mostly at the turn level @xcite @xcite @xcite . different approaches tried to apply the standard coherence tasks to conversational domains such as dialogue and threads, but mainly considering the evaluation of dialogues as-a-whole @xcite @xcite @xcite @xcite . in particular, @xcite found that discrimination might be over-simplistic for dialogue coherence evaluation when considering dialogue act (da) information. in this work, we propose a novel framework to model entities and das information for turn coherence prediction using a weakly supervised training methodology. furthermore, our focus is on predicting coherence of single turns rather than entire dialogues. response selection as a task, response selection has become a standard @xcite @xcite for training both task-oriented and non-task-oriented retrieval-based dialogue models. the task proved to be useful for evaluating models in task-oriented (ubuntu), social media threads (twitter corpus), and movie dialogues (subtle corpus) @xcite . recently the task has also been proposed for pretraining models for task-oriented dialogue @xcite and for dialogue act tagging @xcite . in this work, we investigate response selection as a task for training coherence rating models for spoken dialogue. additionally, while response selection models are usually based on the entire text as input @xcite , we rely solely on entities and das information, in or-der to investigate their effect on turn coherence perception. ## methodology in this work, we are interested in the relation between dialogue acts (das) and entities and how they can be modelled to train automatic predictors of next turn coherence in non-task-based dialogue. our hypothesis is that both entities and das are useful to predict the coherence of the next turn. in order to verify such hypothesis, we first perform an analysis of entities and das patterns of distribution in the switchboard coherence (swbd-coh) corpus, a novel dataset of human-human telephone conversations from switchboard annotated with human coherence ratings per turn. secondly, we hypothesize that we can model entities and das to predict next turn coherence ratings. rather than using supervised data for coherence prediction, we use a weakly supervised training methodology, i.e. training on the task of response selection (which proved useful for other dialogue tasks @xcite ) and testing on coherence ratings. in response selection given a context, i.e. the history of the dialogue up to the current turn, and a list of next turn candidates, models are asked to rank candidates according to their appropriateness with the previous dialogue history. the positive training samples for this task are automatically generated by randomly selecting a given turn in a dialogue, and considering this turn as a positive (coherent) example with the current history of the conversation (the context). negative samples are generated by selecting other random dialogue turns, assuming that they will mostly be not appropriate as the next turn in the dialogue. in particular, we investigate two methodologies to generate negative samples from the training data automatically: internal swap: a random turn is selected from a subsequent part of the same conversation. we assume this task to be harder for coherence evaluation since typically conversations do not have radical topic shifts. external swap: a random turn is selected from other conversations. we assume this task to be easier given the probable shifts in topic. in our first set of experiments, we thus train our models on response selection. one of the possible shortcomings of the data generation procedure used in response selection, however, is the amount of false negatives. although it is assumed that the majority of negative samples generated with this methodology will not be appropriate for the context, there could still be cases in which they are. in order to verify the performance of our models based on das and entities to predict real human coherence judgments, in our second set of experiments models are tested on swbd-coh. analogously to response selection, in turn coherence rating models need to rank next turn candidates given the history of the dialogue. in this case, however, the ranking is not binary but is rather based on a graded coherence rating given by humans for next turn candidates (for further details on the swbd-coh corpus see section 4). ## data the dataset chosen for our experiments is the switchboard dialogue act corpus @xcite (swbd-da), a subset of switchboard annotated with da information. the switchboard corpus is a collection of human-human dyadic telephone conversations where speakers were asked to discuss a given topic. this dataset was chosen both to ensure comparability with previous work on dialogue coherence and because it is open-domain. also, this corpus has da annotations. interestingly, swbd-da is a real-world (transcribed) spoken corpus, so we have sudden topic changes, overlap speech, disfluencies and other typical characteristics of spoken interaction. since our goal was to study coherence in a real-world spoken dialogue setting, rather than removing these features as errors, we considered them an integral part of spoken conversations and did not remove them. response selection source dialogues are split into train, validation, and test sets (see ## models we model dialogue coherence by focusing on two features that have been closely associated to coherence in previous literature: the entities mentioned and the speakers' intents, modelled as dialogue acts (das), in a conversation. our models explore both the respective roles of entities and das and their combination to predict dialogue coherence. we investigate both standard coherence models based on support vector machines (svm) and propose novel neural ones. ## experimental set-up preprocessing entities, i.e. noun phrases (nps), and their syntactic roles were extracted and preprocessed with @xcite 's pipeline foot_3 . following the original entity grid formulation @xcite , only nps heads were kept. the das are taken from annotations on swbd-da (using the standard reduction to 42 tags compared to the damsl ones). evaluation for evaluating response selection, we use pairwise accuracy, the metric used in standard coherence tasks, which evaluates the ability of the model to rank original turns higher than each adversarial one. however, this metric is not indicative of the global ranking of all candidate turns for a given context. for this reason, we add two ranking metrics to evaluate our models: mean reciprocal rank (mrr), which evaluates the average of reciprocal ranks of all candidate turns for a context, and recall at one (r@1) and two (r@2), also used in previous work on response selection @xcite to assess the ability of the model to rank original turns respectively within the first or second rank among all candidates. compared to response selection, where we have a binary choice between coherent and negative turns, in turn coherence rating, we have a set of candidate turns each associated to a coherence score. in this case, we use accuracy, mrr, r@1 and normalized discounted cumulative gain (ndcg) to evaluate our models. accuracy was computed only for cases in which the rating of the turn was not identical across two candidate turns. mrr and r@1 were computed dynamically, that is considering the turn with the highest score within that particular context as the best one in the rank. the ndcg metric @xcite assesses the gain of a candidate according to its rank among all candidates. compared to previous metrics, ndcg allows taking into account the relevance (in our case, the coherence score) of candidates. for all metrics considered, if our models predicts the same score for two candidates, we always assume models made a mistake, i.e. among candidates with the same predicted score positive examples are ranked after the negative ones. ## results in this section, we report the results of our models for response selection. the best performing models on response selection are then evaluated on the turn coherence rating task using the switchboard coherence (swbd-coh) corpus as testset. for both tasks we compare our models to a random baseline. all reported results for neural models are averaged across 5 runs with different seeds. ## conclusions in this work, we investigate how entities and dialogue acts (das) are related to human perception of turn coherence in dialogue. in order to do so, we create a novel dataset, the switchboard coherence (swbd-coh) corpus, of transcribed open-domain spoken dialogues annotated with turn coherence ratings. a statistical analysis of the corpus confirms how both entities and das affect human judgements of turn coherence in dialogue, especially when combined. motivated by these findings, we experiment with different models relying on entities and das to automatically predict turn coherence, i.e. standard coherence models and novel neural ones. in particular, we propose a less sparse alternative, compared to the entity grid, to encode entities and das information. rather than using data annotated explicitly for the task, i.e. coherence prediction, we explore two response selection methodologies for training. we find that our newly proposed architecture outperforms standard ones in response selection. finally, we test our models on the swbd-coh corpus in order to evaluate their ability to predict real human turn coherence ratings. crucially, we find that the combination of das and entities gives the best performances. for the future work, it would be interesting to investigate how to apply large pretrained models to our task, such as bert @xcite . while pretrained models have recently been successfully explored for text-based response selection @xcite , integrating them with our proposed input representation is not a straightforward task since such models typically rely on the whole textual context, while our models do not. while there is still much to understand regarding turn coherence in dialogue, we believe our work could be a first step towards uncovering the relation between das and entities in open-domain spoken dialogue. moreover, we believe that the swbd-coh corpus could become a useful resource for the community to study coherence in open-domain spoken dialogue.