DEAM : Dialogue Coherence Evaluation using AMR -based Semantic Manipulations (2022)

Abstract: Automatic evaluation metrics are essential for the rapid development of open-domain dialogue systems as they facilitate hyper-parameter tuning and comparison between models. Although recently proposed trainable conversation-level metrics have shown encouraging results, the quality of the metrics is strongly dependent on the quality of training data. Prior works mainly resort to heuristic text-level manipulations (e.g. utterances shuffling) to bootstrap incoherent conversations (negative examples) from coherent dialogues (positive examples). Such approaches are insufficient to appropriately reflect the incoherence that occurs in interactions between advanced dialogue models and humans. To tackle this problem, we propose DEAM, a Dialogue coherence Evaluation metric that relies on Abstract Meaning Representation (AMR) to apply semantic-level Manipulations for incoherent (negative) data generation. AMRs naturally facilitate the injection of various types of incoherence sources, such as coreference inconsistency, irrelevancy, contradictions, and decrease engagement, at the semantic level, thus resulting in more natural incoherent samples. Our experiments show that DEAM achieves higher correlations with human judgments compared to baseline methods on several dialog datasets by significant margins. We also show that DEAM can distinguish between coherent and incoherent dialogues generated by baseline manipulations, whereas those baseline models cannot detect incoherent examples generated by DEAM. Our results demonstrate the potential of AMR-based semantic manipulations for natural negative example generation.

## introduction despite the effectiveness of large pretrained language models (radford et al., 2019; lewis et al., 1 our proposed manipulations, data, and trained models can be found at https://github.com/pluslabnlp/ deam not everyone is into this type. have you seen love, death and robots? did you like it? yes, we watched in three hours. do you like science fantasy? yes, i watched it with my friend. yes, i saw one in a factory a few weeks ago. not that much. have you seen love, death and robots? which is your favourite episode? have you seen love, death and robots? did you like it? she lives in nyc. i do know its director. he should have a very creative mind. they spend many years on it. nice to chatting with you! have you work for love, death and robots? did you like me? not that much. no job is about this type. yes, we watched in three hours. i ignored to watch it. yes, i watched it with my friend. figure 1 : examples of human-human (first dialogue) and human-chatbot (second dialogue) conversations alongside manipulations resulted from baseline (indicated with yellow color) and our proposed perturbations (indicated with purple color), respectively. similar to the human-chatbot interaction, our manipulations result in more subtly-incoherent dialogue compared to baseline manipulations. in this work, we focus on the automatic evaluation of the coherence of dialogues, which is underexplored. coherence is a conversation-level metric that measures how well the utterances in a conversation are unified leading to a consistent interaction @xcite . previous works pursue different models such as graph-based @xcite or text-based @xcite approaches to develop automatic trainable coherence evaluation metrics. those models take a contrastive learning approach, where they build binary classifiers to differentiate positive, or coherent examples from negative, or incoherent dialogues. those classifiers are usually trained on datasets constructed by using human-human conversations as positive examples and applying text-level heuristic manipulations to generate incoherent conversations. the text-level manipulations directly change the structures of the conversation such as shuffling the order of utterances, replacing some random utterances from external conversations @xcite @xcite , as shown in the third dialogue of figure 1 . we posit that such text-level manipulations are too simplistic to adequately represent more nuanced coherence errors presented in the current state-of-the-art dialogue systems. for example, the second conversation in figure 1 shows a humansystem interaction from the fed dataset @xcite , where the incoherence is much more subtle than the ones created by textlevel manipulations. in this paper, we investigate manipulation techniques to generate negative samples that represent coherence errors more likely to happen in the stateof-the-art dialogue systems. to this end, we propose deam, a model that uses abstract meaning representation (amr) to apply semantic-level manipulations to generate negative examples. amrs are intended to capture the meaning of a sentence by abstracting away irrelevant syntactic features. thus, injecting targeted and controlled perturbations into an amr is easy and can introduce semantic incoherence into the corresponding sentences. deam starts with parsing conversations into semantic amr representations and then injects incoherence types that are usually observed in current state-of-the-art models into the amr graphs. it concludes this process by translating the manipulated amrs back to conversations as negative examples using a controllable generation model. a fine-tuned roberta model is then trained on the created dataset to distinguish coherent and incoherent conversations as the evaluation metric. our main contributions are as follows: ## related works automatic evaluation of open-domain dialogue systems has a multifaceted nature with many finegrained quality aspects @xcite . turn-level aspects show the quality of the system's utterance given a dialogue context from different perspectives including appropriateness, relevance, engagement, and etc @xcite @xcite . whereas, conversation-level facets such as coherence, diversity, informativeness take into account the whole dialog flow @xcite @xcite . dialogue coherence evaluation is pertinent to discourse coherence since a dialogue is counted as a multi-party discourse. similar to discourse coherence, many original coherence evaluation metrics derived from the centering model for monitoring the local focus of utterances and their entities distribution @xcite @xcite . a group of studies assess the coherence of dialogues with respect to entities and dialogue acts @xcite . another inspected approach for dialogue coherence evaluation is to represent dialogue in a structured graph format where contextually dependent neighbor utterances or concepts are connected nodes in the graph @xcite @xcite . graph convolutional networks are used to complete this task. high-quality training dataset is identified as one of the momentous and indelible components in automatic coherence evaluation. some previous works construct such datasets by collecting human judgments @xcite . while many recent works rely on a more timely and costly affordable approach by automatically generating negative samples. the utterances of the coherent conversations from humanhuman interactions are manipulated by shuffling their order, inserting or replacing irrelevant utterances @xcite @xcite . in this work, we show that such changes can not truly represent machine-generated incoherent conversations. one work that is closely related to us and proposed abstract-level manipulations is ghazarian et al. (2021b). however, their application domain is open-domain story evaluation, and they use story plot, rather than amr, for the manipulation, which is more domain-specific. ## deam overview our goal is to build an evaluation metric that measures the conversation-level coherence of dialogues. we follow the trainable evaluation metrics @xcite to formulate the evaluation as a classification task. we train the evaluator on positive (coherent) and negative (incoherent) conversations, and take the predicted probability for the positive class as the coherence score. as is discussed above, the main challenge for building a reliable metric is to obtain negative samples that can adequately represent the incoherence issues presented in advanced dialogue systems. to this end, we propose to generate negative examples by leveraging amr-based manipulations. we then build a roberta-based classifier as the evaluation metric by fine-tuning roberta on the automatically generated training data. figure 2 illustrates an overview of our proposed evaluation method. the first step of deam is to apply text-to-amr models to the conversations. text-to-amr or amr parsing @xcite @xcite that translates conversation texts to directed and acyclic amr graphs containing relation edges between concept nodes @xcite has been effectively accomplished by transformer-based models in a sequence-to-sequence (seq2seq) training fashion @xcite . we use ## incoherent dialogue generation the challenge that automatic trainable evaluation metrics face is in providing training data that can appropriately replicate moderate to low quality conversations with incoherence sources that usually happen in the current dialogue models. the common solution is to apply manipulations to positive conversations. in this section, we summarize the baselines manipulations and state our proposed amr-based perturbations. ## experimental setup we compare deam and its negative example generation techniques with baseline models and manipulations. we aim to have a data-driven analysis under three setups: setup 1): in this setup, we compare deam with baseline models by varying both the data manipulation strategies and the classification models. we fix the positive examples to be the same set of human-human conversations. setup 2): since baseline models are trained on different datasets, we conduct pairwise comparisons between deam and each baseline evaluator by training on the baseline's dataset. note that we only take the positive examples from the baseline's dataset, and apply different manipulations to get setup 3): this setup is designed to show the effectiveness of different manipulations to generate negative examples. we fix the positive examples and the classifier (i.e., roberta). ## results through our experiments, we report the spearman correlation of evaluation metrics with human annotations under different experimental setups. ## conclusion reliable automatic trainable coherence evaluation metrics that can efficiently measure the dynamics of interactions between interlocutors are principally influenced by the quality of the training instances. we show that leveraging text-level manipulations can not adequately mirror the incoherence errors that current dialogue systems face. according to our study, deam can more effectively accomplish this task by relying on capabilities that amr-based semantic perturbations and pretrained language models present. we leave the investigations regarding the effectiveness of amrs for evaluating sensibility of the generated responses with taking into account knowledge bases for the future.