Automatic Evaluate Dialogue Appropriateness by Using Dialogue Act (2023)

Abstract: Evaluation of dialogue systems requires assessing various aspects, among which appropriateness holds significance as a core element of communicative language competence. However, current evaluations heavily rely on human judgments, which are time-consuming, labor-intensive, prone to biases, and lacking objectivity. In this paper, we introduce Dialogue Act Appropriateness (DAA), a novel method that utilizes the underlying patterns of dialogue act transitions to evaluate the appropriateness of chatbot responses. We learn transition patterns from human-human dialogue corpora, evaluating chatbot appropriateness by measuring the similarity of their transition patterns to those observed in human-human dialogues. To validate DAA, we annotate a test dataset by manually evaluating the appropriateness of dialogues from multiple chatbot systems. The experimental results demonstrate a strong correlation between our evaluation metric and human ratings, establishing the reliability of DAA as a measure of dialogue appropriateness.

## introduction automatic evaluation of text generation quality has long been a challenge in the field of natural language processing (nlp) @xcite . when it comes to assessing the quality of dialogue systems, the task becomes even more complex and multifaceted. dialogue quality encompasses various aspects, including coherence, fluency, engagement, and appropriateness of responses @xcite @xcite . depending on the application and user requirements, different aspects of dialogue quality may need to be evaluated @xcite . consequently, there is a growing demand for diverse evaluation metrics capable of capturing the different facets of dialogue quality. appropriateness is a fundamental characteristic of language usage, serving as a core element of communicative language competence and an evaluative criterion for language users or learners @xcite . it encompasses the notion of using language in a manner that is suitable, contextually relevant, and socially acceptable in various communication settings. evaluating the appropriateness of language use is crucial for assessing the proficiency and effectiveness of individuals' communicative skills in both spoken and written forms of expression. however, the current assessment of appropriateness mainly relies on human judgment, which is not only time-consuming and laborious, but also leads to potential bias and lack of objectivity. according to @xcite , the concept of appropriateness is closely tied to the effective execution of dialogue acts. appropriateness serves as a necessary and sufficient condition for carrying out dialogue acts properly. these acts play a crucial role in identifying the actions performed by utterances in a conversation, shedding light on their intended meaning and purpose in terms of illocutionary force, as discussed by @xcite . by analyzing dialogue acts, researchers can gain a deeper understanding of the interactions between speakers and the objectives they aim to accomplish. this comprehension is further enhanced through the works of @xcite ; @xcite , who emphasize the significance of dialogue acts in unraveling the functions, intended meanings, and purposes of utterances. inspired by this understanding, we consider the dialogues in humanhuman corpora to exhibit a high level of appropriateness, and the dialogue act pattern within these conversations is deemed suitable. therefore, we can leverage human-human dialogues as a guiding reference to assess the appropriateness of humanchatbot dialogues, considering the patterns of dialogue act transitions. specifically, we propose an automatic evaluation method called dialogue act appropriateness (daa) to assess appropriateness based on dialogue acts. initially, we employ a dialogue act classifier to categorize dialogue utterances into sequences of dialogue acts. subsequently, we model the dialogue act transition patterns in human-human dialogue corpora to capture the typical patterns of dialogue act transitions, which serve as the reference dialogue act patterns. finally, we compare the dialogue act transition patterns of the chatbot's responses with the reference dialogue act patterns to measure the appropriateness of the chatbot's responses. to the best of our knowledge, there is currently a lack of annotated dialogue datasets targeting pragmatic appropriateness. to facilitate the automated evaluation of pragmatic appropriateness and validate the efficacy of daa, we curated an evaluation dataset. the dataset comprises conversation logs from six evaluation objects, including five popular chatbot systems and one human for comparison. following the data collection process, seven annotators independently assigned ratings to evaluate the appropriateness of each turn of responses from the objects. experimental results demonstrate a strong correlation between the appropriateness scores generated by daa and the annotators. this indicates that daa effectively captures the notion of appropriateness in dialogues and aligns well with human perceptions. our contributions are summarized as follows: ## method we posit that human-human dialogues exhibit a higher level of appropriateness in the use of dialogue acts. therefore, by measuring the similarity between the dialogue act patterns observed in human-chatbot dialogues and those in humanhuman dialogues, we can assess the appropriateness of chatbot responses. the process of daa, as depicted in figure 1 , involves three key steps. firstly, we classify the utterance sequences in a multi-turn dialogue into sequences of dialogue acts, enabling us to learn and compare the transition patterns of dialogue acts. next, we model the transition patterns of dialogue acts among interlocutors using a human-human dialogue corpus. this allows us to capture the inherent patterns of dialogue act transitions that occur in human conversations, serving as a reference for assessing appropriateness. finally, we compare the dialogue act transition patterns observed in a given human-chatbot dialogue with the learned patterns from the previous step, providing an appropriateness score based on the degree of similarity. ## evaluation data construction while annotated data exists for overall appropriateness and semantic appropriateness, to the best of our knowledge, we did not come across any annotated data specifically designed for evaluating pragmatic appropriateness in conversations. in order to assess the effectiveness of daa, we gathered conversation logs from six objects with different levels of intelligence. each response of these conversations is manually annotated with a score indicating its appropriateness. ## results and analysis in this section, we present the correlation of daa with human ratings and conduct a comparative analysis with other appropriateness metrics, then address the following research questions (rqs): (1) does daa provide a stable and objective evaluation of dialogue appropriateness? (2) is daa sensitive to the accuracy of the dialogue act classifier? (3) how does the domain of the training data impact the evaluation of dialogues? (4) how can we interpret the appropriateness scores provided by daa? (5) how does the direction of contextresponse pairs affect the performance of daa? ## related work early dialogue evaluation methods employed statistical metrics such as bleu, meteor, and rouge @xcite @xcite . these metrics measure word overlap between system-generated and reference responses. embedding-based methods like bertscore @xcite utilize contextual embeddings to compute the similarity between system responses and references. adem @xcite collects human-scored data and employs a hi- some researchers have proposed unsupervised metrics to reduce reliance on manual annotations, such as grade @xcite , dynaeval @xcite , and quantidce @xcite . these methods adopt a responsedifferentiating paradigm, using extensive humanto-human dialogues as positive examples and applying heuristic perturbations to obtain negative examples. the models are trained to distinguish between paired positive and negative examples. however, these heuristic perturbations are often simplistic, such as random utterance order shuffling, limiting the model's ability to learn accurate decision boundaries. on the other hand, pone @xcite performs negative example sampling based on semantic similarity with positive examples. @xcite use a masked language model to replace some words in the response, considering the dialogue history, with words unre-lated to the dialogue history to generate negative examples. deam (ghazarian et al., 2022) analyzes coherent error categories in real dialogues and designs targeted automatic rules for constructing negative examples. fined-eval @xcite supplements training data with additional model scores, such as question-answer relevance scoring and natural language inference contradiction scores, to obtain positive and negative samples. compared to these methods, daa directly models human dialogues without the need for negative sample generation. some methods directly obtain scores based on the token probability distribution provided by language models. holisticeval @xcite to calculate the conditional probability of a response given the dialogue history. mehri and eskenazi (2020a) utilize dialogpt @xcite to pre-given follow-up utterances and take the likelihood as scores. full @xcite improves upon @xcite by validating a large number of follow-up utterances and selecting the follow-up utterance that has the highest correlation with human evaluation. in comparison to these methods, we do not model token-level inputs. instead, we model dialogue act sequences. in the field of evaluation, there is variability in the definition of appropriateness. @xcite focus on the overall impression of the response. @xcite define appropriateness as whether the response is appropriate in terms of grammar, topic, and logic. @xcite consider appropriateness to be synonymous with the overall quality. mehri and eskenazi (2020a) emphasize semantic appropriateness. @xcite associate appropriateness with dialogue act. in this work, we adopt the definition of appropriateness from @xcite and leverage the dialogue act to measure the appropriateness. ## conclusion in this paper, we propose an interpretable and effective method to assess the appropriateness of dialogue agent responses by modeling dialogue act transitions. to facilitate the automated evaluation of pragmatic appropriateness, we annotated a manually scored dataset consisting of 2,716 turn-level appropriateness ratings for six different agents, with evaluations provided by seven annotators. our extensive experiments, using eight pre-trained language models and four human conversation corpora, demonstrate a strong correlation between our evaluation method and human ratings. these results confirm the reliability and effectiveness of daa in assessing the appropriateness of dialogue agent responses.