Joint Goal Segmentation and Goal Success Prediction on Multi-Domain Conversations (2022)

Abstract: To evaluate the performance of a multi-domain goal-oriented Dialogue System (DS), it is important to understand what the users’ goals are for the conversations and whether those goals are successfully achieved. The success rate of goals directly correlates with user satisfaction and perceived usefulness of the DS. In this paper, we propose a novel automatic dialogue evaluation framework that jointly performs two tasks: goal segmentation and goal success prediction. We extend the RoBERTa-IQ model (Gupta et al., 2021) by adding multi-task learning heads for goal segmentation and success prediction. Using an annotated dataset from a commercial DS, we demonstrate that our proposed model reaches an accuracy that is on-par with single-pass human annotation comparing to a three-pass gold annotation benchmark.

## introduction today, commercial conversational ai assistants (e.g., amazon alexa, apple siri, and google assistant) are increasingly popular. however, it is challenging to establish reliable metrics to continuously measure the system performance in business reports and a/b experiments. the commonly used metrics in industry, e.g., monthly active users, dialogue count per user, and downstream impacts such as increased subscriptions and product sales etc., usually move slowly and are not suitable to measure the impact of functionality changes. on the other hand, the dissatisfaction metrics based on manual annotation, which can capture dissatisfying user experiences due to system errors, incomplete service coverage, or poor response quality etc., are sensitive to functionality changes, but not suitable for online monitoring and experimentation due to their offline nature. automatic dialogue evaluation metrics @xcite aim to combine the benefits of the two types of metrics mentioned above by providing an online metric figure 1 : a dialogue that contains two goals: playmusic and turn-on-device. here, a turn is defined as one back-and-forth between the user and agent. a dialogue is a set of turns in quick succession as indicated by the timestamps. a goal represents an objective the user is trying to accomplish in a sequence of turns. human annotation labels for turns (defect/non-defect) and goals (success/failure) are provided for reference. automated dialogue evaluation metrics in the literature could be generally grouped into two categories @xcite : 1) interaction quality (iq) metrics (e.g., @xcite @xcite ) that measures user experiences at turn level; and 2) dialogue quality (dq) metrics (e.g., @xcite ) that measures user experiences for the whole dialogue. however, we argue that, for multi-domain goal-oriented dialogue systems, there is a need to add a goal-level metric with granularity in between turn and dialogue. if we define a dialogue as a set of turns in quick succession, a user may switch her objective mid-dialogue without clear delineations. as shown by the example in figure 1 , the user started the dialogue with a playing music request but switched her objective halfway to turn on a smarthome device. here, we propose a new granularity -goal, which represents an objective the user is trying to accomplish in a sequence of turns. goal segmentation is especially important for accurately evaluating the dialogue quality. in figure 1 example, the user successfully achieved her first goal but failed on the second one. it would be ambiguous to evaluate the user experience at the whole dialogue level. to our best knowledge, there is no other work in the literature that performs goal segmentation and goal success prediction jointly. prior work either assumes that a dialogue automatically terminates upon the completion of a goal e.g., @xcite or the goal boundaries are known beforehand e.g., @xcite . to build a reliable automatic goal evaluation metric, we extend the roberta-iq model @xcite by adding multi-task learning heads for goal segmentation and success prediction. even though roberta-iq model is a model built for turn level metrics, it encodes the turns before and after the reference turn (the turn on which the model will make prediction) to ingest dialogue context information. besides, time difference between turns are encoded in the time bin embeddings and feed into the model together with the context embedding. thus, such characteristics makes roberta-iq model a suitable starting point for goal level evaluation task. for goal segmentation, different from lstm based method @xcite , we leverage self-attention mechanism across turns. the practical benefit of this approach is that it improves inference efficiency on long token sequences, which is a common problem for rnn models. to demonstrate the effectiveness of our proposed methods, we evaluated our method against a golden dataset in which each dialogue is labeled by three annotators and the ground truth labels are determined by majority vote. we then compare the accuracy of the model predictions with that of a single-pass human annotation. the proposed work has two main contributions: 1) a novel goal level dialogue evaluation framework that matches the real-world scenarios in multidomain goal-oriented dialogue system; 2) a deep learning model that jointly learns goal segmentation and goal success prediction, with accuracy on-par with single-pass human annotation. the remainder of this paper is organized as the following: section 2 reviews related work about goal segmentation and dialogue evaluation. section 3 introduces the model architecture. section 4 shows the experimental setup and discusses the performance of the model. we conclude the paper in section 5. ## methodology we extend the roberta-iq @xcite model architecture by adding the goal segmentation task and goal success prediction task. as a brief recap, there are three key differences separating roberta-iq from the vanilla roberta model, as illustrated in figure 2: 1. roberta-iq introduces two special tokens the time bin embedding has two uses: 1) it allows the model to understand the temporal relationship between the reference turn and contextual turns; 2) it enables the model to locate the reference turn -a special bin: bin 0 is reserved for reference turn's tokens. we add the goal segmentation and goal success prediction task heads on top of the [cls] token embedding. for each turn in a dialogue, we mark it as the reference turn and let the model to learn from a binary classification label (b if the reference turn is at the beginning of a goal and i if inside a goal). goal success prediction is modeled as a multi-class classifier with three possible outputs: success, failure, and unactionable. 1 in flattened token sequences form, considering turn-2 as the reference turn (bolded). ## conclusion in this paper, we propose a novel framework to evaluate goal-level performance for multi-domain goal-oriented dialogue systems and a deep learning model that jointly learns goal segmentation and success prediction. our experiments show that the proposed model reaches an accuracy that is on-par with single-pass human annotation and with multitask learning, the model performance is better than single task models for both tasks.