Analyzing Differences in Subjective Annotations by Participants and Third-party Annotators in Multimodal Dialogue Corpus (2023)

Abstract: Estimating the subjective impressions of human users during a dialogue is necessary when constructing a dialogue system that can respond adaptively to their emotional states. However, such subjective impressions (e.g., how much the user enjoys the dialogue) are inherently ambiguous, and the annotation results provided by multiple annotators do not always agree because they depend on the subjectivity of the annotators. In this paper, we analyzed the annotation results using 13,226 exchanges from 155 participants in a multimodal dialogue corpus called Hazumi that we had constructed, where each exchange was annotated by five third-party annotators. We investigated the agreement between the subjective annotations given by the third-party annotators and the participants themselves, on both per-exchange annotations (i.e., participant’s sentiments) and per-dialogue (-participant) annotations (i.e., questionnaires on rapport and personality traits). We also investigated the conditions under which the annotation results are reliable. Our findings demonstrate that the dispersion of third-party sentiment annotations correlates with agreeableness of the participants, one of the Big Five personality traits.

## introduction to achieve adaptive human-machine (or humanrobot) dialogue, it is necessary to estimate the human user's subjective impressions and emotions during the dialogue. the user's satisfaction with the dialogue can be increased by appropriately changing the dialogue content in accordance with the user's emotions. estimated subjective impressions and emotions can also be utilized to evaluate the dialogue. the difficulty here is that such impressions and feelings are inherently subjective, and it is impossible to objectively determine unique references for subjective content. references are necessary participant third-party annotators system (wizard-of-oz) annotations after dialogue watch video annotations • sentiment • 18 rapport items • personality traits for training and evaluating machine learning models. even when manual annotations are performed, the results among annotators do not always agree, which is a common problem in annotations of subjective labels. in this paper, we analyze the disagreement among human annotation results, especially the differences between annotations given by participants themselves and by third-party annotators (fig. 1 ). specifically, we conducted investigations on perexchange annotations, i.e., sentiments, and perdialogue (per-participant) annotations, i.e., questionnaires measuring the participant's rapport and personality traits. we used the osaka university multimodal dialogue corpus hazumi, which we had previously constructed @xcite , for the analysis. our findings show that third-party annotators tend to give subjective annotations on the basis of their rather simple impressions compared to the participants themselves, who may not always fully express their inner states during the dialogue. we also clarify why automatic estimation performances of per-exchange sentiments from multimodal features differ between cases in which the reference sentiments were given by the participants themselves and by the third-party annotators, where the latter usually obtains better performances. we then investigate the conditions under which estimated sentiments given by third-party annotators would be reliable by examining the dispersion of the annotation results. the estimation of users' sentiments based on multimodal data with machine learning will never be perfect, so it would be helpful to know whether the estimation results can be reliable for each user on the basis of other information sources. in this paper, after showing how the dispersion of sentiments given by third-party annotators correlates with machine learning performance, we demonstrate that this dispersion is negatively correlated with one of the personality traits, namely, agreeableness. this finding indicates that a personality trait can be a useful clue for determining the reliability of the sentiment estimation results. ## related work we here describe related studies on adaptive dialogue systems, emotion recognition, datasets of multimodal dialogues, reference labels for subjective annotations, and personality traits, in that order. it is essential that dialogue system responses be adaptive to user states. in task-oriented dialogues, task success rates can be improved and the number of turns to task completion can be reduced by adapting system responses in accordance with several user types @xcite . as for non-task-oriented dialogues, personalization based on the user's domain expertise has been attempted @xcite . system responses are preferably based on various modalities such as vision and prosody in addition to textual input. a variety of studies have examined text-based chatbots based on large pre-trained language models (e.g., @xcite ). currently, studies on dialogue systems have been actively expanded from the text-based perspective to a multimodal one, as evidenced by a recent dialogue competition using a humanoid robot @xcite . user impressions (such as emotions) can be an important clue for adaptive dialogue systems. in particular, adapting to the user's emotions is essential for social interaction @xcite . moreover, different types of information, including multimodal information (e.g., vision and prosody), can be utilized to recognize the user's emotions, as can physiological signals @xcite . in this paper, emotion is treated as sentiments per exchange. a famous multimodal dialogue corpus with emotion labels is the iemocap dataset @xcite , which contains dialogues between actors in role-playing scenarios. the emotional dyadic motion capture (iemocap) dataset is a well-known dataset used to recognize emotion during dialogues @xcite . it is a well-controlled dataset in the sense that data were collected by asking actors to speak with designated emotions. therefore, this dataset contains objective reference labels for each emotion, i.e., the designated emotions. in contrast, our hazumi dataset @xcite utilized in this paper consists of natural and spontaneous dialogues. thus, there are no objective reference labels. we opted to use this dataset because our objective is to analyze the differences between several manual annotation results and discuss reference labels for subjective annotations. prior studies in the fields of social signal processing and affective computing have examined how to determine the ground truth of subjectively assigned labels @xcite @xcite . @xcite proposed three strategies for utilizing selfassessment labels and external assessment labels in training data for two dimensions of a group engagement state (called cohesion) and compared their prediction performances. @xcite recently proposed a method to train a classifier that fits better with the annotation results in medical binary classification tasks. in this paper, we do not train a classifier but analyze what happened in a multimodal dialogue data. we also extend analysis from single to several subjective annotations, i.e., per-exchange annotation and per-dialogue annotations. emotion depends on individual users, e.g., their personality traits (such as the big five (goldberg, 1990)). personality traits also play an important role in a variety of user-adapted interactions @xcite @xcite . the personality traits of a robot and human interlocutors are known to be effective for engagement estimation in human-robot interactions @xcite , and correlation between the engagement and the personality traits given per dialogue has been investigated in human-robot and human-human interactions @xcite . in this work, we comprehensively analyzed the relationship between the user's personality traits on the basis of perdialogue questionnaire results and per-exchange sentiments. we utilized the multimodal dialogue corpus hazumi, which we had previously constructed (komatani and okada, 2021). it is a dataset that can be used extensively for research and development purposes foot_0 . the annotation unit at the utterance level is the exchange. an exchange is defined from the beginning of a system utterance to the beginning of the next system utterance. the data contain 18,162 exchanges in total; the mean duration was 13.10 seconds and its standard deviation was 7.80. ## analyses on relationship between annotations given by participants and third-party annotators we analyzed the correlations between the manual annotations given by the participants themselves and by five third-party annotators. sentiments were analyzed using hazumi1902 and hazumi1911 due to their availability (see if any correlation is found between two metrics corresponding to the annotations, it will provide useful insights for the machine learning design. for example, it would be effective to use one of the metrics as input when estimating the other by machine learning. the correlation would also be ## analyses on dispersion of third-party sentiments we here focus on the dispersion of sentiments given by the five third-party annotators (third-party sentiments). we discuss the conditions under which the third-party sentiments would be reliable. ## conclusion in this paper, we investigated the correlation of subjective annotation results between the participants themselves and five third-party annotators. we found that some are correlated, which will potentially be useful in machine learning to estimate one of the annotation targets, such as the participants' sentiments, their evaluation of dialogues (18 rapport items), or their personality traits. we also investigated the dispersion of the sentiments given by the five third-party annotators. we showed that a difference in annotation results correlates with the estimation error of machine learning and found that the dispersion was negatively cor-related with agreeableness, one of the big five personality traits. these results can provide insights into the development of adaptive dialogue systems: specifically, a personality trait can be used as a clue to determine whether or not to rely on the sentiment recognition results. one of our future works is to estimate the user's personality traits before and during the dialogue. the system can then utilize the personality trait to decide how actively to adapt to the user on the basis of the discussion in this paper. personality traits such as neuroticism are not expressed by users during dialogues such as chit-chat and thus are difficult for the system and third-party annotators to observe. the analyses in this paper considered all of the big five traits, but it will be necessary to select personality traits observable in the dialogue accordingly, e.g., extraversion. the results presented in this paper are based on our japanese dataset hazumi. various factors such as the behavior of the participants and annotators, for example, can be involved. further investigation is needed to confirm the generalizability of the obtained results to other languages and cultures, as well as to different experimental settings including dialogue tasks and instructions to the participants.