Influence of user personality on dialogue task performance: A case study using a rule-based dialogue system (2021)

Abstract: Endowing a task-oriented dialogue system with adaptiveness to user personality can greatly help improve the performance of a dialogue task. However, such a dialogue system can be practically challenging to implement, because it is unclear how user personality influences dialogue task performance. To explore the relationship between user personality and dialogue task performance, we enrolled participants via crowdsourcing to first answer specified personality questionnaires and then chat with a dialogue system to accomplish assigned tasks. A rule-based dialogue system on the prevalent Multi-Domain Wizard-of-Oz (MultiWOZ) task was used. A total of 211 participants’ personalities and their 633 dialogues were collected and analyzed. The results revealed that sociable and extroverted people tended to fail the task, whereas neurotic people were more likely to succeed. We extracted features related to user dialogue behaviors and performed further analysis to determine which kind of behavior influences task performance. As a result, we identified that average utterance length and slots per utterance are the key features of dialogue behavior that are highly correlated with both task performance and user personality.

## introduction personality is the characteristics of behavior, cognition, and emotional patterns that evolve from biological and environmental factors @xcite . as a fundamental difference between individuals, personality has been found to influence the performance of task-oriented dialogue systems, such as the user satisfaction to dialogue systems @xcite . endowing a task-oriented dialogue system with adaptiveness to user personality is therefore essential to achieve better dialogue task performance. in a previous study, mairesse and walker analyzed the differences between introverts and extro-verts in dialogue behavior, such as extroverts being more talkative and using fewer words per utterance than introverts @xcite . they further designed a parameterized natural language generation system named "personage." increasing applications of personality have been put forward on a dialogue system that is able to generate natural, coherent, and personalized utterances @xcite . however, two problems remain that hinder the dialogue system adapting to user personality to achieve better task performance: in this work, we performed a case study analysis of user personality and task performance using a rule-based dialogue system on the prevalent multi-domain wizard-of-oz (multiwoz) task @xcite . specifically, we enrolled participants via crowdsourcing to first take several personality questionnaires and then interact with the dialogue system via text chat to accomplish assigned tasks. participants' personality and dialogue data were collected. we then analyzed the correlation between personality and task performance to identify the personalities highly correlated with task performance. considering that the task performance largely depends on the user's dialogue behavior, we further examined the main features of dialogue behavior influenced by the identified personalities. for data collection, we selected four mainstream personality questionnaires that could possibly influence the dialogue task performance and use a rule-based dialogue system for the multiwoz task @xcite . in total, 211 participants were enrolled, and their personalities and 633 dialogues were collected and analyzed. five task performance metrics are used. our contributions are two-fold: the remainder of this paper is organized as follows. the related work on personality and dialogue systems is presented in the next section. section 3 describes the research approach and analysis procedures. the data collection and data analysis are presented in sections 4 and 5. conclusions and future work are addressed in the last section. ## related work related studies can be classified into: (1) the exploration of correlations between personality and dialogue system performance, (2) the application of personality to a dialogue system, and (3) useradaptive dialogue system. in the past decades, @xcite explored the differences between extroverts and introverts in speaking, such as pitch and volume. @xcite further identified that extroverts feel extroverted voices are generally more credible, whereas introverts feel the opposite. conscientiousness (the tendency to act in an organized or thoughtful way) and agreeableness (the tendency to be compassionate and cooperative toward others) have been found to be important factors influencing the dialogue in customer service @xcite . @xcite examined the consistency of the correlation between personality and word usage in different languages and found that such correlation is independent of language. @xcite recently found that people with different personalities use different persuasion strategies, which can potentially be used by a personalized persuasion dialogue system. although numerous studies have found that personality is more or less related to dialogue behaviors, few studies have comprehensively investigated the influences of user personality on dialogue task performance. as for applying personality to a dialogue system, @xcite applied extraversion as the parameter of a natural language generation system, so as to generate personalized and natural dialogue. specifically, they used an extraversion parameter to control the communication content and the template sentence with different syntactic styles. @xcite applied a profile as personality into an encoder-decoder dialogue system to increase the coherence of generated utterances. @xcite proposed a dataset named "persona-chat" containing a corpus with a corresponding persona description to help build an end-to-end chit-chat dialogue system with consistent personality and coherent utterance generation. towards a more user-adaptive dialogue system, several researchers have investigated user-adaptive dialogue systems. @xcite focused on a system that generates the responses to users' information queries in accordance with their skill with regard to the system, domain-related knowledge, and the degree of hastiness. @xcite developed a dialogue system that adapts to users' behavior regarding pause duration preceding system utterance and gaze to reduce user's discomfort during conversation. to increase user engagement, @xcite proposed a system that is able to select a chat topic in accordance with user interests. the aim of our research is to find out the influences of personality on task performance to benefit further development of a dialogue system that can adapt to user personality to achieve better dialogue task performance. ## approach to explore the correlation between user personality and dialogue task performance, a dataset is needed that contains a variety of user personality traits (e.g., extraversion), dialogue data, and the corresponding dialogue task performances (e.g., task success rate). the data to be analyzed is collected via crowdsourcing, and various features of dialogue behavior related to task feature, utterance feature, and action/utterance intent are extracted from the dialogue data. after collecting a sufficient amount of data, user personality and task performance are explored in three steps: (1) correlation analysis of user personality and dialogue task performance to identify the personality traits highly correlated with task performance; (2) correlation analysis of behavior feature and dialogue task performance to determine their relevance; and (3) analysis of highly correlated personalities, behaviors, and task performances to figure out how a person's personality influences task performance through dialogue behavior. ## data collection we designed a data collection task to obtain user personality and dialogue data on the amazon mechanical turk (amt), a human-intelligence task (hit) crowdsourcing platform. the task has been approved by the ethical review of the relevant organization. ## data analysis the pearson correlation coefficient r (range from -1 to 1) is used for analysis. we first extracted a variety of dialogue behaviors and task performances to be analyzed. we then analyzed the correlation between user personalities and task performances. we further clarified the influences of user personalities on task performances through dialogue behaviors that are highly correlated with personalities and task performance. ## conclusion and future work we presented a case study analysis using a rulebased system on the multi-domain wizard-of-oz (multiwoz) task to explore the correlation between user personality and task performance. we identified the user personalities highly correlated with task performances and found that sociable and extroverted people tended to fail the task, while neurotic people were more likely to succeed. we further selected user personalities, dialogue behaviors, and task performance having high correlations with each other to understand the correlation of personality and task performance in view of dialogue behavior. as a result, we clarified that average utterance length and slots per utterance are the two key dialogue behaviors influenced by the personality, affecting the task performance in the case of our data collection using multiwoz. our work is an initial step to enable a dialogue system to adapt to user personality. since the analyzed correlation results are based on a rule-based system, more dialogue systems, e.g., end-to-end dialogue systems such as simpletod @xcite and ubar @xcite , and different task domains will need to be considered for the analysis to derive more general correlations. the effect of personality and task performance should also be qualitatively explored to better inter-pret the correlations we observed in the experiment. furthermore, the dialogue system adapting to user personality should be implemented to ascertain its dialogue task performance.