“It seemed like an annoying woman”: On the Perception and Ethical Considerations of Affective Language in Text-Based Conversational Agents (2021)

Abstract: Previous research has found that task-oriented conversational agents are perceived more positively by users when they provide information in an empathetic manner compared to a plain, emotionless information exchange. However, users’ perception and ethical considerations related to a dialog systems’ response language style have received comparatively little attention in the field of human-computer interaction. To bridge this gap, we explored these ethical implications through a scenario-based user study. 127 participants interacted with one of three variants of an affective, task-oriented conversational agent, each variant providing responses in a different language style. After the interaction, participants filled out a survey about their feelings during the experiment and their perception of various aspects of the chatbot. Based on statistical and qualitative analysis of the responses, we found language style played an important role in how human-like participants perceived a dialog agent as well as how likable. Language style also had a direct effect on how users perceived the use of personal pronouns ‘I’ and ‘You’ and how they projected gender onto the chatbot. Finally, we identify and discuss ethical implications. In particular we focus on what factors/stereotypes influenced participants’ impressions of gender, and what trade-offs a more human-like chatbot brings.

## introduction conversational agents (cas) have been an area of interest for many years. as technology has advanced, these devices have become more capable and further embedded in users' daily lives. with the advent of personal assistants, like apple's siri and amazon's alexa, focus has begun shifting away from simply conveying correct information to additionally considering the user experience @xcite and how that information is conveyed. rather than seeing a ca as an inanimate tool, users tend to view them as social actors @xcite and infer or assign them personality traits. these perceived personas, whether intentionally designed into the ca or not, can influence how users perceive and interact with these systems. therefore much research has gone into how to design appropriate personas @xcite and investigate how users interpret them @xcite . however, while there is a study exploring ethical issues in text-based affective dialog systems @xcite , comparatively little research has been done in this area. open ethical questions thus remain unanswered around how users perceive and react to different dialog system designs. in this paper, we are particularly interested in how users perceive and emotionally react to a system mimicking human thoughts or emotional awareness, as well as what biases users project onto a dialog system based on cues in its language. one concrete example of this arises around the question whether conversational agents should use the pronoun "i" or convey information in passive voice as using a first person pronoun can imply to the user a false sense of self-perception. therefore, we explore the following research questions and hypotheses: in order to answer these questions, we developed a dialog system to provide information about exam registration to students, an often emotional scenario. for this ca, we designed three different natural language templates, each providing identical information but each using a different language style. two language styles were meant to be affective, reflecting either an empathetic or matterof-fact personality, and the third was meant to be machine-like, using passive voice address. we then performed a user study where 129 participants interacted with and answered a survey about one of the three variants of the ca. the dialogs along with the survey responses from this user study will be publicly available for further research. 1 the main findings of this work are: rq1) users prefer an empathetic language style over a pure information exchange, and generally find the use of the pronoun 'i' natural, suggesting human-like attributes may not be inherently ethically problematic as long as there is transparency about the agent's artificial nature. rq2) a non-gendered, empathetic dialog agent allows users to project a gender (often their own) onto the chatbot, which could help avoid negative stereotypes associated with one gender over the other. ## implementation to create the text-based, task-oriented dialog system for this study, we used the open-source toolkit adviser @xcite . we created a domain 'examination matters' for the system built around the scenario of a 'student advisor chatbot', which can provide information on exam-related topics at the university. this domain provides a good lens through which to investigate users' reactions to a dialog system as this topic is frequently related with negative emotions or stress. to study affective response generation, we implemented pre-defined templates for three different response styles: two sets of affective responses, designed based on myers-briggs type indicator (mbti) personality types; and one set of passive voice templates, meant to be purely informative, inspired by @xcite . depending on the chosen language style for the interaction, the corresponding response template was used to communicate the chatbot's intent. for this study, both affective response templates were designed to reflect the traits "extraversion", "sensing", and "judging", but varied so that one set of templates reflected the trait "feeling" (esfj personality) whereas the other reflected the trait "thinking" (estj personality). these types are both described as decisive, realistic, organized and practical. adjectives describing the esfj personality are: supportive, outgoing, cooperative, warm, sympathetic, friendly, and loyal. in contrast, adjectives describing the estj personality are: assertive, logical, responsible, efficient, straightforward, pragmatic, objective, and structured. these personalities were chosen to help investigate the perception of gender as the adjectives describing "feeling" are more stereotypically associated with women while the adjectives describing "thinking" are more stereotypically associated with men. the esfj language style provides information in a warm, cooperative and empathetic manner. responses contain personal pronouns ('i', 'my', 'you', 'your'), hedging phrases (e.g. "i suggest", "i recommend") and modal verbs like "should". some expressions are vague and open to interpretation. affective esfj responses demonstrate an empathetic understanding of student needs. consequently, this language style can be viewed as affective in the sense that the users' emotional state might be altered. the estj language style provides practical and matter-of-fact information in precise manner. responses contain assertive and also passivevoiced phrases, thereby still addressing the user (e.g. 'you', 'your'). self-reference ('i') is less used, occurring half as many times as in esfj templates. affective estj responses are direct and remind students to follow the rules and requirements. the passive voice language style reflects the artificial nature of a ca. responses are phrased solely to provide information. they do not contain pronouns or emotional phrases, but consist mainly of passive infinitives and modals. example (1) depicts a passive voice, esfj and estj response for the same user utterance. ## future work as this study made use of crowdsourced workers who were roughly within the same age range and from the same cultural background, in the future, it would be interesting to see what role the age and cultural background of the user plays in how they perceive a conversational agent. in particular, it would be interesting to explore if cultural background adds more nuance to the general design recommendations we derived. ## conclusion we presented a user study on the perception of affective language in conversational agents (cas) and discussed ethical concerns in this context. based on participants' survey responses, we showed that users prefer an empathetic, humanlike agent, and that a third of the users viewed the agent as gendered, partially based on linguistic cues. from these findings, we infer that human traits in a ca are not inherently problematic from an ethical standpoint, as long as transparency is always ensured. further, gender biases need to be considered when designing a ca, as the lack of explicit gender cues does not ensure a neutral perception.