A lexa, G oogle, S iri: What are Your Pronouns? Gender and Anthropomorphism in the Design and Perception of Conversational Assistants (2021)

Abstract: Technology companies have produced varied responses to concerns about the effects of the design of their conversational AI systems. Some have claimed that their voice assistants are in fact not gendered or human-like—despite design features suggesting the contrary. We compare these claims to user perceptions by analysing the pronouns they use when referring to AI assistants. We also examine systems’ responses and the extent to which they generate output which is gendered and anthropomorphic. We find that, while some companies appear to be addressing the ethical concerns raised, in some cases, their claims do not seem to hold true. In particular, our results show that system outputs are ambiguous as to the humanness of the systems, and that users tend to personify and gender them as a result.

## introduction following analysis and criticism of the effects of the genderised and anthropomorphic design of conversational agents (cercas @xcite , the producers of some commercial conversational assistant systems have been at pains to claim that their products do not perpetuate negative stereotypes by presenting as gendered, human-like entities. for example, amazon states that their virtual assistant, alexa: 'is not: fully human, fully robotic, artificial ... alexa isn't a person, but she has a persona -amazon personifies alexa as an artificial intelligence (ai) and not as a person with a physical body or a gender identity.' 1 in their editorial guidelines, apple also instructs developers not to use gendered personal pronouns 1 amazon alexa branding guidelines webpage. ## bias statement in this work we address the problem of biased design choices and their potential impact on society. following @xcite , we argue that designing conversational assistants with young, subservient female personas can perpetuate negative gender stereotypes, and lead to abusive, misogynistic behaviour in the real world. as @xcite point out, this becomes especially problematic as these systems appear more human-like. for example, it has been claimed that google's duplex voice assistant is so human-like, that people do not realise they are speaking to a machine and being recorded, which can be a violation of the law in some territories @xcite . nevertheless, people tend to personify nonhuman entities, including technological devices and virtual agents @xcite @xcite . while some argue that this problem can be solved simply by using a 'genderless' voice @xcite , research shows that people will anyway assign binary genders to ambiguous voices (sutton, 2020). 4 thus, a genderless voice is redundant if other elements of an assistant's design cause it to be gendered. in the following, we further examine e which traits beyond voice might contribute to this gendering and to anthropomorphism in general. ## related work personification and anthropomorphism. while definitions vary, we consider personification to be the projection of human qualities onto nonhuman objects (by users) and anthropomorphism to be human-like behaviours or attributes exhibited by those objects (as designed by their creators). several studies have looked at how users directly report perceptions and behaviours towards voice assistants. for example, @xcite conducted a study of the perceptions of 20 users, comparing alexa, google assistant, and siri, classifying perceptions of the agents' characters on five dimensions of anthropomorphic design and personification by users. they found various differences in the perceived human qualities of the various agents, such as intelligence and approachability. however, their study presupposed personification of the agents, with non-human characteristics not considered. in a diary study, @xcite found that seven out of nineteen participants reported using personifying behaviour towards alexa, such as use of politeness. and cercas @xcite found that just over a third of the wide range of virtual assistants and chatbots they examined to have anthropomorphic characteristics. they also found the preferences of members of the public for their idealised voice assistants to be quite mixed, with around half of participants preferring a 'human' identity rather than 'robot', 'animal, or 'other'. similarly to our analysis of 'humanness'(section 4.2), etzrodt and engesser (2021) asked users to classify alexa and google assistant as being a 'thing' or a 'person'. while they used this framework to examine user perceptions in an online survey, we use expert annotators to directly annotate system outputs with coll ardanuy et al. (2020)'s humanness and not humanness labels. as well as collecting direct reports of users, there have been some studies that use text analysis to infer users' implicit attitudes. for example, @xcite manually coded a small number of customer reviews of alexa, finding a roughly even split between use of personal and object pronouns, indicating differences in levels of users' personification. the closest work to our analysis of customer reviews (section 4.1), is that of @xcite , who conducted a large scale analysis of alexa reviews, focusing on user personification. they found that many users develop relationships with the agents that can be characterised as familial or even romantic. however, they did not consider perceptions of gender, or compare with other assistants. gender. there have been relatively fewer studies considering user perception of the agents' genders. cercas @xcite found that a majority of survey participants claim to prefer a hypothetical non-gendered voice (robot or gender-neutral) to recognisably male or female ones. @xcite conducted an analysis of text-based chatbots (rather than voice assistants) according to the developers' design choices of names, avatars, and descriptions, finding them to be overwhelmingly gendered, with more than 75% female-presenting. as in our analysis in section 4.1, they explored use of pronouns to determine the bots' genders, although they did not investigate user perceptions. concerning conversational systems' output, @xcite examined whether chatbots appear to agree with negative gender (and racial) stereotypes in their input. similarly, @xcite found that neural chatbots will generate a biased response dependent on which sentence-based persona description was used to initialise the model (following @xcite ). however, both of these works concentrate on harmful bias in the content generated in response to specific prompts, whereas we consider stylistic gender cues in the chatbots' output overall. summary. the majority of work in this area surveys relatively small samples of users, with much of it concentrating on amazon's alexa (only two of the reviewed publications cover all three systems). in this study, we create and release two corpora comparing amazon alexa, google assistant, and apple siri: (1) a large corpus of user reviews to compare user perceptions of both personification and genderisation of the assistants, and (2) a corpus of system responses to questions from the per-sonachat dataset @xcite . 5 we analyse the systems' outputs to investigate the linguistic markers of gender and persona that they display. ## analysis we examine three of of the most popular and widely available voice-activated assistants: amazon's alexa, google assistant, and apple's siri. each has various default design features, including its name and default voice settings (see ## discussion and conclusion our analysis suggests that people tend to personify and gender the systems, irrespective of the efforts and claims of their designers. this seems to be, at least partly, a result of aspects of their design. we first assessed user perceptions by analysing online comments for use of pronouns and affective language. results in section 4.1 suggest that 16 while these types of analyses have been criticised for breaching privacy and consent @xcite , we do not use them to assign demographic features or social categories to humans, but analyse design choices in system outputs. the name and branding of a system may be highly salient in this respect, with even systems that have male-sounding voice options mostly referred to as 'she' (although we do not know how many users select the male options). google assistant, which has a female voice by default and the most human-like responses, is nevertheless referred to most often using object pronouns, likely as a result of its nongendered name. we then analysed stylistic features in their responses to persona-related questions (section 4.2). we find only weak evidence of gendered language, but large differences in the levels of humanness they seem to express. along with the nature of their voices, this may explain why people personify and subsequently gender conversational assistantseven when they have apparently more neutral design features. while male voice options are available for two of the systems, we can't find any evidence of how many users actually select them. apple's announcement that future users of their systems will have to actively select a voice for siri may lead to more balance in this regard. however, it remains to seen what the users-who are by now accustomed to the idea that these entities are designed as femalewill choose (for their still, after all, female-named assistant). as people are likely to assign gender to objectively non-gendered voices (sutton, 2020), and voice assistants that are designed as or perceived to be female attract abusive behaviour (cercas @xcite , designers may consider attempting to reddress the gender imbalance by designing assistants with servile roles to be male-presenting by default. while there have been examples, such as the bbc's beeb @xcite , this remains an under-explored approach. in terms of the assistants' responses to users, we see a clear difference in approaches. while google assistant, and to a lesser extent, alexa, seem to blur the line between human and machine personas, siri comes across as more practical and task-focused, evading the majority of personalitybased questions. although possibly less engaging, this approach may be a way of avoiding some of the ethical issues discussed in section 2. there is perhaps a tension between companies' commercial aims of seeing high levels of engagement in their products and the ethical considerations discussed here. however, if companies are going to design agents with human-like and gendered char-acteristics and personas, they should not claim the opposite.