Conversational Assistants and Gender Stereotypes: Public Perceptions and Desiderata for Voice Personas (2020)

Abstract: Conversational voice assistants are rapidly developing from purely transactional systems to social companions with “personality”. UNESCO recently stated that the female and submissive personality of current digital assistants gives rise for concern as it reinforces gender stereotypes. In this work, we present results from a participatory design workshop, where we invite people to submit their preferences for a what their ideal persona might look like, both in drawings as well as in a multiple choice questionnaire. We find no clear consensus which suggests that one possible solution is to let people configure/personalise their assistants. We then outline a multi-disciplinary project of how we plan to address the complex question of gender and stereotyping in digital assistants.

## introduction and bias statement biased technology disadvantages certain groups of society, e.g. based on their race or gender. recently, biased machine learning has received increased attention. for example, in the area of natural language processing (nlp), it has been shown that word embeddings @xcite , co-reference resolution @xcite and machine translation systems @xcite are likely to reflect and even amplify social biases in the data. here we address a different type of bias which is not learnt from data, but encoded during the design process. we illustrate this problem on the example of conversational voice assistants (cvas), such as amazon's alexa, apple's siri, microsoft's cortana, or google's assistant, which are predominantly modelled as young, submissive women. according to unesco @xcite , this bears the risk of reinforcing gender stereotypes. in particular, these design choices can create representational harm by reinforcing negative stereotypes society holds about women. the report argues that this becomes even more prevalent in the face of abuse, where most assistants do not answer 'appropriately @xcite , which might impact human-human interactions. in order to tackle the problem of reproducing structural inequality and oppression of marginalised groups when designing new systems, the design justice network 1 proposes to centre the voices of those who are directly impacted by the outcomes of the design process @xcite . similarly, the european commission's ethics guidelines for trustworthy ai (ai hleg, 2019) recommend stakeholder participation during the development of new technologies, as well as paying special attention to the system's societal impact. in this work we aim to unpack and verify some of the statements in the unesco report in a multidisciplinary project including methodologies from human computer interaction (hci), social psychology, and natural language generation. as a first step, we conduct a participatory design workshop to gather public views on this subject to further inform the direction of this research. ## problem definition the persona of a cva can be viewed as a composite of elements of identity (such as demographics and background facts), language behaviour, and interaction style. some of these aspects can be learned from data, including personality-based linguistic style generation (e.g. oraby et al. (2018) ), or generating responses which are factually consistent with a persona profile (e.g. @xcite ). however, demographics and background facts are usually deliberate design choices, and research such as @xcite shows that the gender choices of cvas are conforming to traditional gender roles and social expectations: that is, the majority of cvas have female personas. in interviews with news outlets, companies and developers defend design choices by citing market research showing that female voices are perceived to be more cooperative and helpful, and male voices are considered more trustworthy @xcite . as such, the role of a personal assistant is often assigned to women, whereas in applications where the cva needs to be authoritative, companies tend to choose male voices. however, in 2019 unesco published an in-depth analysis of the gendering of ai, especially focused on conversational assistants @xcite . the report details how the personas of current conversational assistants reinforce and spread existing biases about women as being subservient, modelling acceptance and tolerating sexual harassment and verbal abuse, and 'make women the "face" of glitches and errors'. the report attributes this bias to lack of diversity in the tech sector and concludes that the personas of conversational assistants should not be female by default, and that digital assistants should rather be designed to combat gender-based biases as well as discouraging insults and abusive language. while groups like feminist internet foot_0 and women reclaiming ai foot_1 have addressed the feminisation of cvas through projects such as f'xa foot_2 , we involve the general public in designing alternative personas for conversational assistants as a first step in exploring the needs of diverse users. ## experimental setup methodology: we explore an alternative methodology to the usual way we get 'users' involved in nlp research, i.e. crowd-sourcing or scraping online data (de vries et al., 2020) borrowed from hci. participatory design actively involves all stakeholders in the design process to ensure that the end-product meets their needs @xcite . it aims to assign an active and informed role to everyone affected by the end result. to implement this idea, we organised a public workshop with the help and endorsement of the royal society of edinburgh foot_3 which allowed us to reach a wide population, including potential end-users and people affected by stereotyping. inspired by a previous workshop on voice assistants and feminism @xcite , our workshop aimed to inform and stimulate critical reflection in order to seek an active discourse with the public. as such, we organised two events over two consecutive days: the first event was a short introduction where people where invited to learn more about the underlying technology. on the second day, we introduced the issue of gendered technology, and got two experts presenting their views, including one of the authors of the unesco report and a ux voice designer from the bbc. the overall question the workshop explored was: what would your ideal conversational voice assistant be? participants: we had a total of 128 participants registered to the online event, with 72 participants attending. according to a pre-questionnaire, the majority (72%) of participants identified as female, 28% male. most of them have had experience with using one or more voice assistants although they do not use them regularly. questionnaire: during and after the workshop, we asked participants to complete a form detailing the following characteristics of their ideal personal assistant: anthropomorphism, gender, age group, regional accent, as well as an option to describe other personality traits. in the following study we compare these submitted desiderata with characteristics of 14 existing chatbots: we annotate these chatbots according to the same characteristics as the participants' ideal system. due to the interaction medium of each bot (spoken vs. typed), some characteristics are unavailable such as regional accents. in addition, some systems such as the bbc's beeb have limited availability, in which case we base our annotations on 3rd party reports. we use the following methodology to elicit the information: drawings: prior to the workshop we asked participants to submit two designs: (1) a drawing and a description of how they image alexa's virtual character to be like, and (2) a drawing and description of how their ideal personal assistant's character would be like. this is similar to @xcite , but instead of using an avatar building tool, which restricts participants to a pre-defined set of anthropomorphic choices, we let users submit photographs, e.g. of their own drawings or other objects. ## results questionnaire: 34 participants filled out the questionnaire on desired characteristics, see fig. 1 . gender: the majority of participants prefer a robotic voice (32.4%), followed by female (20.6%) and male (11.8%), see fig. 1a . about one quarter (26.5%) of participants commented that they wanted a gender neutral voice. 7 in contrast, our analysis of current chatbots shows that 71% have female voices by default. age group: most participants (44%) want their chatbot to be either in the age bracket between 25-40 years old, or they have no preference (38%), see fig. 1b . some participants (15%) want the persona to be older than 40. hardly anybody wanted their chatbot to reflect an age group of 24 or younger, which is in stark contrast to existing voice assistants, which are predominentely perceived to be in their 20s. anthropomorphism: most participants would like their assistants to be identifiable as human, followed by animal and robot, see fig. 1c . our annotations revealed that 36% of current systems resemble humans. although none of the commercial systems have a visual avatar, they are all anthropomorphic in other ways such as having a pet, having experienced feelings, or experiencing mental illness (in the case of parry). although the ethics of anthropomorphic ai have been widely discussed, e.g. @xcite , it is beyond the scope of this paper. lect: more than half of participants did not want their chatbot to have a regional or other accent/ language variety, see fig. 1d . amongst the currently available systems, only the bbc chatbot beep has a regional accent (northern england). other personality traits: in addition, we asked participants to submit any other personal characteristics they would want in their ideal assistant. overall, responses varied greatly: friendliness, helpfulness and humour were the most common traits and they are echoed in the design of existing assistants @xcite , but other users called for more transparency, and less anthropomorphism. this directly contradicts some popular approaches to persona design which are centred around the idea of having a digital 'person', e.g. asking "how would we want a person to respond?" @xcite . drawings: here, we select and discuss two submissions for each category: artistic impressions and descriptions of (1) current and (2) future personas. the examples were selected as they represent very different concepts, see appendix a. for impressions of current systems, we got a drawing of a black box, described as 'humourless entity' used for surveillance; and a glamorous woman wearing make-up and an evening dress, described as 'people-pleasing and inoffensive'. for future systems, we got a man holding up a "no bullshit" sign, described as "strong, rational, organised"; and a intriguing piece of ornamental jewellery in form of an octopus, described as "funny, with the ability to invoke laughter but also to empathise and advise. an entity that i could trust completely". ## conclusion and future work in this paper, we investigate alternative designs of the persona of conversational voice agents, which are currently predominantly set to reflect young women. unesco argues that this representation, together with their depiction as subservient assistants, bears the risk of reinforcing gender stereotypes. we involve the public in designing alternatives, both by submitting their preferences and by asking them to sketch their visual conceptions. one critical difference in our methodology, is that we gather the data as apart of a participatory design workshop in order to stimulate critical discussion and active discourse on this matter. the outcomes show a wide range of preferences and possible future designs: participants (@xmath0, gender-neutral or robotic voices, mostly without a regional accent. most people thought that the persona's behaviour and identity should resemble a human in an age bracket between 25-40. descriptions of personality traits ranged from friendly, helpful and humorous to calls for less anthropomorphism. the lack of clear consensus suggests that personalised or configurable digital personas are required to fulfil individual preferences. most commercially available voice assistants allow for a limited number of choices: for example, google assistant lets you choose different voice 'colours', which also include male voices for english. amazon alexa's voice can be changed to various english accents, ranging from southern us to uk english. amazon has also recently added celebrity voices to purchase, such as actor samuel l. jackson. in future work, we will be looking to extend these functionalities to not only reflect personality by different synthesised voices/text-to-speech, but also personality expressed in language behaviour, conversational content and interaction style, building upon previous work on personality-based linguistic style generation (e.g. oraby et al. ( 2018 )), or generating responses which are factually consistent with a persona profile (e.g. @xcite ). in addition, we will be closely working with social psychologists to anticipate the social impacts these artificial personas might have. in particular, we will be studying how digital gendering and personalities of digital assistants influences human online and offline behaviour. and eventually, we aim to build a data-driven mapping between conversational behaviour (e.g. voice, linguistic style and content) and perceived personality traits, such as gender, age, trustworthiness, etc. finally, we hope to repeat similar studies as the one presented in this paper with other subpopulations. for example, we hope to explore perceptions school children might hold about voice assistants, following @xcite . a current and future impressions of conversational personas (a) "strong, rational, organised, and not about to put up with any nonsense!" (b) "funny, with the ability to invoke laughter but also to empathise and advise. an entity that i could trust completely." figure 3: submissions for the "ideal" conversational assistant.