Mirages. On Anthropomorphism in Dialogue Systems (2023)

Abstract: Automated dialogue or conversational systems are anthropomorphised by developers and personified by users. While a degree of anthropomorphism is inevitable, conscious and unconscious design choices can guide users to personify them to varying degrees. Encouraging users to relate to automated systems as if they were human can lead to transparency and trust issues, and high risk scenarios caused by over-reliance on their outputs. As a result, natural language processing researchers have investigated the factors that induce personification and develop resources to mitigate such effects. However, these efforts are fragmented, and many aspects of anthropomorphism have yet to be explored. In this paper, we discuss the linguistic factors that contribute to the anthropomorphism of dialogue systems and the harms that can arise thereof, including reinforcing gender stereotypes and conceptions of acceptable language. We recommend that future efforts towards developing dialogue systems take particular care in their design, development, release, and description; and attend to the many linguistic cues that can elicit personification by users.

## introduction automated dialogue or 'conversational ai' systems are increasingly being introduced to the fabric of society, and quickly becoming ubiquitous. as the capabilities of such systems increase, so does the risk that their outputs are mistaken for humanproductions, and that they are anthropomorphised and personified by people (unesco, 2019). assigning human characteristics to dialogue systems can have consequences ranging from the relatively benign, e.g. referring to automated systems by gender @xcite , to the disastrous, e.g. people following the advice or instructions of a figure 1 : an example of the response of a dialogue system to user input that retains anthropomorphic features, and a de-anthropomorphised version, as envisaged by the authors. recognising such dangers, legislation has been passed to prohibit automated voice systems from presenting as humans @xcite and pre-existing legislation on deceptive trade practices may also apply @xcite . research has also called for wider regulation, e.g. requiring explicit (red) flagging of automated systems @xcite or clarification of the machine nature of manufactured items @xcite . while some developers seek to limit anthropomorphic cues in system outputs (e.g. @xcite , user engagement can be a strong motivation for creating humanlike systems @xcite . as a result, despite appearing to be controlled for such cues, the outputs of systems often retain many anthropomorphic linguistic features, as shown in figure 1 . in this position paper, we make a normative argument against gratuitous anthropomorphic features, grounded in findings from psychology, linguistics, and human-computer interaction: we (i) outline the psychological mechanisms and (ii) linguistic factors that contribute to anthropomorphism and personification, e.g. self-referential personal pronoun use, or generating content which gives the appearance of systems having empathy; and (iii) discuss the consequences of anthropomorphism. we conclude with recommendations that can aid in minimising anthropomorphism, thus providing a path for safer dialogue systems and avoiding the creation of mirages of humanity. ## anthropomorphism anthropomorphism refers to attributing human characteristics or behaviour to non-human entities, e.g. animals or objects. humans have a long history of anthropomorphising non-humans. for example, aesop's fables depict animals reasoning, thinking and even talking like humans @xcite . while aesop used personification to highlight the fictional character of animals, when applied to machines, anthropomorphism can increase user engagement @xcite , reciprocity @xcite , along with more pragmatic factors such as hedonic motivation, price value, and habit. for example, self-disclosure from a system, even when 'patently disingenuous', inspires reciprocity from the user @xcite . by encouraging such types of engagements, developers can foster greater connection between people and systems, which increases user satisfaction @xcite , and plays an important role in systems becoming widely accepted and adopted. 2 this is why, automated evaluations often assess the 'human-likeness' of a response @xcite . thus, developers are incentivised to engage with anthropomorphism to stimulate people to create deeper emotional connections with systems that cannot reciprocate. in the rest of this section, we discuss human and system factors that contribute towards placement of systems on the anthropomorphic continuum. ## linguistic factors prior research has attended to anthropomorphic design features of dialogue system, e.g. gendered names and avatars @xcite and chat-gpt's animated 'three dots' and word-by-word staggered outputs, which give an impression that the system is thinking (venkatasubramonian in @xcite . here, we outline the linguistic factors that engender personification that have been given less consideration, e.g. voice qualities and speech, content, or style of outputs. foot_2 ## consequences of anthropomorphism the anthropomorphism of dialogue systems can induce a number of adverse societal effects, e.g. they can generate unreliable information and reinforce social roles, language norms, and stereotypes. trust and deception when people are unaware that they are interacting with automated systems they may behave differently than if they know the true nature of their interlocutor. @xcite show that system responses which excessively use natural-sounding linguistic phenomena can instil unjustified trust into the factual correctness of a system's answer. thus the trust placed in systems grows as they exhibit anthropomorphic behaviour, whether or not the trust is warranted. this may be even more problematic when users are members of vulnerable populations, such as the very young, the elderly, or people with illnesses or disabilities, or simply lack subject matter expertise. although dialogue systems have been 'put forth' as a possible solution to loneliness, socially disconnected individuals can be particularly vulnerable to such trust issues. children have also been shown to overestimate the intelligence of voice assistants such as amazon alexa, and to be unsure of whether they have emotions or feelings @xcite . given unesco's declaration that children have the right to participate in the design of the technological systems that affect them @xcite , developers may be obliged to bear these considerations in mind. gendering machines people may gender technologies in the face of even minimal gender markers @xcite , as evident in commercial dialogue systems @xcite . even without any gender markers, people still apply binary gender to dialogue systems @xcite , as was the case for the 'genderless' voice assistant q. while some companies now have begun to offer greater diversity of voices and have moved away from default female-gendered voices @xcite , nonbinary or gender-ambiguous dialogue systems such as sam @xcite are almost nonexistent, leaving people who identify as such without representation. summarizing @xcite , @xcite argue that that encouraging or enabling users to predominantly gender systems as female reinforces gender stereotypes of women as inferior to men: [digital assistants] reflect, reinforce and spread gender bias; model acceptance and tolerance of sexual harassment and verbal abuse; send explicit and implicit messages about how women and girls should respond to requests and express themselves; make women the 'face' of glitches and errors that result from the limitations of hardware and software designed predominately by men; and force synthetic 'female' voices and personality to defer questions and commands to higher (and often male) authorities. that is, by designing anthropomorphic systems or even simply leaving space for their (gendered) personification by users, developers risk enabling propagating stereotypes and associated harms. language variation and whiteness considering the narrative and fantasies around autonomous artificial intelligence, @xcite argue that autonomous systems are prescribed attributes such as autonomy, agency, and being powerfulattributes that are frequently ascribed to whiteness, and precluded from people of colour. in such, people of colour are removed, or erased, from the narrative and imagination around a society with autonomous systems @xcite . indeed, from a technical point of view, we see that, historically, nlp technologies have been developed to primarily capture the language use of voices of white demographics @xcite , in part due to their training data. in context of voiced dialogue systems, voices are similarly predominantly white @xcite . while there are many potential benefits to language technologies like dialogue systems, successful human-machine require that people conform their language use to what is recognised by the technologies. given the proclivity of nlp to centre white, affluent american dialects @xcite , language variants that deviate from these socio-linguistic norms are less likely to be correctly processed @xcite , resulting in errors and misrecognition, and forcing users to code switch to have successful engagements with dialogue systems @xcite . this can represent a form of language policing: people can either conform to the machine-recognisable language variant, or forego using it-and its potential benefitsaltogether. consequently, as people conform to language variants that are recognised by dialogue systems, they also conform to whiteness and the continued erasure of marginalised communities. the personification of such systems could exacerbate the erasure of marginalised communities, e.g. through limiting diverse language data. furthermore, system outputs often suffer from standardisation, for instance prioritising specific accents that conform to western notions of acceptability and prestige (see §3). thus, marginalised communities are forced to adopt their accent and (given the tendencies described in §2) personify 'white'-centred dialogue systems that are marketed as 'oracles of knowledge,' reifying hegemonic notions of expertise and knowledge. ## recommendations dialogue systems are used for a wide variety of tasks, and fine-grained recommendations may only be narrowly applicable. we therefore make broad recommendations for consideration: designers should recognise people's tendency to personify, consider which, if any, anthropomorphic tools are appropriate, and reassess both their research goals and the language used to describe their systems. recognise tendencies to personify human languages distinguish between linguistic form (e.g. string prediction in language modelling) and meaning (i.e. the relationship between form and communicative intent) @xcite . @xcite argue that humans reflexively derive meaning from signals, i.e. linguistic forms (within linguistic systems we have competence in), regardless of the presence of communicative intent. whether or not it is a part of a dialogue system's deliberate design to use specific linguistic forms (e.g. the cues outlined in §3), listeners will invariably perceive communicative intent. this is particularly so given that, until recently, open domain dialogue was only possible between humans. thus, unnecessary use of anthropomorphic linguistic cues can cause people to attribute humanlike cognitive abilities to systems-as was the case of google duplex, which excessively leveraged disfluencies. creators of dialogue systems should remain cognisant of these tendencies and carefully consider which anthropomorphic cues people may pick up on, and avoid sending such signals, whether they occur by design or through a lack of consideration (e.g. stemming from datasets). consider the appropriateness of anthropomorphic tools given our inherent nature to attribute meaning to signals, one must consider the appropriateness of the tool and use cases @xcite when designing dialogue systems, in order to avoid the (over-)integration of anthropomorphic cues. indeed, it is only within a given context that one can make judgement on whether anthropomorphism is a concern. for instance, personifying one's vacuum cleaning robot (i.e. shouting at it in frustration for not cleaning properly), is of less concern than the anthropomorphism of a dialogue system marketed as 'social' or 'empathetic', or technology sold as a 'singular oracle of (all) knowledge'. we therefore argue that developers should move towards focusing on the appropriateness of anthropomorphic tools in order to limit the negative consequences of anthropomorphism which can lead to false impressions of a system's capabilities. reassess research goals traditionally, the goal of artificial intelligence research has been to create systems that would exhibit intelligence indistinguishable from humans. tts systems for instance, are evaluated on how natural and fluent the output sounds. though intelligence and understanding should not be conflated with systems that exhibit humanlike behaviour @xcite , the human tendency to anthropomorphise convinces us of a machine's apparent intelligence @xcite . it is in part due to this longstanding goal of anthropomorphic systems that there only exists a small body of work that does not seek anthropomorphism, despite growing awareness of its harms. furthermore, these studies exist in isolation, and the taxonomy introduced in this paper highlights that we lack an approach that quantifies linguistic factors and relates them to possible harms and risks. thus, while it is infeasible to comprehensively map which linguistic cues to use or avoid, we discuss recommendations that arise from prior work. for example, @xcite recommend that developers produce synthesised voices that people recognise as non-human by calibrating mean pitch and pitch shimmer. in an analysis of reviews of commercial voice assistants, @xcite find that the big five personality traits @xcite do not adequately describe user expectations of systems' 'personalities'. the only consistently desired trait was agreeableness, as users expect prompt and reliable responses to queries @xcite . thus, imbuing voice assistants and dialogue systems with humanlike personality traits does not ensure alignment with people's expectation of system behaviour. we therefore recommend that designers and developers reassess the utility of embedding humanlike personality traits in dialogue systems. ## conclusion anthropomorphising dialogue systems can be attractive for researchers in order to drive user engagement. however, production of highly anthropomorphic systems can also lead to downstream harms such as (misplaced) trust in the output (mis-)information. even if developers and designers attempt to avoid including any anthropomorphic signals, humans may still personify systems and perceive them as anthropomorphic entities. for this reason, we argue that it is particularly important to carefully consider the particular ways that systems might be perceived anthropomorphically, and choose the appropriate feature for a given situation. by carefully considering how a system may be anthropomorphised and deliberately selecting the attributes that are appropriate for each context, developers and designers can avoid falling into the trap of creating mirages of humanity.