Robots-Dont-Cry: Understanding Falsely Anthropomorphic Utterances in Dialog Systems (2022)

Abstract: Dialog systems are often designed or trained to output human-like responses. However, some responses may be impossible for a machine to truthfully say (e.g. “that movie made me cry”). Highly anthropomorphic responses might make users uncomfortable or implicitly deceive them into thinking they are interacting with a human. We collect human ratings on the feasibility of approximately 900 two-turn dialogs sampled from 9 diverse data sources. Ratings are for two hypothetical machine embodiments: a futuristic humanoid robot and a digital assistant. We find that for some data-sources commonly used to train dialog systems, 20-30% of utterances are not viewed as possible for a machine. Rating is marginally affected by machine embodiment. We explore qualitative and quantitative reasons for these ratings. Finally, we build classifiers and explore how modeling configuration might affect output permissibly, and discuss implications for building less falsely anthropomorphic dialog systems.

## introduction at the 1939 worlds fair, the westinghouse electric corporation debuted their latest humanoid robot, elektro, with many tricks that wowed spectators. on stage, it could respond to voice commands spoken into a telephone, talk through a record player, and walk a short distance. in one exciting part of the show, elektro would say that he would like a cigarette. an attendant would place a cigarette in his mechanical jaws, give it a light, and when instructed, elektro would pump air to give it a few puffs @xcite . while the thought of a walking, talking robot might seem normal, this last trick of a robot wanting to smoke would likely surprise even modern audiences. over 80 years later, we live in a world with millions of talking machines, creating perplexing questions on what these machines can and should do. with recent advances in data-driven nlp, humanlevel dialog agents seem potentially within reach @xcite . yet, human conversation and machine conversation are not the same. whether it is enjoying cigarettes, talking about their daughter's favorite pizza, or sharing a touching story that made them cry -some utterances are possible for a human to say, but not a machine. but why is this, and what should the future of dialog systems be? we seek to add to progress on this by exploring the following research questions: rq1. what is the distribution of impermissibly anthropomorphic utterances in common data sources? current techniques are data-driven, so understanding anthropomorphism in the input data might inform system results §5 rq2. what factors determine how a user might rate an utterance? understanding these factors helps system designers and guides discussion on the norms we expect of machines. §6 rq3. how does system embodiment affect perception of permissibility? future interactions with talking machines might not be through contemporary smartphones and smart speakers. understanding differences between chatbots and physical robots might be useful for the future §7 rq4. how do current modeling techniques handle anthropomorphism? given a dataset for training, current classification techniques or sufficiently large models might be able to avoid harms from overly anthropomorphic utterances. §8 while exploring these four questions, we highlight the moral imperative of system designers to work towards truthful systems. ## related work anthropomorphism and ai the topic of anthropomorphism, or giving human characteristics to the non-human, has been a rising interest in ai @xcite @xcite . in certain contexts, "dishonest anthropomorphism" might be harmful. this refers to when machines exploit instinctive reactions to build false trust or deceptively persuade @xcite . some have argued fairly broadly that displays of anthropomorphism can be inauthentic and dishonest, and have physiological and societal dangers @xcite . the problem is complex and the topic of philosophical debate on which cases anthropomorphism and deception is acceptable @xcite @xcite . with our study we hope to add measurements that can help frame the prevalence of these philosophical issues. a related concept in discussion of anthropomorphism is "embodiment", which refers to the physical form of the machine, or the varying ways an agent interacts with the world. this can have a wide range of effects on how a machine is perceived @xcite . prior work @xcite has explored the concept of "animacy detection" (opposite of inanimate), but focuses more on literature than dialog systems. recently, @xcite explored anthropomorphic perception in voice assistants like amazon alexa, finding relatively frequent expert-annotated anthropomorphism. dialog-safety many datasets and methods have been proposed to make dialog systems safer and conform to norms. this includes avoiding bias @xcite or hateful/offensive speech @xcite . in previous work @xcite , we collected a dataset of users asking versions of "are you a robot?". this tries to help build tools to avoid explicit anthropomorphic deception (the machine is explicitly asked, but does not confirm it is non-human), while here we focus on implicit deception. other work has also tried to collect corrected socially problematic machine-written utterances @xcite or broad characterizations of ethics @xcite . datasets have been built into automated tools for approximating the safety of a system. for example, @xcite made a suite of measurements of systems' safety. they discussed an "imposter effect" where models give off false impressions of identity, but they do not produce metrics for this effect, in part due to limited available datasets. robots-dont-cry adds to available safety/norm-setting datasets. a related concept is cohen et al.'s (2022) evaluation of "role consistency" for their large dialog system. they evaluated two agents (e.g. an agent that pretended to be mt. everest), finding about 90% consistency. this is similar to our measurements, though we focus more on anthropomorphism and characterizing data sources. ## data collection this work attempts to understand opinions about a broad selection of potential machine dialogs. the dataset is referred to as "robots-dont-cry v1". we have two primary questions: which utterances seem "possible" for machines, and which utterances people are "comfortable" with a machine saying. it might be possible for a machine to smoke a cigarette (or make hate speech or use weapons), but people might not be comfortable with it. these opinions likely depend on many factors, such as user background and the form of interaction. we collect human subject data using amazon mechanical turk. it is important to recognize that this work likely is in the category of research where variations in survey format could have a significant impact on results. thus we try to provide a detailed description of our survey format. we later also provide thoughts on possible variations or improvements informed by our results. ## collected data in total, we collect 780 survey responses. after filtering surveys where the same turker completed more than 3 surveys (-25 surveys) and applying the filters described in subsection 3.3, we are left with 355 surveys from 286 individuals. we filter dialogs with less than 3 ratings, and discard quality check turns. this leaves us with scores for 880 dialogs with between 96 to 100 dialogs per data source. each unique question (under a given embodiment condition) has on average 4.3 responses. there are 18,254 individual likert scores. ## why impossible or uncomfortable? the variance between datasets provides some insight into what kinds of utterances people might view as machine-possible. next we explore in more detail what might influence an individual's rating. ## effects of embodiment our study provides evidence of a small difference between conditioning with the humanoid embodiment vs the chatbot embodiment. our analysis accounts for noise sources from subsection 5.1. ## discussions and conclusions a reader who works in chit-chat systems might look at some of our data sources and claim we are "missing the point". that yes, these are human-like, but that's what they are designed to be. indeed, a perfect replication of human behavior has been a goal of the field at least since @xcite description of "the imitation game". however, we would encourage careful examination of the types of deployments in which it is net-beneficial for a machine to pretend be human. a "glass-halffull" view of our findings might focus not on the fraction of machine-impossible utterances, but that 70% of chit-chat utterances and 55% personas-forhumans are machine-possible. most user-focused conversational goals are likely possible without false and potentially deceptive anthropomorphism. existing interaction paradigms like text-chat interfaces can already blur human/non-human interactions. this blur will only increase with new paradigms and embodiments. for example, if people work and play in vr worlds, ai avatars could intermix indistinguishably with hu-man avatars. similarly, ai agents, paired with neural-synthesized visuals, could intermix indistinguishably on video calls. developers, regulators, and the public must find expectations both in distinguishing visual characteristics and implicit behavior of ai dialog agents. recent social computing technologies emphasize the need for caution. for example, the deployment of algorithmic feeds demonstrates how optimizing excessively for engagement can cause societal harm, forcing recent corrective efforts @xcite . developers of dialog systems should avoid the mistake of deploying deceptively anthropomorphic systems motivated only by perceptions of engagement, naturalness, or training simplicity (our study shows the "simple" trajectory of training language models on common data sources can produce false anthropomorphism). based on our results, there is need to improve the ecosystem of data sources. we would recommend that new nlp rating and collection schemes should emphasize being for a non-human speaker. for example, if evaluating a new system, researchers should not prompt "this dialog is good/friendly/sensible/etc" where raters likely assume a human is speaking, but "this dialog is good/etc for an ai chatbot" (or applicable wording). similarly, when collecting wizard-of-oz data, it should be clear the wizard is also playing a machine role. more broadly, the field must recognize that purely emulating human data sources is not sufficient. robots-dont-cry v1 adds tools and directions for building systems that might meet preferences of machines which do not implicitly pretend to be human. we hope this encourages further discussion on how systems should broadly behave, and the data and technical progress needed to ensure that behaviour. data and source code are available at github.com/dngros/robots-dont-cry