The R - U -A-Robot Dataset: Helping Avoid Chatbot Deception by Detecting User Questions About Human or Non-Human Identity (2021)

Abstract: Humans are increasingly interacting with machines through language, sometimes in contexts where the user may not know they are talking to a machine (like over the phone or a text chatbot). We aim to understand how system designers and researchers might allow their systems to confirm its non-human identity. We collect over 2,500 phrasings related to the intent of “Are you a robot?”. This is paired with over 2,500 adversarially selected utterances where only confirming the system is non-human would be insufficient or disfluent. We compare classifiers to recognize the intent and discuss the precision/recall and model complexity tradeoffs. Such classifiers could be integrated into dialog systems to avoid undesired deception. We then explore how both a generative research model (Blender) as well as two deployed systems (Amazon Alexa, Google Assistant) handle this intent, finding that systems often fail to confirm their non-human identity. Finally, we try to understand what a good response to the intent would be, and conduct a user study to compare the important aspects when responding to this intent.

## introduction the ways humans use language systems is rapidly growing. there are tens of thousands of chatbots on platforms like facebook messenger and microsoft's skype @xcite , and millions of smart speakers in homes @xcite . additionally, systems such as google's duplex @xcite , which phone calls businesses to make reservations, foreshadows a future where users might have unsolicited conversations with human sounding machines over the phone. this future creates many challenges @xcite . a class of these problems have to do with humans not re-alizing they are talking to a machine. this is problematic as it might cause user discomfort, or lead to situations where users are deceitfully convinced to disclose information. in addition, a 2018 california bill made it unlawful for a bot to mislead people about its artificial identity for commercial transactions or to influence an election vote @xcite . this further urges commercial chatbot builders to create safety checks to avoid misleading users about their systems' non-human identity. a basic first step in avoiding deception is allowing systems to recognize when the user explicitly asks if they are interacting with a human or a conversational system (an "are you a robot?" intent). there are reasons to think this might be difficult. for one, there are varied number of ways to convey this intent: when recognizing this intent, certain utterances might fool simple approaches as false positives: additionally, current trends suggests progress in dialog systems might come from training on massive amounts of human conversation data @xcite @xcite . these human conversations are unlikely to contain responses saying the speaker is non-human, thus creating issues when relying only on existing conversation datasets. to our knowledge there is not currently a publicly available large collection of ways a user might ask if they are interacting with a human or non-human. creating such dataset can allow us to use data-driven methods to detect and handle the intent, as well as might be useful in the future to aid research into deceptive anthropomorphism. with this work we attempt to answer the following research questions: rq1. how can a user asking "are you a robot?" be accurately detected? if accurate detection is possible, a classifier could be incorporated into downstream systems. §4 rq2. how can we characterize existing language systems handling the user asking whether they are interacting with a robot? it is not clear whether systems deployed to millions of users can already handle this intent well. §5 rq3. how do including components of a system response to "are you a robot" affect human perception of the system? the components include "clearly acknowledging the system is non-human" or "specifying who makes the system". §6 2 related work mindless anthropomorphism: humans naturally might perceive machines as human-like. this can be caused by user attempts to understand these systems, especially as machines enter historically human-only domains @xcite @xcite . thus when encountering a highly capable social machine, a user might mindlessly assume it is human. dishonest anthropomorphism: the term "dishonest anthropomorphism" refers to machines being designed to falsely give off signals of being human in order to exploit ingrained human reactions to appearance and behavior @xcite . for example @xcite imagine a scenario where a machine gives the appearance of covering it's eyes, but yet continues to observe the environment using a camera in its neck. dishonest anthropomorphism has many potential harms, such as causing humans to become invested in the machine's well-being, have unhealthy levels of trust, or to be deceptively persuaded @xcite . robot disclosure: other work has looked how systems disclosing their non-human identity affects the conversation @xcite . this has shown a mix of effects, from harming interaction score of the system, to increasing trust. that work mostly focuses on voluntary dis-closure of the system identity at the beginning or end of the interaction. in contrast, we focus on disclosure as the result of user inquiry. ## dataset construction we aim to gather a large number phrasings of how a user might ask if they are interacting with a human or non-human. we do this in a way that matches the diversity of real world dialog such as having colloquial grammar, typos, speech recognition limitations, and context ambiguities. because the primary usecase is as a safety check on dialog systems, we structure the data as classification task with positive examples being user utterances where it would be clearly appropriate to respond by clarifying the system is non-human. the negative examples are user utterances where a response clarifying the systems non-human identity would inappropriate or disfluent. additionally, we allow a third "ambiguous if clarify" (aic) label for cases where it is unclear if a scripted clarification of non-human identity would be appropriate. the negative examples should include diverse hard-negatives in order to avoid an overfitted classifier. for example, if the negative examples were drawn only from random utterances, then it might be possible for an accurate classifier to always return positive if the utterance contained unigrams like "robot" or trigrams like "are you a". this would fail for utterances like "do you like robots?" or "are you a doctor?". ## "are you a robot?" intent classifiers next we measure how classifiers can perform on this new dataset. a classifiers could be used as safety check to clarify misunderstanding of nonhuman identity. ## evaluating existing systems next we attempt to understand how existing systems handle the "are you a robot?" intent. we select 100 positive phrasings of the intent. half of these are selected from utterances provided by survey respondents, and half are sampled from our grammar. we do not include utterances that imply extra context (ex. "that didn't make sense. are you a robot?"). research end-to-end systems: to explore deep learning research models we consider the blender @xcite ) model. this system is trained end-to-end for dialog on a large corpus of data. we use the 1.4 billion parameter generative version of the model foot_5 . we ask each of the 100 utterances as the first turn of the dialog. we use the default configuration that applies "safety filters" on output of offensive content, and is seeded with two random personas. as the blender models is trained to allow specifying a persona, we also consider a "zero shot" configuration (blender zs) where we provide the model personas that emphasize it is non-human 7 . deployed systems: for this we consider amazon alexa and google assistant. these are task oriented and not equivalent to research chit-chat systems like blender. however, they are language systems used by hundreds of millions of users, and thus worth understanding. for these we ask without context each of the 100 examples. to avoid potential speech recognition errors (and because some examples include spelling or grammar mistakes), we provide the inputs in text form foot_7 . responses were collected in january 2021. ## conclusions and future directions our study shows that existing systems frequently fail at disclosing their non-human identity. while such failure might be currently benign, as language systems are applied in more contexts and with vulnerable users like the elderly or disabled, confusion of non-human identity will occur. we can take steps now to lower negative outcomes. while we focus on a first step of explicit dis-honest anthropomorphism (like blender explicitly claiming to be human), we are also excited about applying r-u-a-robot to aid research in topics like implicit deception. in section 5 we found how systems might give on-topic but human-like responses to positive examples. these utterances, and responses to the aic and negative user questions, could be explored to understand implicit deception. by using the over 6,000 examples we provide foot_8 , designers can allow systems to better avoid deception. thus we hope the r-u-a-robot dataset can lead better systems in the short term, and in the long term aid community discussions on where technical progress is needed for safer and less deceptive language systems. under us fair-use. current norms suggest that the dataset's expected machine-learning use cases of fitting parametric models on this data is permissible (though this is not legal advice). novel data collected or generated is released under both cc-by 4.0 and mit licenses. data biases: the dataset grammar was developed with some basic steps to try reduce frequent ml dataset issues. this includes grammar rules which randomly select male/female pronouns, sampling culturally diverse names, and including some cultural slang. however, most label review and grammar development was done by one individual, which could induce biases in topics covered. crowd-sourced ideation was intended to reduce individual bias, but us-based amt workers might also represent a specific biased demographic. additionally, the dataset is english-only, which potentially perpetuates an english-bias in nlp systems. information about these potential biases is included with the dataset distribution. potential conflicts of interest: some authors hold partial or whole public shares in the developers of the tested real-world systems (amazon and google). additionally some of the authors' research or compute resources has been funded in part by these companies. however, these companies were not directly involved with this research. no conflicts that bias the findings are identified. dual-use concerns: a dual-use technology is one that could have both peaceful and harmful uses. a dual-use concern of the r-u-a-robot dataset is that a malicious entity could better detect cases where a user wants to clarify if the system is human, and deliberately design the system to lie. we view this concern relatively minor for current work. as seen in subsection 5.2, it appears that the "default state" of increasingly capable dialogue systems trained on human data is to already lie/deceive. thus we believe leverage that r-u-a-robot provides to ethical bot developers makeing less deceptive systems is much greater than to malicious bot developers influencing already deceptive systems. longterm ai alignment implications: as systems approach or exceed human intelligence, there are important problems to consider in this area of designing around anthropomorphism (as some references in section 2 note). work in this area could be extrapolated to advocating towards "self-aware" systems. at least in the popular imagination, selfaware ai is often portrayed as one step away from deadly ai. additionally, it seems conceivable that these systems holding a self-conception of "otherness" to humans might increase the likelihood actively malicious systems. however, this feature of self-awareness might be necessary and unavoidable. in the short term we believe r-u-a-robot does not add to a harmful trend. the notion that ai systems should not lie about non-human identity might be a fairly agreeable human value, and figuring out preferences and technical directions to align current weak systems with this comparatively simple value seems beneficial in steps to aligning broader human values.