How “open” are the conversations with open-domain chatbots? A proposal for Speech Event based evaluation (2021)

Abstract: Open-domain chatbots are supposed to converse freely with humans without being restricted to a topic, task or domain. However, the boundaries and/or contents of open-domain conversations are not clear. To clarify the boundaries of “openness”, we conduct two studies: First, we classify the types of “speech events” encountered in a chatbot evaluation data set (i.e., Meena by Google) and find that these conversations mainly cover the “small talk” category and exclude the other speech event categories encountered in real life human-human communication. Second, we conduct a small-scale pilot study to generate online conversations covering a wider range of speech event categories between two humans vs. a human and a state-of-the-art chatbot (i.e., Blender by Facebook). A human evaluation of these generated conversations indicates a preference for human-human conversations, since the human-chatbot conversations lack coherence in most speech event categories. Based on these results, we suggest (a) using the term “small talk” instead of “open-domain” for the current chatbots which are not that “open” in terms of conversational abilities yet, and (b) revising the evaluation methods to test the chatbot conversations against other speech events.

## introduction there has been a recent surge in the research and development of so-called "open-domain" chatbots (e.g., @xcite @xcite ). these chatbots are typically trained in an end-to-end fashion on large datasets retrieved from different internet sources such as publicly available online discussion forums (e.g., reddit). while the idea of an "open-domain" chatbot (not engineered towards a specific task, but just trained in an agnostic fashion from data), is appealing, there is a lack of clarity on what exactly "open" means. in their paper introducing the meena chatbot, adiwardana et al. (2020) provide the following definition: "unlike closed-domain chatbots, which respond to keywords or intents to accomplish specific tasks, open-domain chatbots can engage in conversation on any topic". is this really the case? the answer to this question will depend on the contexts in which the chatbot is put to test. based on @xcite concept of "language games", @xcite argues that in order to interpret an utterance and generate a meaningful response, the "activity type" in which the exchange takes place is vital. the activity can be described in various terms (e.g., setting, participants, purpose, norms) and it provides the necessary constraints on the interpretation space (e.g., which implications can be made, what can be considered to be a coherent and meaningful contribution to the activity). from this perspective, every conversation has an assumed purpose or motive. this purpose could be characterized as being more "task-oriented" (e.g., planning a trip together or buying a ticket) or just passing time by conversing with each other without a specific task in mind (e.g., gossiping, recapping the day's events). regardless of the activity type, there is always some shared knowledge about the context and expectations of the participants (i.e., humans) from each other (e.g., avoid being rude unless it is intentional) in real-world settings. among other terms (e.g., speech genre, joint action, social episode, frame), goldsmith and baxter (1996) refers to activity types as "speech events" and we will use this term in our paper as well. the nature of the activity or the purpose of the interaction is often not clear in a typical setting for testing open-domain chatbots. usually, a user (often a crowd worker) is asked to "chat about anything" with an agent they have never met before without any further instructions. in most cases, neither the user nor the chatbot has any prior relationship with each other and they will probably never talk again. this type of context is unusual for real-world conversations between humans. the closest example is perhaps engaging in "a small talk" to pass time while waiting at the bus stop or at a dinner party where we might be placed next to a new acquaintance. however, even in those situations, we do have some shared context. since the context restricts the types of speech events that will arise, it is hard to know whether such a chatbot is actually able to engage in conversations freely (i.e., on any topic). therefore, we explore how "speech events" (goldsmith and baxter, 1996) can be applied to the analysis and evaluation of chatbots with two studies. first, we classify the types of "speech events" encountered in a chatbot evaluation data set. second, we conduct a small-scale pilot study to evaluate how well a state-of-the-art chatbot can handle conversations representing a more diverse set of speech event categories. before describing the studies and results in detail, we first provide an overview of the development of open-domain chatbots in the next section. ## open-domain chatbots the term "chatbot" (and its predecessor "chatterbot") has been used since the early 1990's to denote systems that interact with users in the form of a written chat. early examples of such systems include tinymud (mauldin, 1994) and alice (wallace, 2009). however, the term has not been used in academia until recently (adamopoulou and moussiades, 2020). instead, "dialogue system" was a more common term for systems that interact with users in a (written or spoken) conversation. nowadays, the meaning of the term "chatbot" seems to vary, and it is also used interchangeably with the term "dialogue system" (deriu et al., 2020; adamopoulou and moussiades, 2020). deriu et al. (2020) make a distinction between task-oriented, conversational and question-answering chatbots, where conversational chatbots "display a more unstructured conversation, as their purpose is to have open-domain dialogues with no specific task to solve". these chatbots are built to "emulate social interactions" (ibid.). another term sometimes used to differentiate these chatbots from more taskoriented systems is "social chatbots" @xcite . however, it is not entirely clear how the term "social" should be understood in this context. we argue that the notion of "speech events", used in this paper, provides a much richer taxonomy for the various forms of conversations that people engage in. the recent trend of modelling open-domain conversations was sparked by early attempts to use the same kind of sequence-to-sequence models that had been successful in machine translation @xcite . another driving force is the competitions to develop chatbots that can engage in an open-domain conversation coherently for a certain period of time, such as the alexa prize challenge @xcite and the conversational intelligence challenge (convai) @xcite . in terms of evaluation criteria, task-oriented conversations are typically evaluated with task success and efficiency @xcite . however, there is also need for other criteria to evaluate opendomain chatbots which do not necessarily have a clear task. the most famous (and earliest) form of evaluation is perhaps the imitation game (often referred to as the turing test) proposed by @xcite . however, it is not clear that being able to distinguish a chatbot from a human is a sensible test, as this might lead to a focus on handling trick questions rather than modelling a human conversation. another form of evaluation is used in the loebner prize competition, where judges are asked to rate the chatbot responses after asking a fixed set of questions to each chatbot @xcite . this approach faces the problem of not testing the chatbot's ability to have a coherent and engaging interaction over multiple turns. to put more emphasis on the user experience, the users in the alexa prize challenge were asked to rate the chatbot after the interaction, on a scale between 1 and 5, which was then used as a direct measure of performance. however, such a scale is difficult to interpret since it may not reflect the specific criteria utilized by each user to evaluate the relative merits of each chatbot transparently. another method is to let the human users interact with the chatbot and let a third-party (i.e., other humans) rate the conversations on different dimensions either on a turn-by-turn basis or on the level of whole conversations. adiwardana et al. (2020) used a turn-by-turn assessment, and argued that the most important factors in such an evaluation are the "sensibleness" and "specificity" of responses. a common problem with end-to-end chatbots is a tendency to give very generic responses (e.g., "i don't know"), which might be evaluated as sensible (or coherent), but not very specific (and therefore not very engaging). a limitation of turn-by-turn assessments is that they do not capture the global coherence of the conversation as a whole. deriu et al. (2020) criticise the term "high quality" for evaluating human-chatbot conversations due to its subjectivity. instead, they propose appropriateness, functionality and target audience as more objective measures. @xcite proposed a method called acute-eval, in which human judges are asked to make a binary choice between questions (e.g., "who would you prefer to talk to for a long conversation?", "which speaker sounds more human?") for two human-chatbot conversations that are displayed next to each other. regardless of the exact evaluation criteria used, the general assumption behind open-domain chatbots seems to be a set-up where human users are exposed to the chatbot without much introduction or guidance on what to talk about. for example, in the alexa prize challenge, the users are encouraged to start chatting with their alexa devices by just saying "let's chat". to make the conversation more engaging and provide some guidance, some chatbots are given a "persona" @xcite . while such personas can provide some more context, they do not provide much guidance towards the purpose of the conversation. in order to compensate for "naturalness" and "relevance to real-world use cases", shuster et al. (2020) experimented with letting the humans interact with opendomain chatbots in a fantasy game. although the authors praise the ease of data collection through gaming, the extent to which these conversations reflect real-world scenarios is questionable. considering the vagueness around the definitions of an open-domain chatbot and the variation in applications, what does "open domain" mean for conversations with the chatbots? how do we expect the chatbots to handle these conversations? we will explore the answers to these questions by introducing and explaining "speech events" in the next section. ## speech events exploring and categorizing different types of conversations is a daunting task, as the number of potential categories is (in theory) unlimited, and de-pendent on the contexts and participants we should consider. thus, it is important to approach the problem in a systematic way. in this paper, we will base our work on goldsmith and baxter (1996), who use the term "speech events" for different types of conversations. as a note, speech events describe the conversation as a whole, and should not be confused with the term "speech act", which describes the intention of a single conversational turn. in a series of four studies, goldsmith and baxter (1996) developed a descriptive taxonomy of speech events between humans in everyday conversations. first, they collected 903 open-ended diary log entries provided by 48 university students who monitored their daily conversations with other people for a 1-week period. using this data, the authors identified the speech events, labeled and grouped them in a systematic fashion. these categories were also analyzed according to a set of dimensions, including formality, involvement and positivity. in total, 39 speech events were identified, which we group into three major categories (see the definitions for each category in the appendix): our study is based on this categorization of speech events. however, we do not argue that this is the ultimate way to categorize speech events and acknowledge that the exact set of categories are likely to be influenced by the demographics of the group which was under study (university students), and the limited time during which the data was collected. bearing these restrictions in mind, this wider set of speech event categories is appropriate to illustrate our points in this study. ## study i in the first study, we aimed at categorizing the types of speech events that occur in the current evaluations of open-domain chatbots based on the categories defined by goldsmith and baxter (1996) as described above. ## study ii although the results of study i indicate a tendency for one type of speech event ("small talk"), we still do not know how well chatbots could handle other speech event categories and a more thorough analysis of this for various chatbots is beyond the scope of this paper. however, to get an idea about what such an evaluation procedure could look like, we designed a small-scale pilot study with the following research questions in mind: what would be an alternative evaluation scheme involving other speech event categories? how would a state-ofthe-art open-domain chatbot perform in such an evaluation? ## conclusion the results of our two studies show that the typical setting for chatbot evaluations (where the tester is asked to "just chat with the chatbot") tends to limit the conversation to the "small talk" speech event category. therefore, the reported results from such evaluations will only be valid for this type of speech event. in a pilot study where a tester was instructed to follow a broader set of speech event categories, the performance of the state-ofthe-art chatbot (i.e., blender in this case) seems to degrade considerably, as the chatbot struggles to keep a coherent conversation in alignment with the purpose of the conversation. we thus propose that developers of "opendomain chatbots" either explicitly state that their goal is to model small talk (and perhaps use the term "small talk chatbots" instead of "open domain chatbots"), or that they change the way these chat-bots are evaluated. for the second option, these chatbots could be tested against a wider repertoire of speech events, before claiming that they can communicate like humans or have a human-level performance. in addition, there is a pressing need to develop better evaluation frameworks, where the tester is provided with a context for a specific speech event and clear instructions. if this route is followed, it would be possible to evaluate the the performance of the chatbot for the specific speech event category.