The Open-domain Paradox for Chatbots: Common Ground as the Basis for Human-like Dialogue (2023)

Abstract: There is a surge in interest in the development of open-domain chatbots, driven by the recent advancements of large language models. The “openness” of the dialogue is expected to be maximized by providing minimal information to the users about the common ground they can expect, including the presumed joint activity. However, evidence suggests that the effect is the opposite. Asking users to “just chat about anything” results in a very narrow form of dialogue, which we refer to as the “open-domain paradox”. In this position paper, we explain this paradox through the theory of common ground as the basis for human-like communication. Furthermore, we question the assumptions behind open-domain chatbots and identify paths forward for enabling common ground in human-computer dialogue.

## introduction recent advancements of large language models (llms) have given rise to a surge in interest for the development of "open-domain" chatbots @xcite @xcite . unlike task-oriented dialogue systems designed for a specific purpose and typically implemented in a modular fashion, open-domain chatbots are trained end-to-end on large amounts of data. @xcite define their long-term goal as "building a superhuman open-domain conversational agent" that is "preferred on average to an alternative human speaking partner in open conversation". the more specific purpose of such a conversational agent is not stated, and thus it is implied that dialogue is a generic problem that can be abstracted away from the context in which it takes place. in current evaluations of open-domain chatbots, there seems to be a general assumption that the "openness" of the dialogues can be maximized by removing as much instructions and context as possible (e.g., by instructing the user to "just chat about anything"). while this might seem intuitive at first in terms of removing the boundaries for "openness", we argue that this assumption stems from a misconception and that dialogue as a linguistic activity cannot be stripped from its context. the setting in which open-domain chatbots are evaluated does not clearly correspond to any form of human-human dialogue "in the wild". in this position paper, we analyse this misconception as the open-domain paradox: the diversity of the various forms of dialogues found in humanhuman interaction does not stem from the "openness" of the dialogue setting, but rather the opposite: they stem from the diversity of highly specific contexts in which dialogue takes place. if this is true, it means that the current methods for collecting dialogue data and evaluating open-domain chatbots will only give rise to a very narrow form of dialogue which does not correspond closely to human-human dialogues. thus, they will not tell us whether these systems are truly "open". nor will they tell us much about how good these systems actually are at modelling various dialogue phenomena. from the user's perspective, if the common ground and the reason for having the interaction is not clear, there is a risk that the system will not be perceived as meaningful. our contribution has the following goals: first, we provide a critical review (not an extensive survey) of sota open-domain chatbots, in terms of how they are defined, trained and evaluated. we discuss how the lack of common ground has consequences for their limited scope and arguably their "openness", compared to human-human dialogue. secondly, we provide various research directions which might help to mitigate this problem and enable common ground in human-computer dialogue. when we initiate a dialogue as humans, we do not start with a blank slate but we assume some com-mon ground between the speakers/interlocutors. @xcite describes common ground among humans as "the sum of their mutual, common or joint knowledge, beliefs and suppositions". for a successful and meaningful communication to take place, and for coordinating joint actions, it is essential that both parties have a shared understanding of what this common ground is. @xcite makes a distinction between communal and personal common ground. communal common ground refers to the cultural communities (e.g., nationality, profession, hobbies, language, religion, politics) people belong to. in addition, there could also be cultural communities which are shaped around shared expertise specific to the members of that community who may not live in the same place (e.g., english teachers around the world), and it is possible to belong to more than one cultural community at the same time. clark (1996) makes a further distinction in communal common ground between human nature (i.e., same senses, sense organs, types of sensations), communal lexicons (e.g., there are conventions about word meanings even when two interlocutors speak the same language), as well as cultural facts, norms and procedures (which are commonly shared within that community). procedures for joint activities are the underlying notions about common ground for the community members who know the specific "scripts" for the procedures about joint activities in certain contexts (e.g., restaurants, supermarkets vs. school). personal common ground is based on personal joint experiences with someone @xcite , and is further classified into perceptual bases, actional bases and personal diary. one important aspect of the personal common ground, apart from shared memories and commitments, is the linguistic alignment whereby human interlocutors align (or adjust) their language in alliance with their conversational partners @xcite , context and medium of communication @xcite . since childhood, humans learn how to adjust and tolerate linguistic variation (e.g., across conversational partners, contexts, mediums) between different communal and personal common grounds in their environment. the theory of common ground also postulates the principle of least collaborative effort, which means that people in conversation use their assumed common ground to minimize their collabora-tive effort to achieve further understanding @xcite . thus, a brief word might have a significant meaning if the context is highly specific or the interlocutors know each other well. as another example, @xcite showed that conversations between children and their caregivers are hard to transcribe, since their common ground is not known to the transcriber. an important part of common ground is the joint activity that is assumed, that is, the reason why the interaction is taking place. this is similar to @xcite concept of language games or the notion of activity type developed by @xcite . a related concept is that of speech events developed by @xcite (not to be confused with "speech acts"), which refers to the type of activity that the parties are involved in. by analysing transcribed speech diaries from 48 university students over a 1-week period, they developed a taxonomy of 39 speech events: ## lack of common ground in "open-domain" dialogue while the above-mentioned metrics do say something about the relative merits of chatbots, they do not tell us much about their "openness", or the diversity of the speech events they can engage in. when doing so, more attention should be given to the setting in which the chatbots are evaluated. since there is no natural setting in which these open-domain chatbots are used, crowd workers are typically recruited to interact with them, either for data collection or for evaluation purposes. although the crowd workers are typically informed about whether they are interacting with a human or a computer, the setting is similar to that of the turing test mentioned above, in the sense that no information about the assumed common ground is provided, and they are often asked to initiate the interaction with as few instructions as possible. for the meena chatbot, "conversations start with 'hi!' from the chatbot to mark the beginning of the conversation and crowd workers have no expectation or instructions about domain or topic of the conversation" @xcite . for the lamda chatbot, crowd workers were instructed to "start a conversation with the chatbot by posing a question or typing a statement on any topic you want to talk about" @xcite . for the alexa prize, the users were not provided with any details on what they could expect. users were asked (through commercials) to just say "let's chat" to their smart speaker in order to initiate the interaction, but no other instructions were provided @xcite . these forms of generic and minimalistic instructions are perhaps chosen to provide as little bias as possible in terms of what topics will be brought up, and to really stress the "open domain" nature of the chatbots. however, given what has been discussed above about the importance of common ground and a shared understanding of what the speech event is supposed to be in human-human dialogues, the setting of open-domain chatbots without any common ground is quite unnatural. there is also no physical context or visual cues that could be used to infer any common ground. it is hard to find any similar setting for a human-human conversation. even if we initiate a small talk with a stranger when waiting for the bus, we both know that this is the type of activity we are engaged in, which will guide us in what might be appropriate to talk about in that context. the equivalent would rather be to be randomly connected to a person without any knowledge about that person or about what the conversation is supposed to be about. can we expect such a setting to give rise to a wide variety of speech events and topics? if not, how do we know if these systems would be able to handle them? to address this question, dogruöz and skantze (2021) annotated a subset of the publicly released chats from the meena chatbot @xcite based on the closest speech event category from @xcite . the results showed that almost all of them belonged to the small talk category, indicating that they were indeed very limited in scope in terms of speech events. interestingly, the same was found when annotating the human-human chats that was used as a reference in the evaluation of the meena chatbot. those dialogues had been collected by randomly connecting (google) employees through a chat based system and asking them to converse about anything, to create a similar setting as that for the human-chatbot interactions. this indicates that it was not primarily the users' expected (lack of) agency of the interlocutor that limited the scope of the dialogue, but rather the limited instructions about the context for the interaction. these findings point to what we have referred to as the open-domain paradox: a completely "open" setting for conversation, where it is not possible to assume any form of common ground, does not give rise to an "open-domain" dialogue, but rather a very limited form of dialogue in terms of both speech events and factual information. whether the setting for evaluating open-domain chatbots actually gives rise to a diversity of speech events has consequences for our understanding of their capabilities. in their evaluation of the blender chatbot, @xcite reported that it was rated on the same level as the human-human chats taken from the meena evaluation @xcite . however, in an experiment by dogruöz and @xcite , they also tested whether blender could handle other speech events than the small talk that would normally take place in an "open-domain" type of evaluation. this was done by giving a human tester the task of interacting with blender on a set of different speech events, such as decision-making or making plans. in this evaluation, the chatbot performed much worse (compared to a human interlocutor). this shows that the setting for the interaction and the instructions provided to the testers influence the outcome of the evaluation. ## enabling common ground while the idea of an "open-domain" setting for chatbots is quite pervasive in current research, there are also other trends (and forgotten lessons) pointing towards systems where the context is more specific and where the user can potentially assume some form of common ground. in this section, we will discuss those lines of work, and explore to what extent they could increase the diversity of speech events and open a path towards more human-like (and possibly more meaningful) human-computer dialogue. ## conclusion and discussion our goal in this paper was not to survey the latest literature on chatbots but to question the assumptions behind the term "open-domain", and scrutinize to what extent chatbots labelled as such are truly "open". we discussed the notion of common ground in human-human dialogue, and how it is important for human-like dialogue and a diversity of speech events and topics. the general assumption behind sota open-domain chatbots is instead to remove as much context as possible, often presenting users with an empty prompt and asking them to "just chat". however, both linguistic theory and evidence suggests that the absence of context does not give rise to a diversity of speech events, but rather a very limited form of dialogue. we called this the open-domain paradox. to be able to study and model different forms of dialogue between humans and agents, the dialogue needs to be embedded within a highly specific context, where both the agent and the human can assume some form of common ground. we identified a couple of different paths towards this end in section 5. one explanation for the huge interest in the development of (context-less) open-domain chatbots is perhaps that it fits well with the llm paradigm (next-token prediction), which uses a limited prompt with the dialogue history. in a way, it is an example of a solution that has found its problem. it is of course possible to include a larger context in the prompt (i.e., a textual representation of the common ground that could be expected), but this clearly has its technical limits. it will be interesting to see whether there will be a movement towards other solutions, perhaps using more modular architectures (as in the example of cicero). from the perspective presented here, opendomain chatbots (as the term is currently used) are not necessarily more "generic" than task-oriented dialogue systems, given the limited form of dialogue they are typically evaluated against. one option would be to re-brand open-domain chatbots as "small talk chatbots" or, as some have suggested, "social chatbots" @xcite . we do not think this is appropriate either, since even smalltalk (between humans at least) is dependent on the presumed common ground between the speakers. we do not exclude the possibility that the type of dialogue crowdworkers have with open-domain chatbots (i.e., dialogue without common ground) can be regarded as a special speech event category, which we have no existing terminology for yet. for future work, it might be better to characterize dialogue systems based on which contexts they are intended to be used in, and what speech events are expected to take place. when evaluating such systems, it is also important that they are used in the context they were intended for. as we have discussed, according to the "principle of least collaborative effort", humans use common ground to make their interactions more efficient @xcite . thus, users of dialogue systems will likely always prefer to use systems where their common ground is maximized, rather than "open-domain" settings. if one would want to develop a generic "superhuman open-domain conversational agent" @xcite , it would need to be highly context-aware, in order to serve across contexts. this route is perhaps not very realistic, given the incredible richness and diversity of the forms of common ground humans assume and build together. also, this does not even exist for human-human dialogues, as we do not speak about anything with anybody randomly at any given time, without any common ground. instead of open-domain dialogue systems, it might be more fruitful to focus on developing a large variety of highly context-specific dialogue agents.