“It’s our fault!”: Insights Into Users’ Understanding and Interaction With an Explanatory Collaborative Dialog System (2021)

Abstract: Human-AI collaboration, a long standing goal in AI, refers to a partnership where a human and artificial intelligence work together towards a shared goal. Collaborative dialog allows human-AI teams to communicate and leverage strengths from both partners. To design collaborative dialog systems, it is important to understand what mental models users form about their AI-dialog partners, however, how users perceive these systems is not fully understood. In this study, we designed a novel, collaborative, communication-based puzzle game and explanatory dialog system. We created a public corpus from 117 conversations and post-surveys and used this to analyze what mental models users formed. Key takeaways include: Even when users were not engaged in the game, they perceived the AI-dialog partner as intelligent and likeable, implying they saw it as a partner separate from the game. This was further supported by users often overestimating the system’s abilities and projecting human-like attributes which led to miscommunications. We conclude that creating shared mental models between users and AI systems is important to achieving successful dialogs. We propose that our insights on mental models and miscommunication, the game, and our corpus provide useful tools for designing collaborative dialog systems.

## introduction task-oriented dialog systems are designed to help users accomplish a specific goal, such as getting a recommendation or booking an appointment. however rather than interacting as equals, the user maintains control in the dialog, all decisions passing through them. while appropriate for tasks, such as recommender systems, this form of communication leaves the labor of decisions solely on the human and discounts advantages in an artificial *both authors contributed equally to this research. intelligence's (ai) reasoning ability @xcite which can limit the potential of human-ai partnerships @xcite . collaborative dialog offers an alternative @xcite where both parties contribute to the final strategy. it has been a long standing goal in ai to construct computer systems which can effectively act as partners in collaborative activity @xcite . as an example, if a human and robot are working together, collaborative dialog could allow the robot to communicate the scope of its abilities, enabling both partners to work together to develop an effective strategy @xcite . in more complex scenarios, collaborative dialog could be useful for applications, such as search and rescue @xcite or medical diagnosis recommendations @xcite . one important aspect of human-ai collaboration is how well users can form sound mental models of their partner. put simply, a mental model is a person's cognitive representation for how/why a complex system (e.g., a dialog system) works @xcite @xcite . rather than trying to process all aspects of such systems at once, users will create a simplified model in their mind @xcite . these models can range from simple, e.g., "if i say something, the system will say something back" to arbitrarily complex e.g., "the system recognizes that if i take too long to answer then i am upset, so it tries to comfort me." these cognitive representations are shaped through both the users' observations and previous experiences. explainable artificial intelligence (xai) aims at supporting humans to know when, whether, and why to trust and rely on an ai system (or not) @xcite , and therefore helps to build sound mental models. in cooperative tasks, sound mental models can help a user predict the behavior of their partner @xcite and avoid miscommunication @xcite . however, currently it is unclear how users perceive collaborative dialog systems and what mental models they form about them. concretely, we then ask the following research questions: to answer these questions, we first created a novel cooperative puzzle game and explanatory dialog system. we then conducted an experiment with 129 participants, collected logs of their conversations and survey responses of their impressions of their dialog partner. we used these to analyze what mental models they formed, how they navigated miscommunications, and the implications these models have for dialog system design. the corpus we collected is publicly available 1 . the main contributions of this work are 1) we provide a novel combination of two data sources: (a) self-reports (quantitative & qualitative data), and (b) behavioral data from the dialogs to gain new insights on users' mental models about an interactive ai dialog partner. 2) we demonstrate that even when users were not engaged in the task, they perceived the ai-dialog partner as intelligent and likable, implying they saw it as a partner separate from the game. 3) we show users were correctly able to identify the system's abilities, despite this they projected human attributes and motivations onto the system, leading to miscommunications 4) we find user attributes, such as age and previous experience, influenced how they interact with the system. 5) we show that despite users having a sound mental model of the dialog system and employing a variety of strategies to resolve miscommunications, they were largely unable to meaningfully resolve 1 https://github.com/digitalphonetics/ our_fault them, implying a need for shared mental models between users and ai systems. ## main study for the main study 129 participants were recruited from amazon mechanical turk, with the same requirements and payment structure. ## dataset in this study, we collected a dialog corpus for investigating mental models and miscommunication in cooperative settings. the dataset, survey responses, analysis, and code are publicly available foot_3 . ## discussion we gained insights into the participants' impressions of the ai system by analyzing self-reports (quantitative & qualitative) and behavior (dialogs). ## future work to reduce miscommunication in human-ai dialogs, it could be interesting to see the impact of more sophisticated methods (e.g., pretrained language models) and if they would increase the ai system's ability to understand users and lead to different user-perceptions. in addition, we recommend reducing the responsibility for users. as our experiment shows, resolving miscommunication in systems like we created was mainly the responsibility of the user as the system was not able to recognize when the user had a poor understanding of the current problem/task. therefore, future research should focus not only on promoting the user to develop correct mental models of the system, but also on how the ai system can develop a correct mental model of the human counterpart in order to promote successful human-ai cooperation. regarding our correlation results, it would be interesting to explore a more nuanced approach to the role of user attributes (i.e., user age, experience in cooperative games) on dialogs with an ai in a cooperative setting. in particular, it would be inter-esting to further explore the role of user age as well as the question of how pre-existing mental models (e.g., gained through prior experience with cooperative games) promote or hinder collaboration with an ai. in addition, a comparison with other collaborative settings could gain new insights into how task-depended or independent the user perception of an ai system is. ## conclusion in this work, we designed a novel collaborative game and dialog system to collect a new corpus and investigate how users perceive an ai dialog partner and what mental models they formed. we found that users perceived their ai dialog partner as intelligent and likeable. almost all users were able to identify what type of information was important to share with their partner, and many also realized this information needed to be provided in a specific way. however, although users perceived the limitations of the dialog system, they tended to overestimate its abilities and attribute human characteristics to it, resulting in miscommunication. users were also willing to share blame when a misunderstanding occurred, indicating they viewed their partner as sophisticated enough to be responsible for mistakes. based on this, we consider it important for successful collaborative dialog systems that a shared (correct) mental model between user and ai system is developed. our game, corpus, and results provide insight into users' mental models as well as miscommunications during a dialog and serve as a tool for other researchers interested in collaborative dialog.