Limits of Theory of Mind Modelling in Dialogue-Based Collaborative Plan Acquisition (2024)

Abstract: Recent work on dialogue-based collaborative plan acquisition (CPA) has suggested that Theory of Mind (ToM) modelling can improve missing knowledge prediction in settings with asymmetric skill-sets and knowledge. Although ToM was claimed to be important for effective collaboration, its real impact on this novel task remains under-explored. By representing plans as graphs and by exploiting task-specific constraints we show that, as performance on CPA nearly doubles when predicting one’s own missing knowledge, the improvements due to ToM modelling diminish. This phenomenon persists even when evaluating existing baseline methods. To better understand the relevance of ToM for CPA, we report a principled performance comparison of models with and without ToM features. Results across different models and ablations consistently suggest that learned ToM features are indeed more likely to reflect latent patterns in the data with no perceivable link to ToM. This finding calls for a deeper understanding of the role of ToM in CPA and beyond, as well as new methods for modelling and evaluating mental states in computational collaborative agents.

## introduction dialogue-based human-ai collaboration is an interaction in which humans and artificial intelligent (ai) agents converse to achieve a shared goal or task @xcite . when humans collaborate with each other, they rely on two main abilities: verbal communication and theory of mind (tom), i.e. the ability to infer one's own and others' mental states @xcite . to succeed in collaborating with humans, it is therefore imperative for ai agents to possess similar capabilities @xcite . ## problem formulation mindcraft @xcite was introduced as a multi-modal benchmark for studying tom modelling within collaborative tasks. it involves two players collaborating through dialogue in a 3d block world to craft a target material by manipulating blocks using specific tools (see figure 3 , left). players initially receive a partial plan as an incomplete directed and-graph and a tool allowing each to interact with a set of specific blocks. endowed with complementary knowledge and skill sets, playseq. model c a n d i d a t e s a m p l i n g decoder cnn gnn bert {own, partner} missing knowledge tom features dialogue moves dialogue video plan seq. model cnn gnn bert dialogue moves task {status, knowledge, intention} theory of mind modelling collaborative plan acquisition mlp figure 2: models' architecture for theory of mind (tom) modelling and collaborative plan acquisition (cpa). following (bara et al., 2021, 2023), we train one model for each tom task (status, knowledge, intention) and cpa task (own missing knowledge, partner's missing knowledge) and freeze the bert weights during training (indicated by ) for a fair comparison with the baseline. definition 1 consider a joint plan as a directed and-graph @xmath0, e), where the nodes v denote (sub-)goal materials, and edges e denote temporal constraints between the sub-goals. in a collaborative plan acquisition problem, two agents i and j start with partial plans p @xmath1, e i ), e i ⊆ e, and p @xmath2, e j ), e j ⊆ e. given a sequence of visual observations o t i and a joint dialogue history d t at time t, agent i has to infer their own missing knowledge ēi = e \e i and the partner j's missing knowledge ēj = e \ e j . tom and cpa tasks are closely related but fundamentally different: tom tasks focus on exploring players' beliefs about the game state and their partner's mental states, while cpa tasks involve predicting missing information from players' partial plans. additional details are in §a.1 and §a.2. ## limits and future directions for neural theory of mind a key insight of our work is that current approaches for cpa, rather than learning tom, seem to merely exploit latent correlations in the data that have little to do with mental states. this is highlighted by the lack of impact of the proposed tom features on cpa, as shown in instead, recent work proposed the use of openended environments to study tom with the aim of observing whether these capabilities emerge through interactions with other agents @xcite . minecraft represents a good candidate environment for multi-agent collaboration in an open world. however, the way @xcite frame mindcraft is still limited to specific tasks and requires extensive data collection efforts. one possible solution could be to transform mindcraft into a reinforcement learning environment with a focus on less constrained collaborative tasks. while @xcite suggest modelling tom as a supervised learning task, the way humans acquire tom is more nuanced and largely unsupervised @xcite . we believe that the development of open-ended environments combined with learning tom capabilities in an unsupervised, human-like manner is a more promising direction for future research. tom capabilities are deeply linked to language acquisition @xcite . in the context of dialogue-based collaboration, another interesting future direction could be to learn tom from generation instead of classification @xcite . current approaches could be further improved by building a more general and robust world model, e.g., by leveraging a pre-trained language or videolanguage model as a more general prior. finally, in addition to developing suitable environments and learning algorithms, effective and interpretable methods to evaluate whether agents have truly learned tom will be crucial. we see three exciting directions in this regard: probing @xcite , mechanistic interpretability @xcite , and concept learning @xcite . the work of oguntola et al. serves as an inspiring example, where agents learn human-interpretable concepts that represent beliefs about other agents in a simple multi-agent reinforcement learning setting. ## conclusion in this work, we demonstrated that applying taskspecific constraints to plan graphs reduces significantly the performance gap between predicting omk and pmk in mindcraft. at the same time, improvements from tom modelling diminish, raising concerns about current approaches. our ex-periments and analyses consistently suggest that current tom modelling approaches learn features that likely reflect latent patterns in the data, with no perceivable link to tom. this finding calls for a deeper understanding of the role of tom in cpa and beyond, as well as for new methods to model and evaluate mental states in collaborative agents.