Paper: Utility-based Adaptive Teaching Strategies using Bayesian Theory of Mind (ICLR-2024)

Reviewer comment (Theory):
The decision to frame the learning task as a POMDP lacks clear justification. While this relates to the proposed method, more concrete examples are needed for justification.

Author response:
We acknowledged the reviewer’s concern and included a reference that models the learner’s task as a POMDP following a similar approach, page 3 (modification in red). In a teacher-learner setup, it's crucial for the learner to lack access to specific environmental information for the teacher to be beneficial. Thus, introducing the learner’s task within a POMDP framework is a straightforward and elegant method to illustrate the uncertainty the learner faces. This uncertainty is what the teacher attempts to reduce or alleviate through its demonstrations.

Reviewer comment (Theory):
The task formulation for the ToM-teacher isn't clearly presented in the manuscript. From my perspective, the teaching task resembles a POMDP, akin to references [1, 2]. The teacher doesn't fully know the learner's internal state but tries to optimize the student's performance by providing demonstrations. Unlike earlier works, this paper also seeks to estimate the learner's transition function parameters (goals and observation function). Clarifying the task's formulation is crucial for understanding the technical value of the proposed approach, which remains unclear.

Author response:
We concur with the insightful observation made by the reviewer regarding the potential enhancement of our paper through a more structured formalization of the teacher's task. Recognizing this valuable suggestion, we incorporated to the paper a formalization of the teaching task within the contextual MAB framework (which is a simpler particular case of POMDP) and added corresponding references to the paper (modification in blue). 

In our adapted formalization, the teacher has partial observability over the environment’s state which is in our case the learner’s mental state. This strategic integration aims to clarify and strengthen the conceptual underpinning of the teaching task, facilitating a more comprehensive understanding of the teacher-learner dynamics within our proposed framework, see Section 3.3.2 (modification in blue).

Reviewer comment (Writing):
The paper's objective is unclear. The authors claim, "The goal of this work is to study whether learner-specific teachers who model the learner’s internal state are more efficient than learner-agnostic ones." Given the vague problem definition, it's unclear if this captures the paper's essence. Is the primary takeaway that Bayesian ToM-teachers excel at understanding human internal state changes?

Author response:
We acknowledge the reviewer's concern regarding the paper's unclear formulation and objectives. In response, we have clarified these aspects in the revised introduction (modification in red). 

One of the main focuses of our work was to explore the utility of teachers utilizing a Theory of Mind (ToM) model compared to learner-agnostic baselines. While the superiority of ToM-teachers over learner-agnostic teaching strategies was expected, we were intrigued by the discoveries surrounding partial observation of the learner’s trajectory and the utilization of an approximate model of the learner’s policy. In fact, Section 5.2 results demonstrate that in scenarios with limited data where the teacher only has an approximate understanding of the learner's policy, employing a ToM model can be misleading. In this case, a learner-agnostic teaching strategy which aims to maximize the mean utility across all learners proves to be more useful.

We chose a Bayesian model because Bayesian inference is commonly used to model theory of mind mechanisms, as extensively discussed in our related work on cognitive science. However, since our work is confined to simulated environments and virtual agents, further investigation, particularly involving human users, would be necessary to extrapolate our findings and draw conclusions about humans. Yet, we consider such research beyond the scope of this paper.

Reviewer comment (Evaluation):
The evaluation appears lacking. The paper notes that the teacher has a general idea of the student's policy, and during testing, the student's actions are driven by a basic decision tree. This might mean there's minimal uncertainty in the learner's behavior. It would be beneficial to incorporate human-subject experiments to see real-world impact.

Author response:
We appreciate the reviewer’s interest in the real-world impact of our research. At this stage of our research, our work primarily focuses on modeling teaching interactions and, therefore, centers on machine-to-machine teaching scenarios. 
We intentionally use a known policy for the learner. We decided to work with a decision tree for simplicity as learning a policy on large 2D navigation POMDP with highly sparse reward (such as requiring the learner to grab a key to open a door) is a problem in itself and is out of the scope of this work. 

Nevertheless, we believe that the use of a trained agent would not have enhanced our findings. The aligned ToM-teacher accesses the learner’s true policy, whether trained or handcrafted. The rational ToM-teachers operate under the assumption that the learner—whether following the trained policy or the decision tree—is rational (with a certain degree that can be fitted) in its behavior. One can expect that a properly trained agent will demonstrate to a certain extent rational behavior. Consequently, the uncertainty faced by the rational ToM-teacher regarding the trained learner would resemble the uncertainty it encounters with the decision tree learner. In that sense, we believe that the use of a decision tree learner does not affect the value of our contributions.

**While we acknowledge the significance of additional studies involving human-subject experiments to bridge the gap between simulated and real-world scenarios, we consider this to be beyond the scope of our current work.**

Reviewer comment (Theory):
Based on points from point 1 and equation 3, the teaching policy is essentially a greedy policy. Over time, this might not be the best strategy and could lead to suboptimal results. Without a well-defined teaching task, such a policy is not well justified.

Author response:
We concur again with the reviewer. In the fresh perspective of framing the teaching task as maximizing a reward function within a contextual MAB, the ToM-teacher's strategy aligns with choosing the greedy action (i.e., demonstration) based on an estimation of the MAB reward function (modification in blue).

We are grateful for the references highlighted by the reviewer, which we found highly interesting and pertinent to our work. We have incorporated these papers into our related work section (modifications in green).

[1] Srivastava et al., Assistive Teaching of Motor Control Tasks to Humans, 2022.
https://proceedings.neurips.cc/paper_files/paper/2022/file/b6fa3ed9624c184bd73e435123bd576a-Paper-Conference.pdf

[2] Yu et al., COACH: Cooperative Robot Teaching, 2022.
https://proceedings.mlr.press/v205/yu23b/yu23b.pdf

Reviewer comment (Novelty):
I agree with Reviewer C7a4's comment regarding the absence of related works in the submission. To appropriately position this work in the context of existing literature, some major changes are needed in the manuscript. Therefore, I am maintaining my current rating.

Author response:
We are grateful for the references highlighted by the reviewer, which we found highly interesting and pertinent to our work. We have incorporated these papers into our related work section (modifications in green).

[1] Srivastava et al., Assistive Teaching of Motor Control Tasks to Humans, 2022.
https://proceedings.neurips.cc/paper_files/paper/2022/file/b6fa3ed9624c184bd73e435123bd576a-Paper-Conference.pdf

[2] Yu et al., COACH: Cooperative Robot Teaching, 2022.
https://proceedings.mlr.press/v205/yu23b/yu23b.pdf

Reviewer comment (Theory):
The formalization of the environment is unclear and sometimes misaligned with the literature; for example, the environment is introduced as a POMDP but then formalized (seemingly) as an MDP, as there are no observations.

Author response:
The learners’ environment is framed as a POMDP, yet unlike the conventional formalism, the observation functions are considered internal to the learners rather than to the POMDP. As a result, within the same environment, two learners with varying observation functions will receive different observations from the same environment states. We added in the main paper a reference to a similar work using the same formalization (modification in green).

In the revised version of the paper, we've introduced a teaching contextual Multi-Arms Bandit (which is a simpler case of POMDP) that delineates the teacher’s task from that of the learners’ in Section 3.3.2. We hope that the reviewer will find this revision clarifies our method (modifications in blue).

Reviewer comment (Theory):
The Bayes-adaptive POMDP is mentioned, but the uncertainty seems to be limited to the state (belief only over the state?), which is not a problem solved by the BA-POMDP.

Author response:
We referred to the BA-POMDP framework to motivate the use of a belief over the state of the environment which was introduced as a ‘belief state’ in the paper [1] “The belief state specifies the probability of being in each state given the history of action and observation experienced so far, starting from an initial belief b0.” However, as the learner uncertainty relies solely on the state of the environment, it is true that referring to Bayesian approaches of POMDP is more accurate. We thank the reviewer for this useful remark and removed the BA-POMDP reference from the paper, page 4 (modifications in red).

[1] Stéphane Ross, Brahim Chaib-draa, and Joelle Pineau. Bayes-adaptive POMDPs, NeurIPS, 2007.

Reviewer comment (Writing):
Various important notions are not defined, including the learner's initial belief (which the teacher is trying to affect), making it hard to understand what exactly the demonstrations are accomplishing.

Author response:
The initial belief of the learners is said to be uniform (Section 3.2, page 4, “Unless mentioned otherwise, we assume that the learner’s initial belief bi,j_0 on the state of Mj is uniform over the set of possible states Sj_B .“). 

In the revised version of the paper, the initial uniform belief of the learner, which has not yet interacted with the teacher, is considered to be an observation of the learner’s mental state (which is the hidden state of the teaching contextual MAB). This is defined in Section 3.3.2 page 5 (modification in blue) and we hope that the reviewer finds this formalization clarifies this point.

In the demonstration environment, the teacher offers guidance to familiarize the learner with the new environment. Starting from an initial uniform belief, the  learner follows the teacher's demonstration –which is a sequence of actions– and updates its belief about the state of the new environment from the observations it has accumulated by following the teacher’s demonstration. Subsequently, during the evaluation phase, the learner behaves on its own but starts with the initial belief updated by the teacher’s demonstration rather than an uniform initial belief. This is detailed in the first two paragraphs of Section 3.3.1 pages 4 and 5.

Reviewer comment (Writing):
The data observed by the teacher (tau^obs) does not appear to be defined anywhere.

Author response:
The data observed by the teacher was defined in Section 3.3.2 paragraph 4, page 5, “From a past trajectory τ^obs = {(s_k, a^{obs}_k )}^{K−1} _{k=0} of an unknown learner L on the first environment Mobs [...]”.

We thank the reviewer for highlighting this unclear point. In the new version, the teacher’s observation of the learner is more clearly defined as an observation function of the learner’s internal state, see the new Section 3.3.2 page 5 (modifications in blue).

Reviewer comment (Presentation):
The narrative in which the method is introduced is hard to follow; it is a linear specification in which each component is discussed separately, making it unclear which decisions are problem-specific and which are general.

Author response:
We appreciate the reviewer’s feedback regarding the narrative structure of our method's introduction. We understand the importance of a clear and coherent story to elucidate how each component contributes to both the general method and its problem-specific applications, in our case 2D navigation task.

In the paper, we are already presenting a model of learners with internal states aiming at maximizing a reward function in a GC-POMDP framework. Subsequently, we detail the particularity of such learners in a context of 2D navigation. Now, in the revised version of our work, we follow the same general-to-concrete story for the teacher’s presentation. We added a clearer formulation of the teacher’s task in order to dissociate more clearly the general approach of our method from its application to 2D navigation, see Section 3.3.2 page 5 (modifications in blue).

Reviewer comment (Presentation):
It is unclear whether the belief of the learner being just the (initial?) state distribution is specific to the problem or a general property of the method; a general-to-concrete narrative might have made the contribution clearer.

Author response:
We appreciate the reviewer’s feedback regarding the narrative structure of our method's introduction. We understand the importance of a clear and coherent story to elucidate how each component contributes to both the general method and its problem-specific applications, in our case 2D navigation task.

In the paper, we are already presenting a model of learners with internal states aiming at maximizing a reward function in a GC-POMDP framework. Subsequently, we detail the particularity of such learners in a context of 2D navigation. Now, in the revised version of our work, we follow the same general-to-concrete story for the teacher’s presentation. We added a clearer formulation of the teacher’s task in order to dissociate more clearly the general approach of our method from its application to 2D navigation, see Section 3.3.2 page 5 (modifications in blue).

Reviewer comment (Novelty):
(Bayesian) Theory of Mind for teaching learners does not appear novel enough to warrant publication on its own, as similar work has been done in prior studies [1, 2], even though the exact learner-teacher interaction setup differs.

Author response:
Both references pointed out by the reviewer are relevant to our work and deal with similar problems. 
However, while [1] introduces learners with mental states, the objective of this work differs from ours as it addresses meta-learning issues. In this study, the teacher’s demonstrations aim to optimize the learner’s mental state to enhance its learning abilities. Conversely, in our work, the learner's mental state remains fixed and the teacher's demonstrations are aimed at assisting the learner in maximizing rewards for specific tasks.

The setting in [2] shares similarity in the issues it addresses. In that paper, the teacher’s demonstrations aim to assist the learner in maximizing rewards for specific tasks. Additionally, they explore the utilization of a ToM model to enhance teaching. However, this study focuses on the learner’s ToM model of the teacher, used to modulate the processing of the provided demonstrations whereas we implemented a teacher’s ToM model of the learner used to select the demonstration. Furthermore, while our work concentrates on empirical results in a complex 2D navigation environment, [2] emphasizes theoretical findings in relatively simple theoretical environments.

Finally, we consider the primary contribution of our work to be the exploration of ToM models when dealing with limited observation of learners, rather than solely the introduction of a demonstration. We found particularly interesting the discovery in Section 5.2 that, with limited observation of the learner, a teacher using an approximation of the learner’s policy is less useful than a teacher aiming to maximize the mean utility across all potential learners. This insight suggests that ToM models should be employed only when enough observation of the learners and/or precise knowledge of their policy are available. We precise this point in the introduction and in the formulation of the paper’s goals, pages 1 and 2 (modification in red).

We appreciate the reviewer for highlighting these papers; they indeed provide valuable context for our work. We included reference [2] in our related work section (modification in green), as it closely aligns with our research. However, we determined that reference [1] was further from the scope of our work as it was more related to meta-learning.

[1] Celikok, M. M., Murena, P. A., & Kaski, S. (2020). Teaching to learn: sequential teaching of agents with inner states. arXiv preprint arXiv:2009.06227.
https://arxiv.org/pdf/2009.06227.pdf

[2] Peltola, T., Çelikok, M. M., Daee, P., & Kaski, S. (2019). Machine teaching of active sequential learners. Advances in neural information processing systems, 32.
https://proceedings.neurips.cc/paper_files/paper/2019/file/b98a3773ecf715751d3cf0fb6dcba424-Paper.pdf

Reviewer comment (Writing):
The demonstration is likely a key contribution, but the lack of clarity regarding the proposed method makes it hard to accept the paper in its current form.

Author response:
Both references pointed out by the reviewer are relevant to our work and deal with similar problems. 
However, while [1] introduces learners with mental states, the objective of this work differs from ours as it addresses meta-learning issues. In this study, the teacher’s demonstrations aim to optimize the learner’s mental state to enhance its learning abilities. Conversely, in our work, the learner's mental state remains fixed and the teacher's demonstrations are aimed at assisting the learner in maximizing rewards for specific tasks.

The setting in [2] shares similarity in the issues it addresses. In that paper, the teacher’s demonstrations aim to assist the learner in maximizing rewards for specific tasks. Additionally, they explore the utilization of a ToM model to enhance teaching. However, this study focuses on the learner’s ToM model of the teacher, used to modulate the processing of the provided demonstrations whereas we implemented a teacher’s ToM model of the learner used to select the demonstration. Furthermore, while our work concentrates on empirical results in a complex 2D navigation environment, [2] emphasizes theoretical findings in relatively simple theoretical environments.

Finally, we consider the primary contribution of our work to be the exploration of ToM models when dealing with limited observation of learners, rather than solely the introduction of a demonstration. We found particularly interesting the discovery in Section 5.2 that, with limited observation of the learner, a teacher using an approximation of the learner’s policy is less useful than a teacher aiming to maximize the mean utility across all potential learners. This insight suggests that ToM models should be employed only when enough observation of the learners and/or precise knowledge of their policy are available. We precise this point in the introduction and in the formulation of the paper’s goals, pages 1 and 2 (modification in red).

We appreciate the reviewer for highlighting these papers; they indeed provide valuable context for our work. We included reference [2] in our related work section (modification in green), as it closely aligns with our research. However, we determined that reference [1] was further from the scope of our work as it was more related to meta-learning.

[1] Celikok, M. M., Murena, P. A., & Kaski, S. (2020). Teaching to learn: sequential teaching of agents with inner states. arXiv preprint arXiv:2009.06227.
https://arxiv.org/pdf/2009.06227.pdf

[2] Peltola, T., Çelikok, M. M., Daee, P., & Kaski, S. (2019). Machine teaching of active sequential learners. Advances in neural information processing systems, 32.
https://proceedings.neurips.cc/paper_files/paper/2019/file/b98a3773ecf715751d3cf0fb6dcba424-Paper.pdf

Reviewer comment (Theory):
How exactly do the demonstrations affect the learner's belief?

Author response:
The initial belief of the learners is said to be uniform (Section 3.2, page 4, “Unless mentioned otherwise, we assume that the learner’s initial belief bi,j_0 on the state of Mj is uniform over the set of possible states Sj_B .“). 

In the revised version of the paper, the initial uniform belief of the learner, which has not yet interacted with the teacher, is considered to be an observation of the learner’s mental state (which is the hidden state of the teaching contextual MAB). This is defined in Section 3.3.2 page 5 (modification in blue) and we hope that the reviewer finds this formalization clarifies this point.

In the demonstration environment, the teacher offers guidance to familiarize the learner with the new environment. Starting from an initial uniform belief, the  learner follows the teacher's demonstration –which is a sequence of actions– and updates its belief about the state of the new environment from the observations it has accumulated by following the teacher’s demonstration. Subsequently, during the evaluation phase, the learner behaves on its own but starts with the initial belief updated by the teacher’s demonstration rather than an uniform initial belief. This is detailed in the first two paragraphs of Section 3.3.1 pages 4 and 5.

Reviewer comment (Reproducibility):
What is the observation model of the teacher exactly?

Author response:
The data observed by the teacher was defined in Section 3.3.2 paragraph 4, page 5, “From a past trajectory τ^obs = {(s_k, a^{obs}_k )}^{K−1} _{k=0} of an unknown learner L on the first environment Mobs [...]”.

We thank the reviewer for highlighting this unclear point. In the new version, the teacher’s observation of the learner is more clearly defined as an observation function of the learner’s internal state, see the new Section 3.3.2 page 5 (modifications in blue).

Reviewer comment (Theory):
Generalizability: The proposed ToM technique assumes access to an approximate policy of the student. Also all environments discussed in the paper were deterministic. In practice, while the set of student goals are limited, the policy they follow may be far from ideal and the presence of stochasticity may confuse the teacher further to reach a reasonable belief. Would be great to discuss these limitation in the paper.

Author response:
The reviewer brings up an interesting point. We introduce the rational ToM-teacher, characterized by a stochastic approximate policy of the learner. This policy is further defined by a Boltzmann temperature parameter, lambda, representing the assumed degree of rationality of the learner. A higher temperature parameter implies a noisier rationality in the learner's behavior. Therefore, in theory, with adapted temperature parameters, the rational ToM-teacher can be effective for noisily rational learners

Although our experiments only involve learners following a deterministic shortest path policy, representing perfect rationality with the lowest temperature parameter, we could have employed stochastic, noisily rational learners. In such cases, learning the temperature parameter would have been necessary. We added a precision on that point in Appendix B.3 (modification in red). However, learning a good policy within a complex POMDP, such as large Minigrid environments with sparse rewards (for instance, requiring the key of a specific color to open a door), constitutes a distinct research challenge beyond the scope of this paper.

Reviewer comment (Novelty):
Limited novelty: the main idea of the paper is not novel and similar ideas have been explored to infer agent policies and goals as cited by the authors. The main difference is to expand the inference space to include observation model.

Author response:
We acknowledge the reviewer's concern regarding the novelty of our work. We recognize that the introduction of our objectives may have been misleading. In the revised paper, we have clarified our focus, emphasizing the results obtained in the regime of limited observation of the learner’s behavior and ToM-teachers with an approximate understanding of the learner's policy (modification in red). Additionally, we have incorporated references highlighted by the reviewers to better underscore the novelty of our work (modifications in green). We trust that these modifications will demonstrate the significance of our findings and the novelty of our approach to the reviewer.

Reviewer comment (Theory):
Why demonstrations are action sequence only? Why exclude observations? Is it due to deterministic assumption?

Author response:
The observation function differs between the teacher and the learners. The teacher has full observability while the learners can have limited and different observation functions. The idea is that the same demonstration as a sequence of actions will result in different sequences of observations for two learners with different observation functions (i.e. receptive field sizes). Hence while being exposed to the same demonstration, two learners with different observation functions will acquire different knowledge of the environment. 
The demonstration, presented as a sequence of actions, serves as a method for the teacher to guide the learner in the environment. However, the teacher must infer from prior observation what the learner will perceive while being guided. We added a clarification in Section 3.3.1 page 4 (modifications in red).

Reviewer comment (Novelty):
The paper does not engage with, or obviously contribute to, the large literature on models of teaching in machine learning (or cognitive science). Specifically, the introduction is entirely focused on one qualitative theoretical perspective in the cognitive science literature on teaching, without mentioning the extensive literature on formalizing teaching and cooperation. This literature has grown large and quite mature in recent years, including extensive mathematical and computational theories from a variety of perspectives. Several papers touch on topics close to the ideas here, including sequential teaching under perturbations on belief and policy, teaching as a POMDP, and proofs of robustness of standard ToM reasoning in cooperative settings. The results are not surprising given what we already know, but the paper does not make contact with these results nor cite many of the relevant papers.

Author response:
Before preparing our response to all reviewers, we would be glad if the reviewer could help us better understand the weaknesses of our work by pointing to some concrete instances of the relevant literature that we missed and that the reviewer seems to have in mind. The reviewer says that we do not mention “the extensive literature on formalizing teaching and cooperation”, they say that “This literature has grown large and quite mature in recent years, including extensive mathematical and computational theories from a variety of perspectives.” and even say that “several papers touch on topics that are quite close to the ideas here including sequential teaching under perturbations on belief and policy, teaching as a POMDP, proofs of robustness of standard ToM reasoning in cooperative settings, etc.”, but doesn’t give even a single instance of such papers that we missed.

Further in the review, the reviewer says “In recent years, there have been several related NeurIPS papers, searching for ‘teacher’ or ‘teaching’ or ‘cooperation’. There are older papers formalizing teaching as a POMDP problem. There are also newer papers on inferring beliefs of agents.”. Again, mentioning a few instances of such papers that the reviewer may have in mind would help a lot.
Finally, the reviewer says “I would strongly recommend that the authors review the last few years of NeurIPS (also ICML) papers for topics such as ‘teacher’, ‘teaching’, ‘cooperation’ and ‘cooperative’. Not all papers with those words will be related, but one will find quite a lot. Specifically, there are relatively theoretical papers that outline a mathematical framework and imply the results here that appeared in NeurIPS and ICML. There are also papers in NeurIPS that tested human experiments, which necessarily have errors in beliefs. Similarly, please read the literature related to RSA (the Goodman and Frank paper cited) which have a number of interesting models and results. Please also search broadly for teaching and POMDPs as there is related work in that direction also. The current work will benefit from reconceptualization in light of these works.”

Again, the reviewer seems to have some concrete papers in mind, but does not mention them. So we would be extremely grateful to the reviewer if they could provide at least 2 or 3 of the most relevant papers they have in mind, so that we can enrich our perspective with these works if they are actually relevant, or eventually better delineate our scope if we think they are not.

In particular, as the reviewer mentions "cooperation" and "cooperative", we would like to clarify that our work is not much related to the abundant literature about cooperation in multiagent systems, where indeed one can find POMDP-based formalizations and Bayesian inference mechanisms, but teacher-learner interactions are not considered. Rather, our work is inspired by cognitive sciences, and more specifically by recent works on inferential social learning (ISL) (Gweon et al., 2021), which argues that humans learn from evidence generated by others and generate useful evidence to help others learn.

Reviewer comment (Presentation):
The introduction is poorly structured to help readers understand the literature. There has been extensive work before and after the Gweon et al. paper that explores this concept.

Author response:
Our main focus was indeed to leverage the fine-grained understanding of cognitive science research on the human mechanisms, such as ToM, used to transfer information between human individuals. We acknowledge that our work relied on existing modeling of ToM mechanisms, on one hand, and teaching, on the other hand, and we referred to this existing literature in our related work and added the references highlighted by the reviewers (modification in green). 

However, our aim was to provide empirical evidence that demonstrates how employing a ToM model improves the effectiveness of a machine teacher for a machine learner. This differs from existing work, which primarily concentrates on using a ToM model to enhance the learners themselves. Furthermore, our aim was to offer empirical results regarding the impact of an inaccurate prior held by the teacher on the learner's policy, on the ToM teacher’s utility. We clarified this point in the related work and introduction in which we added some of the references pointed by the reviewer (modification in green).

Reviewer comment (Novelty):
The introduction is almost exclusively focused on empirical research. However, there has been extensive work developing models. The contribution of this work in relation to that literature is unclear.

Author response:
Our main focus was indeed to leverage the fine-grained understanding of cognitive science research on the human mechanisms, such as ToM, used to transfer information between human individuals. We acknowledge that our work relied on existing modeling of ToM mechanisms, on one hand, and teaching, on the other hand, and we referred to this existing literature in our related work and added the references highlighted by the reviewers (modification in green). 

However, our aim was to provide empirical evidence that demonstrates how employing a ToM model improves the effectiveness of a machine teacher for a machine learner. This differs from existing work, which primarily concentrates on using a ToM model to enhance the learners themselves. Furthermore, our aim was to offer empirical results regarding the impact of an inaccurate prior held by the teacher on the learner's policy, on the ToM teacher’s utility. We clarified this point in the related work and introduction in which we added some of the references pointed by the reviewer (modification in green).

Reviewer comment (Writing):
The sentence "we explore the limitations of ToM models not being able to recover the learner actual internal state from its behaviour" is awkwardly phrased.

Author response:
We appreciate the reviewer's feedback on the stylistic aspect and have updated the sentence accordingly in the new version (modifications in red).

Reviewer comment (Novelty):
A large number of related papers are omitted. In recent years, several related NeurIPS papers exist (search for teacher, teaching, or cooperation). There are older papers formalizing teaching as a POMDP problem, as well as newer papers on inferring beliefs of agents.

Author response:
Before preparing our response to all reviewers, we would be glad if the reviewer could help us better understand the weaknesses of our work by pointing to some concrete instances of the relevant literature that we missed and that the reviewer seems to have in mind. The reviewer says that we do not mention “the extensive literature on formalizing teaching and cooperation”, they say that “This literature has grown large and quite mature in recent years, including extensive mathematical and computational theories from a variety of perspectives.” and even say that “several papers touch on topics that are quite close to the ideas here including sequential teaching under perturbations on belief and policy, teaching as a POMDP, proofs of robustness of standard ToM reasoning in cooperative settings, etc.”, but doesn’t give even a single instance of such papers that we missed.

Further in the review, the reviewer says “In recent years, there have been several related NeurIPS papers, searching for ‘teacher’ or ‘teaching’ or ‘cooperation’. There are older papers formalizing teaching as a POMDP problem. There are also newer papers on inferring beliefs of agents.”. Again, mentioning a few instances of such papers that the reviewer may have in mind would help a lot.
Finally, the reviewer says “I would strongly recommend that the authors review the last few years of NeurIPS (also ICML) papers for topics such as ‘teacher’, ‘teaching’, ‘cooperation’ and ‘cooperative’. Not all papers with those words will be related, but one will find quite a lot. Specifically, there are relatively theoretical papers that outline a mathematical framework and imply the results here that appeared in NeurIPS and ICML. There are also papers in NeurIPS that tested human experiments, which necessarily have errors in beliefs. Similarly, please read the literature related to RSA (the Goodman and Frank paper cited) which have a number of interesting models and results. Please also search broadly for teaching and POMDPs as there is related work in that direction also. The current work will benefit from reconceptualization in light of these works.”

Again, the reviewer seems to have some concrete papers in mind, but does not mention them. So we would be extremely grateful to the reviewer if they could provide at least 2 or 3 of the most relevant papers they have in mind, so that we can enrich our perspective with these works if they are actually relevant, or eventually better delineate our scope if we think they are not.

In particular, as the reviewer mentions "cooperation" and "cooperative", we would like to clarify that our work is not much related to the abundant literature about cooperation in multiagent systems, where indeed one can find POMDP-based formalizations and Bayesian inference mechanisms, but teacher-learner interactions are not considered. Rather, our work is inspired by cognitive sciences, and more specifically by recent works on inferential social learning (ISL) (Gweon et al., 2021), which argues that humans learn from evidence generated by others and generate useful evidence to help others learn.

Reviewer comment (Novelty):
It is notable that neither the introduction nor the related work discuss POMDPs.

Author response:
We opted to concentrate on the cognitive science literature, particularly exploring how theory of mind has been examined and modeled, with a specific emphasis on Bayesian inference models.

However, we acknowledge the relevance of the literature on POMDPs suggested by other reviewers, and as such, we have incorporated the suggested works into the revised version of the paper (modification in green).

Reviewer comment (Novelty):
The statement "we introduce a teacher equipped with a Theory of Mind (ToM) model that we refer to as ToM-teacher" should acknowledge that most papers have a ToM component.

Author response:
Our objective was to explicitly model the ToM employed by the teacher to instruct the learner. While there are numerous models demonstrating ToM capacities and incorporating ToM components,  to our knowledge, in most of the literature, the ToM model is defined from the learner toward the teacher and is used to modulate how the learner processes the information provided by the teacher. In fact, none of them explicitly defines a model of ToM from the teacher to the learner and shows how this model can be leveraged to modulate teaching in itself. While the current literature focuses on how ToM is used to learn, Gweon research and our work focus on how ToM is used to help others learn.

Reviewer comment (Theory):
Given the introduction’s focus on cognitive science, it is surprising to see decision trees and A* used as part of the learner. There is no explanation for why these would be considered reasonable models of humans.

Author response:
We do not claim that decision trees and A* algorithms are reasonable models of humans. However, we intentionally use a known policy for the learner in order to analyze the impact of inaccurate prior on the learner’s policy. We specifically use a decision tree, due to the complexity of learning in a large 2D navigation POMDP with sparse rewards—a challenge beyond this work's scope. 

We could have used a trained agent. No matter if a trained or handcrafted policy is used, the ToM-teacher assumes the learner behaves rationally. We anticipate a properly trained agent to exhibit rational behavior, creating similar uncertainty for the ToM-teacher, whether the learner follows a trained policy or a decision tree. Thus, we believe that employing a decision tree learner does not impact the value of our contributions.

Reviewer comment (Novelty):
The authors are strongly recommended to review the last few years of NeurIPS (and ICML) papers on topics such as "teacher", "teaching", "cooperation", and "cooperative". While not all papers with those keywords will be related, many will be relevant. Some theoretical papers outline a mathematical framework and imply results similar to those here, and some NeurIPS papers test human experiments with errors in beliefs. The authors should also read the literature related to RSA (e.g., the Goodman and Frank paper cited), which contains interesting models and results, and broadly search for teaching and POMDPs as there is related work there as well. The current work would benefit from being reconceptualized in light of these works.

Author response:
Before preparing our response to all reviewers, we would be glad if the reviewer could help us better understand the weaknesses of our work by pointing to some concrete instances of the relevant literature that we missed and that the reviewer seems to have in mind. The reviewer says that we do not mention “the extensive literature on formalizing teaching and cooperation”, they say that “This literature has grown large and quite mature in recent years, including extensive mathematical and computational theories from a variety of perspectives.” and even say that “several papers touch on topics that are quite close to the ideas here including sequential teaching under perturbations on belief and policy, teaching as a POMDP, proofs of robustness of standard ToM reasoning in cooperative settings, etc.”, but doesn’t give even a single instance of such papers that we missed.

Further in the review, the reviewer says “In recent years, there have been several related NeurIPS papers, searching for ‘teacher’ or ‘teaching’ or ‘cooperation’. There are older papers formalizing teaching as a POMDP problem. There are also newer papers on inferring beliefs of agents.”. Again, mentioning a few instances of such papers that the reviewer may have in mind would help a lot.
Finally, the reviewer says “I would strongly recommend that the authors review the last few years of NeurIPS (also ICML) papers for topics such as ‘teacher’, ‘teaching’, ‘cooperation’ and ‘cooperative’. Not all papers with those words will be related, but one will find quite a lot. Specifically, there are relatively theoretical papers that outline a mathematical framework and imply the results here that appeared in NeurIPS and ICML. There are also papers in NeurIPS that tested human experiments, which necessarily have errors in beliefs. Similarly, please read the literature related to RSA (the Goodman and Frank paper cited) which have a number of interesting models and results. Please also search broadly for teaching and POMDPs as there is related work in that direction also. The current work will benefit from reconceptualization in light of these works.”

Again, the reviewer seems to have some concrete papers in mind, but does not mention them. So we would be extremely grateful to the reviewer if they could provide at least 2 or 3 of the most relevant papers they have in mind, so that we can enrich our perspective with these works if they are actually relevant, or eventually better delineate our scope if we think they are not.

In particular, as the reviewer mentions "cooperation" and "cooperative", we would like to clarify that our work is not much related to the abundant literature about cooperation in multiagent systems, where indeed one can find POMDP-based formalizations and Bayesian inference mechanisms, but teacher-learner interactions are not considered. Rather, our work is inspired by cognitive sciences, and more specifically by recent works on inferential social learning (ISL) (Gweon et al., 2021), which argues that humans learn from evidence generated by others and generate useful evidence to help others learn.

Reviewer comment (Experiments):
It's not clear why the method requires two learning environments - a simple one for the teacher to interact with the learner and gain knowledge of the student's mental state, and another more complex environment where the teacher performs real teaching. Is it possible to unify these two and let the teacher teach on the fly as it interacts with the student?

Author response:
Teaching 'on the fly' while interacting with the learner is possible but still requires sequences of observation, demonstration selection, and evaluation. We decided to split the framework into two environments to separate the Theory of Mind (ToM) modeling (teacher modeling the learner’s internal state) from the teaching (teacher leveraging its model of ToM to provide a demonstration).

Reviewer comment (Evaluation):
If there are significant distinctions between the simple and complex environments, how does it affect the teaching process? Is the knowledge learned by the teacher in the simple environment going to become less effective?

Author response:
The first environment’s complexity can vary, and in additional experiments which we included in Appendix G of the revised version, we show that actually performing the observation stage on a simpler environment causes more ambiguity in the learner’s behavior, causing the teacher to have more difficulty to identify the learner’s goals and beliefs. Indeed, since the learner can have different widths of sensory capacity, being observed in a larger, more complex environment can actually help better identify its sensors. 
However, if the observation environment was overly complex for the learner to engage with objects, its behavior would not be indicative enough of its goal, making inferring internal states impossible. Hence, the need for a reasonably simple environment in which all the learners can interact with the objects without the help of the teacher.

On the other hand, the second environment, where the teacher conducts teaching, needs to be complex enough for some learners to require the teacher’s assistance to achieve the task. If learners can easily accomplish tasks without the teacher's help, providing an empty demonstration would be an optimal solution: no teaching would be needed. Hence, the need for a ‘complex’ environment. Finally, note that a learner’s internal state (goal and observation function) remains consistent across different environments and is independent from the complexity of those environments. Consequently, the insights gained by the teacher during the observation phase can be leveraged for teaching in any environment, regardless of its complexity.
We added in Appendix G of the revised version of the paper a precision on this point discussing the influence of the observation environment size (modifications in orange).

Reviewer comment (Theory):
The teacher requires full knowledge of the underlying POMDP. This is really unrealistic unless in very specific domains. This requirement limits the applicability of the proposed method.

Author response:
Indeed, we assume the teacher's access to the POMDP. Specifically, our method requires the teacher to predict a learner's reward in the environment based on a given policy conditioned by a particular belief. In our method, this involves simulating trajectories within the environment, which relies on the teacher's knowledge of the underlying POMDP. However, for reward prediction, the teacher can use a pre-trained model. We could consider a model trained to predict the reward of a learner following a particular belief-conditioned policy (the teacher’s approximation of the learner’s policy), therefore taking a belief as input. Yet, this would require access to a large amount of trajectories of such learners with various beliefs following the teacher’s approximate policy. 

We chose to focus on uncertainty on the learner’s mental state rather than on the POMDP from the teacher’s point of view. We acknowledge the interest of considering teachers with partial observability but we consider that this study is out of this work’s scope.

Reviewer comment (Novelty):
The idea of the paper does not seem novel to me, and the results are not surprising at all. Of course, if the teacher learned prior for the student's mental state aligns with the real mental state, then it's going to teach better. It would be much more interesting if the authors can provide theoretical justification of the proposed method through some Bayesian inference theory.

Author response:
We understand the reviewer’s concern about the non-surprising nature of the results when interpreted this way. It is definitely obvious that an omniscient teacher will be best, and that an aligned teacher, as defined in the paper, will perform second best. Indeed, these teachers possess knowledge that would not be accessible in a realistic scenario. 

The core of our results, which in our opinion are more interesting and surprising, are related to the rational and non-adaptive teachers. These teachers are based upon more realistic assumptions about the learner, and we showcase how being rational is superior when having access to full trajectories of the learner, and inferior to non-adaptive teachers when having access to limited trajectories. It quantifies the right amount of information needed for the teacher to provide accurate adaptive teaching to the learner, which we believe is a hard problem in teacher-learner scenarios.
 
We hope this clarification can help the reviewer understand the relevance of our results. We clarified this positioning in the introduction in the formulation of the paper’s objectives (modification in red).