COACH: Cooperative Robot Teaching
Abstract: Knowledge and skills can transfer from human teachers to human students.
However, such direct transfer is often not scalable for physical tasks, as they require one-to-one interaction, and human teachers are not available in sufﬁcient numbers.
Machine learning enables robots to become experts and play the role of teachers to help in this situation.
In this work, we formalize cooperative robot teaching as a Markov game, consisting of four key elements: the target task, the student model, the teacher model, and the interactive teaching-learning process.
Under a moderate assumption, the Markov game reduces to a partially observable
Markov decision process, with an efﬁcient approximate solution.
We illustrate our approach on two cooperative tasks, one in a simulated video game and one with a real robot.
Introduction
How do we teach humans to re-orientate a table jointly or play tennis?
Humans often learn by practicing the skills with partners or teachers [1, 2, 3].
This mode of learning is, however, difﬁcult to scale up, as it requires one-to-one interaction and there are not sufﬁcient human teachers [4].
With advances in machine learning, now robots can not only master complex tasks [5, 6, 7], but also collaborate with humans and adapt to human behaviors [8, 9, 10].
In this work, we aim to create robot teachers for physical tasks, thus scaling up teaching and providing learning opportunities to many even when human teachers are not available.
Speciﬁcally, we propose Cooperative RObot TeACHing (COACH), a robot teaching framework to teach humans cooperative skills for two-player physical tasks through interaction.
See Fig.
1 for an illustration.
COACH treats the teaching task as a two-player Markov game over a target task .
One player is the robot teacher, and the other is the human student.
Under a suitable student learning model, COACH transforms the game into a partially observable Markov decision process (POMDP).
The POMDP solution enables the robot teacher to adapt to the different behaviors, according to the history of interactions.
One key challenge of COACH is to represent the student’s knowledge of the target skills and the learning behaviors.
First, we leverage item response theory (IRT), a well-established framework for educational assessment [11].
IRT provides simpliﬁed parametric models that capture the student’s knowledge level with respect to the task difﬁculty in a small number of parameters.
COACH treats these parameters as latent variables in the teaching POMDP and learns them from human-robot interaction data by solving the POMDP.
Next, to teach complex skills, we draw insights from student-centered learning [12] and humanrobot cross-training [13]; we decompose a complex target skill into a set of sub-skills, based on the student’s potential roles in the target task.
With this compact and decomposed skill representation, we can naturally obtain a partially assistive robot teaching curriculum, in which the human learns the sub-skills one at a time, and the robot assists with the sub-skills not yet learned, to complete the task.
While the robot assists the human in the teaching task, its behavior differs from those in common collaborative human-robot interaction settings [14, 15].
There, the primary objective is to complete the task, and the robot is fully assistive: if the human does not perform, the robot then tries to complete the task on its own, if possible.
In the teaching task, the robot is partially assistive
[Figure]
and usually avoids assisting with the speciﬁc sub-skill to be learned, in order to encourage student exploration and learning.
To conclude, this work introduces Cooperative RObot TeACHing (COACH), a formal model for a robot to teach a human in a cooperative task.
The framework assumes the robot teacher has full knowledge about the task, e.g. a set of optimal policies.
As a ﬁrst attempt, we demonstrate an instantiation of solutions to it.
In the approach, we assume the skill can be decomposed into independent sub-skills and the performance of the optimal policy is known.
We conducted human-subject experiments on two challenging human-robot collaboration tasks, Overcooked-AI and Cooperative Ball
Maze (shown in Fig. 2), to show the effectiveness of COACH.
Our results show that COACH allows us to model and reason over adaptive humans in cooperative teaching.
In addition, we ﬁnd out that a fully-assistive teacher may impede humans’ acquisition of skills, and a partially assistive interaction mode indeed motivates students to explore new strategies.
Related Work
Assistance in HRI.
One major aspect of HRI is how the robot could assist humans with a hidden human objective [14, 15].
The objective of the robot is to infer the human’s intention and learns to assist the human.
In its simplest form, the action selection and human intention inference are separated [16, 17, 18].
A decision-theoretic framework, assistant POMDP, is developed to capture the general notion of assistance in HRI [19].
The robot integrates the reward learning and control modules to perform sophisticated reasoning over human feedback [20, 21].
However, these two approaches neglect human learning/adaptation and may hinder humans from improving their skills.
Our work focuses on how to generate behaviors that facilitate human learning during interactions.
Collaboration in HRI.
Another important aspect of HRI is to model interactions as the collaboration between the human and the robot [22], for which the human and the robot share the same objective.
However, the joint optimal policy, e.g. rotating the table counter-clockwise, is unknown to both agents in the ﬁrst place.
Their interaction is mutually adaptative [23, 24, 25].
Particularly, as pointed out in [10], if one side is only aware of partial information about the task, the optimal policy pair naturally induces the behavior of active teaching, active learning, and efﬁcient communication between the robot and human.
In this work, we focus on the following setting: given that the robot teacher knows the optimal policy, how to carry out active teaching.
Teaching Algorithm for Algorithms.
Teaching for algorithms aims to facilitate the learning of the algorithm by choosing or generating training samples.
Various teaching techniques including curriculum learning [26] and machine teaching [27, 28, 29, 30, 31] have been effectively applied to supervised learning and semi-supervised learning problems.
Similar ideas are further extended to train reinforcement learning agents to learn complex skills, e.g., generate training environment for reinforcement learning [32, 33, 34], choose various demonstrations [35] or learn to decompose the skill [36, 37].
Teaching in cooperative multi-agent RL allows agents to simultaneously become teachers and students for each other [3, 38, 39].
However, such approaches generally require relatively more data for training and to some extent the controlled learning behavior of the learner.
Transfer of these approaches to human learning is promising but difﬁcult.
Teaching Algorithm for Human.
Despite the aforementioned practical challenges, some algo- rithms have been successfully deployed for human learning.
Attempts on teaching the crowd on classiﬁcation or concepts prove to be successful [40, 41, 42, 43].
While humans can learn concepts from visual or verbal examples, complex skills like motor control skills can hardly be mastered through these signals.
Recently, skill discovery techniques in reinforcement learning have been introduced to generate a curriculum based on skill decomposition and facilitate humans to learn motor control skills [44].
It focuses on how to adaptively decompose the skill into learnable sub-skills for a human to practice on its own and achieves promising results.
Here, we seek to automate the teaching process for humans to cooperate in a physical task and provide a framework for this teaching mode, e.g., table co-reorientation.
Cooperative Robot Teaching
We formalize Cooperative Robot Teaching by identifying four key elements: (1) the target task, (2) the student learning model, (3) the teacher model, and (4) the interactive teaching-learning process.
The target task.
In this paper, we focus on teaching in a duo cooperative task and we call it the
Target Task , which is the original cooperative game both agents aim to solve.
Deﬁnition 1 .
The target task is a duo player cooperative Markov game between two agents, 1 and 2 , that can be described by a tuple, M =
(cid:104)S , { A 1 , A 2 } , T , R, γ (cid:105) with the following deﬁnitions:
S a set of target task states: s ∈ S .
A 1 a set of actions for the agent 1 : a 1 ∈ A 1 .
A 2 a set of actions for the agent 2 : a 2 ∈
A 2 .
T a conditional distribution on the next target task state, given the previous state and the actions of both agents: T ( s (cid:48) | s, a 1 , a 2 ) .
R a target task reward function that maps target task states and players’ actions to real numbers.
R : S ×
A 1 ×
A 2 → R .
γ the discount factor in the target task.
At each step, 1 and 2 both observe the current task state s t , then select their actions a 1 t ∼ π 1 and a 2 t ∼ π 2 respectively.
π 1 and π 2 refers to the policy of agent 1 and 2 .
They will receive a joint reward r t =
R ( s t , a 1 t , a 2 t ) .
The next state is updated as s t +1 ∼ P T ( s t +1 | s t , a 1 t , a 2 t ) .
Given the deﬁnition of the target task, we ﬁrst answer how to represent the knowledge/skills.
In this work, we choose to represent a skill by the optimal policy π ∗ to the target task.
The optimal policy maximizes the expected cumulative reward when the student is cooperating with a given partner with policy π .
Without loss of generality, we assume the ego-agent is agent 2 .
We deﬁne the optimal policy as π ∗ = arg max π 2 ∈ Π E a 1 t ∼ π, a 2 t ∼ π[ (cid:80) ∞ t =
γ t R ( s t , a 1 t , a 2 t )] .
For example, in the table coreorientation task, the agent needs to learn to deal with either stubborn or adaptive partners.
We recognize that there are other ways to represent knowledge/skills, such as a set of demonstrations and the ground-truth reward function.
However, such representations are indirectly linked with the skill’s performance; therefore, evaluating its proﬁciency is more obscured.
We choose the optimal policy as the representation since it can be directly optimized over and evaluated.
Student.
The student policy is non-stationary since it will improve along with teaching.
We model this evolutionary behavior with a tuple of initial policy and an updating function (cid:104) π S 0 , U(cid:105) .
The student policy represents the student knowledge state.
The updating function U models how the student changes its policy after each teaching step.
Teacher.
We deﬁne the teacher as a knowledgeable agent (expert) who knows a set of optimal policies Π ∗ for a target task.
The teacher aims to acquire a teaching policy π T that can teach any π ∗
i ∈
Π ∗ to the student effectively.
In this general setting, the choice of the optimal student policy π ∗ i depends on the capability, preference, and current knowledge level of the student.
A principled approach to selecting the optimal student policy needs to consider the student’s preference, his/her update model for the knowledge level, and an estimate of his/her current capability, which requires insight beyond our work.
In this paper, we assume that we have an oracle to choose the optimal student policy π ∗
∈
Π ∗ to teach, such that this policy π ∗ matches the preference of the student.
The teacher can be described by a tuple of a student’s optimal policy and the corresponding teaching policy, (cid:104) π ∗ , π T (cid:105) .
The interactive teaching-learning .
We now refer to the robot as the teacher, T and the human as the student, S.
In the target task, T knows the task’s optimal policy π ∗ while the student does not.
The goal of T is to act in the most informative way so that S learns π ∗ fastest.
The choice of π ∗ should account for the student’s preferences.
To embed the objective of teaching and distinguish it from the Target Task , we deﬁne it as the Teaching Task in the following way:
Deﬁnition 2 .
The teaching task is a POMDP M (cid:48) for the teacher given the target task
M = (cid:104)S , { A 1 , A 2 } , T , R, γ (cid:105) , the student (cid:104) π S 0 , U(cid:105) , and the optimal policy to teach π ∗ .
It can be described by a tuple M (cid:48)
=
(cid:104) ˆ S , ˆ A, ˆ T , ˆ O , ˆ Z, ˆ R, ˆ γ (cid:105) with the following deﬁnitions:
ˆ S a set of teaching task state: for ˆ s ∈ ˆ S , ˆ s = { s, π S } .
ˆ A a set of actions:
ˆ A =
A 1 × A 2 .
ˆ T a conditional distribution on the next teaching task state:
ˆ T (ˆ s (cid:48) | ˆ s, ˆ a ) .
ˆ O a set of observations: for ˆ o ∈
ˆ O , ˆ o = { a S , s, r } .
ˆ Z the observation model: ˆ Z (ˆ o | ˆ a, ˆ s ) .
ˆ R the teaching task reward function that measures the effectiveness of teaching.
ˆ R : ˆ S × ˆ A × ˆ S
→ R .
ˆ γ the discount factor in the teaching task.
The objective of the teaching task is to derive a teaching policy π T , enabling students to learn π ∗ for the target task fastest.
More speciﬁcally, the teacher can inﬂuence the student through interactive actions ˆ a ∈ ˆ A .
Given the student initial policy π S 0 and the update function U , the goal of the
Teaching Task is to ﬁnd a teaching policy π T that allows π S
→ π ∗ as fast as possible.
Next, we introduce our choice of the teaching policy π T , the update function U , and the reward function ˆ R .
To devise a student-aware teaching strategy, apart from the current state s t and the target policy π ∗ , our π T also takes the history of observation as input.
The history of observation is h t =
[( s 0 , r 0 , ˆ a 0 , a S 0 ) , ..., ( s t , r t , ˆ a t , a St )] .
The action of the teacher then can be sampled from the policy, i.e., ˆ a t ∼ π T (ˆ a t | h t − 1 , s t , π ∗ ) .
S updates π S with any arbitrary iterative functions conditioned on the history of interactive actions and the joint rewards in the following form: π S t +1 =
U ( π S t , h t ) .
Moreover, to incentivize the teacher to speed up the teaching process, we introduce a step-wise teaching cost to the teacher c t =
C ( s t , ˆ a t ) to penalize unnecessary teaching actions.
To this end, we deﬁne the reward function as
[Equation]
where ω is the weighted factor to trade-off the teaching cost and teaching efﬁciency, and D can be an arbitrary distance measure between two policies, e.g., initial state value in the target task.
The solution to the POMDP M (cid:48) is a teaching policy π T that maximizes the expected sum of rewards
E ˆ a t ∼ π T [ (cid:80) ∞ t =
ˆ γ t
ˆ R (ˆ s, ˆ a t , ˆ s (cid:48) )] .
Method
Require: Maximum Interactions L , Predeﬁned Interactions N 1: for k ∈
ˆ A do
[Table]
In this section, we provide a solution that grounds all the ingredients in the conceptual framework of COACH.
The main spirit of our solution is to parameterize students’ knowledge state with
IRT and decompose complex tasks into a set of role-based independent skills.
This enables easier evaluation of the students’ proﬁciencies and provides a ground to derive the partially assistive interaction mode.
To begin with, we ﬁrst deﬁne the action space, ˆ A .
4.1 Actions
The action space is constructed through sub-skill decomposition.
Sub-skills decomposition is wellstudied for single-agent tasks [45, 46, 47].
However, extending the same idea to a multi-agent setting is still challenging since task completion relies on the interaction among multiple parties.
We observe that in a multi-agent game, the task naturally comprises several roles, of which each agent takes a subset.
The well-established leader-follower model is a particular choice of role-based skill decomposition [48, 49, 50, 51].
Therefore in our work, we propose to decompose skills based on role allocation.
We divide the skill into K locally-independent teachable sub-skills according to the student’s potential roles in the task.
The teacher’s action space ˆ A =
{ k : k ∈ Z , 0 ≤ k < K } consists of teaching each sub-skill.
Such a decomposition of skills naturally leads to a partially assistive mode of interaction.
4.2 States
The state space is constructed with Item Response Theory (IRT) [11].
IRT provides a parametric form to represent student’s skill level.
Given the limited interactions, we adopted the simplest form, the one-parameter logistic model (1PL), to model human skills.
In the 1PL model, each sub-skill k ∈ ˆ
A is assigned a parameter β k representing the difﬁculty, and a parameter α k called the proﬁciency representing a student’s knowledge state.
The probability that a student has mastered sub-skill k is given by P ( k ) :
= σ ( α k − β k ) , where σ is the sigmoid function.
Hence, instead of representing the state with student’s policy π S , we use ( α k , β k ) K to represent the hidden state.
That is, for ˆ s ∈ ˆ S , ˆ s = { s, ( α k , β k ) K } , where ( α k , β k ) K is hidden.
For each student and each k ∈ ˆ A , we assume that α k changes over time while β k does not.
4.3 Transitions
The transition model consists of two main parts, the target task transition model T , and student’s update function U .
While the former one is known to the teacher, we need to make assumptions on the latter one.
Since we deﬁne the state space over student’s proﬁciency α in Sec 4.2, the transition model is also constructed over the proﬁciency.
Following the previous work on online estimation of student proﬁciency [52, 53], we model the student’s proﬁciencies over time on the sub-skill k as a Wiener process: ˆ U ( α kt +∆ t | α kt ) = exp (cid:16) − ( α kt +∆ t − α kt ) 2 2 λ k ∆ t (cid:17) , k ∈
ˆ A, where ∆ t refers to the step interval and λ k is a parameter controlling the “smoothness” with which student’s proﬁciency varies over time.
For each student and for each k ∈ ˆ A , we assume λ k does not change over time.
To this end, we construct the transition model in the POMDP as ˆ T =
{T , ˆ U} , where T is the transition function in the target task.
4.4 Observations
The observation is composed of the target task state, student’s action and the reward received, { a S , s, r } .
Recall that in Sec 4.1, we deﬁne the action as choosing one sub-skill to train the student, which is a macro-action.
With a slight abuse of notation, we use ˆ a = k to represent any action the teacher takes when teaching sub-skill k .
We redeﬁne the observation as the ratio between the target task rewards achieved by the student’s current and optimal policies when the teacher teaches sub-skill k : f
( k ) :
=
R ( s, ˆ a = k,a S ) R ( s, ˆ a = k,a ∗ ) , where a ∗ is the action generated by the optimal policy π ∗ given the same the target task state s .
Therefore, for ˆ o ∈
ˆ
O , ˆ o = { s, f ( k ) } .
Unlike the binary response in conventional knowledge tracing, the response f ( k ) we have is continuous.
Thus, we use the continuous Bernoulli distribution to construct the observation model:
Z ( f ( k ) | P ( k ))
=
P ( k ) f ( k ) (1 − P ( k ))
1 − f ( k ) .
As a result, the observation model can be deﬁned as ˆ Z =
{I , Z } , where I is an identity mapping for the observable target task state, I ( s )
= s .
4.5 Reward
The distance between the student’s policy and the optimal policies can be represented using P ( k ) .
We represent the distance as the average of one minus master probabilities of each sub-skills
D ( π
[Equation] [Equation]
There are other ways to specify the goal according to the decomposition of the skill, e.g. weakest or multiply [54].
We choose the sum due to our local-independence assumption on sub-skills.
In this work, we assume the cost is uniform, thus, given a ﬁnite horizon of interactions, maximizing the reward function deﬁned in Equation (1) is equivalent to maximizing ˆ R (
[Equation] [Equation]
( k )
[Equation]
( k
[Equation]
K , where P t ( k ) =
σ ( α kt − β k ) .
4.6 Model Learning and Decision Making
We use the student’s performance during the interactions to estimate both λ k and α kt , β k .
Let f 1: t ( k ) denote sequences of student’s performance measure against the optimal policy.
We have the posterior P ( λ k , α kt , β k | f 1: t ( k ))
∝ P ( f 1: t ( k ) | λ k , α kt , β k ) P ( λ k , α kt , β k ) .
The conditional probability of the observation and current proﬁciency can be obtained by integrating out all the previous proﬁciencies.
The likelihood can be approximated through P ( f 1: t ( k ) | λ k , α kt , β k ) ≈ (cid:81)
tt
[Equation]
P
[Equation]
k ) |
[Equation]
k , α kt
[Equation]
, β k ) P ( α kt (cid:48) | α kt )
d α kt (cid:48) .
An approximation of the log posterior over the student’s current proﬁciency given previous responses can be derived to learn the parameters λ k and α kt , β k .
Following [52, 53], we employ maximum a posteriori estimation (MAP) to learn these parameters.
Given the estimation of current state using the past history, we use one-step look-ahead to reduce the impact of the inaccuracy in the transition function.
At timestep t , the teacher’s action is given as
[Equation]
In practice, the student is asked to perform on each sub-skill for a few interactions to initialize the parameters.
4.7 Training on Sub-skills
Our overall strategy for training students on each sub-skill is to diversify scenarios the student would encounter during training.
Training students on sub-skills naturally leads to a partially assistive partner on unlearned sub-skills, which allows the student to explore the sub-skill freely.
We adopt an intuitive assumption: an agent learns cooperation better with a diverse group of partners .
Such a teaching strategy is effective when dealing with synthetic students [55, 56].
The student could learn from a diverse set of partially assistive partners or learn to cope with them by acquiring new skills.
Experiments
We carried out two human-subject experiments to demonstrate how COACH works, one in simulation (Overcooked-AI [57]) and the other with a real robot (Cooperative Ball Maze).
Experiment setups are shown in Figure 2.
We investigated the teaching performances of three types of teachers: the fully-assistive teacher who performs optimally concerning the student’s initial capability, the student-aware teacher who behaves according to our teaching strategy, and the random teacher.
The random teacher in the Cooperative Ball Maze experiment chooses sub-skills randomly, while the random teacher in the Overcooked-AI experiment executes actions randomly.
5.1 Setups
Overcooked-AI.
Overcooked-AI is a benchmark environment for fully cooperative human-AI task performance and has become a well-established domain for studying coordination [58, 59, 60, 61].
The goal of the game is to cook and deliver as much soup as possible in a limited time.
[Figure]
We decompose the policy into two sub-skills: putting ingredients in the pot and delivering the soup .
To put ingredients in the pot, there exists one efﬁcient strategy to pass the ingredient through the middle table.
In brief, rather than picking up one onion at a time and putting them into the pot, the efﬁcient strategy is 1) put multiple onions on the middle table; 2) go to the pot; 3) pick up onions from the middle table; 4) put them into the pot.
The overall idea is to reduce the number of movements needed to deliver the same
[Figure] [Figure]
amount of ingredients.
We recruited N =20 (8 females and 12 males) participants and randomly assigned them into three groups, each with a different teaching strategy.
Students are trained with different teachers and are evaluated with a common unseen partner.
We emulate the human partner in evaluation using a trained model.
Each participant was trained for 5 games and then evaluated for 1 game.
Cooperative Ball Maze.
The Cooperative Ball Maze game requires coordination from both the robot and the human.
Each party will hold one side of the maze board and tilt it to move the ball out from one of the two exits.
We deﬁne two sub-skills leading the rotation and following the rotation .
We recruited N =21 (10 females and 11 males) participants to carry out human-subject experiments.
The participants were ﬁrst evaluated in the two sub-skills, then trained for 20 interactions, and ﬁnally evaluated in the two sub-skills again.
Details can be found in the supplementary materials.
5.2 Results
A fully-assistive teacher impedes human’s acquisition of skills .
In the Overcooked-AI experiment shown in Figure 3(a), we observe that the students trained with a fully-assistive teacher perform worse than the students with a random teacher: it seems that a student becomes “lazy” and free rides the teacher when the teacher unilaterally adapts to the student and performs optimally.
We further investigate the learning pattern of the “lazy student” problem and ﬁnd out that this “laziness” does not lie in the student’s reluctance to take actions, but rather in the lack of motivation to explore and improve .
In Figure 3(c), we show the percentage of reward achieved by the student in OvercookedAI during training.
Compared with the student-aware counterpart, the percentage of reward achieved by humans is similar.
However, only 17% of the participants of the group ﬁnd out the efﬁcient strategy (Figure 3(b)), which is crucial to achieving high scores in the evaluation.
Partially assistive or random partner motivates students to explore new strategies.
By leaving some/all work to the student, partially assistive and random teachers both motivate the student to acquire new skills.
This is shown in Figure 3(b) that most of the students under these two teachers can ﬁnd out the efﬁcient strategy in Overcooked-AI.
However, their performance and the robustness of the learned strategies differ signiﬁcantly.
Though multiple explanations could account for it, we hypothesize the student under the random teacher learns a single ﬁxed strategy to ﬁnish the task alone (Figure 3(c)).
Such a strategy that completes the task alone cannot utilize the possibly helpful inputs from the partner, therefore resulting in a poorer performance score.
An individualized curriculum should be designed for the student.
In the post-experiment survey of
Cooperative Ball Maze, we asked the participants “which mode of the robot is easier to cooperate with?”.
Out of the 21 participants, 4 participants preferred to follow the robot and 17 participants preferred to lead the robot.
Moreover, as we evaluated the student performance with partners of different sub-skills, we found that the student performances were consistent with their declared preferences (Figure 4(a)).
That is to say, the student may have a bias over which strategy to acquire, and tailoring the teaching curriculum to focus on that speciﬁc strategy is efﬁcient and more intuitive to the student.
As demonstrated in Figure 5(a), after the ﬁrst 6 trials that estimated the student’s proﬁciency for each sub-skill, the teacher found out this student improved more as the leader, therefore, the teacher allocated 10 trials to perfect the leading sub-skills and only 4 trials for following .
Moreover, one participant in the random teacher group responded “the robot leading mode is too difﬁcult and I gave up”.
This demonstrates the importance of an individualized curriculum: though there are multiple equally optimal strategies, the individual may have strong preferences, and teaching a non-preferable strategy will discourage the student from learning anything at all.
[Figure]
The student improved more when trained in improved more when trained in the following sub-skill.
We refer the readers to the Appendix for the complete data of all participants.
Limitation
Decomposition into sub-skills.
For many tasks, it is not easy to identify distinct roles to fulﬁll the local-independence criteria of sub-skills.
We manually decompose the skill into a few sub-skills according to the role of the student.
Often, such a decomposition may not be possible or requires careful design.
We can mitigate this problem with recent progress on skill decomposition in single-agent task [44] and role-based task decomposition in multiagent tasks [62].
However, it still demands much more effort to verify their efﬁcacy with a real human on real-world tasks.
Teacher’s Knowledge.
In the deﬁnition of the teaching task, we assume the teacher has full knowledge of optimal policies.
However, it can be hard for the robot to know the oracle human policy beforehand.
To make the conceptual framework practical, we need to relax the requirement on the teacher’s prior knowledge.
In our implementation, we reduce such an assumption by approximating the distance to the optimal policy by the difference in performances.
There can be cases where the optimal performance is hard to know or such relaxation results in severe information loss.
We need more insights on tasks to make the framework practical.
Curriculum design.
In this work, we only design the curriculum over different sub-skills.
However, during our experiment, we observe that humans show various responses to the same sub-skill of different difﬁculties.
As a result, a ﬁner-grained curriculum on the sub-skill training shall be found to further facilitate human learning.
Conclusion
In this work, we propose a conceptual framework, Cooperative Robot Teaching, that enables robots to teach humans in cooperative tasks.
We show that, by abstracting a teaching task over the original duo cooperative task, the robot can learn to act as a specialized teacher to humans.
To be more speciﬁc, we model the teaching task as a POMDP with hidden student policy and propose a partially assistive teaching curriculum to support human learning.
We believe that robot teaching ﬁlls in the gap in the bilateral knowledge transfer in HRI: unlike other HRI tasks where the humans instruct the robots how to behave, the role is reversed and robots try to instill the knowledge back into humans.
Despite the challenges that lie ahead, we believe that robot teaching has great potential and is a necessary step forward to bring robots closer to our daily life.
Acknowledgements.
This research is supported in part by the National Research Foundation, Singapore under its Medium Sized Centre Program, Center for Advanced Robotics Technology Innovation (CARTIN), and AI Singapore Programme (AISG Award
No: AISG2-PhD-2022-01-036[T] and AISG2-PhD-2021-08-014), and by the Science and Engineering Research Council, Agency of
Science, Technology and Research, Singapore, under the National Robotics Program (Grant No.