Original: Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration
Revised: Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: Afﬁliation
Revised: Afﬁliation

Original: Abstract: We present a method for learning human-robot collaboration policyfrom human-human collaboration demonstrations.
Revised: Abstract: We present a method for learning human-robot collaboration policyfrom human-human collaboration demonstrations.

Original: An effective robot assistantmust learn to handle diverse human behaviors shown in the demonstrations and berobust when the humans adjust their strategies during online task execution.
Revised: An effective robot assistantmust learn to handle diverse human behaviors shown in the demonstrations and berobust when the humans adjust their strategies during online task execution.

Original: Ourmethod co-optimizes a human policy and a robot policy in an interactive learningprocess: the human policy learns to generate diverse and plausible collaborativebehaviors from demonstrations while the robot policy learns to assist by estimating the unobserved latent strategy of its human collaborator.
Revised: Ourmethod co-optimizes a human policy and a robot policy in an interactive learningprocess: the human policy learns to generate diverse and plausible collaborativebehaviors from demonstrations while the robot policy learns to assist by estimating the unobserved latent strategy of its human collaborator.

Original: Across a 2D strategygame, a human-robot handover task, and a multi-step collaborative manipulationtask, our method outperforms the alternatives in both simulated evaluations andwhen executing the tasks with a real human operator in-the-loop.
Revised: Across a 2D strategygame, a human-robot handover task, and a multi-step collaborative manipulationtask, our method outperforms the alternatives in both simulated evaluations andwhen executing the tasks with a real human operator in-the-loop.

Original: Supplementarymaterials and videos at https://sites.google.com/view/cogail/home
Revised: Supplementarymaterials and videos at https://sites.google.com/view/cogail/home

Original:  Introduction
Revised:  Introduction

Original: Advancing technologies for human-robot collaboration (HRC) has the potential to unlock applications with large societal impact in manufacturing, hospitals, and home settings [1, 2].
Revised: Advancing technologies for human-robot collaboration (HRC) has the potential to unlock applications with large societal impact in manufacturing, hospitals, and home settings [1, 2].

Original: However,robots that are designed to work around humans are still limited in versatility when performing collaborative tasks.
Revised: However,robots that are designed to work around humans are still limited in versatility when performing collaborative tasks.

Original: Recent advances in robot learning focus on robots that work in isolation [3, 4, 5, 6]or alongside other agents that do not exhibit human traits [7, 8, 9, 10].
Revised: Recent advances in robot learning focus on robots that work in isolation [3, 4, 5, 6]or alongside other agents that do not exhibit human traits [7, 8, 9, 10].

Original: Learning to collaborate withhumans presents unique challenges to existing robot learning methods: instead of optimizing onlyfor efﬁcient task completion, the robot assistant must act in coordination and adapt to the diversity ofstrategies and movements of their human counterparts.
Revised: Learning to collaborate withhumans presents unique challenges to existing robot learning methods: instead of optimizing onlyfor efﬁcient task completion, the robot assistant must act in coordination and adapt to the diversity ofstrategies and movements of their human counterparts.

Original: This work aims to develop robot assistantsthat adapt to diverse human strategies and movements in collaborative manipulation tasks.
Revised: This work aims to develop robot assistantsthat adapt to diverse human strategies and movements in collaborative manipulation tasks.

Original: While past works in multi-agent reinforcement learning show that agents could learn to developcollaborative behaviors [7, 9, 11, 8], hand-engineering reward functions for collaborative goals isnon-trivial [10] and the strategies learned from robot-robot collaboration may not align with thatof the human users.
Revised: While past works in multi-agent reinforcement learning show that agents could learn to developcollaborative behaviors [7, 9, 11, 8], hand-engineering reward functions for collaborative goals isnon-trivial [10] and the strategies learned from robot-robot collaboration may not align with thatof the human users.

Original: On the other hand, learning from demonstrations of human-human collaboration could allow the robot to develop plausible collaboration strategies, as well as avoiding rewardengineering.
Revised: On the other hand, learning from demonstrations of human-human collaboration could allow the robot to develop plausible collaboration strategies, as well as avoiding rewardengineering.

Original: However, most prior works in this setting [12, 13, 14] treat the human as a stationarypart of an environment during the learning process and fail to model the human’s reactions to theevolving robot policy.
Revised: However, most prior works in this setting [12, 13, 14] treat the human as a stationarypart of an environment during the learning process and fail to model the human’s reactions to theevolving robot policy.

Original: Modeling such reciprocal behavior adjustment during training is essential todevelop robust assistive policies, as the human collaborator are likely to depart from the exact strategies they previously demonstrated during online task execution.
Revised: Modeling such reciprocal behavior adjustment during training is essential todevelop robust assistive policies, as the human collaborator are likely to depart from the exact strategies they previously demonstrated during online task execution.

Original: In addition, human behaviors tendto be highly diverse both in high-level intents and low-level movements.
Revised: In addition, human behaviors tendto be highly diverse both in high-level intents and low-level movements.

Original: Learning a robot policythat operates effectively with diverse human behaviors is also crucial for HRC tasks.
Revised: Learning a robot policythat operates effectively with diverse human behaviors is also crucial for HRC tasks.

Original: In this work, we propose a method to learn policies that generate diverse human and robot collaboration behaviors from human-human demonstrations with an interactive co-optimization process.
Revised: In this work, we propose a method to learn policies that generate diverse human and robot collaboration behaviors from human-human demonstrations with an interactive co-optimization process.

Original: Instead of treating humans as part of the environment, the human policy is an active participant inthe training process and co-evolves with the robot policy as the learning progresses.
Revised: Instead of treating humans as part of the environment, the human policy is an active participant inthe training process and co-evolves with the robot policy as the learning progresses.

Original: This allowsthe robot policy to adapt to the movements of the human teammate while assisting the human tocomplete the task.
Revised: This allowsthe robot policy to adapt to the movements of the human teammate while assisting the human tocomplete the task.

Original: To overcome the challenge of diverse strategies presented in the human demonstrations, we introduce a latent representation of human collaboration strategies.
Revised: To overcome the challenge of diverse strategies presented in the human demonstrations, we introduce a latent representation of human collaboration strategies.

Original: On one hand, ourlatent strategy learning objective allows the policy to discover and disentangle factors of variationsin the demonstrations, ensuring that the robot assistant is proﬁcient in all collaborative behaviors
Revised: On one hand, ourlatent strategy learning objective allows the policy to discover and disentangle factors of variationsin the demonstrations, ensuring that the robot assistant is proﬁcient in all collaborative behaviors

Original: [Figure]
Revised: [Figure]

Original: shown by the expert even if certain strategies are rare.
Revised: shown by the expert even if certain strategies are rare.

Original: On the other hand, the latent strategy codeserves as a communication protocol that enables the robots to anticipate future behaviors of theirhuman collaborators and to become more effective during human-in-the-loop task execution.
Revised: On the other hand, the latent strategy codeserves as a communication protocol that enables the robots to anticipate future behaviors of theirhuman collaborators and to become more effective during human-in-the-loop task execution.

Original: We test the method in three challenging collaborative domains illustrated in Figure 1: 2D gamethat requires coordination among two agents ( 2D-Fetch-Quest ), human-to-robot handover ( HRHandover ), and sequential collaborative manipulation with handovers ( HR-SeqManip ).
Revised: We test the method in three challenging collaborative domains illustrated in Figure 1: 2D gamethat requires coordination among two agents ( 2D-Fetch-Quest ), human-to-robot handover ( HRHandover ), and sequential collaborative manipulation with handovers ( HR-SeqManip ).

Original: Thesedomains range in difﬁculty from a 2D grid-like game to high-dimensional manipulation tasks with7-DOF arms and task execution within a physics-based simulation environment using Bullet [15] andiGibson [16].
Revised: Thesedomains range in difﬁculty from a 2D grid-like game to high-dimensional manipulation tasks with7-DOF arms and task execution within a physics-based simulation environment using Bullet [15] andiGibson [16].

Original: We conclude by presenting a brief demonstration of the method in interaction withreal human collaborators, showing for the ﬁrst time a learning method that succeeds at adapting toreal users in these challenging tasks.
Revised: We conclude by presenting a brief demonstration of the method in interaction withreal human collaborators, showing for the ﬁrst time a learning method that succeeds at adapting toreal users in these challenging tasks.

Original:  Related work
Revised:  Related work

Original: Multi-agent Reinforcement Learning:
Revised: Multi-agent Reinforcement Learning:

Original: A large body of prior works utilize multi-agent RL(MARL) to learn collaborative strategies for tackling multiplayer games [17, 11], multi-robot manipulation [18, 19], trafﬁc control [20], and social dilemmas [7].
Revised: A large body of prior works utilize multi-agent RL(MARL) to learn collaborative strategies for tackling multiplayer games [17, 11], multi-robot manipulation [18, 19], trafﬁc control [20], and social dilemmas [7].

Original: However, in the HRI setting, therobot needs to learn to adapt to a wide range of human collaborative behaviors, for which it is nontrivial to design effective reward functions.
Revised: However, in the HRI setting, therobot needs to learn to adapt to a wide range of human collaborative behaviors, for which it is nontrivial to design effective reward functions.

Original: Translating the beneﬁts of MARL to HRI in real-worldmanipulation tasks is limited because agents are assumed to have similar policies or don’t exhibithuman traits [21].
Revised: Translating the beneﬁts of MARL to HRI in real-worldmanipulation tasks is limited because agents are assumed to have similar policies or don’t exhibithuman traits [21].

Original: Tampuu et al.
Revised: Tampuu et al.

Original: [10] use multi-agent reinforcement learning to model both agentsand demonstrate how collaborative behavior can arise with the appropriate choice of reward structure.
Revised: [10] use multi-agent reinforcement learning to model both agentsand demonstrate how collaborative behavior can arise with the appropriate choice of reward structure.

Original: In [8] the robot learns to inﬂuence the collaborator’s action by optimizing a long-term rewardover repeated interactions and estimating their policy per interaction episode through a learned latentspace.
Revised: In [8] the robot learns to inﬂuence the collaborator’s action by optimizing a long-term rewardover repeated interactions and estimating their policy per interaction episode through a learned latentspace.

Original: Hand-engineered collaborative rewards can cause the overﬁtting of the learned robot policy[9], limiting the robot’s adaption to diverse human behaviors.
Revised: Hand-engineered collaborative rewards can cause the overﬁtting of the learned robot policy[9], limiting the robot’s adaption to diverse human behaviors.

Original: Multi-agent Inverse Reinforcement Learning: Inverse reinforcement learning (IRL), on the otherhand, shows promising results in learning collaborative behaviors learning from expert demonstrations [22, 23].
Revised: Multi-agent Inverse Reinforcement Learning: Inverse reinforcement learning (IRL), on the otherhand, shows promising results in learning collaborative behaviors learning from expert demonstrations [22, 23].

Original: Notably, recent progress in generative adversarial imitation learning (GAIL) [24, 25]further shows effectiveness in cooperative settings [26].
Revised: Notably, recent progress in generative adversarial imitation learning (GAIL) [24, 25]further shows effectiveness in cooperative settings [26].

Original: However, as noted in prior works [25], thelearned collaborative policy tends to overﬁt strategies that are easy to execute and neglects the morechallenging cases.
Revised: However, as noted in prior works [25], thelearned collaborative policy tends to overﬁt strategies that are easy to execute and neglects the morechallenging cases.

Original: This becomes problematic when the robot encounters a real human collaborator, who may prefer strategies outside of the distribution that the policy focuses on.
Revised: This becomes problematic when the robot encounters a real human collaborator, who may prefer strategies outside of the distribution that the policy focuses on.

Original: Our methodaddresses this problem by explicitly disentangling the behaviors shown in the demonstrations andencourage the robot policy to learn from diverse collaborative strategies.
Revised: Our methodaddresses this problem by explicitly disentangling the behaviors shown in the demonstrations andencourage the robot policy to learn from diverse collaborative strategies.

Original: Modeling diverse behavior: Learning diverse behaviors has been a long-standing challenge in reinforcement learning research [27, 28, 29].
Revised: Modeling diverse behavior: Learning diverse behaviors has been a long-standing challenge in reinforcement learning research [27, 28, 29].

Original: Recent works show that by maximizing the entropy ofstates and minimizing the conditional entropy of states with respect to the actions, policies can learndiverse skills [30, 31] even without reward signals [32].
Revised: Recent works show that by maximizing the entropy ofstates and minimizing the conditional entropy of states with respect to the actions, policies can learndiverse skills [30, 31] even without reward signals [32].

Original: Rather than training policies to behave asdiverse as possible, our method optimizes for covering the behaviors shown in demonstrations.
Revised: Rather than training policies to behave asdiverse as possible, our method optimizes for covering the behaviors shown in demonstrations.

Original: Thisallows the robot to focus on collaborative behaviors that are plausible during human-robot explo-
Revised: Thisallows the robot to focus on collaborative behaviors that are plausible during human-robot explo-

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: ration, which is especially crucial in large environments where behavioral diversity is unbounded.
Revised: ration, which is especially crucial in large environments where behavioral diversity is unbounded.

Original: Clegg et al.
Revised: Clegg et al.

Original: [33] introduces a method to learn diverse and plausible assistive dressing behavior byhand-coding human motor capabilities in the model.
Revised: [33] introduces a method to learn diverse and plausible assistive dressing behavior byhand-coding human motor capabilities in the model.

Original: In contrast, our method does not assume explicit domain knowledge and learns collaborative policies from demonstration data.
Revised: In contrast, our method does not assume explicit domain knowledge and learns collaborative policies from demonstration data.

Original: Wang et al.
Revised: Wang et al.

Original: [34]proposes to learn diverse behaviors through imitation but requires access to demonstrations duringonline execution, whereas our method only assumes access to demonstration at training time.
Revised: [34]proposes to learn diverse behaviors through imitation but requires access to demonstrations duringonline execution, whereas our method only assumes access to demonstration at training time.

Original: Human behavior modeling in human-robot collaboration: Prior works approach the problem ofmodeling human behaviors in interactive learning settings in multiple ways.
Revised: Human behavior modeling in human-robot collaboration: Prior works approach the problem ofmodeling human behaviors in interactive learning settings in multiple ways.

Original: Most works introduceintermediate representation of the human behaviors such as user type classiﬁcation [12, 35], futureaction prediction [36], hidden goal estimation [37], information communication [38] and humanmotion prediction [39].
Revised: Most works introduceintermediate representation of the human behaviors such as user type classiﬁcation [12, 35], futureaction prediction [36], hidden goal estimation [37], information communication [38] and humanmotion prediction [39].

Original: These methods show promising results in HRI applications, including assistive robot for dressing [33, 40], feeding [41, 42, 43], assembling [14] etc.
Revised: These methods show promising results in HRI applications, including assistive robot for dressing [33, 40], feeding [41, 42, 43], assembling [14] etc.

Original: However, most existingworks requires tasks-speciﬁc knowledge such as user types [12, 35].
Revised: However, most existingworks requires tasks-speciﬁc knowledge such as user types [12, 35].

Original: Our method learns a latentspace of collaboration strategy from demonstrations without explicit task-level supervision.
Revised: Our method learns a latentspace of collaboration strategy from demonstrations without explicit task-level supervision.

Original:  Method
Revised:  Method

Original: Our method is designed to overcome two unique challenges imposed by human-robot interaction:(1)
Revised: Our method is designed to overcome two unique challenges imposed by human-robot interaction:(1)

Original: human behaviors are difﬁcult to model because they follow their own policy while reacting to therobot’s actions, and (2) human behaviors are highly diverse both in terms of the high-level strategiesand the low-level actions.
Revised: human behaviors are difﬁcult to model because they follow their own policy while reacting to therobot’s actions, and (2) human behaviors are highly diverse both in terms of the high-level strategiesand the low-level actions.

Original: Below we introduce the general HRC problem setup (3.1), an interactiveimitation learning formulation to the problem (3.2), and how to uncover diverse collaboration behaviors by learning a latent strategy space (3.3).
Revised: Below we introduce the general HRC problem setup (3.1), an interactiveimitation learning formulation to the problem (3.2), and how to uncover diverse collaboration behaviors by learning a latent strategy space (3.3).

Original: Finally, we introduce the learning algorithm
Revised: Finally, we introduce the learning algorithm

Original: Co-GAIL(Algorithm 1) and the human-in-the-loop task execution setup (3.4).
Revised: Co-GAIL(Algorithm 1) and the human-in-the-loop task execution setup (3.4).

Original: 3.1 Learning human-robot co-policy
Revised: 3.1 Learning human-robot co-policy

Original: Although human and robot have different state space and action space, their underlying motivationof completing the collaboration task is shared.
Revised: Although human and robot have different state space and action space, their underlying motivationof completing the collaboration task is shared.

Original: Instead of treating the human as a part of the environment with a ﬁxed policy, allowing the human and the robot to learn and explore concurrentlyhas the potential to achieve a more robust robot policy.
Revised: Instead of treating the human as a part of the environment with a ﬁxed policy, allowing the human and the robot to learn and explore concurrentlyhas the potential to achieve a more robust robot policy.

Original: We consider a human-robot collaborationtask as an extension of Markov Decision Process (MDP) called Markov games [44] for two agents( N = 2 ), MG
Revised: We consider a human-robot collaborationtask as an extension of Markov Decision Process (MDP) called Markov games [44] for two agents( N = 2 ), MG

Original: = ( S , A H , A R , T , R, γ, ρ 0 ) , with state space S , human action space
Revised: = ( S , A H , A R , T , R,  , ⇢ 0 ) , with state space S , human action space

Original: A H , robot action space
Revised: A H , robot action space

Original: A R , transition distribution T ( s t +1 | s t , a Ht , a Rt ) , reward function R ( s t , a Ht , a Rt , s t +1 ) ,discount factor γ ∈ [0 , 1) , and initial state distribution ρ 0 .
Revised: A R , transition distribution T ( s t +1 | s t , a Ht , a Rt ) , reward function R ( s t , a Ht , a Rt , s t +1 ) ,discount factor  2 [0 , 1) , and initial state distribution ⇢ 0 .

Original: The solution to this MDP is a co-policy π co with a shared policy network that outputs action forboth agents.
Revised: The solution to this MDP is a co-policy ⇡ co with a shared policy network that outputs action forboth agents.

Original: At every step, the co-policy observes the current state s t , chooses a set of actions
Revised: At every step, the co-policy observes the current state s t , chooses a set of actions

Original: ( a Ht , a Rt ) ∼ π co ( a Ht , a Rt | s t ) , and observes the next state s t +1 ∼ T ( ·| s t , a Ht , a Rt ) and rewardr t=
Revised: ( a Ht , a Rt ) ⇠ ⇡ co ( a Ht , a Rt | s t ) , and observes the next state s t +1 ⇠ T ( ·| s t , a Ht , a Rt ) and reward 117 r t

Original:  R ( s t , a Ht , a Rt , s t +1 ) shared by both agents.
Revised: = R ( s t , a Ht , a Rt , s t +1 ) shared by both agents.

Original: The goal is to learn a co-policy that maximizesthe shared expected return E [[-(cid:80) ∞ t =
-]
Revised: The goal is to learn a co-policy that maximizesthe shared expected return E [[-(cid:80) ∞ t =
-]

Original: γ{+P 1 t =
 +} t R ( s t , a Ht , a Rt , s t +1 )] .
Revised: {+P 1 t =
 +} t R ( s t , a Ht , a Rt , s t +1 )] .

Original: With a slight abuse of notation, wedenote the part of the policy that outputs human actions as π Hco and that for robot actions as π Rco .
Revised: With a slight abuse of notation, wedenote the part of the policy that outputs human actions as ⇡ Hco and that for robot actions as ⇡ Rco .

Original: 3.2 Learning strategy-conditioned co-policy from 
Revised: 3.2 Learning strategy-conditioned co-policy from demonstration

Original: demonstration

Original: While we could attempt to solve for the co-policy using existing RL methods, designing an effectiveand comprehensive reward function that properly evaluates the quality of human-robot collaborationis non-trivial.
Revised: While we could attempt to solve for the co-policy using existing RL methods, designing an effectiveand comprehensive reward function that properly evaluates the quality of human-robot collaborationis non-trivial.

Original: On the other hand, imitation learning has the advantage of learning the policy directlyfrom demonstrations, bypassing the need to explicitly design a reward function.
Revised: On the other hand, imitation learning has the advantage of learning the policy directlyfrom demonstrations, bypassing the need to explicitly design a reward function.

Original: As such, we recorda set of human-human demonstrations M of two users performing a cooperative task and formulatethe co-policy learning problem based on MA-GAIL [26], a multi-agent extension of the Generative
Revised: As such, we recorda set of human-human demonstrations M of two users performing a cooperative task and formulatethe co-policy learning problem based on MA-GAIL [26], a multi-agent extension of the Generative

Original: Adversarial Imitation Learning (GAIL) [24] framework.
Revised: Adversarial Imitation Learning (GAIL) [24] framework.

Original: Let π E 1 and π E 2 denote two expert policies to which we only have access to demonstrations, M ,a dataset of N
Revised: Let ⇡ E 1 and ⇡ E 2 denote two expert policies to which we only have access to demonstrations, M ,a dataset of N

Original: collaborative task demonstrations
Revised: collaborative task demonstrations

Original: M = { τ i } Ni =1 .
Revised: M = { ⌧ i } Ni =1 .

Original: Each demonstration is a trajectoryτ i =
Revised: Each demonstration is a trajectory⌧ i =

Original: ( s i 0 , a E 1 

Revised: ( s i 0 , a E 1

Original: i , a E 2 

Revised: i , a E 2

Original: i , s i 1 , a E 1 1 i , a E 2 1 i , ..., s iT i ) .
Revised: i , s i 1 , a E 1 1 i , a E 2 1 i , ..., s iT i ) .

Original: The goal of MA-GAIL [26] is to train the co-policyπ co to imitate expert policy ( π E 1 , π E 2 ) by minimizing distance between the generated state-actiondistribution ρ ( π co ) and the expert’s distribution ρ ( π E 1 , π E 2 ) measured by Jensen-Shannon divergence.
Revised: The goal of MA-GAIL [26] is to train the co-policy⇡ co to imitate expert policy ( ⇡ E 1 , ⇡ E 2 ) by minimizing distance between the generated state-actiondistribution ⇢ ( ⇡ co ) and the expert’s distribution ⇢ ( ⇡ E 1 , ⇡ E 2 ) measured by Jensen-Shannon divergence.

Original: We optimize the following objective:
Revised: We optimize the following objective:

Original: [Equation]
Revised: [Equation]

Original: where D is a discriminative classiﬁer which tries to distinguish state-action pairs from the trajectories generated by π co and ( π E 1 , π E 2 ) .
Revised: where D is a discriminative classiﬁer which tries to distinguish state-action pairs from the trajectories generated by ⇡ co and ( ⇡ E 1 , ⇡ E 2 ) .

Original: However, directly applying MA-GAIL and equation 5 results in a poor co-policy due to the diverse behaviors in human-human demonstration.
Revised: However, directly applying MA-GAIL and equation 5 results in a poor co-policy due to the diverse behaviors in human-human demonstration.

Original: Given the same environment state s , human mightperform different types of action a H based on their choice of strategy, which is hidden from theobservable state.
Revised: Given the same environment state s , human mightperform different types of action a H based on their choice of strategy, which is hidden from theobservable state.

Original: To disentangle the diverse human behaviors associated with the same state, we include a latent representation, z , called strategy , to the input of the co-policy, π co ( a H , a R | s , z ) .
Revised: To disentangle the diverse human behaviors associated with the same state, we include a latent representation, z , called strategy , to the input of the co-policy, ⇡ co ( a H , a R | s , z ) .

Original: Weassume that the strategy can be inferred from the history of observations and the the human actionand deﬁne a recognition model z t ∼ ψ ( ·| s t − K : t , a Ht − 1 ) , where K is the length of the history.
Revised: Weassume that the strategy can be inferred from the history of observations and the the human actionand deﬁne a recognition model z t ⇠ ( ·| s t  K : t , a Ht  1 ) , where K is the length of the history.

Original: Tosimplify the notation, we denote the history at time step t as h t =
Revised: Tosimplify the notation, we denote the history at time step t as h t =

Original: ( s t − K : t , a Ht − 1 ) .
Revised: ( s t  K : t , a Ht  1 ) .

Original: The recognitionmodelψ is trained jointly with the co-policy (Figure 2a).
Revised: The recognitionmodel is trained jointly with the co-policy (Figure 2a).

Original: Our framework allows the strategy toevolve as the task progresses in an interaction episode.
Revised: Our framework allows the strategy toevolve as the task progresses in an interaction episode.

Original: As such, the robot must constantly update itsestimate of the human’s latent strategy usingψ ( · ) .
Revised: As such, the robot must constantly update itsestimate of the human’s latent strategy using ( · ) .

Original: 3.3 Uncovering diverse human behaviors
Revised: 3.3 Uncovering diverse human behaviors

Original: Introducing the latent strategy alone does not lead to successful co-policy learning.
Revised: Introducing the latent strategy alone does not lead to successful co-policy learning.

Original: One challengeis that the human policy can simply ignore the latent strategy z and always opt to generate the mostcommon behaviors.
Revised: One challengeis that the human policy can simply ignore the latent strategy z and always opt to generate the mostcommon behaviors.

Original: To this end, we use an information-theoretic regularization [45, 25] to enforcehigh mutual information between the latent strategy and the human action (Figure 2a).
Revised: To this end, we use an information-theoretic regularization [45, 25] to enforcehigh mutual information between the latent strategy and the human action (Figure 2a).

Original: Given a priordistribution of the strategy space p ( z ) , we minimize the objective:
Revised: Given a priordistribution of the strategy space p ( z ) , we minimize the objective:

Original: [Equation]
Revised: [Equation]

Original: where h is the length- K trajectory generated from the acting policy 
Revised: where h is the length- K trajectory generated from the acting policy ⇡

Original: π Hco given a sampled code z .
Revised: Hco given a sampled code z .

Original: While L z indeed incentivizes the human policy to utilize the latent strategy when producing actions,it does not address another issue due to the imbalance distribution of training data, which resultsinψ ( · ) ignoring less frequently seen a H and mapping them to the same strategies that reﬂects themajority of a H .
Revised: While L z indeed incentivizes the human policy to utilize the latent strategy when producing actions,it does not address another issue due to the imbalance distribution of training data, which resultsin ( · ) ignoring less frequently seen a H and mapping them to the same strategies that reﬂects themajority of a H .

Original: Whereas in practice, the human collaborator may interact with the robot with anystrategy included in the demonstration.
Revised: Whereas in practice, the human collaborator may interact with the robot with anystrategy included in the demonstration.

Original: To address this issue, we introduce another objective thatmeasures the reconstruction error of human actions in the training dataset (Figure 2b):
Revised: To address this issue, we introduce another objective thatmeasures the reconstruction error of human actions in the training dataset (Figure 2b):

Original: [Equation]
Revised: [Equation]

Original: In each training iteration, a batch of expert trajectory snippet h will be sampled from the dataset
Revised: In each training iteration, a batch of expert trajectory snippet h will be sampled from the dataset

Original: M .
Revised: M .

Original: The recognition modelψ ( · ) ﬁrst estimates the hidden strategy ˆ z from h .
Revised: The recognition model ( · ) ﬁrst estimates the hidden strategy ˆ z from h .

Original: The human policy π Hco ( s , ˆ z ) is used as a decoder to reconstruct the expert action from the predicted strategy ˆ z .
Revised: The human policy ⇡ Hco ( s , ˆ z ) is used as a decoder to reconstruct the expert action from the predicted strategy ˆ z .

Original: Thisloss encouragesψ ( · ) to encode different a H to different z in order to minimize the reconstructionerror, enforcing the inverse mapping from the human behavior space to the strategy space.
Revised: Thisloss encourages ( · ) to encode different a H to different z in order to minimize the reconstructionerror, enforcing the inverse mapping from the human behavior space to the strategy space.

Original: Puttingit together, the ﬁnal objective for learning diverse collaboration behaviors becomes:
Revised: Puttingit together, the ﬁnal objective for learning diverse collaboration behaviors becomes:

Original: [Equation]
Revised: [Equation]

Original: 4) where λ 1 , λ 2 are the hyperparameters for the intention and expert behavior reconstruction regularization term.
Revised: ) where  1 ,  2 are the hyperparameters for the intention and expert behavior reconstruction regularization term.

Original: The overview of the proposed algorithm is shown in Algorithm 1.
Revised: The overview of the proposed algorithm is shown in Algorithm 1.

Original: Data: M ← given human-human collaboration dataset
Revised: given human-human collaboration dataset

Original: Input: π co← initialize co-policy,ψ ← initializestrategy recognition network, D ← initialize the discriminator, B ← ∅ initialize the replay buffer 1 while not done do for trajectory i=1, ..., m do z i ∼ p ( z ) sample strategy code from prior B ← B ∪ { ( s t , a Ht , a Rt , z i ) } Tt =1 : append experiences from π co ( · , z i ) to the buffer for update step = 1, ..., n do b M , b B ← sample batch of trajectories from M and B
Revised: Input: ⇡ co initialize co-policy, initialize

Original: L z ← calculate strategy reconstruction loss with ψ and π co (Eq. 2) with data from b B L a ← calculate behavior reconstruction loss with ψ and π co (Eq. 3) with data from b M update ψ and π co with the gradient of λ 1 L z + λ 2 L a update discriminator Daccording to Eq. 4 using b M and b B update co-policy π co with PPO[46]
Revised: D

Original: 3.4 Human-in-the-loop task execution
Revised: 4 Human-in-the-loop task execution

Original: During test time, we discard the human policy and only use the robot policy to control the robotwho interacts with a real human user (Figure 2c).
Revised: During test time, we discard the human policy and only use the robot policy to control the robotwho interacts with a real human user (Figure 2c).

Original: Different from prior work [25] where the latentcode reconstruction network is only used during the training process for mutual information maximization, our learned recognition modelψ ( · ) is an important component for the robot policy torecognize its human collaborators’ strategy and anticipate their future behaviors in real-time.
Revised: Different from prior work [25] where the latentcode reconstruction network is only used during the training process for mutual information maximization, our learned recognition model ( · ) is an important component for the robot policy torecognize its human collaborators’ strategy and anticipate their future behaviors in real-time.

Original: Ateach time step,ψ ( · ) takes a history of states s and human’s previous action a H as input and outputsa strategy estimation ˆ z .
Revised: Ateach time step, ( · ) takes a history of states s and human’s previous action a H as input and outputsa strategy estimation ˆ z .

Original: The robot policy will then generate the robot action, a R ∼ π Rco ( s , ˆ z ) .
Revised: The robot policy will then generate the robot action, a R ⇠ ⇡ Rco ( s , ˆ z ) .

Original:  Experiment Setup
Revised:  Experiment Setup

Original: The major advantage of Co-GAIL is its capability of uncovering diverse collaboration behavior fromthe given demonstrations.
Revised: The major advantage of Co-GAIL is its capability of uncovering diverse collaboration behavior fromthe given demonstrations.

Original: In the experiment section, we aim to compare Co-GAIL with state-ofthe-art methods for learning collaboration skills.
Revised: In the experiment section, we aim to compare Co-GAIL with state-ofthe-art methods for learning collaboration skills.

Original: We focus on whether each method can successfullycomplete the task and successfully learn the diverse collaboration strategies.
Revised: We focus on whether each method can successfullycomplete the task and successfully learn the diverse collaboration strategies.

Original: We also invite four realusers to interact with the trained robot policy and evaluate the performance for each method.
Revised: We also invite four realusers to interact with the trained robot policy and evaluate the performance for each method.

Original: 4.1 Baselines
Revised: 4.1 Baselines

Original: We start by comparing with imitation learning designed for a single agent (i.e., treat human as partof the environment) and build towards the proposed method progressively.
Revised: We start by comparing with imitation learning designed for a single agent (i.e., treat human as partof the environment) and build towards the proposed method progressively.

Original: BC-single: Behavior-cloning algorithm that only learns a robot policy.
Revised: BC-single: Behavior-cloning algorithm that only learns a robot policy.

Original: During training, human
Revised: During training, human

Original: behaviors are regarded as a part of the environment and played back in an open-loop fashion.
Revised: behaviors are regarded as a part of the environment and played back in an open-loop fashion.

Original: BC-GAIL: Train the human and robot policy in isolation.
Revised: BC-GAIL: Train the human and robot policy in isolation.

Original: First learn human policy with BC.
Revised: First learn human policy with BC.

Original: Then
Revised: Then

Original: use GAIL algorithm[24] to learn the robot policy by interacting with the learned human policy.
Revised: use GAIL algorithm[24] to learn the robot policy by interacting with the learned human policy.

Original: MA-GAIL: Implementation of MA-GAIL [26].
Revised: MA-GAIL: Implementation of MA-GAIL [26].

Original: Both human and robot policies are learned with a
Revised: Both human and robot policies are learned with a

Original: shared GAIL objective (Equation 5) during the training process.
Revised: shared GAIL objective (Equation 5) during the training process.

Original: MA-InfoGAIL:
Revised: MA-InfoGAIL:

Original: Add the latent strategy code learning objective based on InfoGAIL[25]
Revised: Add the latent strategy code learning objective based on InfoGAIL[25]

Original: (Equation2) as an additional learning objective to MA-GAIL.
Revised: (Equation2) as an additional learning objective to MA-GAIL.

Original: MA-InfoGAIL MA-GAIL
Revised: MA-InfoGAIL MA-GAIL

Original: DIAYN
Revised: DIAYN

Original: Co-GAIL Dataset
Revised: Co-GAIL Dataset

Original: Strategy Strategy
Revised: Strategy Strategy

Original: Strategy
Revised: Strategy

Original: Strategy33% 33% 17% 17%
Revised: Strategy33% 33% 17% 17%

Original: Strategy Strategy
Revised: Strategy Strategy

Original: Strategy
Revised: Strategy

Original: Strategy33% 33% 17% 17%
Revised: Strategy33% 33% 17% 17%

Original: D
Revised: D

Original: Dnumber of episodes number of episodes number of episodes number of episodes number of episodes number of episodes number of episodes number of episodes s u cce ss r a t e s u cce ss r a t e s u cce ss r a t e s u cce ss r a t e s u cce ss r a t e s u cce ss r a t e s u cce ss r a t e s u cce ss r a t e
Revised: Dnumber of episodes number of episodes number of episodes number of episodes number of episodes number of episodes number of episodes number of episodes s u cce ss r a t e s u cce ss r a t e s u cce ss r a t e s u cce ss r a t e s u cce ss r a t e s u cce ss r a t e s u cce ss r a t e s u cce ss r a t e

Original: DIAYN: Implementation of [32] in a collaboration setting.
Revised: DIAYN: Implementation of [32] in a collaboration setting.

Original: Use a diversity reward to encourage thehuman policy to be diverse as possible.
Revised: Use a diversity reward to encourage thehuman policy to be diverse as possible.

Original: Co-GAIL: The complete implementation of our proposed method.
Revised: Co-GAIL: The complete implementation of our proposed method.

Original: Our method differs from MAInfoGAIL in the inclusion of behavior reconstruction loss (Equation 3) to encourage the model tocover all strategies in the given demonstration.
Revised: Our method differs from MAInfoGAIL in the inclusion of behavior reconstruction loss (Equation 3) to encourage the model tocover all strategies in the given demonstration.

Original: 4.2 Collaborative tasks
Revised: 4.2 Collaborative tasks

Original: We evaluate the methods in three continuous-action space domains that involve collaboration, illustrated in Figure 1.
Revised: We evaluate the methods in three continuous-action space domains that involve collaboration, illustrated in Figure 1.

Original: See Appendix for more details.2D-Fetch-Quest (2D-FetchQ.): A collaborative 2D game inspired by the Fetch Quest minigame in
Revised: See Appendix for more details.2D-Fetch-Quest (2D-FetchQ.): A collaborative 2D game inspired by the Fetch Quest minigame in

Original: Super Mario Party.
Revised: Super Mario Party.

Original: Two agents must coordinate their strategy to fetch treasures from two lockedrooms.
Revised: Two agents must coordinate their strategy to fetch treasures from two lockedrooms.

Original: Agents have complementary roles with temporal dependencies: one agent presses a buttonto unlock the room while the other agent fetches the treasure, allowing four different strategiesdepending on the order of the actions and roles.
Revised: Agents have complementary roles with temporal dependencies: one agent presses a buttonto unlock the room while the other agent fetches the treasure, allowing four different strategies depending on the order of the actions and roles.

Original: The human-human demonstrations exhibit fourdistinctive strategies.
Revised: Human-human demonstrations exhibit four distinctivestrategies.

Original: Strategy 1 and 2 differs in which party takes the treasure ﬁrst.
Revised: Strategy 1 and 2 differs in which party takes the treasure ﬁrst.

Original: Strategy 3 and4 are when the two parties decide to switch to the other side of the space to complete the task.
Revised: Strategy 3 and 4 are whenthe two parties decide to switch to the other side of the space to complete the task.

Original: HR-Handover (HR-H.): A human-to-robot object handover task, in which the human holds anobject and hands it over to the robot.
Revised: HR-Handover (HR-H.): A human-to-robot object handover task, in which the human holds anobject and hands it over to the robot.

Original: The handover can take place in any region between the twoagents with any approaching angle at any speed.
Revised: The handover can take place in any region between the twoagents with any approaching angle at any speed.

Original: The action space of each agent is the 6D position ofthe agent’s hand relative to the previous time step and a ﬂoat value describing the gripper state.
Revised: The action space of each agent is the 6D position ofthe agent’s hand relative to the previous time step and a ﬂoat value describing the gripper state.

Original: Thehuman-human demonstrations exhibit diverse handover locations, varying in all three dimensions.
Revised: Thehuman-human demonstrations exhibit diverse handover locations, varying in all three dimensions.

Original: We visualized the distribution of the position when handover happens (X-Y locations) in Figure 4a.
Revised: We visualized the distribution of the position when handover happens (X-Y locations) in Figure 4a.

Original: HR-SeqManip (HR-S.): Place an object in a drawer by collaboratively manipulating both the objectand the drawer.
Revised: HR-SeqManip (HR-S.): Place an object in a drawer by collaboratively manipulating both the objectand the drawer.

Original: There are two possible strategies to solve this task: the robot assists by opening thedrawer while the human drops the object, or the human hands over the object to the robot and opensthe drawer, while the robot drops the object into the drawer.
Revised: There are two possible strategies to solve this task: the robot assists by opening thedrawer while the human drops the object, or the human hands over the object to the robot and opensthe drawer, while the robot drops the object into the drawer.

Original: Other forms of diversity include theobject handover locations and details in motion such as ways to open the drawer.
Revised: Other forms of diversity include theobject handover locations and details in motion such as ways to open the drawer.

Original: The action spaceis the same as in HR-Handover.
Revised: The action spaceis the same as in HR-Handover.

Original: These domains are implemented in an interactive simulation environment in which the human userscontrol the agents via a control interface.
Revised: These domains are implemented in an interactive simulation environment in which the human userscontrol the agents via a control interface.

Original: Demonstrations in 2D-Fetch-Quest are collected througha joystick interface.
Revised: Demonstrations in 2D-Fetch-Quest are collected througha joystick interface.

Original: HR-Handover and HR-SeqManip are implemented in iGibson environments[16] with Bullet physics engine [15].
Revised: HR-Handover and HR-SeqManip are implemented in iGibson environments[16] with Bullet physics engine [15].

Original: Collaborative demonstrations are collected from two humanusers controlling the two agents using the Roboturk teleoperation interface [47, 48].
Revised: Collaborative demonstrations are collected from two humanusers controlling the two agents using the Roboturk teleoperation interface [47, 48].

Original: More details ofthe environment design could be found in Appendix.
Revised: More details ofthe environment design could be found in Appendix.

Original: 4.3 Experimental setup
Revised: 4.3 Experimental setup

Original: We assess the performance of π R in terms of task completion (success rate) and coverage of diversestrategies.
Revised: We assess the performance of ⇡ R in terms of task completion (success rate) and coverage of diversestrategies.

Original: We employ three evaluation protocols to show different facets of the problem of learningdiverse and plausible collaborative behaviors.
Revised: We employ three evaluation protocols to show different facets of the problem of learningdiverse and plausible collaborative behaviors.

Original: Interpolation: Uniformly sample the learned strategy space and simulate a robot-human trajectoryfor each strategy using the learned co-policy.
Revised: Interpolation: Uniformly sample the learned strategy space and simulate a robot-human trajectoryfor each strategy using the learned co-policy.

Original: We evaluate the success rate and diversity of thesetrajectories for our method and the baselines.
Revised: We evaluate the success rate and diversity of thesetrajectories for our method and the baselines.

Original: This evaluation provides an insight into the effect ofeach strategy on the co-policy, and the coverage of diversity presented in the training data.
Revised: This evaluation provides an insight into the effect ofeach strategy on the co-policy, and the coverage of diversity presented in the training data.

Original: Replay: We subject the robot policy to interact with the open-loop replays of human trajectoriespreviously unseen during training.
Revised: Replay: We subject the robot policy to interact with the open-loop replays of human trajectoriespreviously unseen during training.

Original: Unlike interpolation evaluation, the replay evaluation tests of thepolicy can correctly respond to diverse and previously unseen collaborative behaviors.
Revised: Unlike interpolation evaluation, the replay evaluation tests of thepolicy can correctly respond to diverse and previously unseen collaborative behaviors.

Original: The settingallows scalable evaluation as we do not need a real human in the loop, but it is limited because thehuman policy does not react reciprocally to the robot.
Revised: The settingallows scalable evaluation as we do not need a real human in the loop, but it is limited because thehuman policy does not react reciprocally to the robot.

Original: Real-human evaluation: Proof-of-concept evaluation with task execution with a real human userin-the-loop interacting with the robot’s policy as illustrated in Figure 2c.
Revised: Real-human evaluation: Proof-of-concept evaluation with task execution with a real human userin-the-loop interacting with the robot’s policy as illustrated in Figure 2c.

Original: We report the success ratefor all domains and methods.
Revised: We report the success ratefor all domains and methods.

Original:  Results
Revised:  Results

Revised: [Table]

Revised: Modeling human behavior improves

Original: [Table]
Revised: [Table]

Original: Modeling human behavior improvesπ Rco : The results show that modelinghuman behavior improves the robustness of the learned π R when facing unseen human behavior.
Revised: : The results show that modelinghuman behavior improves the robustness of the learned ⇡ R when facing unseen human behavior.

Original: Our proposed CoGAIL outperforms BC-single in replayevaluation in Table 1 for more than 20%in all three tasks.
Revised: Our proposed CoGAIL outperforms BC-single in replayevaluation in Table 1 for more than 20%in all three tasks.

Original: Co-optimization outperforms learning in isolation: Comparing the replayevaluation results (Table 1) for Co-GAIL and MA-GAIL indicates that co-optimizing the π co hasbeneﬁts over the sequential training of human and robot policy in BC-GAIL.
Revised: Co-optimization outperforms learning in isolation: Comparing the replayevaluation results (Table 1) for Co-GAIL and MA-GAIL indicates that co-optimizing the ⇡ co hasbeneﬁts over the sequential training of human and robot policy in BC-GAIL.

Original: Co-GAIL learns more diverse strategies compared with the baselines:
Revised: Co-GAIL learns more diverse strategies compared with the baselines:

Original: Unlike existing methods, we designed Co-GAIL to obtain a π Rco that can interact in the scenariothat a real human approaches the task with diverse strategies.
Revised: Unlike existing methods, we designed Co-GAIL to obtain a ⇡ Rco that can interact in the scenariothat a real human approaches the task with diverse strategies.

Original: Using the 2D-FetchQuest domain, which can be solved with 4 distinct strategies (see Appendix), we designed two datasets containing different distribution of demonstrated strategies as follows:
Revised: Using the 2D-FetchQuest domain, which can be solved with 4 distinct strategies (see Appendix), we designed two datasets containing different distribution of demonstrated strategies as follows:

Original: D 1 = { 17% , 17% , 33% , 33% } and D 2 = { 33% , 33% , 17% , 17% } , as illustrated in Figure 3.
Revised: D 1 = { 17% , 17% , 33% , 33% } and D 2 = { 33% , 33% , 17% , 17% } , as illustrated in Figure 3.

Original: [Table]
Revised: [Table]

Original: We train relevant baselines over eachdataset and use the interpolation evaluation to analyze the distribution oflearned strategies in the latent space.
Revised: We train relevant baselines over eachdataset and use the interpolation evaluation to analyze the distribution oflearned strategies in the latent space.

Original: Weobserve that Co-GAIL is able to recoverall the strategies, while the baselines arenot able to generate the less representedstrategies or have comparatively fewerof them.
Revised: Weobserve that Co-GAIL is able to recoverall the strategies, while the baselines arenot able to generate the less representedstrategies or have comparatively fewerof them.

Original: Similar results are also observed in the high-dimensional task HRHandover.
Revised: Similar results are also observed in the high-dimensional task HRHandover.

Original: We analyze the range of 3D locations in which the model interpolation performs a successful human-to-robot handover (HR-Handover).
Revised: We analyze the range of 3D locations in which the model interpolation performs a successful human-to-robot handover (HR-Handover).

Original: A top view of the space between the humanand the robot is shown in Figure 4a for two different random seeds.
Revised: A top view of the space between the humanand the robot is shown in Figure 4a for two different random seeds.

Original: Handover locations of MAInfoGAIL tend to concentrate instead of covering the demonstrated distribution.
Revised: Handover locations of MAInfoGAIL tend to concentrate instead of covering the demonstrated distribution.

Original: We also observethat, since DIAYN encourages the learned policy as diverse as possible, it also explores strategiesthat are less plausible in real human-robot collaboration.
Revised: We also observethat, since DIAYN encourages the learned policy as diverse as possible, it also explores strategiesthat are less plausible in real human-robot collaboration.

Original: Co-GAIL (green dots) shows better coverage across the range of the dataset on both the X and Y dimensions (Z in Appendix due to space).
Revised: Co-GAIL (green dots) shows better coverage across the range of the dataset on both the X and Y dimensions (Z in Appendix due to space).

Original: Wefurther investigate the interpolation results of each method and ﬁnd that MA-GAIL, MA-InfoGAIL,
Revised: Wefurther investigate the interpolation results of each method and ﬁnd that MA-GAIL, MA-InfoGAIL,

Original: DIAYN have high success rates during interpolation (Table 2) while performing poorly in replayevaluation (Table 1), indicating that previous methods tend to overﬁt, while Co-GAIL explores thedataset more broadly.
Revised: DIAYN have high success rates during interpolation (Table 2) while performing poorly in replayevaluation (Table 1), indicating that previous methods tend to overﬁt, while Co-GAIL explores thedataset more broadly.

Original: The learned latent space maps to qualitatively different human strategies (Figure 4b): In thisexperiment, we verify if regions of the learned latent space correspond to distinct recognizablehuman strategies.
Revised: The learned latent space maps to qualitatively different human strategies (Figure 4b): In thisexperiment, we verify if regions of the learned latent space correspond to distinct recognizablehuman strategies.

Original: For HR-Handover, we divide the handover locations projected on the horizontal
Revised: For HR-Handover, we divide the handover locations projected on the horizontal

Original: [Figure]
Revised: [Figure]

Original: MA-InfoGAIL
Revised: MA-InfoGAIL

Original: DIAYN
Revised: DIAYN

Original: Co-GAIL
Revised: Co-GAIL

Original: [Figure]
Revised: [Figure]

Original: Strategy
Revised: Strategy

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Dataset MA-InfoGAIL
Revised: Dataset MA-InfoGAIL

Original: DYIAN
Revised: DYIAN

Original: Co-GAIL
Revised: Co-GAIL

Original: [Figure]
Revised: [Figure]

Original: ) HR-Handover interpolation distribution
Revised: a) HR-Handover interpolation distribution

Original: Dataset
Revised: Dataset

Original: MA-InfoGAIL
Revised: MA-InfoGAIL

Original: DIAYN
Revised: DIAYN

Original: Co-GAIL
Revised: Co-GAIL

Original: [Figure]
Revised: [Figure]

Original: plane XY into three regions, obtaining handovers executed at the left ( S 1 ) , middle ( S 2 ) , and right( S 3 ) sides.
Revised: plane XY into three regions, obtaining handovers executed at the left ( S 1 ) , middle ( S 2 ) , and right( S 3 ) sides.

Original: For HR-SeqManip, we determined two possible strategies ( S 1 , S 2 ) depending on thehuman and robot roles (see Appendix).
Revised: For HR-SeqManip, we determined two possible strategies ( S 1 , S 2 ) depending on thehuman and robot roles (see Appendix).

Original: We use the recognition model of each method to compute anaverage latent code for each unseen trajectory and visualize the code in Figure 4b, where each codeis colored by its strategy category.
Revised: We use the recognition model of each method to compute anaverage latent code for the unseen trajectories and visualize the code in Figure 4b, where each codeis colored by its strategy category.

Original: Codes for the same strategy tend to cluster in the same region.
Revised: Codes for the same strategy tend to cluster in the same region.

Original: We observe a better separation of latent strategy learned using Co-GAIL than the other two methods.
Revised: We observe a better separation of latent strategy learned using Co-GAIL than the other two methods.

Original: This shows the value of our diversity objective in Equation 3.
Revised: This shows the value of our diversity objective in Equation 3.

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: Real human user interacting withπ Rco : We performed a proof-of-conceptanalysis with real users to assess representative methods.
Revised: Real human user interacting with 293 ⇡ Rco : We performed a proof-of-conceptanalysis with real users to assess representative methods.

Original: Users interactedwith all methods in randomized order.
Revised: Users interactedwith all methods in randomized order.

Original: We report the success rate out of 20 trials per user per method per domain in
Revised: We report the success rate out of 20 trials per user per method per domain in

Original: Table 3.
Revised: Table 3.

Original: Co-GAIL consistently outperforms the other methods except for one case in the HR-Handover.
Revised: Co-GAIL consistently outperforms the other methods except for one case in the HR-Handover.

Original: In this case, the user performedhandover in a small area that overlapped with the region that MA-InfoGAIL overﬁt to.
Revised: In this case, the user performedhandover in a small area that overlapped with the region that MA-InfoGAIL overﬁt to.

Original: Co-GAILshowed the best adaptation to diverse strategies across users and trials.
Revised: Co-GAILshowed the best adaptation to diverse strategies across users and trials.

Original: While a small sample size,these promising results are the ﬁrst demonstration of a learning method that succeeds at adapting toreal users in these challenging tasks with continuous motion and varied strategies.
Revised: While a small sample size,these promising results are the ﬁrst demonstration of a learning method that succeeds at adapting toreal users in these challenging tasks with continuous motion and varied strategies.

Original:  Conclusions and Future Work
Revised:  Conclusions and Future Work

Original: We present Co-GAIL, a method for training robot assistants that can handle diverse human behaviors.
Revised: We present Co-GAIL, a method for training robot assistants that can handle diverse human behaviors.

Original: Our method represents a novel approach for taking advantage of simulated environments in
Revised: Our method represents a novel approach for taking advantage of simulated environments in

Original: HRI research.
Revised: HRI research.

Original: This is achieved by leveraging data of human-human collaboration demonstrationsas guidance to concurrently generate simulated interactive behaviors and train a human-robot collaborative policy.
Revised: This is achieved by leveraging data of human-human collaboration demonstrationsas guidance to concurrently generate simulated interactive behaviors and train a human-robot collaborative policy.

Original: We test Co-GAIL in a 2D game and two high-dimensional manipulation tasks,with both procedural evaluation and online human-in-the-loop evaluation.
Revised: We test Co-GAIL in a 2D game and two high-dimensional manipulation tasks,with both procedural evaluation and online human-in-the-loop evaluation.

Original: While Co-GAIL outperforms other state-of-the-art methods in challenging HRC tasks, a few challenges remain open forfuture works.
Revised: While Co-GAIL outperforms other state-of-the-art methods in challenging HRC tasks, a few challenges remain open forfuture works.

Original: In our current implementation, the roles of leaders and followers must be assignedto the human agent and the robot agent in advance.
Revised: In our current implementation, the roles of leaders and followers must be assignedto the human agent and the robot agent in advance.

Original: Enabling the agents to ﬂexibly switch rolesonline would further improve the collaboration experience.
Revised: Enabling the agents to ﬂexibly switch rolesonline would further improve the collaboration experience.

Original: Another future direction is to apply thelearned policy to a real robot.
Revised: Another future direction is to apply thelearned policy to a real robot.

Original: A possible approach is to align the observation space of the simulatedenvironment and the real world through state estimation and employ domain randomization [49] toallow the agent to be robust against domain shift in sim-to-real transfer.Appendix A Implementation Details This section describes the network architectures of each model used in the main paper.
Revised: A possible approach is to align the observation space of the simulatedenvironment and the real world through state estimation and employ domain randomization [49] toallow the agent to be robust against domain shift in sim-to-real transfer.

Original: Co-policy: Our co-policy model π co consists of an actor and a critic network.

Original: Both networks aremultilayer perceptrons with 4 fully-connected layers of size 128 each.

Original: The output layer of the criticis (128, 1).

Original: The output feature of the actor network is further processed by a fully-connected layerinto a normal distribution with the mean and standard deviation for each action dimension.

Original: Theoutput action is then branched into the human action a H and robot action a R .

Original: These actions will besampled from the distribution during training and the one with maximum probability will be selectedduring testing.

Original: Strategy recognition network: The strategy recognition network ψ is a multilayer perceptronswith 2 fully-connected layers with the output size of 2 (The strategy code in our experiments is 2ﬂoat numbers in the range ( − 1 . 0 , 1 . 0) ).

Original: To regularize the output strategy prediction, we use a tanhfunction to process the output prediction.

Original: Discriminator network: The discriminator networks D is a multilayer perceptrons with 3 fullyconnected layers (64 , 64 , 1) , followed by a tanh function to regularize the output within ( − 1 . 0 , 1 . 0) .

Original: B Training DetailsB.1 GAIL

Original: The standard GAIL objective detailed in Equation 5 typically uses a sigmoid cross-entropy lossfunction.

Original: However, we empirically ﬁnd this loss will cause the vanishing of gradients because of thesigmoid function, especially in the high-dimensional manipulation tasks (HR-Handover and HRSeqManip).

Original: In this work, we use the loss function proposed by least-squares GAN (LSGAN)[50]for the high-dimensional manipulation tasks.

Original: The training objective of the discriminator D is:

Original: [Equation]

Original: During the training, the strategy code z is randomly sampled.

Original: The original implementation ofinfoGAIL[25] uses a uniformly random sampling over code space.

Original: However, we empirically foundthat random sampling might cause unstable results between different training seeds.

Original: Therefore, weuse a grid-based sampling on our strategy code.

Original: The code space ( − 1 . 0 , 1 . 0) × ( − 1 . 0 , 1 . 0) is ﬁrstdiscretized into 25 grid points.

Original: During the training, each grid point will be selected in a cycle orderand a uniformly sampled noise ( [ − 0 .

Original: 2] for each code dimension in our experiments) will beadded to the selected grid point.

Original: This setting could make sure that the replay buffer covers most ofthe strategies of the learned policy and improve the stability of the training results across differenttraining seeds.

Original: In the ﬁnal learning objective of Co-GAIL (Equation 4), two hyperparameters λ 1 and λ 2 are introduced.

Original: For all three experiments, we used λ 1 = λ 2 = 0 .

Original: We found that large λ 1 and λ 2would cause unstable training procedures of the GAIL algorithm, where the objectives that measurethe reconstruction error of the strategy code and human actions in Equation 4 dominate the lossfunction and cause the model to neglect the ﬁrst GAIL learning objective.

Original: We empirically foundλ 1 = λ 2 = 0 .

Original: 1 achieve the most stable learning performance in all three task environments.

Original: B.2 PPO

Original: The co-policy model is trained by the PPO[46].

Original: [-For all three experiments, we use the learning rate
-]
Revised: [-For all three experiments, we use the learning rate
-]

Original: e − 4 with a linear decay based on the percentage of the completed training epochs over the totalnumber of episodes.

Original: The size of the replay buffer is 6000 for the ﬁrst two experiments (2D-FetchQuest, HR-Handover) and 10000 for the HR-SeqManip.

Original: The rest of the hyperparameters are thesame as the default PPO algorithm[46].

Original: [Figure]

Original: Robot

Original: [Figure]

Original: Human

Original: Robot

Original: Button

Original: Door

Original: Treasure

Original: Destination

Original: C Experimental Details

Original: [Figure]

Original: C.1 2D-Fetch-Quest

Original: [Figure]

Original: Task details.

Original: This environment is adapted from the Fetch-Quest collaborative game in Super Mario

Original: Party.

Original: The human and robot agents need to work together to fetch the treasure and reach the destination.

Original: In Figure 5, we illustrate the meaning of each icon in the ﬁrst row.

Original: At the beginning of thegame, two treasures are locked in the rooms located in two corners of the map.

Original: For each agent, theonly way to fetch a treasure is to let its collaborator press the button beside the room for it.

Original: Afterthe collaborator press the button, the agent could enter the room and get the treasure.

Original: To win thegame, both agents should get a treasure and reach the closest destination.

Original: Therefore, the agent whohas already get the treasure will help the other agent to fetch its treasure.

Original: There are four types ofstrategies to tackle this game.

Original: As is shown in Figure 5, strategy 1 and 2 is different in the order ofwho gets the treasure ﬁrst.

Original: In strategies 3 and 4, the human and robot will switch their roles and goto the other side of the map to complete the collaboration.

Original: State and action space.

Original: The continuous action space of both human and robot agents is a 2Dtranslation (∆ x, ∆ y ) in the map, where ∆ x ∈ [ − 1 . 0 , 1 . 0] , ∆ y ∈ [ − 1 . 0 , 1 . 0] .

Original: [-The size of the map is8 . 
-]
Revised: [-The size of the map is8 . 
-]

Original: × 8 . 0 .

Original: The input state of the co-policy model consists of the 2D locations of the human and robotagent, the positions of the two treasures in the map, and two binary values that indicate whether eachdoor of the room is opened or not.

Original: Demonstrations.

Original: The human-human collaboration demonstrations are collected by two users withtwo joysticks on an Xbox gaming controller.

Original: During training, 60 human-human demonstrations aregiven.

Original: The average step size of the given human-human demonstration is 145 .

Original: No external rewardsignals are given during the training.

Original: Training and evaluation details.

Original: For each method, the training takes 1000 episodes and eachepisode contains 6000 environment steps.

Original: The checkpoint is saved every 30 training episodes.

Original: Eachmethod is trained with three different seeds: 300 , 400 , 500 in our experiments.

Original: The maximum ofthe mean success rate among all checkpoints across three seeds and its 95% conﬁdence interval arereported for both replay evaluation and interpolation in Table.

Original: 1 and Table.

Original: 2 for each method.

Original: During the replay evaluation, 60 testing demonstrations (not included in the training set) are used totest each method.

Original: During the interpolation, 100 uniformly sampled strategies are used to generatedifferent behaviors for each method.

Original: [Figure]

Original: C.2 HR-Handover

Original: Task details.

Original: In this environment, the human agent will ﬁrst fetch the object in front of it andhandover the object to the robot.

Original: A successful completion of the task happens only when the robotholding the object and the distance between the end effectors of the human and robot is larger than 10centimeter.

Original: The diversity of the collaborative behaviors in this task is the position of the object whenhandover happens.

Original: As shown in Figure 6, the handover can happen in a wide range of locations.

Original: Wepost-hoc categorize these handover locations to left, right, or center only for analysis purposes.

Original: State and action space.

Original: The continuous action space of both human and robot agents is the 6D posechanges (∆ x, ∆ y, ∆ z, ∆ R x , ∆ R y , ∆ R z ) of the end-effector relative to the previous time step and aﬂoat value g ∈ [ − 1 . 0 , 1 . 0] describing the gripper state ( − 1 refers to gripper fully closed and 1 refersto gripper fully opened), where ∆ x, ∆ y, ∆ z ∈ [ − 1 . 0 , 1 . 0] centimeters and ∆ R x , ∆ R y , ∆ R z ∈[ − 1 . 0 , 1 . 0] degrees.

Original: The environment state of this task includes the 6D pose of the end-effectors ofboth human and robot in their base frame, the relative position between the object to the end-effectorof each agent and the orientation of the object.

Original: Demonstrations.

Original: The human-human collaboration demonstrations are collected by two users withthe phone teleportation system RoboTurk[47, 48].

Original: Without haptic feedback, it is hard for the humanagent to identify whether the robot has held the object ﬁrmly or not.

Original: Therefore, we simplify thecontrol system that after the robot holds the object, the gripper of the human agent will automaticallyopen.

Original: During training, 160 human-human demonstrations with an average step size 211 is used totrain the model.

Original: Training and evaluation details.

Original: For each method, the training takes 300 episodes with 6000 environment steps each episode.

Original: The checkpoints are saved every 10 training episodes.

Original: The evaluationsettings are the same as the 2D-Fetch-Quest.

Original: To visualize the mapping between the latent space todifferent strategies (Figure. 4), we classiﬁed the strategies into three categories based on the positionwhere handover happens from the top-down view (Figure. 6).

Original: Additional results.

Original: Here we show an additional interpolation result on the handover positions ofeach method in Figure.

Original: Our proposed method Co-GAIL successfully covers the data distributionwhile the baseline methods focus on a partial region.

Original: C.3 HR-SeqManip

Original: In this environment, the human agent will ﬁrst fetch the object on the cabinet.

Original: The goal is to put theobject into the drawer.

Original: As is shown in Figure.

Original: 7, there are two strategies to complete the task.

Original: Instrategy 1, the human will ﬁrst handover the object to the robot and then open the drawer.

Original: After therobot putting the object into the drawer, the human will close the drawer to complete the task.

Original: Instrategy 2, the robot will open the drawer for the human to put the object into.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: The action space of both human and robot agents is the same as the HR-Handover task.

Original: The environment state has an additional relative position between the handle of the drawer to the end-effectorof each agent.

Original: During training, 120 human-human demonstrations with an average step size 404 isused to train the model.

Original: For each method, the training takes 1000 episodes with 10000 environmentsteps each episode.

Original: The checkpoints are saved every 30 training episodes.

Original: The evaluation settingsare the same

Original: [Figure]

Original: ca ti on [ m ]

Original: Y - l o ca ti on [ m ]

Original: [Figure]

Original: D Real-human Evaluation Details

Original: Evaluation protocol.

Original: To test whether each method can handle diverse human behavior, we invitefour real-human operators to conduct the real-human evaluation.

Original: We select three methods that havethe best performance in the replay evaluation ( MA-GAIL , MA-InfoGAIL , Co-GAIL ) to compare inthis experiment.

Original: The BC baseline is also included for reference.

Original: The checkpoint with the highestreplay evaluation success rate of each method will be ﬁrst loaded.

Original: Each human operator will perform20 rounds of evaluation.

Original: In each round, the robot will be controlled by the models learned byeach method in random order.

Original: The human operator does not know which model is it currentlyworking with.

Original: To further make sure the comparison between the methods is fair, human operatorsare suggested to maintain similar motions for different trials in the same round.

Original: The ﬁnal averagesuccess rate over 20 rounds of each method is reported in Table.

Original: Controller.

Original: For the 2D-Fetch-Quest, the human operator will use one joystick to control the humanagents in the game.

Original: For the HR-Handover and HR-SeqManip, the human operator will use the phoneteleportation system RoboTurk[47, 48] to control the 6D pose of the end-effector of the humanoid.

Original: E Applicability to real robot

Original: Due to COVID-19, we were not able to explore the potential of deploying our system onto a realworld robot platform.

Original: Our plan for simulation-to-real world transfer involves two steps.

Original: First, ourmodels currently operate on low-dimensional state space including (1) robot and human proprioceptive information and (2) object pose information.

Original: In the real world, we could easily obtain robotproprioceptive information through the robot’s internal APIs and match that of the simulator.

Original: Wecould obtain human’s hand information through an off-the-shelf hand tracker [51], and the objectpose through a category-agnostic 6DoF pose tracking system [52].

Original: This allows us to match the statespace between the simulated environment and the real world.

Original: Second, as indicated by prior works,there are likely mismatches between the simulated and real-world physics.

Original: We plan to address suchgap in physical dynamics through domain randomization, i.e., we will expose our agent with wideranges of possible physical dynamics settings during training.

Original: Past work [49] showed that suchtraining strategy could largely alleviate the domain shift in dynamics, allowing the agent to adapt tovarying dynamics in the physical world.