Original: HERD: Continuous Human-to-Robot Evolution for Learning from Human Demonstration
Revised: HERD: Continuous Human-to-Robot Evolution for Learning from Human Demonstration

Original: [Figure]
Revised: [Figure]

Original: Abstract: The ability to learn from human demonstration endows robots with the ability to automate various tasks.
Revised: Abstract: The ability to learn from human demonstration endows robots with the ability to automate various tasks.

Original: However, directly learning from human demonstration is challenging since the structure of the human hand can be very different from the desired robot gripper.
Revised: However, directly learning from human demonstration is challenging since the structure of the human hand can be very different from the desired robot gripper.

Original: In this work, we show that manipulation skills can be transferred from a human to a robot through the use of micro-evolutionary reinforcement learning, where a ﬁve-ﬁnger human dexterous hand robot gradually evolves into a commercialtwo-ﬁnger-gripper robot, while repeated interacting in a physics simulator to continuously update the policy that is ﬁrst learned from human demonstration.
Revised: In this work, we show that manipulation skills can be transferred from a human to a robot through the use of micro-evolutionary reinforcement learning, where a ﬁve-ﬁnger human dexterous hand robot gradually evolves into a commercial robot, while repeated interacting in a physics simulator to continuously update the policy that is ﬁrst learned from human demonstration.

Original: To deal with the high dimensions of robot parameters, we propose an algorithm for multi-dimensional evolution path searching that allows joint optimization of both the robot evolution path and the policy.
Revised: To deal with the high dimensions of robot parameters, we propose an algorithm for multi-dimensional evolution path searching that allows joint optimization of both the robot evolution path and the policy.

Original: Through experiments on human object manipulation datasets, we show that our framework can efﬁciently transfer the expert human agent policy trained from human demonstrations in diverse modalities to target commercial robots.
Revised: Through experiments on human object manipulation datasets, we show that our framework can efﬁciently transfer the expert human agent policy trained from human demonstrations in diverse modalities to target commercial robots.

Original:  Introduction
Revised:  Introduction

Original: Learning from human demonstrations [3] is a promising direction to enable robots to perform diverse manipulation capabilities.
Revised: Learning from human demonstrations [3] is a promising direction to enable robots to perform diverse manipulation capabilities.

Original: Recent large-scale visual datasets of human activity [2, 4, 5, 6, 7] have highlighted the need for enabling robotic agents to learn from human activities to perform diverse tasks  environments generally follow the pipeline of demonstration conversion , i.e. (1) convert demonstrated human states to robot states and then (2) train the robot policy .
Revised: Recent large-scale visual datasets of human activity [2, 4, 5, 6, 7] have highlighted the need for enabling robotic agents to learn from human activities to perform diverse tasks in real-world environments generally follow the pipeline of demonstration conversion , i.e. (1) convert demonstrated human states to robot states and then (2) train the robot policy .

Original: Though step (2) can be solved by existing imitation learning approaches [8, 9, 10, 1], step (1) is extremely difﬁcult.
Revised: Though step (2) can be solved by existing imitation learning approaches [8, 9, 10, 1], step (1) is extremely difﬁcult.

Original: Efforts to address step (1) include human-robot observation matching such as [11], and converting human demonstrations to robots by human-robot interaction [12, 13] or human-in-the-loop teleoperation [14, 15].
Revised: Efforts to address step (1) include human-robot observation matching such as [11], and converting human demonstrations to robots by human-robot interaction [12, 13] or human-in-the-loop teleoperation [14, 15].

Original: However, these solutions are highly situational and task-speciﬁc and require signiﬁcant human intervention for each individual task, therefore are not scalable and cannot automatically generalize to new tasks.
Revised: However, these solutions are highly situational and task-speciﬁc and require signiﬁcant human intervention for each individual task, therefore are not scalable and cannot automatically generalize to new tasks.

Original: We propose a new paradigm for learning from human demonstration — policy transformation , i.e.
Revised: We propose a new paradigm for learning from human demonstration — policy transformation , i.e.

Original: (1) train policy on embodied human hand dexterous robot and then (2) transfer the policy from dexterous hand robot to the target robot .
Revised: (1) train policy on embodied human hand dexterous robot and then (2) transfer the policy from dexterous hand robot to the target robot .

Original: Many existing methods have been shown successful in learning control policy for dexterous hand robots from human demonstration [1, 16] and can easily solve step (1) of our pipeline.
Revised: Many existing methods have been shown successful in learning control policy for dexterous hand robots from human demonstration [1, 16] and can easily solve step (1) of our pipeline.

Original: A general and automatic solution of step (2) would render a scalable solution of learning from human demonstration.
Revised: A general and automatic solution of step (2) would render a scalable solution of learning from human demonstration.

Original: The key challenge to step (2) is the huge dynamics discrepancy between the dexterous hand robot and the target robot.
Revised: The key challenge to step (2) is the huge dynamics discrepancy between the dexterous hand robot and the target robot.

Original: The discrepancy stems from the mismatch in robot morphology and kinematics.
Revised: The discrepancy stems from the mismatch in robot morphology and kinematics.

Original: Unfortunately, statistical matching imitation learning approaches such as [8, 9, 10, 17] assume the teacher and student robotic agents share the same or similar transition dynamics, therefore are ineffective in solving the problem.
Revised: Unfortunately, statistical matching imitation learning approaches such as [8, 9, 10, 17] assume the teacher and student robotic agents share the same or similar transition dynamics, therefore are ineffective in solving the problem.

Original: In this paper, we present a framework named HERD (short for H uman
Revised: In this paper, we present a framework named HERD (short for H uman

Original: E volution to R obot for Learning from D emonstration) with a new perspective on solving the problem of learning from human demonstration — continuous robot evolution .
Revised: E volution to R obot for Learning from D emonstration) with a new perspective on solving the problem of learning from human demonstration — continuous robot evolution .

Original: Following the philosophy introduced in the seminal work of REvolveR [18], the core idea of our framework is to deﬁne a continuous interpolation between the dexterous hand robot and a target robot that have different morphology.
Revised: Following the philosophy introduced in the seminal work of REvolveR [18], the core idea of our framework is to deﬁne a continuous interpolation between the dexterous hand robot and a target robot that have different morphology.

Original: Then an expert dexterous hand robot policy learned with methods such as [1, 16] can be transferred to the target robot through training on a sequence of intermediate robots that gradually evolve into the target robot.
Revised: Then an expert dexterous hand robot policy learned with methods such as [1, 16] can be transferred to the target robot through training on a sequence of intermediate robots that gradually evolve into the target robot.

Original: The policy is continuously updated through repeated interaction in a physics engine that contains the intermediate robots, as illustrated in Figure 1.
Revised: The policy is continuously updated through repeated interaction in a physics engine that contains the intermediate robots, as illustrated in Figure 1.

Original: To instantiate the idea, we develop two solutions for the evolution from ﬁve-ﬁnger dexterous robot to two target commercial robots respectively: a Sawyer robot with a two-ﬁnger Rethink gripper, and a Jaco robot with a three-ﬁnger Jaco gripper.
Revised: To instantiate the idea, we develop two solutions for the evolution from ﬁve-ﬁnger dexterous robot to two target commercial robots respectively: a Sawyer robot with a two-ﬁnger Rethink gripper, and a Jaco robot with a three-ﬁnger Jaco gripper.

Original: The solutions precisely match the dynamics of target commercial grippers.
Revised: The solutions precisely match the dynamics of target commercial grippers.

Original: To deal with the high dimensions of robot parameters, we propose an algorithm for automatically searching evolution paths in high-dimensional robot parameter space that allows joint optimization of both robot evolution and the policy, based on online estimation of reward gradient with respect to robot evolution.
Revised: To deal with the high dimensions of robot parameters, we propose an algorithm for automatically searching evolution paths in high-dimensional robot parameter space that allows joint optimization of both robot evolution and the policy, based on online estimation of reward gradient with respect to robot evolution.

Original: We conduct experiments on transferring human expert demonstrations of manipulation tasks in diverse modalities to the target commercial robots.
Revised: We conduct experiments on transferring human expert demonstrations of manipulation tasks in diverse modalities to the target commercial robots.

Original: We show that HERD can successfully transfer the expert dexterous hand robot policy trained from Hand Manipulation Suite sensor glove demonstrations [1] to the target robots even with sparse rewards.
Revised: We show that HERD can successfully transfer the expert dexterous hand robot policy trained from Hand Manipulation Suite sensor glove demonstrations [1] to the target robots even with sparse rewards.

Original: On DexYCB dataset [2], we demonstrate that our approach can also transfer the expert policy trained with visual human demonstrations to the target robots.
Revised: On DexYCB dataset [2], we demonstrate that our approach can also transfer the expert policy trained with visual human demonstrations to the target robots.

Original: In addition, we show that the proposed multi-dimensional robot evolution path search algorithm can signiﬁcantly improve the policy transferring efﬁciency.
Revised: In addition, we show that the proposed multi-dimensional robot evolution path search algorithm can signiﬁcantly improve the policy transferring efﬁciency.

Original:  Preliminaries
Revised:  Preliminaries

Original: MDP Preliminary We consider a continuous control problem formulated as Markov Decision
Revised: MDP Preliminary We consider a continuous control problem formulated as Markov Decision

Original: Process (MDP).
Revised: Process (MDP).

Original: It is deﬁned by a tuple ( S , A , T , R, γ ) , where S ⊆ R S is the state space, A ⊆ R A is the action space, T : S ×
Revised: It is deﬁned by a tuple ( S , A , T , R, γ ) , where S ⊆ R S is the state space, A ⊆ R A is the action space, T : S ×

Original: A → S is the transition function, R : S ×
Revised: A → S is the transition function, R : S ×

Original: A → R is the reward function, and γ ∈ [0 , 1] is the discount factor.
Revised: A → R is the reward function, and γ ∈ [0 , 1] is the discount factor.

Original: A policy π : S → A maps a state to an action where π ( a | s ) is the probability of choosing action a at state s .
Revised: A policy π : S → A maps a state to an action where π ( a | s ) is the probability of choosing action a at state s .

Original: Suppose M is the set of all MDPs and ρ π,M = (cid:80) ∞
Revised: Suppose M is the set of all MDPs and ρ π,M = (cid:80) ∞

Original: t =

Revised: t =

Original: γ t R ( s t , a t ) is the episode discounted reward with policy π on MDP M ∈ M .
Revised: γ t R ( s t , a t ) is the episode discounted reward with policy π on MDP M ∈ M .

Original: The optimal policy π ∗ M on MDP M is the one that maximizes the expected value of ρ π,M .
Revised: The optimal policy π ∗ M on MDP M is the one that maximizes the expected value of ρ π,M .

Original: REvolveR [18] Preliminary Liu et al.
Revised: REvolveR [18] Preliminary Liu et al.

Original: [18] proposed a technique named REvolveR for transferring policies from a source robot to a target robot.
Revised: [18] proposed a technique named REvolveR for transferring policies from a source robot to a target robot.

Original: The core idea is to deﬁne an evolutionary sequence of intermediate robots that connects the source to the target robot.
Revised: The core idea is to deﬁne an evolutionary sequence of intermediate robots that connects the source to the target robot.

Original: Given source and target robots represented by two MDPs M S , M T ∈ M respectively, REvolveR deﬁnes a continuous function
Revised: Given source and target robots represented by two MDPs M S , M T ∈ M respectively, REvolveR deﬁnes a continuous function

Original: E : [0 , 1] → M where E (0) =
Revised: E : [0 , 1] → M where E (0) =

Original: M S , E (1) =
Revised: M S , E (1) =

Original: M T .
Revised: M T .

Original: Then an expert policy π ∗ E (0) on the source robot E (0) is optimized by sequentially interacting with each intermediate robot in the sequence
Revised: Then an expert policy π ∗ E (0) on the source robot E (0) is optimized by sequentially interacting with each intermediate robot in the sequence

Original: [Figure]
Revised: [Figure]

Original: Notation We use bold letters to denote the vector variables.
Revised: Notation We use bold letters to denote the vector variables.

Original: Specially, 0 and 1 are the all-zero and all-one vectors with proper dimensions respectively.
Revised: Specially, 0 and 1 are the all-zero and all-one vectors with proper dimensions respectively.

Original: S D − 1 ( ξ ) =
Revised: S D − 1 ( ξ ) =

Original: { x ∈ R D
Revised: { x ∈ R D

Original: |
Revised: |

Original: ||
Revised: ||

Original: x
Revised: x

Original: || 2 = ξ } is a ( D − 1) -sphere
Revised: || 2 = ξ } is a ( D − 1) -sphere

Original: (i.e. a sphere in R D space) with radius ξ ∈ R + .
Revised: (i.e. a sphere in R D space) with radius ξ ∈ R + .

Original: (cid:12) is element-wise product.
Revised: (cid:12) is element-wise product.

Original:  Method 3.1 Human Demonstration to Dexterous Robot Expert Policy
Revised:  Method 3.1 Human Demonstration to Dexterous Robot Expert Policy

Original: Recent works [1, 16] have shown success in learning ﬁve-ﬁnger dexterous robot policy from human hand demonstrations in diverse modalities, such as sensor glove signals [1] and visual data [16], which made the ﬁrst step of our proposed paradigm possible.
Revised: Recent works [1, 16] have shown success in learning ﬁve-ﬁnger dexterous robot policy from human hand demonstrations in diverse modalities, such as sensor glove signals [1] and visual data [16], which made the ﬁrst step of our proposed paradigm possible.

Original: Inspired by this research progress, we adopt similar approaches to train expert policies on a dexterous robot as illustrated in Figure 1(1)(2).
Revised: Inspired by this research progress, we adopt similar approaches to train expert policies on a dexterous robot as illustrated in Figure 1(1)(2).

Original: Sensor Glove Demonstration When recorded in sensor glove signals, human demonstrations are directly converted to states and actions in a simulator [19] from which a policy is trained with algorithms such as DAPG [1].
Revised: Sensor Glove Demonstration When recorded in sensor glove signals, human demonstrations are directly converted to states and actions in a simulator [19] from which a policy is trained with algorithms such as DAPG [1].

Original: We use the same procedure and obtain the expert policy as in [1].
Revised: We use the same procedure and obtain the expert policy as in [1].

Original: Visual Demonstration We develop a toolchain for training dexterous policies from human visual demonstrations of manipulation.
Revised: Visual Demonstration We develop a toolchain for training dexterous policies from human visual demonstrations of manipulation.

Original: Our toolchain includes human and object pose estimation with visual perception followed by pose ﬁtting of the dexterous robot via inverse kinematics (IK).
Revised: Our toolchain includes human and object pose estimation with visual perception followed by pose ﬁtting of the dexterous robot via inverse kinematics (IK).

Original: We then train the policy by setting the episodes’ initial state to be the state of the grasping frame, including poses and joint velocity.
Revised: We then train the policy by setting the episodes’ initial state to be the state of the grasping frame, including poses and joint velocity.

Original: During training, the initial state gradually moves backward in time until the desired starting state.
Revised: During training, the initial state gradually moves backward in time until the desired starting state.

Original: For more details, please refer to the supplementary.
Revised: For more details, please refer to the supplementary.

Original: 3.2 Continuous Dexterous Hand to Target Commercial Robots Evolution
Revised: 3.2 Continuous Dexterous Hand to Target Commercial Robots Evolution

Original: A core hypothesis in our proposed paradigm of learning from human demonstration is the feasibility of interpolation between a human-hand-like ﬁve-ﬁnger dexterous robot and a target commercial robot.
Revised: A core hypothesis in our proposed paradigm of learning from human demonstration is the feasibility of interpolation between a human-hand-like ﬁve-ﬁnger dexterous robot and a target commercial robot.

Original: In this paper, we present solutions for dexterous-to-commercial-robot interpolation for two target commercial robots to show this.
Revised: In this paper, we present solutions for dexterous-to-commercial-robot interpolation for two target commercial robots to show this.

Original: Since both the dexterous hand robot and the target robots are complicated in structure, how to design the evolution from the former to the latter is not straightforward and has a large design space.
Revised: Since both the dexterous hand robot and the target robots are complicated in structure, how to design the evolution from the former to the latter is not straightforward and has a large design space.

Original: The design choice can signiﬁcantly affect the policy transfer performance.
Revised: The design choice can signiﬁcantly affect the policy transfer performance.

Original: Following [18], we ﬁrst match the robot morphology between the dexterous and the target robot.
Revised: Following [18], we ﬁrst match the robot morphology between the dexterous and the target robot.

Original: With the same morphology, we then design a shared high-dimensional robot parameter space that allows deﬁning an intermediate robot by simply setting these parameters.
Revised: With the same morphology, we then design a shared high-dimensional robot parameter space that allows deﬁning an intermediate robot by simply setting these parameters.

Original: The evolution includes both the dexterous hand and arm.
Revised: The evolution includes both the dexterous hand and arm.

Original: Dexterous Hand to Commercial Gripper Evolution To match the number of ﬁngers, we keep the thumb and gradually shrink the redundant ﬁngers to be zero-size and zero-mass so that they eventually disappear.
Revised: Dexterous Hand to Commercial Gripper Evolution To match the number of ﬁngers, we keep the thumb and gradually shrink the redundant ﬁngers to be zero-size and zero-mass so that they eventually disappear.

Original: We also gradually change the range of the ﬁnger joints to match the desired joint conﬁguration.
Revised: We also gradually change the range of the ﬁnger joints to match the desired joint conﬁguration.

Original: In addition, we add new position servo joints to the remaining ﬁngers.
Revised: In addition, we add new position servo joints to the remaining ﬁngers.

Original: The position servo joints are initially frozen and evolve to have the same range as the target ﬁngers.
Revised: The position servo joints are initially frozen and evolve to have the same range as the target ﬁngers.

Original: Dexterous Arm to Commercial Robot Arm Evolution A major challenge of arm evolution is the different ways of arm mounting.
Revised: Dexterous Arm to Commercial Robot Arm Evolution A major challenge of arm evolution is the different ways of arm mounting.

Original: In simulation, the dexterous robot elbow joint is usually modeled as a virtual free joint that can move freely in the 3D space, while the commercial robot arms are usually mounted on ﬁxed bases.
Revised: In simulation, the dexterous robot elbow joint is usually modeled as a virtual free joint that can move freely in the 3D space, while the commercial robot arms are usually mounted on ﬁxed bases.

Original: Another challenge is the huge difference in the numbers of arm joints and joint physical parameters.
Revised: Another challenge is the huge difference in the numbers of arm joints and joint physical parameters.

Original: Our solution is to attach the free joint of the dexterous arm to the end effector of the target robot and treat the entire dexterous robot as a de facto large gripper.
Revised: Our solution is to attach the free joint of the dexterous arm to the end effector of the target robot and treat the entire dexterous robot as a de facto large gripper.

Original: During evolution, both the dexterous arm and the elbow free joint gradually shrink to zero so that the dexterous robot eventually evolves to be ﬁrmly attached to the target robot end effector.
Revised: During evolution, both the dexterous arm and the elbow free joint gradually shrink to zero so that the dexterous robot eventually evolves to be ﬁrmly attached to the target robot end effector.

Original: Our robot evolution solution includes the changing of D independent robot parameters such as body sizes and mass, and joint ranges and damping, where D = 40 for target Sawyer robot and D =for target Jaco robot.
Revised: Our robot evolution solution includes the changing of D independent robot parameters such as body sizes and mass, and joint ranges and damping, where D = 40 for target Sawyer robot and D =for target Jaco robot.

Original: The evolution process for Sawyer robot is illustrated in Figure 2.
Revised: The evolution process for Sawyer robot is illustrated in Figure 2.

Original: 3.3 Multi-dimensional Continuous Robot Evolution for Policy Transfer
Revised: 3.3 Multi-dimensional Continuous Robot Evolution for Policy Transfer

Original: As shown in Section 3.2, the evolution from a human-like dexterous robot to a target commercial robot is an extremely high-dimensional problem in robot parameter space (e.g. D = 42 ).
Revised: As shown in Section 3.2, the evolution from a human-like dexterous robot to a target commercial robot is an extremely high-dimensional problem in robot parameter space (e.g. D = 42 ).

Original: This means there exist numerous choices of intermediate robot sequences that connect source to target robots.
Revised: This means there exist numerous choices of intermediate robot sequences that connect source to target robots.

Original: However, previous work REvolveR [18] assumes the same and uniform evolution progress hard-coded for all robot parameters and could be sub-optimal.
Revised: However, previous work REvolveR [18] assumes the same and uniform evolution progress hard-coded for all robot parameters and could be sub-optimal.

Original: We envision that when transferring the policy through intermediate robots, the optimal robot evolution should be ﬂexible and able to smartly bypass difﬁcult robot conﬁgurations to reach the target robot efﬁciently.
Revised: We envision that when transferring the policy through intermediate robots, the optimal robot evolution should be ﬂexible and able to smartly bypass difﬁcult robot conﬁgurations to reach the target robot efﬁciently.

Original: In this section, we introduce an algorithm for automatically ﬁnding such optimized evolution strategy.
Revised: In this section, we introduce an algorithm for automatically ﬁnding such optimized evolution strategy.

Original: We formulate the problem as a joint optimization of both robot evolution and the policy.
Revised: We formulate the problem as a joint optimization of both robot evolution and the policy.

Original: Problem Statement Suppose the source and target robot morphology is matched using the method in [18] or Section 3.2.
Revised: Problem Statement Suppose the source and target robot morphology is matched using the method in [18] or Section 3.2.

Original: Therefore the D kinematics parameters of the source and target robots can be mapped to the same space denoted as θ S , θ T ∈ R D .
Revised: Therefore the D kinematics parameters of the source and target robots can be mapped to the same space denoted as θ S , θ T ∈ R D .

Original: Continuous function F : [0 , 1] D
Revised: Continuous function F : [0 , 1] D

Original: → M deﬁnes an intermediate robot by interpolation between all pairs of kinematic parameters θ =
Revised: → M deﬁnes an intermediate robot by interpolation between all pairs of kinematic parameters θ =

Original: ( 1 − α ) (cid:12) θ S
Revised: ( 1 − α ) (cid:12) θ S

Original: + α (cid:12) θ T where α ∈ [0 , 1] D is the evolution parameter that describes the evolution progress of each of the D kinematics parameters.
Revised: + α (cid:12) θ T where α ∈ [0 , 1] D is the evolution parameter that describes the evolution progress of each of the D kinematics parameters.

Original: Given an expert policy π ∗ F ( 0 ) on the source robot F ( 0 ) , the overall goal is to ﬁnd the optimal policy π ∗ F ( 1 ) on target robot F ( 1 ) .
Revised: Given an expert policy π ∗ F ( 0 ) on the source robot F ( 0 ) , the overall goal is to ﬁnd the optimal policy π ∗ F ( 1 ) on target robot F ( 1 ) .

Original: Note that the formulation of
Revised: Note that the formulation of

Original: REvolveR [18] is a special case of our problem setting with α = α · 1 and α ∈ [0 , 1] .
Revised: REvolveR [18] is a special case of our problem setting with α = α · 1 and α ∈ [0 , 1] .

Original: Differentiable Evolution Path Search (DEPS)
Revised: Differentiable Evolution Path Search (DEPS)

Original: We adopt multi-dimensional robot evolution as a solution to the above problem as illustrated in Figure 3(a).
Revised: We adopt multi-dimensional robot evolution as a solution to the above problem as illustrated in Figure 3(a).

Original: Suppose there is a robot evolution path τ
Revised: Suppose there is a robot evolution path τ

Original: = ( F ( α 0 ) , F ( α 1 ) , F ( α 2 ) , . .
Revised: = ( F ( α 0 ) , F ( α 1 ) , F ( α 2 ) , . .

Original: , F ( α K − 1 ) , F ( α K )) where α 0 = 0 , α K = 1 .
Revised: , F ( α K − 1 ) , F ( α K )) where α 0 = 0 , α K = 1 .

Original: In k -th phase, the policy optimization objective is to maximize the expected reward E [ ρ π,F ( α k ) ] on robot F ( α k ) .
Revised: In k -th phase, the policy optimization objective is to maximize the expected reward E [ ρ π,F ( α k ) ] on robot F ( α k ) .

Original: For all k , the evolution parameter step size ξ = ||
Revised: For all k , the evolution parameter step size ξ = ||

Original: α k − α k
Revised: α k − α k

Original: +

Revised: +

Original: || 2 is sufﬁciently small.
Revised: || 2 is sufﬁciently small.

Original: In this way, we decompose the difﬁcult robot-to-robot policy transfer problem into a sequence of K easy problems.
Revised: In this way, we decompose the difﬁcult robot-to-robot policy transfer problem into a sequence of K easy problems.

Original: The optimal solution to the above problem requires ﬁnding the optimal evolution path τ together with policy optimization.
Revised: The optimal solution to the above problem requires ﬁnding the optimal evolution path τ together with policy optimization.

Original: We propose an algorithm named Differentiable Evolution Path Search (DEPS) to ﬁnd both the optimized robot evolution path τ in tandem with policy optimization.
Revised: We propose an algorithm named Differentiable Evolution Path Search (DEPS) to ﬁnd both the optimized robot evolution path τ in tandem with policy optimization.

Original: Given the current evolution parameter α k , we aim to ﬁnd the next best evolution parameter α k +

Revised: Given the current evolution parameter α k , we aim to ﬁnd the next best evolution parameter α k +

Original: =
Revised: =

Original: α k + l k as well as its optimal policy π ∗ F ( α k +1 ) .
Revised: α k + l k as well as its optimal policy π ∗ F ( α k +1 ) .

Original: The optimization objective is formulated as
Revised: The optimization objective is formulated as

Original: [Equation]
Revised: [Equation]

Original: where λ ∈ R + is a weighting factor.
Revised: where λ ∈ R + is a weighting factor.

Original: The ﬁrst term optimizes the policy reward while the second term encourages the evolved robot to move close to the target robot.
Revised: The ﬁrst term optimizes the policy reward while the second term encourages the evolved robot to move close to the target robot.

Original: Since step size ξ is small, we assume L is locally differentiable w.r.t.
Revised: Since step size ξ is small, we assume L is locally differentiable w.r.t.

Original: l k .
Revised: l k .

Original: Then the direction of l k can be estimated by
Revised: Then the direction of l k can be estimated by

Original: [Equation]
Revised: [Equation]

Original: where J = ∇ α E [ ρ π,F ( α ) ] is the Jacobian of expected reward w.r.t.
Revised: where J = ∇ α E [ ρ π,F ( α ) ] is the Jacobian of expected reward w.r.t.

Original: evolution parameter α .
Revised: evolution parameter α .

Original: The
Revised: The

Original: Jacobian J can be estimated by ﬁnite difference with Monte-Carlo sampling of α .
Revised: Jacobian J can be estimated by ﬁnite difference with Monte-Carlo sampling of α .

Original: Concretely, we sample n random vectors δ 1 , δ 2 , . . .
Revised: Concretely, we sample n random vectors δ 1 , δ 2 , . . .

Original: , δ n with (cid:96) 2 -norm of ξ .
Revised: , δ n with (cid:96) 2 -norm of ξ .

Original: Suppose δ 0 = 0 .
Revised: Suppose δ 0 = 0 .

Original: For each δ i , we α
Revised: For each δ i , we α

Original: =α =(a)
Revised: =α =(a)

Original: S D − 1 ( ξ ) α =α =δ i α k
Revised: S D − 1 ( ξ ) α =α =δ i α k

Original: J 1 − α k α k + l k (b)
Revised: J 1 − α k α k + l k (b)

Original: 1: Notation Summary: 2: α ∈ [0 , 1] D : robot evolution parameter; F : [0 , 1] D
Revised: 1: Notation Summary: 2: α ∈ [0 , 1] D : robot evolution parameter; F : [0 , 1] D

Original: → M : continuous robot evolution function; π F ( 0 ) : expert policy on the source robot; ξ ∈ R + : evolution step size; q ∈ R : reward threshold; λ ∈ R + : weight factor; λ 1 ∈ (0 , 1) : shrink ratio; 3: α ← 0 , π ← π F ( 0 ) , l ∼ S D − 1 ( ξ ) // initialization 4: while α (cid:54) = 1 do 5: if E [ ρ π,F ( α + l ) ] < q then 6: δ 1 , δ 2 , . .
Revised: → M : continuous robot evolution function; π F ( 0 ) : expert policy on the source robot; ξ ∈ R + : evolution step size; q ∈ R : reward threshold; λ ∈ R + : weight factor; λ 1 ∈ (0 , 1) : shrink ratio; 3: α ← 0 , π ← π F ( 0 ) , l ∼ S D − 1 ( ξ ) // initialization 4: while α (cid:54) = 1 do 5: if E [ ρ π,F ( α + l ) ] < q then 6: δ 1 , δ 2 , . .

Original: , δ n ∼ S D − 1 ( ξ ) , δ 0 ← 0 // sample δ i vectors 7: execute π and ρ i ← ρ π,F ( α + δ i ) , ∀ i ∈ { 0 , · · · , n } 8: compute Jacobian J from δ j and ρ j according to Eq.(4) 9:
Revised: , δ n ∼ S D − 1 ( ξ ) , δ 0 ← 0 // sample δ i vectors 7: execute π and ρ i ← ρ π,F ( α + δ i ) , ∀ i ∈ { 0 , · · · , n } 8: compute Jacobian J from δ j and ρ j according to Eq.(4) 9:

Original: l ← J / || J || 

Revised: l ← J / || J ||

Original: + λ ( 1 − α ) /
Revised: + λ ( 1 − α ) /

Original: || 

Revised: ||

Original: − α || 2 10: l ← l / || l || 2 · ξ 11: for e in 0 , 1 , . . .
Revised: − α || 2 10: l ← l / || l || 2 · ξ 11: for e in 0 , 1 , . . .

Original: , N e do 12: β ∼ Uniform (1 − λ e 1 , 1) 13: sample rollouts and train π on robot F ( α + β · l ) 14: end for 15: end if 16: α ← min { max { α + l , 0 } , 1 } // make sure stay in [0 , 1] 17: end while 18: return π execute π on robot F ( α k + δ i ) to get episode reward ρ i = ρ π,F ( α k + δ i ) .
Revised: , N e do 12: β ∼ Uniform (1 − λ e 1 , 1) 13: sample rollouts and train π on robot F ( α + β · l ) 14: end for 15: end if 16: α ← min { max { α + l , 0 } , 1 } // make sure stay in [0 , 1] 17: end while 18: return π execute π on robot F ( α k + δ i ) to get episode reward ρ i = ρ π,F ( α k + δ i ) .

Original: By the deﬁnition of
Revised: By the deﬁnition of

Original: Jacobian, if there were no noise, ideally we should have
Revised: Jacobian, if there were no noise, ideally we should have

Original: [Equation]
Revised: [Equation]

Original: However, there is usually inevitable noise in the policy rollout and ρ i .
Revised: However, there is usually inevitable noise in the policy rollout and ρ i .

Original: Let ρ =
Revised: Let ρ =

Original: [ ρ 1 , ρ 2 , . .
Revised: [ ρ 1 , ρ 2 , . .

Original: , ρ n ] (cid:62) and ∆ =
Revised: , ρ n ] (cid:62) and ∆ =

Original: [ δ 1 , δ 2 , . . . ,
Revised: [ δ 1 , δ 2 , . . . ,

Original: δ n ] .
Revised: δ n ] .

Original: The gradient can be estimated by Least Squares Minimization
Revised: The gradient can be estimated by Least Squares Minimization

Original: [Equation]
Revised: [Equation]

Original: Since the scale of the episode reward ρ π,F ( α ) and therefore J is task-dependent and can be arbitrary, to stabilize the evolution path search we scale both J and 1 − α k in Equation (2) to be unit vectors to obtain the direction of evolution progression as follows
Revised: Since the scale of the episode reward ρ π,F ( α ) and therefore J is task-dependent and can be arbitrary, to stabilize the evolution path search we scale both J and 1 − α k in Equation (2) to be unit vectors to obtain the direction of evolution progression as follows

Original: [Equation]
Revised: [Equation]

Original: The next evolution parameter is obtained by α k +

Revised: The next evolution parameter is obtained by α k +

Original: =
Revised: =

Original: α k + ξ · l k / || l k || 2 , where ξ is the evolution step size.
Revised: α k + ξ · l k / || l k || 2 , where ξ is the evolution step size.

Original: The geometric interpretation of Equation (5) is illustrated in Figure 3(b).
Revised: The geometric interpretation of Equation (5) is illustrated in Figure 3(b).

Original: We point out that the optimization objectives of our DEPS in Equations (2) and (5) can be viewed as a generalization of multiple previous works.
Revised: We point out that the optimization objectives of our DEPS in Equations (2) and (5) can be viewed as a generalization of multiple previous works.

Original: When λ → + ∞ , the problem reduces to vanilla
Revised: When λ → + ∞ , the problem reduces to vanilla

Original: REvolveR [18] where α = α · 1 ; when λ → 0 , the problem reduces to unrestricted robot and policy joint optimization such as [20] and [21].
Revised: REvolveR [18] where α = α · 1 ; when λ → 0 , the problem reduces to unrestricted robot and policy joint optimization such as [20] and [21].

Original: Note that when λ > 1 , { α k } k is guaranteed to converge to 1 , because it is easy to show for all l with (cid:96) 2 -norm of ξ , || 1 − ( α k +1 + l ) || 2 (cid:54) ||
Revised: Note that when λ > 1 , { α k } k is guaranteed to converge to 1 , because it is easy to show for all l with (cid:96) 2 -norm of ξ , || 1 − ( α k +1 + l ) || 2 (cid:54) ||

Original: 1 − α k || 2 − ξ | λ − 1 | .
Revised: 1 − α k || 2 − ξ | λ − 1 | .

Original: Policy Optimization along the Evolution Path Given the next evolution parameter α k
Revised: Policy Optimization along the Evolution Path Given the next evolution parameter α k

Original: +

Revised: +

Original: =
Revised: =

Original: α k + l k and the optimized policy π F ( α k ) , the next goal is to ﬁnd the optimal policy π ∗ F ( α k +1 ) on robot
Revised: α k + l k and the optimized policy π F ( α k ) , the next goal is to ﬁnd the optimal policy π ∗ F ( α k +1 ) on robot

Original: F ( α k +1 ) .
Revised: F ( α k +1 ) .

Original: Instead of directly ﬁne-tuning π F ( α k ) on α k +1 , we adopt the following optimization objective with randomized evolution progression similar to [18] to achieve the goal:
Revised: Instead of directly ﬁne-tuning π F ( α k ) on α k +1 , we adopt the following optimization objective with randomized evolution progression similar to [18] to achieve the goal:

Original: [Equation]
Revised: [Equation]

Original: where λ 1 ∈ (0 , 1) is the ratio of range shrinkage and e ∈ N is the iteration counter.
Revised: where λ 1 ∈ (0 , 1) is the ratio of range shrinkage and e ∈ N is the iteration counter.

Original: The above optimization objective allows the policy to be ﬁne-tuned on a set of randomly sampled robots within a range along the l k to improve the stability and robustness of training while also ensuring the sampled evolution parameter α to converge to α k
Revised: The above optimization objective allows the policy to be ﬁne-tuned on a set of randomly sampled robots within a range along the l k to improve the stability and robustness of training while also ensuring the sampled evolution parameter α to converge to α k

Original: +

Revised: +

Original: =
Revised: =

Original: α k + l k when e → + ∞ .
Revised: α k + l k when e → + ∞ .

Original: In practice, a batch of β in Equation (6) is sampled in each iteration.
Revised: In practice, a batch of β in Equation (6) is sampled in each iteration.

Original: The overall algorithm is illustrated in Algorithm 1.
Revised: The overall algorithm is illustrated in Algorithm 1.

Original:  Related Work
Revised:  Related Work

Original: Learning from Human Demonstration Human demonstrations are often collected in terms of trajectories that could be obtained by teleoperation via virtual reality, motion capture markers, or kinesthetic control of robot.
Revised: Learning from Human Demonstration Human demonstrations are often collected in terms of trajectories that could be obtained by teleoperation via virtual reality, motion capture markers, or kinesthetic control of robot.

Original: A policy is then learned by cloning the demonstrations [22, 23, 24, 25, 26, 27] or through inverse reinforcement learning [25, 28, 29, 30, 31].
Revised: A policy is then learned by cloning the demonstrations [22, 23, 24, 25, 26, 27] or through inverse reinforcement learning [25, 28, 29, 30, 31].

Original: In contrast to these methods, in our approach, the human does not control the robot to which the demonstration is transferred.
Revised: In contrast to these methods, in our approach, the human does not control the robot to which the demonstration is transferred.

Original: Human demonstrations need not always be in the form of state space information and a recent line of work rely on visual demonstration to enhance the scalability of data collection as it is much easier to watch humans than ask them to collect demos [32, 33, 34, 35, 36, 37, 38, 39, 40, 41].
Revised: Human demonstrations need not always be in the form of state space information and a recent line of work rely on visual demonstration to enhance the scalability of data collection as it is much easier to watch humans than ask them to collect demos [32, 33, 34, 35, 36, 37, 38, 39, 40, 41].

Original: Although in our case, the demonstrations are converted to the state space of the dexterous hand robot through inverse kinematics, they are originally collected by watching humans.
Revised: Although in our case, the demonstrations are converted to the state space of the dexterous hand robot through inverse kinematics, they are originally collected by watching humans.

Original: Different from these prior works, we use the demonstration to transfer the policy to very different robots than the demonstrator.
Revised: Different from these prior works, we use the demonstration to transfer the policy to very different robots than the demonstrator.

Original: Learning Controllers for New Robot Morphology Evolutionary methods to get new shapes date back to the early days of automata theory from Von Neumman [42].
Revised: Learning Controllers for New Robot Morphology Evolutionary methods to get new shapes date back to the early days of automata theory from Von Neumman [42].

Original: In early 90s, Karl Sims showed the effectiveness of evolutionary methods in optimizing robot shapes and controllers [43, 44].
Revised: In early 90s, Karl Sims showed the effectiveness of evolutionary methods in optimizing robot shapes and controllers [43, 44].

Original: These works laid the foundation of the area where not only robots are optimized but also controllers are transferred across evolved robots.
Revised: These works laid the foundation of the area where not only robots are optimized but also controllers are transferred across evolved robots.

Original: In recent years, many works have revived this idea using deep learning based graph evolution [45, 46, 47, 48, 49].
Revised: In recent years, many works have revived this idea using deep learning based graph evolution [45, 46, 47, 48, 49].

Original: Our work does not treat the morphological evolution in the form of graph networks but rather gradually morphs the source robot to target robot.
Revised: Our work does not treat the morphological evolution in the form of graph networks but rather gradually morphs the source robot to target robot.

Original: Relation to Path Planning
Revised: Relation to Path Planning

Original: Our robot evolution path search problem is similar to the problem of path planning [50, 51, 52, 53, 54, 55].
Revised: Our robot evolution path search problem is similar to the problem of path planning [50, 51, 52, 53, 54, 55].

Original: However, unlike ordinary planning where the obstacles are known, in our problem the difﬁcult robot conﬁgurations are unknown.
Revised: However, unlike ordinary planning where the obstacles are known, in our problem the difﬁcult robot conﬁgurations are unknown.

Original: Moreover, in ordinary path planning, evaluating the cost of a path is cheap, while in our case, transferring the policy to a new robot requires a large number of policy optimization iterations and is computationally costly.
Revised: Moreover, in ordinary path planning, evaluating the cost of a path is cheap, while in our case, transferring the policy to a new robot requires a large number of policy optimization iterations and is computationally costly.

Original:  Experiments
Revised:  Experiments

Original: Our experiment is designed to verify the following two hypotheses: (1) our proposed DEPS algorithm can ﬁnd optimized evolution paths in high-dimensional robot parameter space to improve policy transfer over vanilla linear evolution path [18]; (2) our HERD framework can learn from human demonstrations recorded in diverse modalities including sensor glove signal and visual data.
Revised: Our experiment is designed to verify the following two hypotheses: (1) our proposed DEPS algorithm can ﬁnd optimized evolution paths in high-dimensional robot parameter space to improve policy transfer over vanilla linear evolution path [18]; (2) our HERD framework can learn from human demonstrations recorded in diverse modalities including sensor glove signal and visual data.

Original: Experiment Settings We conduct experiments in MuJoCo physics simulation engine [56].
Revised: Experiment Settings We conduct experiments in MuJoCo physics simulation engine [56].

Original: We adopt the ﬁve-ﬁnger dexterous hand robot provided in the ADROIT platform [57].
Revised: We adopt the ﬁve-ﬁnger dexterous hand robot provided in the ADROIT platform [57].

Original: The high-ﬁdelity models of the target robots and grippers are imported from robosuite environment [58] and have been used in previous robotics publications that eventually transferred their experiments to real robots such as [59, 60].
Revised: The high-ﬁdelity models of the target robots and grippers are imported from robosuite environment [58] and have been used in previous robotics publications that eventually transferred their experiments to real robots such as [59, 60].

Original: We use NPG [61] as the RL algorithm for policy optimization.
Revised: We use NPG [61] as the RL algorithm for policy optimization.

Original: Due to the nature of robot-to-robot policy transfer, the total number of RL iterations it takes to reach the goal of a certain success rate or episode reward cannot be set beforehand.
Revised: Due to the nature of robot-to-robot policy transfer, the total number of RL iterations it takes to reach the goal of a certain success rate or episode reward cannot be set beforehand.

Original: So we instead report the number of policy training iterations and simulation epochs needed to reach 80% success rate on the tasks.
Revised: So we instead report the number of policy training iterations and simulation epochs needed to reach 80% success rate on the tasks.

Original: 5.1 Learning from Sensor Glove Signal Demonstrations
Revised: 5.1 Learning from Sensor Glove Signal Demonstrations

Original: Tasks and Demonstrations
Revised: Tasks and Demonstrations

Original: We use the three tasks from the Hand Manipulation Suite (HMS) [1]:
Revised: We use the three tasks from the Hand Manipulation Suite (HMS) [1]:

Original: Hammer , Relocate , and Door .
Revised: Hammer , Relocate , and Door .

Original: In Hammer , the task is to pick up the hammer and smash the nail into the board; In Relocate , the task is to pick up the ball and move it to the target position; In Door , the task is to turn the door handle and fully open the door.
Revised: In Hammer , the task is to pick up the hammer and smash the nail into the board; In Relocate , the task is to pick up the ball and move it to the target position; In Door , the task is to turn the door handle and fully open the door.

Original: We use a sparse reward setting where only task completion is rewarded.
Revised: We use a sparse reward setting where only task completion is rewarded.

Original: For each task, HMS [1] provided 25 human demonstrations recorded as palm and ﬁnger locations using a sensor glove named CyberGlove III.
Revised: For each task, HMS [1] provided 25 human demonstrations recorded as palm and ﬁnger locations using a sensor glove named CyberGlove III.

Original: Then the expert policy on the dexterous robot is trained using DAPG [1].
Revised: Then the expert policy on the dexterous robot is trained using DAPG [1].

Original: Qualitative Results We provide visualization of the policies on the intermediate robots learned during evolution with our HERD in Figure 4.
Revised: Qualitative Results We provide visualization of the policies on the intermediate robots learned during evolution with our HERD in Figure 4.

Original: Starting from the dexterous robot, our proposed DEPS algorithm is able to ﬁnd the next optimized evolved robot.
Revised: Starting from the dexterous robot, our proposed DEPS algorithm is able to ﬁnd the next optimized evolved robot.

Original: The policy is also able to iteratively adapt to new evolved robots and successfully transfer to the target robots.
Revised: The policy is also able to iteratively adapt to new evolved robots and successfully transfer to the target robots.

Original: Quantitative Comparison with Baselines We compare our HERD against the following baseline methods for learning a policy on the target robot: (1) From Scratch : we train policy on the target robot from scratch with the RL algorithm under both sparse and dense shaped rewards; (2) DAPG
Revised: Quantitative Comparison with Baselines We compare our HERD against the following baseline methods for learning a policy on the target robot: (1) From Scratch : we train policy on the target robot from scratch with the RL algorithm under both sparse and dense shaped rewards; (2) DAPG

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [1]: a variant of NPG [61] with demonstration augmented policy gradient.
Revised: [1]: a variant of NPG [61] with demonstration augmented policy gradient.

Original: of transferred HERD policy on the target robots as the demonstrations for DAPG;
Revised: of transferred HERD policy on the target robots as the demonstrations for DAPG;

Original: (3) REvolveR [18]: robot-to-robot policy transfer with continuous and linear robot evolution path which is a special case of our HERD.
Revised: (3) REvolveR [18]: robot-to-robot policy transfer with continuous and linear robot evolution path which is a special case of our HERD.

Original: The results are illustrated in Table 1.
Revised: The results are illustrated in Table 1.

Original: Training RL from scratch never completes the task to receive reward and is not able to learn on these tasks.
Revised: Training RL from scratch never completes the task to receive reward and is not able to learn on these tasks.

Original: Robot-to-robot evolution solutions with sparse rewards are able to successfully transfer the policy and even signiﬁcantly outperform training from scratch with shaped rewards.
Revised: Robot-to-robot evolution solutions with sparse rewards are able to successfully transfer the policy and even signiﬁcantly outperform training from scratch with shaped rewards.

Original: Moreover, our
Revised: Moreover, our

Original: HERD outperforms REvolveR [18], though the margin in simulation epochs is smaller than RL iterations in that HERD spends additional simulation epochs to obtain Jacobian as in Equation (4).
Revised: HERD outperforms REvolveR [18], though the margin in simulation epochs is smaller than RL iterations in that HERD spends additional simulation epochs to obtain Jacobian as in Equation (4).

Original: We notice that our HERD also has advantage over DAPG trained using expert rollouts directly collected on the target robot.
Revised: We notice that our HERD also has advantage over DAPG trained using expert rollouts directly collected on the target robot.

Original: This shows that training a policy while maintaining sufﬁcient reward/success during robot evolution has advantage over exploration from scratch even with demonstration.
Revised: This shows that training a policy while maintaining sufﬁcient reward/success during robot evolution has advantage over exploration from scratch even with demonstration.

Original: 5.2 Learning From Visual Demonstrations
Revised: 5.2 Learning From Visual Demonstrations

Original: Task and Demonstrations
Revised: Task and Demonstrations

Original: In addition to the demonstration captured by sensor glove, we are also interested in showing learning from visual human demonstration with our HERD.
Revised: In addition to the demonstration captured by sensor glove, we are also interested in showing learning from visual human demonstration with our HERD.

Original: We use DexYCB dataset [2] as the source of human demonstration.
Revised: We use DexYCB dataset [2] as the source of human demonstration.

Original: DexYCB dataset is a multi-view RGB-D video dataset capturing human hand manipulating YCB objects [62].
Revised: DexYCB dataset is a multi-view RGB-D video dataset capturing human hand manipulating YCB objects [62].

Original: We deﬁne the task as grasping and then moving the object of interest to the desired position within a distance threshold as demonstrated in the DexYCB videos.
Revised: We deﬁne the task as grasping and then moving the object of interest to the desired position within a distance threshold as demonstrated in the DexYCB videos.

Original: We use 5 YCB objects in Table 2.
Revised: We use 5 YCB objects in Table 2.

Original: For each object, we take one video which results in 5 demonstrations in total.
Revised: For each object, we take one video which results in 5 demonstrations in total.

Original: The human hand joint poses and object 6D poses are estimated by HRNet32 [63] and CozyPose [64] respectively.
Revised: The human hand joint poses and object 6D poses are estimated by HRNet32 [63] and CozyPose [64] respectively.

Original: Then the expert dexterous policy is trained using the method in Section 3.1.
Revised: Then the expert dexterous policy is trained using the method in Section 3.1.

Original: The reward is sparse task completion reward.
Revised: The reward is sparse task completion reward.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Result Analysis and Comparison We compare against REvolveR [18] and illustrate the experiment results in Table 2.
Revised: Result Analysis and Comparison We compare against REvolveR [18] and illustrate the experiment results in Table 2.

Original: Our HERD outperforms REvolveR in terms of policy iterations as well as most reported simulation epoch metrics.
Revised: Our HERD outperforms REvolveR in terms of policy iterations as well as most reported simulation epoch metrics.

Original: One interesting ﬁnding is that the policy for manipulating 011 banana object is the easiest task among all objects.
Revised: One interesting ﬁnding is that the policy for manipulating 011 banana object is the easiest task among all objects.

Original: A possible reason is that 011 banana object has a geometry most suitable for both the dexterous hand and target gripper.
Revised: A possible reason is that 011 banana object has a geometry most suitable for both the dexterous hand and target gripper.

Original: Qualitative Results We visualize the policies on the intermediate robots learned during evolution with our HERD in Figure 5.
Revised: Qualitative Results We visualize the policies on the intermediate robots learned during evolution with our HERD in Figure 5.

Original: Again, our proposed DEPS algorithm is able to ﬁnd the next optimized evolved robot while successfully transferring the policy to the target robot.
Revised: Again, our proposed DEPS algorithm is able to ﬁnd the next optimized evolved robot while successfully transferring the policy to the target robot.

Original: For more details on the visualizations in this section, please refer to the supplementary video.
Revised: For more details on the visualizations in this section, please refer to the supplementary video.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we propose a framework named HERD for learning from human demonstration.
Revised: In this paper, we propose a framework named HERD for learning from human demonstration.

Original: HERD framework utilizes the continuous evolution of robots to transfer the expert policy from embodied dexterous hands to a target robot.
Revised: HERD framework utilizes the continuous evolution of robots to transfer the expert policy from embodied dexterous hands to a target robot.

Original: We conduct experiments using the human demonstrations from Human Manipulation Suite [1] and DexYCB dataset [2].
Revised: We conduct experiments using the human demonstrations from Human Manipulation Suite [1] and DexYCB dataset [2].

Original: We show that the proposed approach is able to transfer expert policy trained using human demonstrations in sensor glove signals or visual data to target two-ﬁnger Sawyer and three-ﬁnger Jaco robots, even with challenging sparse rewards.
Revised: We show that the proposed approach is able to transfer expert policy trained using human demonstrations in sensor glove signals or visual data to target two-ﬁnger Sawyer and three-ﬁnger Jaco robots, even with challenging sparse rewards.

Original: Limitations We expect that our HERD would fail in learning from human demonstrations for robots with extremely different functions than humans, e.g. learning object manipulation policy for a robot arm mounted with a suction cup from human hand manipulation demonstrations.
Revised: Limitations We expect that our HERD would fail in learning from human demonstrations for robots with extremely different functions than humans, e.g. learning object manipulation policy for a robot arm mounted with a suction cup from human hand manipulation demonstrations.

Original: This is because our proposed human-to-robot continuous evolution solution can only handle different morphology but cannot handle different functions of the robot end-effector.
Revised: This is because our proposed human-to-robot continuous evolution solution can only handle different morphology but cannot handle different functions of the robot end-effector.

Original: We leave the exploration of more general human-to-robot continuous evolution solutions as future work.
Revised: We leave the exploration of more general human-to-robot continuous evolution solutions as future work.

Original: Acknowledgement This work is in part funded by JST AIP Acceleration, Grant Number JPMJCR20U1, Japan.
Revised: Acknowledgement This work is in part funded by JST AIP Acceleration, Grant Number JPMJCR20U1, Japan.

Original: Deepak Pathak is supported by NSF grant IIS-2024594.
Revised: Deepak Pathak is supported by NSF grant IIS-2024594.