Original: HERD: Continuous Human-to-Robot Evolution for Learning from Human Demonstration
Revised: HERD: Continuous Human-to-Robot Evolution for Learning from Human Demonstration

Original: [Figure]
Revised: [Figure]

Original: email

Original: [Figure] Gripper [Figure] Abstract: The ability to learn from human demonstration endows robots with theability to automate various tasks.
Revised: Abstract: The ability to learn from human demonstration endows robots with the ability to automate various tasks.

Original: However, directly learning from human demonstration is challenging since the structure of the human hand can be very differentfrom the desired robot gripper.
Revised: However, directly learning from human demonstration is challenging since the structure of the human hand can be very different from the desired robot gripper.

Original: In this work, we show that manipulation skillscan be transferred from a human to a robot through the use of micro-evolutionaryreinforcement learning, where a five-finger human dexterous hand robot graduallyevolves into a commercial two-finger-gripper robot, while repeated interacting ina physics simulator to continuously update the policy that is first learned fromhuman demonstration.
Revised: In this work, we show that manipulation skills can be transferred from a human to a robot through the use of micro-evolutionary reinforcement learning, where a five-finger human dexterous hand robot gradually evolves into a commercial two-finger-gripper robot, while repeated interacting in a physics simulator to continuously update the policy that is first learned from human demonstration.

Original: To deal with the high dimensions of robot parameters, wepropose an algorithm for multi-dimensional evolution path searching that allowsjoint optimization of both the robot evolution path and the policy.
Revised: To deal with the high dimensions of robot parameters, we propose an algorithm for multi-dimensional evolution path searching that allows joint optimization of both the robot evolution path and the policy.

Original: Through experiments on human object manipulation datasets, we show that our framework canefficiently transfer the expert human agent policy trained from human demonstra- tions in diverse modalities to a target commercial robot.
Revised: Through experiments on human object manipulation datasets, we show that our framework can efficiently transfer the expert human agent policy trained from human demonstrations in diverse modalities to a target commercial robot.

Original:  Introduction
Revised:  Introduction

Original: Learning from human demonstrations [3] is a promising direction to enable robots to perform diverse manipulation capabilities.
Revised: Learning from human demonstrations [3] is a promising direction to enable robots to perform diverse manipulation capabilities.

Original: Recent large-scale visual datasets of human activity [2, 4, 5, 6, 7]have highlighted the need for enabling robotic agents to learn from human activities to perform di- verse tasks in real-world environments. Existing paradigm for learning from human demonstrationgenerally follows the pipeline of demonstration conversion , i.e. (1) convert demonstrated humanstates to robot states and then (2) train the robot policy .
Revised: Recent large-scale visual datasets of human activity [2, 4, 5, 6, 7] have highlighted the need for enabling robotic agents to learn from human activities to perform diverse tasks in real-world environments states to robot states and then (2) train the robot policy .

Original: Though step (2) can be solved by existingimitation learning approaches [8, 9, 10, 1], step (1) is extremely difficult.
Revised: Though step (2) can be solved by existing imitation learning approaches [8, 9, 10, 1], step (1) is extremely difficult.

Original: Efforts to address step (1)include human-robot observation matching such as [11], and converting human demonstrations torobots by human-robot interaction [12, 13] or human-in-the-loop teleoperation [14, 15].
Revised: Efforts to address step (1) include human-robot observation matching such as [11], and converting human demonstrations to robots by human-robot interaction [12, 13] or human-in-the-loop teleoperation [14, 15].

Original: However,these solutions are highly situational and task-specific and require significant human interventionfor each individual task, therefore are not scalable and cannot automatically generalize to new tasks.
Revised: However, these solutions are highly situational and task-specific and require significant human intervention for each individual task, therefore are not scalable and cannot automatically generalize to new tasks.

Original: We propose a new paradigm for learning from human demonstration — policy transformation , i.e.(1) train policy on embodied human hand dexterous robot and then (2) transfer the policy fromdexterous hand robot to the target robot .
Revised: We propose a new paradigm for learning from human demonstration — policy transformation , i.e. (1) train policy on embodied human hand dexterous robot and then (2) transfer the policy from dexterous hand robot to the target robot .

Original: Many existing methods have been shown successful inlearning control policy for dexterous hand robots from human demonstration [1, 16] and can easilysolve step (1) of our pipeline.
Revised: Many existing methods have been shown successful in learning control policy for dexterous hand robots from human demonstration [1, 16] and can easily solve step (1) of our pipeline.

Original: A general and automatic solution of step (2) would render a scalablesolution of learning from human demonstration.
Revised: A general and automatic solution of step (2) would render a scalable solution of learning from human demonstration.

Original: The key challenge to step (2) is the huge dynamicsdiscrepancy between the dexterous hand robot and the target robot.
Revised: The key challenge to step (2) is the huge dynamics discrepancy between the dexterous hand robot and the target robot.

Original: The discrepancy stems fromthe mismatch in robot morphology and kinematics.
Revised: The discrepancy stems from the mismatch in robot morphology and kinematics.

Original: Unfortunately, statistical matching imitationlearning approaches such as [8, 9, 10, 17] assume the teacher and student robotic agents share thesame or similar transition dynamics, therefore are ineffective in solving the problem.
Revised: Unfortunately, statistical matching imitation learning approaches such as [8, 9, 10, 17] assume the teacher and student robotic agents share the same or similar transition dynamics, therefore are ineffective in solving the problem.

Original: In this paper, we present a framework named HERD (short for H uman
Revised: In this paper, we present a framework named HERD (short for H uman

Original: E volution to R obot for Learning from D emonstration) with a new perspective on solving the problem of learning from humandemonstration — continuous robot evolution .
Revised: E volution to R obot for Learning from D emonstration) with a new perspective on solving the problem of learning from human demonstration — continuous robot evolution .

Original: The core idea  is to define a continuous interpolation between the dexterous hand robot and a target robot that have different morphology.
Revised: Following the philosophy introduced in the seminal work of REvolveR [18], the core idea of our framework is to define a continuous interpolation between the dexterous hand robot and a target robot that have different morphology.

Original: Then anexpert dexterous hand robot policy learned with methods such as [1, 16] can be transferred to thetarget robot through training on a sequence of intermediate robots that gradually evolve into the target robot.
Revised: Then an expert dexterous hand robot policy learned with methods such as [1, 16] can be transferred to the target robot through training on a sequence of intermediate robots that gradually evolve into the target robot.

Original: The policy is continuously updated through repeated interaction in a physics engine thatcontains the intermediate robots, as illustrated in Figure 1.
Revised: The policy is continuously updated through repeated interaction in a physics engine that contains the intermediate robots, as illustrated in Figure 1.

Original: To instantiate the idea, we develop a solution specifically for  evolution from  dexterous hand robotto a Sawyer robot with a two-finger Rethink  gripper.
Revised: To instantiate the idea, we develop two solutions for the evolution from five-finger dexterous robot to two target commercial robots respectively: a Sawyer robot with a two-finger Rethink gripper, and a Jaco robot with a three-finger Jaco gripper.

Original: The solution handles the evolution of bothdexterous hand and arm to precisely match the dynamics of Rethink gripper.
Revised: The solutions precisely match the dynamics of target commercial grippers.

Original: To deal with the highdimensions of robot parameters, we propose an algorithm for automatically searching evolutionpaths in high-dimensional robot parameter space that allows joint optimization of both robot evolution and the policy, based on online estimation of reward gradient with respect to robot evolution.
Revised: To deal with the high dimensions of robot parameters, we propose an algorithm for automatically searching evolution paths in high-dimensional robot parameter space that allows joint optimization of both robot evolution and the policy, based on online estimation of reward gradient with respect to robot evolution.

Original: We conduct experiments on transferring human expert demonstrations of manipulation tasks in diverse modalities to a target commercial robot.
Revised: We conduct experiments on transferring human expert demonstrations of manipulation tasks in diverse modalities to the target commercial robots.

Original: We show that HERD can successfully transfer theexpert dexterous hand robot policy trained from Hand Manipulation Suite sensor glove demonstrations [1] to the target Sawyer robot even with sparse rewards.
Revised: We show that HERD can successfully transfer the expert dexterous hand robot policy trained from Hand Manipulation Suite sensor glove demonstrations [1] to the target robots even with sparse rewards.

Original: On DexYCB dataset [2], we show thatour approach can also transfer the expert policy trained with visual human demonstrations to thetarget robot.
Revised: On DexYCB dataset [2], we demonstrate that our approach can also transfer the expert policy trained with visual human demonstrations to the target robots.

Original: In addition, we show that the proposed multi-dimensional robot evolution path searchalgorithm can significantly improve the policy transferring efficiency.
Revised: In addition, we show that the proposed multi-dimensional robot evolution path search algorithm can significantly improve the policy transferring efficiency.

Original:  Preliminaries
Revised:  Preliminaries

Original: MDP Preliminary We consider a continuous control problem formulated as Markov Decision
Revised: MDP Preliminary We consider a continuous control problem formulated as Markov Decision

Original: Process (MDP).
Revised: Process (MDP).

Original: It is defined by a tuple ( S , A , T , R, γ ) , where S ⊆ R S is the state space, A ⊆ R Ais the action space, T : S ×
Revised: It is defined by a tuple ( S , A , T , R, γ ) , where S ⊆ R S is the state space, A ⊆ R A is the action space, T : S ×

Original: A → S is the transition function, R : S ×
Revised: A → S is the transition function, R : S ×

Original: A → R is the rewardfunction, and γ ∈ [0 , 1] is the discount factor.
Revised: A → R is the reward function, and γ ∈ [0 , 1] is the discount factor.

Original: A policy π : S → A maps a state to an action whereπ ( a | s ) is the probability of choosing action a at state s .
Revised: A policy π : S → A maps a state to an action where π ( a | s ) is the probability of choosing action a at state s .

Original: Suppose M is the set of all MDPs andρ π, M = (cid:80) ∞
Revised: Suppose M is the set of all MDPs and ρ π,M = (cid:80) ∞

Original: t =

Revised: t =

Original: γ t R ( s t , a t ) is the episode discounted reward with policy π on MDP M ∈ M .
Revised: γ t R ( s t , a t ) is the episode discounted reward with policy π on MDP M ∈ M .

Original: Theoptimal policy π ∗ M on MDP M is the one that maximizes the expected value of ρ π,M .
Revised: The optimal policy π ∗ M on MDP M is the one that maximizes the expected value of ρ π,M .

Original: REvolveR [18] Preliminary Liu et al.
Revised: REvolveR [18] Preliminary Liu et al.

Original: [18] proposed a technique named REvolveR for transferringpolicies from a source robot to a target robot.
Revised: [18] proposed a technique named REvolveR for transferring policies from a source robot to a target robot.

Original: The core idea is to define an evolutionary sequenceof intermediate robots that connects the source to the target robot.
Revised: The core idea is to define an evolutionary sequence of intermediate robots that connects the source to the target robot.

Original: Given source and target robots represented by two MDPs M S , M T ∈ M respectively, REvolveR defines a continuous function
Revised: Given source and target robots represented by two MDPs M S , M T ∈ M respectively, REvolveR defines a continuous function

Original: E : [0 , 1] → M where E (0) =
Revised: E : [0 , 1] → M where E (0) =

Original: M S , E (1) =
Revised: M S , E (1) =

Original: M T .
Revised: M T .

Original: Then an expert policy π ∗ E (0) on the sourcerobot E (0) is optimized by sequentially interacting with each intermediate robot in the sequence
Revised: Then an expert policy π ∗ E (0) on the source robot E (0) is optimized by sequentially interacting with each intermediate robot in the sequence

Original: E ( α 1 ) , E ( α 2 ) , . . .
Revised: E ( α 1 ) , E ( α 2 ) , . . .

Original: , E ( α K )
Revised: , E ( α K )

Original: where 0 <
Revised: where 0 <

Original: α 1 <
Revised: α 1 <

Original: α 2 < · · · < α K = 1 , until the policy is able to act(near) optimally on each intermediate robot and eventually transfer the policy to target robot E (1) .
Revised: α 2 < · · · < α K = 1 , until the policy is able to act (near) optimally on each intermediate robot and eventually transfer the policy to target robot E (1) .

Original: For all i , | α i +1 − α i | is small enough so that each policy fine-tuning step is a much easier task.
Revised: For all i , | α i +1 − α i | is small enough so that each policy fine-tuning step is a much easier task.

Revised: Our method is built upon the philosophy of REvolveR to use robot evolution for policy transfer.

Original: Notation We use bold letters to denote the vector variables.
Revised: Notation We use bold letters to denote the vector variables.

Original: Specially, 0 and 1 are the all-zeroand all-one vectors with proper dimensions respectively.
Revised: Specially, 0 and 1 are the all-zero and all-one vectors with proper dimensions respectively.

Original: S D − 1 ( ξ ) =
Revised: S D − 1 ( ξ ) =

Original: { x ∈ R D
Revised: { x ∈ R D

Original: |
Revised: |

Original: ||
Revised: ||

Original: x
Revised: x

Original: || 2 = ξ } is a( D − 1) -sphere
Revised: || 2 = ξ } is a ( D − 1) -sphere

Original: (i.e. a sphere in R D space) with radius
Revised: (i.e. a sphere in R D space) with radius

Original: ξ ∈ R + .
Revised: ξ ∈ R + .

Original: ⊙ is element-wise product.
Revised: ⊙ is element-wise product.

Original:  Method 
Revised:  Method 3.1 Human Demonstration to Dexterous Robot Expert Policy

Original: 3.1 Human Demonstration to Dexterous Robot Expert Policy

Original: Recent works [1, 16] have shown success in learning five-finger dexterous robot policy from humanhand demonstration in diverse modalities, such as sensor glove signals [1] and visual data [16],which made the first step of our proposed paradigm possible.
Revised: Recent works [1, 16] have shown success in learning five-finger dexterous robot policy from human hand demonstrations in diverse modalities, such as sensor glove signals [1] and visual data [16], which made the first step of our proposed paradigm possible.

Original: Inspired by these research progress, weadopt similar approaches to train expert policies on  dexterous robot as illustrated in Figure 1(1)(2).
Revised: Inspired by this research progress, we adopt similar approaches to train expert policies on a dexterous robot as illustrated in Figure 1(1)(2).

Original: Sensor Glove Demonstration When recorded in sensor glove signals, human demonstrations aredirectly converted to states and actions in  simulator [19] from which a policy is trained with algorithms such as DAPG [1].
Revised: Sensor Glove Demonstration When recorded in sensor glove signals, human demonstrations are directly converted to states and actions in a simulator [19] from which a policy is trained with algorithms such as DAPG [1].

Original: We use the same procedure and obtain the expert policy as  [1].
Revised: We use the same procedure and obtain the expert policy as in [1].

Original: Visual Demonstration We develop a toolchain for training dexterous policy from human visualdemonstration of manipulation.
Revised: Visual Demonstration We develop a toolchain for training dexterous policies from human visual demonstrations of manipulation.

Original: Our toolchain includes human and object pose estimation withvisual perception followed by pose fitting of the dexterous robot via inverse kinematics (IK).
Revised: Our toolchain includes human and object pose estimation with visual perception followed by pose fitting of the dexterous robot via inverse kinematics (IK).

Original: Wethen train the policy by setting the episodes’ initial state to be the state of the grasping frame,including poses and joint velocity.
Revised: We then train the policy by setting the episodes’ initial state to be the state of the grasping frame, including poses and joint velocity.

Original: During training, the initial state gradually moves backward intime until the desired starting state.
Revised: During training, the initial state gradually moves backward in time until the desired starting state.

Original: For more details, please refer to the supplementary.
Revised: For more details, please refer to the supplementary.

Original: 3.2 Continuous Dexterous Hand to Target Two-finger Robot Evolution
Revised: 3.2 Continuous Dexterous Hand to Target Commercial Robots Evolution

Original: A core hypothesis in our proposed paradigm of learning from human demonstration is the feasibilityof interpolation between a human-hand-like five-finger dexterous robot and a target commercialrobot.
Revised: A core hypothesis in our proposed paradigm of learning from human demonstration is the feasibility of interpolation between a human-hand-like five-finger dexterous robot and a target commercial robot.

Original: In this paper, we use a commercial Sawyer robot with a two-finger Rethink gripper as anexample to show this.
Revised: In this paper, we present solutions for dexterous-to-commercial-robot interpolation for two target commercial robots to show this.

Original: Since both the dexterous hand robot and the Sawyer robot are complicated instructure, how to design the evolution from the former to the latter is not straightforward and has alarge design space.
Revised: Since both the dexterous hand robot and the target robots are complicated in structure, how to design the evolution from the former to the latter is not straightforward and has a large design space.

Original: The design choice can significantly affect the policy transfer performance.
Revised: The design choice can significantly affect the policy transfer performance.

Original: In thissection, we present a solution for dexterous-to-two-finger evolution.

Original: Following [18], we first matchthe robot morphology between the dexterous and the target robot.
Revised: Following [18], we first match the robot morphology between the dexterous and the target robot.

Original: With the same morphology, wethen design a shared high-dimensional robot parameter space that allows defining an intermediaterobot by simply setting these parameters.
Revised: With the same morphology, we then design a shared high-dimensional robot parameter space that allows defining an intermediate robot by simply setting these parameters.

Original: The evolution includes both the dexterous hand and arm.
Revised: The evolution includes both the dexterous hand and arm.

Original: Dexterous Hand to Two-finger Gripper Evolution The main difference between a dexteroushand and a robot gripper is the number of fingers and the configuration of the finger joints.

Original:  To matchthe number of fingers, we keep the thumb andthe first finger while gradually shrinking the otherthree fingers to be zero-size and zero-mass so that they eventually disappear.
Revised: Dexterous Hand to Commercial Gripper Evolution To match the number of fingers, we keep the thumb and gradually shrink the redundant fingers to be zero-size and zero-mass so that they eventually disappear.

Original: To match the jointconfiguration, we force increasingly strict limits on the range of the proximal and distal joints andfreeze both of them.
Revised: We also gradually change the range of the finger joints to match the desired joint configuration.

Original: As a result, proximal, middle, and distal phalanges eventually evolve to be asingle rigid body.

Original: In addition, we adda newprismatic joint to the same position of the knuckle.
Revised: In addition, we add new position servo joints to the remaining fingers.

Original: Theprismatic joint is initially frozen and evolves to have the same range as the target prismatic gripper.
Revised: The position servo joints are initially frozen and evolve to have the same range as the target fingers.

Original: Dexterous Arm to Sawyer Robot Arm Evolution A major challenge of arm evolution is thedifferent ways of arm mounting.
Revised: Dexterous Arm to Commercial Robot Arm Evolution A major challenge of arm evolution is the different ways of arm mounting.

Original: To simulate the human elbow, the dexterous robot elbow  is usuallyattached to a virtual free jointso thatit can move freely in  space, while thetarget commercial robotarms are usually mounted on fixed bases.
Revised: In simulation, the dexterous robot elbow joint is usually modeled as a virtual free joint that can move freely in the 3D space, while the commercial robot arms are usually mounted on fixed bases.

Original: Another challenge is the huge difference in the numbers ofarm joints and joint physical parameters.
Revised: Another challenge is the huge difference in the numbers of arm joints and joint physical parameters.

Original: Our solution is to attach the free joint of the dexterous armto the end effector of the target robot and treat the entire dexterous robot as a de facto large gripper.
Revised: Our solution is to attach the free joint of the dexterous arm to the end effector of the target robot and treat the entire dexterous robot as a de facto large gripper.

Original: [Figure] α = During evolution, both the dexterous armsize and the elbow free joint gradually shrink to zero sothat the dexterous robot eventually evolves to be firmly attached to the target robot end effector. 
Revised: During evolution, both the dexterous arm and the elbow free joint gradually shrink to zero so that the dexterous robot eventually evolves to be firmly attached to the target robot end effector. [Figure]

Original: The above evolution process is illustrated in Figure 2.
Revised: The above evolution process is illustrated in Figure 2.

Original: Our evolution solution includes the changingof D= 42 independent robot parameters, such as body sizes and mass, and joint ranges and damping.
Revised: Our evolution solution includes the changing of D independent robot parameters such as body sizes and mass, and joint ranges and damping.

Original: After evolution, the robot parameters are set to match the dynamics of the targettwo-finger Sawyer  robot.More details on the evolution process can be found in the supplementary material.
Revised: In our design, D = 40 for target Sawyer robot and D = 42 for target Jaco robot.

Original: 3.3 Multi-dimensional Continuous Robot Evolution for Policy Transfer
Revised: 3.3 Multi-dimensional Continuous Robot Evolution for Policy Transfer

Original: As shown in Section 3.2, the evolution from a human-like dexterous robot to a target commercialrobot is an extremely high-dimensional problem in robot parameter space (e.g. D = 42 ).
Revised: As shown in Section 3.2, the evolution from a human-like dexterous robot to a target commercial robot is an extremely high-dimensional problem in robot parameter space (e.g. D = 42 ).

Original: Thismeans there exist numerous choices of intermediate robot sequences that connect source to targetrobots.
Revised: This means there exist numerous choices of intermediate robot sequences that connect source to target robots.

Original: However, previous work REvolveR [18] assumes the same and uniform evolution progresshard-coded for all robot parameters and could be sub-optimal.
Revised: However, previous work REvolveR [18] assumes the same and uniform evolution progress hard-coded for all robot parameters and could be sub-optimal.

Original: We envision that when transferringthe policy through intermediate robots, the optimal robot evolution should be flexible and able tosmartly bypass difficult robot configurations to reach the target robot efficiently.
Revised: We envision that when transferring the policy through intermediate robots, the optimal robot evolution should be flexible and able to smartly bypass difficult robot configurations to reach the target robot efficiently.

Original: In this section, weintroduce an algorithm for automatically finding such optimized evolution strategy.
Revised: In this section, we introduce an algorithm for automatically finding such optimized evolution strategy.

Original: We formulatethe problem as a joint optimization of both robot evolution and the policy.
Revised: We formulate the problem as a joint optimization of both robot evolution and the policy.

Original: Problem Statement Suppose the source and target robot morphology is matched using the methodin [18] or Section 3.2.
Revised: Problem Statement Suppose the source and target robot morphology is matched using the method in [18] or Section 3.2.

Original: Therefore the D kinematics parameters of the source and target robots can bemapped to the same space denoted as θ S , θ T ∈ R D .
Revised: Therefore the D kinematics parameters of the source and target robots can be mapped to the same space denoted as θ S , θ T ∈ R D .

Original: Continuous function F : [0 , 1] D
Revised: Continuous function F : [0 , 1] D

Original: → M definesan intermediate robot by interpolation between all pairs of kinematic parameters θ = ( 1 − α ) ⊙θ S + α ⊙ θ T where α ∈ [0 , 1] D is the evolution parameter that describes the evolution progressof each of the D kinematics parameters.
Revised: → M defines an intermediate robot by interpolation between all pairs of kinematic parameters θ = ( 1 − α ) ⊙ θ S + α ⊙ θ T where α ∈ [0 , 1] D is the evolution parameter that describes the evolution progress of each of the D kinematics parameters.

Original: Given an expert policy π ∗ F ( 0 ) on the source robot F ( 0 ) ,the overall goal is to find the optimal policy π ∗ F ( 1 ) on target robot F ( 1 ) .
Revised: Given an expert policy π ∗ F ( 0 ) on the source robot F ( 0 ) , the overall goal is to find the optimal policy π ∗ F ( 1 ) on target robot F ( 1 ) .

Original: Note that  formulation of
Revised: Note that the formulation of

Original: REvolveR [18] is a special case of our problem setting with α = α · 1 and α ∈ [0 , 1] .
Revised: REvolveR [18] is a special case of our problem setting with α = α · 1 and α ∈ [0 , 1] .

Original: Differentiable Evolution Path Search (DEPS)
Revised: Differentiable Evolution Path Search (DEPS)

Original: We adopt multi-dimensional robot evolution as asolution to the above problem as illustrated in Figure 3(a).
Revised: We adopt multi-dimensional robot evolution as a solution to the above problem as illustrated in Figure 3(a).

Original: Suppose there is a robot evolution pathτ =
Revised: Suppose there is a robot evolution path τ

Original:  ( F ( α 0 ) , F ( α 1 ) , F ( α 2 ) , . ..
Revised: = ( F ( α 0 ) , F ( α 1 ) , F ( α 2 ) , . .

Original: , F ( α K − 1 ) , F ( α K )) where α 0 = 0 , α K = 1 .
Revised: , F ( α K − 1 ) , F ( α K )) where α 0 = 0 , α K = 1 .

Original: In k -th phase, thepolicy optimization objective is to maximize the expected reward E [ ρ π,F ( α k ) ] on robot F ( α k ) .
Revised: In k -th phase, the policy optimization objective is to maximize the expected reward E [ ρ π,F ( α k ) ] on robot F ( α k ) .

Original: Forall k , the evolution parameter step size ξ = ||
Revised: For all k , the evolution parameter step size ξ = ||

Original: α k − α k
Revised: α k − α k

Original: +

Revised: +

Original: || 2 is sufficiently small.
Revised: || 2 is sufficiently small.

Original: In this way, wedecompose the difficult robot-to-robot policy transfer problem into a sequence of K easy problems.
Revised: In this way, we decompose the difficult robot-to-robot policy transfer problem into a sequence of K easy problems.

Original: The optimal solution to the above problem requires finding the optimal evolution path τ togetherwith policy optimization.
Revised: The optimal solution to the above problem requires finding the optimal evolution path τ together with policy optimization.

Original: We propose an algorithm named Differentiable Evolution Path Search(DEPS) to find both the optimized robot evolution path τ in tandem with policy optimization.
Revised: We propose an algorithm named Differentiable Evolution Path Search (DEPS) to find both the optimized robot evolution path τ in tandem with policy optimization.

Original: Giventhe current evolution parameter α k , we aim to find the next best evolution parameter α k +

Revised: Given the current evolution parameter α k , we aim to find the next best evolution parameter α k +

Original: =
Revised: =

Original: α k + l kas well as its optimal policy π ∗ F ( α k +1 ) .
Revised: α k + l k as well as its optimal policy π ∗ F ( α k +1 ) .

Original: The optimization objective is formulated as
Revised: The optimization objective is formulated as

Original: [Equation]
Revised: [Equation]

Revised: where λ ∈ R + is a weighting factor.

Revised: The first term optimizes the policy reward while the second term encourages the evolved robot to move close to the target robot.

Revised: Since step size ξ is small, we α

Original: α =α =(a)
Revised: =α =(a)

Original: S D − 1 ( ξ ) α =α =δ i α k
Revised: S D − 1 ( ξ ) α =α =δ i α k

Original: J 1 − α k α k + l k (b)
Revised: J 1 − α k α k + l k (b)

Original: 1: Notation Summary: 2: α ∈ [0 , 1] D : robot evolution parameter; F : [0 , 1] D
Revised: 1: Notation Summary: 2: α ∈ [0 , 1] D : robot evolution parameter; F : [0 , 1] D

Original: → M : continuous robot evolution function; π F ( 0 ) : expert policy on the source robot; ξ ∈ R + : evolution step size; q ∈ R : reward threshold; λ ∈ R + : weight factor; λ 1 ∈ (0 , 1) : shrink ratio; 3: α ← 0 , π ← π F ( 0 ) , l ∼ S D − 1 ( ξ ) // initialization 4: while α ̸ = 1 do 5: if E [ ρ π,F ( α + l ) ] < q then 6: δ 1 , δ 2 , . .
Revised: → M : continuous robot evolution function; π F ( 0 ) : expert policy on the source robot; ξ ∈ R + : evolution step size; q ∈ R : reward threshold; λ ∈ R + : weight factor; λ 1 ∈ (0 , 1) : shrink ratio; 3: α ← 0 , π ← π F ( 0 ) , l ∼ S D − 1 ( ξ ) // initialization 4: while α ̸ = 1 do 5: if E [ ρ π,F ( α + l ) ] < q then 6: δ 1 , δ 2 , . .

Original: , δ n ∼ S D − 1 ( ξ ) , δ 0 ← 0 // sample δ i vectors 7: execute π and ρ i ← ρ π,F ( α + δ i ) , ∀ i ∈ { 0 , · · · , n } 8: compute Jacobian J from δ j and ρ j according to Eq.(4) 9:
Revised: , δ n ∼ S D − 1 ( ξ ) , δ 0 ← 0 // sample δ i vectors 7: execute π and ρ i ← ρ π,F ( α + δ i ) , ∀ i ∈ { 0 , · · · , n } 8: compute Jacobian J from δ j and ρ j according to Eq.(4) 9:

Original: l ← J / || J || 

Revised: l ← J / || J ||

Original: + λ ( 1 − α ) /
Revised: + λ ( 1 − α ) /

Original: || 

Revised: ||

Original: − α || 2 10: l ← l / || l || 2 · ξ 11: for e in 0 , 1 , . . .
Revised: − α || 2 10: l ← l / || l || 2 · ξ 11: for e in 0 , 1 , . . .

Original: , N e do 12: β ∼ Uniform (1 − λ e 1 , 1) 13: sample rollouts and train π on robot F ( α + β · l ) 14: end for 15: end if 16: α ← min { max { α + l , 0 } , 1 } // make sure stay in [0 , 1] 17: end while 18: return π where λ ∈ R + is a weighting factor.
Revised: , N e do 12: β ∼ Uniform (1 − λ e 1 , 1) 13: sample rollouts and train π on robot F ( α + β · l ) 14: end for 15: end if 16: α ← min { max { α + l , 0 } , 1 } // make sure stay in [0 , 1] 17: end while 18: return π assume L is locally differentiable w.r.t.

Original: The first term optimizes the policy reward while the second term encourages the evolved robot to move close to the target robot.

Original: Since step size ξ is small, weassume L is locally differentiable w.r.t.

Original: l k .
Revised: l k .

Original: Then the direction of l k can be estimated by
Revised: Then the direction of l k can be estimated by

Original: [Equation]
Revised: [Equation]

Original: where J = ∇ α E [ ρ π,F ( α ) ] is the Jacobian of expected reward w.r.t.
Revised: where J = ∇ α E [ ρ π,F ( α ) ] is the Jacobian of expected reward w.r.t.

Original: evolution parameter α .
Revised: evolution parameter α .

Original: The
Revised: The

Original: Jacobian J can be estimated by finite difference with Monte-Carlo sampling of α .
Revised: Jacobian J can be estimated by finite difference with Monte-Carlo sampling of α .

Original: Concretely, wesample n random vectors δ 1 , δ 2 , . . 
Revised: Concretely, we sample n random vectors δ 1 , δ 2 , . . .

Original: , δ n with ℓ 2 -norm of ξ .
Revised: , δ n with ℓ 2 -norm of ξ .

Original: Suppose δ 0 = 0 .
Revised: Suppose δ 0 = 0 .

Original: For each δ i , weexecute π on robot F ( α k + δ i ) to get episode reward ρ i = ρ π,F ( α k + δ i ) .
Revised: For each δ i , we execute π on robot F ( α k + δ i ) to get episode reward ρ i = ρ π,F ( α k + δ i ) .

Original: By the definition of
Revised: By the definition of

Original: Jacobian, if there were no noise, ideally we should have
Revised: Jacobian, if there were no noise, ideally we should have

Original: [Equation]
Revised: [Equation]

Original: However, there is usually inevitable noise in the policy rollout and ρ i .
Revised: However, there is usually inevitable noise in the policy rollout and ρ i .

Original: Let ρ =
Revised: Let ρ =

Original: [ ρ 1 , ρ 2 , . .
Revised: [ ρ 1 , ρ 2 , . .

Original: . , ρ n ] ⊤and ∆ =
Revised: . , ρ n ] ⊤ and ∆ =

Original: [ δ 1 , δ 2 , . . . ,
Revised: [ δ 1 , δ 2 , . . . ,

Original: δ n ] .
Revised: δ n ] .

Original: The gradient can be estimated by Least Squares Minimization
Revised: The gradient can be estimated by Least Squares Minimization

Original: [Equation]
Revised: [Equation]

Original: Since the scale of the episode reward ρ π,F ( α ) and therefore J is task-dependent and can be arbitrary,to stabilize the evolution path search we scale both J and 1 − α k in Equation (2) to be unit vectorsto obtain the direction of evolution progression as follows[Equation]
Revised: Since the scale of the episode reward ρ π,F ( α ) and therefore J is task-dependent and can be arbitrary, to stabilize the evolution path search we scale both J and 1 − α k in Equation (2) to be unit vectors to obtain the direction of evolution progression as follows

Original: [Equation]
Revised: [Equation]

Original: The next evolution parameter is obtained by α {+k +
+}
Revised: The next evolution parameter is obtained by α {+k +
+}

Original: [Equation]
Revised: =

Original:  || l k ||  , where ξ is the evolution stepsize.
Revised: α k + ξ · l k / || l k || 2 , where ξ is the evolution step size.

Original: The geometric interpretation of Equation (5) is illustrated in Figure 3(b).
Revised: The geometric interpretation of Equation (5) is illustrated in Figure 3(b).

Original: We point out that the optimization objectives of our DEPS in Equations (2) and (5) can be viewedas a generalization of multiple previous works.
Revised: We point out that the optimization objectives of our DEPS in Equations (2) and (5) can be viewed as a generalization of multiple previous works.

Original: When λ → + ∞ , the problem reduces to vanilla
Revised: When λ → + ∞ , the problem reduces to vanilla

Original: REvolveR [18] where α = α · 1 ; when λ → 0 , the problem reduces to unrestricted robot and policyjoint optimization such as [20] and [21].
Revised: REvolveR [18] where α = α · 1 ; when λ → 0 , the problem reduces to unrestricted robot and policy joint optimization such as [20] and [21].

Original: Note that when λ > 1 , { α k } k is guaranteed to converge to1 , because it is easy to show for all l with ℓ 2 -norm of ξ , || 1 − ( α k +1 + l ) ||
Revised: Note that when λ > 1 , { α k } k is guaranteed to converge to 1 , because it is easy to show for all l with ℓ 2 -norm of ξ , || 1 − ( α k +1 + l ) ||

Original: 2 ⩽ || 1 − α k || 2 − ξ | λ − 1 | .
Revised: 2 ⩽ || 1 − α k || 2 − ξ | λ − 1 | .

Original: Policy Optimization along the Evolution Path Given the next evolution parameter α k
Revised: Policy Optimization along the Evolution Path Given the next evolution parameter α k

Original: +

Revised: +

Original: =
Revised: =

Original: α k +l k and the optimized policy π F ( α k ) , the next goal is to find the optimal policy π ∗ F ( α k +1 ) on robot
Revised: α k + l k and the optimized policy π F ( α k ) , the next goal is to find the optimal policy π ∗ F ( α k +1 ) on robot

Original: F ( α k +1 ) .
Revised: F ( α k +1 ) .

Original: Instead of directly fine-tuning π F ( α k ) on α k +1 , we adopt the following optimizationobjective with randomized evolution progression similar to [18] to achieve the goal:
Revised: Instead of directly fine-tuning π F ( α k ) on α k +1 , we adopt the following optimization objective with randomized evolution progression similar to [18] to achieve the goal:

Original: [Equation]
Revised: [Equation]

Original: where λ 1 ∈ (0 , 1) is the ratio of range shrinkage and e ∈ N is the iteration counter.
Revised: where λ 1 ∈ (0 , 1) is the ratio of range shrinkage and e ∈ N is the iteration counter.

Original: The aboveoptimization objective allows the policy to be fine-tuned on a set of randomly sampled robots withina range along the l k to improve the stability and robustness of training while also ensuring thesampled evolution parameter α to converge to α k
Revised: The above optimization objective allows the policy to be fine-tuned on a set of randomly sampled robots within a range along the l k to improve the stability and robustness of training while also ensuring the sampled evolution parameter α to converge to α k

Original: +

Revised: +

Original: =
Revised: =

Original: α k + l k when e → + ∞ .
Revised: α k + l k when e → + ∞ .

Original: In practice, a batchof β in Equation (6) is sampled in each iteration.
Revised: In practice, a batch of β in Equation (6) is sampled in each iteration.

Original: The overall algorithm is illustrated in Algorithm 1.
Revised: The overall algorithm is illustrated in Algorithm 1.

Original:  Related Work
Revised:  Related Work

Original: Learning from Human Demonstration Human demonstrations are often collected in terms oftrajectories that could be obtained by teleoperation via virtual reality, motion capture markers, orkinesthetic control of robot.
Revised: Learning from Human Demonstration Human demonstrations are often collected in terms of trajectories that could be obtained by teleoperation via virtual reality, motion capture markers, or kinesthetic control of robot.

Original: A policy is then learned by cloning the demonstrations [22, 23, 24, 25,26, 27]. or through inverse reinforcement learning [25, 28, 29, 30, 31].
Revised: A policy is then learned by cloning the demonstrations [22, 23, 24, 25, 26, 27] or through inverse reinforcement learning [25, 28, 29, 30, 31].

Original: In contrast to these methods,in our approach, the human does not control the robot to which the demonstration is transferred.
Revised: In contrast to these methods, in our approach, the human does not control the robot to which the demonstration is transferred.

Original: Human demonstrations need not always be in the form of state space information and a recent line ofwork rely on visual demonstration to enhance the scalability of data collection as it is much easier towatch humans than ask them to collect demos [32, 33, 34, 35, 36, 37, 38, 39, 40, 41].
Revised: Human demonstrations need not always be in the form of state space information and a recent line of work rely on visual demonstration to enhance the scalability of data collection as it is much easier to watch humans than ask them to collect demos [32, 33, 34, 35, 36, 37, 38, 39, 40, 41].

Original: Although in ourcase, the demonstrations are converted to the state space of the dexterous hand robot through inversekinematics, they are originally collected by watching humans.
Revised: Although in our case, the demonstrations are converted to the state space of the dexterous hand robot through inverse kinematics, they are originally collected by watching humans.

Original: Different from these prior works, weuse the demonstration to transfer the policy to very different robots than the demonstrator.
Revised: Different from these prior works, we use the demonstration to transfer the policy to very different robots than the demonstrator.

Original: Learning Controllers for New Robot Morphology Evolutionary methods to get new shapes dateback to the early days of automata theory from Von Neumman [42].
Revised: Learning Controllers for New Robot Morphology Evolutionary methods to get new shapes date back to the early days of automata theory from Von Neumman [42].

Original: In early 90s, Karl Sims showedthe effectiveness of evolutionary methods in optimizing robot shapes and controllers [43, 44].
Revised: In early 90s, Karl Sims showed the effectiveness of evolutionary methods in optimizing robot shapes and controllers [43, 44].

Original: Theseworks laid the foundation of the area where not only robots are optimized but also controllers aretransferred across evolved robots.
Revised: These works laid the foundation of the area where not only robots are optimized but also controllers are transferred across evolved robots.

Original: In recent years, many works have revived this idea using deeplearning based graph evolution [45, 46, 47, 48, 49].
Revised: In recent years, many works have revived this idea using deep learning based graph evolution [45, 46, 47, 48, 49].

Original: Our work does not treat the morphologicalevolution in the form of graph networks but rather gradually morphs the source robot to target robot.
Revised: Our work does not treat the morphological evolution in the form of graph networks but rather gradually morphs the source robot to target robot.

Original: Relation to Path Planning
Revised: Relation to Path Planning

Original: Our robot evolution path search problem is similar to the problem ofpath planning [50, 51, 52, 53, 54, 55].
Revised: Our robot evolution path search problem is similar to the problem of path planning [50, 51, 52, 53, 54, 55].

Original: However, unlike ordinary planning where the obstacles areknown, in our problem the difficult robot configurations are unknown.
Revised: However, unlike ordinary planning where the obstacles are known, in our problem the difficult robot configurations are unknown.

Original: Moreover, in ordinary pathplanning, evaluating the cost of a path is cheap, while in our case, transferring the policy to a newrobot requires a large number of policy optimization iterations and is computationally costly.
Revised: Moreover, in ordinary path planning, evaluating the cost of a path is cheap, while in our case, transferring the policy to a new robot requires a large number of policy optimization iterations and is computationally costly.

Original:  Experiments
Revised:  Experiments

Original: Our experiment is designed to verify the following two hypotheses: (1) our proposed DEPS algorithm can find optimized evolution paths in high-dimensional robot parameter space to improvepolicy transfer over vanilla linear evolution path [18]; (2) our HERD framework can learn fromhuman demonstrations recorded in diverse modalities including sensor glove signal and visual data.
Revised: Our experiment is designed to verify the following two hypotheses: (1) our proposed DEPS algorithm can find optimized evolution paths in high-dimensional robot parameter space to improve policy transfer over vanilla linear evolution path [18]; (2) our HERD framework can learn from human demonstrations recorded in diverse modalities including sensor glove signal and visual data.

Original: Experiment Settings We conduct experiments in MuJoCo physics simulation engine [56].
Revised: Experiment Settings We conduct experiments in MuJoCo physics simulation engine [56].

Original: Weadopt the five-finger dexterous hand robot provided in the ADROIT platform [57].
Revised: We adopt the five-finger dexterous hand robot provided in the ADROIT platform [57].

Original: The high-fidelitymodels of the target Sawyer robot and Rethink gripper are imported from robosuite environment [58]and have been used in previous robotics publications that eventually transferred their experiments toreal robots such as [59].
Revised: The high-fidelity models of the target robots and grippers are imported from robosuite environment [58] and have been used in previous robotics publications that eventually transferred their experiments to real robots such as [59, 60].

Original: We use NPG [60] as the RL algorithm for policy optimization.
Revised: We use NPG [61] as the RL algorithm for policy optimization.

Original: Due to thenature of robot-to-robot policy transfer, the total number of RL iterations it takes to reach the goal ofa certain success rate or episode reward cannot be set beforehand.
Revised: Due to the nature of robot-to-robot policy transfer, the total number of RL iterations it takes to reach the goal of a certain success rate or episode reward cannot be set beforehand.

Original: So we instead report the numberof policy training iterations and simulation epochs needed to reach 80% success rate on the tasks.
Revised: So we instead report the number of policy training iterations and simulation epochs needed to reach 80% success rate on the tasks.

Original: 5.1 Learning from Sensor Glove Signal Demonstrations
Revised: 5.1 Learning from Sensor Glove Signal Demonstrations

Original: Tasks and Demonstrations
Revised: Tasks and Demonstrations

Original: We use the three tasks from the Hand Manipulation Suite (HMS) [1]:
Revised: We use the three tasks from the Hand Manipulation Suite (HMS) [1]:

Original: Hammer , Relocate , and Door .
Revised: Hammer , Relocate , and Door .

Original: In Hammer , the task is to pick up the hammer and smash the nail into
Revised: In Hammer , the task is to pick up the hammer and smash the nail into

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: the board; In Relocate , the task is to pick up the ball and move it to the target position; In Door , the task is to turn the door handle and fully open the door.
Revised: the board; In Relocate , the task is to pick up the ball and move it to the target position; In Door , the task is to turn the door handle and fully open the door.

Original: We use a sparse reward setting where onlytask completion is rewarded.
Revised: We use a sparse reward setting where only task completion is rewarded.

Original: For each task, HMS [1] provided 25 human demonstrations recordedas palm and finger locations using a sensor glove named CyberGlove III.
Revised: For each task, HMS [1] provided 25 human demonstrations recorded as palm and finger locations using a sensor glove named CyberGlove III.

Original: Then the expert policy onthe dexterous robot is trained using DAPG [1].
Revised: Then the expert policy on the dexterous robot is trained using DAPG [1].

Original: Qualitative Results We provide visualization of the policies on the intermediate robots learnedduring evolution with our HERD in Figure 4.
Revised: Qualitative Results We provide visualization of the policies on the intermediate robots learned during evolution with our HERD in Figure 4.

Original: Starting from the dexterous robot, our proposed DEPSalgorithm is able to find the next optimized evolved robot.
Revised: Starting from the dexterous robot, our proposed DEPS algorithm is able to find the next optimized evolved robot.

Original: The policy is also able to iteratively adaptto new evolved robots and successfully transfer to the target Sawyer robot.
Revised: The policy is also able to iteratively adapt to new evolved robots and successfully transfer to the target robots.

Original: Quantitative Comparison with Baselines We compare our HERD against the following baselinemethods for learning a policy on the target robot: (1) From Scratch : we train policy on the targetrobot from scratch with the RL algorithm under both sparse and dense shaped rewards; (2) DAPG[1]: a variant of NPG [60] with demonstration-augmented policy gradient.
Revised: Quantitative Comparison with Baselines We compare our HERD against the following baseline methods for learning a policy on the target robot: (1) From Scratch : we train policy on the target robot from scratch with the RL algorithm under both sparse and dense shaped rewards; (2) DAPG [1]: a variant of NPG [61] with demonstration-augmented policy gradient.

Original: We use the rollouts oftransferred HERD policy on the target Sawyer robot as the demonstrations for DAPG; (3) REvolveR[18]: robot-to-robot policy transfer with continuous and linear robot evolution path which is a specialcase of our HERD.
Revised: We use the rollouts of transferred HERD policy on the target robots as the demonstrations for DAPG; (3) REvolveR [18]: robot-to-robot policy transfer with continuous and linear robot evolution path which is a special case of our HERD.

Original: The results are illustrated in Table 1.
Revised: The results are illustrated in Table 1.

Original: Training RL from scratch never completes the task to receive reward and is not able to learn on thesetasks.
Revised: Training RL from scratch never completes the task to receive reward and is not able to learn on these tasks.

Original: Robot-to-robot evolution solutions with sparse rewards are able to successfully transfer thepolicy and even significantly outperform training from scratch with shaped rewards.
Revised: Robot-to-robot evolution solutions with sparse rewards are able to successfully transfer the policy and even significantly outperform training from scratch with shaped rewards.

Original: Moreover, our
Revised: Moreover, our

Original: HERD outperforms REvolveR [18], though the margin in simulation epochs is smaller than RL iterations in that HERD spends additional simulation epochs to obtain Jacobian as in Equation (4).
Revised: HERD outperforms REvolveR [18], though the margin in simulation epochs is smaller than RL iterations in that HERD spends additional simulation epochs to obtain Jacobian as in Equation (4).

Original: Wenotice that our HERD also has advantage over DAPG trained using expert rollouts directly collectedon the target robot.
Revised: We notice that our HERD also has advantage over DAPG trained using expert rollouts directly collected on the target robot.

Original: This shows that training  policy while maintaining sufficient reward/successduring robot evolution has advantage over exploration from scratch even with demonstration.
Revised: This shows that training a policy while maintaining sufficient reward/success during robot evolution has advantage over exploration from scratch even with demonstration.

Original: 5.2 Learning From Visual Demonstrations
Revised: 5.2 Learning From Visual Demonstrations

Original: Task and Demonstrations
Revised: Task and Demonstrations

Original: In addition to the demonstration captured by sensor glove, we are alsointerested in showing learning from visual human demonstration with our HERD.
Revised: In addition to the demonstration captured by sensor glove, we are also interested in showing learning from visual human demonstration with our HERD.

Original: We use DexYCBdataset [2] as the source of human demonstration.
Revised: We use DexYCB dataset [2] as the source of human demonstration.

Original: DexYCB dataset is a multi-view RGB-D videodataset capturing human hand manipulating YCB objects [61].
Revised: DexYCB dataset is a multi-view RGB-D video dataset capturing human hand manipulating YCB objects [62].

Original: We define the task as grasping and
Revised: We define the task as grasping and

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: then moving the object of interest to the desired position within a distance threshold as demonstratedin the DexYCB videos.
Revised: then moving the object of interest to the desired position within a distance threshold as demonstrated in the DexYCB videos.

Original: We use 5 YCB objects in Table 2.
Revised: We use 5 YCB objects in Table 2.

Original: For each object, we take 2 videos indifferent scenes resulting in 10 demonstrations in total.
Revised: For each object, we take 2 videos in different scenes resulting in 10 demonstrations in total.

Original: The human hand joint poses and object 6Dposes are estimated by HRNet32 [62] and CozyPose [63] respectively.
Revised: The human hand joint poses and object 6D poses are estimated by HRNet32 [63] and CozyPose [64] respectively.

Original: Then the expert dexterouspolicy is trained using the method in Section 3.1.
Revised: Then the expert dexterous policy is trained using the method in Section 3.1.

Original: The reward is sparse task completion reward.
Revised: The reward is sparse task completion reward.

Original: Result Analysis and Comparison We compare against REvolveR [18] and illustrate the experiment results in Table 2.
Revised: Result Analysis and Comparison We compare against REvolveR [18] and illustrate the experiment results in Table 2.

Original: Our HERD outperforms REvolveR in terms of policy iterations as well asmost reported simulation epoch metrics.
Revised: Our HERD outperforms REvolveR in terms of policy iterations as well as most reported simulation epoch metrics.

Original: One interesting finding is that the policy for manipulating011 banana object is the easiest task among all objects.
Revised: One interesting finding is that the policy for manipulating 011 banana object is the easiest task among all objects.

Original: A possible reason is that 011 bananaobject has a geometry most suitable for both the dexterous hand and Sawyer gripper.
Revised: A possible reason is that 011 banana object has a geometry most suitable for both the dexterous hand and target gripper.

Original: Qualitative Results We visualize the policies on the intermediate robots learned during evolutionwith our HERD in Figure 5.
Revised: Qualitative Results We visualize the policies on the intermediate robots learned during evolution with our HERD in Figure 5.

Original: Again, our proposed DEPS algorithm is able to find the next optimizedevolved robot while successfully transferring the policy to the targetSawyer robot.
Revised: Again, our proposed DEPS algorithm is able to find the next optimized evolved robot while successfully transferring the policy to the target robot.

Original: For more detailson the visualizations in this section, please refer to the supplementary video.
Revised: For more details on the visualizations in this section, please refer to the supplementary video.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we propose a framework named HERD for learning from human demonstration.
Revised: In this paper, we propose a framework named HERD for learning from human demonstration.

Original: HERD framework utilizes the continuous evolution of robots to transfer the expert policy from embodied dexterous hands to a target robot.
Revised: HERD framework utilizes the continuous evolution of robots to transfer the expert policy from embodied dexterous hands to a target robot.

Original: We conduct experiments using the human demonstrationsfrom Human Manipulation Suite [1] and DexYCB dataset [2].
Revised: We conduct experiments using the human demonstrations from Human Manipulation Suite [1] and DexYCB dataset [2].

Original: We show that the proposed approachis able to transfer expert policy trained using human demonstrations in sensor glove signals or visualdata toa target two-finger Sawyer robot, even with challenging sparse rewards.
Revised: We show that the proposed approach is able to transfer expert policy trained using human demonstrations in sensor glove signals or visual data to target two-finger Sawyer and three-finger Jaco robots, even with challenging sparse rewards.

Original: Limitations We expect that our HERD would fail in learning from human demonstrations for robotswith extremely different morphology or function than humans, e.g. learning object manipulationpolicy for a robot arm mounted with a suction cup from human manipulation demonstrations.
Revised: Limitations We expect that our HERD would fail in learning from human demonstrations for robots with extremely different morphology or function than humans, e.g. learning object manipulation policy for a robot arm mounted with a suction cup from human manipulation demonstrations.

Revised: Acknowledgement This work is in part funded by JST AIP Acceleration, Grant Number JPMJCR20U1, Japan.

Revised: DP was supported by NSF IIS-2024594.