M ETA -E VOLVE : C ONTINUOUS R OBOT E VOLUTION FOR O NE - TO - MANY P OLICY T RANSFER
Anonymous authors
Paper under double-blind review
[Figure]
A BSTRACT
We investigate the problem of transferring an expert policy from a source robot to multiple different robots.
To solve this problem, we propose a method name
Meta-Evolve that uses continuous robot evolution to efficiently transfer the policy to a newly defined meta robot and then to each target robot.
Since the meta robot is closer to the target robots, our approach can significantly naive one-to-one policy transfer.
We also present three heuristic approaches with theoretical results to determine the meta robot.
Experiments have shown that with three target robots, our method is able to improve over the baseline of launching multiple independent one-to-one robot-to-robot policy transfers by up to 2.4 × in terms of training and exploration needed.
I NTRODUCTION
The robotics industry has designed and developed a large number of commercial robots deployed in various applications.
How to efficiently learn robotic skills on diverse robots in a scalable fashion?
A popular solution is to train a policy for every new robot on every new task from scratch.
This is not only inefficient in terms of sample efficiency but also impractical for complex robots due to a large exploration space.
Statistic matching Imitation learning (IL) methods that optimize to match the distribution of actions (Ross et al., 2011), transitioned states (Liu et al., 2019; Radosavovic et al., 2020), or reward function (Ng et al., 2000; Ho & Ermon, 2016) could be possible solutions.
However, they can only be applied to robots with similar dynamics to yield optimal performance.
Recent advances in evolution-based imitation learning (Liu et al., 2022a;b) inspires us to view this problem from the perspective of policy transferring from one robot to another.
The core idea is to interpolate two different robots by producing a large number of intermediate robots between them which gradually evolve from the source robot toward the target robot.
These continuously evolving robots act as the bridge for transferring the policy from the source robot to the target robot.
The source robot is usually selected as a robot such that it is easy to collect sufficient demonstrations to train a high-performance expert policy, e.g. a Shadow Hand robot (ShadowRobot, 2005) that can be trained from large-scale human hand demonstration data such as Grauman et al.
and Damen et al.
While the continuous robot evolution has shown success in learning challenging robot manipulation tasks (Liu et al., 2022a), the policy transfer is limited to being between a pair of robots.
As illustrated in Figure 1(a), for N different target robots, it requires launching N independent runs of robot-to-robot policy transfer and is not scalable.
How can one efficiently transfer a well-trained policy from one source robot to multiple different target robots?
Our intuition is that when robots are designed to complete certain tasks, they often share similar forms of morphology and dynamics to interact with other objects in similar ways.
Examples include robot grippers that are all designed to close their fingers to grasp objects and multi-legged robots that are all designed to stretch their legs for agile motion.
To transfer the policy to N different target robots to complete similar tasks, it may be possible to find a meta robot that is close to all N target robots during continuous robot evolution.
Then the robot evolution and policy transfer can first reach the meta robot and then launch N independent robot-to-robot policy transfers toward each of the N target robots.
In this way, the initial part of the robot evolution path between the source and the meta robot can be shared among N target robots, and can significantly reduce the amount of exploration and training cost needed if the meta robot is much closer to the target robots.
The idea is illustrated in Figure 1(b).
We propose a method named Meta-Evolve to instantiate the above idea that uses robot microevolution for efficiently transferring a policy from one source robot to multiple target robots.
Given the source and an arbitrary number of target robots, our method first matches their morphology.
This allows the source and target robots to be represented in the same high-dimensional continuous space of physical parameters and also allows the sampling of new intermediate robots.
We propose three different ways to determine the meta robot within the continuous space by heuristics for minimizing the total cost of policy transfer training and exploration.
We showcase our Meta-Evolve on three Hand Manipulation Suite (Rajeswaran et al., 2018) manipulation tasks where the source robot is a five-finger dexterous hand and the target robots are three robot grippers with two, three, and four fingers respectively.
Our Meta-Evolve is able to reduce the total amount of simulation epochs and training iterations by up to 2.4 × compared to pairwise robot policy transfer methods (Liu et al., 2022a;b) to reach the same performance on the three target robots.
To summarize, our contributions are as follows:
• We introduce a new research problem of one-to-many policy transfer where the goal is to transfer an expert policy from a source robot to multiple target robots.
Different from traditional multi-task learning, we propose to use continuous robot evolution as the formulation which allows studying this problem from a robot or task evolutionary perspective.
• We develop a method named Meta-Evolve as a viable solution to the new problem.
We propose to employ a meta robot for efficient evolution path sharing among target robots and proposed three different ways of determining the meta robots heuristically.
• We conduct experiments on Hand Manipulation Suite tasks and our Meta-Evolve shows significant gains compared to pairwise policy transfer baselines.
This shows that our Meta-Evolve can allow more scalable imitation learning and learning from demonstration for diverse robots.
P RELIMINARY
Notation We use bold letters to denote the vector variables.
Specially, 0 and 1 are the all-zero and all-one vectors with proper dimensions respectively.
⊙ is the element-wise product.
For a vector θ , we use θ ( j ) to denote its j -th element.
We use MAX ( · ) and MIN ( · ) to denote element-wise maximum and minimum of a set of vectors respectively.
MDP Preliminary We consider a continuous control problem formulated as Markov Decision
Process (MDP).
It is defined by a tuple ( S , A , T , R, γ ) , where S ⊆ R S is the state space, A ⊆ R A is the action space, T : S ×
A → S is the transition function, R : S ×
A → R is the reward function, and γ ∈ [0 , 1] is the discount factor.
A policy π : S → A maps a state to an action where π ( a | s ) is the probability of choosing action a at state s .
Suppose M is the set of all MDPs and ρ π,M = (cid:80) ∞
t =
γ t R ( s t , a t ) is the episode discounted reward with policy π on MDP M ∈ M .
The optimal policy π ∗ M on MDP M is the one that maximizes the expected value of ρ π,M .
REvolveR and HERD Preliminary Liu et al.
(2022b) proposed a technique named REvolveR for transferring policies from one robot to a different robot.
Given a well-trained expert policy π ∗ M S on a source robot M S ∈ M , its goal is to find the optimal policies π ∗ M T on another target robot M T ∈ M .
The core idea is to define an evolutionary sequence of intermediate robots that connects the two robots through linear interpolation of physical parameters and sequentially fine-tune the policy on each intermediate robot in the sequence.
Liu et al.
(2022a) proposed HERD, which extends the idea to robots represented in high-dimensional parameter space, and proposes to optimize the robot evolution path together with the policy.
Concretely, given source and target robots M S , M T ∈ M that are parameterized in D -dimensional space respectively, HERD defines a continuous function
F : [0 , 1] D → M where F ( 0 ) =
M S , F ( 1 ) =
M T .
Suppose the physical parameters of the source and target robots are θ S , θ T ∈ R D respectively.
For any evolution parameter α ∈ [0 , 1] D , F ( α ) defines an intermediate robot whose physical parameters are θ =
(1 − α ) ⊙ θ S + α ⊙ θ T .
Then an expert policy π ∗ F ( 0 ) on the source robot F ( 0 ) is optimized by sequentially interacting with each intermediate robot in the sequence F ( α 1 ) , F ( α 2 ) , . .
, F ( α K ) where α K = 1 , until the policy is able to act (near) optimally on each intermediate robot.
At robot F ( α k ) , the optimization objective for finding the next best intermediate robot F ( α k +1 ) =
F ( α k + l k ) is formulated as
[Equation]
(1) which optimizes both the expected policy reward E [ ρ π,F ( α k + l k ) ] and the L 2 distance to the target robot || 1 − ( α k + l k ) ||
For all k , the evolution step size ξ =
|| l k || 2 is small enough so that each policy fine-tuning step is a much easier task.
After K steps, the policy will eventually be transferred to the target robot F ( 1 ) .
The idea is illustrated in Figure 1(a).
O NE - TO -M ANY R OBOT - TO -R OBOT P OLICY T RANSFER 3. P ROBLEM S TATEMENT
We investigate the problem of transferring a policy from one source robot to multiple target robots.
Formally, we consider a source robot M S ∈ M and N target robots M T , 1 , M T , 2 , . . .
, M T ,N ∈ M respectively.
We assume the state space S , and action space A , reward function R and discount factor γ of M S and all M T ,i are shared and the difference is their transition dynamics T .
Given a well-trained expert policy π ∗ M S on a source robot M S , the goal is to find the optimal policies π ∗ M T ,i on each of the target robot M T ,i .
We would like to investigate using the information in π M S to improve the sample efficiency as well as the final performance of π M T ,i and also study how the learning of each individual π M T ,i can help each other during the process.
We approach this problem by defining a meta robot M Meta ∈ M that shares the same state space, action space, reward and discount factor.
The robot M Meta is designed such that M Meta sits in the middle of M S and all M T ,i , i.e. in terms of transition dynamics, M Meta is closer to all M T ,i than M S while also being closer to M S than all M T ,i .
Therefore, instead of repeating the process of directly transferring a source robot policy π S from M S to M T ,i for N times, we first transfer π S to M Meta and then from M Meta to each individual M T ,i .
3. M ULTI - ROBOT M ORPHOLOGY M ATCHING AND R OBOT R EPRESENTATIONS
Our problem setting and our proposed solution are based on an assumption that the source robot M S and target robots M T ,i share the same state and action space based on which the intermediate robots can be defined.
The assumption is true for a pair of robots as shown in Liu et al.
(2022a;b) where the intermediate robots are produced by robot morphology matching and kinematic interpolation.
In this subsection, we show that this assumption can be extended to more than two robots.
Morphology Matching
The kinematic tree of a robot describes the connection of the bodies and joints.
The morphology of the robot, i.e. the topology of its kinematic tree, represents the kinematic behavior of the robot.
Given two different robots with different morphology, it has been shown in Liu et al.
(2022b) that their kinematic trees can be matched by adding extra nodes and edges.
This step [Figure] can be easily extended to N robots where N > 2 .
As illustrated in Figure 2(a), by matching proper root nodes and key leaf nodes, the kinematic tree of the morphology-matched robot is essentially a graph union of the kinematic trees of all robots.
This means each robot needs to create additional bodies, joints and motors, though they may be all zero in their numbers.
The above kinematic matching process can be automated by an algorithm that achieves the best matching of nodes and edges across N robots.
However, in practice, we would like the matching process to include reasonable human intervention with enough robotics knowledge, e.g. matching human hand fingers to robot gripper fingers where the knuckle joints are matched correctly.
Kinematic Interpolation After morphology matching, the state and action space of the robots are matched.
The difference in the robot transition dynamics is now only due to the differences in physical parameters, such as shapes and mass of robot bodies, gain and armature of joint motors, etc.
Suppose there are D physical parameters.
Then each θ ∈ R D uniquely represents a new robot.
Suppose the physical parameters of the source robot and the N target robots are represented in Ddimensional space as θ S ∈ R D and θ T , 1 , θ T , 2 , . .
, θ T ,N ∈ R D respectively.
In each dimension, we compute the upper and lower bounds of the physical parameters
[Equation]
where θ u and θ l essentially defines the convex hull of the set of robot physical parameters in Ddimensional space H
=
[ θ l (1) , θ u (1)]
× [ θ l (2) , θ u (2)] ×· · · [ θ l ( N ) , θ u ( N )]
∈ R D that encompasses the source and all target robots.
We can now use continuous function F : [0 , 1] D → M to define an intermediate robot by interpolation between all pairs of kinematic parameters
[Equation]
where α ∈ [0 , 1] D is the evolution parameter that describes the normalized position of a robot in the convex hull H .
Note that by limiting α to be between 0 and 1 , we assume the convex hull H is the set of all possible intermediate robots.
This is a reasonable assumption since an out-of-range parameter can be physically dangerous and also unlikely to be useful in robot continuous interpolation.
The convex hull H also serves as the metric space that measures the distance between any two robots.
In HERD (Liu et al., 2022a), the source and target robots are always represented as 0 and 1 respectively because when there are only two robots, one of them must be either lower or upper bound.
Different from HERD, the source and target robots in our problem are not necessarily 0 and 1 .
An example of the resulting robot evolution space and the positions of the source and target robots in the evolution space is illustrated in Figure 2(b).
More details can be found in Section B.
3. O NE - TO - MANY R OBOT E VOLUTION FOR P OLICY T RANSFER
Suppose the source robot and the N target robots are represented by F ( β 0 ) :
= M S and
F ( β 1 ) :
= M T , 1 , . .
, F ( β N ) :
= M T ,N respectively where β i ∈ [0 , 1] D .
Similar to
HERD (Liu et al., 2022a), we employ N robot evolution paths of intermediate robots
τ i =
( F ( α i, 1 )) , F ( α i, 2 )) , . . .
, F ( α i,K i ))) ,
i = 1 , 2 , . .
, N where K i ∈
Z + , F ( α i, 1 ) =
F ( β 0 ) is the source robot and F ( α
i,K i ) =
F ( β i ) is the target robot i .
Following HERD (Liu et al., 2022a), we use intermediate robots F ( α i,k ) as the bridge to transfer the policy to target robot i .
We use K i training phases of policy optimizations.
At training phase k , the policy is trained on policy rollouts from robots sampled on the line α k α k
+1 .
The sampling window gradually converges to α k +1 during training until the policy is able to achieve enough success rate on F ( α k +1 ) before moving on to the next training phase k + 1 .
For all k , we set the evolution step size ||
α i,k − α i,k +1 || 2 = ξ to be small enough so that each training phase is an easy sub-task with little policy fine-tuning.
Naively following HERD would require training through all the τ i for N time in total.
However, if the target robots β i are mutually similar but are very different from the source robot, during the start of the training phases, the robot would evolve roughly in similar directions.
Therefore, the N robot evolution paths would be close to each other near the start of the paths and the policy optimization might be redundant, as illustrated in Figure 1(a).
We propose to design the N evolution paths by forcing the first m ∈ Z + intermediate robots to be shared among the N paths to address the redundancy issue at the start of the training.
Formally,
[Equation]
This means that all path will first reach a shared robot F ( α i,m ) before splitting their ways towards each target robot.
We name the last shared robot “ meta robot ”, denoted by F ( β Meta ) :
=
F ( α i,m ) .
To summarize, given a defined meta robot, the policy transfer to N target robots consists of the following two steps:
Transfer source expert policy π ∗ F ( β 0 ) using HERD from the source robot F ( β 0 ) to the meta robot F ( β Meta ) to obtain a well-trained meta robot policy π F ( β Meta ) ; 2.
For i = 1 , 2 , . . .
, N , transfer the meta robot policy π F ( β Meta ) using HERD from the meta robot F ( β Meta ) to the target robot F ( β i ) to obtain target robot policy π F ( β i ) ;
In this way, both simulation for exploration and training during policy transfer can be saved significantly.
Theoretically, if the target robots are close enough to the target robots, or the policy transfer from the meta robot to the target robots is easy enough, we could expect our Meta-Evolve a speedup up to O ( N ) compared to one-to-one policy transfer baselines.
3. M ETA R OBOT D ETERMINATION
Given the source and target robots, the meta robot significantly impacts the overall performance of the policy transfer.
However, due to the complexity of the robots’ physical parameter space and its relation to the actual MDP transition dynamics, it is extremely difficult to develop a universal solution.
We hereby provide the following three heuristics for determining the meta robot.
Geometric Median of Source and Target Robots (denoted as “ Geom-Med All ”
) We aim to minimize the total L 2 travel distance in robot evolution parameter space from the source robot F ( β 0 ) to the meta robot F ( β Meta ) and then to all target robots β i .
Mathematically, a point that minimizes the sum of distances to a set of points is called the geometric median of the point set.
Then the meta robot will be the geometric median of evolution parameter set { β 0 , β 1 , . .
. , β N } :
[Equation]
{+Note that by using geometric median, we assume that the training cost for transferring the policy from robot F ( α i,k ) to robot F ( α i,k +1 ) is proportional to || α i,k − α i,k +
 || 2 . We believe this is a reasonable assumption since the training cost should be proportional to the distribution difference D KL ( F ( α i,k ) , F ( α i,k +1 ))+} of the MDPs of the two robots F ( α i,k ) , F ( α i,k +1 ) and should be proportional to the robot hardware difference
||
{+α i,k − α i,k +
+}
{+|| 
+}
when ||
α i,k − α i,k +1 || 2 → 0 , ∀ k .
Geometric Median of Target Robots Only (denoted as “ Geom-Med Target only ”)
At each training phase of the policy transfer, the algorithm should aim to reduce the expected future cost of training instead of the past.
Therefore, at training phase k , the algorithm should act greedily to minimize the total L 2 travel distance from the current robot F ( α i,k ) to the meta robot F ( β Meta ) and then to all target robots F ( β i ) , i.e. the temporary meta robot at phase k should be determined as the geometric median of evolution parameter set { α i,k , β 1 , . .
. , β N } :
[Equation]
As long as α i,k moves towards the goal of β Meta ,k and reduces its distance to α i,k , in training phase k , the β Meta ,k will converge to the geometric median of evolution parameter set { β 1 , . .
. , β N } as shown by Theorem 3.1.
For simplicity and to avoid introducing additional noise due to the change of meta robot goal, we directly use the following β Meta, Target as the meta robot in our algorithm.
[Equation]
Theorem 3.1.
Suppose β 1 , β 2 , . .
, β N are not collinear.
Suppose β Meta ,k is defined by Equation (6) , and for all k, ||
α i,k +1 − β Meta ,k || 2 <
||
α i,k − β Meta ,k || 2 , then lim k →∞ β Meta ,k = β Meta, Target .
The proof of Theorem 3.1 is in Section A.
Minimum Distance to Target Robots’ Convex Hull (denoted as “ Min Dist to Cvx Hull ”)
In general, we cannot have complete knowledge about how the change of a certain parameter affects the MDP transition dynamics of a robot.
Therefore, it may be wise to set the meta robot to be outside the ranges of target robot parameters and let the subsequent independently launched HERD runs figure out the optimized evolution path themselves.
At the same time, we would prefer the meta robot to be as close to the target robots as possible.
Thus, we can choose the point on the surface of the convex hull spanned by the target robots that has the minimum distance to the source robot: ∀ d, β Meta, Min
[Equation]
otherwise
[Equation] [Equation]
R ELATED W ORK
Imitation Learning across Different Robots
Traditional imitation learning is designed for learning on the same robots such as Ross et al.
(2011); Ng et al.
; Ho & Ermon (2016); Duan et al.
(2017); Ke et al.
However, due to a huge mismatch in transition dynamics, these works often struggle in learning across different robots.
Our work can be viewed as imitation learning across different robots.
Compared to previous imitation learning methods that aim to learn across different robots directly (Radosavovic et al., 2020; Liu et al., 2019), we aim to employ robot evolution to gradually adapt the policy.
Furthermore, our Meta-Evolve focuses on one-to-many imitation where the policy must work on multiple target robots.
Learning Controllers for Diverse Robot Morphology
Recent work has studied the problem of learning a policy/controller for diverse robots.
For instance, Pathak et al.
(2019) uses dynamic graph neural networks to control and develop robots with different morphology simultaneously to build agents that can generalize to new scenarios.
Besides, GNNs have been used to control diverse robot morphology in NerveNet (Wang et al., 2018) to control different robots.
Huang et al.
leverages modularity using graph neural networks across limbs of robots to train agent-agnostic policies.
Hierarchical controllers have also been shown to be effective while transferring across morphology (Hejna et al., 2020).
In contrast to these works, we do not co-develop the controller with morphology but transfer the policy from a source robot to multiple target robots by simulating an evolutionary process.
Meta Learning Our Meta-Evolve is closely related to the formulation of meta-learning (Finn et al., 2017; 2018; Rajeswaran et al., 2019).
Different from meta reinforcement learning where only the policies π are meta learned, our formulation can be viewed as the continuous update of both the policy π as well as the transition dynamics T instantiated by different robot parameters θ . Moreover, [Figure]
[Figure] [Table] while meta-learning aims to learn a meta policy from scratch, in our problem, the source expert policy is provided and used in policy transfer.
Closely related to our approach is task interpolation for meta-learning (Yao et al., 2021).
Different from task interpolation, our policy transfer does not require working on multiple robots at the same time but only needs each transferred policy to work on one target robot.
E XPERIMENTS
The design of our Meta-Evolve method is motivated by the hypothesis that by sharing the evolution paths among multiple robots through the design of the meta robot, the overall cost of one-to-many policy transfer can be reduced.
To show this, we apply our Meta-Evolve on Hand Manipulation
Suite (HMS) (Rajeswaran et al., 2018).
5. E XPERIMENT S ETTINGS
Source and Target Robots
We utilize the five-finger dexterous hand provided in the ADROIT platform (Kumar et al., 2013) as the source robot and follow Rajeswaran et al.
(2018) for the initial environment settings.
The target robots are three robot grippers with two, three, and four fingers respectively.
The target robots can be produced by gradually shrinking the fingers of the five-finger dexterous hand.
The robot evolution is illustrated in Figures 1 and 2(b).
Task, Reward and RL Algorithm We use the three tasks from the the task suite in (Rajeswaran et al., 2018):
Hammer , Relocate and Door .
In Hammer , the task is to pick up the hammer and smash the nail into the board; in Relocate , the task is to pick up the ball and take it to the target position; in Door , the task is to turn the door handle and fully open the door.
We use a challenging sparse reward function where only the task completion is rewarded.
We use NPG (Rajeswaran et al., 2017) as the RL algorithm in all compared methods.
Evaluation Metrics For each compared method, the goal is to reach 80% success rate on all three target robots.
Due to the nature of one-to-many policy transfer, the total number of RL iterations or simulation epochs it takes to reach this goal cannot be set beforehand.
So we instead report the number of policy training iterations and simulation epochs needed to reach the goal. [Figure] [Figure] [Table]
[Table]
5. O NE - TO - THREE R OBOT - TO - ROBOT P OLICY T RANSFER
We compare our Meta-Evolve against one-to-one policy transfer baseline HERD (Liu et al., 2022a).
The results on Hammer , Door , Relocate are illustrated in Tables 1, 2 and 3 respectively.
In terms of the total number of simulation epochs and training iterations needed, our method is able to achieve up to 2.4 × improvement on these three tasks.
Breaking down each part of the evolution path, we noticed that in our method, the source robot to meta robot part of the path is usually the most costly and constitutes the largest portion of simulation epochs as well as training, e.g. on Relocate and Door tasks.
However, the cost after splitting the path at the meta robot is much smaller which yields a smaller total cost.
A more interesting observation is that, for some target robots and tasks, e.g. two- and four-finger target robots on Hammer task, the total cost of path “source to meta robot” plus the cost of path “ meta robot to target robot” is even smaller than the cost of directly transferring the policy to that target robot using HERD.
It shows that, transferring the policies to multiple related target through a meta robot can possibly help each robot improve their learning efficiency.
We believe this phenomena deserve more future research attention.
5. A BLATION S TUDIES
We provide ablation studies on the design choice of meta robot.
The results are illustrated in Tables 1, 2 and 3.
Choosing the meta robot as the minimum distance point to the target robot convex hull (i.e. “Min Dist to Cvx Hull”) and geometric median of only the target robots (i.e. “Geom-Med
Target only”) yield similar performance and are much better than choosing the geometric median of both source and target robots (i.e. “Geom-Med All”).
After further analysis, a possible reason is that the first two approaches produce meta robots that are much closer to the targets than the last approach.
It shows that the meta robot should be selected to be closer to the target robot if possible.
5. V ISUALIZATIONS
We provide visualizations of the transferred policies on each of the three target robots as well as the meta robot, on all three HMS tasks.
The policy is able to be transferred to all target robots and successfully complete the task.
Moreover, we observe that the meta robot obtained from the heuristics we introduced in Section 3.4 looks like a deformed three-finger gripper which is indeed
[Figure]
[Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
close to all three target robots, considering the target robots include two-, three- and four-finger grippers respectively. For more details on the visualization, please refer to the supplementary video.
D ISCUSSIONS
When does Meta-Evolve Fail?
One core assumption behind our Meta-Evolve method is that the target robots are mutually similar while very different from the source robot.
If the evolution goal is in opposite directions, our Meta-Evolve will not work.
For example, given the source robot of a five-finger hand, if one target robot is a ten-finger hand while another target robot is a twofinger gripper, the optimal meta robot might simply be the source robot itself, because the evolution directions toward the target robots are opposite (growing multiple fingers vs. shrinking multiple fingers).
However, in practice, most industrial robots are indeed mutually similar in morphology and kinematics while very different from the human hand, so our assumption still stands and MetaEvolve can still be useful in such cases.
Can the Meta Robot be Learned or Optimized?
We envision the learning or optimization of the meta robot being very challenging.
Policy transfer through robot evolution relies on local optimization in the robot evolution space.
On the other hand, the optimization of the meta robot requires optimizing the robot evolution path globally and needs an accurate “guess” of the results of future policy transfer.
We leave the problem of finding an optimized meta robot as the future work.
C ONCLUSION
In this paper, we introduce a new research problem of transferring an expert policy from a source robot to multiple target robots.
To solve this new problem, we introduce a new method name MetaEvolve that utilizes continuous robot evolution to efficiently transfer the policy to a meta robot and then to each target robot.
We also present three different heuristic approaches with theoretical results to determine the meta robot.
We conduct experiments on Hand Manipulation Suite tasks and show that our one-to-many policy transfer method can significantly outperform the one-to-one policy transfer baseline.
A P ROOF OF T HEOREM 3.
Proof.
For all k , the definition of β Meta ,k is
[Equation]
Thus we have
[Equation]
Since for all k, ||
α i,k +1 − β Meta ,k || 2 <
||
α i,k − β Meta ,k || 2 , therefore ||
[Equation]
Let d k = ||
β Meta ,k − α i,k ||
+ (cid:80) Ni
=
|| β Meta ,k − β
i || 2 , Equation (11) can be re-written as
[Equation]
(12) which means the sequence { d k } is a decreasing sequence.
By definition, d k is a summation of distance, so ∀ k, d k > 0 .
Therefore, according to the Monotone Convergence Theorem, the sequence { d k } must converge to its lower bound inf { d k } .
Since ∀ β ,
[Equation]
The last inequality is due to the definition of β Meta, Target .
Since there exists a solution of α i,k = β Meta, Target = arg min β (cid:80) Ni
=1 || β −
β
i || 2 that can turn all two inequalities in Equation (13) into equality.
Also considering that β 1 , β 2 , . .
, β N are not collinear, according to Vardi & Zhang (2000), the solution to the geometric median β Meta, Target is unique given β 1 , β 2 , . .
. , β N .
This means the lower bound inf { d k } is achieved if and only at β Meta,
Target = arg min β (cid:80) Ni
=1 || β −
β
i || 2 which also means
[Equation]
B R OBOT E VOLUTION S PECIFICS
We illustrate the kinematic tree of the source robot in Figure 6.
During evolution, all revolute joints gradually freeze to have a range of 0.
On the other hand, the prismatic joints are initially frozen with a range of 0, and some of their ranges gradually increase until the same full range.
During robot
P
R
F
RR
R
RPR
R
R
RPR
R
R R
R R
R
RR
R
R R
R R
F
RR
Free Joint
Prismatic Joint
Revolute Joint
Joints Frozen in Source Robot, Evolve to be Active in
Target Robot
Joints Active in Source Robot Hand, Evolve to be Frozen in
Target Robot
Body
P P
Hyperparameter
Value
RL Discount Factor γ 0.
GAE 0.
NPG Step Size 0.
Policy Network Hidden Layer Sizes (32,32)
Value Network Hidden Layer Sizes (32,32)
Simulation Epoch Trajectory Length
RL Traning Batch Size
Evolution Progression Step Size ξ 0.
Number of Sampled δ i for Jacobian Estimation n
Evolution Direction Weight Factor λ 1.
Sample Range Shrink Ratio λ0.
[Table]
Threshold
[Table]
evolution, the body of the ring finger gradually shrinks to be zero-size and disappears.
Our evolution solution includes the changing of D = 65 independent robot parameters.
C H YPERPARAMETER AND T RAINING D ETAILS
We present the hyperparameters and training procedures of our robot evolution and policy optimization.
We use PyTorch (Paszke et al., 2019) as our deep learning framework and NPG (Rajeswaran et al., 2017) as the RL algorithm in all experiments.
To fairly compare against HERD (Liu et al., 2022a), we use the same evolution progression step size ξ as HERD in the experiments.
The hyperparameters are illustrated in Table 4.