Original: GenLoco: Generalized Locomotion Controllers for Quadrupedal Robots
Revised: GenLoco: Generalized Locomotion Controllers for Quadrupedal Robots

Original: Abstract: Recent years have seen a surge in commercially-available and affordablequadrupedal robots, with many of these platforms being actively used in researchand industry.
Revised: Abstract: Recent years have seen a surge in commercially-available and affordable quadrupedal robots, with many of these platforms being actively used in research and industry.

Original: As the availability of legged robots grows, so does the need forcontrollers that enable these robots to perform useful skills.
Revised: As the availability of legged robots grows, so does the need for controllers that enable these robots to perform useful skills.

Original: However, most learningbased frameworks for controller development focus on training robot-speciﬁccontrollers, a process that needs to be repeated for every new robot.
Revised: However, most learningbased frameworks for controller development focus on training robot-speciﬁc controllers, a process that needs to be repeated for every new robot.

Original: In thiswork, we introduce a framework for training generalized locomotion controllers(GenLoco) for quadrupedal robots.
Revised: In this work, we introduce a framework for training generalized locomotion (GenLoco) controllers for quadrupedal robots.

Original: Our framework synthesizes general-purposelocomotion controllers that can be deployed on a large variety of quadrupedalrobots with similar morphologies.
Revised: Our framework synthesizes general-purpose locomotion controllers that can be deployed on a large variety of quadrupedal robots with similar morphologies.

Original: We present a simple but effective morphologyrandomization method that procedurally generates a diverse set of simulated robotsfor training.
Revised: We present a simple but effective morphology randomization method that procedurally generates a diverse set of simulated robots for training.

Original: We show that by training a controller on this large set of simulatedrobots, our models acquire more general control strategies that can be directlytransferred to novel simulated and real-world robots with diverse morphologies,which were not observed during training. 
Revised: We show that by training a controller on this large set of simulated robots, our models acquire more general control strategies that can be directly transferred to novel simulated and real-world robots with diverse morphologies, which were not observed during training. (Code and pretrained policies: https://github.com/ HybridRobotics/GenLoco , Video: https://youtu.be/5QUs32MjNu4 )

Original:  Introduction
Revised:  Introduction

Revised: Just as more general-purpose models have gained prominence in supervised learning domains, with broadly applicable language models that can solve a variety of NLP tasks [1, 2] and languageconditioned visual recognition or image generation models that can be applied to a variety of problems and settings [3, 4], so too we might expect that more powerful robotic learning systems might enable more broadly applicable robotic controllers.

Revised: In particular, in settings where a variety of robotic platforms share the same basic morphology (e.g., the set of commonly used quadrupedal bodies, the set of 7 DoF robotic arms, the set of quadcopters, etc.), we might expect that it should be possible to train control policies that can be applied broadly to all robots within a particular set.

Revised: If this were possible, then practitioners who want to make use of learned policies would not need to start by training policies of their own for their own robot, but could instead simply download a pre-trained policy for the general robot class from the web, and then deploy directly on their platform.

Revised: In this paper, we take a step toward this vision in the particular setting of quadrupedal locomotion.

Original: Quadrupedal locomotion offers the potential for robotic agents to traverse and operate in complexunstructured environments.
Revised: Quadrupedal locomotion offers the potential for robotic agents to traverse and operate in complex unstructured environments.

Original: However, designing effective locomotion controllers for quadrupedalrobots is challenging, as it typically requires detailed knowledge of the dynamics of a particularsystem and careful controller design for each desired skill.
Revised: However, designing effective locomotion controllers for quadrupedal robots is challenging, as it typically requires detailed knowledge of the dynamics of a particular system and careful controller design for each desired skill.

Original: Model-free reinforcement learning (RL)provides a paradigm that can automate much of the controller engineering process, where an agentlearns locomotion skills automatically through trial-and-error.
Revised: Model-free reinforcement learning (RL) provides a paradigm that can automate much of the controller engineering process, where an agent learns locomotion skills automatically through trial-and-error.

Original: RL techniques have been effectivefor developing locomotion controllers for a large variety of quadrupedal robots [1, 2, 3, 4, 5, 6, 7].
Revised: RL techniques have been effective for developing locomotion controllers for a large variety of quadrupedal robots [5, 6, 7, 8, 9, 10, 11, 12, 13].

Original: While RL provides a general framework that can in principle be applied to any robot, the resultingcontrollers are most often speciﬁc to the particular robot that they were trained on.
Revised: While RL provides a general framework that can in principle be applied to any robot, the resulting controllers are most often speciﬁc to the particular robot that they were trained on.

Original: Therefore, thesecontrollers will generally be ineffective when deployed on another robot, and new  controllers willneed to be trained from scratch speciﬁcally for the new system.
Revised: Therefore, these controllers will generally be ineffective when deployed on another robot, and new [Figure] controllers will need to be trained from scratch speciﬁcally for the new system.

Original: This requires either extensive datacollection or, if using simulation to real-world transfer, detailed simulated models.
Revised: This requires either extensive data collection or, if using simulation to real-world transfer, detailed simulated models.

Original: Recent years have seen the emergence of a growing catalog of commercially available quadrupedalrobots, with many of these systems converging on similar body plans.
Revised: Recent years have seen the emergence of a growing catalog of commercially available quadrupedal robots, with many of these systems converging on similar body plans.

Original: This similarity in the morphologies of quadrupedal robots may lead one to wonder: is it possible to create a generalized locomotioncontroller that can be broadly applied to different quadrupedal robots?
Revised: This similarity in the morphologies of quadrupedal robots may lead one to wonder: is it possible to create a generalized locomotion controller that can be broadly applied to different quadrupedal robots?

Original: If so, such generalizedcontrollers can greatly reduce the labor-intensive process of continually developing robot-speciﬁccontrollers for new systems.
Revised: If so, such generalized controllers can greatly reduce the labor-intensive process of continually developing robot-speciﬁc controllers for new systems.

Original: In this work, we present a framework for developing more generallocomotion controllers, which can be applied to a range of different quadrupedal robots.
Revised: In this work, we present a framework for developing more general locomotion controllers, which can be applied to a range of different quadrupedal robots.

Original: We focusour investigation on the setting where the test-time morphology is unknown, and thus the goal is todevelop controllers that generalize to new robots not observed during training.
Revised: We focus our investigation on the setting where the test-time morphology is unknown, and thus the goal is to develop controllers that generalize to new robots not observed during training.

Original: To this end, we proposea morphology randomization method that reduces the need for robot speciﬁc information duringtraining by procedurally generating a diverse set of morphologies.
Revised: To this end, we propose a morphology randomization method that reduces the need for robot speciﬁc information during training by procedurally generating a diverse set of morphologies.

Original: By randomizing the morphology[Figure] and dynamics of the simulated robots, our system is able to train generalized controllers that can bedeployed across a variety of different robots, as shown in Fig.
Revised: By randomizing the morphology and dynamics of the simulated robots, our system is able to train generalized controllers that can be deployed across a variety of different robots, as shown in Fig.

Original: The core contribution of this work is the development of a reinforcement learning framework thatproduces generalized locomotioncontrollers (GenLoco)  for quadrupedal locomotion skills, whichcan be deployed on a large variety of different robot morphologies.
Revised: The core contribution of this work is the development of a reinforcement learning framework that produces generalized locomotion (GenLoco) controllers for quadrupedal locomotion skills, which can be deployed on a large variety of different robot morphologies.

Original: By training controllers on a widerange of randomly generated robot morphologies, our system is able to learn policies that generalizeto new robots not seen during training.
Revised: By training controllers on a wide range of randomly generated robot morphologies, our system is able to learn policies that generalize to new robots not seen during training.

Original: We demonstrate the effectiveness of our model on manynotable quadrupedal robots in simulation and in real-world experiments.
Revised: We demonstrate the effectiveness of our model on many notable quadrupedal robots in simulation and in real-world experiments.

Original: We show that the learnedcontrollers can be deployed directly on a number of commonly used quadrupedal robots, includingthe Unitree’s A1 and MIT’s Mini Cheetah [8].
Revised: We show that the learned controllers can be deployed directly on a number of commonly used quadrupedal robots, including the Unitree’s A1, MIT’s Mini Cheetah [14] and CUHK’s Sirius.

Original: We also demonstrate that, by introducing morphologyrandomization into the training process, our framework is able to more effectively transfer policiesfrom simulation to the real-world than policies trained on a speciﬁc robot.
Revised: We also demonstrate that, by introducing morphology randomization into the training process (varying robot parameters such as body size, leg length, and mass during training), our framework is able to more effectively transfer policies from simulation to the real world than policies trained on a speciﬁc robot.

Original: In this paper, we focus onthe randomized robot morphology while keeping  number of Degree of Freedoms (DoFs) and linksconstant.
Revised: In this paper, we focus on the randomized robot morphology while keeping the number of Degree of Freedoms (DoFs) and links constant.

Original: Code and obtained policies in our work will be released upon publication.
Revised: The open-source GenLoco policies could be used as baselines to test controllers for newly developed quadrupedal robots without the need to train a robot-speciﬁc policy from scratch.

Original:  Related Work
Revised:  Related Work

Original: RL provides a general framework for learning robotic controllers for a large array of tasks.
Revised: RL provides a general framework for learning robotic controllers for a large array of tasks.

Original: Insteadof requiring tedious manual controller engineering, RL techniques can automatically synthesizecontrollers for a desired task by optimizing the controller against an objective function [9].
Revised: Instead of requiring tedious manual controller engineering, RL techniques can automatically synthesize controllers for a desired task by optimizing the controller against an objective function [15].

Original: RL hasbeen applied to develop a wide range of motor skills for agents in simulation [10, 11, 12, 13], and inthe real world [14, 15, 16, 17, 18, 19].
Revised: RL has been applied to develop a wide range of motor skills for agents in simulation [16, 17, 18, 19], and in the real world [20, 21, 22, 23, 24, 5, 12].

Original: But due to challenges associated with applying RL algorithmson real-world systems [20], sim-to-real techniques are commonly used to adapt and transfer controllerstrained in simulation to a physical system [21, 22, 4, 3, 19].
Revised: But due to challenges associated with applying RL algorithms on real-world systems [25], sim-to-real techniques are commonly used to adapt and transfer controllers trained in simulation to a physical system [26, 27, 8, 7, 5, 28, 29, 13].

Original: Domain randomization is one of the mostcommonly used sim-to-real transfer techniques, where the dynamics  of a simulator are randomizedduring training in order to develop controllers that can adapt to differences between simulated andreal-world dynamics [23, 24, 21, 4, 25, 26].
Revised: Domain randomization is one of the most commonly used sim-to-real transfer techniques, where the dynamics [Figure] of a simulator are randomized during training in order to develop controllers that can adapt to differences between simulated and real-world dynamics [30, 31, 26, 8, 32, 33, 29, 34].

Original: Real-world data can also be leveraged to further improvereal-world performance by adapting the dynamics of the simulator [27, 28, 19, 29, 30], or directlyﬁnetuning a controller’s behaviors [31, 3, 5].
Revised: Real-world data can also be leveraged to further improve real-world performance by adapting the dynamics of the simulator [35, 36, 5, 37, 38], or directly ﬁnetuning a controller’s behaviors [39, 7, 9, 13].

Original: In this work, we also take a domain randomizationapproach to transfer controllers trained in simulation to robots operating in the real world.
Revised: In this work, we also take a domain randomization approach to transfer controllers trained in simulation to robots operating in the real world.

Original: However,unlike many prior sim-to-real methods, which assume a known morphology for the real-world robot,our work explores the development of more general locomotion controllers deployable on a varietyof different robots, including morphologies not known during training.
Revised: However, unlike many prior sim-to-real methods, which assume a known morphology for the real-world robot, our work explores the development of more general locomotion controllers deployable on a variety of different robots, including morphologies not known during training.

Original: Developing general-purpose controllers that can be deployed on robots with different embodimentscan greatly mitigate the overhead of creating individual robot-speciﬁc controllers.
Revised: Developing general-purpose controllers that can be deployed on robots with different embodiments can greatly mitigate the overhead of creating individual robot-speciﬁc controllers.

Original: Devin et al.[32] utilized a modular network architecture to transfer manipulation skills across a small set ofmanually-crafted simulated robotic arms.
Revised: Devin et al. [40] utilized a modular network architecture to transfer manipulation skills across a small set of manually-crafted simulated robotic arms.

Original: Chen et al.
Revised: Chen et al.

Original: [33] proposed conditioning policies on learnedencodings of a robot’s morphology, which allowed a controller to be deployed on a number ofdifferent robotic arms.
Revised: [41] proposed conditioning policies on learned encodings of a robot’s morphology, which allowed a controller to be deployed on a number of different robotic arms.

Original: A similar morphology encoding approach has also been applied to trainfull-body motion controllers, which can be deployed on a large variety of simulated humanoid agents[34].
Revised: A similar morphology encoding approach has also been applied to train full-body motion controllers, which can be deployed on a large variety of simulated humanoid agents [42].

Original: Graph convolutional networks have been used to implicitly encode the morphological structureof simulated robots with varying numbers of degrees-of-freedom [35, 36].
Revised: Graph convolutional networks have been used to implicitly encode the morphological structure of simulated robots with varying numbers of degrees-of-freedom [43, 44].

Original: Gupta et al.
Revised: Gupta et al.

Original: [37] traineddomain invariant features spaces in order to transfer manipulation skills across simulated robotswith different morphologies.
Revised: [45] learned domain invariant feature spaces in order to transfer manipulation skills across simulated robots with different morphologies.

Original: While these transfer learning techniques have shown promising results[Figure] in simulation, they have yet to be demonstrated on robots in the real world.
Revised: While these transfer learning techniques have shown promising results in simulation, they have yet to be demonstrated on robots in the real world.

Original: In this work, we aimto develop general locomotion controllers that can be deployed on a large variety of robots withdifferent morphologies.
Revised: In this work, we aim to develop general locomotion controllers that can be deployed on a large variety of robots with different morphologies.

Original: Although prior works suggest that recurrent or memory-based policies canimprove transfer by performing “implicit system identiﬁcation,” we found that our approach was ableto produce effective policies with fully feedforward neural networks.
Revised: Although prior works suggest that recurrent or memory-based policies can improve transfer by performing “implicit system identiﬁcation” [26, 34], we found that our approach was able to produce effective policies with fully feedforward neural networks.

Original: By taking in past observationsand actions as input, the model is able to infer the morphological structure of a robot, and successfullytransfer locomotion skills across a diverse set of quadrupedal morphologies without requiring anexplicit representationor encoding of a particular robot’s body structure.
Revised: By taking in past observations and actions as input, the model is able to to implicitly encode task-relevant information about the robot’s morphology, and successfully transfer locomotion skills across a diverse set of quadrupedal morphologies without requiring an explicit representation of a particular robot’s body structure.

Original:  Generalized Quadrupedal Locomotion Controllers
Revised:  Generalized Quadrupedal Locomotion Controllers

Original: In this section, we present a framework for training  Generalized Locomotion Controller (GenLoco),where a single controller can be deployed on a large variety of quadrupedal robots with differentmorphologies.
Revised: In this section, we present a framework for training a Generalized Locomotion Controller (GenLoco), where a single controller can be deployed on a large variety of quadrupedal robots with different morphologies.

Original: An overview of our system is shown in Fig.
Revised: An overview of our system is shown in Fig.

Original: The controllers are trained throughmodel-free reinforcement learning.
Revised: The controllers are trained through model-free reinforcement learning.

Original: But unlike most prior RL frameworks for robotic locomotion,which assume a ﬁxed morphology for the robot during both training and testing, our work explores thesetting where the embodiment of the robot can vary, and the particular embodiment at test time is notknown a priori.
Revised: But unlike most prior RL frameworks for robotic locomotion, which assume a ﬁxed morphology for the robot during both training and testing, our work explores the setting where the embodiment of the robot can vary, and the particular embodiment at test time is not known a priori.

Original: Our Models are trained by randomizing the morphology and other dynamics propertiesduring training in simulation, thereby encouraging the controller to learn adaptable strategies, whichcan be effectively applied on different robots.
Revised: Our models are trained by randomizing the morphology and other dynamics properties during training in simulation, thereby encouraging the controller to learn adaptable strategies, which can be effectively applied on different robots.

Original: At every timestep, the controller receives a history ofsensory observations as input, which can be used to infer the morphology of the robot it is currentlydeployed on. 
Revised: At every timestep, the controller receives a history of sensory observations as input, which can be used to infer task-relevant information about the robot it is currently deployed on. The controller then outputs actions, which are added to time-invariant [Figure]

Original: The controller then outputs actions, which specify target motor positions for each joint.
Revised: [Table] nominal joint positions to specify target motor positions for each joint.

Original: The  motor targets areﬁrst passed through a low-pass ﬁlter[3], to mitigate undesirable high-frequencymovements, before being applied to joint-level PD controllers to generate motor torques.
Revised: The desired motor positions are passed through a low-pass ﬁlter to mitigate undesirable high-frequency movements [7], before being applied to joint-level PD controllers to generate motor torques [Table]

Original: 3.1 Morphology Generation
Revised: 3.1 Morphology Generation

Original: The key to developing generalized locomotion controllers lies in training the controller on a diversecollection of robots, which encourages the controller to learn strategies that can generalize to newrobots not observed during training.
Revised: The key to developing generalized locomotion controllers lies in training the controller on a diverse collection of robots, which encourages the controller to learn strategies that can generalize to new robots not observed during training.

Original: At the start of each training episode, we procedurally generate arandom robot morphology based on a predeﬁned morphology template.
Revised: At the start of each training episode, we procedurally generate a random robot morphology based on a predeﬁned morphology template.

Original: As illustrated in Fig.
Revised: As illustrated in Fig.

Original: 3, manycommonly used quadrupedal robots follow a similar body structure, consisting of a robot base andfour legs, each of which has 3 DoFs: hip, thigh, and knee joints.
Revised: 3, many commonly used quadrupedal robots follow a similar body structure, consisting of a robot base and four legs, each of which has 3 DoFs: hip, thigh, and knee joints.

Original: The robot’s feet can be modeled asspheres, and the contacts between the feet and the ground can be approximated as point contacts.
Revised: The robot’s feet can be modeled as spheres, and the contacts between the feet and the ground can be approximated as point contacts.

Original: Thisdesign template has been followed in widely used quadrupedal robots, including Unitree’s A1, Go1,
Revised: This design template has been followed in widely used quadrupedal robots, including Unitree’s A1, Go1,

Original: Laikago and Aliengo, Boston Dynamics’ Spot, MIT’s Mini Cheetah, and CUHK’s Sirius.
Revised: Laikago and Aliengo, Boston Dynamics’ Spot, MIT’s Mini Cheetah, and CUHK’s Sirius.

Original: Therefore,our system generates random morphological variations based on this template by randomizing theproportions of the various body parts and their respective dynamics properties.
Revised: Therefore, our system generates random morphological variations based on this template by randomizing the proportions of the various body parts and their respective dynamics properties.

Original: A sample of robotsproduced by our morphology generation procedure are shown in Fig.
Revised: A sample of robots produced by our morphology generation procedure are shown in Fig.

Original: The randomized morphologyparameters and their respective ranges are recorded in Table 1.
Revised: The randomized morphology parameters and their respective ranges are recorded in Table 1.

Original: Below, we provide a more detailedaccount of the major parameters of variations in our morphology generation procedure.[Figure] [Table]
Revised: Below, we provide a more detailed account of the major parameters of variations in our morphology generation procedure.

Original: Size factor.
Revised: Size factor.

Original: We introduce a size factor α , which uniformly scales the size of each body part (e [Table] g.robot base and legs).
Revised: We introduce a size factor ↵ , which uniformly scales the size of each body part (e.g. robot base and legs).

Original: Because α multiplicatively scales along every dimension, it is a sensitiveparameter, and we found randomly sampling values of α from the range [0 . 8 , 1 . 2] to be sufﬁcient incapturing the variation among most robots.
Revised: Because ↵ multiplicatively scales along every dimension, it is a sensitive parameter, and we found randomly sampling values of ↵ from the range [0 . 8 , 1 . 2] to be sufﬁcient in capturing the variation among most robots.

Original: This factor facilitates positive correlation between robotcomponent sizes, reducing the likelihood of generating morphologies with distorted proportions.
Revised: This factor facilitates positive correlation between robot component sizes, reducing the likelihood of generating morphologies with distorted proportions (e.g., a robot with an extremely large base but extremely small legs).

Original: Robot base parameters.
Revised: Robot base parameters.

Original: The parameters of the robot base include its size and density.
Revised: The parameters of the robot base include its size and density.

Original: We modelthe base geometrically as a rectangular box.
Revised: We model the base geometrically as a rectangular box.

Original: The dimensions, such as length b l , width b w , and heightb h , of the base are randomized to cover a wide range of robot base sizes.
Revised: The dimensions, such as length b l , width b w , and height b h , of the base are randomized to cover a wide range of robot base sizes.

Original: The density of the base isalso randomized between [400 , 1200] kg / m 3 .
Revised: The density of the base is also randomized between [400 , 1200] kg / m 3 .

Original: The ranges of values, detailed in Table 1, are selectedto encapsulate the sizes and masses of popular industrial robots.
Revised: The ranges of values, detailed in Table 1, are selected to encapsulate the sizes and masses of popular industrial robots as listed in Appendix E.

Original: Leg parameters.
Revised: Leg parameters.

Original: The robot’s upper leg (thigh) and the lower leg (calf) are modeled as cylindricalsolids, and the robot’s feet are represented by spheres, as shown in Fig.
Revised: The robot’s upper leg (thigh) and the lower leg (calf) are modeled as cylindrical solids, and the robot’s feet are represented by spheres, as shown in Fig.

Original: Speciﬁcally, the sizes ofthe robot’s thigh and foot are chosen based on the robot’s calf link dimensions, as detailed in Table 1.
Revised: Speciﬁcally, the sizes of the robot’s thigh and foot are chosen based on the robot’s calf link dimensions, as detailed in Table 1.

Original: The range of values for the thigh length l t is speciﬁed to be ± 25% of the calf length l c .
Revised: The range of values for the thigh length l t is speciﬁed to be ± 25% of the calf length l c .

Original: Similarly,the range of values for the thigh radius is deﬁned as ± 25% of the calf radius r c .
Revised: Similarly, the range of values for the thigh radius is deﬁned as ± 25% of the calf radius r c .

Original: By correlating theproperties of different segments of the legs, we can prevent the morphology generator from producingimplausible designs, such as a robot with large calves but very small thighs and feet, which wouldhinder the robot’s locomotion capabilities.
Revised: By correlating the properties of different segments of the legs, we can prevent the morphology generator from producing implausible designs, such as a robot with large calves but very small thighs and feet, which would hinder the robot’s locomotion capabilities.

Original: PD gains.
Revised: PD gains.

Original: Larger and heavier robots typically require stiffer joint-level PD gains in order togenerate larger motor torques.
Revised: Larger and heavier robots typically require stiffer joint-level PD gains in order to generate larger motor torques.

Original: Therefore, the gains used in the PD controllers are also scaled withrespect to the mass of the robot.
Revised: Therefore, the gains used in the PD controllers are also scaled with respect to the mass of the robot.

Original: The nominal mass is set to the mass of the A1 ( 12 . 458 kg).
Revised: The nominal mass is set to the mass of the A1 ( 12 . 458 kg).

Original: Thisrandomization scheme helps to ensure that the PD controllers are sufﬁciently strong for larger andheavier morphologies.
Revised: This randomization scheme helps to ensure that the PD controllers are sufﬁciently strong for larger and heavier morphologies.

Original:  Training
Revised:  Training

Original: Our controllers are trained to perform various locomotion skills using a reinforcement learning-basedmotion imitation framework [3].
Revised: Our controllers are trained to perform various locomotion skills using a reinforcement learning-based motion imitation framework based on Peng et al.

Original: The goal of these controllers is to imitate a given reference motionq r = { q r 0 , q r 1 , . . .
Revised: The goal of these controllers is to imitate a given reference motion q r = { q r 0 , q r 1 , . . .

Original: , q rT } , which speciﬁes target poses q rt at each timestep t , The reward r t at timestept is computed according to:
Revised: , q rT } , which speciﬁes target poses q rt at each timestep t , The reward r t at timestep t is computed according to:

Original: [Equation]
Revised: [Equation]

Original:  with w p = 0 . 6 , w v = 0 . 1 , w bp = 0 . 15 , w bv = 0 . 15 .
Revised: , with w p = 0 . 6 , w v = 0 . 1 , [Equation] [Equation]

Original: The pose reward r p t = exp (cid:104) − 5 (cid:80) 12 j =1 (cid:107) ˆ q jt − q jt (cid:107) 2 (cid:105) encourages the robot to match its local jointrotations q t with those speciﬁed by the reference motion, where q jt denotes the rotation of thej -th joint.
Revised: The pose reward r p t = exp [Equation] [Equation] i encourages the robot to match its local joint rotations q t with those speciﬁed by the reference motion, where q jt denotes the rotation of the j -th joint.

Original: The velocity reward r v t follows a similar form, and encourages the robot to match thejoint velocities of the reference motion.
Revised: The velocity reward r v t follows a similar form, and encourages the robot to match the joint velocities of the reference motion.

Original: The base position reward r bp t and base velocity reward r bv tencourage the robot to track the motion of the base from the reference motion.
Revised: The base position reward r bp t and base velocity reward r bv t encourage the robot to track the motion of the base from the reference motion.

Original: A more detaileddescription of the reward function is available in Appendix B.
Revised: A more detailed description of the reward function is available in Appendix B, and additional implementation details are described in Appendix C.

Original: State and action spaces.
Revised: State and action spaces.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, actions from the controller a t specify target jointpositions q d ∈ R 12 , which are used by the joint-level PD controllers to determine desired motortorques.
Revised: 2, actions from the controller a t specify target joint positions q d 2 R 12 , which are used by the joint-level PD controllers to determine desired motor torques.

Original: The controller operates at 30 Hz.
Revised: The controller operates at 30 Hz.

Original: To encourage smoother motions, actions are processedwith a low-pass ﬁlter before being applied to the robot.
Revised: To encourage smoother motions, actions are processed with a low-pass ﬁlter before being applied to the robot.

Original: The observation s t of the controller includesthree components: 1) a 15-timestep history of robot raw sensor feedback q base t − 15: t and q t − 15: t− 1 , 2)a 15-timestep history of past actions a t − 15: t − 1 , and 3) a phase variable φ ∈ [0 , 1]  that indicates therobot’s progress along the reference motion, where 0 denotes the start of the motion and 1 denotesthe end [10].
Revised: The observation s t of the controller includes three components: 1) a 15-timestep history of robot raw sensor feedback q base t  15: t and q t  15: t , 2) a 15-timestep history of past actions a t  16: t  1 , and 3) a phase variable  2 [0 , 1] is the normalized time that indicates the robot’s progress along the reference motion, where 0 denotes the start of the motion and 1 denotes the end [16].

Original: The robot’s sensor feedback at each timestep consists of the base orientation q base ∈ R 4from IMU sensors, recorded as a quaternion, and the measured joint positions q ∈ R 12 .
Revised: The robot’s sensor feedback at each timestep consists of the base orientation q base 2 R 4 from IMU sensors, recorded as a quotation, and the measured joint positions q 2 R 12 .

Original: Note thatthe observations do not include quantities that require an explicitly state estimator, such as the linearvelocity of the base [2, 3, 26, 7] or contact sensors [4].
Revised: Note that the observations do not include quantities that require an explicit state estimator, such as the linear velocity of the base [6, 7, 33, 11] or contact sensors [8].

Original: The history of sensor readings and actionsprovides the GenLoco policy some information that can be used to perform state estimation and toinfer the robot’s dynamics and morphology.
Revised: The history of sensor readings and actions provides the GenLoco policy some information that can be used to perform state estimation and to infer the robot’s dynamics and morphology.

Original: Dynamics randomization.
Revised: Dynamics randomization.

Original: In addition to randomizing the morphological structure of the robots,we also randomize the dynamics parameters of the simulation during training [38] to improve therobustness of our policy and facilitate sim-to-real transfer.
Revised: In addition to randomizing the morphological structure of the robots, we also randomize the dynamics parameters of the simulation during training [31] to improve the robustness of our policy and facilitate sim-to-real transfer.

Original: The randomized dynamics parametersinclude three categories: 1) link mass, link inertia, and ground friction to deal with modeling errors,2) motor strength, joint-level PD gains and motor friction to mitigate the uncertainties of motordynamics, and 3) the latency between the policy and the joint-level controllers.
Revised: The randomized dynamics parameters include three categories: 1) link mass, link inertia, and ground friction to deal with modeling errors, 2) motor strength (torque limits), joint-level PD gains and motor damping ratio to mitigate the uncertainties of motor dynamics, and 3) the latency between the policy and the joint-level controllers.

Original: The randomizationranges of each parameter are detailed in Table 3 in Appendix C.
Revised: The randomization ranges of each parameter are detailed in Table 3 in Appendix D.

Original: Episode design and curriculum learning.
Revised: Episode design.

Original: Each episode hasa horizon of 100 timesteps, lastingabout 3 seconds.
Revised: Each episode has 100 timesteps, lasting about 3 seconds.

Original: Early termination is applied when the robot deviates too far from the reference baseposition and orientation [3].
Revised: Early termination is applied if the robot deviates too far from the reference base position and orientation [7]. At the start SpotMicro Pacing Aliengo Pacing

Revised: ANYmal

Revised: [Figure]

Original: At the start of each episode, a new morphology is generated accordingto the procedure described in Sec. 3.1, and the dynamics parameters are also randomly selectedaccording to Sec. 4.
Revised: A1 Pacing Laikago Pacing ANYmal-C Pacing Mini Cheetah Pacing

Revised: Sirius Pacing

Revised: Spot Pacing

Revised: SpotMicro Spinning

Revised: Aliengo Spinning

Revised: ANYmal

Original: The introduction of the morphology randomization signiﬁcantly increases thedifﬁculty of the task, and the agent can fail to learn desired skills.
Revised: [Figure] A1 Spinning Laikago Spinning ANYmal

Original: Therefore, we utilize a trainingcurriculum to assist in training with our morphology generation procedure.
Revised: [Figure] C Spinning Mini Cheetah Spinning

Original: During training, welinearly increase the width of the random interval detailed in Table 1 with respect to the trainingiteration. Speciﬁcally, in the ﬁrst training iteration, we use a ﬁxed morphology (i.e., sample parametersfrom a 0-width interval centered on the average of the min and max values in Table 1).
Revised: Sirius Spinning

Original: In eachsubsequent training iteration, the widths of  the intervals from which we sampled parameter valuesare linearly increased until the interval bounds reach the minimum and maximum values.
Revised: Spot Spinning of each episode, a new morphology is generated according to the procedure described in Sec. 3.1, and the dynamics parameters are also randomized according to Sec. 4.

Original: GenLoco policies π θ are trained using Proximal Policy Optimization (PPO) [39], and all simulationsare performed using Pybullet simulation [40].
Revised: GenLoco policies ⇡ ✓ are trained using Proximal Policy Optimization (PPO) [46], and all simulations are performed in Pybullet [47].

Original: Each policy is trained withabout 800 million samples,requiring approximately 2 weeks on 16 CPU workers.
Revised: Each policy was trained with 800 million samples, taking approximately 2 weeks on 16 CPU workers.

Original: The morphology randomization curriculumlinearly anneals the range of parameter values from 0 to their respective maximum range over thecourse of 200 million samples.

Original: Learning curves are presented in Appendix D.

Original: [Figure]

Original:  Simulation Validation
Revised:  Simulation Validation

Original: In this section, we validate the effectiveness of our models on controlling quadrupedal robots withdifferent morphologies by direct transfer in simulation.
Revised: In this section, we validate the effectiveness of our models on controlling quadrupedal robots with different morphologies by direct transfer in simulation.

Original: We compare our GenLoco policies with thepolicies trained on speciﬁc robots, which use the same training settings introduced in Sec. 4.
Revised: We compare our GenLoco policies with the policies trained on speciﬁc robots, which use the same training settings introduced in Sec. 4.

Original: Wetrain controllers for imitating two reference motions: a forward moving pacing gait and an in-placespinning motion.
Revised: We train controllers for imitating two reference motions: a forward moving pacing gait and an in-place spinning motion.

Original: The reference motions are retargeted to match our robot’s morphology from motioncaptured data of a real dog [3].
Revised: The reference motions (using motion capture data from a real dog [7]) are retargeted to the A1’s morphological features, and rescaled for each new morphology to account for differences in sizes and proportions. Thus, new reference motions are not required during test-time.

Original: Our controllers are trained using procedurally generated robots, and then tested on simulated modelsof commercially available robots.
Revised: After being trained on procedurally generated morphologies, our controllers were tested on simulated models of commercially available robots using nominal standing poses and PD gains found in [48].

Original: The robots used for testing include the A1, Aliengo, Go1, Mini 
Revised: Robots used for testing include the A1, Aliengo, Go1, Mini Cheetah, Sirius, Laikago, Spot, SpotMicro,

Original: Cheetah, Sirius, Laikago, and ANYmal-B. The A1 and Go1 are small robots with a mass of about 12kg and a  height ofabout 0 . 4 m. The Sirius, Aliengo and Laikago are heavier and larger with eachstanding at 0 .
Revised: ANYmal-B, and ANYmal-C. The SpotMicro [49] is relatively tiny, with a mass of 4 . 8 kg and a fully-standing height of 0 . 26 m. The A1 and Mini Cheetah are small robots weighing about 10 kg and standing about 0 .

Original: 6 m in height.
Revised: 4 m in height.

Revised: The Sirius, Aliengo, Laikago, and Spot are heavier and larger, each having height  0 .

Revised: 5 m. The morphological parameters of these robots generally lie within the training distributions.

Original: Furthermore, ANYmal-B has a different knee joint design from the designtemplate used for training.
Revised: However, the ANYmal-B and ANYmal-C have a different knee joint design from the design template used for training.

Original: Detailed speciﬁcations of these robots are available in Appendix E.
Revised: Detailed speciﬁcations of these robots are listed in Appendix E.

Original: 5.1 Zero-Shot Transfer to Novel Robots
Revised: 5.1 Zero-Shot Transfer to Novel Robots

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 4 and the supplementary video, our controllers can be directly deployed on a largevariety of quadrupedal robots.
Revised: 4 and the supplementary video, our controllers can be directly deployed on a large variety of quadrupedal robots.

Original: None of the robots we tested on were used during training.
Revised: None of the robots we tested on were used during training.

Original: Notably,
Revised: Notably,

Original: GenLoco policies successfully transfer to the ANYmal-B robot, which possesses an inverted kneejoint design different from the template used during training (Fig. 3), without additional tuning.
Revised: GenLoco policies successfully transfer to the ANYmal-series robots, which possess an inverted knee joint design different from the template used during training (Fig. 3), without additional tuning.

Original: Thishighlights the generalizability of our proposed GenLoco framework.
Revised: This highlights the generalizability of our proposed GenLoco framework.

Original: Our GenLoco policies are some of the ﬁrst locomotion controllers that can be deployed directly ondifferent quadrupedal robots without further ﬁne-tuning.
Revised: Our GenLoco policies are some of the ﬁrst locomotion controllers that can be deployed directly on different quadrupedal robots without further ﬁne-tuning.

Original: By providing a history of past observationsand actions to the controller, we enable it to leverage the history to infer the morphology of the robotit is currently deployed on, and by randomizing the morphology during training the model developsmore general control strategies that can effectively execute a desired skill on different morphologies.
Revised: By providing a history of past observations and actions to the controller, we enable it to leverage the history to infer the morphology of the robot [Figure] it is currently deployed on, and by randomizing the morphology during training the model develops more general control strategies that can effectively execute a desired skill on different morphologies.

Original: 5.2 Out-of-Distribution Generalization
Revised: 5.2 Out-of-Distribution Generalization

Original: To further evaluate the ability of GenLoco to generalize learned locomotion skills to differentmorphologies, we test the controllers extensively on a large range of simulated robot morphologies.
Revised: To further evaluate the ability of GenLoco to generalize learned locomotion skills to different morphologies we test the controllers extensively on a large range of simulated robot morphologies.

Original: We evaluate the controller’s performance when varying four morphology parameters: 1) body size,[Figure] [Figure] which is scaled by size factor α , 2) calf length l c , 3) thigh length l t, and 4) body mass b m .
Revised: We evaluate the controller’s performance when varying four morphology parameters: 1) body size, which is scaled by size factor ↵ , 2) calf length l c , 3) thigh length l t and 4) body mass b m .

Original: Thetesting range is set to be much larger than the range used during training detailed in Table 1.
Revised: The testing range is set to be much larger than the range used during training detailed in Table 1.

Original: Wecompare GenLoco to policies trained speciﬁcally for the A1, since the nominal values used in themorphology generation process (Table 1) are based on the original A1 robot.
Revised: We compare GenLoco to policies trained speciﬁcally for the A1, since the nominal values used in the morphology generation process (Table 1) are based on the original A1 robot.

Original: The comparisons withthe pacing gait controllers and spinning gait controllers are shown in Fig.
Revised: The comparisons with the pacing gait controllers and spinning gait controllers are shown in Fig.

Original: 5a and Fig.
Revised: 5a and Fig.

Original: 5b, respectively.
Revised: 5b, respectively.

Original: The GenLoco policies are overall able to better generalize to out-of-distribution robots, exhibitingmore robust behaviors and maintaining higher returns across larger variations in the morphologicalparameters than the A1-speciﬁc policies.
Revised: The GenLoco policies are overall able to better generalize to out-of-distribution robots, exhibiting more robust behaviors and maintaining higher returns across larger variations in the morphological parameters than the A1-speciﬁc policies.

Original: In the case of the pacing skill, GenLoco’s performance doesnot degrade until the robot size is more than two times larger than the maximum training size.
Revised: In the case of the pacing skill, GenLoco’s performance does not degrade until the robot size is more than two times larger than the maximum training size.

Original: Whilethe A1-speciﬁc policy shows some robustness to small variations in the morphology parameters, thepolicy fails when deployed on other robots, such as the Sirius.
Revised: While the A1-speciﬁc policy shows some robustness to small variations in the morphology parameters, the policy fails when deployed on other robots, such as the Sirius.

Original: In order to understand the advantages of the morphology randomization during training, we test therobustness of the GenLoco policies and A1-speciﬁc policies by varying the dynamics parameters.
Revised: In order to understand the advantages of the morphology randomization during training, we test the robustness of the GenLoco policies and A1-speciﬁc policies by varying the dynamics parameters.

Original: Inaddition to generalization to different morphologies, our model is also robust to large variations in thedynamics of a system.
Revised: In addition to generalization to different morphologies, our model is also robust to large variations in the dynamics of a system.

Original: Fig.
Revised: Fig.

Original: 5c compares the performance of GenLoco policies to A1-speciﬁc policiesthat were also trained using the same range of dynamics randomization detailed in Table 3.
Revised: 5c compares the performance of GenLoco policies to A1-speciﬁc policies that were also trained using the same range of dynamics randomization detailed in Table 3.

Original: Asdemonstrated in Fig.
Revised: As demonstrated in Fig.

Original: 5c, 5d, for the same range of dynamics randomization during training, GenLocopolicies consistently outperform the A1-speciﬁc policies.
Revised: 5c, 5d, for the same range of dynamics randomization during training, GenLoco policies consistently outperform the A1-speciﬁc policies.

Original: In particular, GenLoco policies demonstratenotable robustness to changes in the dynamics parameters related to actuated joints, such as motorfriction and strength.
Revised: In particular, GenLoco policies demonstrate notable robustness to changes in the dynamics parameters related to actuated joints, such as motor friction and strength.

Original: This improved performance on out-of-distribution settings is likely in part dueto random morphologies introducing a more diverse range of dynamics for the policy to train on.
Revised: This improved performance on out-of-distribution settings is likely in part due to random morphologies introducing a more diverse range of dynamics for the policy to train on.

Original:  Real-World Deployment
Revised:  Real-World Deployment

Original: Finally, we evaluate the effectiveness of the learned controllers on robots in the real world.
Revised: Finally, we evaluate the effectiveness of the learned controllers on robots in the real world.

Original: We testthe controllers on two robots, Unitree’s A1 and MIT’s Mini Cheetah.
Revised: We test the controllers on three robots, Unitree’s A1, MIT’s Mini Cheetah and CUHK’s Sirius.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 6 and the 
Revised: 6 and the supplementary video, our GenLoco policies can be deployed on the real robots in a [Figure]

Original: [Table]
Revised: [Table]

Original: supplementary video, our GenLoco policies can be deployed on the real robots in a zero-shot manner,enabling the robots to perform agile maneuvers such as pacing and spinning.
Revised: zero-shot manner, enabling the robots to perform agile maneuvers such as pacing and spinning.

Original: We further compare thereal-world performance of GenLoco to policies trained speciﬁcally for the A1 and Mini Cheetah.
Revised: We further compare the real-world performance of GenLoco to policies trained speciﬁcally for the A and Mini Cheetah.

Original: Theperformance of these policies are available in Table 2.
Revised: We performed 6 trials for each policy using the same experimental setup, and the performance of these policies are available in Table 2.

Original: We note that the performance of GenLocopolicies, which work on both robots and were trained on neither, is overall better than the policiestrained for speciﬁc robots, as shown in the supplementary video.
Revised: We note that the performance of GenLoco policies, which work on both robots and were trained on neither, is overall better than the policies trained for speciﬁc robots, as shown in the supplementary video.

Original: Furthermore,there is a failure whenwe deploy the Mini Cheetah-speciﬁc policy for pacing on the real robot, even though the same policyworks well in simulation as shown in Fig.4.
Revised: Furthermore, the Mini Cheetahspeciﬁc pacing policy failed in all 6 real robot deployment trials, despite working well in simulation as shown in Fig.

Original: Moreover, when testing the A1’s robot-speciﬁc spinpolicy on the hardware, the robot drifts in the transverse plane during its motion.

Original: Notably, thesetwo performance degradations during sim-to-real transfer do not occur for the GenLoco policiesunder identical environment circumstances, as shown in the supplementary video.
Revised: Notably, such performance degradations during sim-to-real transfer do not occur for the GenLoco policies under identical environment circumstances.

Original: This highlights theadvantages of GenLoco in terms of robustness and generalization during sim-to-real transfer. 
Revised: This suggests that solely randomizing the dynamics for a single robot may be insufﬁcient due to differences in morphological characteristics between simulation and the real world. By training with randomized morphologies, the GenLoco policy better adapts to kinematics variations and therefore maintains robustness during sim-to-real transfer. An additional note of interest is that the timeline from completion of the Sirius hardware to our deployment of GenLoco control on Sirius was mere days. The successful zero-shot transfer observed in this experiment further highlights the utility of our model.

Original:  Limitations
Revised:  Limitations

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5a, 5b, the performance of GenLoco policies drops dramatically for larger robots,such as Boston Dynamics’ Spot, which has a mass of 31 kg and is 0 .
Revised: 5a and 5b, the performance of GenLoco policies drops dramatically for larger robots.

Original: 7 m tall. This may suggestthat more aggressive randomization anda more expressive model architecture with recurrence areneeded during training, like [41].
Revised: This may suggest that using more aggressive randomization and more expressive model architectures with recurrence, such as in [50], could improve the model.

Original: Moreover, our model does not generalize well to robots that havemorphologies drastically different from the template used in our morphology generation procedure(Fig. 3), and it fails to transfer to the ANYmal-C robot [42].

Original: Lastly, our models cannot be deployedon robots with a different number of DoFs.
Revised: Moreover, our models cannot be deployed on robots with a different number of DoFs.

Original: More ﬂexible architectures, such as graph neural networks[35, 36], may allow formore adaptable models that can handle variable numbers of DoFs.
Revised: More ﬂexible architectures, such as graph neural networks [43, 44], may allow for adaptable models that can handle variable numbers of DoFs.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we present an RL-based framework for training generalized locomotion controllersthat can be deployed on a diverse set of quadrupedal robots.
Revised: In this paper, we presented an RL-based framework for training generalized locomotion controllers that can be deployed on a diverse set of quadrupedal robots.

Original: We show that a simple history-basedmodel, trained on procedurally generated robots in simulation, can be successfully transferred to alarge variety of new robots, which were not observed during training.
Revised: We show that a simple history-based model, trained on procedurally generated robots in simulation, can be successfully transferred to a large variety of new robots, which were not observed during training.

Original: The trained models can also bedeployed directly on real robots, without requiring any additional training on the physical systems.
Revised: The trained models can also be deployed directly on real robots, without requiring any additional training on the physical systems.

Original: While our experiments have been focused on quadrupedal robots, our method is general and can alsobe applied to robots in other domains, such as robotic arms and quadrotors.
Revised: While our experiments have been focused on quadrupedal robots, our method is general and can also be applied to robots in other domains, such as robotic arms and quadrotors.

Original: However, the effectivenessof our models remains limited to robots that have the same number of DoFs and follow a predeﬁnedmorphological template.
Revised: However, the effectiveness of our models remains limited to robots that have the same number of DoFs and follow a predeﬁned morphological template.

Original: Despite these limitations, we hope our work will provide a stepping stonetowards more general-purpose controllers that can be widely and conveniently deployed on a diversecatalog of robots.Appendix A Nominal Morphology Generation Parameter Values [Table] Appendix B Reward Function
Revised: Despite these limitations, we hope our work will provide a stepping stone towards more general-purpose controllers that can be widely and conveniently deployed on a diverse catalog of robots.

Original: Similar to Peng et al.

Original: [3], we compute our reward function as follows: [Equation] w [Equation]
Revised: Acknowledgments

Original: where the pose reward r p t , velocity reward r v t , base position reward r bp t , base velocity reward r bv tterms are deﬁned according to:

Original: [Equation]

Original: where at timestep t ,

Original: • q jt represents the local joint rotation of joint j of the robot• ˙ q jt represents the local joint angular velocity of joint j of the robot• x base t represents the global position of the robot• ˙ x base t represents the linear velocity of the robot• q base t represents the 3D base rotation of the robot

Original: • ˙ q base t represents the angular velocity of the robot

Original: • ˆ q jt represents the local joint rotation of joint j from the reference motion

Original: [Table]

Original: • ˆ˙ q jt represents the local joint angular velocity of joint j from the reference motion• ˆ x base t represents the global position from the reference motion

Original: • ˆ˙ x base t represents the linear velocity from the reference motion• ˆ q base t represents the 3D base rotation from the reference motion• ˆ˙ q base t represents the angular velocity from the reference motion

Original: The pose reward encourages the robot to match its local joint rotations q t with those speciﬁed by thereference motion.

Original: The velocity reward r v t encourages the robot to match the joint velocities of thereference motion. The base position reward r bp t encourages the robot to track the base position androtation of the reference motion. The base velocity reward r bv t encourages the robot to move at thesame speed as the reference motion. Appendix C Dynamics Randomization Ranges
Revised: This work was supported in part by Hong Kong Centre for Logistics Robotics and in part by ARL

Original: [Table]

Original: The default values used in the table refer to the values (mass, inertia, etc.) from the underlyingmorphology—note that during training, these values were themselves produced by randomizationwithin the morphology generation procedure.

Original: Appendix D Training Curriculum

Original: [Figure]

Original: We demonstrate the importance of the training curriculum introduced in Sec. 4.

Original: Fig.

Original: 7shows the performance of the controllers overthe course of training.

Original: The pacing controllertrained with a curriculum achieves higher returnthan the controller trained without a curriculum.

Original: The curriculum allows the agent to learn moreefﬁciently by gradually expanding the range ofmorphology randomization.

Original: This prevents theagent from adopting overly conservative behavior when subjected with a large amount of randomization at the beginning of training, potentially leading to a less optimal policy.

Original: Appendix E Existing RobotSpeciﬁcations

Original: Parameter

Original: A

Original: [Table] Laikago [Table] Aliengo
Revised: DCIST CRA W911NF-17-2-0181.

Original:  Mini Cheetah ANYmal-B Total weight (kg)   15 . Base length (m) 0 .0 .0 .0 .0 .
Revised: We thank Prof. Sangbae Kim, the MIT Biomimetic Robotics Lab, and NAVER LABS for lending the Mini Cheetah for experiments

Original: [Table]