Human-Robotic Prosthesis as Collaborating Agents for Symmetrical Walking
Anonymous Author(s)
AffiliationAddress
Abstract
This is the first attempt at considering human influence in the reinforcement learning control of a robotic lower limb prosthesis toward symmetrical walking in realworld situations.
We propose a collaborative multi-agent reinforcement learning(cMARL) solution framework for this highly complex and challenging humanprosthesis collaboration (HPC) problem.
The design of an automatic controller ofthe robot within the HPC context is based on accessible physical features or measurements that are known to affect walking performance.
Comparisons are madewith the current state-of-the-art robot control designs, which are single-agent based,as well as existing MARL solution approaches tailored to the problem, includingmulti-agent deep deterministic policy gradient (MADDPG) and counterfactualmulti-agent policy gradient (COMA).
Results show that, when compared to theseapproaches, treating the human and robot as coupled agents and using estimatedhuman adaption in robot control design can achieve lower stage cost, peak error,and symmetry value to ensure better human walking performance.
Additionally,our approach accelerates learning of walking tasks and increases learning successrate.
The proposed framework can potentially be further developed to examine howhuman and robotic lower limb prosthesis interact, an area that little is known about.
Advancing cMARL toward real world applications such as HPC for normativewalking sets a good example of how AI can positively impact on people’s lives.
Introduction
The concept, design, and applications of human-robot cooperation have advanced rapidly due to newdemands in AI-enabled applications fueled by powerful deep learning and reinforcement learningalgorithms [1, 2, 3, 4].
Human-robot collaboration can take on a variety of forms depending on tasksto be solved, how information is shared, and the nature of interaction [2, 3, 4].
Examples of complexcollaborative tasks include picking up or carrying objects together, cooperating on a production line,in which cases a robot can learn to imitate human demonstrations [5, 6, 7, 8, 9].
Other applicationscenarios may include intermittent robotic correction of human driving, or vice versa [10].
In essence,most of these recent studies involve interactions between a human and a robot in a way such thatthere is either space between the agents, or there is time for predictive counter measures to take place.
By contrast, in the HPC problem the human and robot agents are highly coupled, and there is oftenlittle time for the human agent to correct balance/gait disturbances.
As a result, innovative solution frameworks are needed to first address the design, analysis, and testingof complex control systems to restore locomotion for amputees.
Based on these platforms, humanrobot interactions can be studied, and fundamental issues such as user preference and prosthesisembodiment can be examined.
The knowledge in turn, will further help design seamless automaticrobot control systems.
As such, challenges arising from controlling a wearable robotic lower limb prosthesis to meet the human user’s needs are unique, and in some aspects, these challenges aregreater than what have been studied [11, 12, 13, 14].
Achieving normative walking is fundamentally a real time control problem that involves continuousstates and continuous controls of the human-robot system.
State-of-the-art automatic control ofrobotic prosthesis has been reported recently with successful real human subject testing [15, 16].
These approaches are single agent-based reinforcement learning controls, specific tasks such aslevel ground and ramp walking were accomplished via designer prescribed or specified robotic jointmovement profiles.
A most recent single-agent based reinforcement learning solution demonstratedthat human amputee subject can perform level ground and ramp walking with the robot controlleraimed at mimicking the intact joint movement.
Note, however, that all previous results have notconsidered human influence on the human-robot system performance in the automatic control ofthe robotic prosthesis, a phenomena that has direct impact not only on restoring walking but alsoon human health [17].
Multi-agent reinforcement learning control is a natural candidate to addressthis challenge.
Yet, only a handful of results are available, and they are not directly in the problemdomain under our consideration.
For example, existing work on the physical human-exoskeletoncollaboration treated the problem in a leader (human)-follower (exoskeleton) setting.
Leader-followercoordination was achieved by measured/predicted interaction force [18].
Such an approach was tohave the robot follow the learned behavior from human.
Great effort has been made develop continuous control reinforcement learning algorithms which cansubstantively address real world applications.
Several algorithms, such as deep deterministic policygradient (DDPG) [19], proximal policy optimization (PPO) [20], soft actor-critic (SAC) [21], andtwin delayed DDPG (TD3) [22], have demonstrated success with solving complex control problems.
For example, in simulated human locomotion control [23, 24], over 20 independent control signals areused to control a humanoid robot to achieve different walking tasks.
Robotic upper-limb control hasundergone intense development, especially in the realms of robot surgical assistance [25, 26], patientrehabilitation [25], and industrial applications [27].
However, these problems differ fundamentallyfrom the human-robot walking problem, as their control target usually consists of well-defined endpoints generated by a decoupled trajectory generation exosystem [28].
On the other hand, walkingtasks are difficult to associate with an end point task goal due to heavy dynamic coupling betweenthe human reference and robotic limb.
Single agent-based continuous control has also demonstratedpromise in engineering applications such as stabilization, tracking, and reconfiguring control of
Apache helicopters [29, 30, 31], stabilization and control of large power grids [32, 33, 34], roboticmanipulation and locomotion via MuJoCo and OpenAI Gym [35, 36], and wearable robots withhuman in the loop [37, 38, 39].
While these works are encouraging toward solving realistic singleagent continuous control problems, it is not obvious how they can directly address the multi-agenthuman-robot normative walking problem.
Related Work and Challenges
Single agent RL for automatic control of a robotic limb.
Simulation.
Most state-of-the-art
RL control design approaches to enable continuous human-robot walking are single-agent based.
Important milestones have been achieved by two major classes of RL algorithms: 1) actor-criticalgorithms, including direct heuristic dynamic programming (dHDP) and Q-learning [37, 38, 40, 41],and 2) variants of policy iteration algorithms, including flexible policy iteration (FPI)
[42] and policyiteration with constraint embedded (PICE) [43].
Both types of single-agent based control algorithmswere demonstrated in simulation first.
Human Tests.
Then these algorithms were tested on subjectswalking with a robotic knee prosthesis [15, 44].
Note that all of the above methodologies and testsused designer-prescribed robot joint movement profiles[45] generated a priori for the specific subjectsand walking tasks.
In real-world use scenarios, joint movement profiles must evolve dynamically inreal-time to accommodate internal human walking objectives[16, 39].
A more recent single-agent RLcontrol work [46] replaced designer-prescribed joint movement profiles with the intact joint motion.
Note also that, none of the above results have taken into account human influence on human-robotwalking in the control design, a novel contribution of this work.
Multi-agent reinforcement learning(MARL).
A cMARL problem usually cannot be transformed to an equivalent single-agent problem,as additional agent(s) and their control policies introduce uncertainties and/or non-stationarity tothe environment [47].
Partial observability is common in cMARL problems, which are usuallyaddressed by the decentralized POMDP framework.
Centralized training decentralized execution
(CTDE) is a popular paradigm to address multi-agent coordination problems.
A central featureof most CTDE approaches is factorization of the joint state-action value function into individualutility functions.
MADDPG [48] is a popular CTDE-based method to solve both cooperative andcompetitive MARL problems.
COMA [49] is another popular MARL algorithm which utilizes asingle centralized critic by using global state information and actions of all agents.
We therefore usethem in benchmark evaluations.
Shared Autonomy (SA).
The human-prosthesis problem under ourconsideration falls into the area of physical human-robot interaction (pHRI) as the human and therobotic prosthesis are in direct contact at all times.
It is not, however, the extensively studied pHRIproblem archetype (e.g., cooperative object manipulation, human operating in a remote environment),wherein interactions are usually mediated by a third object.
For similar reasons, our pHRI problem isdifferent in several important aspects from the approaches to the existing shared autonomy, such ascross-training, bounded memory adaptation, predict and blend, and model-free RL [50, 51, 52, 53].
The wearable exoskeleton control problem seems relevant to our HPC problem.
Yet, there are stillfundamental differences.
Recent exoskeleton approach also takes into account human-exoskeletoninteracting effects [54, 55].
However, these exoskeletons mainly focus on end point performanceof a foot or lower limb joints where user intent is quantified by estimating human joint torque orinteractive torque.
Thus, human-exoskeleton system performance goal is for the robot to producea well-defined joint or endpoint trajectory [56, 57, 58].
For HPC problem we consider, there is noclear end point goal as there is in most existing current pHRI problems, including robotic upper limbthat has been intensively addressed in literature.
Therefore, how to define a performance goal forour HPC problem is a challenge.
We address this study from existing limited yet proved knowledge[46, 16].
Modeling Challenges.
The HPC walking problem involves two strongly coupled agents,the interacting dynamics of which are difficult or nearly impossible to describe by ordinary differenceor differential equations in large part because there is no clear performance goal in HPC as in studiedcases of shared autonomy.
HPC may be affected by several factors, such as lower limb mechanics,inter-limb neuromechemical coupling, and physical structure of the human body including thelower limb [59, 60].
Physical and physiological differences in individual human subjects furthercomplicates modeling and control design.
Even though our knowledge of human motor control andmotor learning in cases such as post-stroke or general loss of normative locomotion capabilitieshave expanded greatly [61], little is known about how an active robotic prosthesis, not a traditionalstick-type passive prosthesis, affects human and vice versa in walking tasks.
This is because that thehuman-robotic prosthesis interacts continuously, the intricate human neurocontrol circuits includingsensing, perception and feedback control, which are necessary to facilitate normative walking, isdisrupted after amputation.
Human Utility Challenges.
Because of the reasons above, also becauseof no clear end point target as performance goals as in most studied shared autonomy cases, seamlesscollaboration between a robotic lower limb prosthesis serially attached to its human user poses newissues that requires to be answered [11, 12].
In this study, we based on latest understanding on the
HPC problem, to provide an innovative solution approach to account for human influence in HPC.
Contributions.
Human-robot collaborative tasks will continue to play an increasingly vital rolein modern life, and existing single- and multi-agent control frameworks have only covered a smallsubset of the problems.
The contributions of this work are as follows.
1) We introduce an innovativeapproach to automatically control a robotic lower limb prosthesis by treating the human user as acollaborating agent.
We thus address a new challenge in the domain of shared autonomy problems.2)
To solve the control problem, we introduce a new cMARL approach to solve our HPC problem, aproblem that cannot be readily solved to satisfaction by existing MARL approaches such as COMAor MADDPG. 3)
This is the very first attempt in the field of wearable lower limb robots that humaninfluence is explicitly considered in the robot control design.
Fundamental control problem
statement
In powered lower limb prosthesis, the finite state impedance control (FS-IC) framework mostfrequently serves to provide intrinsic control of a robotic joint, i.e., it is the built-in controller fromrobotic prosthesis manufacturers.
While position control is common in industrial robots, when ahuman is paired with a robotic joint, robot trajectory tracking using position control may precludeany dynamic interaction of the robot with its human user and the environment [62].
This may causean amputee to react to the awkwardness of the prosthesis rather than to interact constructively with it[63].
As FS-IC impedance control framework is considered to provide compliant control for human use of the robot, it is expected to generate stable and predictable human-robot walking behavior, andthus, it is adopted in this study which aims for its controller designs at real world applications.
The FS-IC treats a gait cycle, or a step, as four consecutive gait phases (Figure 1, Left): stanceflexion (STF) as Phase 1, stance extension (STE) as Phase 2, swing flexion (SWF) as Phase 3,and swing extension (SWE) as Phase 4.
In what follows, we design four individual controllers foreach of the four phases.
Since the four controllers are conceptualized similarly and designed usingthe same approach, for the sake of clarity we carry out subsequent discussion without explicitlyreferring to the specific phase numbers (but we emphasize that each of the four phases requires itsown independently-trained controller).
Additional details on FS-IC such as phase detection andtransition are provided in Appendix A.3,
At the k -th gait cycle, a robot controller (solved by cMARL in this paper) is to determine threeimpedance parameters, I k
=
[ K k , B k , ( θ e ) k ] T ∈ R 3 under the FS-IC framework, representingstiffness K k , damping coefficient B k , and equilibrium position ( θ e ) k , respectively.
The prostheticjoint motor torque T k ∈ R is then generated based on joint kinematics (knee joint angle θ and angularvelocity ω ) according to the following impedance control law,
[Equation]
The control problem formulation requires automatically determining 12 control inputs or impedancecontrol parameters (3 for each of the 4 gait phases) for individual users aiming at walk in real worldsituations.
The initial set of feasible baseline impedance control parameters I 0 can be obtained frommanufacturers or rehabilitation clinics.
Impedance updates take place according to
[Equation]
where u k ∈ R 3 is to be determined from our proposed cMARL approach.
Method
The human user and the robotic limb are considered collaborating agents.
Our cMARL solutionapproach toward human-robot symmetrical walking problem is formulated based on physical featuresand measurements that have been shown affecting human-robot walking performance, and they arerespectively available to the human and the robot, but not necessarily to each other.
State and control variables.
Refer to Figure 1.
At the k -th gait cycle, let τ ik and λ ik , respectively,represent the stance time (time of foot on the ground) and step length (length between two consecutivesteps when toe touching the ground) of the human intact leg.
Similarly, τ pk and λ pk , respectively, forthe prosthetic leg.
Let ∆ τ k , ∆ λ k denote respectively the difference of stance time and step length,
[Equation]
[Equation]
We thus define human state variable z k ∈ R 2 as z k =
[∆ τ k , ∆ λ k ] T .
Robot states include kinematic features determined from the knee movement profiles as well as steplength and stance time (or z k ).
For robot kinematic state variables, we extract gait kinematic features from knee motion profiles of both limbs (Figure 1).
Let P ik and D ik represent the knee angle andtime duration of the intact limb, and similarly, P pk and D pk for the prosthetic limb.
Let ∆ P k and ∆ D kdenote the error of peak knee angles and the error of duration time between the intact and the roboticknees, respectively,
[Equation]
We then define robot state x k ∈ R 4 to include the following four variables, the first two of which areshared between the human and the robot, x k = [∆ τ k , ∆ λ k , ∆ P k , ∆ D k ] T .
As such, human states are available to the robot, but a subset of the robot state (the kinematic features)is not available to the human.
This reasonably reflects practical situations in real life applications.
The robot control policy is a state feedback law from robot state to robot impedance control parameters,namely, control u k consists of increments to the impedance control parameters (Eq. 2),
[Equation]
Determining human control policy which originates from human neurocontrol circuit is a dauntingtask, especially now that sensing, perception and feedback control circuits are interrupted after alower limb amputation.
While agents’ policy solutions are solved from Bellman optimality-likeequations, a human cannot interpret and implement an MARL solution via their complicated neuralcircuits.
But this is not our intention.
We instead use the human control variable as an estimated inputto the robot controller design, i.e., the robot is informed by an estimated human influence when theyshare the same goal of symmetrical walking.
As the very first step to demonstrate this idea, we consider the human user of the wearable robotintentionally or voluntarily walk at a reference step length, which may also be from an instructionalfeedback.
For real life scenarios, such behavioral cues can change over time, task, or environment.
But it is important to validate such formulation of considering human influence in the robot controldesign.
Toward this end, we consider a practical reference cue denoted as a desired step lengthλ o .
We let v k represent an endogenous control signal of the human which is a function of humanphysical and mental states that involve activities ranging from neural level to muscular and joint level.
Accordingly, we represent the endogenous human influence solved from the MARL design as a steplength λ dk .
We therefore define human control as,
[Equation]
Including this estimated human control into the problem of robot control design, we take into accounthuman influence on human-robot walking performance while they share the same symmetricalwalking goal.
Symmetrical walking as shared task goal.
We consider the stage cost to be shared between thehuman and the robot.
[Equation]
where R x ∈ R 4 × 4 and R u ∈ R 3 × 3 are positive semi-definite weighting matrices, R v and µ arepositive weighting constants, and h k = v k − ¯ v k .
In the above, the difference between actual humanstep length λ ik and reference λ
o , denoted as ¯ v k = λ
ik − λ
o , is considered practical and available.
In this formulation, the shared control objective is represented in two ways.
First, the robot kinematicstate variables in x k are to match the intact knee, i.e., the robotic joint angle profile is to matchthat of the intact joint.
Additionally, a walking symmetry measure is directly considered by thedifferences in step length and stance time between human and robot, commonly used gait symmetrymeasures [64, 65, 66].
The human is assumed to perceive the gait symmetry but do not have accessto or understand robot kinematic data (peak knee deflection error ∆ P k and peak knee time error∆ D k ).
The works [16, 41] show from test subject data that human control and adaptation has adirect influence on robot kinematics through dynamic interactions [62].
To account for this dynamiclearning phenomenon, this framework penalizes the estimation error
h k = v k − ¯ v k = λ dk − λ ikbetween human perceived difference in step length.
An unexpected or undesired human action willincrease the cost represented by h k , while a diminishing h k indicates that human input that originatedfrom the neuromuscular system matches actual human performance.
Minimal controller energyexpenditures are also included in the cost structure.
The human-robot control objective to be solve by MARL as a function of the state variablesand the the controls is formulated as an infinite horizon, discounted cost Q ( x k , v k , u k ) = 227 (cid:80) ∞
j = k γ j − k U
x j , v j , u j ) where γ ( 0 < γ < 1 ) is the discount factor for the infinite-horizon problem.
Solutions to control actions.
Let v k
= π v ( z k ) and u k =
π u ( x k ) , where π v and π u are the humanand the robot control policies solved from cMARL, respectively.
Solving optimal control policies π ∗ vand π ∗ u requires solving the optimal Q function that satisfies Bellman optimality equation:
[Equation]
When using a neural network-based actor-critic solution framework, we approximately solve theoptimal control problem using the following iterative procedure ( i is the iteration index),
[Equation]
where π v i ( z k )
=
[Equation]
=
[Equation]
k Q
[Equation]
( x k , v k , u k ) , and 234 Q i ( x k , v k , u k ) are iterative actor policies and iterative Q value function.
During training, the actor and critic back-propagate their respective squared error to update theirweights.
The prediction error of actor e a v ,k , e a u ,k ∈ R is,
[Equation]
The prediction error for the critic e c,k is formulated based on the Bellman error,ϵ c,k
= U + γQ
i ( x k +1 , π v i ( z k +1 ) , π u i ( x k +1 ))
− Q i +1 ( x k , v k , u k ) , (11) and the critic neural network is trained to minimize e c,k
= 12 ϵ 2 c,k .
The optimal state-action cost-to-go function Q ∗ ( x k , v k , u k ) is approximated by a neural network viabackpropagation to minimize the Bellman error on the shared cost signal, not on a local cost signalfor either the human or the robot as the human and the robot are strongly coupled.
The human controland the robot control, in turn, are learned via backpropagation from the shared cost signal, which hasbeen shown successful in learning several complex, real-time control solutions.
Further details on thecMARL automatic control solution is provided in Appendix D.
Experiment
We conduct design evaluations of the proposed cMARL for symmetrical walking using OpenSim, awell-established open source biomechanical modeling tool for conducting biomechanics research andmotor control science [67].
The robot knee control is realized within an FS-IC framework.
In real life,initial impedance parameters can be selected based on manufacturer and/or rehabilitation clinician’srecommendations.
Similar care is given to OpenSim simulated walking yet in all evaluations, theinitial impedance values vary from a large range of settings for fair examination.
As in [37], weenforce realistic safety constraints to prevent the human from stumbling or falling.
Impedanceparameters are reset to initial impedance values if any of the state variables exceed the safety bounds.
The safety protocols followed in this work are fully described in Appendix A.4.
To make the simulations reflective of real world conditions, sensor and actuator noise data extractedfrom real human experimental testing sessions is applied to all the simulations in this study.
Appendix
A.2 provides the complete procedure followed of extracting noise data from experiments involvinghuman subjects and injecting it into all the simulations.
In this section, we provide results of a large set of simulation studies aiming at answering thefollowing questions: 1) Does our cMARL solution framework provide better performance than stateof-the-art baselines, including MADDPG and COMA? 2) Does including human influence in robotcontrol design accelerate learning and improve success rate of policy?
Is our cMARL applicableto different and realistic walking tasks?
To provide answers, we show three sets of evaluations:benchmark, ablation and reliability.
Benchmark and ablation evaluations are based on a level ground walking task with a pace of 1m/ s.
Reliability evaluations are based two new walking tasks: slope walking on a degree ramp and level ground walking at an increased pace of
12m/ s.
Performance Criteria.
In order to ensure that amputee subjects walk safely and continuously, weconsider several performance metrics: 1)
This is an optimal control problem and thus the objective is to minimize regulation cost (smaller is better).
Peak knee error can directly reflect amputee safety,preventing unbalance/stumbling (smaller is better).
3) Symmetry in walking can prevent secondaryinjury (close to 0 is better).
Faster learning in terms of fewer tuning steps is practically importantto amputees (fewer steps is better).
5) High success rate boosts amputees’ confidence and hencenatural walking performance (higher is better).
Details of how the data was obtained can be found in
Appendix E.2.
The main evaluation results are presented in Table 1.
The second value in each entry(i.e.
, after the ± symbol) represents the standard deviation of the performance metrics.
[Table]
Benchmark study.
MADDPG and COMA differ from our proposed cMARL approach towardsymmetry walking.
MARL problems can vary greatly, so are expected for their solutions [68, 69].
To perform benchmark studies, we tailor MADDPG and COMA, respectively.
The details weredescribed in Appendix D.
Figure 2 shows that our cMARL solution has the best convergence profile and success rate overthe baselines.
Training and evaluation results are shown in Figures 3 and 4 which compare theenvironment sample efficiency as well as algorithm performances.
To answer the first questionbased on benchmarking, our cMARL solution outperforms the baselines both in terms of kinematicand symmetry measurements as shown in Figures 3 and 4, center and right panels for training andevaluation, respectively.
Additionally, the left panels of Figures 2, 3 and 4 show that our cMARLsolution outperforms benchmarks with at least 30% less environment samples.
Ablation study.
To gain insights on how human control influences human-robot walking performance,an ablation study is carried out with direct human influenced terms removed, including h k and v k ,from the problem formulation.
This only leaves the kinematics and symmetry measurements in thestage cost.
Similar to the benchmark study, the ablation study has a training session and evaluationsession.
For comparable results, the total cost in evaluation is obtained as U =
( x ) T ( R x ) x + u T R u u. Figures 3 and 4 show that with an estimated human control influence accounted for in the robotcontrol design, our cMARL solution (the green curves) outperforms the one without it (orange curves).
Treating human and robot as collaborating agents toward a shared performance goal, our cMARLsolution approach achieves increased success rate and accelerated learning speed (Figure 2).
Thisresult makes sense as an estimated human control provides a predictive signal to the robot controlwhich aims at duplicating the intact human joint movement.
Additional information on the quality ofestimated human control is given in Appendix B.2.
Reliability study.
Driven by making the proposed cMARL method practical and useful in reallife, we setup two different walking tasks: (1) slope walking (11.5 degree ramp) and (2) walking at an increased pace (1.12m/s).
They will result in different walking patterns from those used in thebaseline study, and thus different knee joint profiles.
For slope walking, knee flexion will be morepronounced during the stance phase since it walks inclined.
In the case of faster pace, stance timewill be compressed.
To carry out the tests, the same training procedure is used as in benchmark andablation studies.
Figure.
5 shows the performance of cMARL during slope walking and increased pace walking tasks.
Performance of the two new tasks follow the same trend as that of the level ground walking at anominal pace.
These results again validate our design approach of using the intact knee movementtrajectory as the target for the robotic knee to copy.
By doing so, we have removed a major controldesign barrier in the way of performing different walking tasks by automatic control.
Limitations of This Study.
Here we conduct an in-depth simulation-based analyses as an importantfirst step to evaluate this novel cMARL solution to the HPC problem.
Simulations are criticallyimportant as exploration of problem formulation, control algorithm design, and systematic evaluationare necessary to be performed prior to human experiment due to factors such as human fatigue,human safety, human loss of interest/confidence caused by repeated trial-and-error, time spent, andsignificant cost associated with testing amputee subjects.
However, our framework is still to betested in human experiment.
Based on several important works in the literature [45, 37, 42, 70],extensive simulation studies followed by real life human test studies [38, 46, 71] have proven a highlysuccessful development procedure for human/robot control.
This will be the next step of this study.
For scenarios in which the terrain or task has changed significantly, a task planner will becomenecessary, making the problem a planning and control problem (as opposed to automatic control,the focus of this work).
This expanded automatic control algorithm must be extensively verified insimulation and then in human tests before it can be integrated into a real-world planning frameworkfor daily use cases.
Such planning frameworks constitute intended future works.
Conclusion and discussion
In this work, we develop a novel cMARL framework towards systematically integrating thehuman and robot as collaborative agents to achieve normative walking toward solving real worldproblems.
With reaching symmetric locomotion as shared control performance goal, we demonstrateimproved walking performance.
2) Symmetry is selected as the shared goal for the collaboratingagents because asymmetric walking has been linked to secondary health complications includingback pain and osteoarthiritis [72, 73, 74].
Although human-robot walking performance goals aredifficult to systematically catalogue, additional considerations such as embodiment of the robot intothe human will be considered in future works.
By breaking apart the shared cost for the human andthe robot (cf.
Section 5.1), the symmetrical walking task is treated and evaluated by MADDPG and
COMA, respectively.
Simulation results show that the factorization-based CTDE paradigm strugglesto address the human-robot problem.
The observed performance issues with factorization likely stemfrom the intrinsic coupling between the human and robot agents.
4) While ensuring human user safetyhas been carefully considered during control design, additional analysis of important properties suchas convergence of learning, (sub)optimality of control policy, and human-robot closed-loop stabilityis still needed for this framework.
Encouragingly, previous related works [75, 76, 77, 78] indicatethat these theoretical results are very likely to be provable.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes]
• Did you include the license to the code and datasets?
[Yes] Simulation details and exact hyperparameters are reported in the supplemental material. Code will be released along withthe camera ready version with a URL to the github repository.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A] We don’t have theoretical analysis
(b) Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)?
[Yes] Detail instructions in supplemental material
(b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] In supplement file
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
[Table]
a) If your work uses existing assets, did you cite the creators?
[Yes]
) Did you mention the license of the assets?
[Yes]
Did you include any new assets either in the supplemental material or as a URL?
[N/A]
Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] In supplement file
) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] In supplement file
If you used crowdsourcing or conducted research with human subjects...
[Table]
) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
Did you describe any potential participant risks, with links to Institutional Review
Board ) approvals, if applicable?
[Yes]
Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]