Human-Robotic Prosthesis as Collaborating Agents for Symmetrical Walking
Anonymous Author(s)
AffiliationAddress
Abstract
This is the first attempt at considering human influence in the reinforcement learningcontrol of a robotic lower limb prosthesis toward symmetrical walking in realworld situations.
We propose a collaborative multi-agent reinforcement learning(cMARL) solution framework for this highly complex and challenging humanprosthesis collaboration (HPC) problem.
The design of an automatic controllerof the robot within the HPC context is based on accessible physical features ormeasurements that are known to affect walking performance.
Comparisons aremade with the current state-of-the-art robot control designs, which are single-agentbased, as well as MARL solution approaches tailored to the problem, includingmulti-agent deep deterministic policy gradient (MADDPG) and counterfactualmulti-agent policy gradient (COMA).
Results show that, when compared to theseapproaches, treating human and robot as coupled agents and using estimated humanadaption in robot control design accelerates learning of walking tasks and increaseslearning success rate.
The proposed framework can potentially be further developedto examine how human and robotic lower limb prosthesis interact, an area that littleis known about.
Advancing cMARL toward real world applications such as HPCfor normative walking sets a good example of how AI can positively impact onpeople’s lives.
Introduction
The concept, design, and applications of human-robot cooperation have advanced rapidly due to newdemands in AI-enabled applications fueled by powerful deep learning and reinforcement learningalgorithms [1, 2, 3, 4].
Human-robot collaboration can take on a variety of forms depending on tasksto be solved, how information is shared and the nature of interaction [2, 3, 4].
Examples of complexcollaborative tasks include picking up or carrying objects together, cooperating on a production line,in which cases a robot can learn to imitate human demonstrations [5, 6, 7, 8, 9].
Other applicationscenarios may include intermittent robotic correction of human driving, or vice versa [10].
In essence,most of these recent studies involve interactions between a human and a robot in a way such thatthere is either space between the agents, or there is time for predictive counter measures to takeplace.
Lower limb exoskeleton is another major type of wearable robot. The conceptualizationof the control problem is different even though recent designs also take into account human-robotinteracting effects [11, 12] as these exoskeletons mainly focus on end point performance of a footor lower limb joints where user intent is quantified by estimating human joint torque or interactivetorque. Thus, human-exoskeleton system performance goal is for the robot to produce a well-definedjoint or endpoint trajectory. Seamless collaboration between a robotic lower limb prosthesis serially attached to its human user is critical to achieve normative walking [16, 17].
Yet, the problem is unique and it is challenging to appropriately set up human-robot performance goals. The human-robot collaborative walkingproblem involves two strongly coupled agents, the interacting dynamics of which are difficult ornearly impossible to describe by an ordinary difference or differential equation. Additionally, humanperception and cognition change when wearing a prosthesis [18]. Locomotion of a human-robotsystem may be affected by several factors, such as lower limb mechanics, inter-limb neuromechemicalcoupling, and physical structure of the human body including the lower limb [19, 20].
Physicaland physiological differences in individual human subjects further complicates the robot controldesign. Even though our knowledge of human motor control and motor learning, in cases such aspost stroke or a general loss of normative locomotion capabilities, have expanded greatly [21], little isknown about how an active robotic prosthesis, not a traditional stick-type passive prosthesis, affectshuman and vice versa in walking tasks. This is because that the human-robotic prosthesis interactscontinuously, the intricate human neurocontrol circuits including sensing, perception and feedbackcontrol, which are necessary to facilitate normative walking, is disrupted after amputation. Innovative solution frameworks are needed to first address the design, analysis, and testing of complexcontrol systems to restore locomotion for amputees, based on which, human-robot interactions can bestudied, and fundamental issues such as user preference and prosthesis embodiment can be examined.
The knowledge in turn, will further help design seamless automatic robot control systems.
As such,challenges arising from controlling a wearable robotic lower limb prosthesis to meet the human user’sneeds are unique, and in some aspects, these challenges are greater than what have been studied[16, 17, 22, 23].
Achieving normative walking is fundamentally a real time control problem that involves continuousstates and continuous controls of the human-robot system.
State-of-the-art automatic control ofrobotic prosthesis has been reported recently with successful real human subject testing [24, 25].
These approaches are single agent-based reinforcement learning controls, specific tasks such aslevel ground and ramp walking were accomplished via designer prescribed or specified robotic jointmovement profiles.
A most recent single-agent based reinforcement learning solution demonstratedthat human amputee subject can perform level ground and ramp walking with the robot controlleraimed at mimicking the intact joint movement.
Note, however, that all previous results have notconsidered human influence on the human-robot system performance in the automatic control ofthe robotic prosthesis, a phenomena that has direct impact not only on restoring walking but alsoon human health [26].
Multi-agent reinforcement learning control is a natural candidate to addressthis challenge.
Yet, only a handful of results are available, and they are not directly in the problemdomain under our consideration.
For example, existing work on the physical human-exoskeletoncollaboration treated the problem in a leader (human)-follower (exoskeleton) setting.
Leader-followercoordination was achieved by measured/predicted interaction force [27].
Such an approach was tohave the robot follow the learned behavior from human.
Great effort has been made develop continuous control reinforcement learning algorithms which cansubstantively address real world applications.
Several algorithms, such as deep deterministic policygradient (DDPG) [28], proximal policy optimization (PPO) [29], soft actor-critic (SAC) [30], andtwin delayed DDPG (TD3) [31], have demonstrated success with solving complex control problems.
For example, in simulated human locomotion control [32, 33], over 20 independent control signals areused to control a humanoid robot to achieve different walking tasks.
Robotic upper-limb control hasundergone intense development, especially in the realms of robot surgical assistance [34, 35], patientrehabilitation [34], and industrial applications [36].
However, these problems differ fundamentallyfrom the human-robot walking problem, as their control target usually consists of well-defined endpoints generated by a decoupled trajectory generation exosystem [37].
On the other hand, walkingtasks are difficult to associate with an end point task goal due to heavy dynamic coupling betweenthe human reference and robotic limb.
Single agent-based continuous control has also demonstratedpromise in engineering applications such as stabilization, tracking, and reconfiguring control of
Apache helicopters [38, 39, 40], stabilization and control of large power grids [41, 42, 43], roboticmanipulation and locomotion via MuJoCo and OpenAI Gym [44, 45], and wearable robots withhuman in the loop [46, 47, 48].
While these works are encouraging toward solving realistic singleagent continuous control problems, it is not obvious how they can directly address the multi-agenthuman-robot normative walking problem.
Related work
Single agent RL for automatic control of a robotic limb.
Most state-of-the-art RL control designapproaches to enable continuous human-robot walking are single-agent based.
Important milestoneshave been achieved by two major classes of RL algorithms: 1) actor-critic algorithms, including directheuristic dynamic programming (dHDP) and Q-learning [46, 47, 25, 49, 50, 51], and 2) variants ofpolicy iteration algorithms, including flexible policy iteration (FPI)
[52] and policy iteration withconstraint embedded (PICE) [53, 54].
Both types of single-agent based control algorithms were testedon amputee subjects walking with a robotic knee prosthesis.
In human tests, 12 prosthesis controlinputs were tuned within approximately 10 minutes of level ground walking.
The tuned single-agentcontrollers were then tested on human subjects walking at different speeds and in different terrainconditions [53].
Note that all of the above methodologies and tests used designer-prescribed robotjoint movement profiles generated a priori for the specific subjects and walking tasks.
In real worlduse scenarios, joint movement profiles must evolve dynamically in real-time to accommodate internalhuman walking objectives.
Additionally, profiles will vary significantly between individuals, even forexecution of the same task in the same environment.
A more recent single-agent RL control workreplaced designer-prescribed joint movement profiles with the intact joint motion, achieving greatsuccess in human amputee tests .
The current study, therefore, makes use of this idea as a constituentelement of controller design.
Note also that, none of the above results have taken into account humaninfluence on human-robot walking in the control design, a novel contribution of this work.
Multi-agent reinforcement learning (MARL).
A cMARL problem usually cannot be transformedto an equivalent single-agent problem, as additional agent(s) and their control policies introduceuncertainties and/or non-stationarity to the environment [55].
Partial observability is common incMARL problems, which are usually addressed by the decentralized POMDP framework.
Centralizedtraining decentralized execution (CTDE) is a popular paradigm to address multi-agent coordinationproblems.
A central feature of most CTDE approaches is factorization of the joint state-action valuefunction into individual utility functions.
MADDPG [56] is a popular CTDE-based method to solveboth cooperative and competitive MARL problems.
Agents are assumed to communicate completeand perfect information with each other, so a centralized critic is trained from global state/action data,allowing agents to have individual reward functions.
However, since it is unrealistic for the human tohave direct access to robot kinematic sensor data, this structure is not immediately appropriate forthe human-robot collaborative walking problem.
COMA [57] is another popular MARL algorithmwhich utilizes a single centralized critic by using global state information and actions of all agents.
Additionally, COMA addresses the multi-agent credit assignment challenge by using a counterfactualbaseline which keeps all other agents’ actions fixed while treating a single marginalized agent.
Ingame-playing problems
, COMA significantly improves performance over other MARL methodssuch as independent Q-learning [58], multiagent bidirectionally-coordinated nets [59], and variant
DQN [60].
However, due to the heavy dynamic coupling between the human and prosthesis, it isimpossible to keep one agent action fixed and marginalize out the other agent’s action. As a result,this counterfactual baseline is not appropriate for our problem, either.
Contributions.
Human-robot collaborative tasks will continue to play an increasingly vital role inmodern life, and existing single- and multi-agent control frameworks fail to substantively addressthese classes of problems.
As such, the contributions of this work are as follows.
1) We introducean innovative cMARL design approach to automatically control a robotic lower limb prosthesis bytreating the human user as a collaborating agent whose control action is used to represent humaninfluence on human-robot symmetrical walking performance.
We demonstrate that by having theprosthetic knee track the intact knee, the proposed framework does not require a control designer toprescribe a robot control target for each walking task and for each person, overcoming a fundamentaldesign barrier facing existing methods in real life applications. Based on three sets of simulationstudies (benchmark, ablation, and evaluation), results show that the proposed design outperformsbenchmark MARL methods tailored to the problem, as well as state-of-the-art single agent basedcontrol designs.
Fundamental control problem statement
In powered lower limb prosthesis, the finite state impedance control (FS-IC) framework mostfrequently serves to provide intrinsic control of a robotic joint, i.e., it is the built-in controller from robotic prosthesis manufacturers.
While position control is common in industrial robots, when ahuman is paired with a robotic joint, robot trajectory tracking using position control may precludeany dynamic interaction of the robot with its human user and the environment [61].
This may causean amputee to react to the awkwardness of the prosthesis rather than to interact constructively with it[62].
As FS-IC impedance control framework is considered to provide compliant control for humanuse of the robot, it is expected to generate stable and predictable human-robot walking behavior, andthus, it is adopted in this study which aims for its controller designs at real world applications.
The FS-IC treats a gait cycle, or a step, as four consecutive gait phases (Figure 1, Left): stanceflexion (STF) as Phase 1, stance extension (STE) as Phase 2, swing flexion (SWF) as Phase 3,and swing extension (SWE) as Phase 4.
In what follows, we design four individual controllers foreach of the four phases.
Since the four controllers are conceptualized similarly and designed usingthe same approach, for the sake of clarity we carry out subsequent discussion without explicitlyreferring to the specific phase numbers (but we emphasize that each of the four phases requires itsown independently-trained controller).
Additional details on FS-IC such as phase detection andtransition are provided in Appendix A.3,
At the k -th gait cycle, a robot controller (solved by cMARL in this paper) is to determine threeimpedance parameters, I k
=
[ K k , B k , ( θ e ) k ] T ∈ R 3 under the FS-IC framework, representingstiffness K k , damping coefficient B k , and equilibrium position ( θ e ) k , respectively.
The prostheticjoint motor torque T k ∈ R is then generated based on joint kinematics (knee joint angle θ and angularvelocity ω ) according to the following impedance control law,
[Equation]
The control problem formulation requires automatically determining 12 control inputs or impedancecontrol parameters (3 for each of the 4 gait phases) for individual users aiming at walk in real worldsituations.
The initial set of feasible baseline impedance control parameters I 0 can be obtained frommanufacturers or rehabilitation clinics.
Impedance updates take place according to
[Equation]
where u k ∈ R 3 is to be determined from our proposed cMARL approach.
Method
The human user and the robotic limb are considered collaborating agents.
Our cMARL solutionapproach toward human-robot symmetrical walking problem is formulated based on physical featuresand measurements that have been shown affecting human-robot walking performance, and they arerespectively available to the human and the robot, but not necessarily to each other.
State and control variables.
Refer to Figure 1.
At the k -th gait cycle, let τ ik and λ ik , respectively,represent the stance time (time of foot on the ground) and step length (length between two consecutive steps when toe touching the ground) of the human intact leg.
Similarly, τ pk and λ pk , respectively, forthe prosthetic leg.
Let ∆ τ k , ∆ λ k denote respectively the difference of stance time and step length,
[Equation]
We thus define human state variable z k ∈ R 2 as z k =
[∆ τ k , ∆ λ k ] T .
Robot states include kinematic features determined from the knee movement profiles as well as steplength and stance time (or z k ).
For robot kinematic state variables, we extract gait kinematic featuresfrom knee motion profiles of both limbs (Figure 1).
Let P ik and D ik represent the knee angle andtime duration of the intact limb, and similarly, P pk and D pk for the prosthetic limb.
Let ∆ P k and ∆ D kdenote the error of peak knee angles and the error of duration time between the intact and the roboticknees, respectively,
[Equation]
We then define robot state x k ∈ R 4 to include the following four variables, the first two of which areshared between the human and the robot, x k = [∆ τ k , ∆ λ k , ∆ P k , ∆ D k ] T .
As such, human states are available to the robot, but a subset of the robot state (the kinematic features)is not available to the human.
This reasonably reflects practical situations in real life applications.
The robot control policy is a state feedback law from robot state to robot impedance control parameters,namely, control u k consists of increments to the impedance control parameters (Eq. 2),
[Equation]
Determining human control policy which originates from human neurocontrol circuit is a dauntingtask, especially now that sensing, perception and feedback control circuits are interrupted after alower limb amputation.
While agents’ policy solutions are solved from Bellman optimality-likeequations, a human cannot interpret and implement an MARL solution via their complicated neuralcircuits.
But this is not our intention.
We instead use the human control variable as an estimated inputto the robot controller design, i.e., the robot is informed by an estimated human influence when theyshare the same goal of symmetrical walking.
As the very first step to demonstrate this idea, we consider the human user of the wearable robotintentionally or voluntarily walk at a reference step length, which may also be from an instructionalfeedback.
For real life scenarios, such behavioral cues can change over time, task, or environment.
But it is important to validate such formulation of considering human influence in the robot controldesign.
Toward this end, we consider a practical reference cue denoted as a desired step lengthλ o .
We let v k represent an endogenous control signal of the human which is a function of humanphysical and mental states that involve activities ranging from neural level to muscular and joint level.
Accordingly, we represent the endogenous human influence solved from the MARL design as a steplength λ dk .
We therefore define human control as,
[Equation]
Including this estimated human control into the problem of robot control design, we take into accounthuman influence on human-robot walking performance while they share the same symmetricalwalking goal.
Symmetrical walking as shared task goal.
We consider the stage cost to be shared between thehuman and the robot.
[Equation]
where R x ∈ R 4 × 4 and R u ∈ R 3 × 3 are positive semi-definite weighting matrices, R v and µ arepositive weighting constants, and h k = v k − ¯ v k .
In the above, the difference between actual humanstep length λ ik and reference λ
o , denoted as ¯ v k = λ
ik − λ
o , is considered practical and available.
In this formulation, the shared control objective is represented in two ways.
First, the robot kinematicstate variables in x k are to match the intact knee, i.e., the robotic joint angle profile is to matchthat of the intact joint.
Additionally, a walking symmetry measure is directly considered by thedifferences in step length and stance time between human and robot, commonly used gait symmetrymeasures [63, 64, 65].
The human is assumed to perceive the gait symmetry but do not have accessto or understand robot kinematic data (peak knee deflection error ∆ P k and peak knee time error ∆ D k ).
The works [25, 50] show from test subject data that human control and adaptation has adirect influence on robot kinematics through dynamic interactions [61].
To account for this dynamiclearning phenomenon, this framework penalizes the estimation error
h k = v k − ¯ v k = λ dk − λ ikbetween human perceived difference in step length.
An unexpected or undesired human action willincrease the cost represented by h k , while a diminishing h k indicates that human input that originatedfrom the neuromuscular system matches actual human performance.
Minimal controller energyexpenditures are also included in the cost structure.
The human-robot control objective to be solve by MARL as a function of the state variablesand the the controls is formulated as an infinite horizon, discounted cost Q ( x k , v k , u k ) = 230 (cid:80) ∞
j = k γ j − k U
x j , v j , u j ) where γ ( 0 < γ < 1 ) is the discount factor for the infinite-horizon problem.
Solutions to control actions.
Let v k
= π v ( z k ) and u k =
π u ( x k ) , where π v and π u are the humanand the robot control policies solved from cMARL, respectively.
Solving optimal control policies π ∗ vand π ∗ u requires solving the optimal Q function that satisfies Bellman optimality equation:
[Equation]
When using a neural network-based actor-critic solution framework, we approximately solve theoptimal control problem using the following iterative procedure ( i is the iteration index),
[Equation]
where π v i ( z k )
=
[Equation]
[Equation]
π u i
[Equation]
x k ) = arg min u k Q i ( x k , v k , u k ) , and 237 Q i ( x k , v k , u k ) are iterative actor policies and iterative Q value function.
During training, the actor and critic back-propagate their respective squared error to update theirweights.
The prediction error of actor e a v ,k , e a u ,k ∈ R is,
[Equation]
The prediction error for the critic e c,k is formulated based on the Bellman error,
[Equation]
11) and the critic neural network is trained to minimize e c,k
= 12 ϵ 2 c,k .
The optimal state-action cost-to-go function Q ∗ ( x k , v k , u k ) is approximated by a neural network viabackpropagation to minimize the Bellman error on the shared cost signal, not on a local cost signalfor either the human or the robot as the human and the robot are strongly coupled.
The human controland the robot control, in turn, are learned via backpropagation from the shared cost signal, which hasbeen shown successful in learning several complex, real-time control solutions.
Further details on thecMARL automatic control solution is provided in Appendix B.3.
Experiment
We conduct design evaluations of the proposed cMARL for symmetrical walking using OpenSim, awell-established open source biomechanical modeling tool for conducting biomechanics research andmotor control science [66].
The robot knee control is realized within an FS-IC framework.
In real life,initial impedance parameters can be selected based on manufacturer and/or rehabilitation clinician’srecommendations.
Similar care is given to OpenSim simulated walking yet in all evaluations, theinitial impedance values vary from a large range of settings for fair examination.
As in [46], weenforce realistic safety constraints to prevent the human from stumbling or falling.
Impedanceparameters are reset to initial impedance values if any of the state variables exceed the safety bounds.
The safety protocols followed in this work are fully described in Appendix A.4.
To make the simulations reflective of real world conditions, sensor and actuator noise data extractedfrom real human experimental testing sessions is applied to all the simulations in this study.
Appendix
A.2 provides the complete procedure followed of extracting noise data from experiments involvinghuman subjects and injecting it into all the simulations.
In this section, we provide results of a large set of simulation studies aiming at answering thefollowing questions: 1) Does our cMARL solution framework provide better performance than stateof-the-art baselines, including MADDPG and COMA? 2) Does including human influence in robot control design accelerate learning and improve success rate of policy?
Is our cMARL applicableto different and realistic walking tasks?
To provide answers, we show three sets of evaluations:benchmark, ablation and reliability.
Benchmark and ablation evaluations are based on a level groundwalking task with a pace of 1m/s.
Reliability evaluations are based two new walking tasks: slopewalking on a 11.5 degree ramp and level ground walking at an increased pace of 1.12m/s.
A trial consists of 300 gait cycles of continuous walking, which is chosen according to publishedresults based on single-agent design studies using simulations and human experiments. Each gaitcycle will generate one data tuple ( x k , u k , v k , U v , U u , x k +1 ) .
A trial is considered successful if allstate variables in x k reach their respective error tolerance bound (Appendix A.4, Table 1, bottom row). Additionally, one gait phase at a time, if 8 out of 10 consecutive gaits meet the state error tolerancebound condition, training has converged.
If all 4 phases have converged within 300 gait cycles, thetrial is a success.
Results reported in this paper are based on 16 trials of OpenSim simulations foreach study below. Different code implementation of different RL algorithms is known to cause large variances andimpact on performance.
To minimize such potential influence, all algorithms are implemented using
Matlab [67], and all actor-critic implementations in all comparisons share the same code base [46, 68].
Benchmark study.
MADDPG and COMA differ from our proposed cMARL approach towardsymmetry walking (Figure 1, Right).
MARL problems can vary greatly, so are expected for theirsolutions [69, 70].
To perform benchmark studies, we tailor MADDPG and COMA, respectively as follows.
Referto Figure 7.
For MADDPG, we allow human androbot have different stage cost as U v =
z T R z z + R v v 2 + µh 2 and U u = ( x − ) T ( R x − ) x − +u T R u u , respectively where x − only includes therobotic joint kinematic variables.
These stagecosts respectively go to update critic Q v and
Q u for human and robot.
Human policy ( π v )(function of z ) and robot policy ( π u ) (function ofx − ) are determined accordingly.
For COMA, wetrain a single centralized critic with joint stagecost U
=
U v
+ U u where U v , U u are the same asin MADDPG.
Policies will execute locally.
Weupdate the policies using respective TD errorsin place of a non-biased advantage function [57]because we are dealing with a strongly coupledhuman-robot system.
Therefore it does not make sense to implement the counterfactual advantagefunction as in COMA.
Also note that, we use real-time training instead of batch memory becausecollecting batch data for policy update requires an amputee subject to endure multiple gait cyclesbefore one learning update.
This may not be realistic for practical use.
Nonetheless, an evaluation isperformed (Appendix D).
Details on the derivations and implementations of the tailored MADDPGand COMA are in Appendix C.
Each of the three approaches used the same random seeds during training and evaluation.
Evaluationtrials are performed on 5 randomly selected ones out of the 16 training trials, and evaluations areconducted every 2 gaits during the first 50 gaits and then every 10 gaits afterwards, each usingthe latest policy at the time of evaluation.
Due to very low success rate of the tailored MADDPGmethod and very low chance of retrieving a successful policy (Figure 4, grey lines), also that itscurrent implementation may be viewed as a decomposed COMA, we have removed MADDPG fromevaluation.
Figure 3 shows that our cMARL solution has the best convergence profile and success rate overthe baselines.
Training and evaluation results are shown in Figures 4 and 5 which compare theenvironment sample efficiency as well as algorithm performances.
To answer the first questionbased on benchmarking, our cMARL solution outperforms the baselines both in terms of kinematicand symmetry measurements as shown in Figures 4 and 5, center and right panels for training andevaluation, respectively.
Additionally, the left panels of Figures 3, 4 and 5 show that our cMARLsolution outperforms benchmarks with at least 30% less environment samples.
Ablation study.
To gain insights on how human control influences human-robot walking performance,an ablation study is carried out with direct human influenced terms removed, including h k and v k ,from the problem formulation.
This only leaves the kinematics and symmetry measurements in thestage cost.
Similar to the benchmark study, the ablation study has a training session and evaluationsession.
For comparable results, the total cost in evaluation is obtained as U =
( x ) T ( R x ) x + u T R u u. Figures 4 and 5 show that with an estimated human control influence accounted for in the robotcontrol design, our cMARL solution (the green curves) outperforms the one without it (orange curves).
Treating human and robot as collaborating agents toward a shared performance goal, our cMARLsolution approach achieves increased success rate and accelerated learning speed (Figure 3).
Thisresult makes sense as an estimated human control provides a predictive signal to the robot controlwhich aims at duplicating the intact human joint movement.
Additional information on the quality ofestimated human control is given in Appendix E.
Reliability study.
Driven by making the proposed cMARL method practical and useful in reallife, we setup two different walking tasks: (1) slope walking (11.5 degree ramp) and (2) walking atan increased pace (1.12m/s).
They will result in different walking patterns from those used in thebaseline study, and thus different knee joint profiles.
For slope walking, knee flexion will be morepronounced during the stance phase since it walks inclined.
In the case of faster pace, stance timewill be compressed.
To carry out the tests, the same training procedure is used as in benchmark andablation studies.
Figure.
6 shows the performance of cMARL during slope walking and increased pace walking tasks.
Performance of the two new tasks follow the same trend as that of the level ground walking at anominal pace.
These results again validate our design approach of using the intact knee movementtrajectory as the target for the robotic knee to copy.
By doing so, we have removed a major controldesign barrier in the way of performing different walking tasks by automatic control.
[Figure]
Conclusion and discussion
[Figure] In this work, we develop a novel cMARL framework towards systematically integrating the humanand robot as collaborative agents to achieve normative walking in real world scenarios.
With reachingsymmetric locomotion as shared control performance goal, we demonstrate expedited learning timeand improved symmetry of walking.
Although this new collaborative control framework can inprinciple be expanded to considering additional locomotion performance measures (which representsa fundamental and unique challenge in robotic lower limb prosthesis), additional work is needed tofully and appropriately represent endogenous human controls and their effect on human-robot systemduring walking.
Further extensive simulation studies and human subject testings are also needed tomake the proposed control framework fully utilizable in real life situations.
2) Symmetry is selected asthe shared goal for the collaborating agents because asymmetric walking has been linked to secondaryhealth complications including back pain and osteoarthiritis [71, 72, 73].
Although human-robotwalking performance goals are difficult to systematically catalogue, additional considerations suchas human energy expenditure and embodiment of the robot into the human will be considered infuture works.
By breaking apart the shared cost for the human and the robot (cf.
Section 5.1), thesymmetrical walking task is treated and evaluated by MADDPG and COMA, respectively.
Simulationresults show that the factorization-based CTDE paradigm struggles to address the human-robotproblem.
The observed performance issues with factorization likely stem from the intrinsic couplingbetween the human and robot agents.
4) While ensuring human user safety has been carefullyconsidered during control design, additional analysis of important properties such as convergence oflearning, (sub)optimality of control policy, and human-robot closed-loop stability is still needed forthis framework.
Encouragingly, previous related works [74, 75, 76] indicate that these theoreticalresults are very likely to be provable.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A] We don’t have
theoretical analysis
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] Detail
instructions in supplemental material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] In supplement file
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] In supplement file
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] In supplement file
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[Yes]
[Table]
[Table]