Distributed Inverse Constrained Reinforcement Learning for Multi-agent Systems
Anonymous Author(s)
AffiliationAddress email
Abstract
This paper considers the problem of recovering the policies of multiple interacting experts by estimating their reward functions and constraints where the demonstration data of the experts is distributed to a group of learners.
We formulatethis problem as a distributed bi-level optimization problem and propose a novelbi-level “distributed inverse constrained reinforcement learning” (D-ICRL) algorithm that allows the learners to collaboratively estimate the constraints in theouter loop and learn the corresponding policies and reward functions in the innerloop from the distributed demonstrations through intermittent communications.
We formally guarantee that the distributed learners asymptotically achieve consensus which belongs to the set of stationary points of the bi-level optimizationproblem.
Introduction
Multi-agent systems (MASs) have become an effective way to model large-scale networked systems,e.g. , multi-robot systems and the Internet-of-Things.
Multi-agent reinforcement learning (MARL) isa systematic framework to coordinate MASs.
It extends RL to address the decision-making problemof multiple interacting agents [1].
One challenge in (MA)RL is that manually crafting the rewardfunctions or predesigning the policies can be infeasible for humans when the tasks are too complicated.
Inverse reinforcement learning (IRL) presents a way to tackle this problem where a learneraims to recover a reward function (and a corresponding policy) that best explains the behavior in thedemonstrations of an expert.
Current state-of-the-arts on IRL [2, 3, 4, 5] can successfully solve thereward function inference problem but still face two other major challenges in MARL and MASs:(i) it is often the case that the expected behavior (experts’ behavior) is more precisely explained bya reward function combined with a set of constraints instead of a single reward function; (ii) thedemonstration data in MASs is usually distributed over multiple learners and unable to be shareddue to some practical reasons including privacy concerns and communication burdens.
This paperaims to bridge the gap and proposes the first algorithm to solve the three challenges (i.e., reward function inference, constraint inference, and distributed demonstration data) at once.
Related works .
IRL is an ill-posed and ambiguous problem [6] as different reward functions could explain the demonstrations.
To solve this problem, several fundamental approaches are proposed including feature expectation matching [7], maximum margin planning [8], maximum entropy [9], andmaximum causal entropy (MCE) [10].
Recently, some advanced techniques baring more features are proposed.
Paper [2] uses adversarial learning to learn reward functions robust to environmentalchanges, paper [4] applies strongly convex regularizers to the learner’s policy to avoid the ambiguity problem, and paper [5] proposes a scalable Bayesian-based method working for large control tasks.
In real-world settings, numerous constraints are present and a single reward function is not enoughto explain complicated behavior.
Paper [11] extends ME IRL to environments with constraints, formulates the constraint inference problem as a maximum likelihood estimation (MLE) problem,and leverages a greedy method to solve the problem.
Paper [12] extends paper [11] to continuousenvironments.
These two works both assume the access to the ground truth rewards.
While the discussions in the last two paragraphs are limited to single agents, there have been recentworks which extend IRL techniques to multiple agents [13, 14, 15, 16, 17].
Notice that previousworks on multi-agent IRL do not explicitly distinguish the notions of “experts” and “learners”,their “multiple agents” means “multiple experts” and they only consider centralized learning ( i.e. ,a single learner who obtains all demonstrated behavior of the multiple experts).
In this paper, weexplicitly distinguish the notions of “experts” and “learners”, and use “multiple experts” to representthe commonly used “multiple agents” in literature.
Moreover, the aforementioned papers do notconsider the more general situation where the demonstrations are distributed over multiple learnersand unable to be shared.
To handle this situation, the learners need to perform distributed learning.
As a basic topology in distributed learning, peer-to-peer (P 2 P) architecture has no central server andeach node can communicate with a subset of all the nodes.
It reduces the risk of single points offailures [18] in the system.
This paper proposes D-ICRL based on the P 2 P structure where ICRLcan learn policies by estimating the reward functions and constraints through demonstrations anddistributed learners can perform ICRL without sharing the distributed demonstration data.
In specific, we formulate D-ICRL as a distributed bi-level optimization problem where the learnerscooperatively learn the constraints through the outer problem and estimate the corresponding rewardfunctions and policies through the inner problem.
Paper [19] provides asymptotic convergence forcentralized bi-level optimization problems by assuming that the inner problem can be completelysolved and its optimal solution can be obtained.
Papers [20, 21, 22] relax this assumption by partiallysolving the inner problem and provide convergence rate analysis for problems with strongly convexinner objective function.
Paper [23] solves a special case of distributed bi-level optimization wherethe outer and inner problems use identical decision variables.
To the best of our knowledge, there isno work on bi-level optimization which (i) assumes that the inner objective function is merely strictlyconvex and partially solves the inner problem; (ii) solves the general form of bi-level optimizationin a distributed way.
Notice that these two challenges arise in our D-ICRL problem.
Contribution statement .
Our contributions are threefold.
First, we extend IRL to a multi-expertmulti-learner (MEML) setting where a group of learners collaboratively learn policies by estimatingthe reward functions and constraints without sharing the distributed demonstrations performed by aset of cooperative experts.
We formulate this D-ICRL problem as a distributed bi-level optimizationproblem.
Second, we propose a distributed bi-level learning algorithm to solve this problem wherethe learners cooperatively estimate the constraints in the outer problem and learn the correspondingreward functions and policies in the inner problem.
Third, we provide convergence rate analysis tothe optimal solution of the inner problem and asymptotic convergence to the set of stationary pointsof the outer problem.
Model
In this section, we present the models for cooperative experts and distributed learners.
Experts .
There are N E experts whose decision making is based on a constrained Markov game(CMG)
A CMG ( S , A , γ, P 0 , P, r E , c E , b ) is defined via state set S ≜ (cid:81) N E i =1 S
[ i ] , actionset A ≜ (cid:81)
N E i =
1 A [ i ] , discount factor γ ∈ (0 , 1) , and initial state distribution P 0 .
The state-actionspace can be either discrete or continuous.
The system state transition function is given by P suchthat the probability (or probability density in continuous state-action space) of state transition tos ′ from s by taking action a ≜ ( a [1] , · · · , a [ N E ] ) is P ( s ′ | s, a ) .
The reward function of expert iis r
[ i ] E : S ×
A → R and the experts are cooperative ( i.e. , r E ≜ (cid:80) N E i =1 r [ i ] E ).
The cost functionof expert i is c
[ i ] E : S ×
A → R + and the cost function of the whole system is c E ≜ (cid:80) N E i
=1 c [ i ] E .
Expert i ’s policy π
[ i ] E ( a [ i ] | s ) represents the probability (or probability density) of expert i takingaction a [ i ] at state s and the joint policy of all the experts is denoted by π E ( a | s ) ≜ (cid:81) N E i =
1 π [ i ] E ( a [ i ] | s ) .
Define J r E ( π ) ≜ E πS,A [ (cid:80) ∞ t =
γ t r E ( S t , A t )] as the expected cumulative reward where the initialstate is drawn from P 0 and J c E ( π ) ≜ E πS,A
[ (cid:80) ∞ t =
γ t c E ( S t , A t )] as the expected cumulative cost.
The experts’ policy π E wants to maximize J r E ( π ) subject to J c E ( π ) ≤ b .
Define the constraintset indicated by c [ i ] E as C [ i ] E ≜ { ( s, a ) ∈ S
× A : c [ i ] E ( s, a ) > 0 } .
Following [12], we study thecase where b = 0 , i.e., hard constraint.
Thus, the constraint reduces to J c E ( π ) = 0 , implying thatthe probability of reaching any ( s, a ) ∈ (cid:83)
i C
[ i ] E is zero under policy π .
These experts use π E todemonstrate a set D of trajectories which is partitioned into N L subsets.
Each subset D [ v ] consistsof m [ v ] examples { ζ j } m [ v ] j =1 where each example ζ j ≜ s j 0 , a j 0 , s j 1 , a j 1 · · · is a state-action sequence ofthe experts. Thus the total demonstration D has m ≜ (cid:80) N L v =1 m [ v ] trajectories.
Learners .
There is a group of learners where each learner v knows ( S , A , γ, P 0 , P, b, D [ v ] ) andcan communicate with other learners.
The learners choose a reward feature vector ϕ
[ i ] r : S × A →[0 , d 1 ] l [ i ] r for expert i where d 1 is constant and it is assumed that r [ i ] E =
( ω [ i ] r E )
⊤ ϕ
[ i ] r , where ω [ i ] r E ∈ R l [ i ] ris the (unknown) reward weight vector.
The learners also choose a cost feature vector ϕ
[ i ] c : S×A →[0 , d 2 ] l [ i ] c where d 2 is constant and it is assumed that c
[ i ] E = ( ω
[ i ] c E ) ⊤ ϕ
[ i ] c , where ω
[ i ] c E is the (unknown)cost weight vector and ω
[ i ] c E ,j is the j -th component of ω
[ i ] c E .
Let C
[ i ] j ≜ { ( s, a ) ∈ S×A : ϕ
[ i ] c,j ( s, a ) >0 } be the constraint set indicated by the j -th cost feature ϕ
[ i ] c,j of expert i .
As the budget b = 0 , theeffect of ω [ i ] c E ,j > 0 is the same, i.e., C
[ i ] j cannot be visited, thus the learners assume ω
[ i ] c E ,j ∈ [0 , 1] .
Communication network .
The learners execute a two-time-scale learning algorithm over a communication network.
The fast-time-scale (inner) algorithm is executed at discrete time k over atime-varying directed graph G ( k ) ≜ ( V , E ( k )) where V ≜ { 1 , · · · , N L } is the node (learner) set and
E ( k ) ⊆
V × V is the set of directed edges (communication links) at time k .
The edge ( v, v ′ )
∈ E ( k )means that learner v receives information from learner v ′ at time k
and ( v, v ) ∈ E ( k ) for all k ≥ 0 .
The adjacency matrix of the graph at time k is W ( k ) ≜
[ W [ vv ′ ]
( k )] v,v ′ ∈V ∈ R N L × N L where
W [ vv ′ ] ( k ) =
if and only if ( v, v ′ ) / ∈ E ( k ) .
We use N [ v ] ( k ) ≜ { v ′ ∈ V| ( v, v ′ )
∈ E ( k ) }
to denotethe set of neighbors of learner v at time k .
The slow-time-scale (outer) algorithm is carried out atdiscrete time n over a graph ¯ G ( n ) ≜
( V , ¯ E ( n )) , where ¯ E ( n ) , ¯ W ( n ) and ¯ N [ v ] ( n ) are defined in ananalogous way.
The time unit for n is much larger than that for k .
Assumption 1.
There exists an integer B ≥ 1 such that the graph ( V , E ( k ) ∪ · · · ∪ E ( k + B − 1))is strongly connected for all k ≥ 0 .
Assumption 2.
The adjacency matrix W ( k ) has the following properties:(i) (doubly stochastic) 1 ⊤ W ( k ) = 1 ⊤ and W ( k ) 1 = 1 where 1 is the column vector whose entriesare all ones.
(ii) (non-degenerate
There is an ϵ ∈ (0 , 1) such that W [ vv ] ( k ) ≥ ϵ for all v ∈ V and
W [ vv ′ ] ( k ) ≥ ϵ
if ( v, v ′ ) ∈ E ( k ) .
Assumption 3.
The set ¯ N [ v ] ( n ) =
V for any v ∈ V and n ≥ 0 .
The adjacency matrix ¯ W ( n ) isdoubly stochastic and non-degenerate with constant ¯ ϵ .
Notions and notations
This section provides notions and notations which will be used in the remaining of the paper.
Reward feature expectation matching (RFEM) .
Given policy π , the reward feature expectationvector of expert i is defined as µ [ i ] r ( π ) ≜ E πS,A
[ (cid:80) ∞ t =0 γ t ϕ [ i ] r ( S t , A t )] and the demonstration D isneeded to estimate µ [ i ] r ( π E ) :
ˆ µ [ i ] r ≜ 1 m (cid:80) mj =1 (cid:80) ∞
t =0 γ t ϕ
[ i ] r ( s jt , a jt ) , where ( s jt , a jt ) ∈ ζ j .
The RFEMrequires µ [ i ] r ( π ) =
ˆ µ [ i ] r for every expert i .
However, each learner v can only observe the subset D [ v ]and its local estimate is ˆ µ [ i,v ] r ≜ 1 m [ v ] (cid:80) ζ j ∈D [ v ] (cid:80) ∞
t
=0 γ t ϕ
[ i ] r ( s jt , a jt ) .
Cost expectation matching (CEM) .
Similar to RFEM, CEM requires J c E ( π ) =
J c E ( π E ) .
As thetrue cost function c
E is unknown, given an estimate c ω c = (cid:80) N E i =
( ω [ i ] c )
⊤ ϕ
[ i ] c , we define the empiricalcost estimate of the experts as ˆ b ω c ≜ 1 m (cid:80) mj
=1 (cid:80) ∞ t =
γ t c ω c ( s jt , a jt ) .
The local estimate from learnerv is ˆ b [ v ] ω c ≜ 1 m [ v ] (cid:80) ζ j ∈D [ v ] (cid:80) ∞
t
=
γ t c ω c ( s jt , a jt ) .
Remark 1.
As RFEM is a standard notion in IRL literature, it is “natural” to follow this and use“cost feature expectation matching” (CFEM).
However, here we use CEM instead of CFEM because we want to leverage the budget information provided in CMG.
We still use RFEM instead of “rewardexpectation matching” (REM) because REM does not embed any additional useful information as
CMG does not have any “budget information” for cumulative reward.
Notations .
We use ˆ µ r ≜ [(ˆ µ [1] r ) ⊤ , · · · , (ˆ µ [ N E ] r ) ⊤ ] ⊤ to represent the concatenated empirical rewardfeature expectation vector.
Similarly, we define learner v ’s concatenated empirical reward featureexpectation vector estimate ˆ µ [ v ] r , concatenated reward feature expectation vector µ r ( π ) under π ,and concatenated cost feature vector ϕ c .
Let Π be the set of all valid stochastic policies such thatevery π ∈ Π satisfies π ( a | s ) ≥ 0 for any s ∈ S and a ∈ A and (cid:82) a ∈A π ( a | s ) da = 1 for anys ∈ S .
We define the expected cumulative cost of policy π under cost function estimate c ω c as
J ω c ( π ) ≜ E πS,A [ (cid:80) ∞ t =
γ t c ω c ( S t , A t )] and use Project Ω c ( · ) to denote the projection onto the set Ω c .
Problem formulation
In an MEML D-ICRL problem, each learner v aims to use all the available information to collaboratively recover the policy π E through communications by estimating the reward function r E andthe cost function c E .
In the learning process, learner v is unwilling to share its private data ( i.e. ,local demonstration and local estimate of reward feature expectation and cost expectation).
Noticethat estimating c E is in essence estimating the constraint set (cid:83) i C
[ i ] E .
We formulate the constraintlearning problem as a distributed MLE problem:
[Equation]
where Ω c ≜ [0 , 1] (cid:80) NEi =1 l [ i ] c , F [ v ] ( ω c ) ≜ (cid:80) j ∈D [ v ] (cid:80) ∞
t =
γ t ln π ω c ( a jt | s jt ) is learner v ’s local likelihood function and π ω c is a policy parameterized by ω c .
To find a statistical distribution model forπ ω c , we formulate the following problem based on MCE scheme:
[Equation]
where H ( π ) ≜ (cid:80) ∞
t =0 E S,A [ − γ t ln π ( A t | S t )] is the (infinite-horizon) causal entropy [25].
Remark 2.
Directly matching the ground truth budget (i.e., J ω c ( π ) =
b ) may cause infeasibility of(2), i.e., the feasible set of (2) is not guaranteed to be nonempty.
Therefore, we match ˆ b ω c , which isan estimate of J ω c ( π E ) , to ensure the feasibility because now, π E is a feasible solution of (2).
The dual function of (2) is G ( ω r , λ ; ω c ) ≜
max π ∈ Π H ( π )+ (cid:80) N E i =1 ( ω [ i ] r ) ⊤ ( µ [ i ] r ( π ) − ˆ µ [ i ] r )+ λ ( J ω c ( π )− ˆ b ω c ) where the dual variable ω r =
[( ω [1] r ) ⊤ , · · · , ( ω [ N E ] r ) ⊤ ] ⊤ is to estimate ω r E .
Let η ≜[ ω ⊤ r , λ ] ⊤ , it is well-known that G ( η ; ω c ) is convex in η since it is the pointwise maximum of afamily of affine functions of η .
As G ( η ; ω c ) is global, we introduce the local convex dual function
G [ v ] ( η ; ω c ) ≜ max π ∈ Π H ( π ) + (cid:80) N E i =
[Equation]
( ω [ i ]
[Equation]
) ⊤ ( µ
[Equation]
i ] r
[Equation]
−
[Equation] [Equation]
] r ) + λ ( J ω c ( π ) − ˆ b [ v ] ω c ) .
Lemma 1.
(i)
The optimal solution of problem (2) is the constrained soft Bellman policy π η ∗ ( ω c ); ω c ,where η ∗ ( ω c ) is an optimal solution of min η G ( η ; ω c ) .
(ii)
The local dual function G [ v ] ( η ; ω c ) isdifferentiable and its gradient is [( µ r ( π η ; ω c ) − ˆ µ [ v ] r ) ⊤ , J ω c ( π η ; ω c ) − ˆ b [ v ] ω c ] ⊤ .
The proof of Lemma 1 is similar to [10, 25, 26].
For the sake of completeness, we still include theproof and the expression of the constrained soft Bellman policy in the Appendix.
Lemma 1 indicates that π η ∗ ( ω c ); ω
c is the statistical model we need in problem (1).
Therefore, problem (1) can be reformulated as the following distributed bi-level optimization problem:
[Equation]
where F [ v ] and m [ v ] G [ v ] are known only to learner v .
Remark 3.
Problem (4) and arg min η G ( η ; ω c ) are equivalent because (cid:80) N L v =1 m [ v ] G [ v ]
[Equation]
η ; ω c
[Equation]
=
[Equation]
N L v
[Equation]
m [
[Equation]
] H
[Equation]
+
[Equation] [Equation]
Algorithm and convergence guarantee
In this section, we develop a bi-level distributed learning algorithm to solve problem (3)-(4) and provide convergence rate to the optimal solution of the inner problem (4) and asymptotic convergenceto the set of stationary points of the outer problem (3).
Input : { ω [ v ] c (0) } N L v =1 , { η [ v ] (0) } N L v =1 , W ( k ) , ¯ W ( n )
Output : ω [ v ] c ( n ) , ¯ η [ v ] ( ω [ v ] c ( n )) , π ¯ η [ v ] ( ω [ v ] c ( n )); ω [ v ] c ( n ) ∀ v ∈ V 1: for n = 0 , 1 , · · · do 2: for v ∈ V do 3:
Receives ω
[ v ′ ]
c ( n ) from v ′ ∈ ¯ N [ v ] ( n ) 4: ¯ η [ v ] ( ω [ v ] c ( n )) , π ¯ η [ v ] ( ω [ v ] c ( n )); ω [ v ] c ( n ) =
Inner process ( ω [ v ] c ( n )) 5: if n = 0 then 6: ¯ ∇ [ v ] (0) =
¯ ∇ F [ v ] ( ω [ v ] c (0) , ¯ η
[ v ] ( ω [ v ] c (0))) 7: else 8:
Receives ¯ ∇ [ v ′ ] ( n − 1) from v ′ ∈ ¯ N [ v ] ( n ) 9: ¯ ∇ [ v ] ( n ) =
(cid:80) N L v ′
=1 ¯ W [ vv ′ ] ( n ) ¯ ∇ [ v ′ ] ( n − 1) + ¯ ∇ F [ v ] ( ω [ v ] c ( n ) , ¯ η [ v ] ( ω [ v ] c ( n )))
− ¯ ∇ F [ v ] ( ω [ v ] c ( n − 1) , ¯ η [ v ] ( ω [ v ] c ( n − 1)))
10: end if 11: ˜ ω
[ v ] c ( n ) = Project Ω c (cid:18) ω [ v ] c ( n ) + N L ¯ ∇
[ v ] ( n ) (cid:19) 12: ω [ v ] c ( n + 1 / 2) =
ω [ v ] c ( n ) + β ( n )(˜ ω
[ v ] c ( n ) − ω
[ v ] c ( n )) 13:
Receives ω
[ v ′ ]
c ( n + 1 / 2) from v ′ ∈ ¯ N [ v ] ( n ) 14: ω [ v ] c ( n + 1) =
(cid:80) N L v ′
=1 ¯ W [ vv ′ ] ( n ) ω [ v ′ ]
c ( n + 1 / 2) 15: end for 16: end for
At outer iteration n in Algorithm 1, the learners first cooperatively solve the inner problem foreach learner through Inner process (lines 3 - 4 ) and then use the obtained result ¯ η [ v ] ( ω [ v ] c ( n )) tocollaboratively solve the outer problem (3) through the outer process (lines 5 - 14 ).
In what follows,we will elaborate each process.
Algorithm
Input : ω [ v ] c , { η [ v ] (0) } N L v =1 , W ( k )
Output : ¯ η [ v ] ( ω [ v ] c ) , π ¯ η [ v ] ( ω [ v ] c ); ω [ v ] c 1: for k = 0 , 1 , · · · , K − 2 do 2: for ¯ v ∈ V do 3:
Receives η
[¯ v ′ ]
( ω [ v ] c , k ) from ¯ v ′ ∈
N
[¯ v ] ( k ) 4: η
[¯ v ] ( ω [ v ] c , k + 1)
= (cid:80) N L v ′
=1 W [¯ v ¯ v ′ ]
( k ) η
[¯ v ′ ]
( ω [ v ] c , k ) − α ( k ) ∇ η G [¯ v ] ( η [¯ v ] ( ω [ v ] c , k ); ω [ v ] c ) 5: end for 6: end for 7: ¯ η [ v ] ( ω [ v ] c ) ≜ (
cid:80) K − 1 k =
α ( k ) η [ v ] ( ω [ v ] c ,k ) (cid:80) K −k =
α ( k ) 8: π ¯ η [ v ] ( ω [ v ] c ); ω [ v ] c is the corresponding constrained soft Bellman policy.
Inner process .
As ¯ N [ v ] ( n ) =
V , learner v knows the cost weight vector of all the learners.
Givenω [ v ] c , at inner iteration k , each learner ¯ v receives η
[¯ v ′ ] ( ω [ v ] c , k ) from neighbors and updates its estimateof the reward weight vector (embedded in η [¯ v ] ).
In specific, at line 4 in Algorithm 2, the first term(convex combination) on the right hand side encourages consensus and the second term (gradient)drives to the optimal solution.
It is shown in Lemma 3 that each learner can reach η ∗ ( ω [ v ] c ) at therate of O ( 1 √ log K ) .
Outer process .
Once obtaining ¯ η [ v ] ( ω [ v ] c ) , each learner v faces three challenges in solving the outerproblem: (i) the gradient of F [ v ] ( ω [ v ] c , η ∗ ( ω [ v ] c )) cannot be obtained because the inner problem is notfully solved, i.e., learner v does not find η ∗ ( ω [ v ] c ) but its approximation ¯ η [ v ] ( ω [ v ] c ) , (ii) (cid:80) v ′ ̸ = v F [ v ′ ]is unknown, and (iii) the local likelihood function F [ v ] is non-convex.
To solve these challengesrespectively, we propose three techniques: local gradient approximation (LGA), global gradientapproximation (GGA), and local successive convex approximation (LSCA).
By subtly designingthese three approximations, our algorithm is guaranteed to converge to the set of stationary points.
Notice that the second challenge also arises in the inner problem but we solve it in a simpler waybecause the inner problem is convex.
To approximate the local gradient ∇ F [ v ] ( ω c , η ∗ ( ω c )) , we propose the following LGA of learnerv :
[Equation]
; ωc S,A [ (cid:80) ∞ t =0 γ t ϕ c ( S t , A t )] , whosederivation is based on the strict convexity of G ( η ; ω c ) and G [ v ] ( η ; ω c ) .
The derivation and the proofof strict convexity are both included in the Appendix.
Moreover, it can be shown (in the Appendix)that the approximation error
|| ¯ ∇ F [ v ] ( ω c , ¯ η ( ω c )) −∇ F [ v ] ( ω c , η ∗ ( ω c )) || is upper bounded and reducesto zero as K → ∞ .
While learner v can use LGA to get a sound approximation of ∇ F [ v ] , in order to solve problem (3), it still needs the knowledge of the gradients (cid:80) v ′ ̸ = v ∇ F [ v ′ ] of other learners to getthe global gradient.
Therefore, we leverage a GGA ¯ ∇ [ v ] (lines 5 - 10 ) who aims to track theaverage global gradient 1 N L ∇ F ( ω [ v ] c , η ∗ ( ω [ v ] c )) .
However, ∇ F ( ω [ v ] c , η ∗ ( ω [ v ] c )) is not availabledue to the inaccessibility of η ∗ ( ω [ v ] c ) .
Therefore, ¯ ∇ [ v ] is designed to track the approximation
[Equation]
of the average global gradient.
To tackle the non-convexity issue, learner v solves a LSCA problem (line 11 ) and then followstwo-step updates of the local cost weight vector ω [ v ] c .
LSCA proposes to solve a local convexification of problem (3).
In specific, learner v solves the problem arg max ω c ∈ Ω c ˜ F [ v ] ( ω c ; ω
[ v ] c ) where− ˜ F
[ v ] ( ω c ; ω [ v ] c ) is a local strongly-convex surrogate of − F ( ω c , η ∗ ( ω c )) at ω [ v ] c satisfying the gradient consistency condition ∇ ˜ F [ v ] ( ω c ; ω [ v ] c )
=
N L ¯ ∇ [ v ] .
An attractive property of LSCA [27] isthat its fixed point exists and is a stationary point of functions with gradient N L ¯ ∇ [ v ] under certainconditions (in the Appendix).
Once obtaining ˜ ω
[ v ] c , learner v executes two-step updates.
The first update towards ˜ ω
[ v ] c (line 12 )drives learner v to a stationary point of problem (3) and the second update (lines 13 - 14 ) encouragesthe consensus among different learners.
Lemma 2.
The result ˜ ω
[ v ] c ( n ) at line 11 in Algorithm 1 is the optimal solution of a LSCA problemat ω [ v ] c ( n ) under the GGA N L ¯ ∇ [ v ] ( n ) .
Lemma 3.
(Convergence rate of the inner problem) Suppose Assumptions 1 and 2 hold and letα ( k ) =
¯ αk +1 where ¯ α is a positive constant.
Then, for every learner v ∈ V in Algorithm 2,
[Equation]
where C [ v ] 1 and C [ v ] 2 are positive constants whose expression can be found in the Appendix.
We define the set of KKT points (also called stationary points) of problem (3)-(4) as [28] [29]: Ω ∗ c ≜{ ω ∗ c ∈ Ω c : max ω c ∈ Ω c { ( ∇ F ( ω ∗ c , η ∗ ( ω ∗ c )))
⊤ ( ω c − ω ∗ c ) } = 0 } , and study the convergence propertyof the following metric [30]: J ( ω c ( n )) ≜ max ω c ∈ Ω c { ( ∇ F ( ω c ( n ) , η ∗ ( ω c ( n ))))
⊤ ( ω c − ω c ( n )) } .
Theorem 1.
(Asymptotic convergence of the outer problem) Suppose Assumptions 1, 2, and 3 hold.
Let β ( n ) ∈ (0 , 1) , (cid:80) ∞ n =0 β ( n ) = + ∞ , and (cid:80) ∞ n =0 ( β ( n )) 2 < + ∞ , then in Algorithm 1:( consensus ) : lim n →∞ max v,v ′ ∈V || ω
[ v ] c ( n ) − ω
[ v ′ ]
c ( n ) ||
[Equation]
0 , ( convergence ) : lim
[Equation]
where ¯ M is a positive constant whose existence is proved in the Appendix.
Lemma 3 and Theorem 1 show that the distributed learners can achieve common values of ω c and η .
As the constrained soft Bellman policy is continuous in ( ω c , η ) (proved in the Appendix), they willalso achieve a common policy.
Simulation
This section presents two simulation examples.
In the first example, the experts are programmed tofollow the optimal policy under discrete state and action spaces.
In the second example, the expertsare humans, i.e., may not be optimal, and the states and actions are both continuous.
6.1 Synthetic grid world
We consider the grid world introduced in [11].
Different from paper [11]’s
MDP, we implement a CMG on this environment where three experts perform motion planning from s 0 /s ′
/s ′′ 0to s G /s
G /s ′′ G while avoiding collision with obstacles and each other.
The environment consists ofa 9 -by- 9 grid of states, and the actions of each expert are to stay still, move up, down, left, right, ordiagonally by one cell.
The dynamics is stochastic as each action of an expert has 20% probability of failure, resulting in staying still.
While the grid world looks small, the CMG has more than387 , 000 , 000 state-action pairs due to multiple experts.
Each state-action pair produces a distancerelevant reward feature and the ground truth reward of each expert increases if the distance to itsgoal decreases.
The true CMG (Figure 1a), from which the experts generate demonstration, includes constraints (i.e., obstacles) and the nominal MG does not.
We use heat maps to show therelevant visitation frequency of each state and action pair.
Each expert will terminate its motion ifviolating a constraint, i.e, colliding with an obstacle.
In our algorithm, there are four distributed learners where each learner can only respectively get10 , 20 , 30 , 40 demonstrated trajectories of the experts.
Notice that the learners and experts arenot the same entities, i.e., the learners stand aside observing the experts.
The four learners jointlyrecover the experts’ policies by learning the constraints and the reward functions over an underlyingcommunication network.
The detailed simulation setup is included in the Appendix.
Furthermore, we use the following three baselines for comparisons:
ME-greedy : This method is introduced in
where a
greedy method is used to
estimate
theconstraints
based on the
ME framework.
MCE-greedy : We construct this method by extending ME-greedy to the same greedy method based
on the MCE framework which is more suitable for stochastic
environments.
Centralized inverse constrained inverse reinforcement learning (C-ICRL)
: This method is aspecial case of D-ICRL where there is a centralized learner obtaining all the demonstration data.
Notice that ME-greedy and MCE-greedy assume the access to the ground truth rewards while CICRL does not have this assumption.
All the three baselines are centralized learning where a singlelearner can get the total 100 demonstrated trajectories.
Our algorithm is distributed learning and doesnot obtain the ground truth rewards, therefore, it solves a more difficult problem than the baselinesdo.
However, the simulation shows that our algorithm can be on par with and even outperform thebaselines in some aspects.
In our bi-level algorithm, the outer loop is to estimate the constraintsand the inner loop is to recover the corresponding policy by estimating the reward function.
Toreason about the performance of our algorithm, we use five metrics: cumulative rewards (CR),false positive rate (FPR), false negative rate (FNR), constraint violation rate (CVR), and rewardfunction distance (RFD).
The CR, commonly used in IRL and imitation learning literature [31, 32],illustrates the total reward collected by the three experts in an episode, FPR, introduced in [11], isthe proportion of learned constraints that are not the ground truth constraints, FNR is the proportion of the ground truth constraints that are not learned, CVR [12] is the average percentage of the threeexperts’ violating any constraint in an episode.
Since the reward function is a linear combinationof the reward features, we propose RFD to measure the distance between the learned reward
weightvector ω r and experts’ reward weight vector ω r E : || ω r − ω r E
|| / || ω
r E || .
[Figure]
Figure 1b shows the constraints learned by our algorithm (each learner recovers the same con- straints).
Several action constraints are not recovered due to the reason that the experts will barelytake these actions even if they are feasible.
Figures 1c 1f show that our learning algorithm can successfully imitate the experts’ behavior according to CR and CVR.
Figures 1d 1e show that D-ICRLand C-ICRL have better FPR performance compared to the greedy-based methods while MCEgreedy has the best FNR performance.
From Figure 1e and the small window in Figure 1c, we cansee that the distributed learners can quickly achieve consensus even if they start from different initialconditions in both outer and inner loops.
Figures 1c and 1g show the performance of inner processin the last outer iteration.
Figure 1g shows that D-ICRL and C-ICRL can approach the ground truthreward functions and stabilize
[Table]
Table 1 shows the comparison result where each learner in our distributed algorithm can be on parwith C-ICRL in all the five metrics.
Our algorithm can outperform ME-greedy introduced in [11] in almost every metric except being on par with it in CVR.
While MCE-greedy is centralized learningand assumes the access to the ground truth rewards, our method can still outperform it in threemetrics, i.e., CR, FPR, and CVR.
6.2 Drones motion planning with obstacles
In the second example, we analyze our algorithm in a physical setting with continuous state andaction spaces.
We build a simulator in gazebo (Figure 2a) where each drone aims to reach the doorin its diagonal direction while avoiding collisions.
We (humans) first control the simulated drones totheir target doors, record nine pairs of trajectories, and distribute the demonstration to two learners.
The demonstration is shown in Figure 2b and each cell in the figure represents the constraint setindicated by a cost feature of each expert, i.e., C
[ i ] j .
We define the reward feature of each state as itslocation multiplied with a coefficient.
Figure 2c shows the learned constraints and trajectories.
[Figure]
We keep the three metrics, FPR, FNR, and CVR in last example.
Because the ground truth rewardis not accessible in this physical setting, we cannot use RFD and we use success rate (SR) insteadof CR.
The SR is defined as the percentage of the drones’ successfully reaching the doors withoutany collision.
We use the reward function learned from D-ICRL as the input reward function of
ME-greedy.
Because this environment is deterministic, MCE-greedy reduces to ME-greedy.
[Table]
Discussion and future work
We propose D-ICRL, the first multi-expert-multi-learner MCE ICRL framework that is effective to
CMG with both discrete and continuous environments.
We employ MLE to estimate the constraints,use an MCE-based optimization problem to learn the corresponding reward function and policy,and derive our algorithm based on a novel distributed bi-level optimization theoretical framework.
Experimental results show that D-ICRL can imitate the experts’ behavior as well as recover theenvironmental constraints.
Despite its benefits, one drawback of our approach is that the learnersassume that the reward/cost function is a linear combination of the reward/cost feature vector.
In thefuture work, we will investigate approaches that can relax this assumption.