Distributed Inverse Constrained Reinforcement Learning for Multi-agent Systems
Abstract
This paper considers the problem of recovering the policies of multiple interacting experts by estimating their reward functions and constraints where the demonstration data of the experts is distributed to a group of learners.
We formulate this problem as a distributed bi-level optimization problem and propose a novel bi-level “distributed inverse constrained reinforcement learning” (D-ICRL) algorithm that allows the learners to collaboratively estimate the constraints in the outer loop and learn the corresponding policies and reward functions in the inner loop from the distributed demonstrations through intermittent communications.
We formally guarantee that the distributed learners asymptotically achieve consensus which belongs to the set of stationary points of the bi-level optimization problem.
Simulations are done to validate the proposed algorithm.
Introduction
Multi-agent systems (MASs) have become an effective way to model large-scale networked systems, e g. , multi-robot systems and the Internet-of-Things.
Multi-agent reinforcement learning (MARL) is a systematic framework to coordinate MASs.
It extends RL to address the decision-making problem of multiple interacting agents [1].
One challenge in (MA)RL is that manually crafting the reward functions or predesigning the policies can be infeasible for humans when the tasks are too complicated.
Inverse reinforcement learning (IRL) presents a way to tackle this problem where a learner aims to recover a reward function (and a corresponding policy) that best explains the behavior in the demonstrations of an expert.
Current state-of-the-arts on IRL [2, 3, 4, 5] can successfully solve the reward function inference problem but still face two other major challenges in MARL and MASs: (i) it is often the case that the expected behavior (experts’ behavior) is more precisely explained by a reward function combined with a set of constraints instead of a single reward function; (ii) the demonstration data in MASs is usually distributed over multiple learners and unable to be shared due to some practical reasons including privacy concerns and communication burdens.
This paper aims to bridge the gap and proposes the first algorithm to solve the three challenges (i.e., reward function inference, constraint inference, and distributed demonstration data) at once.
Related works .
IRL is an ambiguous problem [6] as different reward functions could explain the demonstrations.
To solve this problem, several fundamental approaches are proposed including feature expectation matching [7], maximum margin planning [8], maximum entropy (ME) [9], and maximum causal entropy (MCE) [10].
Recently, some advanced techniques baring more features are proposed.
Paper [2] uses adversarial learning to learn reward functions robust to environmental changes, paper [4] applies strongly convex regularizers to the learner’s policy to avoid the ambiguity problem, and paper [5] proposes a scalable Bayesian-based method working for large control tasks.
In real-world settings, numerous constraints are present and a single reward function is not enough to explain complicated behavior.
Paper [11] extends ME IRL to environments with constraints, formulates the constraint inference problem as a maximum likelihood estimation (MLE) problem, and leverages a greedy method to solve the problem.
Paper [12] extends paper [11] to continuous environments.
These two works both assume the access to the ground truth rewards.
While the discussions in the last two paragraphs are limited to single agents, there have been recent works which extend IRL techniques to multiple agents [13, 14, 15, 16, 17].
Notice that previous works on multi-agent IRL do not explicitly distinguish the notions of “experts” and “learners”, their “multiple agents” means “multiple experts” and they only consider centralized learning ( i.e. , a single learner who obtains all demonstrated behavior of the multiple experts).
In this paper, we explicitly distinguish the notions of “experts” and “learners”, and use “multiple experts” to represent the commonly used “multiple agents” in literature.
Moreover, the aforementioned papers do not consider the more general situation where the demonstrations are distributed over multiple learners and unable to be shared.
To handle this situation, the learners need to perform distributed learning.
As a basic topology in distributed learning, peer-to-peer (P 2 P) architecture has no central server and each node can communicate with a subset of all the nodes.
It reduces the risk of single points of failures [18] in the system.
This paper proposes D-ICRL based on the P 2 P structure where ICRL can learn policies by estimating the reward functions and constraints through demonstrations and distributed learners can perform ICRL without sharing the distributed demonstration data.
In specific, we formulate D-ICRL as a distributed bi-level optimization problem where the learners cooperatively learn the constraints through the outer problem and estimate the corresponding reward functions and policies through the inner problem.
Paper [19] provides asymptotic convergence for centralized bi-level optimization problems by assuming that the inner problem can be completely solved and its optimal solution can be obtained.
Papers [20, 21, 22] relax this assumption by partially solving the inner problem and provide convergence rate analysis for problems with strongly convex inner objective function.
Paper [23] solves a special case of distributed bi-level optimization where the outer and inner problems use identical decision variables.
To the best of our knowledge, there is no work on bi-level optimization which (i) assumes that the inner objective function is merely strictly convex and partially solves the inner problem; (ii) solves the general form of bi-level optimization in a distributed way.
Notice that these two challenges arise in our D-ICRL problem.
Contribution statement .
Our contributions are threefold.
First, we extend IRL to a multi-expertmulti-learner (MEML) setting where a group of learners collaboratively learn policies by estimating the reward functions and constraints without sharing the distributed demonstrations performed by a set of cooperative experts.
We formulate this D-ICRL problem as a distributed bi-level optimization problem.
Second, we propose a distributed bi-level learning algorithm to solve this problem where the learners cooperatively estimate the constraints in the outer problem and learn the corresponding reward functions and policies in the inner problem.
Third, we provide convergence rate analysis to the optimal solution of the inner problem and asymptotic convergence to the set of stationary points of the outer problem.
Simulations are conducted to validate the proposed method.
Model
In this section, we present the models for cooperative experts and distributed learners.
Experts .
There are N E experts whose decision making is based on a constrained Markov game (CMG) [24].
A CMG ( S , A , γ, P 0 , P, r E , c E , b ) is defined via state set S ≜ (cid:81) N E i =1 S
[ i ] , action set A ≜ (cid:81) N E i =1 A [ i ] , discount factor γ ∈ (0 , 1) , and initial state distribution P 0 .
The state-action space can be either discrete or continuous.
The system state transition function is given by P such that the probability (or probability density in continuous state-action space) of state transition to s ′ from s by taking action a ≜ ( a [1] , · · · , a [ N E ] ) is P ( s ′
| s, a ) .
The reward function of expert i is r
[ i ] E : S ×
A → R and the experts are cooperative ( i.e. , r E ≜ (cid:80) N E i =1 r [ i ] E ).
The cost function of expert i is c
[ i ] E : S ×
A → R + and the cost function of the whole system is c E ≜ (cid:80) N E i
=1 c [ i ] E .
Expert i ’s policy π
[ i ] E ( a [ i ] | s ) represents the probability (or probability density) of expert i taking action a [ i ] at state s and the joint policy of all the experts is denoted by π E ( a | s ) ≜ (cid:81) N E i =
1 π [ i ] E ( a [ i ] | s ) .
Define J r E ( π ) ≜ E πS,A [ (cid:80) ∞ t =
γ t r E ( S t , A t )] as the expected cumulative reward where the initial state is drawn from P 0 and J c E ( π ) ≜ E πS,A
[ (cid:80) ∞ t =
γ t c E ( S t , A t )] as the expected cumulative cost.
The experts’ policy π E wants to maximize J r E ( π ) subject to J c E ( π ) ≤ b .
Define the constraint set indicated by c [ i ] E as C [ i ] E ≜ { ( s, a ) ∈ S
× A : c [ i ] E ( s, a ) > 0 } .
Following [12], we study the case where b = 0 , i.e., hard constraint.
Thus, the constraint reduces to J c E ( π ) = 0 , implying that the probability of reaching any ( s, a ) ∈ (cid:83) i C
[ i ] E is zero under policy π .
These experts use π E to demonstrate a set D ≜ { ζ j } mj =1 of m trajectories, which is partitioned into N L subsets.
Each trajectory ζ j ≜ s j 0 , a j 0 , s j 1 , a j 1 · · · is a state-action sequence of the experts.
Learners .
There is a group of N L learners where each learner v knows ( S , A , γ, P 0 , P, b, D [ v ] ) and can communicate with other learners.
The local demonstration subset D [ v ] has m [ v ] trajectories.
The learners choose an l [ i ] r -dimensional reward feature vector ϕ
[ i ] r : S ×A → [0 , d 1 ] l [ i ] r for expert i where d 1 is constant and it is assumed that r [ i ] E =
( ω [ i ] r E )
⊤ ϕ
[ i ] r , where ω [ i ] r E ∈ R l [ i ] r is the (unknown) reward weight vector.
The learners also choose an l [ i ] c -dimensional cost feature vector ϕ
[ i ] c : S ×
A → [0 , d 2 ] l [ i ] c where d 2 is constant and it is assumed that c
[ i ] E = ( ω
[ i ] c E ) ⊤ ϕ
[ i ] c , where ω
[ i ] c E is the (unknown) cost weight vector and ω
[ i ] c E ,j is the j -th component of ω
[ i ] c E .
Let C
[ i ] j ≜ { ( s, a ) ∈ S×A : ϕ
[ i ] c,j ( s, a ) > 0 } be the constraint set indicated by the j -th cost feature ϕ
[ i ] c,j of expert i .
As the budget b = 0 , the effect of ω [ i ] c E ,j > 0 is the same, i.e., C
[ i ] j cannot be visited, thus the learners assume ω
[ i ] c E ,j ∈ [0 , 1] .
Communication network .
The learners execute a two-time-scale learning algorithm over a communication network.
The fast-time-scale (inner) algorithm is executed at discrete time k
over a time-varying directed graph G ( k ) ≜ ( V , E ( k )) where V ≜ { 1 , · · · , N L } is the node (learner) set and
E ( k ) ⊆
V × V is the set of directed edges (communication links) at time k .
The edge ( v, v ′ )
∈ E ( k ) means that learner v receives information from learner v ′ at time k
and ( v, v ) ∈ E ( k ) for all k ≥ 0 .
The adjacency matrix of the graph at time k is W ( k ) ≜
[ W [ vv ′ ]
( k )] v,v ′ ∈V ∈ R N L × N L where
W [ vv ′ ] ( k ) =
if and only if ( v, v ′ ) / ∈ E ( k ) .
We use N [ v ] ( k ) ≜ { v ′ ∈ V| ( v, v ′ )
∈ E ( k ) } to denote the set of neighbors of learner v at time k .
The slow-time-scale (outer) algorithm is carried out at discrete time n over a graph ¯ G ( n ) ≜ ( V , ¯ E ( n )) , where ¯ E ( n ) , ¯ W ( n ) and ¯ N [ v ] ( n ) are defined in an analogous way.
The time unit for n is much larger than that for k .
Assumption 1.
There exists an integer B ≥ 1 such that the graph ( V , E ( k ) ∪ · · · ∪ E ( k + B − 1)) is strongly connected for all k ≥ 0 .
Assumption 2.
The adjacency matrix W ( k ) has the following properties: (i) (doubly stochastic) 1 ⊤ W ( k ) = 1 ⊤ and W ( k ) 1 = 1 where 1 is the column vector whose entries are all ones.
(ii) (non-degenerate)
There is an ϵ ∈ (0 , 1) such that W [ vv ] ( k ) ≥ ϵ for all v ∈ V and
W [ vv ′ ] ( k ) ≥ ϵ
if ( v, v ′ ) ∈ E ( k ) .
Assumption 3.
The set ¯ N [ v ] ( n ) =
V for any v ∈ V and n ≥ 0 .
The adjacency matrix ¯ W ( n ) is doubly stochastic and non-degenerate with constant ¯ ϵ .
Figure 1 provides an illustration of the relation between the experts and the learners.
The experts operate in a multi-agent system and demonstrate a set D of trajectories and the learners stand outside the system where each learner v can observe a subset D [ v ] of the global demonstration.
Notions and notations
This section provides notions and notations which will be used in the remaining of the paper.
Reward feature expectation matching (RFEM) .
Given policy π , the reward feature expectation vector of expert i is defined as µ [ i ] r ( π ) ≜ E πS,A
[ (cid:80) ∞ t =0 γ t ϕ [ i ] r ( S t , A t )] and the demonstration D is needed to estimate µ [ i ] r ( π E ) :
ˆ µ [ i ] r ≜ 1 m (cid:80) mj =1 (cid:80) ∞
t =0 γ t ϕ
[ i ] r ( s jt , a jt ) , where ( s jt , a jt ) ∈ ζ j .
The RFEM requires µ [ i ] r ( π ) =
ˆ µ [ i ] r for every expert i .
However, each learner v can only observe the subset D [ v ] and its local estimate is ˆ µ [ i,v ] r
[Equation]
Cost expectation matching (CEM) .
Similar to RFEM, CEM requires J c E ( π ) =
J c E ( π E ) .
As the true cost function c E is unknown, given an estimate c ω c = (cid:80) N E i =
( ω [ i ] c )
⊤ ϕ
[ i ] c , we define the empirical cost estimate of the experts as ˆ b ω c ≜ 1 m (cid:80) mj
=1 (cid:80) ∞ t =
γ t c ω c ( s jt , a jt ) .
The local estimate from learner v is ˆ b [ v ] ω c ≜ 1 m [ v ] (cid:80) ζ j ∈D [ v ] (cid:80) ∞
t
=
γ t c ω c ( s jt , a jt ) .
Remark 1.
As RFEM is a standard notion in IRL literature, it is “natural” to follow this and use “cost feature expectation matching” (CFEM).
However, here we use CEM instead of CFEM because we want to leverage the budget information provided in CMG.
We still use RFEM instead of “reward expectation matching” (REM) because REM does not embed any additional useful information as
CMG does not have any “budget information” for cumulative reward.
Notations .
We use ˆ µ r ≜ [(ˆ µ [1] r ) ⊤ , · · · , (ˆ µ [ N E ] r ) ⊤ ] ⊤ to represent the concatenated empirical reward feature expectation vector.
Similarly, we define learner v ’s concatenated empirical reward feature expectation vector estimate ˆ µ [ v ] r , concatenated reward feature expectation vector µ r ( π ) under π , and concatenated cost feature vector ϕ c .
Let Π be the set of all valid stochastic policies such that every π ∈ Π satisfies π ( a | s ) ≥ 0 for any s ∈ S and a ∈ A and (cid:82) a ∈A π ( a | s ) da
= 1 for any s ∈ S .
We define the expected cumulative cost of policy π under cost function estimate c ω c as
J ω c ( π ) ≜ E πS,A [ (cid:80) ∞ t =
γ t c ω c ( S t , A t )] and use Project Ω c ( · ) to denote the projection onto the set Ω c .
Problem formulation
In an MEML D-ICRL problem, each learner v aims to use all the available information to collaboratively recover the policy π E through communications by estimating the reward function r E and the cost function c E .
In the learning process, learner v is unwilling to share its private data ( i.e. , local demonstration and local estimate of reward feature expectation and cost expectation).
Notice that estimating c E is in essence estimating the constraint set (cid:83)
i C
[ i ] E .
We formulate the constraint learning problem as a distributed MLE problem:
[Equation] [Equation]
where Ω c ≜ [0 , 1] (cid:80) NEi =1 l [ i ] c , F [ v ] ( ω c ) ≜ (cid:80) j ∈D [ v ] (cid:80) ∞
t =
γ t ln π ω c ( a jt | s jt ) is learner v ’s local likelihood function and π ω c is a policy parameterized by ω c .
To find a statistical distribution model for π ω c , we formulate the following problem based on MCE scheme:
[Equation]
where H ( π ) ≜ (cid:80) ∞
t =0 E S,A [ − γ t ln π ( A t | S t )] is the (infinite-horizon) causal entropy [25].
Remark 2.
Directly matching the ground truth budget (i.e., J ω c ( π ) =
b ) may cause infeasibility of (2), i.e., the feasible set of (2) is not guaranteed to be nonempty.
Therefore, we match ˆ b ω c , which is an estimate of J ω c ( π E ) , to ensure the feasibility because now, π E is a feasible solution of (2).
The dual function of (2) is G ( ω r , λ ; ω c ) ≜
max π ∈ Π H ( π )+ (cid:80) N E i =1 ( ω [ i ] r ) ⊤ ( µ [ i ] r ( π ) − ˆ µ [ i ] r )+ λ ( J ω c ( π ) − ˆ b ω c ) where the dual variable ω r =
[( ω [1] r ) ⊤ , · · · , ( ω [ N E ] r ) ⊤ ] ⊤ is to estimate ω r E .
Let η ≜ [ ω ⊤ r , λ ] ⊤ , it is well-known that G ( η ; ω c ) is convex in η since it is the pointwise maximum of a family of affine functions of η .
As G ( η ; ω c ) is global, we introduce the local convex dual function
G [ v ] ( η ; ω
[Equation] [Equation]
,v ] r ) + λ ( J ω c ( π ) − ˆ b [ v ] ω c ) .
Lemma 1.
(i)
The optimal solution of problem (2) is the constrained soft Bellman policy π η ∗ ( ω c ); ω c , where η ∗ ( ω c ) is an optimal solution of min η G ( η ; ω c ) .
(ii)
The local dual function G [ v ] ( η ; ω c ) is differentiable and its gradient is [( µ r ( π η ; ω c ) − ˆ µ [ v ] r ) ⊤ , J ω c ( π η ; ω c ) − ˆ b [ v ] ω c ] ⊤ .
The proof of Lemma 1 is similar to [10, 25, 26].
For the sake of completeness, we still include the proof and the expression of the constrained soft Bellman policy in the Appendix.
Lemma 1 indicates that π η ∗ ( ω c ); ω
c is the statistical model we need in problem (1).
Therefore, problem (1) can be reformulated as the following distributed bi-level optimization problem:
[Equation]
where F [ v ] and m [ v ] G [ v ] are known only to learner v .
Remark 3.
Problem (4) and arg min η G ( η ; ω c ) are equivalent because (cid:80) N L v =1 m [ v ] G [ v ] ( η ; ω c )
= (cid:80) N L v =1 m [ v ] H ( π η ; ω c ) + (cid:80) N L v =1 m [ v ] (cid:80) N E i =1 ( ω
[ i ] r ) ⊤ ( µ [ i ] r ( π η ; ω c ) − ˆ µ [ i,v ] r )
[Equation]
λ (cid:80) N L
[Equation]
m
[Equation] [Equation]
J
[Equation]
c
[Equation]
π η
[Equation]
ω c ) − ˆ b
[Equation]
v ] ω
[Equation]
Algorithm and convergence guarantee
In this section, we develop a bi-level distributed learning algorithm to solve problem (3)-(4) and provide convergence rate to the optimal solution of the inner problem (4) and asymptotic convergence to the set of stationary points of the outer problem (3).
Input : { ω [ v ] c (0) } N L v =1 , { η [ v ] (0) } N L v =1 , W ( k ) , ¯ W ( n )
Output : ω [ v ] c ( n ) , ¯ η [ v ] ( ω [ v ] c ( n )) , π ¯ η [ v ] ( ω [ v ] c ( n )); ω [ v ] c ( n ) ∀ v ∈ V 1: for n = 0 , 1 , · · · do 2: for v ∈ V do 3:
Receives ω
[ v ′ ]
c ( n ) from v ′ ∈ ¯ N [ v ] ( n ) 4: ¯ η [ v ] ( ω [ v ] c ( n )) , π ¯ η [ v ] ( ω [ v ] c ( n )); ω [ v ] c ( n ) =
Inner process ( ω [ v ] c ( n )) 5: if n = 0 then 6: ¯ ∇ [ v ] (0) =
¯ ∇ F [ v ] ( ω [ v ] c (0) , ¯ η
[ v ] ( ω [ v ] c (0))) 7: else 8:
Receives ¯ ∇ [ v ′ ] ( n − 1) from v ′ ∈ ¯ N [ v ] ( n ) 9: ¯ ∇ [ v ] ( n ) =
(cid:80) N L v ′
=1 ¯ W [ vv ′ ] ( n ) ¯ ∇ [ v ′ ] ( n − 1) + ¯ ∇ F [ v ] ( ω [ v ] c ( n ) , ¯ η [ v ] ( ω [ v ] c ( n )))
− ¯ ∇ F [ v ] ( ω [ v ] c ( n − 1) , ¯ η [ v ] ( ω [ v ] c ( n − 1)))
10: end if 11: ˜ ω
[ v ] c ( n ) = Project
[Equation]
n
[Equation]
12: ω
[Equation] [Equation]
=
[Equation]
n ))
Receives ω
[ v ′ ]
c ( n + 1 / 2) from v ′ ∈ ¯ N [ v ] ( n ) 14: ω [
[Equation]
( n
[Equation]
= (cid:80)
[Equation]
) 15: end for 16: end for
At outer iteration n in Algorithm 1, the learners first cooperatively solve the inner problem for each learner through Inner process (lines 3 - 4 ) and then use the obtained result ¯ η [ v ] ( ω [ v ] c ( n )) to collaboratively solve the outer problem (3) through the outer process (lines 5 - 14 ).
In what follows, we will elaborate each process.
Algorithm
Input : ω [ v ] c , { η [ v ] (0) } N L v =1 , W ( k )
Output : ¯ η [ v ] ( ω [ v ] c ) , π ¯ η [ v ] ( ω [ v ] c ); ω [ v ] c 1: for k = 0 , 1 , · · · , K − 2 do 2: for ¯ v ∈ V do 3:
Receives η
[¯ v ′ ]
( ω [ v ] c , k ) from ¯ v ′ ∈
N
[¯ v ] ( k ) 4: η
[¯ v ] ( ω [ v ] c , k + 1)
= (cid:80) N L v ′
=1 W [¯ v ¯ v ′ ]
( k ) η
[¯ v ′ ]
( ω [ v ] c , k ) − α ( k ) ∇ η G [¯ v ] ( η [¯ v ] ( ω [ v ] c , k ); ω [ v ] c ) 5: end for 6: end for 7: ¯ η [ v ] ( ω [ v ] c ) ≜ (
cid:80) K − 1 k =
α ( k ) η [ v ] ( ω [ v ] c ,k ) (cid:80) K −k =
α ( k ) 8: π ¯ η [ v ] ( ω [ v ] c ); ω [ v ] c is the corresponding constrained soft Bellman policy.
Inner process .
As ¯ N [ v ] ( n ) =
V , learner v knows the cost weight vector of all the learners.
Given ω [ v ] c , at inner iteration k , each learner ¯ v receives η
[¯ v ′ ] ( ω [ v ] c , k ) from neighbors and updates its estimate of the reward weight vector (embedded in η [¯ v ] ).
In specific, at line 4 in Algorithm 2, the first term (convex combination) on the right hand side encourages consensus and the second term (gradient) drives to the optimal solution.
It is shown in Lemma 3 that each learner can reach η ∗ ( ω [ v ] c ) at the rate of O ( 1 √ log K ) .
Outer process .
Once obtaining ¯ η [ v ] ( ω [ v ] c ) , each learner v faces three challenges in solving the outer problem: (i) the gradient of F [ v ] ( ω [ v ] c , η ∗ ( ω [ v ] c )) cannot be obtained because the inner problem is not fully solved, i.e., learner v does not find η ∗ ( ω [ v ] c ) but its approximation ¯ η [ v ] ( ω [ v ] c ) , (ii) (cid:80) v ′ ̸ = v F [ v ′ ] is unknown, and (iii) the local likelihood function F [ v ] is non-convex.
To solve these challenges respectively, we propose three techniques: local gradient approximation (LGA), global gradient approximation (GGA), and local successive convex approximation (LSCA).
By subtly designing these three approximations, our algorithm is guaranteed to converge to the set of stationary points.
Notice that the second challenge also arises in the inner problem but we solve it in a simpler way because the inner problem is convex.
To approximate the local gradient ∇ F [ v ] ( ω c , η ∗ ( ω c )) , we propose the following LGA of learner v : ¯
[Equation]
F [
[Equation] [Equation]
c , ¯
[Equation] [Equation]
[Equation] [Equation]
t =
[Equation]
γ t ϕ c
[Equation]
S t , A t )] , whose derivation is based on the strict convexity of G ( η ; ω c ) and G [ v ] ( η ; ω c ) .
The derivation and the proof of strict convexity are both included in the Appendix.
Moreover, it can be shown (in the Appendix) that the approximation error (defined in the Appendix) is upper bounded and reduces to zero as
K → ∞ .
While learner v can use LGA to get a sound approximation of ∇ F [ v ] , in order to solve problem (3), it still needs the knowledge of the gradients (cid:80) v ′ ̸ = v ∇ F [ v ′ ] of other learners to get the global gradient.
Therefore, we leverage a GGA ¯ ∇ [ v ] (lines 5 - 10 ) who aims to track the average global gradient 1 N L ∇ F ( ω [ v ] c , η ∗ ( ω [ v ] c )) .
However, ∇ F ( ω [ v ] c , η ∗ ( ω [ v ] c )) is not available due to the inaccessibility of η ∗ ( ω [ v ] c ) .
Therefore, ¯ ∇ [ v ] is designed to track the approximation 1 N L (cid:80) N L v ′ =
1 ¯ ∇ F [ v ′ ]
( ω [ v ′ ]
c , ¯ η [ v ′ ] ( ω [ v ′ ] c )) of the average global gradient.
To tackle the non-convexity issue, learner v solves a LSCA problem (line 11 ) and then follows two-step updates of the local cost weight vector ω [ v ] c .
LSCA proposes to solve a local convexification of problem (3).
In specific, learner v solves the problem arg max ω c ∈ Ω c ˜ F [ v ] ( ω c ; ω
[ v ] c ) where − ˜ F [ v ] ( ω c ; ω [ v ] c ) is a local strongly-convex surrogate of − F ( ω c , η ∗ ( ω c )) at ω [ v ] c satisfying the gradient consistency condition ∇ ˜ F [ v ] ( ω c ; ω [ v ] c )
=
N L ¯ ∇ [ v ] .
An attractive property of LSCA [27] is that its fixed point exists and is a stationary point of functions with gradient N L ¯ ∇ [ v ] under certain conditions (in the Appendix).
Once obtaining ˜ ω
[ v ] c , learner v executes two-step updates.
The first update towards ˜ ω
[ v ] c (line 12 ) drives learner v to a stationary point of problem (3) and the second update (lines 13 - 14 ) encourages the consensus among different learners.
Lemma 2.
The result ˜ ω
[ v ] c ( n ) at line 11 in Algorithm 1 is the optimal solution of a LSCA problem at ω [ v ] c ( n ) under the GGA N L ¯ ∇ [ v ] ( n ) .
Lemma 3.
(Convergence rate of the inner problem) Suppose Assumptions 1 and 2 hold and let α ( k ) =
¯ αk +1 where ¯ α is a positive constant.
Then, for every learner v ∈ V in Algorithm 2,
[Equation]
where C [ v ] 1 and C [ v ] 2 are positive constants whose expression can be found in the Appendix.
We define the set of KKT points (also called stationary points) of problem (3)-(4) as [28] [29]: Ω ∗ c ≜ { ω ∗ c ∈ Ω c : max ω c ∈ Ω c { ( ∇ F ( ω ∗ c , η ∗ ( ω ∗ c )))
⊤ ( ω c − ω ∗ c ) } = 0 } , and study the convergence property of the following metric [30]: J ( ω c ( n )) ≜ max ω c ∈ Ω c { ( ∇ F ( ω c ( n ) , η ∗ ( ω c ( n ))))
⊤ ( ω c − ω c ( n )) } .
Theorem 1.
(Asymptotic convergence of the outer problem) Suppose Assumptions 1, 2, and 3 hold.
Let β ( n ) ∈ (0 , 1) , (cid:80) ∞ n =0 β ( n ) = + ∞ , and (cid:80) ∞ n =0 ( β ( n )) 2 < + ∞ , then in Algorithm 1: ( consensus ) : lim
[Equation]
where ¯ M is a positive constant whose existence is proved in the Appendix.
Lemma 3 and Theorem 1 show that the distributed learners can achieve common values of ω c and η .
As the constrained soft Bellman policy is continuous in ( ω c , η ) (proved in the Appendix), they will also achieve a common policy.
Simulation
This section presents two simulation examples.
In the first example, the experts are programmed to follow the optimal policy under discrete state and action spaces.
In the second example, the experts are humans, i.e., may not be optimal, and the states and actions are both continuous.
6.1 Synthetic grid world
We consider the grid world introduced in [11].
Different from paper [11]’s
MDP, we implement a CMG on this environment where three experts perform motion planning from s 0 /s ′
/s ′′to s G /s ′
G /s ′′ G while avoiding collision with obstacles and each other.
The environment consists of a 9 -by- 9 grid of states, and the actions of each expert are to stay still, move up, down, left, right, or diagonally by one cell.
The dynamics is stochastic as each action of an expert has 20% probability of failure, resulting in staying still.
While the grid world looks small, the CMG has more than 387 , 000 , 000 state-action pairs due to multiple experts.
Each state-action pair produces a distancerelevant reward feature and the ground truth reward of each expert increases if the distance to its goal decreases.
The true CMG (Figure 2a), from which the experts generate demonstration, includes constraints (i.e., obstacles) and the nominal MG does not.
We use heat maps to show the relevant visitation frequency of each state and action pair.
Each expert will terminate its motion if violating a constraint, i.e, colliding with an obstacle.
In our algorithm, there are four distributed learners where each learner can only respectively get 10 , 20 , 30 , 40 demonstrated trajectories of the experts.
Notice that the learners and experts are not the same entities, i.e., the learners stand aside observing the experts.
The four learners jointly recover the experts’ policies by learning the constraints and the reward functions over an underlying communication network.
The detailed simulation setup is included in the Appendix.
Furthermore, we use the following three baselines for comparisons:
ME-greedy : This method is introduced in [11] where a greedy method is used to estimate the constraints based on the ME framework.
MCE-greedy : We construct this method by extending ME-greedy to the same greedy method based on the MCE framework which is more suitable for stochastic environments.
Centralized inverse constrained inverse reinforcement learning (C-ICRL)
: This method is a special case of D-ICRL where there is a centralized learner obtaining all the demonstration data.
Notice that ME-greedy and MCE-greedy assume the access to the ground truth rewards while CICRL does not have this assumption.
All the three baselines are centralized learning where a single learner can get the total 100 demonstrated trajectories.
Our algorithm is distributed learning and does not obtain the ground truth rewards, therefore, it solves a more difficult problem than the baselines do.
However, the simulation shows that our algorithm can be on par with and even outperform the baselines in some aspects.
In our bi-level algorithm, the outer loop is to estimate the constraints and the inner loop is to recover the corresponding policy by estimating the reward function.
To reason about the performance of our algorithm, we use five metrics: cumulative rewards (CR), false positive rate (FPR), false negative rate (FNR), constraint violation rate (CVR), and reward function distance (RFD).
The CR, commonly used in IRL and imitation learning literature [31, 32], illustrates the total reward collected by the three experts in an episode, FPR, introduced in [11], is the proportion of learned constraints that are not the ground truth constraints, FNR is the proportion of the ground truth constraints that are not learned, CVR [12] is the average percentage of the three experts’ violating any constraint in an episode.
Since the reward function is a linear combination of the reward features, we propose RFD to measure the distance between the learned reward weight vector ω r and experts’ reward weight vector ω r E : || ω r − ω r E
|| / || ω
r E || .
[Figure]
Figure 2b shows the constraints learned by our algorithm (each learner recovers the same constraints).
Several action constraints are not recovered due to the reason that the experts will barely take these actions even if they are feasible.
Figures 2c 2f show that our learning algorithm can successfully imitate the experts’ behavior according to CR and CVR.
Figures 2d 2e show that D-ICRL and C-ICRL have better FPR performance compared to the greedy-based methods while MCEgreedy has the best FNR performance.
From Figure 2e and the small window in Figure 2c, we can see that the distributed learners can quickly achieve consensus even if they start from different initial conditions in both outer and inner loops.
Figures 2c and 2g show the performance of inner process in the last outer iteration.
Figure 2g shows that D-ICRL and C-ICRL can approach the ground truth reward functions and
[Table]
Table 1 shows the comparison result where each learner in our distributed algorithm can be on par with C-ICRL in all the five metrics.
Our algorithm can outperform ME-greedy introduced in [11] in almost every metric except being on par with it in CVR.
While MCE-greedy is centralized learning and assumes the access to the ground truth rewards, our method can still outperform it in three metrics, i.e., CR, FPR, and CVR.
6.2 Drones motion planning with obstacles
In the second example, we simulate in a physical setting with continuous state-action space.
We build a simulator in Gazebo where each drone aims to reach the door in its diagonal direction while avoiding collisions.
We first control the simulated drones to their target doors, record nine pairs of trajectories, and distribute four and five pairs to two learners respectively.
The demonstration is shown in Figure 3b and each cell in the figure represents the constraint set indicated by a cost feature of each expert, i.e., C
[ i ] j .
We define the reward feature of each state as its location multiplied with a coefficient.
Figure 3c shows the learned constraints and trajectories.
[Figure]
We keep the three metrics, FPR, FNR, and CVR in last example.
Because the ground truth reward is not accessible in this physical setting, we cannot use RFD and we use success rate (SR) instead of CR.
The SR is defined as the percentage of the drones’ successfully reaching the doors without any collision.
We use the reward function learned from D-ICRL as the input reward function of
ME-greedy.
Because this environment is deterministic, MCE-greedy reduces to ME-greedy.
[Table]
Discussion and future work
We propose D-ICRL, the first multi-expert-multi-learner MCE ICRL framework that is effective to
CMG with both discrete and continuous environments.
We employ MLE to estimate the constraints, use an MCE-based optimization problem to learn the corresponding reward function and policy, and derive our algorithm based on a novel distributed bi-level optimization theoretical framework.
Experimental results show that D-ICRL can imitate the experts’ behavior as well as recover the environmental constraints.
Despite its benefits, the limitations are that (i) the learners assume that the reward/cost function is a linear combination of some features; (ii) the communication between learners is frequent.
We will address the issues in the future.
Acknowledgements
This work is partially supported by National Science Foundation through grants CNS 1830390,
ECCS 1846706, and ECCS 2140175.
We would like to thank the reviewers for their insightful and constructive suggestions.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] We have the formal contribution statement in Section 1 .
(b) Did you describe the limitations of your work?
[Yes] We discuss the limitations in Section 7 .
(c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] We have the formal assumption statements in Section(b)
Did you include complete proofs of all theoretical results?
[Yes] We include all the proofs in the Appendix (Section 9 ) 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We include the simulation setups in Section 6 and also in the Appendix (Section 10 ).
We also include the code in the supplementary materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] We include some details in Section 6 and the rest details are included in the Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] In the first experiment (subsection 6 . 1 ), we include the mean and standard deviation of the cumulative reward.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] The compute type is included in the Appendix. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] The first experiment uses the environment in a reference and we cite it.
The second experiment is in Gazebo based on hector quadrotor.
(b) Did you mention the license of the assets?
[Yes] We include the licenses in the supplementary materials.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] In the second experiment, we create a new simulator in Gazebo based on hector quadrotor.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[No] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
We do not conduct research on human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
We do not conduct research on human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A] We do not conduct research on human subjects.
A Appendix: Notions and notations
We define the concatenated reward feature vector as ϕ r ≜ [( ϕ [1] r ) ⊤ , · · · , ( ϕ [ N E ] r ) ⊤ ] ⊤ and the augmented reward feature under ω c as ϕ r,ω c ≜ [ ϕ ⊤ r , c ω c ] ⊤ .
The augmented reward feature expectation starting from state-action pair ( s, a ) under policy π is µ πr,
ω c ( s, a ) ≜ ϕ r,ω c ( s, a ) + γ (cid:82) s ′ ∈S P
( s ′ | s, a ) µ πr,ω c ( s ′ )
ds ′ and the augmented reward feature expectation starting from state s is µ πr,ω c ( s ) ≜ (cid:82) a ∈A π ( a | s ) µ πr,ω c ( s, a ) da .
Moreover, we denote µ r,ω c ( π ) ≜ [( µ r ( π ))
⊤ , J ω c ( π )] ⊤ .
The empirical augmented reward feature expectation vector is defined as ˆ µ r,ω c ≜ [ˆ µ ⊤ r , ˆ b ω c ] ⊤ and its estimate from learner v is defined as ˆ µ [ v ] r,ω c ≜ [(ˆ µ [ v ] r ) ⊤ , ˆ b [ v ] ω c ] ⊤ .
For a given vector ¯ η , we define the augmented Q-function as Q π ¯ η,ω c ( s, a ) =
¯ η ⊤ µ πr,ω
c
( s, a ) and the augmented value-function as V π ¯ η,ω c ( s ) =
¯
η ⊤ µ πr,ω
c
( s ) .
The concatenated cost feature expectation starting from state-action pair ( s, a ) under policy π is µ πc ( s, a ) ≜
ϕ c ( s, a ) + γ (cid:82) s ′ ∈S P
s ′ | s, a ) µ πc ( s ′ )
ds ′ and the concatenated cost feature expectation starting from state s is µ πc ( s ) ≜ (cid:82) a ∈A π ( a | s ) µ πc ( s, a ) da .
We denote µ c ( π ) ≜ E πS,A
[ (cid:80) ∞ t =
γ t ϕ c ( S t , A t )] .
The empirical concatenated cost feature expectation is ˆ µ c ≜ 1 m (cid:80) mj
=1 (cid:80) ∞ t =
γ t ϕ c ( s jt , a jt ) .
The causal entropy staring from ( s, a ) under policy π is
H π ( s, a ) =
− ln π ( a | s ) + γ (cid:82) s ′ ∈S P ( s ′ | s, a ) H π
( s ′ )
ds ′ and the casual entropy from state s is
H π ( s ) =
(cid:82) a ∈A π ( a | s )
H π ( a | s ) da .
We define the state-action visitation frequency as ψ π ( s, a ) ≜
E π [ (cid:80) ∞ t =
γ t 1 { S t = s } 1 { A t = a } ] and state visitation frequency as ψ π ( s ) ≜
E π [ (cid:80) ∞ t =
γ t 1 { S t = s } ] , where 1 {·} is the indicator function.
It is obvious that ψ π ( s ) ≤ 11 − γ for any s ∈ S and (cid:82) s ∈S ψ π ( s ) ds = 11 − γ .
We use 0 m × n to denote an m × n matrix whose entries are all zero.
Lemma 4.
For any ( s, a ) ∈ S × A , any ω c ∈ Ω c , and any π , || µ πr,ω c ( s ) || , || µ πr,ω c ( s, a ) || , || µ πc ( s ) || , and || µ πc ( s, a ) || are bounded.
Proof.
We know that µ πr,ω c ( s, a ) =
ϕ r,ω c ( s, a ) + E πS,A [ (cid:80) ∞ t =1 γ t ϕ r,ω c ( S t , A t ) | S
= s,
A
= a ] .
Since || ϕ r,ω c
( s, a ) || ≤ (cid:113)(cid:80) N E i =
( l [ i ] r d 21 ) + ( (cid:80) N E i =1 l [ i ] c d 2 ) 2 ) , we know that || µ πr,ω c ( s, a ) || ≤ 11 − γ (cid:113)(cid:80) N E i =1 ( l [ i ] r d 21 )
+ ( (cid:80) N E i =1 l [ i ] c d 2 ) 2 ) .
Because µ πr,ω c ( s ) = (cid:82) a ∈A π ( a | s ) µ πr,ω c ( s, a ) da , we can see that || µ πr,ω c ( s ) || is bounded given that || µ πr,ω c ( s, a ) || is bounded.
Analogously, || µ πc ( s ) ||
and || µ πc ( s, a ) || are also bounded.
Constrained soft Bellman policy .
We provide the formula of the constrained soft Bellman policy under continuous state-action space and the policy can be approximated through soft Q learning [26]:
[Equation]
It is obvious that the constrained soft Bellman policy is continuous in ( ω c , η ) as it is a composition of continuous functions of ( ω c , η ) .
B Appendix: Proof
This section focuses on the continuous state-action space and has seven subsections.
Subsection
B.1 includes some preliminary results for the later subsections, subsection B.2 presents the proof of Lemma 1 , subsection B.3 provides some intermediate results for the remaining subsections, subsection B.4 proves Lemma 3 , subsection B.5 presents the derivation of LGA, subsection B.6 proves
Lemma 2 , and subsection B.7 presents the proof of the theorem.
B.1 Preliminary results
In this section
[Equation]
we prove two lemmas which serve as building blocks for the remaining subsections.
Lemma
[Equation]
Proof.
Define Z s,η ; ω c ≜ exp( V soft η ; ω c ( s )) and Z a | s,η ; ω c ≜ exp( Q soft η ; ω c ( s, a ))
[Equation]
therefore Z a | s,η ; ω c is smooth in η because it is a composition of logarithmic, exponential and linear functions of η .
From
Leibniz integral rule
[Equation]
we know that ∇ η (cid:82) a Z a | s,η ; ω c da = (cid:82) a ∇ η Z a | s,η ; ω c da .
Therefore,
[Equation]
Continuing the expansion, we can get:
[Equation]
µ
[Equation]
; ωc
[Equation]
Therefore
[Equation]
∇ η ln π η ; ω c ( a | s ) =
∇ η ln Z a | s,η ; ω c − ∇ η ln Z s,η ; ω c = µ π η ; ωc r,ω c ( s, a ) − µ π η ; ωc r,ω c ( s ) .
Lemma 6.
The gradients ∇
[Equation]
µ
[Equation]
da
[Equation]
− 1) dads .
Proof.
[Equation] [Equation]
Keep the expansion and we can get:
[Equation]
Analogously, we have that
[Equation]
Keep the expansion, we have:
[Equation]
B.2 Proof of Lemma
The Lagrangian of problem (2) is L ( π, η ; ω
[Equation]
=
H
[Equation]
+
[Equation] [Equation]
−
[Equation]
µ
[Equation] [Equation]
+
[Equation]
) − ˆ b ω c ) .
To find the maximizer of max π ∈ Π L ( π, η ; ω c ) , we remove the constant terms and formulate the following problem:
[Equation]
From [26], we can see that the continuous constrained soft Bellman policy is the optimal solution of (5) by setting the hyperparameter α = 1 in equation (2) in [26].
Let p ∗ be the optimal value of the primal problem (2) and d ∗ be the optimal value of the dual problem min η G ( η ; ω c ) .
We know that p ∗ exists (proof of Lemma 2 in [25]).
For any η , G ( η ; ω c ) is an upper bound of p ∗ because any optimal solution of the primal problem (2) is a feasible solution of max π ∈ Π L ( π, η ; ω c ) , therefore, d ∗ is finite.
We change the policy π to be time-dependent but force it to be stationary, then
[Equation]
[Equation]
where P ( S t = s ) is the probability (density) of state s being reached at time t .
From the formula of the constrained soft Bellman policy, we know that π η ; ω c ( a | s ) > 0 for any s ∈ S and a ∈ A , thus π η ; ω c is an interior point of Π .
Therefore ∂L ( π,η ; ω c ) ∂π ( a | s ) = 0 at π η ; ω c and from (6) we have that for any ( s, a ) ∈ S ×
A such that P ( S t = s ) ̸ = 0 for at least one time t :
[Equation]
Now, we derive the gradient of G ( η ; ω c )
=
H ( π
[Equation]
+
[Equation]
+ λ
[Equation]
[Equation]
where the second equality follows from Lemma 6 and the third equality follows from (7) and the fact that ψ π η ; ωc ( s )
=
if P ( S t = s ) = 0 for all time t .
Notice that µ r,ω c ( π η ; ω c ) − ˆ µ r,ω c
=
[( µ r ( π η ; ω c ) − ˆ µ r ) ⊤ , J ω c ( π η ; ω c ) − ˆ b ω c ] ⊤ .
Analogously, we can see that ∇ η G [ v ] ( η ; ω c ) =
[( µ r ( π η ; ω c ) − ˆ µ [ v ] r ) ⊤ , J ω c ( π η ; ω c ) − ˆ b
[ v ] ω c ] ⊤ .
As G ( η ; ω c ) attains its minimum at η ∗ ( ω c ) , given that min η G ( η ; ω c ) is unconstrained, the gradient of G ( η ; ω c ) with respect to η should be 0 at this point ( i.e. µ r ( π η ; ω c ) − ˆ µ r =
and J ω c ( π η ; ω c ) − ˆ b ω c = 0 ).
It implies that π η ; ω c is a feasible solution of the primal problem (2).
Thus, we have:
[Equation]
Therefore, we know that p ∗ =
d ∗ and p ∗ is obtained at π η ∗ ( ω c ); ω c .
B.3 Intermediate results
In this section, we prove three lemmas which will be used in the remaining subsections.
Lemma 7.
The dual functions G ( η ; ω c ) and G [ v ] ( η ; ω c ) are both strictly convex in η for any nonzero cost weight vector ω c .
We know that
[Equation]
where the second equality follows from Lemma 6 and the last equality follows from Lemma 5.
For any nonzero vector ¯ η , we have:
[Equation]
where Var ( Q π η ; ωc ¯ η,ω c ( s, · )) is the variance of the augmented Q-function Q π η ; ωc ¯ η,ω c at state s .
We know that Q π η ; ωc ¯ η ; ω c ( s, a ) =
¯ η ⊤ µ π η ; ωc r,ω c ( s, a ) and µ π η ; ωc r,ω c ,j ( s, a ) is in essence a discounted cumulative sum of ϕ r,ω c ,j .
Because each component of ϕ r,ω c can be chosen independently, the variance
Var ( Q π η ; ωc ¯ η,ω c ( s, · )) =
(cid:80) j ¯ η 2 j Var ( µ π η ; ωc r,ω c ,j ( s, · )) .
From the formula of the constrained soft Bellman policy, we know that whatever η and ω c are, π η ; ω c ( a | s ) > 0 for any a ∈ A at state s .
Therefore, the variance Var ( µ π η ; ωc r,ω c ,j ( s, · )) is zero only under the situation that µ π η ; ωc r,ω c ,j ( s, a ) is a constant almost everywhere over A and we can choose ϕ r,ω c ,j whose values always vary over a to avoid this.
Therefore, Var ( µ π η ; ωc r,ω c ,j ( s, · )) > 0 and thus Var ( Q π η ; ωc ¯ η,ω c ( s, · )) > 0 .
Because ψ π η ; ωc ( s ) ≥ 0 for all s ∈ S and (cid:82) s ∈S ψ π η ; ωc ( s ) ds = 11 − γ , the measure of the set where ψ π η ; ωc ( s ) > 0 is strictly greater than 0 , otherwise (cid:82) s ∈S ψ π η ; ωc ( s ) ds = 0 .
Thus, (cid:82) s ∈S ψ π η ; ωc ( s )
Var ( Q π η ; ωc ¯ η,ω c
( s, · )) ds > 0 and ∇ 2 ηη G ( η ; ω c ) is positive definite.
Analogously, ∇ 2 ηη G [ v ] ( η ; ω c ) is also positive definite.
Therefore, G ( η ; ω c ) and G [ v ] ( η ; ω c ) are both strictly convex for any nonzero ω c .
Lemma 8.
(i)
There is a positive constant C ∇ η G [ v ] such that for any ω c ∈ Ω c and η ∈ R (cid:80) NEi =1 l [ i ] r +1 , it holds that ||∇ η G [ v ] ( η ; ω c ) || ≤ C ∇ η G [ v ] and ||∇ η G ( η ; ω c ) || ≤ C ∇ η G ≜ (cid:80) N L v =1 C ∇ η G [ v ] .
(ii)
For any ω c ∈ Ω c and η ∈ R (cid:80) NEi =1 +1 , G [ v ] ( η ; ω c ) and G ( η ; ω c ) are continuously twice differentiable in ( ω c , η ) .
Proof.
(i) From Lemma 1 , we know that ||∇ η G [ v ] ( η ; ω c ) ||
= || µ r,ω c ( π η ; ω c ) − ˆ µ [ v ] r,ω c || ≤ || µ r,ω c
( π η ; ω c ) ||
+ || ˆ µ [ v ] r,
ω c
|| , thus from Lemma 4, we know that ||∇ η G [ v ] ( η ; ω c ) || ≤ C ∇ η G [ v ] for some positive constant C ∇ η G [ v ] .
Therefore, ||∇ η G ( η ; ω c ) ||
= || (cid:80) N L v =1 G [ v ] ( η ; ω c ) || ≤ (cid:80) N
[Equation]
(cid:80)
[Equation]
C
[Equation]
G
[Equation]
≜
[Equation]
G (ii) From the proof in Lemma 7, we know that:
[Equation]
Thus,
[Equation]
where the formula of ∇ η µ π η ; ωc r,ω c ( s ) can be found in Lemma 6 and the formula of ∇ η µ π η ; ωc r,ω c ( s, a ) can be derived in a similar way to Lemma
[Equation]
With similar process
[Equation]
we can also get the formulas of ∇ 3 ω c ω c ω c G ( η ; ω c ) , ∇ 3 ηω c ω c G ( η ; ω c ) , ∇ 3 ηηω c G ( η ; ω c ) , and ∇ 3 ηω c η G ( η ; ω c ) .
Therefore, G ( η ; ω c ) is three-times differentiable.
Analogously, G [ v ] ( η ; ω c ) is also three-times differentiable.
Thus, they are both continuously twice differentiable.
Lemma 9.
There are positive constants τ ∇ 2 ηη G and τ ∇ 2 ηη G [ v ] such that ∇ 2 ηη G ( ω c , η ∗ ( ω ′ c ))
⪰ τ ∇ 2 ηη G I , ∇ 2 ηη G ( ω c , ¯ η [ v ] ( ω c ))
⪰ τ ∇ 2 ηη G I , ∇ 2 ηη G [ v ] ( ω c , η ∗ ( ω ′ c ))
⪰ τ ∇ 2 ηη G [ v ] I and ∇ 2 ηη G [ v ] ( ω c , ¯ η [ v ] ( ω c ))
⪰ τ ∇ 2 ηη G [ v ] I for all ω c ∈ Ω c \{ 0 }
Proof.
The proof of Lemma 9 is divided into two steps where the first step shows that ∇ 2 ηη G ( ω c , η ∗ ( ω ′ c )) ⪰
τ ∇ 2 ηη G I and the second step shows that ∇ 2 ηη G ( ω c , ¯ η [ v ] ( ω c ))
⪰ τ ∇ 2 ηη G I .
The part for ∇ 2 ηη G [ v ] can be derived analogously.
Step (i) .
Lemma 7 shows that G ( η ; ω c ) is strictly convex in η , thus there is a unique η ∗ ( ω c ) .
Since the problem min η G ( η ; ω c ) is unconstrained and ∇ η G ( η ∗ ( ω c ); ω c ) = 0 holds for any ω c ∈
R (cid:80) NEi =1 l [ i ] c , taking derivative with respect to ω c on both sides renders
[Equation]
where M ( ω c , η ) ≜ ∇ 2 ω c η G ( η ; ω c ) (cid:2) ∇ 2 ηη G ( η ; ω c ) (cid:3) − 1 .
Since η ∗ ( ω ′ c ) is differentiable (equation (8)) and ∇ 2 ηη G ( η ; ω c ) is continuous in ( ω c , η ) (Lemma 8 (ii)), ∇ 2 ηη G ( η ∗ ( ω ′ c ); ω c ) is continuous in ( ω c , ω ′ c ) .
Since Ω c is compact, Ω c × Ω c is compact (Tychonoff’s theorem, Theorem 1 . 9 .
7 in [33]), then the image of ∇ 2 ηη G ( η ∗ ( ω ′ c ); ω c ) is compact (Theorem 4 .
14 in [34]).
We denote the eigenvalues of ∇ 2 ηη G ( η ∗ ( ω ′ c ); ω c ) by λ i ( ω c , ω ′ c ) and we know that
λ i ( ω c , ω ′ c ) > 0 for any ω c and ω ′ c because ∇ 2 ηη G ( η ∗ ( ω ′ c ); ω c ) is positive definite and symmetric.
Now, we prove that the image of every λ i ( ω c , ω ′ c ) is compact.
The characteristic polynomial of ∇ 2 ηη G ( η ∗ ( ω ′ c ); ω c ) is ( λ − λ 1 ( ω c , ω ′ c )) · · · ( λ − λ d ( ω c , ω ′ c ))
=
λ
d + p d − 1 ( ω c , ω ′ c ) λ
d − 1 + · · · + p 1 ( ω c , ω ′ c ) λ
+ p 0 ( ω c , ω ′ c ) , where λ i ( ω c , ω ′ c ) is the root and p i ( ω c , ω ′ c ) is the coefficient.
Each p i ( ω c , ω ′ c ) is in essence a polynomial function of the entries of ∇ 2 ηη G ( η ∗ ( ω ′ c ); ω c ) and the entries of ∇ 2 ηη G ( η ∗ ( ω ′ c ); ω c ) are continuous in ( ω c , ω ′ c ) as ∇ 2 ηη G ( η ∗ ( ω ′ c ); ω c ) is continuous in ( ω c , ω ′ c ) , then p i ( ω c , ω ′ c ) is continuous in ( ω c , ω ′ c )
(Theorem 4 . 7 in [34]).
Then λ i ( ω c , ω ′ c ) is also continuous in ( ω c , ω ′ c )
(Theorem 3 . 9 .
1 in [35]), thus the image of λ
i ( ω c , ω ′ c ) is compact.
According to Heine-Borel theorem (Theorem 2 . 41 in [34]), the image of λ i ( ω c , ω ′ c ) is closed and bounded.
Therefore, min λ i ( ω c , ω ′ c ) exists and belongs to the image of λ
i ( ω c , ω ′ c ) , then min λ
i ( ω c , ω ′ c ) is positive.
Thus, we can choose a positive number τ ∇ 2 ηη G ∗ =
min { min λ 1 ( ω c , ω ′ c ) , · · · , min λ d ( ω c , ω ′ c ) } and ∇ 2 ηη G ( η ∗ ( ω ′ c ); ω c ) ⪰ τ ∇ 2 ηη G ∗ I .
Step (ii) .
For the distributed gradient descent in Algorithm 2 , we know that (equation (5) in [36]) η
[Equation]
N
[Equation]
v ′ =
[Φ
[Equation] [Equation]
[Equation]
ω
[Equation]
) , where Φ( k, s ) ≜ W ( s ) W ( s + 1) · · · W ( k ) is the state transition matrix and [Φ( k, s )] vv ′ is the entry at the v -th row and v ′ -th column.
We define y ( ω c , k ) ≜ 1 N L (cid:80) N L v ′ =
1 η [ v ′ ] (0) − (cid:80) ks
=
α ( s − 1) (cid:80) N L v ′ =
1 1 N L ∇ η G [ v ′ ]
( η [ v ′ ]
( ω c , s − 1); ω c ) , then y ( ω c , k + 1) =
y ( ω c , k ) − α ( k ) N L (cid:80) N L v ′ =
1 ∇ η G [ v ′ ]
( η [ v ′ ]
( ω c , k ); ω c ) .
Following the proof of Lemma 5 (a) and proposition 3 in [36], we can get ||
[Equation]
|
[Equation] [Equation]
N
[Equation]
where B
=
( N L − 1) B .
We can find c η such that (cid:80) N L v =1 || η [ v ] (0) ||
≤ c η .
Therefore,
[Equation]
Notice that the second term in (11) does not exist if k < 2 .
Claim
[Equation]
is bounded for any K
[Equation]
Proof.
Because the upper bound of 4 α ( k ) N L (cid:80) N L v ′
=1 C ∇ η G [ v ′ ]
|| y ( ω c , k ) − η
[ v ′ ] ( ω c , k ) || , i.e., formula (11), is positive, it suffices to show that (cid:80) ∞
k
=1 4 α ( k ) N L (cid:80) N L v ′ =
1 C ∇ η G [ v ′ ]
|| y ( ω c , k ) − η
[ v ′ ] ( ω c , k ) || is bounded.
Now, we prove that the summation of each term in (11) is finite one by one.
First,
[Equation]
is bounded
[Equation]
Second, ∞ (cid:88)
[Equation]
Let S
k
= (cid:80) k − 1 s =1 ¯ αs (1 − ϵ B 0 )
k − 1 − s B 0 , then k ¯ α S k − 1 − k +1¯
[Equation]
Because (1 − ϵ B 0 ) k − 2 B 0 decays faster than 1 ( k − 2)( k − 1)( k +1) , there exists a positive integer ¯ K such that
[Equation]
k
[Equation]
¯ K .
Therefore, k ¯ α S k − 1 − k +1¯ α S k >
if k > ¯ K .
We can find a positive number M such that k +1 ¯ α S k < M for any k > 0 .
Then, ¯ α k +1 S k < ¯ α 2 M ( k +
[Equation]
2 and (cid:80) ∞ k =2 ¯ αk +1 S k is finite.
Third,
[Equation]
Equation (10) shows that
||
y ( ω c , k + 1)
[Equation]
Telescoping from k = 0 to K − 1 , we have:
[Equation]
where D max
[Equation]
is finite.
Therefore
[Equation]
+ (cid:112) D max ≤ ¯ D
[Equation]
where ¯ D
[Equation]
=
[Equation]
+ √
D max is finite.
Because η ∗ ( ω c ) is bounded within a compact set and || η [ v ] ( ω c , k ) || ≤ || η ∗ ( ω c ) ||
+ ¯
D max , we know that η [ v ] ( ω c , k ) is also bounded within a compact set.
Then, || ¯ η [ v ] ( ω c )
||
= || (cid:80) K − 1 j =
α ( k ) η [ v ] ( ω c ,k ) (cid:80) K −j =
α ( k ) || ≤ max || η
[ v ] ( ω c , j ) || ≤ || η ∗ ( ω c ) ||
+ ¯ D max so that ¯ η [ v ] ( ω c ) is also bounded within a compact set.
Following the same idea of η ∗ , we can find a positive constant τ G ¯ η ¯ η such that ∇ 2 ηη G (¯ η [ v ] ( ω c ); ω c ) ⪰
τ
G ¯ η ¯ η I for all k > 0 .
Then, we can find the positive constant τ ∇ 2 ηη G = min { τ ∇ 2 ηη G ∗ , τ
G ¯ η ¯ η } .
With similar derivation, we can find the positive constant τ ∇ 2 ηη G [ v ] .
B.4 Proof of Lemma
From Claim 1, we know that there is a positive constant c max such that (cid:80)
Kk
=0 2 α ( k ) (cid:80) N L v ′ =
1 C ∇ η G [ v ′ ]
|| y ( ω c , k ) − η
[ v ′ ] ( ω c , k ) || ≤ c max for all K > 0 .
From (9), we know that ||
y ( ω c , k +1) − η ∗ ( ω c ) ||
2 ≤ || y ( ω c , k ) − η ∗ ( ω c ) ||
2 + 4 α ( k ) N L (cid:80) N L v ′
=1 C ∇ η G [ v ] || y ( ω c , k ) − η
[ v ] ( ω c , k ) || − 2 α ( k ) N L (cid:20)
G ( y ( ω c , k ); ω c ) −
G ( η ∗ ( ω c ); ω c ) (cid:21) + ( α ( k ))
2 N L C 2 ∇ η G , then we have:
K
[Equation]
(cid:88)
[Equation] [Equation]
≤
[Equation]
⇒
[Equation]
Then, ||
[Equation]
+ ||
[Equation]
c , k ) − y ( ω c , k ) || (cid:80) K − 1 k =
α ( k ) + || (cid:80) K − 1 k =
α ( k ) y ( ω c , k ) (cid:80) K − 1 k =
α ( k ) − η ∗ ( ω c ) || .
We know that c max≥ (cid:80) K − 1 k =
α ( k ) (cid:80) N L v ′ =
1 C ∇ η G [ v ′ ]
|| y ( ω c , k ) − η
[ v ′ ] ( ω c , k ) || ≥ C ∇ η G [ v ] (cid:80) K − 1 k
=
α
( k ) ||
y ( ω c , k ) − η
[ v ] ( ω c , k ) || ,
then || (cid:80) K − 1 k =
α ( k ) η [ v ] ( ω c , k ) (cid:80) K − 1 k =
α ( k ) − η ∗ ( ω c ) || ≤ log K c max 2 C ∇ η G [ v ] + (cid:118)(cid:117)(cid:117)(cid:116)
τ ∇ 2 ηη G log K (cid:20) N L || y
( ω c , 0) − η ∗ ( ω c ) ||2¯ α + c max + C 2 ∇ η G π 2 ¯ α 12 N L (cid:21) , =
log K c max 2 C ∇ η G [ v ] + 1 √ log K (cid:118)(cid:117)(cid:117) (cid:116) (cid:20) N L
|| y ( ω c , 0) − η ∗ ( ω c ) ||τ ∇ 2 ηη G ¯ α + 2 c max τ ∇ 2 ηη G +
C 2 ∇ η G π 2 ¯ α 6 τ ∇ 2 ηη G N L , = log K c max 2 C ∇ η G [ v ] + 1 √ log K (cid:118)(cid:117)(cid:117)(cid:116)
[Equation]
We let C [ v ] 1 =
[Equation]
and
[Equation]
B.5 The derivation of learner v ’s LGA
Using the chain rule, we know that:
[Equation]
where M ( ω c , η ) is defined in (8).
Lemma 10.
The gradient ∇ F ( ω, η ∗ ( ω c ))
= ∇ ω c F ( ω c , η ∗ ( ω c )) .
Proof.
The global likelihood function is:
[Equation]
where P D ( S t = s, A t = a ) is the empirical probability of ( s, a ) occurring at time t in the demonstrations D :
[Equation]
From the proof of Lemma
[Equation]
we know that π η ; ω c can be formulated as:
[Equation]
Thus
[Equation]
where the last inequality follows from Lemma
[Equation]
Here,
[Equation] [Equation]
Therefore,
[Equation]
The above derivation shows that ∇ η F ( ω c ; η ) =
− m ∇ η G ( η ; ω c ) .
Therefore, we have that ∇ F ( ω c , η ∗ ( ω
[Equation]
= ∇
[Equation]
, η ∗ ( ω c ))
− M ( ω c , η ∗ ( ω c )) ∇
[Equation]
( ω
[Equation]
, η
[Equation]
( ω c ))
= ∇ ω c F ( ω c , η ∗ ( ω c ))
+
M ( ω c , η ∗ ( ω c )) ∇ η mG ( η
[Equation]
( ω
[Equation]
); ω c ) = ∇ ω c F ( ω c , η ∗ ( ω c )) .
However, we cannot obtain the gradient ∇ F ( ω c , η ∗ ( ω c )) because each learner v does not find η ∗ ( ω c ) but its approximation ¯ η [ v ] ( ω c ) .
Therefore, we propose a surrogate gradient to approximate ∇ F ( ω c , η ∗ ( ω c )) : ¯
[Equation]
η [
[Equation] [Equation]
ω
[Equation]
Notice that this surrogate gradient is global, to decompose it, we propose a local gradient approximation from learner v : ¯ ∇ F [ v ] ( ω c , ¯ η [ v ] ( ω c )) ≜ ∇ ω c F [ v ] ( ω c , ¯ η [ v ] ( ω c )) .
This local approximation aims to approximate ¯ ∇ F [ v ] ( ω c , η ∗ ( ω c ))
= ∇ ω c F [ v ] ( ω c , η ∗ ( ω c )) .
Notice that ∇ F ( ω c , η ∗ ( ω c ))
= ∇ ω c F ( ω c , η ∗ ( ω c )) =
(cid:80) N L v =1 ∇ ω c F [ v ] ( ω c , η ∗ ( ω c ))
= (cid:80) N L
[Equation]
=1 ¯ ∇ F [ v ] ( ω c , η ∗ ( ω c )) .
Similar to the derivation of ∇ η F ( ω c , η ) , we can get the gradient ∇ ω c F [ v ] ( ω c , η )
=
(cid:80) ζ j ∈D [ v ] (cid:80)
∞ t = γ t ϕ c ( s jt , a jt ) − m [ v ] E π η ; ωc S,A [ (cid:80) ∞ t =
γ t ϕ c ( S t , A t )] .
We still use this approximation when ω c = 0 .
Lemma 11.
The approximation error ||
¯ ∇ F [ v ] ( ω c , ¯ η ( ω c ))
− ¯ ∇ F [ v ] ( ω c , η ∗ ( ω c )) || is upper bounded and decreases to zero at the rate of O ( 1 √ log K ) .
Proof.
At first, we first prove the following claim showing that bounded gradient can imply Lipschitz condition of a function if the domain of the function is convex.
Claim 2.
Suppose a function f : E ⊆ R n → R m is differentiable on the convex set E , and ||∇ f ( x ) || ≤ M for any x ∈ E , where M is a finite positive constant.
Then f satisfies the Lipschitz condition, i.e., || f ( b ) − f
( a ) ||
≤ M ||
b − a || for any a, b ∈ E .
Proof.
Let y ( t ) =
(1 − t )
a + tb where t ∈ [0 , 1] and h ( t ) =
f ◦ y ( t ) .
Let z = f ( b ) − f ( a ) and define v ( t ) =
z ⊤ h ( t ) .
Then v is real-valued and differentiable on (0 , 1) .
By the mean value theorem, there exists ¯ t ∈ (0 , 1) such that v (1) − v (0) = v ′ ( t ) ⇒ ||
f ( b ) − f ( a ) ||
=
( f ( b ) − f ( a )) ⊤ ( ∇ y f
( y ( t )))
⊤ ( b − a ) ≤ || f ( b ) − f
( a ) || · ||∇ y f ( y ( t ))
|| · || b − a || ⇒ ||
f ( b ) − f ( a ) || ≤
M || b − a || .
Claim 3.
For any ω c ∈ Ω c , ∇ ω c F [ v ] ( ω c , η ) is Lipschitz continuous (w.r.t.
η ) with positive constant
L ∇ ωc F [ v ] .
Proof.
From Claim 2, it suffices to show that ||∇ ω c η F [ v ] ( ω c , η ) || is bounded.
Following similar derivation in the proof of Lemma 10, we can get ∇ ω c F [ v ] ( ω c , η ) =
m
[ v ] (ˆ µ c − λµ c ( π η ; ω c )) .
Then following the similar idea of the proof in Lemma 7, we can see that
[Equation]
[Equation]
Because ψ π η ; ωc ( s ) , π η ; ω c ( a | s ) , µ π
[Equation]
; ωc r,ω c ( s ) , µ π η ; ωc r,ω c ( s, a ) , µ π η ; ωc c ( s ) , and µ π η ; ωc c ( s, a ) are finite (Lemma 4), and (cid:82) s ∈S ds = S and (cid:82) a ∈A da =
A are finite, we know that ||∇ ω c η F [ v ] ( ω c , η ) || is bounded and thus L ∇ ωc F [ v ] exists.
Therefore, the approximation error is
[Equation]
From Lemma 3 , we know that || ¯ η ( ω c ) − η ∗ ( ω c ) || decreases to 0 at the rate of O ( 1 √ log K ) , thus || ¯ ∇ F [ v ] ( ω c , ¯ η ( ω c ))
− ¯ ∇ F [ v ] ( ω c , η ∗ ( ω c ))
|| decreases to 0 at the rate of O ( 1 √ log K ) .
B.6 Proof of Lemma
Learner v ’s LSCA problem at ω [ v ] c is arg max ω c Ω c ˜ F
[ v ] ( ω c ; ω [ v ] c ) where ˜ F [ v ] ( ω c ; ω [ v ] c ) needs to satisfy the following four conditions (Assumption 3 . 14 in [37]):
(i) ∇ ˜ F [ v ] ( ω c ; ω [ v ] c ) =
N L ¯ ∇ [ v ] at ω [ v ] c .
(ii) − ˜ F
[ v ] ( ω c ; ω [ v ] c ) is strongly convex in ω c .
(iii) ˜ F [ v ] ( ω c ; ω
[ v ] c ) is continuously differentiable in ω c .
(iv) ∇ ˜ F [ v ] ( ω c ; ω [ v ] c ) is Lipschitz continuous in ω [ v ] c .
To satisfy these assumptions, we choose the function ˜ F [ v ] ( ω c ; ω [ v
[Equation]
c
[Equation]
−
[Equation] [Equation]
||
+ ( N L ¯ ∇ [ v ] ) ⊤ ( ω c − ω
[ v ] c ) .
Then ˜ ω
[ v ] c = arg max
[Equation]
c −
[Equation]
|| ω c − ω
[Equation] [Equation]
|
[Equation]
+ (cid:18)
N L ¯ ∇ [ v ] (cid:19) ⊤ (cid:18) ω
c − ω [ v ] c (cid:19) .
Now, we prove that ˜ ω [ v ] c
= Project Ω c (cid:18) ω
[ v ] c
+ N L ¯ ∇ [ v ] (cid:19) .
Because ω c ∈ Ω c , by the property of the projection operator, we know that ˜ ω [ v ] c = arg min
[Equation]
∇
[Equation]
] ||
[Equation]
=
[Equation]
B.7 Proof of Theorem
With the the result of LSCA in subsection B.6, SONATA (Theorem 4 in [38]) can prove that ( consensus )
[Equation]
Then,
[Equation] [Equation]
where the last inequality follows from Claim 3 and || η ∗ ( ω [ v ] c
[Equation]
n )
[Equation] [Equation]
(Lemma 3 ).
Then, we have
[Equation]
where ¯ M is well-defined because l [ i ] c and L ∇ ωc F [ v ] are both positive constants.
C Appendix: Simulation details
In this section, we provide the simulation details.
The Python3 code was run on a laptop with one
Intel Core i7-9750H 2.60GHz CPU and 16 GB of RAM under Ubuntu 18.04 operating system.
Due to the well-known curse of dimensionality, the reinforcement learning (or dynamic programming) of multiple experts is hard to compute.
To alleviate this issue, we model the experts as separate MDPs for most of the time and only model them as an MG when they are close to each other.
C.1 Synthetic grid world
The inner communication network for the four learners has two stages: in stage 1 , learnersand 2 can communicate and learners 3 and 4 can communicate; in stage 2 , learners 1 and 4 can communicate and learners 2 and 3 can communicate.
Thus the adjacency matrix in stage 1 is   0 .
5 0 .
 and in stage 2 is   0 .  0 . 0 .
0 .
 .
The outer communication graph also has two stages, as we need to ensure ¯ N [ v ] ( n ) =
N , the adjacency matrix in two stages are respectively:   0 . 49999 0 .
[Table]

 and  
[Table]

 .
The discount factor is 0 .
For each ω [ i ] c,j , we also need a threshold ω
c, th to judge whether C [ i ] j is identified as a part of the ground truth constraint, i.e., C
[ i ] j is identified as a part of the ground truth constraint if ω [ i ] c,j ≥ ω c, th and is not if otherwise.
The principle of choosing the value of ω c, th is that the probability of entering C [ i ] j when ω
[ i ] c,j = ω c, th is less than 1% of the probability when ω [ i ] c,j = 0 .
In this example, we choose ω c, th = 0 . 1 .
C.2 Drones motion planning with obstacles
The simulator is built in Gazebo based on a package called hector quadrotor [39].
In this experiment, there are only two learners who obtain, respectively, four and five pairs of demonstrated trajectories.
The inner and outer communication graphs are the same and only have one stage.
The adjacency matrix is (cid:20) 0 .
5 0 .(cid:21) .
The discount factor is 1 and the drones fly at most 500 steps and we choose ω c, th = 0 .
D Appendix: Supplementary contexts D.1 Intuition of the bi-level formulation
The intuition behind this bi-level formulation is inspired by hyperparameter learning [22] and an
IRL literature [40].
Paper [40] interprets IRL as an bi-level problem where the outer level is to learn the reward function and the inner level is to learn the corresponding policy.
In our case, we treat the constraint as a hyperparameter in our learned environment and we need to recover the corresponding reward function and policy given current constraint.
Therefore, in our problem, the outer level is to learn the constraints and the inner level is to learn the corresponding reward function and policy.
D.2 Derivation of centralized bi-level problem and decomposition to the distributed bi-level problem
The global log likelihood of the global demonstration set D = { ζ j } mj
=1 is F ( ω c )
=
(cid:80) mj
=1 (cid:80) ∞ t =0 γ t ln π ω c ( a jt | s jt ) .
To solve this MLE problem, we need to find a parametric policy model of π ω c .
To find such a model, we use an optimization problem (2) based on MCE scheme.
Notice that this optimization problem (2) is parameterized by ω c , thus its optimal solution is also parameterized by ω c .
Its optimal solution is the policy model we want in the likelihood function F ( ω c ) .
Following well-established MCE IRL literature [25], we can see that the optimal solution of this optimization problem is the constrained soft Bellman policy π η ∗ ( ω c ); ω c , where η ∗ ( ω c ) is the optimization solution of the problem min η G ( η ; ω c ) .
To fully define the MLE problem, we also need the optimal solution η ∗ ( ω c ) of the problem min η G ( η ; ω c ) .
Therefore, we formulate the MLE problem as a bi-level optimization problem: max ω c F ( ω c , η ∗ ( ω c )) =
(cid:80) mj
=1 (cid:80) ∞ t =0 γ t ln π η ∗ ( ω c ); ω c ( a jt | s jt ) s.t.
η ∗ ( ω c ) =
arg min η G ( η ; ω c ) .
Now, we finish the derivation of the centralized bi-level problem.
As no learner knows the global demonstration data D and no learner can formulate F and G , the centralized bi-level problem cannot be directly solved.
Therefore, we need to decompose the centralized problem into an equivalent distributed problem that the learners can solve even if each learner only knows its local demonstration.
Therefore, we decompose F into multiple F [ v ] and G into multiple G [ v ] .
In remark 3, we show that the distributed problem (3)-(4) is equivalent to the centralized bi-level problem we derive here.
D.3 Explanation of the theoretical results
In our setting, each distributed learner only has a portion of the global demonstration data set and one goal of our distributed algorithm is that the distributed learners can learn as good as a centralized learner who obtains the global data set even if the distributed learners do not share their local demonstration set.
In Lemma 3, it is shown that given a cost feature estimate ω c , the distributed learners can converge to a consensus and the consensus is the optimal solution of the inner problem, i.e., their consensus is the best solution the centralized learner can achieve.
In Theorem 1, it is shown that the distributed learners can achieve consensus on the cost weight ω c which belongs to the stationary point set of the outer problem.
Combining Lemma 3 and Theorem 1, we can see that the distributed learners will converge to the consensus on both the reward function and cost function where the learned cost weight belongs to the stationary point set of the outer problem and the learned reward weight is the optimal solution of the inner problem.