Envy-free Policy Teaching to Multiple Agents
Abstract
We study envy-free policy teaching.
A number of agents independently explore acommon Markov decision process (MDP), but each with their own reward functionand discounting rate.
A teacher wants to teach a target policy to the diverse groupof agents, by way of modifying the agents’ reward functions, providing additionalbonus to certain behaviors or penalizing others.
These reward modiﬁcations arepersonalized for each agent.
An important question in this setting concerns howa teaching program can be designed so that the agents think that they are treatedfairly.
We adopt the fairness notion of envy-freeness (EF) to formalize this questionand deﬁne three different EF notions, each imposing stronger requirements thanthe previous one.
Using these notions, we then investigate several fundamentalquestions, including the existence of EF solutions in the policy teaching setting,the computation of cost-minimizing solutions, and the price of fairness (PoF), i.e.,the increase in cost due to consideration of fairness.
We show that an EF solutionmay not exist when penalties are not allowed, but exists otherwise.
Depending onthe cost measures, computing a cost-minimizing EF solution can be formulated asconvex or linear programming and hence solved efﬁciently.
Asymptotically, the
PoF increases but at most linearly with the geometric sum of the discount factor ingeneral, the size of the MDP, and the number of agents involved.
Thus, fairnesscan be incorporated in multi-agent teaching without signiﬁcant computational orprice-of-fairness burdens.
Introduction
Incentive design is an important approach to inﬂuencing rational agents’ behavior.
In reinforcementlearning (RL), the incentive of an agent is expressed through their reward function, which plays a keyrole in determining the policy the agent learns [1].
One can thus teach a desired policy to an agent bymodifying the agent’s reward function, in a way that makes the target policy optimal with respectto the new rewards.
This can be very useful in many scenarios.
In safe RL, for example, penaltiescan be imposed on dangerous actions to prevent an agent from executing them [2].
A ride sharingplatform may reward its users with bonuses to induce user behaviors that incur socially optimalbehavior.
In many cases, personalized teaching programs are useful when agents to be taught comewith diversities: they might have very different innate reward functions, or apply different discountfactors when evaluating a policy.
As a result, agents may ﬁnd them rewarded/penalized differentlyfor performing the same action in the same situation (see Figure 1).
Concerns of fairness may arisedue to these differences, leading us to the question of how to design personalized teaching programsso that agents being taught think that they are treated fairly.
Towards answering this question, the ﬁrst step is to understand what it means to be fair in the settingof policy teaching.
Indeed, in a world with growing awareness of equality and transparency, fairnesshas discussed and evaluated in a wide range of applications.
Various fairness notions and conceptshave been proposed and used across many domains in technology [3].
We borrow the well-studied s m s l s z  s m s l s z  fairness notion of envy-freeness (EF) from the literature on fair division, which naturally applies tothe policy teaching setting.
The notion is widely used for settling disputes over property divisions,or ﬁnding resource allocations that make participants with different valuations unenvious of eachother (e.g., the fair cake-cutting problem [4]).
By adapting EF to policy teaching, we aim to ﬁnd a setof personalized teaching programs for a group of agents, such that no agent would prefer to switchthe program they receive with another agent.
At the same time, as a basic requirement of policyteaching, each program should also incentivize the corresponding agent who receives it to use thetarget policy.
Besides this basic notion, we also consider two stronger variants of it—one allows anagent to further deviate from the target policy when evaluating the potential beneﬁt had they receivedanother agent’s teaching program; the other simply requires all teaching programs to be identical,which gives complete fairness in a sense.
Main Results We investigate several fundamental questions related to the above laid out objective.
• Existence of a Solution.
The ﬁrst question is about the existence of an EF solution under thethree EF notions of interest.
We show that an EF solution always exists and one can be obtainedsimply by penalizing undesired actions by a sufﬁciently large value.
Nevertheless, the reverse doesnot hold true: one cannot hope to ﬁnd an EF solution by rewarding actions desired by the targetpolicy, no matter how large the rewards are set.
We demonstrate instances that do not admit any EFsolution even with the weakest EF notion when only bonuses are allowed, but also prove that thisnon-existence issue is resolved if the agents have an identical discount factor.• Cost Minimization.
Since reward modiﬁcation can be very costly, we are also interested in ﬁndingout a solution with the least cost.
We consider two cost measures in this paper, which considerthe norm of the reward modiﬁcations made and the cumulative cost the teacher actually pays,respectively.
Depending on the choice of the cost measure, we show that computing a costminimizing EF solution can be formulated as convex or linear programming and can hence besolved efﬁciently.
• Price of Fairness.
Finally, we analyze the price of fairness (PoF), a quantity that measures the (multiplicative) increase of cost due to consideration of fairness—in a similar spirit of the PoA(price of anarchy) in game theory.
We show that, asymptotically, the PoF increases linearly with the geometric sum of the discount factor in general, while it may also grow, at most linearly, with the size of the MDP and the number of agents involved depending on the speciﬁc EF notion considered.
In summary, our results indicate that the consideration
of
fairness in addition to the original goal of policy teaching may result in
non-
existence
of
workable solutions
but the existence is guaranteed
in a fairly wide range of important settings.
It does not appear to complicate the problem in terms of computational complexity.
The additional cost it incurs grows moderately with the size of the problem.
Thus, fairness can be incorporated in multi-agent teaching without signiﬁcant computationalor price-of-fairness burdens.
Related Work
Our work lies at the intersection of policy teaching and envy-free resource allocation.
Without the fairness constraints, our model can be seen as a policy teaching problem for eachindividual agent in the model.
A number of studies have looked at this problem [5, 6].
The problemcan be computationally harder though when the target is to hit one in a set of policies rather than a single target [7].
When the teacher is targeting a malicious policy, policy teaching can also beinterpreted as reward poisoning [8, 9, 10, 11, 12].
From a technical point of view, these two problemsare almost identical and can be solved by using the same techniques.
However, conceptually, it isless likely that one would take fairness into consideration when designing a poisoning attack.
Morebroadly, policy teaching can be seen as a sub-ﬁeld of reward design, a broader area that studies howto inﬂuence agents’ behaviors thought tweaking the reward function.
The objectives of these studiesare not limited to inducing a target policy.
A notable example is reward shaping [13, 14, 15], whichaims to accelerate an agent’s learning process through reward design.
Indeed, while our focus is onpolicy teaching, the same question of how to design rewards fairly can be asked with other objectivesas well.
These can be potential directions for future work.
The study of fair division dates back to the early work of Foley [16], and the formal concept ofenvy-freeness appeared even earlier [17].
Research on fair division has since evolved into a largebody of work, with focuses on allocation of divisible or indivisible items [18, 19, 20].
Our work is inparticular related to fair allocation of indivisible goods with subsidies [21], as external beneﬁts areprovided to change the agents’ original incentives in both settings.
Our work considers the additionalgoal of teaching the target policy; indeed, without this goal, an envy-free solution can be achievedtrivially by no making any modiﬁcation to the original reward functions.
We note that there are also other studies on machine teaching settings with multiple agents, ormultiple teachers [22, 23, 24], though with very different models from ours.
From a mechanismdesign perspective, our model can also be viewed as one version of the contract design problem [25],where a principal offers an agent a contract for performing a target policy, but might be uncertainabout the agent’s type (i.e., the original reward function).
Our EF solutions correspond exactly totruthful mechanisms that elicit the agent’s true type.
Preliminaries
[Figure]
There is a set of n agents 1 , . . . , n .
Let [ n ] = { 1 , . . . , n } .
Each agent i 2 [ n ] faces an MDP
M i = h S, A, R i , P, z ,  i i with shared state space S , action space A , transition function P : S ⇥
A ⇥ 104 S !
[0 , 1] describing the state transition dynamics, and initial state distribution z .
Moreover, there isa personal reward function R i : S ⇥ A !
R and discount factor  i .
Whenever agent i takes an action 106 a in state s , a reward R i ( s, a ) is generated for this agent; meanwhile the state transitions to a nextone s 0 2 S with probability P ( s, a, s 0 ) .
We consider the setting where each agent is concerned withthe (expected) cumulative reward, i.e., the discounted sum of rewards with respect to  i , obtainedover an inﬁnite horizon.
Without loss of generality these are actions generated by following somedeterministic policy ⇡ : S !
A , and the cumulative reward of this policy for agent i is
[Equation]
[Equation] where the expectation is taken over the trajectory ( s t , a t ) 1 t =0 resulting from an initial state s 0 sampledfrom z and the agent executing ⇡ subsequently.
The agent aims to ﬁnd an optimal policy, whichmaximizes ⇢ ⇡ i , and this can usually be handled by standard planning and reinforcement learningalgorithms.
Note that in the setting we consider throughout this paper, the agents operate independently inseparate environments.
2.1 Single-agent Policy Teaching
Consider the situation where we want agent i to execute a target policy ⇡ ? , but the agent ﬁnds adifferent policy ⇡ 0 optimal for M i .
We hope to inﬂuence the agent’s decision making and incentivizethem to use ⇡ ? .
A typical way is by modifying the agent’s reward function by providing additionalrewards.
We follow the literature and consider only deterministic target policies—indeed, in general,one cannot hope to incentivize an agent to use precisely a non-deterministic policies only throughtweaking the reward function.
Speciﬁcally, the teacher chooses a reward adjustment function  i : S ⇥ A !
R , or adjustment for 125 short, whereby an additional reward  i ( s, a ) is provided whenever the agent takes an action a 2 Ain a state s 2 S .
Effectively, the adjustment changes the agent’s reward function to e
R i ( s, a ) =
R i ( s, a ) +  i ( s, a ) .
The agent then optimizes their policy with respect to e
R i , and will be incentivizedto use ⇡ ?
if it offers the maximum cumulative reward with respect to e
R i .
The cumulative reward ofagent i obtained by executing a policy ⇡ is deﬁned as as the sum of discounted rewards as follows;we view it as a function of  i :
[Equation]
We can deﬁne the V-function and Q-function of ⇡ conditioned on adjustment  i as follows.
For alls 2 S and a 2 A :
[Equation]
The V-function captures the cumulative reward the agent obtains by starting from s and following⇡ , whereas the Q-function captures the cumulative reward by starting from s and taking action aat the ﬁrst step.
We have ⇢ ⇡ i (  i )
=
V ⇡ i ( z |  i ) :
=
E s
⇠ z V ⇡ i ( s 0 |  i ) .
Using these two concepts, the 136 Bellman equation further gives the optimal policy in the MDP: a policy ⇡ is optimal if and only if
Q ⇡ i ( s, ⇡ ( s ) |  i )  Q ⇡ i ( s, a |  i ) for all s 2 S and a 2 A .
Incentive Constraint Hence, we require that ⇡ ? satisﬁes the condition described by the Bellmanequation.
Since the agent may ﬁnd multiple policies optimal, a robustness guarantee ✏ > 0 isconsidered to strictly incentivize the agent to use ⇡ ? .
The incentive constraint requires that
[Equation]
for all a 6 = ⇡ ?
( s )
[Equation]
so that ⇡ ? remains optimal even if there is a small noise in the agent’s perception of the values.
Cost Measures
The goal of the teacher is to ﬁnd the most cost-efﬁcient way of teaching.
The costcan either be measured by the norm of the adjustment, i.e.,
[Equation]
or the cumulative payment
[Equation]
where  is a discount factor from the teacher’s perspective.
The cumulative payment measure appliesin particular when the teacher uses non-negative rewards (i.e., subsidies) as additional incentives forthe agent.
Teaching Multiple Agents and Fairness
In the multi-agent setting, the teacher needs to provide adjustments to all the agents in [ n ] .
We calla collection of adjustments (  i ) i 2 [ n ] an adjustment scheme .
A basic approach for this setting is to 151 deal with each agent separately, by solving a single-agent teaching problem for each of them.
Thesolution obtained via this approach provides personalized adjustments to the agents and it minimizesthe teacher’s overall cost.
Nevertheless, it might not be fair as we showed in the example of Figure 1.
To be more speciﬁc, we deﬁne the following notions of fairness, each being stronger than the previousone.
Deﬁnition 3.1 ( Envy-freeness (EF)).
An adjustment scheme (  i ) i 2 [ n ] is envy-free if it holds for alli 2 [ n ] that
[Equation]
In other words, no agent i would prefer the adjustment for another agent j to their own.
The above notion only compares the agents’ beneﬁts under ⇡ ? .
Indeed, we require that  i incentivizesagent i to use ⇡ ? , so the left side of (3) is exactly also the highest possible extra beneﬁt i can obtaingiven
adjustment  i .
But this is not true for the adjustment on the right side: ⇡ ? need not be optimal foragent i with respect to R i +   ; a higher cumulative reward might be attainable if the agent switchesto another policy.
In some scenarios, this higher potential reward may be a legitimate considerationwhen fairness is evaluated.
The following stronger notion takes this aspect into consideration.
Deﬁnition 3.2 ( Strong envy-freeness (SEF)).
An adjustment scheme (  i ) i 2 [ n ] is strongly envy-freeif it holds for all i 2 [ n ] that
[Equation]
where ⇢ Opt i (  j ) denotes the maximum cumulative reward attainable given adjustment  j .
An even stronger fairness notion deﬁned below simply requires the same adjustment to be applied toall the agents.
Deﬁnition 3.3 ( Complete fairness (CF)).
An adjustment scheme (  i ) i 2 [ n ] is completely fair if 171
[Equation]
Let D EF , D SEF , and D CF denote the sets of adjustment schemes complying with the above fairness 173 notions respectively.
It is not hard to see that:
[Equation]
Besides the fairness guarantees, we also want the adjustment scheme to—as our initial objective—incentivize the agents to use the target policy, i.e., to satisfy the incentive constraints we deﬁned above;we call such adjustment schemes feasible schemes (Deﬁnition 3.4).
Indeed, without the feasibilityguarantee, all the fairness notions above can be trivially satisﬁed by providing zero additional rewardto every agent.
In addition, sometimes only non-negative additional rewards are allowed, e.g., whenwe can provide the agents with subsidies but cannot penalize them.
Hence, we deﬁne non-negativeadjustment schemes (Deﬁnition 3.5).
Deﬁnition 3.4 ( feasibility).
An adjustment scheme (  i ) i 2 [ n ] is feasible (with respect to a givenrobustness guarantee ✏ > 0 ) if for all i 2 [ n ] , it holds that
[Equation]
Deﬁnition 3.5 ( non-negativity).
An adjustment scheme (  i ) i 2 [ n ] is non-negative if  i ( s, a )  0 forall
i 2 [ n ] , s 2 S , and a 2 A .
Similarly to the single-agent teaching problem, we are interested in obtaining a cost-minimizingsolution.
We consider the sum of the costs and deﬁne the cost of an adjustment scheme  as
[Equation]
Next, before we delve into the computation of a cost-minimizing solution, we ﬁrst investigate theexistence of a solution with respect to the above deﬁned fairness notions.
Existence of a Fair Feasible Solution
Throughout this section, we assume that the original rewards are bounded in the interval [  h, h ] , i.e.,
R i ( s, a )
[  h, h ] for all s , a , and i .
Our ﬁrst result shows that a fair and feasible solution alwaysexists under all of the above fairness notions, in particular under the strongest CF notion.
Theorem 4.1.
For any robustness guarantee ✏ > 0 , a CF and feasible adjustment scheme alwaysexists.
Proof.
The idea is to penalize actions off the target policy by a sufﬁciently large value.
We constructan adjustment scheme (  i )
i 2 i where
[Equation]
[Figure] =
The target policy ⇡ ? , highlighted as the red edges, is such that ⇡ ?
( s ) =
a for all s 2 S , i.e., it always chooses action a .
for all s 2 S
and i 2 [ n ] .
The scheme is CF as  i is the same for all the agents.
To see that it is also feasible, observe that by following the target policy ⇡ ? , an agent obtains rewardat least  h in every step. Hence, for all s 2 S and all a 6 = ⇡ ? ( s ) , we have
[Equation]
It then follows that
[Equation] where we used the fact that V ⇡ ? i ( s 0 |  i )  h 1   i for all s 0 , which is due to the fact that the rewardobtained at every step is at most h . Nevertheless, if we only allow non-negative schemes, the existence of a feasible solution cannot betaken for granted.
As we prove in Theorem 4.2, the example illustrated in Figure 2, albeit involvingonly two agents, does not admit any SEF feasible solution (and hence neither a CF one).
Theorem 4.2.
For any robust guarantee ✏  0 , a feasible adjustment scheme that is EF and nonnegative may not exist, even when there are only two agents and the agents’ reward functions are thesame (but discount factors are different).
Proof.
We show that there exists no feasible adjustment scheme that is EF and non-negative in theexample illustrated in Figure 2.
Suppose for the sake of contradiction that there exists a scheme(  1 ,  2 ) which is SEF, non-negative, and feasible.
Without loss of generality, we can assume that  1 ( s, b ) =
2 ( s, b ) =
for all s 2 S .
Indeed, it isnot hard to see that if there exists an EF and feasible scheme with some or all of these values beingstrictly positive, it will remain EF and feasible if these values are reset to 0 .
Hence, it remains to pindown the values for action a in the adjustment scheme.
For ease of description, let x i =
i ( s y , a )and y i
=  i ( s x , a ) for i 2 { 1 , 2 } .
We ﬁrst argue that the following two inequalities hold:
[Equation]
To see this, consider the EF constraints.
The adjustment scheme considered is EF, so it satisﬁes (3),which implies that ⇢ ⇡ ?
i (  i )  ⇢ ⇡ ?
i (   i ) , where  i means the other index in { 1 , 2 } .
Hence,
[Equation]
It is easy to derive the values of s x and s y as neither of them depends on the values of any otherstates.
When agent i uses ⇡ ? , we have
[Equation]
Plugging these two equations back into (8) gives
[Equation]
where 0 .
5 is the probability of the initial states.
Replacing  i with the corresponding values gives
[Equation]
It follows that (11) + (12) gives y 1  y 2 , and (11) +5 · (12) gives x 2  x 1 .
Next, we turn to the feasibility constraints.
The assumption that  i is feasible means that
[Equation]
and
[Equation]
Plugging the value function and Q-function ((9) and (10)) into the above two equations gives:
[Equation]
Hence, we have
[Equation]
which follows by (15), (15), (15), and (6), respectively.
This means that y 1 < 0 and contradicts theassumption that  is a non-negative scheme.
It turns out that the agents’ discount factors also play a crucial role: an identical discount factor issufﬁcient for ensuring the existence of a feasible CF solution.
We present this result below but leavethe proof to the appendix.
Theorem 4.3.
When the agents have an identical discount factor, a feasible adjustment scheme thatis also CF and non-negative always exists, for any robustness guarantee ✏ > 0 .
Computing an Optimal Fair Solution
In this section, we investigate the computational problems of fair policy teaching.
Our main result isthat for each of the fair notions we deﬁned the set of fair solutions lie in a convex polytope deﬁned bypolynomially many linear constraints.
To ﬁnd out a cost minimizing solution from this convex setcan then be formulated as a convex optimization problem, given that the two types of cost functionswe consider (i.e., (1) and (2)) are convex functions of the adjustment scheme.
We show how towrite down the various types of constraints considered as linear constraints next.
We start from thefeasibility constraints.
Feasibility Constraints A feasible scheme  is characterized by the following linear constraints,where we add an auxiliary variable V
i ( s ) for each s 2 S , and Q i ( s, a ) for pair ( s, a ) 2 S ⇥ A —these auxiliary variables correspond to the V- and Q-functions of the target policy when  is applied.
[Equation]
Indeed, the ﬁrst two lines above follow from
We then consider each of the fairness notions.
PoEF PoSEF
PoCF
[Equation]
CF Constraints To enforce CF simply amounts to the following constraints for each pair of agentsi, j 2 [ n ] , which enforces the schemes to be identical:
[Equation]
EF Constraints To enforce EF, we add additional variables
V i,j and Q i,j to capture the values V ⇡ ?
i ( s |  j ) and Q ⇡ ?
i ( s, a |  j ) , i.e., the values agent i would have got had they been offered theadjustment for agent j .
We then add the following constraints—the same format as the Bellmanequation—to force these additional variables to acquire the desired values.
[Equation]
Thus, EF simply amounts to the following constraints for each pair of agents i, j 2 [ n ] (recall that zis the distribution of the initial state):
[Equation]
SEF Constraints Similarly to the approach for handling the EF constraints, we need additionalvariables to capture the values of each agent i had then been offered adjustment  j .
Indeed, we alsouse Constraints in (18) and (19) but replace (18a) with the following one:
[Equation]
) 8 i, j, s, a (20a) which associates V i,j ( s ) to the maximum Q i,j ( s, a ) , instead of Q i,j ( s, ⇡ ?
( s )) .
It should be noted that under these constraints, V i,j ( s ) in a solution will not necessarily be equal tothe V-values of the optimal policy—it will only be an upper bound of them.
However, this will notcause any issue to our approach as our goal is not to compute the V-values.
Indeed, if the actual
V-values does not satisfy (19), then these constraints cannot be satisﬁed by any upper bounds of theactual V-values, either.
Non-negativity Constraints Finally, to enforce non-negativity, we simply need the additionalconstraints:  i ( s, a )  0 for all i , s , and a .
Price of Fairness
We consider the price of fairness (PoF) in this section.
The PoF measures the increase of teachingcost due to consideration of fairness.
It is in a similar spirit of the well-known concept of the price ofanarchy in economics and game theory.
More speciﬁcally, we deﬁne the PoEF, PoSEF, and PoCF forour three fairness notions, which stand for the prices of EF, SEF, and CF, respectively.
Formally, let I n,m,  be the set of instances with n agents, m state-action pairs (i.e., m = | S | · | A | ), and [Equation]
[Equation]   274 for all i 2 [ n ] .
We deﬁne
[Equation]
PoSEF and PoCF can be deﬁned the same way with the corresponding notions.
We consider bothtypes of cost functions we deﬁned in Section 2.
When cumulative payment is considered (i.e., (2)), werequire adjustment schemes to be non-negative for this measure to be meaningful.
Since a feasible fair s m s l s r s ⇤ b , (0 , 0) d , ( ✏ , ✏ ) a ( ✏ , ✏ ) e (0 , 0) c , (  1 , 0) | {z } L c op i e s solution may not exist with this requirement, we only consider instances with an identical discountfactor for all the agents, in which case the existence of a feasible fair solution is guaranteed (see
Theorem 4.3).
We analyze the asymptotic growth of the PoF.
The results are summarized in Table 1.
The PoFincreases linearly with  in all the cases, and with the number of agents involved, with the onlyexception of PoEF when the norm is used as the cost function.
It also increases but sub-linearly withthe size of the MDP with this cost function but not with the cumulative payment.
Our results are allasymptotically tight.
Due to space limit, we leave the detailed proofs of the bounds to the appendix and only providesome intuition about the growth of PoF here.
Consider the example illustrated in Figure 3.
Withoutfairness consideration, all agents except agent 1 already ﬁnd the target policy optimal, whereas agent1 prefers action e to d at state s r .
Overall, it sufﬁces to give a bonus of 1 on ( s ⇤ , c ) for agent 1 , thecost of which is 1 .
Now we consider fairness constraints, and suppose that we still provide a bonus 1 ( s ⇤ , c ) = 1 .
The consequence is that agents 2 , . . .
, n will be envious of this bonus to agent 1 .
Toachieve CF for example, the same bonus will have to be offered to these agents as well.
Nevertheless,while this removes envies, it also incentivizes the agents to take action b instead of a , leading toviolation of the feasibility constraint.
Inevitably, to construct a feasible and fair in this example, wecannot hope to only modify the reward for action c (or only for agent 1 with SEF and CF).
Modifyingthe other rewards is however much more costly (under the norm cost function) since each one ofthem has L copies of themselves, which requires the same modiﬁcation by symmetry.
Conclusion
We studied the fairness issue in policy teaching and adopted the notion of envy-freeness to formalizethe problem.
Several fundamental questions regarding the existence of a fair solution, the computationof cost-minimization solution, and the price of considering fairness have been answered in the paper.
For future work, it would be interesting to generalize the model to other reward design settings,where a larger set of design objectives can be considered.
In the case of non-negative adjustmentschemes, we prove that a fair solution exists for one setting.
An interesting future research directionwould be to prove this result for a broader class of settings.
As we mentioned earlier in the paper,policy teaching is equivalent to reward poisoning from a technical point of view.
Hence, almostany techniques that applies to policy teaching also applies immediately to solve reward poisoningproblems.
We note this potential negative social impact of our results but also remark that since ourconsideration is fairness we are not aware of any scenario where a malicious party intends to launchpoisoning attack while considering fairness at the same time.