PCRL: Reinforcement Learning Based on Priority Conventions for Microscopically Sequenceable Multi-agent Problems
Abstract
Reinforcement learning (RL) has played an important role in tackling the decisionproblems emerging from agent ﬁelds.
However, RL still has challenges in tackling multi-agent large-discrete-action-space (LDAS) problems, possibly resultingfrom large agent numbers.
At each decision step, a multi-agent LDAS problemis often faced with an unaffordable number of candidate actions.
Existing workhas mainly tackled these challenges utilizing indirect approaches such as continuation relaxation and sub-sampling, which may lack solution quality guaranteesfrom continuation to discretization.
In this work, we propose to embed agreedpriority conventions into reinforcement learning (PCRL) to directly tackle themicroscopically sequenceable multi-agent LDAS problems.
Priority conventionsinclude position-based agent priority to break symmetries and prescribed actionpriority to break ties.
In a microscopically sequenceable multi-agent problem,the centralized planner, at each decision step of the whole system, generates anaction vector (each component of the vector is for an agent and is generated in amicro-step) by considering the conventions.
The action vector is generated sequentially when microscopically viewed, and such generation will not miss the optimalaction vector, and can help RL’s exploitation around the lexicographic-smallestoptimal action vector.
Proper learning schemes and action-selection schemes havebeen designed to make the embedding reality.
The effectiveness and superiorityof PCRL have been validated by experiments on multi-agent applications, including the multi-agent complete coverage planning application (involving up to4 18 > 6 . 8 × 10 10 candidate actions at each decision step) and the cooperative ponggame (state-based and pixel-based, respectively), showing PCRL’s LDAS dealingability and high optimality-ﬁnding ability than the joint-action RL methods andheuristic algorithms.
Introduction
Backgrounds Multi-agent systems have been applied to various civil (see Fig. 1), military andentertainment applications [1, 2].
With the applications becoming broader and deeper, the agentnumber in a multi-agent system has become larger and larger [3, 4], which results in the large discreteaction space (LDAS) problem.
A multi-agent problem with even a dozen of agents is hard to besolved by reinforcement learning (RL) [5] for its explosive joint-action combinations [6].
To tackle the multi-agent LDAS problems, some researchers resorted to independent reinforcementlearning regardless of other agents, but that may form loose cooperation and yield unstable or suboptimal solutions; some researchers resorted to distributed reinforcement learning and communicationto cooperate , but that lacks a global knowledge and is hard to design cooperation mechanisms [7];
R 1 R
R
R R
R R
R R
R R
R R
R R
R R
R R
R(c) (a)
R 1 R(b)
R
R(d) many other researchers [8, 9] turned to centralized RL for high performance.
For a centralized RLmethod, a trivial joint-action representation (Fig. 2(a)) can be applied to a small-scale multi-agentsystem.
However, when the agent number increases to a dozen, it becomes unaffordable because theaction space is exponential to the number of agents (see Fig.
1(a) ∼ (c)).
Previous centralized RL [8]has tried relaxing large discrete action space to continuous action space and generating a few discretesamples near the optimal continuous action vector to ﬁnd the locally optimal discrete action vector.
Nevertheless, this subspace may miss the optimal discrete action vector thus leading to sub-optimal; Moreover, the relaxing methods lack solution quality guarantees from continuation to discretization.
Therefore, it remains challenging for a centralized RL to directly, effectively and efﬁciently tackle themulti-agent LDAS problems, difﬁcult in representing the actions and ﬁnding the optimal action from10 10 or more candidate action vectors.
Moreover, a multi-agent LDAS problem often has many tieoptimal actions; see Fig.
1 for an example.
With multiple ties, RL for LDAS is also difﬁcult to trainbecause multiple return-maximum peak landscapes may impact the convergence and exploitation of RL, so new action selection and training schemes are required.
Motivations Some multi-agent problems are microscopically sequenceable, i.e., an optimal jointaction can be selected out one component by one component in micro-steps according to the conventions, besides of being selected out simultaneously, i.e., the "microscopically sequenceable" deﬁned in
RL language is that the joint action π ∗ ( s ) = ( a 1 , a 2 , ..., a n ) where each component can be producedas a 1 = π ∗ 1 ( s ) , a 2 = π ∗ 2 ( s | a 1) , a 3 = π ∗ 3 ( s | a 2) , ... and π i functions are related. For example, in
Fig. 1(b), the planner (can be set to the smallest id agent R 1 , without loss of generalization) selectsthe action for R 4 , say "right"; then, the planner selects the action for R 2 by referring to R 4 ’s "right",and selects out "up"; Then selects out R 1 to "up" and R 3 to "left".
After the planner decides the actionfor each agent, the planner sends the action to corresponding agents and these agents simultaneouslyexecute the joint action.
The decision ﬂow at a decision step has been shown in Fig. 2(c) and can be expressed by naturallanguage (formal description in later paragraphs) as follows: (1) The planner makes clear theorder of agents.
(2)The planner decides one action (from A ) for the highest priority agent that theplanner believes to produce the largest Q value for the whole system, with the following agents fullysupporting the highest priority agent.
And if the ﬁrst prioritized agent has tie actions, the actionpriority will be obeyed.
The planner refers to the ﬁrst prioritized agent’s action decision and thendecides the action for the second prioritized agent, also aiming for the largest Q value for the wholesystem.
The procedure continues until the last agents.
[Figure] Priority conventions The agreed priority conventions for a speciﬁc multi-agent problem are prescribed by humans before the learning starts, and can be problem-speciﬁc designed.
The priorityconventions usually consist of (1) Agent priority convention : This is for determining the orderof agents to decide action and breaking symmetries: Dynamically, at each decision step of thecentralized system, the agents’ priorities are determined by current states, say, the relative positionsof agents.
For example, in Fig.
1(c), the agents’ priorities can be determined from left to right andupper to lower position, so the priorities (deciding order) of agents are R 1 ≺ R 2 ... ≺ R 16 .
Here,x ≺ y means x has a higher priority than y .
If the priority is not position based but ID based, thenthe state space will have many mirror states.
(2) Agent ID convention : This is for coinciding agentsand breaking symmetries. Statically assign each agent a unique ID before learning to identify itselffrom others.
When at least two agents coincide in the same position during the application, thesmaller id agent has a higher priority.
(3) Action priority convention : This is for breaking tiesof equally optimal actions. The actions of an agent have prescribed priority, say, up action ≺ leftaction ≺ down action ≺ right action, to break action ties if two actions both will bring about optimality,and to converge to one minimum lexicographic action vector.
Hereafter, we deﬁne that two actionvector (cid:126)α ≺ (cid:126)β (or called " (cid:126)α is lexicographic smaller than (cid:126)β ") if and only if ∃ i ∈ [1 , N ] , α i ≺ β iand ∀ j ∈ [1 , i − 1] , α j = β j .
The contribution of this paper lies in (1) Priority conventions and proofs: this paper proposes theconcept of "microscopically sequenceable" to tackle LDAS multi-agent problems without missing theoptimality, and can help exploitation around the lexicographic-smallest optimal action vector.
(2) Newschemes: it proposes new learning schemes that fully exploit the agreed conventions, such as auxiliaryequality constraint, and neural network action selection schemes for PCRL.
(3) Proof-of-conceptpractices: applying PCRL to tackle seemingly different problems, including 10 10 magnitude LDASmulti-agent path planning problems and the pixel cooperative task, and achieves 20% fewer steps andcompetitive performance.
Related Work
Nowadays, deep reinforcement learning has been successful in large state problems [5, 10].
However,it is still very challenging to deal with the LDAS problems, especially in the action representationand selection.
2.1 Action representation
For the action representation, previous work has followed several lines to circumvent the largediscrete-action-space obstructions:
One line is to parameterize actions into continuous spaces and then to ﬁnd discrete actions nearthe optimal continuous action [11] because continuous actions are differentiable.
For example, [8] exploited the k-nearest-neighbor to ﬁnd the optimal policy π ∗ ( s) ≈
[Equation] k ( | [Equation]
− ˆ a | 2 ) Q ( s , a) in adiscrete subspace near the optimal continuous actions ˆ a .
It has been validated in discrete cartpoleproblems, recommending systems and multi-agent systems.
One step further, work [12] exploitedlyutilized the action space structure and without assuming the structure is provided a priori.
Thisline will likely miss the optimal discrete solutions for inappropriate continuation and will lead tosuboptimality.
Also, it lacks a quality guarantee from continuation to discretization, which also hasbeen encountered and studied in the randomized rounding theory of integer programming [13].
Another line is to factor or eliminate action spaces [14–16], using binary coding and coarse coding toeliminate actions that are not optimal with high probability.
However, the elimination probabilityis hard to model and eliminations are not always possible.
Some researchers employed new factorand combination methods [17–20] for action representation and selection, when the problem satisﬁes
Individual-Global Maximization (IGM) Constraint or monotonic properties.
2.2 Action selection
Action selection usually has deep entanglement with action representation.
[21–23] uses DQN foraction selection and has achieved success in Atari games and human-level control.
[24] applieda recurrent neural network (RNN) to select actions for POMDP.
[25] used bidirectional RNN torepresent the actions and select actions, successfully dealing with the continuous multi-actions andhave achieved success in SC2 problem and other problems.
Some work resorted to social conventionsfor selecting actions but not for LDAS problems, for example, [26, 27].
And some work relied onsophisticated communications for exchanging selected action [28–30], which may be vulnerable tobe impacted by unstable communication in practical cooperation/coordination.
Our way of action selection resembles Stackelberg model and the sequence-to-sequence paradigm(for example image-to-caption [31]).
[32] also proposed seq-to-seq and on-policy policy gradientto tackle LDAS problems, along with others [33] but we offer a large agent number off-policy andoptimality-preserved one.
Meanﬁeld reinforcement learning [4] also has LDAS dealing ability.
Ititeratively learns each agent’s best response to the mean effect from its neighbors and is proved toconverge to Nash Q-value.
However, the Nash Q-value is not always the global optimal value inscenarios like the prisoner’s dilemma.
Methodology
This section introduces the basics of PCRL, then the mathematics and details of how to learn theagreed priority conventions in PCRL. PCRL algorithm
Our concern problems are markovian when macroscopically viewed and can beestablished as a classic MDP M = (cid:104)S , A, P , R , γ (cid:105) , where the letters are as usually deﬁned while
A =
A N where N is the agent number and A is the action set of agent 1 (the studied agents arehomogeneous and the action set of each agent is the same as agent 1).
PCRL aims to learn the optimalpolicies to maximize the accumulated discounted return of an MDP.
Without losing generality, we suppose agents R 1 , R 2 , ..., R N are priority-ordered according to theagreed conventions, where N is the number of agent.
Corresponding to the above basics and Fig.
2,the overall pseudocode of PCRL is in Algorithm 1 in appendix.
It is modiﬁed from the DQN [5],differing in aspects of action representation-selection, neural network architecture and learningschemes.
State representation When macroscopically viewed as in Fig. 2(b), the planner maps current states (consisting of features of current physical map state and agent state) to an action vector:(cid:126)a t = f
( s t ) , (1) where f is a trainable RNN-based function and is to be optimized.
For cleanness, we will omit thesubscription of t for the current state hereafter.
The state s can be convoluted features or simply the concatenation of raw map state and agentstate, i.e., s t = concat ( MAP t , AP t ) .
Using the CCPP (in Fig. 1(b)) as an example, s =
S grid _ map | S agent _ positions , where " | " means concatenation; grid_map can be binary vector indicating whether each grid has been covered (i.e., visited), and agent_position are an integer vectorwith length equal to grid_map indicating the agent number upon each cell.
For simplicity, the state s will be supplied as the initial hidden state (other usages are also possible),i.e. , the h and c , of the RNN of the planner.
Action representation and selection If the action space is not large enough, then a joint-actionrepresentation and selection will work.
However, large action brings in intractability.
The action in PCRL is represented using recurrent neural networks.
When generating (cid:126)a = ( a 1 , a 2 , ..., a n ) , PCRLresembles the image-to-caption [31] technology in that when given a state s , PCRL learns to optimizeand produces a sequence of a i , i ∈ [1 , N ] with a 1 assigned to the highest priority agent and thissequence aims to bring about the maximum return.
Mathematically, the action for the highest priorityagent (i.e., the a 1 ) is generated as below:
[Equation]
where h 0 = c 0 is the feature of state s , and supplied as the initial state of the RNN.
(cid:126) 0 is the input ofthe RNN in this micro-step and means no agent has been decided an action (can refer to Fig.
2(c)).cargmax (stands for consistent argmax) function means selecting the action for the highest priorityagent, i.e., R 1 according to the agreed action priority convention if ties exist.
RNN () function is aone-step forward function of the RNN, which is with only |A| output entries.
If ties exist in outputentries, argmax should consistently select the smallest optimal action according to action priority, sohereafter we write argmax as cargmax .
For the cargmax function, say, if RNN () outputs a valuevector of [5 . 1 , 5 . 32 , 5 . 2 , 5 . 32] , then we consistently select the ﬁrst "5.32" (index=1) in case of tieswhereas trivial argmax in pytorch may return unguaranteed and inconsistent index.
Remark 1: Better comprehended with CCPP in Fig. 1(b).
−→ Q 1 means the expected Q value for thewhole system when agent 1 selects each action from A , so −→ Q 1 [ j ] , j ∈ |A| tries to approximate
[Equation]
This approximation is possible since s contains all information of the current state including thetotal number of agents, and the following agents refer to and fully cooperate a 1 = action j .
(Eq. 2)can also be viewed from state mapping, i.e., −→ Q 1 : (grid states, agent state, total= N , priority=1 (orhighest))
→ 4 real values → one index.
This mapping can be learned by reinforcement learning.
Similarly, for the remaining agents, the planner conditions at previous agents’ actions and the priorityconventions as well, so
[Equation]
where onehot function maps an integer into a 0-1 input in which there is only one 1 in index a
i − 1 . Remark 2: Assume now i = 2 , then now this RNN ( . ) ( ∆ = −→ Q 2 )outputs
A entries and approximates to
[Equation]
where a 1 has been determined at previous micro-step and conditioned at this micro-step.
The approximation in each micro-step has physical meaning: If sufﬁciently trained, then
[Equation]
where function len is the number of output entries.
The physical meaning can be comprehended asthat the RNN is to approximate the optimal value of the monothetic system.
The last line is importantbecause V ∗ ( s ) links all the −→ Q ∗ i and can be exploited to achieve better learning.
Reward The reward at each step is the total reward that all agents earn.
The total reward isbeneﬁcial for learning and regards the system as a monolithic system.
In Fig.
1(c), the reward is thenumber of uncovered grids that the whole system visits at this step.
An uncovered grid will give areward of 1 only for the ﬁrst time it was visited.
If R 1 ∼ R 16 executed the solid arrow action, thereward would be 16.
Sometimes, a step penalty is added to the step reward so that the hypothesisreward [34] in RL holds 1 , so that, maximizing the return of the MDP ↔ minimizing the step to coverall shaded cells.
Training As Fig.
2 and Algorithm 1 depict, the training strategy needs specially designation.
Whenin state s and takes action (cid:126)a , the MDP will transfer to some s (cid:48) and get a reward r , then Q ( s,(cid:126)a ) shouldconverge to the target Q target ,
[Equation]
(12) where (cid:126)a (cid:48) can be selected out on the target network according to (Eq. 2) to (Eq. 4), rather thanenumerating all A N action combinations.
The Q ( s,(cid:126)a ) =
Q ( s, ( a 1 , a 2 , ..., a N ) can be calculated byfeeding (cid:126) 0 , onehot ( a 1 ) , ..., onehot ( a N − 1 ) to the RNN in the N micro-steps and gathering the a N -thentry in −→ Q N (see Fig. 2(c)).
Moreover, considering the uniqueness of our PCRL decision ﬂow and (Eq. 11), we can use anauxiliary loss to accelerate and smoothen the training.
The auxiliary loss is equalities among agents’ Q i values (as explained in (Eq. 11) and that paragraph), i.e.,
[Equation]
where ss and −→ aa are batch data from the experience replay buffer.
L aux guides the mean-squarederror of Q s of each agents to converge to target Q target .
Other type of (Eq. 13) is also possible.
We propose the following, and the proofs are in the appendix.
Theorem 1.
The action representation scheme and selection process will not lose the optimality ofthe action space and can select out the lexicographic-smallest optimal action vector.
Theorem 2.
The action selection process breaks the exponential action space into linearly expressiblespace complexity.
Experiments and Discussions
We evaluate the performance of PCRL on two seemingly different but both can be viewed asmicroscopically sequenceable multi-agent applications: multi-agent complete coverage path planning(CCPP) problem and Cooperative Pong (CP).
We compared our method with joint-action RL orheuristics on their efﬁcacy and efﬁciency.
All experiments were carried out on a computer of amdryzen 9 5950x, with a single 3090 RTX GPU.
Every method has run 10 times with different randomseeds at each experiment setting and reports the averages and the standard deviations (STD), to gainthe reliability of the evaluation.
The hyperparameters are listed in the appendix, also in the sourcecode that is supplemented.
Our experiments consist of (1) The PCRL’s LDAS dealing ability and
4.1 Multi-agent complete
coverage planning
PCRL illustration and consistent argmax Table 1 depicts the sufﬁciently trained (for 1 hour) andconverged results of the two-agent complete coverage path planning example on the 3 × 3 gridworld.
The ﬁrst column is the current state, where "1" stands for shaded grid cell and R stands for a agent.
The agents are homogeneous and the priorities are position-determined, so we ignore the ID of them.
The second column shows the outputs of the RNN (i.e., the Q values of RL) in two micro-steps.
The ﬁrst micro-step has the Q value for the highest priority agent.
It is the expectation for the ﬁrstagent to decide an action and the following agent to decide a corresponding action .
And the secondmicro-step is the second agent’s Q value referring to the ﬁrst agent’s action and fully supportingthe ﬁrst agent.
As can be seen, the bolded values are near, which is almost the V ∗ ( s )
(The groundtruth V ∗ ( s ) at each current state can be easily calculated).We can see that the total return receivedin the ﬁrst state is six, which is eight plus the penalty multiply minimal steps. The third column isthe optimal action for each agent, obeying to the priority. The fourth column is the reward for thewhole two-agent system.
The last line column is some explanation for data interpretation. Due to thecargmax function, this RNN has converged to the smallest lexicographic action vector at each state,and in training stage helped exploitation. [Table] [Table]
[Table] PCRL and DQN in deterministic transition environment CCPP
We compared the ﬁnal performance of PCRL, joint-action DQN on grid world on various numbers of agents.
The details of theresults are shown in the Fig.
Fig. 3(left) shows that when trained enough steps, PCRL can get a close ﬁnal performance to jointaction DQN.
That means the PCRL network can successfully learn the priority conventions andthe proper Q values, and has an ability similar to the monothetic system.
Moreover, PCRL has thepotential to deal with over 10 10 actions: the needed parameters of DQN are exponentially increasingwhile PCRL has a linearly increasing number of parameters, about 562176 for 12 agents (see themiddle of Fig. 3).
The performance of DQN is slightly better than PCRL, however the performancedecreases quite rapidly and soon fails to tackle 12 or more agents scenarios (see the right of Fig.
3,where the averaged return of PCRL is normalized as 100%). [Figure]
Large space and non-deterministic transition environment
The Fig.
4 depicts 18 agents on a9 × 9 grid world the same as Fig. 1(c), with 54 uncovered cells in total.
This case, at each step, ownsover 10 10 action vector candidates.
The experiment is further made tougher to a non-deterministic(stochastic) environment.
[-PCRLDQN 0 5E+ 1E+10 1.5E+ 2E+ 2 3 4 5 6 7 8 9 10 
-]
In the non-deterministic transition environment experiment, there is anon-zero probability p of “leftwards slipping” when agents take actions (say agent takes "up" then ithas p probability being moved to the "left" cell), where agents’ do not have any prior knowledge about the environment transition.
Fig.
4 shows the averaged score and averaged steps with sliding-windowwidth 100 in the training episode.
As shown, the training is converging gradually and smoothly.
Theaveraged steps is approaching 20 and the averaged return is about 44.
To further compare the steps, which is the focus of CCPP, we employ a heuristic search algorithmwhere the agent selects the minimum Manhattan distance cell and goes towards the cell.
The Violinplot is depicted in Fig.
5, where the ﬁrst "violin" is the performance of RNN in mid-course of training,while the second one is of the ﬁnal RNN after training.
The third one is for the heuristic algorithm.
PCRL can learn to optimize and use 20% smaller steps (19.7 VS 24.5) to cover all the shaded cells,and is more stable.
pong
Cooperative pong Fig. 1(d) is a game where the objective is to keep the ball in play for the longesttime.
The game is over when the ball goes out of bounds from either the left or right edge of thescreen.
All collisions of the ball are elastic.
Cooperative Pong has 480 pixels × 280 pixels, so a ballwith speed 9 will cost about 60 steps to go from the left paddle to the right paddle.
Algorithms thatcan keep the ball for 400 steps are rare [35].
The CP can be established as a joint-action DQN withand a microscopically sequenceable multi-agent problem.
To show the applicability of PCRL in CPtask, we applied PCRL on the state-based and pixel-based CP, respectively.
State-based CP and pixel-based CP We trained PCRL and the joint-action DQN on CP with asimilar scale of network parameters, and the performance is illustrated in Fig.
6a.
Set {+ball_speed=
+}
We can see
even outperforms the joint which treats the two agents as a monothetic agent.
The keyreason for this result may be that PCRL has higher parameters efﬁciency than DQN of a similar scaleof parameters.
To better demonstrate the effectiveness of our approach, we also use the pixel states as the Q-netinputs.
We resize the original pixel state to 84 * 84 * 3 tensor.
We use a 3-layer convolution networktransforming the pixel state to a vector state ﬁrst, and then feed the vector state as the RNN hiddenstate.
The performance in training with pixel states (in about four hours) is shown in Fig.
6b, fromwhich we can see that our algorithm converges after about 1000 episodes and get a satisfactoryperformance.
It indicates that PCRL is effective and efﬁcient even in the end-to-end setting withoutlosing much performance compared to the state-based pong performance.      (SLVRGHV         [Figure]
Conclusion
This paper tackled the microscopically sequenceable multi-agent problems via reinforcement learningthat combines agreed priority conventions.
The agreed priority conventions can successfully breakthe exponential scale actions into a sequence of actions (linear scale expressible) without losingoptimality.
From the multi-agent complete coverage planning and cooperative pong experiments, thealgorithm shows closeness or superiority in effectiveness and efﬁciency to the joint-action DQN andheuristics.
The results suggest the potentials of PCRL for a larger number of multi-agent systems(say recommending systems) or swarm robots.
In the future, further work can be done, such as (1) larger agent number multi-agent systems toverify the ability of PCRL for larger action space problems, say swarm robots.
(2) application ofthis algorithm to fully-distributed or centralized-training-and-decentralized-execution multi-agentsystems.
(3) studies of PCRL for heterogeneous agents.
(4) studies on partial observable domains.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes] See the abstract and introduction.
Moreover,
The
abstract, introduction and experiment results are consistent.
(b) Did you describe the limitations of your work?
[Yes] See the conclusion part.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
The
technique is neutral.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] Our paper is strictly comforms to them.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes]
In the
supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they [Table]
were chosen)?
[Yes]
In the article, appendix and codes.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Each ran at least ten times.
[Table] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
In an ordinary desktop computer.
[Table]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[No]
[Table]
(b) Did you mention the license of the assets?
[No]
[Table] (c) Did you include any new assets either in the supplemental material or as a URL? [No] [Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re [Table] using/curating?
[No] [Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable [Table] information or offensive content?
[No]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[No]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[No]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[No]
[Table] A Appendix A.1 Pseudocode of PCRL
1: Initialize replay memory D ; Initialize the evaluation RNN (the action-value function Q ) with random weights θ ; Initialize the target RNN (the action-value function ˆ Q ) with weights θ − = θ 2: for episode=1 to E do 3:
Observe initial state s 4: while unterminated do 5:
Select an action (cid:126)a : with probability ε select a random action vector by uniform sampling; otherwise, generate (cid:126)a as Fig.
2 and (Eq. 2) to (Eq. 4) 6:
Carry out action (cid:126)a 7:
Observe reward r and new state s (cid:48) 8:
Store experience < s,(cid:126)a, r, s (cid:48) > into replay memory D 9:
Sample random transitions < ss, −→ aa, rr, ss (cid:48) > from replay memory D 10:
Calculate target for each minibatch transition: if ss (cid:48) is terminal state, then tt =
rr otherwise tt =
rr + γ ∗ max (cid:126)a (cid:48)
ˆ Q (cid:16) ss (cid:48) , (cid:126)a (cid:48) (cid:17) where (cid:126)a (cid:48) is generated as Fig. 2(c) and (Eq. 2) to (Eq. 4) 11:
Train the evaluation Q -network using ( tt − Q ( ss, −→ aa ))
2 + L aux ( ss, −→ aa, rr, ss (cid:48) )
as loss with respect to θ , where L aux is the auxiliary convergence loss in (Eq. 11) and (Eq. 13) 12:
Update target network as θ (cid:48)
= (1 − τ ) θ (cid:48)
+ τθ 13: s = s (cid:48) 14: end while 15: end for
A.2 Microscopic generation proofs
Theorem: The action representation scheme and selection process will not lose the optimality of theaction space and can select out the lexicographic-smallest optimal action vector.
Proof.
Suppose (cid:126)α =
( α 1 , α 2 , ..., α N ) and (cid:126)β = ( β 1 , β 2 , ..., β N ) are two (More than two tie optimalaction vectors can also hold similar conclusion.)
optimal tie actions for a state s , and in the articlehas deduced that V ∗ ( s )
=
Q ( s, (cid:126)α ) =
Q ( s, (cid:126)β ) .
Since the state s is the same, i.e., the map and agent states are the same, thus the agent priority ordersare identical.
Suppose (cid:126)α is lexicographic smaller than (cid:126)β (this is possible, since each component of (cid:126)αand (cid:126)β are comparable according to agreed action priority, so either (cid:126)α is lexicographic smaller than (cid:126)βor (cid:126)β is lexicographic smaller than (cid:126)α .
The latter can prove the following similarly).
Let j = min
i ( α i ≺ β i ) , the planner’s procedure will select α i instead of β i for agent i .
Since the smallest-lexicographic action vector can be selected out and learned, PCRL will not losethe optimality for the concerned problem.
Theorem: The action selection process breaks the exponential action space into linearly expressiblespace complexity.
Proof.
The recurrent neural network has N micro-step outputs, each micro-step output consists of| A | actions.
Suppose that at each micro-step, the planner can select one action, so the possible totalaction combination space is still | A | N , so RNN will not miss expressing the optimal vector.
However,the optimum (maximum) action vector can be expressed by and selected from |A| per micro-step ×
N micro-steps.
A.3 Common hyper-parameters
The common hyper-parameters for experiments are:
[Table]
The auxiliary loss type 1 is as (Eq. 11), and type 2 are the mean squared error of max Q 1 to max Q 2 ,max Q 2 to max Q 3 etc.
This two can get similar results.
The γ of CCPP is set as listed because of the reward hypothesis.
Reinforcement learning is based on the reward hypothesis, that is, RL is to maximize G t = 457 (cid:80) ∞ {+k =
+}
γ k r t + k +1 .
When t = 0 , it is maximizing G = (cid:80) ∞ k =
γ k r k +
= R 1 + γ ∗ r 2 + γ 2 ∗ r 3 + ... ,i.e., the maximum expected return from the initial setting of the environment.
If one wants to maximize the reward G to get the fewest step coverage route of the agent, then oneneeds to set appropriate parameters such as γ so that G ﬁts the aim.
Generally, 0 ≤ γ ≤ 1 , when γis 0, it is a single-step reﬂective and myopic behavior.
In this appendix, we want to maximize thereward G to obtain the fewest steps of multi-agent complete coverage.
Because the multi-agent istreated as a monothetic system, so one agent’s proofs can naturally be extended to a multi-agentsystem.
Through mathematical analysis, the following theorems and proofs can be obtained:
Theorem 3.
When γ < 1 and there is no single-step penalty for each step, maximizing the returndoes not always get the fewest step complete coverage.
When γ = 1 and each step has a single-steppenalty (step_penalty < 0), maximizing the return is equivalent to the fewest step complete coverage.
Before the proof, use Fig.
7 to illustrate the ﬁrst sentence when γ < 1 .
The ﬁgure shows the sameenvironment conﬁguration (setting), including the same map state and agent position.
The two pathsof the agent are path (1) and path (2).
Denote their rewards are respectively G (1) , G (2) .
It can be seenin the ﬁgure that path (1) is shorter than path (2).
However, when γ < 1 and each step has no penalty,the return of path (2) is larger, that is, G (1) < G (2) :
According to the return formula, the return forpaths (1) and (2) is G (1) =
1 + γ 2 + γ 3 , G (2)
=
+ γ 1 + γ 4 , at this time
[Equation]
[Equation] [Equation]
[Equation]
The last step is because when 0 < γ < 1 , γ 2 < γ 1 and 1 > γ 2 .
However, When γ = 1 and a single-step penalty step _ penalty = − 0 . 2 , the return of path (1) is 2.2,and the return of path (2) is 2.0.
This ﬁts the requirements of CCPP: fewest steps and maximumreturn.
From the illustration of the ﬁgure, a mathematical proof [Figure]
Proof.
Because there are examples shown in Fig.
7, the ﬁrst sentence of the theorem can be provedby contradiction.
For the second sentence of the theorem, it can be equivalent to "When γ = 1 , andeach step has a single-step penalty (step_penalty<0), a greater return is equivalent to a fewer stepcomplete coverage path".
The idea of the proof is to assume that there are path (1) and path (2), and the number of steps usedare n and m , respectively.
Assume n < m, n > 0 , m > 0 and step _ penalty < 0 .
First to prove the necessity, that is, under the same environment conﬁguration, fewer steps will get agreater return, that is, one needs to prove G (1) >
G (2) .
Denote C i the cell at step i and C i = 1 if thecell has never been visited.
Because G (1) = ( C 1 + step _ penalty ) ∗ γ
+( C 2 + step _ penalty ) ∗ γ 1 + ...
+( C n + step _ penalty )
∗γ n − 1 =
( C 1 + step _ penalty )
+ ( C 2 + step _ penalty )
+ ...
+ ( C n + step _ penalty )
=
( C 1 + C 2 +... + C n ) + n ∗ step _ penalty =
number _ of _ uncovered _ cells + n ∗ step _ penalty .
Similarly, G (2) =
( C 1 + C 2 + ... + C m )
+ m ∗ step _ penalty =
number _ of _ uncovered _ cells +m ∗ step _ penalty .
Due to the penalty of each step, we get G (1) > G (2) , that is, a smaller numberof steps gets a higher reward.
Proof of adequacy, that is, greater returns can lead to fewer steps.
Because the rewards of a scene canbe written as number _ of _ uncovered _ cells + k ∗ step _ penalty , where k is the number of stepsvisiting all shaded cells, a larger reward corresponds to a smaller number of steps.
A.4 Planning difﬁculty for CCPP
CCPP can be established as a Euclidean Traveling Salesman Problem which is NP-hard.
Here weestablish it as an MDP.
To realize the difﬁculty of the problem to set a proper problem scale, we ﬁrstconducted a agent random wandering experiment to cover all grids and the experimental results areshown in Fig.
8: the x -axis represents the n of the square n × n gridworld, where the y -axis is theexpected steps (averaged among 1000 runs) for a agent to visit all n 2 − 1 cells (the start cell is indown-rightmost and is empty).
It can be seen that the number of steps required increases almostexponentially (the log linearly interpolation curve has been ﬁt and has been plotted in the sameﬁgure).
When n is 10, the number of random steps to completely cover all cells is more than 1000steps, so we set it as the scale of the experiments.
Roughly estimated, the total number of paths for aproblem scale of 10 × 10 map size and 2 agents can reach 100!
(about 10 158 ).
So, this pre-experimenthas partially shown the complexity of problems that PCRL can tackle and have strong optimizationability.
A.5 Deterministic transition environment of 18 agents CCPP
Fig.
9 shows the settings of Fig. 1(c) and with a deterministic transition environment.
It shows thereturn and steps of each training episode, the training cost and the sliding-window averaged steps.
The ground truth return is 52.5 and PCRL sometimes can get 51.5 and an average near 50.
Thispicture again shows the LDAS dealing ability and high optimality-ﬁnding ability of PCRL. [Figure]
As shown in (Eq. 11), the maximum of −→ Q 1 will equal to the maximum of −→ Q 2 at the state s , theoretically.
Respecting this, we evaluate our trained neural network to analyze the convergence of Q i s. Deﬁnethe gap as (max −→ Q 2 − max −→ Q 1 ) / max −→ Q 1 .
The left part of Fig.
10 shows the gap at every state alongone test episode, while the right part shows the averaged gap in the training history of an RNN.
Theformer demonstrates an only 5% gap in the states.
The latter demonstrates that as training goes, thegap from substantially large becomes narrow and narrow.
Both parts proved the convergence of gapsbetween −→ Q i , i = 1 , ..., N .
         (SLVRGHV  
      
U H O D W L Y H  H UU R U V  U D W L R    4YDOXHUHODWLYHHUURUVUDWLRRQHHSLVRGH        (SLVRGHV          U H O D W L Y H  H UU R U V  U D W L R    4YDOXHUHODWLYHHUURUVUDWLR