Original: M ULTI -A GENT N EURAL

Original:  R EWRITER FOR V EHICLE R OUTING WITH L IMITED D ISCLOSURE OF C OSTS
Revised: M ULTI -A GENT N EURAL R EWRITER FOR V EHICLE R OUTING WITH L IMITED D ISCLOSURE OF C OSTS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We interpret solving the multi-vehicle routing problem as a team Markov game with partially observable costs.
Revised: We interpret solving the multi-vehicle routing problem as a team Markov game with partially observable costs.

Original: For a given set of customers to serve, the playing agents (vehicles) have the common goal to determine the team-optimal agent routes with minimal total cost.
Revised: For a given set of customers to serve, the playing agents (vehicles) have the common goal to determine the team-optimal agent routes with minimal total cost.

Original: Each agent thereby observes only its own cost.
Revised: Each agent thereby observes only its own cost.

Original: Our multi-agent reinforcement learning approach, the so-called multi-agent Neural Rewriter, builds on the single-agent Neural Rewriter to solve the problem by iteratively rewriting solutions.
Revised: Our multi-agent reinforcement learning approach, the so-called multi-agent Neural Rewriter, builds on the single-agent Neural Rewriter to solve the problem by iteratively rewriting solutions.

Original: Parallel agent action execution and partial observability require new rewriting rules for the game.
Revised: Parallel agent action execution and partial observability require new rewriting rules for the game.

Original: We propose the introduction of a so-called pool in the system which serves as a collection point for unvisited nodes.
Revised: We propose the introduction of a so-called pool in the system which serves as a collection point for unvisited nodes.

Original: It enables agents to act simultaneously and exchange nodes in a conflict-free manner.
Revised: It enables agents to act simultaneously and exchange nodes in a conflict-free manner.

Original: We realize limited disclosure of agent-specific costs by only sharing them during learning.
Revised: We realize limited disclosure of agent-specific costs by only sharing them during learning.

Original: During inference, each agents acts decentrally, solely based on its own cost.
Revised: During inference, each agents acts decentrally, solely based on its own cost.

Original: First empirical results on small problem sizes demonstrate that we reach a performance close to the employed OR-Tools benchmark which operates in the perfect cost information setting.
Revised: First empirical results on small problem sizes demonstrate that we reach a performance close to the employed OR-Tools benchmark which operates in the perfect cost information setting.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: While the logistics market is typically determined by strongly competing players, there also exist initiatives to encourage collaboration for, e.g., more cost-efficient and sustainable resource usage 12 .
Revised: While the logistics market is typically determined by strongly competing players, there also exist initiatives to encourage collaboration for, e.g., more cost-efficient and sustainable resource usage 12 .

Original: A simplified model of the logistics market is given by a set of customers and a set of vehicles belonging 
Revised: A simplified model of the logistics market is given by a set of customers and a set of vehicles belonging to different logistics companies with vehicle-specific costs.

Original: The setup of cooperating companies, which seek to jointly optimize the overall costs while serving all customers, is formalized by the (multi-)vehicle routing problem (Laporte, 1992) (cf.
Revised: The setup of cooperating companies, which seek to jointly optimize the overall costs while serving all customers, is formalized by the (multi-)vehicle routing problem (Laporte, 1992) (cf.

Original: Figure 1).
Revised: Figure 1).

Revised: It involves assigning customers to the companies as well as determining the corresponding vehicle routes.

Original: A natural additional constraint is that companies  want to keep certain types of their costs private.
Revised: A natural additional constraint is that companies hereby want to keep certain types of their costs private.

Original: E.g., company-specific driver costs, fuel consumptions and also fixed costs for marketing or customer service, reflect businessinternal information.
Revised: E.g., company-specific driver costs, fuel consumptions and also fixed costs for marketing or customer service, reflect business-internal information.

Original: Sharing such information would lead to a decisive competitive disadvantage.
Revised: Sharing such information would lead to a decisive competitive disadvantage.

Original: We interpret solving the multi-vehicle routing problem as a team Markov game (Wang & Sandholm, 2002) with partially observable costs: The parallel acting vehicles play a cooperative game to build the (ideally) team-optimal vehicle routes, while each vehicle can observe only its own local cost information.
Revised: We interpret solving the multi-vehicle routing problem as a team Markov game (Wang & Sandholm, 2002) with partially observable costs: The parallel acting vehicles play a cooperative game to build the (ideally) team-optimal vehicle routes, whereby each vehicle can observe only its own local cost information.

Revised: While other approaches for collaborative vehicle routing have studied the scenario of self-interested vehicles (companies) which can form coalitions involving only a subset of vehicles (Mak et al., 2021), we focus on the setup of selfless, team-oriented vehicles which all participate in the collaboration.

Original: As reinforcement learning (RL) has lately become very attractive for learning heuristics for NP-hard combinatorial optimization problems like vehicle routing problems (Nazari et al., 2018; Kool et al., 2019; Chen & Tian, 2019), we build on an existing RL framework to implement our game.
Revised: As reinforcement learning (RL) has lately become very attractive for learning heuristics for NP-hard combinatorial optimization problems like vehicle routing problems (Nazari et al., 2018; Kool et al., 2019; Chen & Tian, 2019), we build on an existing RL framework to implement our game.

Original: The so-called Neural Rewriter (Chen & Tian, 2019) considers the setup of a single vehicle and learns how to iteratively improve a given solution within a rewriting episode of fixed length.
Revised: The so-called Neural Rewriter (Chen & Tian, 2019) considers the setup of a single vehicle and learns how to iteratively improve a given solution within a rewriting episode of fixed length.

Original: In each step of the episode the current routing solution, given by a sequence of visited nodes, gets slightly changed by swapping two nodes.
Revised: In each step of the episode the current routing solution, given by a sequence of visited nodes, gets slightly changed by swapping two nodes.

Original: Our proposed multiagent Neural Rewriter (MANR) extends the Neural Rewriter to a multi-agent system with multiple vehicles (agents) with individual costs.
Revised: Our proposed multi-agent Neural Rewriter (MANR) extends the Neural Rewriter to a multi-agent system with multiple vehicles (agents) with individual costs.

Original: We assume the individual cost matrices to originate from the same underlying distribution, i.e., we consider a set of homogeneous agents.
Revised: We assume the individual cost matrices to originate from the same underlying distribution, i.e., we consider a set of homogeneous agents.

Original: To ensure the required parallel action execution in the team Markov game, we have to avoid conflicting rewriting actions of agents.
Revised: To ensure the required parallel action execution in the team Markov game, we have to avoid conflicting rewriting actions of agents.

Original: The assumption of partially observable costs poses additional requirements on the rewriting game structure:
Revised: The assumption of partially observable costs poses additional requirements on the rewriting game structure:

Original: Given the incomplete knowledge about its team members, an agent must never modify the routes of other agents but can solely alter its own local route.
Revised: Given the incomplete knowledge about its team members, an agent must never modify the routes of other agents but can solely alter its own local route.

Original: Yet at the same time, agents must be able to exchange nodes.
Revised: Yet at the same time, agents must be able to exchange nodes, i.e., to change the customer-company-assignment.

Original: This raises the question: How should agents know whether excluding a  node or integrating a new  node is beneficial for the team if they only observe their own costs?
Revised: This raises the question: How should agents know whether excluding a customer node or integrating a new customer node is beneficial for the team if they only observe their own costs?

Original: There is a trade-off between non-disclosure and optimality; some sort of information exchange between the team members is inevitable for finding a global optimum.
Revised: There is a trade-off between non-disclosure and optimality; some sort of information exchange between the team members is inevitable for finding a global optimum.

Original: We solve the described challenges on different levels of the MANR.
Revised: We solve the described challenges on different levels of the MANR.

Original: On the game setup level, the key is the introduction of a pool set to coordinate agent actions and prevent conflicts.
Revised: On the game setup level, the key is the introduction of a pool set to coordinate agent actions and prevent conflicts.

Original: It serves as a collection point for agents to drop  nodes and take  nodes from.
Revised: It serves as a collection point for agents to drop customer nodes and take customer nodes from.

Original: Providing the local agents with some global knowledge for deciding about exchanging  nodes is ensured by the employed learning approach.
Revised: Providing the local agents with some global knowledge for deciding about exchanging customer nodes is ensured by the employed learning approach.

Original: We ultimately seek to limit the necessary exchange of cost information between the players as much as possible.
Revised: We ultimately seek to limit the necessary exchange of cost information between the players as much as possible.

Original: As a first step, we realize limited disclosure of vehicle-specific costs only during inference and consider shared costs during training:
Revised: As a first step, we realize limited disclosure of vehicle-specific costs only during inference and consider shared costs during training.

Revised: This is a valid assumption from an application perspective as training can be performed based on fictitious company cost information sampled from a realistic cost distribution.

Original: Our actor-critic setup follows the idea of centralizedlearning-decentralized-execution (Zhang et al., 2021).
Revised: Our actor-critic setup follows the idea of centralized-learning-decentralized-execution (Zhang et al., 2021):

Original: During training, all vehicle cost information is shared to learn a centralized critic for estimating the expected team benefit of rewriting actions.
Revised: During training, all vehicle cost information is shared to learn a centralized critic for estimating the expected team benefit of rewriting actions.

Original: The critic passes this global knowledge on to an agent policy, which itself observes only local information.
Revised: The critic passes this global knowledge on to an agent policy, which itself observes only local information.

Original: An agent thus learns to locally behave as a team player, assuming a representative team.
Revised: An agent thus learns to locally behave as a team player, assuming a representative team.

Original: During inference, each agent uses only his local cost (and not the cost of other agents) to determine his actions.
Revised: During inference, each agent uses only his local cost (and not the cost of other agents) to determine his actions.

Original: We give an overview of the related work in Section 2.
Revised: We give an overview of the related work in Section 2.

Original: In Section 3 we define the considered collaborative vehicle routing problem and present and discuss the adapted Neural Rewriter, the so-called multi-agent Neural Rewriter as a solution approach.
Revised: In Section 3 we define the considered collaborative vehicle routing problem and present and discuss the adapted Neural Rewriter, the so-called multi-agent Neural Rewriter as a solution approach.

Original: Section 4 empirically evaluates the MANR on simulated data for different setups varying the size of the routing problem as well as the number of vehicles and compares its performance to OR-Tools.
Revised: Section 4 empirically evaluates the MANR on simulated data for different setups varying the size of the routing problem as well as the number of vehicles and compares its performance to OR-Tools.

Original: Section 5 summarizes the results and outlines plans for future work.
Revised: Section 5 summarizes the results and outlines plans for future work.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: RL for vehicle routing RL has been successfully used to tackle NP-hard vehicle routing problems.
Revised: RL for vehicle routing RL has been successfully used to tackle NP-hard vehicle routing problems.

Original: Most of the work focuses on setups involving a single vehicle.
Revised: Most of the work focuses on setups involving a single vehicle.

Original: E.g., Chen & Tian (2019) consider a capacitated vehicle routing problem where a single vehicle has to visit a set of customers in (possibly) multiple tours starting and ending at its depot without exceeding the vehicle’s capacity within one tour.
Revised: E.g., Chen & Tian (2019) consider a capacitated vehicle routing problem where a single vehicle has to visit a set of customers in (possibly) multiple tours starting and ending at its depot without exceeding the vehicle’s capacity within one tour.

Original: Their approach follows the idea of local search.
Revised: Their approach follows the idea of local search, a well-established heuristic solution approach in the operations research community.

Original: Based on an initial solution, they iteratively create locally rewritten solutions via local modifications.
Revised: Based on an initial solution, they iteratively create locally rewritten solutions via local modifications.

Original: They define a local modification as swapping two nodes in the current routing solution sequence and let a RL algorithm learn how to create good neighbouring solutions therewith.
Revised: They define a local modification as swapping two nodes in the current routing solution sequence and let a RL algorithm learn how to create good neighbouring solutions therewith.

Original: Other RL-based approaches for vehicle routing typically build a solution sequentially by visiting one customer more in each step of the episode.
Revised: Other RL-based approaches for vehicle routing typically build a solution sequentially by visiting one customer more in each step of the episode.

Original: E.g., both Kool et al.
Revised: E.g., both Kool et al.

Original: ; Nazari et al.
Revised: ; Nazari et al.

Original: (2018) consider multiple variants of vehicle routing problems which however also concern only a single vehicle.
Revised: (2018) consider multiple variants of vehicle routing problems which however also concern only a single vehicle.

Original: Nazari et al.
Revised: Nazari et al.

Original: (2018) describe the idea of multiple vehicles in a collaborative or competetive setting as an interesting direction and already mention the need to solve emerging conflicts between multiple vehicles.
Revised: (2018) describe the idea of multiple vehicles in a collaborative or competetive setting as an interesting direction and already mention the need to solve emerging conflicts between multiple vehicles.

Original: We decided to build on the rewriting approach of Chen & Tian (2019) as itnaturally allowsagents to perceive an immediate reward for a rewritten solution in each step of  the episode and thus avoids a challenging sparse reward environment.
Revised: We decided to build on the rewriting approach of Chen & Tian (2019) as it allows to handle conflicts more naturally. In an iteratively built solution agents either need to perform their actions sequentially to avoid conflicts, which would violate our assumption of a Markov game with simultaneously acting agents, or they need a (stochastically influenced) a posterio conflict solver which would be challenging for agents to learn about. The rewriting approach allows to introduce a pool mechanism for ensuring conflict-free parallel agent actions (details are given below). Moreover, the rewriting setup avoids a challenging sparse reward environment: At each step the cost of the current solution indicates the quality of this solution.

Original: Limited disclosure in multi-agent RL
Revised: Limited disclosure in multi-agent RL

Original: In our setup, agents seek to keep their local costs private, whereby these costs influence the reward and also local state representations.
Revised: In our setup, agents seek to keep their local costs private, whereby these costs influence the reward and also local state representations.

Original: Limiting the disclosure of such agent-specific information is an active research topic for both cooperative and competitive multi-agent systems.
Revised: Limiting the disclosure of such agent-specific information is an active research topic for both cooperative and competitive multi-agent systems.

Original: The extreme case of independent learning (Matignon et al., 2012) considers agents as independent entities which observe solely their local environment and are not allowed to share information at all.
Revised: The extreme case of independent learning (Matignon et al., 2012) considers agents as independent entities which observe solely their local environment and are not allowed to share information at all.

Original: Other approaches allow for some sort of information exchange as it is known to generally improve performance over independent learning for cooperative tasks (Tan, 1993).
Revised: Other approaches allow for some sort of information exchange as it is known to generally improve performance over independent learning for cooperative tasks (Tan, 1993).

Original: In centralized-learning-decentralized-execution schemes (Zhang et al., 2021; Lowe et al., 2017), one typically assumes perfect global knowledge about the agents’ local states, actions and rewards during training.
Revised: In centralized-learning-decentralized-execution schemes (Zhang et al., 2021; Lowe et al., 2017), one typically assumes perfect global knowledge about the agents’ local states, actions and rewards during training.

Original: This information is leveraged to train policies which themselves observe solely local information.
Revised: This information is leveraged to train policies which themselves observe solely local information.

Original: For execution, only the policies are used in a decentral manner and thus guarantee limited disclosure during inference.
Revised: For execution, only the policies are used in a decentral manner and thus guarantee limited disclosure during inference.

Original: The literature discusses also a stricter interpretation of limited disclosure by additionally requiring it during the training phase.
Revised: The literature discusses also a stricter interpretation of limited disclosure by additionally requiring it during the training phase.

Original: One possible approach is to share solely locally learned model parameters instead of the local raw data in spirit of distributed machine learning (McMahan et al., 2017).
Revised: One possible approach is to share solely locally learned model parameters instead of the local raw data in spirit of distributed machine learning (McMahan et al., 2017).

Original: E.g., in Zhang et al.
Revised: E.g., in Zhang et al.

Original: (2018), the cooperating agents jointly learn a global value function by sharing the agents’ individual parameters of their local estimate of the global value function over a time-varying communication network.
Revised: (2018), the cooperating agents jointly learn a global value function by sharing the agents’ individual parameters of their local estimate of the global value function over a time-varying communication network.

Original: As a first step, we consider the scenario of limited disclosure during inference, realized by a centralized-learning-decentralized-execution approach.
Revised: As a first step, we consider the scenario of limited disclosure during inference, realized by a centralized-learning-decentralized-execution approach.

Original: To the best of our knowledge, there exists no
Revised: To the best of our knowledge, there exists no

Original: RL-based approach to solve a multi-vehicle routing problem with simultaneously acting cooperating agents which can observe solely their own individual cost.
Revised: RL-based approach to solve a multi-vehicle routing problem with simultaneously acting cooperating agents which can observe solely their own individual cost.

Original:  M ULTI -A GENT N EURAL R EWRITER
Revised:  M ULTI -A GENT N EURAL R EWRITER

Original: The developed multi-agent Neural Rewriter (MANR) uses building blocks from the original Neural Rewriter (Chen & Tian, 2019) to solve a multi-vehicle routing problem with vehicle-specific costs and depot locations in spirit of a team Markov game with partially observable costs.
Revised: The developed multi-agent Neural Rewriter (MANR) uses building blocks from the original Neural Rewriter (Chen & Tian, 2019) to solve a multi-vehicle routing problem with vehicle-specific costs and depot locations in spirit of a team Markov game with partially observable costs.

Original: During conception of the multi-agent setup, we considered the following guiding principles:
Revised: During conception of the multi-agent setup, we considered the following guiding principles:

Original: • Self-determined agents: Whether and how an agent’s route is changed can be decided only by the agent itself and not by other agents.
Revised: • Self-determined agents: Whether and how an agent’s route is changed can be decided only by the agent itself and not by other agents.

Original: This is essential since, due to the partial cost observability, the agents do not have the necessary information for deciding about optimal routes of other agents.
Revised: This is essential since, due to the partial cost observability, the agents do not have the necessary information for deciding about optimal routes of other agents.

Original: • Conflict-free parallel decisions: Any decision of an agent cannot be in conflict with other agent decisions at the same time step.
Revised: • Conflict-free parallel decisions: Any decision of an agent cannot be in conflict with other agent decisions at the same time step.

Original: A naive extension of the Neural Rewriter to multiple vehicles by retaining the original action (and state) space locally for single agents does not satisfy these criteria:
Revised: A naive extension of the Neural Rewriter to multiple vehicles by retaining the original action (and state) space locally for single agents does not satisfy these criteria:

Original: An agent swapping any two customer nodes in the overall solution can result in changing another agent’s route and also gives room for contradicting agent actions.
Revised: An agent swapping any two customer nodes in the overall solution can result in changing another agent’s route and also gives room for contradicting agent actions.

Original: Solving these issues by simply restricting agents to swap only customer nodes in their own routes would prohibit the necessary node exchange between agents.
Revised: Solving these issues by simply restricting agents to swap only customer nodes in their own routes would prohibit the necessary customer exchange between agents.

Original: Hence, we needed to redefine the rules for our team Markov game.
Revised: Hence, we needed to redefine the rules for our team Markov game.

Original: The approach we propose is the introduction of a pool set over which single agents can get rid of  nodes or integrate nodes.
Revised: The approach we propose is the introduction of a pool set over which single agents can get rid of customer nodes or integrate new ones.

Original: Agents can interact with each other only via the pool, a direct interaction is not possible.
Revised: Agents can interact with each other only via the pool, a direct interaction is not possible.

Original: The pool coordinates the node exchange and with this guarantees the self-determination of agents as well as the conflict-free agent decisions which will be described below in more detail. ”
Revised: The pool coordinates the node exchange and with this guarantees the self-determination of agents as well as the conflict-free agent decisions which will be described below in more detail. ”

Original: Agentizing” the
Revised: Agentizing” the

Original: Neural Rewriter also required adaptions of the used models.
Revised: Neural Rewriter also required adaptions of the used models.

Original: For a detailed listing of modifications the reader is referred to Appendix A.1.
Revised: For a detailed listing of modifications the reader is referred to Appendix A.1.

Original: In the following, we first introduce the problem statement in
Revised: In the following, we first introduce the problem statement in

Original: Section 3.1 and our translation to a team Markov game in Section 3.2.
Revised: Section 3.1 and our translation to a team Markov game in Section 3.2.

Original: The implementation of the game with the corresponding MANR model components is described in Section 3.3.
Revised: The implementation of the game with the corresponding MANR model components is described in Section 3.3.

Original: 3. P ROBLEM S TATEMENT
Revised: 3. P ROBLEM S TATEMENT

Original: We consider a multi-vehicle routing problem where n vehicles (agents), characterized by individual costs and depots, collaborate on serving a set of customers V .
Revised: We consider a multi-vehicle routing problem where n vehicles (agents), characterized by individual costs and depots, collaborate on serving a set of customers V .

Original: Each customer node v ∈ V has to be visited exactly once in total by any of the agents.
Revised: Each customer node v ∈ V has to be visited exactly once in total by any of the agents.

Original: Each agent route starts and ends at its own depot.
Revised: Each agent route starts and ends at its own depot.

Original: A collection of agent routes which satisfies these two criteria is called a feasible solution.
Revised: A collection of agent routes which satisfies these two criteria is called a feasible solution.

Original: The goal is to find an optimal solution which is a feasible solution and has minimal team average costs (cf.
Revised: The goal is to find an optimal solution which is a feasible solution and has minimal team average costs (cf.

Original: Figure 1).
Revised: Figure 1).

Original: The team average cost is given by the average over all agent route costs.
Revised: The team average cost is given by the average over all agent route costs.

Original: 3. T EAM M ARKOV G AME
Revised: 3. T EAM M ARKOV G AME

Original: Chen & Tian (2019) modelled the rewriting procedure of local search for a one-vehicle routing problem as a Markov decision process: States represent routing solutions, actions local modifications of a solution and rewards indicate the cost improvement of a local modification.
Revised: Chen & Tian (2019) modelled the rewriting procedure of local search for a one-vehicle routing problem as a Markov decision process: States represent routing solutions, actions local modifications of a solution and rewards indicate the cost improvement of a local modification.

Original: In our multi-agent system we generally differentiate between two perspectives on the problem: the local perspectives of single agents referring to their own local states and actions and the global perspective which observes all agents’ states and actions simultaneously.
Revised: In our multi-agent system we generally differentiate between two perspectives on the problem: the local perspectives of single agents referring to their own local states and actions and the global perspective which observes all agents’ states and actions simultaneously.

Original: We consider a single global episode which involves rewriting global states with global actions where all agents execute one local action in parallel.
Revised: We consider a single global episode which involves rewriting global states with global actions where all agents execute one local action in parallel.

Original: The system obtains global rewards which reflect the overall success of rewriting steps for the whole agent team.
Revised: The system obtains global rewards which reflect the overall success of rewriting steps for the whole agent team.

Original: To address the challenge of non-conflicting parallel local agent actions, we establish the pool set and model it as an additional component in the system’s global state.
Revised: To address the challenge of non-conflicting parallel local agent actions, we establish the pool set and model it as an additional component in the system’s global state.

Original: The pool offers the agents the opportunity to drop customer nodes which they want to exclude from their routes and also to integrate new customer nodes.
Revised: The pool offers the agents the opportunity to drop customer nodes which they want to exclude from their routes and also to integrate new customer nodes.

Original: It also plays an important role for the global team reward as improper usage of the pool leads to a collective penalty.
Revised: It also plays an important role for the global team reward as improper usage of the pool leads to a collective penalty.

Original: The precise Markov game setup including the kinds of agent interaction with the pool is described in the following.
Revised: The precise Markov game setup including the kinds of agent interaction with the pool is described in the following.

Original: States
Revised: States

Original: A global state s t =
Revised: A global state s t =

Original: ( s 1 t , s 2 t , ..., s nt , P t ) at time t is defined by the concatenation of all local agent states at time t and the corresponding current state of the pool P t .
Revised: ( s 1 t , s 2 t , ..., s nt , P t ) at time t is defined by the concatenation of all local agent states at time t and the corresponding current state of the pool P t .

Original: A local state s it of agent i at time t is given by the agent’s route at time t characterized by the sequence of visited nodes starting and ending at the agent’s depot.
Revised: A local state s it of agent i at time t is given by the agent’s route at time t characterized by the sequence of visited nodes starting and ending at the agent’s depot.

Original: The pool state P t is a set of nodes which is either empty or contains customer nodes which were dropped there by agents and which are thus unvisited at that time step.
Revised: The pool state P t is a set of nodes which is either empty or contains customer nodes which were dropped there by agents and which are thus unvisited at that time step.

Original: We note that not all global states are necessarily feasible solutions to the routing problem but only those with an empty pool.
Revised: We note that not all global states are necessarily feasible solutions to the routing problem but only those with an empty pool.

Original: See Figure 2 for exemplary global states.
Revised: See Figure 2 for exemplary global states.

Original: Actions A global action a t =
Revised: Actions A global action a t =

Original: ( a 1 t , a 2 t , ...a nt ) at time t is defined by the concatenation of all local agent actions at time t .
Revised: ( a 1 t , a 2 t , ...a nt ) at time t is defined by the concatenation of all local agent actions at time t .

Original: A local action a it of agent i at time t involves making two successive decisions which specify the region of the solution to be changed as well as the rule of how to change it.
Revised: A local action a it of agent i at time t involves making two successive decisions which specify the region of the solution to be changed as well as the rule of how to change it.

Original: Technically, it means to select a region node w it and afterwards a corresponding rule node u it with a it =
Revised: Technically, it means to select a region node w it and afterwards a corresponding rule node u it with a it =

Original: ( w it , u it ) implying the region node to be moved by being placed after the rule node.
Revised: ( w it , u it ) implying the region node to be moved by being placed after the rule node.

Original: We allow local actions only to rewrite the agent’s own local state and update the pool state.
Revised: We allow local actions only to rewrite the agent’s own local state and update the pool state.

Original: Semantically, an agent can re-arrange one node within its local state, integrate a new node by taking it from the pool, exclude a node by giving it to the pool or also keep its local state unchanged (cf.
Revised: Semantically, an agent can re-arrange one node within its local state, integrate a new node by taking it from the pool, exclude a node by giving it to the pool or also keep its local state unchanged (cf.

Original: Figure 2 for examples).
Revised: Figure 2 for examples).

Original: To encourage the frequent generation of feasible solutions, we restrict the set of allowed local actions dependent on the pool state:
Revised: To encourage the frequent generation of feasible solutions, we restrict the set of allowed local actions dependent on the pool state:

Original: Each time the pool is filled and hence the global state is an infeasible solution, agents are asked to integrate nodes from the pool to reestablish a feasible solution and are not allowed to make any other changes to their local states.
Revised: Each time the pool is filled and hence the global state is an infeasible solution, agents are asked to integrate nodes from the pool to reestablish a feasible solution and are not allowed to make any other changes to their local states.

Original: We note that the local agent actions which involve local re-ordering or giving nodes to the pool cannot interfere with each other.
Revised: We note that the local agent actions which involve local re-ordering or giving nodes to the pool cannot interfere with each other.

Original: To guarantee the same for taking nodes from the pool, the pool has a coordinating mechanism which offers nodes to agents for integration in a conflict-free manner (cf.
Revised: To guarantee the same for taking nodes from the pool, the pool has a coordinating mechanism which offers nodes to agents for integration in a conflict-free manner (cf.

Original: Section 3.3).
Revised: Section 3.3).

Original: The designed pool component thus enables us to meet the principles of conflict-free decisions and also self-determined agents, since all communication regarding exchanging nodes flows through the pool and nodes from there are integrated on a voluntarily basis.
Revised: The designed pool component thus enables us to meet the principles of conflict-free decisions and also self-determined agents, since all communication regarding exchanging nodes flows through the pool and nodes from there are integrated on a voluntarily basis.

Original: The rules of the game, including the correspondence of the described rewriting actions to the choices of region and rule, are formalized in Appendix A.2.
Revised: The rules of the game, including the correspondence of the described rewriting actions to the choices of region and rule, are formalized in Appendix A.2.

Original: Rewards
Revised: Rewards

Original: The global reward generally reflects the improvement in the team average cost for two feasible global states and penalizes the agent team if infeasible solutions were created consecutively for a too long time.
Revised: The global reward generally reflects the improvement in the team average cost for two feasible global states and penalizes the agent team if infeasible solutions were created consecutively for a too long time.

Original: The team average cost in the global state s t is given by c ( s t )
Revised: The team average cost in the global state s t is given by c ( s t )

Original: = 1 n (cid:80) ni c i ( s it ) where c i ( s i t ) denotes the local cost of agent i at time t .
Revised: = 1 n (cid:80) ni c i ( s it ) where c i ( s i t ) denotes the local cost of agent i at time t .

Original: The global reward at time t is then defined by
Revised: The global reward at time t is then defined by

Original: [Equation]
Revised: [Equation]

Original:  where c ( s prev f ( t ) ) denotes the team average cost of the last feasible solution before time step t and m is a hyperparameter.
Revised: feasible [Equation] if s t is infeasible and the last m global states were infeasible, else, (1) where c ( s prev f ( t ) ) denotes the team average cost of the last feasible solution before time step t and m is a hyperparameter.

Original: A strictly positive reward thus indicates an improved corresponding current feasible solution compared to the last one.
Revised: A strictly positive reward thus indicates an improved corresponding current feasible solution compared to the last one.

Revised: For m > 1 the reward becomes non-Markovian which is currently left to the RL algorithm to learn about instead of being explicitly handled in the state representation.

Revised: The choice of m is discussed in Appendix A.7.

Original: Note that computing the global reward requires the agent-specific costs in a global state to be revealed during training.
Revised: Note that computing the global reward requires the agent-specific costs in a global state to be revealed during training.

Original: Rewriting episode
Revised: Rewriting episode

Original: The globally observed episode starts with an initial feasible global state at time zero and is limited by a fixed number of rewriting steps T : ( s 0 , a 0 , r 1 , s 1 , a 1 , r 2 , s 2 , ..., s T − 1 , a T − 1 , r T , s T ) .
Revised: The globally observed episode starts with an initial feasible global state at time zero and is limited by a fixed number of rewriting steps T : ( s 0 , a 0 , r 1 , s 1 , a 1 , r 2 , s 2 , ..., s T − 1 , a T − 1 , r T , s T ) .

Original: The final solution to the routing problem is defined by the last feasible global state in the rewriting episode.
Revised: The final solution to the routing problem is defined by the last feasible global state in the rewriting episode.

Original: 3. M ODEL O VERVIEW
Revised: 3. M ODEL O VERVIEW

Original: In this section, we present the RL-based models used to implement the described rewriting game.
Revised: In this section, we present the RL-based models used to implement the described rewriting game.

Original: We saw that a local agent action is a two-step procedure, requiring a region and a corresponding rule node to be chosen.
Revised: We saw that a local agent action is a two-step procedure, requiring a region and a corresponding rule node to be chosen.

Original: Defining one policy over the tuple of regions and rules would result in a discrete distribution with a sample space size that is quadratic in the problem size.
Revised: Defining one policy over the tuple of regions and rules would result in a discrete distribution with a sample space size that is quadratic in the problem size.

Original: Hence we follow Chen & Tian (2019) and reduce the space by considering two separate distributions for regions and rules.
Revised: Hence we follow Chen & Tian (2019) and reduce the space by considering two separate distributions for regions and rules.

Original: In our current implementation, only the rule distribution is modelled as a neural network, the region distribution is random (while following some rules).
Revised: In our current implementation, only the rule distribution is modelled as a neural network, the region distribution is random (while following some rules).

Original: It can harm the overall rewriting procedure only in terms of slowing it down, as described below in more detail.
Revised: It can harm the overall rewriting procedure only in terms of slowing it down, as described below in more detail.

Original: Since we assume homogeneous and thus interchangeable agents, we learn a single agent-agnostic policy for choosing rules.
Revised: Since we assume homogeneous and thus interchangeable agents, we learn a single agent-agnostic policy for choosing rules.

Original: To ensure limited disclosure of local costs in the execution phase, we require the agent policy to only observe local cost information for decision-making.
Revised: To ensure limited disclosure of local costs in the execution phase, we require the agent policy to only observe local cost information for decision-making.

Original: This is enough information for agents to help the team by optimizing the order of their own currently visited nodes.
Revised: This is enough information for agents to help the team by optimizing the order of their own currently visited nodes.

Original: However, it is not sufficient for deciding whether exchanging nodes is beneficial for the team.
Revised: However, it is not sufficient for deciding whether exchanging nodes is beneficial for the team.

Original: Hence, we must provide them with global cost knowledge during learning.
Revised: Hence, we must provide them with global cost knowledge during learning.

Original: This is realized by an actor-critic approach following the scheme of centralized-learning-decentralized-execution:
Revised: This is realized by an actor-critic approach following the scheme of centralized-learning-decentralized-execution:

Original: Based on perfect knowledge about all agent costs, the centralized critic learns to judge global actions in given global states with respect to their benefit for the team.
Revised: Based on perfect knowledge about all agent costs, the centralized critic learns to judge global actions in given global states with respect to their benefit for the team.

Original: The critic is used to guide the centralized agent policy.
Revised: The critic is used to guide the centralized agent policy.

Original: It enables the agent policy to learn about the underlying cost matrix distribution and thus to locally assess if integrating or excluding a node is helpful for the team.
Revised: It enables the agent policy to learn about the underlying cost matrix distribution and thus to locally assess if integrating or excluding a node is helpful for the team.

Original: To facilitate the training process of the critic, we don’t learn it by a posteriori showing it a global action defined by the chosen local agent actions, but let the critic centrally select the global action itself during training.
Revised: To facilitate the training process of the critic, we don’t learn it by a posteriori showing it a global action defined by the chosen local agent actions, but let the critic centrally select the global action itself during training.

Original: It allows a better trade-off between exploration and exploitation and thus helps learning, especially given the high-dimensional joint action space.
Revised: It allows a better trade-off between exploration and exploitation and thus helps learning, especially given the high-dimensional joint action space.

Original: We force agents to centrally coordinate their local actions only during training.
Revised: We force agents to centrally coordinate their local actions only during training.

Original: During inference, solely the agent policy is used in a decentral manner, see Figure 3.
Revised: During inference, solely the agent policy is used in a decentral manner, see Figure 3.

Original: In the following, we summarize the workflow for generating a global action during training and inference together with the necessary model components.
Revised: In the following, we summarize the workflow for generating a global action during training and inference together with the necessary model components.

Original: Due to space limitations, we refer the reader to Appendix A.3 for a discussion of the corresponding loss functions.
Revised: Due to space limitations, we refer the reader to Appendix A.3 for a discussion of the corresponding loss functions.

Original: Encoding nodes: For each node in the problem, we learn high-dimensional embeddings which incorporate information about the current state.
Revised: Encoding nodes: For each node in the problem, we learn high-dimensional embeddings which incorporate information about the current state.

Original: Each agent encodes its currently visited nodes solely based on local information with an LSTM-based agent-agnostic local state encoder.
Revised: Each agent encodes its currently visited nodes solely based on local information with an LSTM-based agent-agnostic local state encoder.

Original: For nodes in the pool we introduce an additional model, the pool state encoder, as the pool state differs from the local ones in its semantic structure.
Revised: For nodes in the pool we introduce an additional model, the pool state encoder, as the pool state differs from the local ones in its semantic structure.

Original: These learned embeddings are fed into our centralized critic and agent rule policy for decision-making.
Revised: These learned embeddings are fed into our centralized critic and agent rule policy for decision-making.

Original: See Appendix A.4 for more details on the node encoders. 
Revised: See Appendix A.4 for more details on the node encoders. [Figure]

Original: Determining local actions: During training, we sample Z > 1 candidate local actions for each agent.
Revised: Determining local actions: During training, we sample Z > 1 candidate local actions for each agent.

Original: During inference, each agent selects one local action ( Z = 1 ) (cf.
Revised: During inference, each agent selects one local action ( Z = 1 ) (cf.

Original: Figure 3).
Revised: Figure 3).

Original: A local action requires to choose a region node to be moved and based thereon a rule node.
Revised: A local action requires to choose a region node to be moved and based thereon a rule node.

Original: These decisions are made successively with two different models: A region node is determined with the region selector which is influenced by randomness.
Revised: These decisions are made successively with two different models: A region node is determined with the region selector which is influenced by randomness.

Original: The corresponding rule node is chosen by a learned agent-agnostic but agent-cost dependent rule policy, the local rule selector.
Revised: The corresponding rule node is chosen by a learned agent-agnostic but agent-cost dependent rule policy, the local rule selector.

Original: [Figure]

Original: Region selector
Revised: Region selector

Original: Region nodes are chosen randomly while following some rules.
Revised: Region nodes are chosen randomly while following some rules.

Original: For local re-ordering or exclusion of a node, agents can independently sample from a uniform distribution defined over their currently visited customers to get candidate local region nodes.
Revised: For local re-ordering or exclusion of a node, agents can independently sample from a uniform distribution defined over their currently visited customers to get candidate local region nodes.

Original: If the pool is filled and a region node must thus be one of the customer nodes in pool, we coordinate the assignment of region nodes to agents to avoid conflicts.
Revised: If the pool is filled and a region node must thus be one of the customer nodes in pool, we coordinate the assignment of region nodes to agents to avoid conflicts.

Original: The coordination can be viewed as an automated mechanism of the pool which simply informs each agent about its (for it) selected region.
Revised: The coordination can be viewed as an automated mechanism of the pool which simply informs each agent about its (for it) selected region.

Original: This mechanism is also driven by randomness but at the same time equipped with a little intelligence.
Revised: This mechanism is also driven by randomness but at the same time equipped with a little intelligence.

Original: It neither offers a node first to the agent who just dropped the node in the pool, nor asks the same agent to integrate a node multiple times in a row (throughout the rewriting episode).
Revised: It neither offers a node first to the agent who just dropped the node in the pool, nor asks the same agent to integrate a node multiple times in a row (throughout the rewriting episode).

Original: We note that a random region selector does not necessarily aggravate the rewriting procedure in terms of leading to a bad rewritten state since an agent can always choose to do nothing via the rule node if a bad region node was selected.
Revised: We note that a random region selector does not necessarily aggravate the rewriting procedure in terms of leading to a bad rewritten state since an agent can always choose to do nothing via the rule node if a bad region node was selected.

Original: Nevertheless it can slow down the rewriting procedure.
Revised: Nevertheless it can slow down the rewriting procedure.

Original: Local rule selector Given an agent’s region node, the learned agent-agnostic local rule selector completes a local action by selecting a corresponding rule node.
Revised: Local rule selector Given an agent’s region node, the learned agent-agnostic local rule selector completes a local action by selecting a corresponding rule node.

Original: The decision is based on a predicted probability distribution over all possible rule candidates.
Revised: The decision is based on a predicted probability distribution over all possible rule candidates.

Original: It relies on an attention mechanism which processes node encodings of the already selected region and all respective possible rules as well as information as of how choosing the rule would affect the agent’s local state (in terms of its node representations).
Revised: It relies on an attention mechanism which processes node encodings of the already selected region and all respective possible rules as well as information as of how choosing the rule would affect the agent’s local state (in terms of its node representations).

Original: During training, the rule is sampled from the predicted probability distribution while for inference we choose the rule with the highest probability.
Revised: During training, the rule is sampled from the predicted probability distribution while for inference we choose the rule with the highest probability.

Original: Determining a global action: During training, the Z candidate local actions per agent are centrally collected and zipped together to build Z candidate global actions.
Revised: Determining a global action: During training, the Z candidate local actions per agent are centrally collected and zipped together to build Z candidate global actions.

Original: We choose one global action out of these candidates for rewriting based on an over the joint action space learned actionvalue function.
Revised: We choose one global action out of these candidates for rewriting based on an over the joint action space learned actionvalue function.

Original: During inference, the single chosen local agent actions automatically determine the global action (cf.
Revised: During inference, the single chosen local agent actions automatically determine the global action (cf.

Original: Figure 3) .
Revised: Figure 3) .

Original: Global action scorer The learned MLP-based global action scorer quantifies the expected team benefit of rewriting a given global state with a given global action.
Revised: Global action scorer The learned MLP-based global action scorer quantifies the expected team benefit of rewriting a given global state with a given global action.

Original: During training, we use it in an epsilon-greedy strategy to establish a good trade-off between exploration and exploitation.
Revised: During training, we use it in an epsilon-greedy strategy to establish a good trade-off between exploration and exploitation.

Original:  E MPIRICAL E VALUATION
Revised:  E MPIRICAL E VALUATION

Original: We empirically evaluate the MANR for vehicle routing in different setups on simulated data varying the number of agents as well as the problem size, i.e., the number of customer nodes in the routing problem.
Revised: We empirically evaluate the MANR for vehicle routing in different setups on simulated data varying the number of agents as well as the problem size, i.e., the number of customer nodes in the routing problem.

Original: The procedure for data simulation is summarized in Section 4.1.
Revised: The procedure for data simulation is summarized in Section 4.1.

Original: Section 4.2 discusses the experiment results for all setups and compares them to  the widely used OR-Tools optimization software.
Revised: Section 4.2 discusses the experiment results for all setups and compares them to a benchmark. There exists no comparable (RL-based) approach which could be used in the same collaborative limited disclosure setting out of the box. For this reason, we compare our results to an established approach with perfect cost knowledge and evaluate the competitiveness of our method. We chose the widely used OR-Tools optimization software as a benchmark which is also based on the concept of local search.

Revised: In Appendix A.5 we furthermore demonstrate the benefit of collaboration for the average agent in our approach by comparing it with a non-collaborative setup.

Original: 4. D ATA G ENERATION
Revised: 4. D ATA G ENERATION

Original: Customer nodes and agent depot nodes are sampled within the unit square, see Figure 4.
Revised: Customer nodes and agent depot nodes are sampled within the unit square, see Figure 4.

Original: We draw a random fraction of customer nodes near the agent depots to enforce the participation of all agents in an optimal routing solution.
Revised: We draw a random fraction of customer nodes near the agent depots to enforce the participation of all agents in an optimal routing solution.

Original: The precise node sampling procedure is explained in Appendix
Revised: The precise node sampling procedure is explained in Appendix

Original: A.5.
Revised: A.6.

Original: The agent-specific costs between two customer nodes are modelled via agent-specific velocities.
Revised: The agent-specific costs between two customer nodes are modelled via agent-specific velocities.

Original: Each agent velocity η i is uniformly sampled from [0 . 95 , 1] , i.e., an agent can be at most 5% faster than other agents.
Revised: Each agent velocity η i is uniformly sampled from [0 . 95 , 1] , i.e., an agent can be at most 5% faster than other agents.

Original: The cost for agent i to travel between two nodes v , z ∈ [0 , 1]is then given by their inverse-velocity-scaled Euclidean Distance as c i ( v , z ) = 1 η i ∥ v − z ∥ 2 .
Revised: The cost for agent i to travel between two nodes v , z ∈ [0 , 1]is then given by their inverse-velocity-scaled Euclidean Distance as c i ( v , z ) = 1 η i ∥ v − z ∥ 2 .

Original: The MANR requires an initial feasible solution as a starting point for the rewriting procedure.
Revised: The MANR requires an initial feasible solution as a starting point for the rewriting procedure.

Original: To enforce the pool usage need for successful rewriting, we simply randomly assign the customer nodes to agents as evenly as possible.
Revised: To enforce the pool usage need for successful rewriting, we simply randomly assign the customer nodes to agents as evenly as possible.

Original: Each agent then applies the nearest neighbour heuristic (Rosenkrantz et al., 1977) to its customer node set, see Figure 4 for an exemplary initial solution.
Revised: Each agent then applies the nearest neighbour heuristic (Rosenkrantz et al., 1977) to its customer node set, see Figure 4 for an exemplary initial solution.

Original: Further examples of initial states are depicted in Appendix A.5.
Revised: Further examples of initial states are depicted in Appendix A.6.

Original: 4. E XPERIMENTS
Revised: 4. E XPERIMENTS

Original: We performed experiments for vehicle routing problems of sizes 10 and 20 with 2 , 3 and 5 agents respectively.
Revised: We performed experiments for vehicle routing problems of sizes 10 and 20 with 2 , 3 and 5 agents respectively.

Original: Each data set consists of 6280 vehicle routing problems and is split into three parts of 80% - 10% - 10% for training, validating and testing.
Revised: Each data set consists of 6280 vehicle routing problems and is split into three parts of 80% - 10% - 10% for training, validating and testing.

Original: Hyperparameter tuning was performed on the validation set with Ray Tune 3 .
Revised: Hyperparameter tuning was performed on the validation set with Ray Tune 3 .

Original: During training, we chose 30 rewriting steps in all setups with 10 customer nodes and 40 rewriting steps for those with 20 nodes.
Revised: During training, we chose 30 rewriting steps in all setups with 10 customer nodes and 40 rewriting steps for those with 20 nodes.

Original: For inference, we increased the rewriting steps to compensate the stochastic region selection:
Revised: For inference, we increased the rewriting steps to compensate the stochastic region selection:

Original: In contrast to the training phase, each agent now just samples a single random region in a step; we don’t have the luxury of choosing from a set of candidate actions with (most probably) different random region suggestions.
Revised: In contrast to the training phase, each agent now just samples a single random region in a step; we don’t have the luxury of choosing from a set of candidate actions with (most probably) different random region suggestions.

Original: We consider 100 rewriting steps throughout all experiments for evaluation which we have found to work out well for all setups.
Revised: We considerrewriting steps throughout all experiments for evaluation which we have found to work out well for all setups.

Original: We refer to Appendix A.6 for a complete overview of selected hyperparameter values for each of the experiments.
Revised: We refer to Appendix A.7 for a complete overview of selected hyperparameter values for each of the experiments.

Original: We compare our results to the routing solver from OR-Tools which is based on local search and specifically tuned for vehicle routing 4 .
Revised: We compare our results to the routing solver from OR-Tools which is based on local search and specifically tuned for vehicle routing 4 .

Original: We use the default search parameters and start with the same initial solutions as the MANR.
Revised: We use the default search parameters and start with the same initial solutions as the MANR.

Original: Note that the solver necessitates complete knowledge about all vehicle costs for optimization.
Revised: Note that the solver necessitates complete knowledge about all vehicle costs for optimization

Original:  Evaluation
Revised: . Evaluation

Original: For each experiment, we make 20 inference runs on the test set (due to the region stochasticity) and compute the test set performance averaged over all runs.
Revised: For each experiment, we make 20 inference runs on the test set (due to the region stochasticity) and compute the test set performance averaged over all runs.

Original: Performance is measured with the team average cost of the last feasible solution in the rewriting episode and the initial solutions are kept equal throughout all runs.
Revised: Performance is measured with the team average cost of the last feasible solution in the rewriting episode and the initial solutions are kept equal throughout all runs.

Original: We also compute the mean performance when choosing the best run for each test sample individually (”MANR best”).
Revised: We also compute the mean performance when choosing the best run for each test sample individually (”MANR best”).

Revised: This is a natural and valid metric when inference time is not decisive in the application.

Original: We compare these values to the mean test set performance of the initial solutions as well as of the solutions produced by OR-Tools: We report the performance gaps in terms of percentage cost reductions relative to the intitial solution and relative to OR-Tools.
Revised: We compare these values to the mean test set performance of the initial solutions as well as of the solutions produced by OR-Tools: We report the performance gaps in terms of percentage cost reductions relative to the intitial solution and relative to OR-Tools.

Original: We also report the average run times for one routing problem, when performing evaluation on a server with a single GPU (Tesla V100S-PCIE-32GB) and CPU core.
Revised: We also report the average run times for one routing problem, when performing evaluation on a server with a single GPU (Tesla V100S-PCIE-32GB) and CPU core.

Original: The results for the setup of 10 nodes are presented in Table 1.
Revised: The results for the setup of 10 nodes are presented in Table 1.

Original: The results fornodes as well as the absolute performance values for all setups can be found in Appendix A.7.
Revised: The results for 20 nodes as well as the absolute performance values for all setups can be found in Appendix A.8.

Original: [Table]
Revised: [Table]

Original: Discussion For all experimental setups (both 10 and 20 nodes), the MANR significantly improves over the initial solutions: We have an average percentage cost reduction of 40% for the mean MANR solution over all runs (”MANR”) and 50% when considering the best solution for each routing problem per run (”MANR best”).
Revised: Discussion For all experimental setups (both 10 and 20 nodes), the MANR significantly improves over the initial solutions: We have an average percentage cost reduction of 40% for the mean MANR solution over all runs (”MANR”) and 50% when considering the best solution for each routing problem per run (”MANR best”).

Original: Generally, the more agents the bigger the percentage cost reduction.
Revised: Generally, the more agents the bigger the percentage cost reduction.

Original: [Table]

Original: This stems from the fact that the more agents, the higher the probability that a node gets assigned to a wrong agent in the randomly generated initial solution.
Revised: This stems from the fact that the more agents, the higher the probability that a node gets assigned to a wrong agent in the randomly generated initial solution.

Original: Hence, we expect worse initial solutions and thus more room for improvement with an increasing amount of agents.
Revised: Hence, we expect worse initial solutions and thus more room for improvement with an increasing amount of agents.

Original: The performance of the OR-Tools benchmark cannot be reached by the MANR but it gets reasonably close, taking into account the imperfect cost knowledge of the MANR agents.
Revised: The performance of the OR-Tools benchmark cannot be reached by the MANR but it gets reasonably close, taking into account the imperfect cost knowledge of the MANR agents.

Original: Evaluating the results of both 10 andnodes, there is an average percentage cost reduction of − 41% of the MANR solution quality relative to the one of OR-Tools, respectively, − 14% when considering MANR best.
Revised: Evaluating the results of both 10 andnodes, there is an average percentage cost reduction of − 41% of the MANR solution quality relative to the one of OR-Tools, respectively, − 14% when considering MANR best.

Original: The more agents the more pronounced the gap.
Revised: The more agents the more pronounced the gap.

Original: This can be explained with the increasing global action space dimensionality which makes it more difficult for the global action scorer (the critic) to learn.
Revised: This can be explained with the increasing global action space dimensionality which makes it more difficult for the global action scorer (the critic) to learn.

Original: Looking at individual rewriting episodes of the MANR, we saw that agents learned to use the newly introduced pool in a meaningful way (see Appendix A.7.1 for an excerpt of an exemplary rewriting rollout).
Revised: Looking at individual rewriting episodes of the MANR, we saw that agents learned to use the newly introduced pool in a meaningful way (see Appendix A.8.1 for an excerpt of an exemplary rewriting rollout).

Original: We also observed, that the MANR even solves a few isolated problems better than OR-Tools.
Revised: We also observed, that the MANR even solves a few isolated problems better than OR-Tools.

Original: It will be interesting to analyze the kind of these routing problems in the future.
Revised: It will be interesting to analyze the kind of these routing problems in the future.

Original: Comparing inference times, the well-tuned, in C++-implemented OR-Tools software is significantly faster.
Revised: Comparing inference times, the well-tuned, in C++-implemented OR-Tools software is significantly faster.

Original: It is also less influenced by the number of agents.
Revised: It is also less influenced by the number of agents.

Original: However, it requires to be perfectly informed about all agent costs in contrast to the MANR.
Revised: However, it requires to be perfectly informed about all agent costs in contrast to the MANR.

Original: Also, up to now, we haven’t optimized our Python code for efficiency.
Revised: Also, up to now, we haven’t optimized our Python code for efficiency.

Original: There is room for improvement, e.g., by parallelizing the decentral agent action generation during inference which currently runs sequentially in our implementation.
Revised: There is room for improvement, e.g., by parallelizing the decentral agent action generation during inference which currently runs sequentially in our implementation.

Original: We could also investigate in more detail, which number of rewriting steps suffices in the respective settings and save time with shorter rewriting rollouts.
Revised: We could also investigate in more detail, which number of rewriting steps suffices in the respective settings and save time with shorter rewriting rollouts.

Original: Moreover, we expect the run times to approach each other when scaling to larger problem sizes as it was observed for the Neural Rewriter.
Revised: Moreover, we expect the run times to approach each other when scaling to larger problem sizes as it was observed for the Neural Rewriter.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented the multi-agent Neural Rewriter (MANR) for collaboratively solving a multi-vehicle routing problem with limited disclosure of vehicle-specific costs in spirit of a team Markov game.
Revised: We presented the multi-agent Neural Rewriter (MANR) for collaboratively solving a multi-vehicle routing problem with limited disclosure of vehicle-specific costs in spirit of a team Markov game.

Original: Vehicle-specific costs are only explicitly shared in the training phase to allow agents to learn about the underlying cost distribution.
Revised: Vehicle-specific costs are only explicitly shared in the training phase to allow agents to learn about the underlying cost distribution.

Original: During inference, each agent performs its action solely based on local cost information.
Revised: During inference, each agent performs its action solely based on local cost information.

Original: We enable parallel conflict-free agent actions in the game by introducing a pool mechanism which coordinates the necessary node exchange between agents.
Revised: We enable parallel conflict-free agent actions in the game by introducing a pool mechanism which coordinates the necessary node exchange between agents.

Original: The introduced pool comes at the cost of generating also infeasible solutions within the rewriting episode, but is counteracted by teaching agents a proper pool usage during training.
Revised: The introduced pool comes at the cost of generating also infeasible solutions within the rewriting episode, but is counteracted by teaching agents a proper pool usage during training.

Original: Our agent-agnostic policy, which must solely process local information, is provided with some global knowledge during training via its cost-omniscient critic.
Revised: Our agent-agnostic policy, which must solely process local information, is provided with some global knowledge during training via its cost-omniscient critic.

Original: Our empirical results demonstrate that the approach indeed enables the solely local cost observing agents to act for the sake of the team, i.e., to exchange nodes with other agents via the pool in a meaningful way: The MANR improves an initial solution where nodes get simply randomly assigned to agents by 50% on average.
Revised: Our empirical results demonstrate that the approach indeed enables the solely local cost observing agents to act for the sake of the team, i.e., to exchange nodes with other agents via the pool in a meaningful way: The MANR improves an initial solution where nodes get simply randomly assigned to agents by 50% on average.

Original: Agents learn to assess the capabilities of representative team members and can base their rewriting decisions thereon.
Revised: Agents learn to assess the capabilities of representative team members and can base their rewriting decisions thereon.

Original: The experiments also confirm the inescapable trade-off between non-disclosure and optimality.
Revised: The experiments also confirm the inescapable trade-off between non-disclosure and optimality.

Original: The performance of our benchmark, the OR-Tools heuristic, which assumes perfect cost information cannot be reached on average.
Revised: The performance of our benchmark, the OR-Tools heuristic, which assumes perfect cost information cannot be reached on average.

Original: However, the MANR gets close and even solves a few isolated problems better.
Revised: However, the MANR gets close and even solves a few isolated problems better.

Original: In the future, we plan to further empirically evaluate the scalability of our approach by increasing the number of nodes in the routing problem.
Revised: In the future, we plan to further empirically evaluate the scalability of our approach by increasing the number of nodes in the routing problem.

Original: We want to improve our current implementation by fine-tuning the model architectures and enhancing the code efficiency as we expect a significant performance boost from it.
Revised: We want to improve our current implementation by fine-tuning the model architectures and enhancing the code efficiency as we expect a significant performance boost from it.

Revised: Another adaption of the current setup could involve explicitly representing the pool history in the state by, e.g., introducing a counter counting the consecutive steps of a filled pool, to guarantee the Markov property in the game setup and thus simplify learning.

Original: Another adaption of the current setup could involve replacing the random component in the region selection with a learned model.
Revised: The approach could furthermore be improved by replacing the random component in the region selection with a learned model.

Original: Also the issue of limited agent scalability due to joint action space learning needs to be addressed.
Revised: Also the issue of limited agent scalability due to joint action space learning needs to be addressed.

Original: Moreover, we plan to extend to other setups.
Revised: Moreover, we plan to extend to other setups.

Original: One direction could be to consider a heterogeneous agent team where the agent cost matrices are not sampled from the same distribution.
Revised: One direction could be to consider a heterogeneous agent team where the agent cost matrices are not sampled from the same distribution.

Original: It requires to provide the agent policy with more global cost knowledge than in the current setup and hence intensifies the trade-off between non-disclosure and optimality.
Revised: It requires to provide the agent policy with more global cost knowledge than in the current setup and hence intensifies the trade-off between non-disclosure and optimality.

Original: Another interesting direction concerns to tighten the limited disclosure requirement by also demanding it during the learning phase.
Revised: Another interesting direction concerns to tighten the limited disclosure requirement by also demanding it during the learning phase.

Original: A possible way to avoid explicit revelation of agentspecific costs during training is to employ distributed machine learning approaches, i.e., to only share local model parameters.
Revised: A possible way to avoid explicit revelation of agent-specific costs during training is to employ distributed machine learning approaches, i e., to only share local model parameters.

Original: We could also let agents exchange abstract information via learned vectorbased embeddings.
Revised: We could also let agents exchange abstract information via learned vector-based embeddings.

Original: For both options it would be interesting trying to quantify the non-disclosure: We could introduce an opponent to the game which tries to re-construct agent costs from the shared information. 
Revised: For both options it would be interesting trying to quantify the non-disclosure: We could introduce an opponent to the game which tries to re-construct agent costs from the shared information. A CKNOWLEDGMENTS

Revised: The research of N. Paul was supported by the Fraunhofer Society within the project “SWAP –

Revised: Hierarchical swarms as production architecture with optimized utilization”.

Original:  A A PPENDIX A. D IFFERENCES TO THE ORIGINAL N EURAL R EWRITER
Revised: The work of T. Wirtz was funded by the German Federal Ministry of Education and Research, ML2R - no. 01S18038B. S. Wrobel contributed as part of the University of Bonn and the Fraunhofer Center for Machine Learning within the Fraunhofer Cluster for Cognitive Internet Technologies. A A PPENDIX A. D IFFERENCES TO THE ORIGINAL N EURAL R EWRITER

Original: We give an overview about the most essential differences of the MANR compared to the original
Revised: We give an overview about the most essential differences of the MANR compared to the original

Original: Neural Rewriter.
Revised: Neural Rewriter.

Original: Application
Revised: Application

Original: The Neural Rewriter is used for solving a capacitated vehicle routing.
Revised: The Neural Rewriter is used for solving a capacitated vehicle routing.

Original: We don’t consider the setup of vehicle capacities and customer demands and thus removed all associated information, e.g., in the node representations.
Revised: We don’t consider the setup of vehicle capacities and customer demands and thus removed all associated information, e.g., in the node representations.

Original: Game Setup
Revised: Game Setup

Original: The MANR extends the setup of the Neural Rewriter by considering multiple vehicles instead of a single one.
Revised: The MANR extends the setup of the Neural Rewriter by considering multiple vehicles instead of a single one.

Original: As described in Section 3, it required a redefinition of the RL setup, which goes beyond simply scaling from local states and actions to global states and actions.
Revised: As described in Section 3, it required a redefinition of the RL setup, which goes beyond simply scaling from local states and actions to global states and actions.

Original: A key difference is the introduction of the pool set which is modelled as an additional component in the MANR’s global state.
Revised: A key difference is the introduction of the pool set which is modelled as an additional component in the MANR’s global state.

Original: For the routing application, the Neural Rewriter technically allows an action to swap two nodes in the route.
Revised: For the routing application, the Neural Rewriter technically allows an action to swap two nodes in the route.

Original: We use a different ruleset which is also mentioned by the authors Chen & Tian (2019), that allows an action to move one node to a different position in the route.
Revised: We use a different ruleset which is also mentioned by the authors Chen & Tian (2019), that allows an action to move one node to a different position in the route.

Original: This is essential for the MANR as swapping two nodes would condemn an agent to always stay with the same amount of visited nodes and never allow it to reduce or enrich its visited node set.
Revised: This is essential for the MANR as swapping two nodes would condemn an agent to always stay with the same amount of visited nodes and never allow it to reduce or enrich its visited node set.

Original: At last, we adapt the reward definition of the Neural Rewriter to include feedback to proper and improper pool usage.
Revised: At last, we adapt the reward definition of the Neural Rewriter to include feedback to proper and improper pool usage.

Original: Models
Revised: Models

Original: We follow the approach of the Neural Rewriter to consider separate models for the region and rule selection, but realize them differently.
Revised: We follow the approach of the Neural Rewriter to consider separate models for the region and rule selection, but realize them differently.

Original: Moreover, in contrast to the Neural Rewriter, we learn a model to evaluate a complete (joint) action.
Revised: Moreover, in contrast to the Neural Rewriter, we learn a model to evaluate a complete (joint) action.

Original: The Neural Rewriter selects a region via a learned action-value function, while we currently model it with a rule-based random component.
Revised: The Neural Rewriter selects a region via a learned action-value function, while we currently model it with a rule-based random component.

Original: We instead learn an action-value function for the joint global action as our approach relies on an entity with perfect global knowledge.
Revised: We instead learn an action-value function for the joint global action as our approach relies on an entity with perfect global knowledge.

Original: The action-value function of the Neural Rewriter learns to predict the expected (discounted) maximal cost improvement compared to the given global state when choosing a region in the global state.
Revised: The action-value function of the Neural Rewriter learns to predict the expected (discounted) maximal cost improvement compared to the given global state when choosing a region in the global state.

Original: This is meaningful taking into account that one is only interested in the best solution which was produced within the rewriting episode.
Revised: This is meaningful taking into account that one is only interested in the best solution which was produced within the rewriting episode.

Original: This idea is however not transferable to our setup with limited cost disclosure:
Revised: This idea is however not transferable to our setup with limited cost disclosure:

Original: In order to determine the best team solution in a rolled out rewriting episode, each agent would need to share its occuring costs for all produced solutions.
Revised: In order to determine the best team solution in a rolled out rewriting episode, each agent would need to share its occuring costs for all produced solutions.

Original: For this reason, we instead always take the last feasible solution in the episode as the final solution, without evaluating any team average costs of solutions.
Revised: For this reason, we instead always take the last feasible solution in the episode as the final solution, without evaluating any team average costs of solutions.

Original: We hence adapt the loss function for our action-value function and let it predict the (discounted) cumulative team cost improvement.
Revised: We hence adapt the loss function for our action-value function and let it predict the (discounted) cumulative team cost improvement.

Original: Our rule policy is almost identical to the one of the Neural Rewriter, we only provide it with (partly) different input: To decide which node is chosen as the rule, some fictitious node representations in the spirit of ”what-would-happen-if” are provided (see definition of node representations within the local state and pool encoder in Appendix A.4).
Revised: Our rule policy is almost identical to the one of the Neural Rewriter, we only provide it with (partly) different input: To decide which node is chosen as the rule, some fictitious node representations in the spirit of ”what-would-happen-if” are provided (see definition of node representations within the local state and pool encoder in Appendix A.4).

Original: For the Neural Rewriter they create a fictitious node representation for the candidate rule node after being swapped with the region and for another self-defined node (which is not part of the real nodes).
Revised: For the Neural Rewriter they create a fictitious node representation for the candidate rule node after being swapped with the region and for another self-defined node (which is not part of the real nodes).

Original: In the MANR we create such fictitious node representations for each node whose representation would be affected by the local action, i.e., each node whose predecessor would change since node representations contain predecessor information.
Revised: In the MANR we create such fictitious node representations for each node whose representation would be affected by the local action, i.e., each node whose predecessor would change since node representations contain predecessor information.

Original: Consequently, for the employed local action type of placing the region node after the rule, we create fictitious node representations for the region node, the region node successor and the rule successor.
Revised: Consequently, for the employed local action type of placing the region node after the rule, we create fictitious node representations for the region node, the region node successor and the rule successor.

Original: Analogously to the Neural Rewriter, we train the rule policy with an actor-critic approach but replace the critic with our action-value function which assesses global actions.
Revised: Analogously to the Neural Rewriter, we train the rule policy with an actor-critic approach but replace the critic with our action-value function which assesses global actions.

Original: At last, for encoding nodes in the local state, we take over the LSTM-based encoder of the Neural
Revised: At last, for encoding nodes in the local state, we take over the LSTM-based encoder of the Neural

Original: Rewriter.
Revised: Rewriter.

Original: For encoding nodes in the pool we introduce a new attention-based encoder.
Revised: For encoding nodes in the pool we introduce a new attention-based encoder.

Original: A. M ORE DETAILS ON THE RULES OF THE GAME
Revised: A. M ORE DETAILS ON THE RULES OF THE GAME

Original: We formalize the set of allowed local agent actions which depend on the pool state.
Revised: We formalize the set of allowed local agent actions which depend on the pool state.

Original: Notation:
Revised: Notation:

Original: • V denotes the set of customer nodes, • s t =
Revised: • V denotes the set of customer nodes, • s t =

Original: ( s 1 t , s 2 t , ..., s nt , P t ) denotes the global state at time t , consisting of local agent states s it at time t and the corresponding current state of the pool P t .
Revised: ( s 1 t , s 2 t , ..., s nt , P t ) denotes the global state at time t , consisting of local agent states s it at time t and the corresponding current state of the pool P t .

Original: Local actions A local action of agent i generally consists of selecting a region node w i and afterwards a corresponding rule node u i with a i = ( w i , u i ) implying the region node to be moved by being placed after the rule node.
Revised: Local actions A local action of agent i generally consists of selecting a region node w i and afterwards a corresponding rule node u i with a i = ( w i , u i ) implying the region node to be moved by being placed after the rule node.

Original: For the interaction with the pool we introduce a representative pool node denoted by p .
Revised: For the interaction with the pool we introduce a representative pool node denoted by p .

Original: The allowed local actions for agent i at time t are defined as follows:
Revised: The allowed local actions for agent i at time t are defined as follows:

Original: • If the pool P t is empty: The region node to be moved can be any customer node from the agent’s own local state w
Revised: • If the pool P t is empty: The region node to be moved can be any customer node from the agent’s own local state w

Original: it ∈ V | s it .
Revised: it ∈ V | s it .

Original: The fixed rule set u it ∈ { s it } ∪ { p } then semantically translates to either keeping its local state unchanged by choosing the current region predecessor in its local state as the rule, re-arranging the region node locally by choosing any other node from its local state as the rule or excluding the region node from its local state by choosing the pool node as the rule.
Revised: The fixed rule set u it ∈ { s it } ∪ { p } then semantically translates to either keeping its local state unchanged by choosing the current region predecessor in its local state as the rule, re-arranging the region node locally by choosing any other node from its local state as the rule or excluding the region node from its local state by choosing the pool node as the rule.

Original: • If the pool P t is filled: The region node to be moved can be any customer node from the pool state w it ∈ P t .
Revised: • If the pool P t is filled: The region node to be moved can be any customer node from the pool state w it ∈ P t .

Original: The fixed rule set u it ∈ { s it }∪{ p } then semantically translates to either accepting an offer to integrate the region node locally by choosing any node from its local state as the rule or declining an offer and thus leave its local state unchanged by choosing the pool node as the rule.
Revised: The fixed rule set u it ∈ { s it }∪{ p } then semantically translates to either accepting an offer to integrate the region node locally by choosing any node from its local state as the rule or declining an offer and thus leave its local state unchanged by choosing the pool node as the rule.

Original: A. T RAINING DETAILS
Revised: A. T RAINING DETAILS

Original: All presented models (local state encoder, pool state encoder, local rule selector, global action scorer) are trained centrally and simultaneously with a combined loss function:
Revised: All presented models (local state encoder, pool state encoder, local rule selector, global action scorer) are trained centrally and simultaneously with a combined loss function:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: • θ denote the parameters of the global action scorer Q with corresponding loss L a ( · ) , • ϕ denote the parameters of the local rule selector π u with corresponding loss L u ( · ) , • ψ = [ ψ 1 , ψ 2 ] denote the parameters of the local state and pool state encoder, which are implicitly trained with both losses L a ( · ) and L u ( · ) .
Revised: • θ denote the parameters of the global action scorer Q with corresponding loss L a ( · ) , • ϕ denote the parameters of the local rule selector π u with corresponding loss L u ( · ) , • ψ = [ ψ 1 , ψ 2 ] denote the parameters of the local state and pool state encoder, which are implicitly trained with both losses L a ( · ) and L u ( · ) .

Original: The global action scorer Q is fitted to the cumulative discounted observed reward within the rewriting episode as
Revised: The global action scorer Q is fitted to the cumulative discounted observed reward within the rewriting episode as

Original: [Equation]
Revised: [Equation]

Original: where γ < 1 denotes the discount factor.
Revised: where γ < 1 denotes the discount factor.

Original: The agent-agnostic local rule selector π u is learned by an actor-critic approach with the global action scorer Q serving as the critic:
Revised: The agent-agnostic local rule selector π u is learned by an actor-critic approach with the global action scorer Q serving as the critic:

Original: [Equation]
Revised: [Equation]

Original: with 
Revised: with [Equation]

Original: [Equation] where A ( s t , u it , a − i t , w it ) denotes the advantage of choosing rule node u it in the global state s t given region node w it and all other agents’ local actions a − 
Revised: where A ( s t , u it , a − i t , w it ) denotes the advantage of choosing rule node u it in the global state s t given region node w it and all other agents’ local actions a − i t [Equation] ( a

Original: i t =
Revised: [Equation]

Original: ( a 1 t , ..., a i − 1 t , a i +1 t , ..., a nt ) .
Revised: [Equation] , a [Equation] , ..., a nt ) .

Original: Note that it’s arguably correct to criticize a local rule node in the context of all other agents’ local actions since the choice of a − i t does not effect the optimal action a it for agent i under Q , see A.3.1 below for a formalization of this statement.
Revised: Note that it’s arguably correct to criticize a local rule node in the context of all other agents’ local actions since the choice of a − i t does not effect the optimal action a it for agent i under Q , see A.3.1 below for a formalization of this statement.

Original: An agent can thus not be penalized or praised for his rule node’s choice due to bad or good actions of the other agents.
Revised: An agent can thus not be penalized or praised for his rule node’s choice due to bad or good actions of the other agents.

Original: The hyperparameter α < 1 in equation 2 weakens the loss of the local rule selector to give its critic, the global action scorer, a head first to perform well.
Revised: The hyperparameter α < 1 in equation 2 weakens the loss of the local rule selector to give its critic, the global action scorer, a head first to perform well.

Original: The total loss is minimized with the Adam optimizer.
Revised: The total loss is minimized with the Adam optimizer.

Original: A.3. P ROPERTY OF THE GLOBAL CRITIC Q
Revised: A.3. P ROPERTY OF THE GLOBAL CRITIC Q

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: denote the expected return when choosing global action a t =
Revised: denote the expected return when choosing global action a t =

Original: ( a 1 t , a 2 t , ..., a nt ) in the global state s t and following the joint agent policy π .
Revised: ( a 1 t , a 2 t , ..., a nt ) in the global state s t and following the joint agent policy π .

Original: Define
Revised: Define

Original: [Equation]
Revised: [Equation]

Original: as the local action a i of agent i which maximizes Q π for given other agent actions a − i = ( a 1 , ..., a i − 1 , a i +1 , ..., a n ) and the global state s .
Revised: as the local action a i of agent i which maximizes Q π for given other agent actions a − i = ( a 1 , ..., a i − 1 , a i +1 , ..., a n ) and the global state s .

Original: Then it holds
Revised: Then it holds

Original: [Equation]
Revised: [Equation]

Original: for any other agent actions (cid:103) a − i .
Revised: for any other agent actions (cid:103) a − i .

Original: I.e., the optimal action of agent i under Q π in a given global state s is independent of the other agents’ actions at the same time step
Revised: I.e., the optimal action of agent i under Q π in a given global state s is independent of the other agents’ actions at the same time step.

Original: . A. M ORE DETAILS ON THE MODELS Local state encoder
Revised: A. M ORE DETAILS ON THE MODELS Local state encoder

Original: Each node v in the local state of vehicle agent i is represented by a 5-dimensional vector ( v x , v y , v px , v py , c i ( v, v p )) containing x,y coordinates of the node itself and of its current predecessor in the route v p as well as the agent-specific cost to travel between them.
Revised: Each node v in the local state of vehicle agent i is represented by a 5-dimensional vector ( v x , v y , v px , v py , c i ( v, v p )) containing x,y coordinates of the node itself and of its current predecessor in the route v p as well as the agent-specific cost to travel between them.

Original: Each such node representation is encoded by a learned bidirectional LSTM which follows the trajectory of the local agent route and incorporates the sequential information.
Revised: Each such node representation is encoded by a learned bidirectional LSTM which follows the trajectory of the local agent route and incorporates the sequential information.

Original: Pool state encoder
Revised: Pool state encoder

Original: Each node v in the pool state is represented by a 2-dimensional vector ( v x , v y ) containing its x,y coordinates.
Revised: Each node v in the pool state is represented by a 2-dimensional vector ( v x , v y ) containing its x,y coordinates.

Original: Since there exists no semantic sequential order in the pool state, each such node representation is encoded by a learned self-attention mechanism.
Revised: Since there exists no semantic sequential order in the pool state, each such node representation is encoded by a learned self-attention mechanism.

Original: A. D ATA GENERATION
Revised: A. C OLLABORATION BENEFIT

Revised: In the following, we empirically demonstrate the incentive for the average agent to collaborate on serving customers: Assume each agent has its own set of customers to be visited.

Revised: For each agent separately, we solve a travelling salesman problem (TSP) computing an optimized route visiting all agent’s customer nodes.

Revised: The non-collaborative baseline cost c non-collab is then given by the average cost over all agent TSP solutions.

Revised: We compare the result to our collaborative approach where agents combine their customer orders to find an optimized assignment of customers to agents as well as the corresponding routes.

Revised: Analogously to Section 4.2, we compute the team average cost for our setup, i.e., the cost for the average agent when collaborating, denoted by c team (both for ”MANR” and ”MANR best”).

Revised: The collaboration benefit is then defined by the percentage cost reduction relative to the non-collaborative baseline cost as c non-collab − c team c non-collab .

Revised: We report the collaboration benefit for all in Section 4.2 discussed experimental setups of 10 and 20 customer nodes together with 2,3,5 agents respectively on the test set.

Revised: For the non-collaborative baseline we consider the same agent-customer-assignments as in the initial solutions of the MANR (cf.

Revised: Section 4.1).

Revised: The resulting agent routes thus only differ from those in the initial solutions in such a way that they are not obtained by the nearest neighbour heuristic but are optimized with the

Revised: TSP-solver from OR-Tools 5 .

Revised: Table 2 summarizes the empirical results.

Revised: For 10 customer nodes we have an average collaboration benefit of 40% for MANR, resp.

Revised: 50% for MANR best.

Revised: For 20 customer nodes we have an average collaboration benefit of 33% for MANR, resp.

Revised: 45% for MANR best.

Revised: Generally, the more agents, the higher the profit for collaborating.

Revised: This stems from the fact that the more (random) agent depots the higher the chance that a different agent can visit a given customer node cheaper than the agent initially assigned to it.

Original:  Nodes When sampling all customer nodes uniformly in the unit square as in Chen & Tian (2019), we observed that it is often overall cheaper if only one agent sets off to do the job alone.
Revised: [Table] A. D ATA GENERATION Nodes When sampling all customer nodes uniformly in the unit square as in Chen & Tian (2019), we observed that it is often overall cheaper if only one agent sets off to do the job alone.

Original: To diversify the data set and enforce the participation of all agents in an optimal routing solution, we only draw a random fraction of customer nodes uniformly and the rest near the agents’ depots.
Revised: To diversify the data set and enforce the participation of all agents in an optimal routing solution, we only draw a random fraction of customer nodes uniformly and the rest near the agents’ depots.

Original: More precisely, we first sample agent depot node coordinates d i , i ∈ { 1 , ..., n } , uniformly in the unit square [0 , 1] 2 .
Revised: More precisely, we first sample agent depot node coordinates d i , i ∈ { 1 , ..., n } , uniformly in the unit square [0 , 1] 2 .

Original: For each agent i we define a bivariate truncated normal distribution in the unit square N ( d i , 0 .
Revised: For each agent i we define a bivariate truncated normal distribution in the unit square N ( d i , 0 .

Original: 1 2 ) centered at its depot with a standard deviation of 0 . 1 .
Revised: 1 2 ) centered at its depot with a standard deviation of 0 . 1 .

Original: We randomly sample the fraction of customer nodes which shall be drawn uniformly in the unit square.
Revised: We randomly sample the fraction of customer nodes which shall be drawn uniformly in the unit square.

Original: The remaining customer nodes are assigned to the agents as evenly as possible and their coordinates are drawn from the described agent-specific normal distributions.
Revised: The remaining customer nodes are assigned to the agents as evenly as possible and their coordinates are drawn from the described agent-specific normal distributions.

Revised: much room for improvement whereby agents will have to use the pool to exchange nodes.

Original: A. H YPERPARAMETERS
Revised: A. H YPERPARAMETERS

Original: Hyperparameter tuning was performed on the validation set with Ray Tune 5 .
Revised: Hyperparameter tuning was performed on the validation set with Ray Tune 6 .

Original: Some hyperparameters could be set equally for all experiments, while others depend on the problem size or the number of agents.
Revised: Some hyperparameters could be set equally for all experiments, while others depend on the problem size or the number of agents.

Original: For a fixed problem size k ∈ { 10 , 20 } we could use the same number of candidate actions Z as well as the same amount of rewriting steps T .
Revised: Forafixed problem size k ∈ { 10 , 20 } we could use the same number of candidate actions Z as well as the same amount of rewriting steps T .

Original: Apart from one exception (see below) we could also use the same learning rates for Q and for π u (controlled by α in equation 2) for a fixed problem size.
Revised: Apart from one exception (see below) we could also use the same learning rates for Q and for π u (controlled by α in equation 2) for a fixed problem size.

Original: For a fixed number of agents n ∈ { 2 , 3 , 5 } we used the same value of m in equation 1 which manages the maximum number of allowed infeasible solutions in a row.
Revised: For a fixed number of agents n ∈ { 2 , 3 , 5 } we used the same value of m in equation 1 which manages the maximum number of allowed infeasible solutions in a row.

Original: We set m = n + 1 to allow each of the n agents to decline the offered integration of a node from the pool once.
Revised: We set m = n + 1 to allow each of the n agents to decline the offered integration of a node from the pool once.

Original: The penalty is awarded as soon as one agent declines for a second time.
Revised: The penalty is awarded as soon as one agent declines for a second time.

Original: See Table 2 for a detailed overview about all experiment configurations.
Revised: See Table 3 for a detailed overview about all experiment configurations.

Original: [Table]
Revised: [Table]

Original: rewriting steps T      num candidates actions Z      max num pool filled m      num trained epochs      policy learning rate factor α 1e1e- 1e1e- 1e5e- 5 https:

Original: A. M ORE DETAILS ON THE EVALUATION
Revised: A M ORE DETAILS ON THE EVALUATION

Original: We report the performance gaps for the MANR, respectively MANR best, in terms of the percentage cost reduction relative to the intitial solution and relative to OR-Tools for the setup of 20 customer nodes together with the corresponding run times in Table 3.
Revised: We report the performance gaps for the MANR, respectively MANR best, in terms of the percentage cost reduction relative to the intitial solution and relative to OR-Tools for the setup of 20 customer nodes together with the corresponding run times in Table 4.

Original: Absolute performance values for the setups of 10 and 20 nodes are visualized in Figures 6 and 7 .
Revised: Absolute performance values for the setups of 10 and 20 nodes are visualized in Figures 6 and 7 .

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: A.7. E XEMPLARY R EWRITING ROLLOUT
Revised: A.8. E XEMPLARY R EWRITING ROLLOUT

Original: We visualize a rewriting rollout of a routing problem with 10 nodes and 3 agents.
Revised: We visualize a rewriting rollout of a routing problem with 10 nodes and 3 agents.

Original: We show the first 15 global states.
Revised: We show the first 15 global states.

Original: For the rest of the rewriting episode of 100 steps, the MANR stayed in the last depicted solution.
Revised: For the rest of the rewriting episode of 100 steps, the MANR stayed in the last depicted solution.