Learning-based Synthesis of Social Laws in STRIPS
Abstract
In a multi-agent environment, each agent must take into ac-
count not only the actions it must perform to achieve its goals, but also the behavior of other agents in the system, which usu- ally requires some sort of coordination between the agents.
One way to avoid the complexity of centralized planning and online negotiation between agents is to design an artiﬁcial social system.
This system enacts a social law that restricts the behavior of the agents.
A robust social law enables the agents to reach their goals while keeping them from interfer- ing with each other.
However, the problem of efﬁcient synthesis of such laws is computationally hard, and previously proposed search techniques do not scale well.
In this paper, we propose the use of graph neural networks to predict so- cial laws from a graph-based representation of multi-agent systems.
However, as this prediction can be wrong, we use heuristic search to correct possible mistakes in the network’s prediction ensuring that the produced social law is indeed ro- bust.
Our empirical evaluation shows that this approach beat the previous state-of-the-art in social law synthesis and that it can learn from an imperfect expert, even in the presence of noise.
Introduction
Multi-agent systems are becoming increasingly more common, with applications such as warehouse order fulﬁllment, autonomous cars, drone delivery, and more.
In this paper, we focus on multi-agent systems with classical planning capabilities, known as multi-agent planning (MAP) (Borrajo and Fern ´ andez 2019).
Automated MAP solves the problem of generating sequences of actions that achieve the individual goals of the agents starting from a given initial state.
The design of such agents and environments is quite different from the design of an isolated agent and should take into account conﬂicts between agents, communication, privacy, etc.
In a multi-agent setting, a plan that would have allowed an agent to obtain its goals in solitude may yield unexpected consequences.
As Helmut von Moltke put it “no plan survives contact with the enemy”.
Broadly speaking, the approaches to solve multi-agent planning can be divided into two categories: centralized and distributed, where each of the approaches has its drawbacks (Wooldridge 2009; Shoham and Leyton-Brown 2009).
Centralized planning methods, usually, refer to the whole system as one planning problem, which neglects the privacy issues and usually do not scale well due to exponential growth in the number of agents.
While being useful in numerous domains, this approach might suffer from limitations, such as the mentioned bottlenecks, or system-wide vulnerability to failure.
On the other hand, the distributed approach requires some sort of coordination between the agents, where the coordination can be either implicit or explicit.
Explicit coordination requires agents to negotiate (usually repeatedly) for conﬂict resolution, which requires time and physical embodiment of the communication devices.
Implicit coordination, in turn, involves an element of system design that either imposes some rules of encounter on the agents or enacts a social law restricting the agents from performing certain actions under some conditions.
Our focus here is on the social law approach (Tennenholtz and Moses 1989), which is a form of implicit coordination.
A social law restricts the allowed behaviors of the agents.
A social law is called robust if, assuming all agents follow the law, they are all guaranteed to achieve their goals.
Recent work (Nir, Shleyfman, and Karpas 2020) proposed an automatic method for synthesizing robust social laws for a given multi-agent environment.
This is done by treating this problem as a heuristic search problem through the space of possible social laws.
However, as this search space is huge (exponential in the number of actions), this technique does not scale well to large problems.
Morales et al.
(2018) proposed a technique of social law synthesis in the context of normative systems.
Their approach is based on an evolutionary algorithm and does not guarantee completeness.
Theoretically, if we had a complete speciﬁcation of the possible instances in a given MAP domain, we could use formal methods to synthesize a generic social law for the domain.
However, MAP domains typically do not have such a speciﬁcation, rather only a description of the schematic predicates and operators, without any speciﬁcation of the possible combinations of initial states and goals (Shleyfman and Karpas 2018).
Thus, the generalization of social laws, and the study of domain structures, though an interesting topic, can not provide guarantees about unseen problem instances.
Instead, we focus on a statistical machine learning approach, in which we generalize from labeled examples – social laws for MAP instances (MAPs), to social laws in unseen MAPs of the same domain.
Speciﬁcally, we assume access to a domain-expert, who provides us with a set of examples – small instances of the same domain, with a robust social law for each instance.
We introduce a technique for learning from such examples and show that it is possible to generalize from such small examples and use the learned knowledge to synthesize social laws in large problem instances.
Our aim here is to produce a robust social law for a given MAP based on the knowledge provided by the domain-expert.
To do so, we use graph neural networks, combined with a search procedure to correct possible mistakes in the network’s prediction, as a fast, domainindependent solution for the problem of social law generation.
Our technique works by representing multi-agent planning problems as graphs (extending the notion of Problem
Description Graph (PDG) (Pochter, Zohar, and Rosenschein 2011) to the multi-agent setting).
We train a Graph Convolutional Neural network (GCN) (Kipf and Welling 2017) on our labeled examples and use it to predict social laws on larger problems.
However, as there are no guarantees that the predicted social law will be robust, we use the network output to speed up the search for social laws (Nir, Shleyfman, and Karpas 2020) by (a) introducing new heuristics based on the prediction, and (b) seeding the open list with potential social law candidates.
We empirically show that our technique can exploit a few small labeled examples to synthesize social laws for larger
MAPs in the same domain, even when the domain-expert is imperfect.
We also address the drawbacks of this approach, i.e., the domain-expert may not be able to locate all of the possible conﬂicts or produce a fully robust social law for each of the proposed examples.
This is simulated by introducing noise to the social laws produced by the expert.
We show that our approach is noise-tolerant, and can sometimes even beneﬁt from such noise.
Preliminaries
In this work we consider a multi-agent planning (MAP) setting that reasons about multiple non-collaborative, noncommunicating agents.
We start with the deﬁnition of the modiﬁed version of MA - STRIPS (Brafman and Domshlak 2008) introduced by Karpas et al.
This modiﬁcation employs individual goals for agents rather than a single goal for the entire problem.
Formally, a MAP problem in this setting is deﬁned as a tuple Π = (cid:104) F, { A i } ni =1 , I, { G i } ni =1 (cid:105) , where: n ∈ N represents the number of agents, F is a set of facts, I ⊆
F is the initial state, and
G i ⊆ F and A i deﬁne the goal and the actions of agent i , correspondingly.
The set of all actions of the problem and the overall goal of the problem are written as A =
(cid:83) ni
=
A i and G = (cid:83) ni
=
G i .
We say that s is a state of the system Π if s ⊆ F , and S =
F is the set of all states of Π .
Each action a ∈ A i is described by preconditions pre (
a ) ⊆ F , negative preconditions npre ( a ) ⊆
F 1 , add effects add ( a ) ⊆ F , and delete effects
del ( a ) ⊆ F .
We say that the action a is well-deﬁned if pre ( a ) ∩ npre ( a )
= add ( a ) ∩ del ( a ) = ∅ .
The action a is applicable in the state s if pre ( a ) ⊆
s and s ∩ npre ( a ) = ∅ .
The result of such application is be denoted by s (cid:74) a (cid:75) :
= ( s \ del ( a )) ∪ add ( a ) .
In what follows we assume well-deﬁned, unit-cost actions.
We deﬁne a projection of Π for an agent i to be Π i = (cid:104) F, A
i , I, G i (cid:105) .
Note that Π is a regular (single-agent) STRIPS task (Fikes and Nilsson 1971).
A sequence of actions π i is called a plan for the task Π i if the actions are consequently applicable starting from the state I and resulting in a state s , s.t.
G i ⊆ s .
The single-agent plans in the MAP context are referred to as individual plans .
Following, Fiˇser et al.
(2019), for an action set
L ⊆ A , we denote Π
i \ L := (cid:104) F, A i \
L, I, G i (cid:105) , generalizing it to the MAP setting as Π \ L = (cid:104) F, { A i \ L } ni =1 , I, { G i } ni =1 (cid:105) .
We consider an execution model where each agent plans its actions in advance, and a given arbitrary scheduler executes the actions one by one with no given priorities, i.e., the scheduler can execute the plans provided by the agents in any possible order.
At each step of the joint plan execution, the agent chosen by the scheduler acts according to its pending action, changing the state of the world accordingly.
The scheduler repeats this procedure until one of the following happens: either all individual plans are executed, or some agent fails to execute its pending action.
This failure to execute an action is denoted as a conﬂict .
In what follows, we consider the scheduler to be adversarial, i.e., we would like to avoid any possible joint execution that results in either a conﬂict or in reaching a state s where one of the agents has not achieved its goal (for some i it holds that G i (cid:54)⊆ s ).
To ensure that no execution results in such a failure,
Karpas et al.
introduced the notion of social law for MAP– a set restrictions on agents’ behavior that are meant to prevent conﬂicts.
They described a social law as a modiﬁcation to a MAP task Π that can add, delete or modify facts, actions, the initial state, or the goal, and showed that some restrictions require complex conditions, for which the original set of facts is insufﬁcient.
In this work, we follow in the footsteps of Nir et al.
(2020), which concentrated only on social laws that restrict conﬂict-inducing actions, while preserving the reachability of all agent goals, without altering the initial set of facts.
Robust Social Laws
In this work we follow the deﬁnition of rational robustness given by Karpas et al. , a social law is considered to be robust if it prevents any possibility for a conﬂict between the agents, whatever plans the agents choose to execute.
Formally characterizing it as follows:
Deﬁnition 1.
A MAP Π is robust to rational iff an action sequence that arbitrarily interleaves any individual plans of all agents achieves the individual goals of these agents.
In what follows, we address the social laws under which a MAP Π is robust to rational, or simply robust .
To this end, we refer to the deﬁnition of social laws by Nir et al.
Deﬁnition 2.
For a MAP Π with total action set A , we call a set of restricted actions L ⊆
A a social law.
We say that a social law L is robust if the restricted MAP Π \ L is robust.
In other words, a social law L is robust if it allows any scheduling of the agents’ plans: preventing any possible conﬂicts and ensuring that each agent will reach its goal at the end of the execution, i.e., that any scheduling of plans for the modiﬁed agent projections
Π i \ L yields no conﬂict.
The procedure proposed by Karpas et al.
to verify that a
MAP Π \ L is robust can be seen as two independent subprocedures, that both solve classical planing problems:
IS SOLVABLE FORALL
(Π \ L ) checks whether for each agent i the projection Π
i \ L is solvable, i.e. agent i can reach its goal.
We say that a social law L is feasible if the procedure returns true .
FIND CONFLICT (Π \ L ) ensures that there are no conﬂicts in any possible schedule.
This is achieved by the construction of a classical planning task whose goal is to reach a conﬂict, where unsolvability means that there are no scheduling interferences.
The social law L is considered to be robust if the ﬁrst subprocedure returns true (the social law is feasible), and the second returns unsolvable (there are no conﬂicts).
Here it is important to note that while the unsolvable problems were mostly ignored in classical planning in the previous decades, considerable progress was achieved in the past years (Muise and Lipovetzky 2015).
Example: We illustrate these concepts with a toy problem from the ROVERS domain (see Figure 1) with two locations
L 1 , L 2 , two rovers R 1 , R 2 and, one rock sample to collect.
The actions that can be taken by the rovers are move – that moves rover between locations and collect – collects a rock sample while destroying the rock (the action is irreversible).
Suppose that the rock sample is to be taken by R 1 rover from L 2 and both rovers are initially positioned at L 1 .
For simplicity, the goal of R 2 is considered to be an empty set.
Two possible robust social laws for this task are: forbid the rover R 2 to take the sample sample ( R 2 , L 2 ) , or forbid the rover R 2 to move to location L 2 , which results in the social law L = { move ( R 2 , L 1 , L 2 ) } .
Note that if R 2 cannot move to L 2 , it also cannot collect the rock sample located at L 2 .
The Search for Social Laws
In their work, Nir et al. introduced a forward search approach that returns a robust social law for a given MASTRIPS problem Π , if there is one.
The problem consists of the following components:
• The original MA - STRIPS problem is considered to be the initial state s 0 .
This is the MAP Π equipped with the empty social law L 0 = ∅ .
• The transition function τ ( s, a ) , where s is the state associated with the social law L ( s = Π \ L ) and a is an action of the original MAP Π .
The application of a in the state s results in the state/MAP s (cid:48)
=
Π \ ( L ∪ { a } ) , with the associated social law L (cid:48)
=
L ∪ { a } .
• The goal test function, as mentioned in the previous subsection, is the test for two independent sub-procedures: IS SOLVABLE
FORALL
( s ) ∧ ¬ FIND CONFLICT ( s )
The size of this search space is 2 | A | where A is set of all actions in Π , and given a ﬁxed planning task, each state can be seen as a set of the actions forbidden by its social law.
Unfortunately, uninformed search algorithms, such as BFS or DFS, do not scale well.
Thus, Nir et al.
proposed various techniques, such as pruning, heuristics and preferred operators identiﬁcation, to speed up the search.
In this paper, we will rely on learning using graph convolutional neural networks, described next, to better guide the search.
Graph Convolutional Neural Networks
To predict a social law based on previously seen social laws for other MAPs, we need to introduce some compact representation that encapsulates the complex relations between the components of a given MAP.
We use a graph representation where facts, actions, and agents correspond to graph vertices, while the relations between them are encapsulated in graph edges.
Next, we need a scalable approach for supervised learning on graph-structured data.
The method we chose is based on an efﬁcient variant of convolutional neural networks which operates directly on graphs, i.e., Graph Convolutional Neural networks (GCN) (Kipf and Welling 2017).
This model scales linearly in the number of graph edges and learns hidden layer representations that encode both local graph structure and features of nodes.
The GCN models we use classify nodes by propagating information through a multi-layer architecture with convolutional layers that capture structural patterns.
Speciﬁcally, given a graph G =
( V , E )
, the network’s input is:
• A feature vector x v ∈ R d for every node v ∈ V ; summarized in a |V| × d feature matrix X , where d is the feature vector dimension, and • An adjacency matrix E ∈ { 0 , 1 } |V|×|V| (where the matrix entry e uv = 1 if e uv ∈ E or u = v , and 0 otherwise).
Note that E represents the edges of the graph G .
Every layer in this network is written as a non-linear activation function σ , where
[Equation]
with H (0) =
X and H ( K ) the output feature matrix, K being the number of layers.
The various GCN models presented in the literature mostly differ in how σ ( · , · ) is constructed and parameterized (Duvenaud et al. 2015; Li et al.
2016; Jain et al. 2016).
In the following sections, we introduce a method that, given some training MAP examples and their corresponding social law, yields a robust social law for unseen examples from the same domain.
In other words, given training data: { (Π j , L j ) } mj =1 , where: Π j = (cid:104) F, { A i } ni =1 , I, { G i } ni =1 (cid:105) is a
MA - STRIPS planning problem with | A | = k being the number of total grounded actions and, L j ∈ 2 k is a label vector which denotes which actions are to be restricted by a possible robust social law for this planning problem, it is required to build a model that can formulate a robust social law, i.e. a subset of actions L ⊆ A .
Social Laws: Graph Representation
In the next subsection, we show what is the graph representation of a given MA - STRIPS planning problem and the corresponding social law.
Problem Description Graphs
The concept of the problem description graph (PDG) is not new.
In classical planning, PDGs were introduced by
Pochter et al.
(2011) in the setting of symmetry detection for problem formulated in SAS+ (B ¨ ackstr ¨ om and Nebel 1995).
Later on Shleyfman et al.
(2015) rewrote this formulation for STRIPS , and Sievers et al.
(2019) used this approach for learning a portfolio planner selection.
We, however, adopt the PDG for the MA - STRIPS formalism.
Deﬁnition 3.
Let Π = (cid:104) F, { A i } ni =1 , I, { G i } ni =1 (cid:105) be a
MA - STRIPS planning task.
The problem description graph (PDG) of Π is the graph (cid:104) V, E (cid:105) with • Graph vertices V = V F ∪
V A ∪ V Ag
, such that,
V F = V + F ∪ V − F = { v + f , v − f | f ∈ F } ,
V A =
V main A ∪ V
pre A ∪ V eﬀ A
= { v main a , v pre a , v eﬀ a | a ∈ A } ,
V Ag = { v i | i ∈ { 1 , . . . , n }} .
• E is the graph edges set:
E = E F ∪ E A ∪ E Ag , where
E F = {{ v + f , v − f } | f ∈ F } ,
E A = (cid:91) a ∈ A (cid:0) E main a ∪ E pre a ∪ E eﬀ a (cid:1) , where
E main a = {{ v main a , v pre a } , { v main a , v eﬀ a } | a ∈ A }
E pre a = {{ v + f , v pre a } | f ∈ pre ( a ) }∪ {{ v − f , v pre a } | f ∈ npre ( a ) }
E eﬀ a
= {{ v + f , v eﬀ a } | f ∈ add ( a ) }∪ {{ v − f , v eﬀ a } | f ∈ del ( a ) } , and
E Ag = {{ v i , v + f } | f ∈ G i } ∪ {{ v i , v main a } | a ∈ A i } have_sample(R1,L2) ¬rock_at(L2) rock_at(L2) at(R1,L2)
R1 isAgent = T collect(R1,L2) collect( ... ).EFF
collect( ... ).PRE
isActionMain =
T isActionEff =
T isActionPre = T isPredicate + = T isGoal= T isPredicate + = T isInit
= T isPredicate + = T isPredicate - = T
• Each v ∈ V has 8 binary attributes as follows:
isPredicate + (
v )
= T for v ∈ V + F else F isPredicate − ( v ) =
T for v ∈ V − F else F isAgent ( v ) =
T for v
∈ V Ag else F isActionMain ( v ) =
T for v ∈ V
main A else F isActionPre ( v ) =
T for v ∈ V pre A else F isActionEff ( v ) =
T for v ∈ V eﬀ
A else F isInit ( v ) =
T for v ∈
( V + f ∩ I ) ∪ ( V − f \ I ) else F isGoal ( v ) =
T for v ∈ V + f ∩ G else F where G =
(cid:83) ni
=1 G i
Informally, the PDG has three types of vertex clusters: V F – each fact can be either true or false, and thus represented by two vertices; V A – actions have three vertices each, the vertices of the action a correspond to the elements of the tuple (cid:104) pre ( a ) , npre ( a ) , add ( a ) , del ( a ) (cid:105) ; V Ag – each vertex here represents an agent.
For example, if f ∈ F is a predicate such that f ∈ I ∩ G j for some agent j , the following attributes hold true for v f : isPredicate
+ ( v f ) , isInit ( v f ) , and isGoal ( v f ) .
The edges indicate the connections between the vertices.
Firstly, we connect the vertices of the clusters of actions and facts.
Secondly, the precondition and effect vertices of each action are connected to their corresponding facts.
Finally, we connect the agents’ vertices to their actions and goal facts.
The vertex attributes are introduced to mark vertex type (agent, fact, etc.) and to specify such conditions as being in the initial state or being in one of the agents’ goals.
Overall, there are 4 attributes for facts, 3 attributes for actions, andattribute to designate agent vertices.
It is important to note that all agent vertices have the attributes, therefore the agents are interchangeable, i.e., there is no particular order on the agent that will make one agent more “important” than the other.
Figure 2 shows part of the PDG for our ROVERS toy problem, corresponding to the action collect ( R 1 , L 1) .
The node
R 1 represents the ﬁrst rover agent.
The node collect ( R 1 , L 2 ) represents the rover’s collect action and, is connected to its corresponding precondition and effect nodes that, in turn, are connected to 4 additional fact nodes.
Data Representation
In our case, we start with m MAP examples { Π 1 , . . .
, Π m } from the same domain, and a domain-expert.
For each of these tasks, the expert produces a social law L i (we assume that these social laws do exist), such that the augmented
MAP Π i \ L i is robust.
Each task Π i is represented by a graph PDG (Π i ) =
(cid:104) V i , E i (cid:105) , where E i ∈ { 0 , 1 } | V i |×|
V
i | is the corresponding adjacency matrix, V i ∈ { 0 , 1 } | V |× 8 is the nodes feature matrix, i.e. each row represents a node and its attributes.
The social law is represented as a vector of vertex labels, i.e. y i
= {− 1 , 0 , 1 } | V i | , where -1 is the label of the nonaction vertices, 0 is the label for action vertices that are not included in the social law, and 1 is the label for the action vertices that are included in the robust social law for Π i .
Using this notation the training data above can be represented as a series of tuples
[Equation]
This data is used to train the GCN-based models.
The exact architecture of the GCN is described in Table 1 and, discussed in the Empirical Evaluation section.
Learning-Based Social Law Synthesis Method
In this section, we introduce a method that utilizes a given dataset, as described above, to train a predictive model.
This model, in turn, is exploited to synthesize a robust social law for an unseen MAP.
The output of such a predictor may contain some errors.
One of our contributions is a procedure that can handle such inaccuracies.
The model may fail on both counts: it can restrict a necessary and completely harmless action, or, on the contrary, allow a conﬂict-inducing action; in both cases, the predicted social law is not robust.
Thus, it is necessary to establish a method that uses the knowledge from the predictor while guaranteeing the robustness of the synthesized social law.
The learning-based social law synthesis method proposed here has three primary steps, which we further describe in the following subsections:
Train and Predict given a dataset with m MAPs and their corresponding social laws from a certain MA - STRIPS domain, we train a network to predict a social law for an unseen MAP from the same domain.
Preprocessing given a predicted social law for an unseen
MAP, the ﬁrst step we take to ﬁnd a robust social law is performing some relatively low-cost preprocessing to eliminate some easy-to-ﬁnd mistakes resulting in a feasible social law, i.e., one where every agent can still achieve its goal individually.
Search for Robust Social Law after the preprocessing is done, we perform a search through the space of possible social laws, guided by the prediction of our network.
Train and Predict with GCN based models
Let us ﬁrst brieﬂy describe how we train and evaluate GCNbased models to predict social laws.
Network Structure: All models have 4 or 6 convolutional layers, as described in Table 1, followed by a batch normalization layer, a ReLU activation layer and, a 50% dropout.
The network output is passed through a sigmoid function.
[Table] Training: First, we randomly split the given dataset to train, validation, and test subsets according to the following ratios: 50%, 25%, and 25% respectively.
Then, we trainnetworks (see Table 1) with the Adam optimizer (Kingma and Ba 2015) using negative log-likelihood as the criterion.
The training parameters are as follows: two possible learning rates: (1e-4, 5e-5), three possible batch sizes: (2,8,16) and, the default Adam optimizer settings.
We also use an early stop mechanism with 50 epochs of patience, testing for improvement in validation set loss.
We are interested in social laws that are permissive, i.e. restrict only a small fraction of the agents’ actions, which is why the labels in our dataset are imbalanced – with more allowed actions (false labels) than restricted actions (true labels).
Furthermore, as described in the next section, detecting false-positives requires less effort.
Thus, the procedure emphasizes recall over precision and gives a higher weight (x3) to false-negative errors.
Finally, the procedure picks the best network according to its balanced accuracy score, i.e. the arithmetic mean of the true positive and the true negative rates.
Prediction: the predicted social law L M is the set of all graph nodes (actions) that got a positive prediction by the network, i.e., a score higher than 0.5.
Let us deﬁne
M : A → [0 , 1] , a function that maps an action node to its network’s score.
Thus, the predicted social law L M is the set of all actions that got a positive score L M = { a ∈ A |
M ( a ) > 0 . 5 } .
As previously mentioned, enacting L M on Π gives a similar multi-agent problem only without the actions included in L M .
Note that L M is not guaranteed to be robust thus, the purpose of the following steps is to detect and correct possible errors in L M , to produce a robust social law.
Preprocessing of the Predicted Social Law
Here we describe a relatively low-cost procedure to process the predicted social law and ﬁx possible errors.
The procedure goes through L M and detects what restrictions prevent agents from achieving their goal individually, making L M an infeasible social law.
Then, to establish a feasible social law, the procedure removes these parts from the social law, that is, it allows these actions.
Let us note that, while this preprocessing procedure ensures that the produced social law is feasible, there still are no guarantees about its robustness.
Theprocedure consists of two main steps: (a) checking every single-action restriction in L M individually and removing the restrictions that make it infeasible, and (b) per- forming a binary search through the remaining subsets of the social law, aiming to ﬁnd one that is feasible, trying to keep the majority of L M .
In the following, we further describe the above-mentioned steps.
We implement the ﬁrst step by constructing MSL =
{ L 1 , L 2 , . . .
, L n } a set of n atomic (one-sized) social laws s.t.
each L ∈ MSL consists of only one action restriction from L M .
Then, for every L ∈ MSL , the procedure conﬁrms that the affected agent can achieve its goal, i.e. L is feasible, by solving the single-agent projection of the planning problem.
We denote these safe restrictions as applicable restrictions.
We ﬁnish the ﬁrst step by constructing
L Mr ⊆ L M , the reﬁned social law that consists of all applicable restrictions from the predicted social law.
In other words, we discard from L M actions which, if we include them in the social law, will prevent one of the agents from achieving its goal individually.
Despite the ﬁrst step, L Mr may be an infeasible social law, as restricting two actions could prevent some agent from achieving its goal, even if restricting each of these separately does not.
Of course, checking all possible subsets of L M r is not tractable so, to acquire some feasible social law, we introduce the second step of the procedure that executes a binary search to remove parts of L Mr such that the resulting social law is feasible.
The intuition behind this step lies in the assumption that the actions with a lower M ( a ) score (that is, where the network is less certain) should be restricted with lower priority.
Thus, in case of an infeasible social law, the procedure would prefer to discard these actions ﬁrst.
The binary search procedure is implemented by, ﬁrst, constructing a list l =
[ a 1 , a 2 , . .
with the actions in L Mr , sorted by M ( a ) , i.e., the higher score actions (the actions that should be restricted according to the network’s prediction) are at the sufﬁx of the list.
Then, the procedure performs a binary search in l to detect the largest contiguous sufﬁx s.t.
the corresponding social law is feasible, i.e., all agents can achieve their goals — this is veriﬁed by calling
IS SOLVABLE FORALL for every sufﬁx that is checked.
We denote the resulting social law as L Mf – the feasible predicted social law.
Let us note that this step could result in an empty social law.
It is easy to see that L Mf is a feasible social law for Π as the second phase conﬁrms it before producing it.
Also, our empirical evaluations shows that this procedure scales well and can produce large ( ∼ 150 actions) and feasible social laws in less than a minute.
Searching for a Robust Social Law
In the previous section, we described a procedure that ﬁxes one type of mistake in the predicted social law L M , producing a feasible social law L Mf ⊆
L M .
However, we may need to add necessary and more complex modiﬁcations to L Mf to obtain a robust social law.
Here we describe a method, based on the search technique from Nir et al. , to tackle this challenge.
We adapt the social law search technique from Nir et al.
to correct the remaining errors, if there are any, in L Mf .
Most of the search problem settings, as brieﬂy described in the Domain Empty Open List Modiﬁed Open List
Base
[Table]
[Table] [Table] [Table] [Equation]
preliminaries, remain the same as well as the search safe pruning techniques.
Additionally, we suggest 2 ways to use the predictor function.
First, we suggest inserting L Mf to the open list at the beginning of the search, as we assume it will be closer to a robust social law than the standard initial social law, ∅ .
We remark that the empty social law is also put on the open list, so our method remains complete.
Second, we introduce 3 new heuristic functions which rely on the social law predictive function M : A → R to guide the search.
Let us note that our greedy search is set to explore nodes (social laws) with higher heuristics value.
Sum of Scores
[Equation]
This heuristic sums M ( a ) for all actions restricted by the social law ( a ∈ L ), making the search explore social laws that restrict high-score actions, i.e. actions that, according to the network, are more likely to be included in a robust social law.
Note that this heuristic will always prefer to add more actions to the social law, without regard to its size.
Average Score
[Equation]
This heuristic computes the arithmetic mean of M ( a ) restricted by the social law, and thus guides the search towards smaller, less-restrictive social laws.
However, its preference for smaller social laws that include only high-score actions may limit its use when handling larger problems.
Count
[Equation]
This heuristic returns the number of restricted actions with a positive M ( a ) score.
One may use this heuristic when
M ( a ) is noisy and the true robust social law may need to restrict actions that got a negative score.
Empirical Evaluation
We begin with a description of our data generation procedure, and then evaluate our approach on problems from Nir et al.
(2020) that are based on domains from the ﬁrst Competition of Distributed and Multi Agent Planners (Komenda,
Stolba, and Kovacs 2016).
Data Generation
To evaluate our approach, we need a way to generate robust social laws for multiple problems in each domain we use.
Therefore, we implemented automatic social law generators for 4 domains: DRIVERLOG , ZENOTRAVEL , TAXI and,
ROVERS , for which we also have random problem generators.
These social law generators are meant to simulate a domain-expert, but are not perfect – we implemented them based on a small number of problems, and in some cases they do not generalize correctly to other randomly generated problems.
Thus, they simulate an imperfect expert.
As we will see, our technique is still able to use the labels from such an imperfect expert to generate robust social laws.
This setup simulates a single expert who applies the same class of social law to problems from a subset of a multi agent planning domain.
It is true that if multiple experts were to provide social laws for different problems in the same domain, they could formulate different classes of social laws – this is left for future work.
Note that although we manually wrote these social law generators for each domain, for the purpose of the empirical evaluation, the social law learning technique we present and evaluate is domain-independent – the same technique is used in all domains.
In addition to the imperfect labeling mentioned above, we wanted to simulate and control additional errors.
One type of error a domain-expert might make is needlessly restricting some actions, leading to an overly restrictive social law.
To simulate this in our empirical evaluation we added artiﬁcial noise to the social law generator output.
We implement this by randomly adding some percentage of the agents’ actions to the generated social law.
Note that, mistakes that make a social law not-robust can be detected using social law robustness veriﬁcation (Karpas, Shleyfman, and Tennenholtz 2017) thus, the noise we induce is unidirectional, and affects only allowed actions.
Additionally, the number of the allowed actions is usually much smaller than the size of the restricted actions so adding only 5% noise can actually double the size of the social law.
Robust Social Law Synthesis
We have described three different network-based heuristics h sum , h cnt and, h avg , as well as the technique for seeding the open list with the predicted social law.
Thus, we have 6 different greedy search conﬁgurations to compare: using one out of the three heuristics, either with open list modiﬁcation (denoted MOL ) or without (denoted NM ).
These are tested on 80 planning problems from 4 different domains.
Similarly to previous work on automated social law synthesis, the search goal test calls the Fast Downward Stone
Soup (FD) (R ¨ oger, Pommerening, and Seipp 2014) and
SymPA (Torralba 2016) planners for the goal test – checking if the current social law is robust.
We too deploy a sequential order of timed attempts (2-minute attempts (FD and SymPA) followed by 30-minute attempts (again, FD, then SymPA)).
We consider a search successful if it either found or disproves the existence of a robust social law.
Table 2 reports the number of successful searches in each domain with 0% noise and 64 training samples.
The baseline – the best social law synthesis conﬁguration from the previous work (2020) – was able to solve 16 of these problems.
All of the proposed conﬁgurations solved more problems.
These results show that inserting the predicted social law to the search open list (denoted MOL ) leads to at least 5 more successful searches.
Our best conﬁguration, using h sum with MOL solves twice as many problems as the baseline.
The results in the TAXI domain show that there is no robust social law present for all of the solved problems.
In fact, we suspect that this result holds for the rest of the domain.
As expected, this shows that our approach does not have an advantage in proving a robust social law does not exist.
Next, we evaluate the effects of the size and quality of the training set.
Figure 3 shows the number of problems solved ( y axis of the plots in the top row) and the balanced accuracy score ( y axis of the plots on the bottom row) in each domain with various noise levels, for varying sizes of the training set ( x axis).
We calculate the balanced accuracy score on an unseen test set of generated problems and their corresponding social law.
Putting aside TAXI , the results show that 4 training samples are enough to beat the baseline that is marked on the graph as a horizontal dashed line.
Note that the trends in the graphs point that there is a correlation between the number of successful searches and the balanced accuracy score.
Interestingly, a limited amount of noise does not seem to hurt and, in fact, our results show that noise may sometimes help the network in achieving higher balanced accuracy and
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table] [Table]
[Table]
[Table]
[Table] [Table]
[Table]
[Table]
ZNTL248.
ZNTL2071.
ZNTL309.
TAXI7.
NSL
NSL NSL
NSL NSL
NSL
TAXI13.
NSL
NSL NSL
NSL NSL
NSL
TAXI32.
NSL
NSL NSL
NSL NSL
NSL
TAXI39.
DLOG10.
DLOG147.
A VG 320.
= 0%, NSL = robust social law does not exist, ZNTL = zenotravel domain) solving more problems.
Note that adding noise to training data is considered to be a standard data augmentation technique in deep learning because it expands the size of the training dataset.
However, in our procedure, it is not the case because the noise is encoded in training samples.
Of course, we do not add noise to the test set samples.
All in all, 20 problems were solved by all of the conﬁgurations and are used to compare search times and social law length as reported in Table 2.
As our results show, using MOL improves search times, mostly when using h sum and h avg .
The most likely explanation for that is that the predicted social law is closer to the search goal than an empty social law, and thus the search can be shallower.
However, using MOL tends to result in more restrictive social laws.
This may be handled by employing h avg that, as expected, may be slower but tends to regulate the length of the resulted social law.
Conclusion
In this paper, we explored the use of graph neural networks to better guide the process of social law synthesis.
Our empirical evaluation shows that the proposed techniques are scalable; we solve twice as many problems as the previous state-of-the-art.
Furthermore, the results show that these techniques can learn from an imperfect expert, even in the presence of noise.
Given the PDG representation we used is domainindependent, we see two interesting directions for future research: adding domain-speciﬁc features (predicate name for fact nodes, operator name for action nodes, etc.) and, transfer learning between domains that exploits possible domain structure similarities.