G ROUNDING P LANNABLE L IFTED A CTION M ODELS FOR V ISUALLY G ROUNDED L OGICAL P REDICATES
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose FOSAE++, an unsupervised end-to-end neural system that generates a compact discrete state transition model (dynamics / action model) from raw visual observations.
Our representation can be exported to Planning Domain Description
Language (PDDL), allowing symbolic state-of-the-art classical planners to perform high-level task planning on raw observations.
FOSAE++ expresses states and actions in First-Order Logic (FOL), a superset of so-called object-centric representation.
It is the ﬁrst unsupervised neural system that fully supports FOL in
PDDL action modeling, while existing systems are limited to continuous, propositional, or property-based representations, and/or require manually labeled input.
I NTRODUCTION
Learning a high-level symbolic transition model of an environment from raw input (e.g., images) is a major challenge in the integration of connectionism and symbolism.
Doing so without manually deﬁned symbols is particularly difﬁcult as it requires solving both the Symbol Grounding (Harnad, 1990; Taddeo & Floridi, 2005; Steels, 2008) and the Action Model Learning/Acquisition problem.
Recently, seminal work by Asai & Fukunaga (2018, Latplan) that learns discrete planning models from images has opened the door to applying symbolic Classical Planning systems to a wide variety of raw, noisy data.
Latplan uses discrete variational autoencoders to generate propositional latent states and its dynamics (action model) directly from images.
Unlike existing work, which requires several machine learning pipelines (SVM/decision trees) and labeled inputs (e.g., a sequence of high-level options) (Konidaris et al., 2014), Latplan is an end-to-end unsupervised neural network that requires no manually labeled inputs.
Numerous extensions and enhancements have been proposed: Causal InfoGAN (Kurutach et al., 2018) instead uses GAN framework to obtain propositional representations.
Latplan’s representation was shown to be compatible with symbolic Goal
Recognition (Amado et al., 2018).
First-Order State AutoEncoder (Asai, 2019, FOSAE) extends
Latplan to generate predicate symbols.
Cube-Space AutoEncoder (Asai & Muise, 2020, CSAE) regularized the latent space to a particular form which directly exports to a learned propositional PDDL model (Fikes et al., 1972).
Discrete Sequential Application of Words (DSAW) learns a plannable propositional word embedding from a natural language corpus (Asai & Tang, 2020).
In this paper, we obtain a lifted action model expressed in First-Order Logic (FOL), which is a superset of object-centric (property-based) representation that Machine Learning community recently began to pay attention to 1 , but has long been the central focus of the broader AI community.
In propositional action models, the environment representation is a ﬁxed-sized binary array and does not transfer to a different or a dynamically changing environment with a varying number of objects.
In contrast, lifted FOL representations are generalized over objects and environments, as we demonstrate in Blocksworld with different number of blocks, or Sokoban with different map sizes.
We propose Lifted First-Order Space AutoEncoder (FOSAE++) neuro-symbolic architecture, which learns a lifted PDDL action model by integrating and extending the FOSAE, the CSAE and the
Neural Logic Machine (Dong et al., 2019, NLM) architectures.
The overall task of our system is illustrated in Fig.
The system takes a transition dataset containing a set of pairs of raw observations which are single time step away.
Each observation consists of multiple visual segmentations of the objects.
It learns a lifted action model of the environment by generating the symbols and emits a PDDL (Haslum et al., 2019) encoding for state-of-the-art planning systems.
Contribution Table 1 contains a taxonomy of existing model acquisition systems in chronological order.
FOSAE++ is the ﬁrst system that satisﬁes all features readily available in symbolic action model acquisition systems, while not relying on human-derived symbols.
FOSAE++ generates unnamed symbols by itself — Effectively addressing the long-standing Knowledge Acquisition bottleneck (Cullen & Bryman, 1988) and the Symbol Grounding problem, showing a future direction for high-level symbolic autonomy
[Figure]
then used to reason in the environment using classical planners.
[Table]
P RELIMINARIES AND B ACKGROUND
We denote a multi-dimensional array (tensor) in bold and its elements with a subscript (e.g., x ∈ R N × M , x 2 ∈ R M ), an integer range n ≤ i ≤ m by n..m , a concatenation of tensors a and b in the last axis by a ; b , and the i -th data point of a dataset by a superscript i which we may omit for clarity.
We use the same symbol for a set and its size (e.g., S , and not | S | ) to avoid the clutter.
Finally, B = [0 , 1] ⊂ R .
We assume background knowledge of discrete VAEs with continuous relaxations (included in the appendix Sec. A.1), such as Gumbel-Softmax (GS) and Binary-Concrete (BC)
(Jang et al., 2017; Maddison et al., 2017).
Their activations are denoted as GS and BC , respectively.
2. L IFTED STRIPS/PDDL P LANNING
Planning Domain Description Language (PDDL) is a modeling language for a Lifted STRIPS planning formalism (Fikes et al., 1972) and its extensions (Haslum et al., 2019).
Let F ( T ) be a formula consisting of logical operations {∧ , ¬} and a set of terms T .
For example, when
T = { have ( I , food ) , full ( I ) } , then have ( I , food ) ∧
¬ full ( I ) ∈ F ( T ) .
We denote a lifted STRIPS planning problem as a 5-tuple (cid:104) O, P, A, I, G (cid:105) .
O is a set of objects ( (cid:51) food ), P is a set of predicates ( (cid:51) full ( x ) ), and A is a set of lifted actions ( (cid:51) eat ).
Each predicate p ∈ P has an arity # p ≥ 0 .
Predicates are instantiated/ grounded into propositions P ( O ) =
(cid:83) p ∈ P (cid:0) { p } × O × # p . . .
× O (cid:1) , such as have(I, food) .
A state s ⊆ P ( O ) represents truth assignments to the propositions, e.g., s = { have(I, food) } represents have(I, food)
= (cid:62) .
We can also represent it as a bitvector of size (cid:80) p O # p .
Each lifted action a ( X ) ∈
A has an arity # a and parameters X
= ( x 1 , · · · , x # a ) , such as eat( x 1 , x 2 ) .
Lifted actions are instantiated into ground actions A ( O )
= (cid:83) a ∈ A (cid:0) { a } × O × # a . . .
× O (cid:1) , such as eat ( I , food ) .
a ( X ) is a 3-tuple (cid:104) PRE ( a ) , ADD ( a ) , DEL ( a ) (cid:105) , where PRE ( a ) , ADD ( a ) , DEL ( a ) ∈ F ( P ( X )) are preconditions, add-effects, and delete-effects: e.g., eat( x 1 , x 2 )
= (cid:104){ have ( x 1 , x 2 ) } , { full ( x 1 ) } , { have ( x 1 , x 2 ) }(cid:105) .
The semantics of these three elements are as follows:
A ground action a † ∈ A
( O ) is applicable when a state s satisﬁes
PRE (cid:0) a † (cid:1) , i.e., PRE (cid:0) a † (cid:1) ⊆
s , and applying an action a † to s yields a new successor state a † ( s ) =
( s \ DEL (cid:0) a † (cid:1) ) ∪ ADD (cid:0) a † (cid:1) , e.g., eat ( I , food ) =
“ I can eat a food when I have one, and if I eat one I am full but the food is gone. ”
Finally, I, G ⊆ P ( O ) are the initial state and a goal condition, respectively.
The task of classical planning is to ﬁnd a plan ( a † 1 , · · · , a † n ) which satisﬁes a † n ◦ · · · ◦ a † 1 ( I ) ⊆
G .
2. N EURAL P ROPOSITIONAL /A CTION S YMBOL G ENERATION WITH L ATPLAN
Latplan is a framework for domain-independent image-based classical planning (Asai & Fukunaga, 2018).
It learns a propositional state representation and transition rules entirely from image-based observations of the environment with discrete VAEs and solves the problem using a classical planner.
Latplan is trained on a transition input Tr : a set of pairs of raw data randomly sampled from the environment.
The i -th transition ( o i, 0 , o i, 1 ) ∈
Tr is a pair of observations made before and after an unknown high-level action is performed.
Once trained, Latplan can process a planning input ( o I , o G ) , a pair of raw images corresponding to an initial and goal state of the environment.
The output of Latplan is a data sequence representing the plan execution ( o I , . . .
o G ) that reaches o G from o I .
While the original paper used an image-based implementation, conceptually any form of temporal data is viable for this methodology, e.g., an NLP corpus (Asai & Tang, 2020).
The latest Latplan Asai & Muise (2020) has a training phase and a planning phase.
In the training phase, it trains an end-to-end neural network called Cube-Space AutoEncoder (CSAE) on Tr (Fig. 2, top left).
CSAE is a variational autoencoder modeled by binary and categorical random variables each representing the propositional states and the actions in the classical planning.
The dynamics modeled by these actions directly compiles into a PDDL model.
The network combines BinaryConcrete VAE to produce binary state representation, and a Gumbel-Softmax VAE to produce a a action apply
[Equation]
BinConcrete
Back-to-Logit (BTL) z i,1 ~
z i,
BN + BC a effect =
BatchNormalization apply
MLP (Input) Object feature vectors
Reconstruction @ (1,2) @ (2,0)
Raw observation @ (1,2) @ (0,0) @ (2,0) @ (0,0) o
[Figure]
detection
[Figure]
) z/ z/ d e c o d e
Simplified First Order State AE (SimpleFOSAE) = continuous repr.
discreterepr.
discrete repr.
= sum
[Figure]
o ~ categorical bottleneck layer which assigns a categorical label to each input.
Let o
and o 1 be a pair of observed states in a transition, z
and z 1 be the corresponding binary latent states, and a be the one-hot vector that represents a discrete action label assigned to the transition.
CSAE is a variational autoencoder network that can be formalized as follows:
[Equation]
where ENCODE , DECODE , ACTION , APPLY are arbitrary multilayer perceptrons.
The outputs of
ENCODE and APPLY are activated by Binary Concerete, and the output of ACTION is activated by
Gumbel-Softmax of A categories (hyperparameter).
Assuming that the prior distributions p ( z 0 ) and p ( a ) are uniform Bernoulli and Categorical distributions, respectively, a lower bound (ELBO) of the log likelihood of observing a pair of states ( o 0 , o 1 ) can be as derived follows (Appendix Sec.
A.3.3):
[Equation]
After the training, it generates a propositional classical planning problem ( z I , z G ) from a planning input ( o I , o G ) and export it into PDDL ﬁles together with the action model obtained in (2), which are then solved by Fast Downward (Helmert, 2006), an optimized C++-based solver independent from the neural network.
Finally, it obtains a step-by-step, human-comprehensible visualization of the plan execution by decoding the intermediate states of the plan into images.
The A -dimensional one-hot categorical variable a i in the network performs a clustering on the state transitions with the maximum number of clusters A speciﬁed as a hyperparameter.
The cluster ID is used as its action symbol.
The clustering is performed by its encoder, ACTION ( z i, 0 , z i, 1 )
= a i , which takes a propositional state pair ( z i, 0 , z i, 1 ) and returns a one-hot vector a i of A categories using Gumbel-Softmax.
AAE’s decoder APPLY takes the current state
z
i, 0 and the action a i and reconstructs a successor state z ∼
i, 1 ≈ z i, 1 , acting as a progression function APPLY ( a i , z i, 0 ) =
z ∼ i, 1 .
APPLY is typically just called a “model” in model-based RL literature.
While APPLY can be any network from the training standpoint, such a neural black-box function does not directly translates to a STRIPS action model, preventing efﬁcient search with state-ofthe-art classical planner.
Cube-Space AE (Asai & Muise, 2020) addresses this issue by Back-ToLogit technique (BTL Fig.
2, bottom-left) which modiﬁes APPLY .
Latent state transitions learned by BTL guarantees that the actions and the transitions satisfy the STRIPS state transition rule s (cid:48) =
( s \ DEL ( a )) ∪ ADD ( a ) , thus enabling a direct translation from neural network weights to PDDL modeling language.
Details of the network, the translation method and the proof can be found in the appendix Sec.
A.3.
2. P REDICATE S YMBOL G ENERATION WITH S IMPLE F IRST -O RDER S TATE A UTO E NCODER First-Order State AutoEncoder
(Asai
, Fig. 2, bottom) is an autoencoder that takes a set of object feature vectors , identiﬁes its latent predicate representation, then outputs a reconstruction of the input.
Unlike prior work on relational modeling (Santoro et al., 2017; Zambaldi et al., 2019; Battaglia et al., 2018), this system obtains discrete logical predicates compatible with symbolic systems.
The input is similar to the setting of Ugur & Piater (2015) and James et al.
(2020b): Each feature vector can represent each object in the environment in an arbitrary complex manner.
In this paper, we use segmented pixels combined with bounding box information ( x, y and width, height).
Let o n ∈ R F be an F -dimensional feature vector representing each object and o =
( o 1 , . . .
o O ) ∈
R O × F be the input matrix representing the set of O objects.
FOSAE generates a multi-arity latent representation .
Assume we represent the input with a combination of predicates of different arities.
We denote a set of predicates of arity n as P/n (Prolog notation) and its propositions as P/n ( O ) .
We denote the binary tensor representation of P/n ( O ) as
z /n ∈ B
O × n ... ×
O × P/n , and the latent space is a tuple z =
( z / 1 , . .
, z /N ) , where N is the largest arity.
The total size of its latent space is (cid:80) n = N n =1 O n P/n .
To convert the input o into unary predicates P/ 1 , we apply a 1D pointwise convolutional ﬁlter f 1 of P/ 1 output features over the objects and activate it by BinaryConcrete for discretization, i.e., z / 1 i = BC ( f 1 ( o i ))
∈ B
P/ 1 .
Similarly, for binary relationships P/ 2 , we apply a ﬁlter f 2 over concatenated pairs of objects, i.e., z / 2 ij = BC ( f 2 ( o i ; o j ))
∈ B
P/ 2 .
We can extend this framework to an arbitrary arity n .
The multiarity latent tensors ( z / 1 , . . .
, z /N ) can be ﬂattened, concatenated, then decoded to the reconstruction ˜
o ∈ R
O × F by an arbitrary decoder network (MLP in the original paper).
There is a potential exponential blowup due to O N parameter combinations.
This problem is addressed by attentions in the original work (Asai, 2019), but we omit this feature in this paper for simplicity and due to relatively small N ( < 4 ).
We call our variant as SimpleFOSAE.
2. N EURAL L OGIC M ACHINE (NLM)
The NLM (Dong et al., 2019) is a neural Inductive Logic Programming (ILP) (Muggleton, 1991) system whose inputs and outputs are hand-crafted binary tensors representing propositional groundings of FOL statements.
These binary tensors are in the multi-arity representation ( z / 1 , . .
, z /N )
identical to the latent space of SimpleFOSAE.
NLM provides a key feature that is necessary for learning a lifted First-Order Logic action model: Invariance to the number and the order of objects in the multi-arity representation.
The latter implies permutation equivariance, i.e., when the ﬁrst n axes of each input
z /n ∈
B
O × n ... × O × P/n are reordered by a permutation π , the output axes are also reordered by π .
A more detailed explanation of NLM can be found in the appendix Sec.
A.2.
An NLM contains N COMPOSE operations which are applied to the corresponding elements of ( z / 1 , · · · , z /N ) .
Each COMPOSE is denoted as COMPOSE Q,σ ( z , n ) ∈ B O n ...O × Q , where Q is a constant that speciﬁes the number of output features, and σ is a nonlinearlity.
An NLM output is denoted as NLM Q,σ ( z )
=
( COMPOSE Q,σ ( z , 1) , · · · , COMPOSE Q,σ ( z , N )) .
L IFTING THE A CTION M ODEL
In order to obtain a lifted action model in an unsupervised setting, there are three requirements: (1) white-box action model which is trivially convertible to STRIPS formalism, (2) invariance to the number/order of objects, (3) unsupervised generation of multi-arity predicate symbols.
To our knowledge, no existing systems fulﬁll all requirements: (Simple)FOSAE lacks 1 and 2, CSAE lacks 2 and 3, and NLM lacks 1 and 3 (designed for hand-crafted symbolic data).
[Equation]
action parameters selector in NLM)
[Equation]
(extract the parameter-bound subspace)
[Equation]
(unsupervised action assignment)
[Equation]
(learning the bounded dynamics
[Equation]
(reﬂection to the global dynamics)
[Equation]
decoder in NLM
[Equation]
1 , o ∼∼ i,= DECODE ( z i, 0 ) , DECODE ( z i, 1 ) , DECODE ( z ∼
i, 1 )
Loss: (cid:96) ( o i, 0 , o ∼ i, 0 ) + (cid:96) ( o i, 1 , o ∼ i, 1 ) + (cid:96) ( o i, 1 , o ∼∼ i, 1 )+ (cid:96) ( z i, 1 , z ∼ i, 1 ) + (cid:96) ( z i, 1 † , z ∼ i, 1 † ) + regularization .
We now propose FOSAE++ whose overview is shown above, which addresses all issues at once.
Its overall design follows the CSAE, which ENCODE s the input, identiﬁes the action a i , APPLY -es the action, then DECODE s the results.
It uses FOSAE’s encoder to generate multi-arity latent representation, which is then consumed by NLMs used as a basic building block of other components.
This architecture’s key element is a new component PARAMS and a unique pair of operations called BIND and UNBIND , which intuitively reﬂects the structure of lifted action models.
Suppose we model a lifted action (move ?block ?
from ?to) in Blocksworld (Fig. 1), with effects such as (on ?block ?to) .
Since a lifted model always refers to the objects through its parameters such as ?
to , it cannot affect
( ) binding for move(?from, ?
to) where (?from, ?
to)
=
(c,a) ?
from ?to product with abcd bind( z /3,(c,a)) truth cube z /3 (boolean tensor) of a ternary predicate p(?x,?y,?z) a b c d a bc d a b c d ?x ?y ?
z c a c a c a ?
z ?
x ?
y
Note: axis ordering has changed.
the objects that does not appear in the parameter list, e.g., action (move a b c) cannot affect (on d c) because d (cid:54)∈ { a , b , c } .
We represent this restriction as differentiable matrix operations.
To implement this idea, we ﬁrst added a new attention network x =
PARAMS
( z i, 0 ; z i, 1 ) = ( x 1 , . . .
, x # a ) which attends to the objects, essentially learning the parameters of the action.
We assume all actions have the same number of parameters, and thus # a is a hyperparameter.
Each parameter x
i is a one-hot hard attention vector in B O ( (cid:80)
j
= O j =1 x ij = 1 ) activated by Gumbel-Softmax.
This can later extract a speciﬁc object from the object feature matrix o by an inner product x i · o ( B O R O × F → R F ).
PARAMS has several NLM layers, ending with NLM # a, IDENTITY .
We then extract its unary part of the results ( → R O × # a ), transpose it ( → R # a × O ), then apply # a
-way GumbelSoftmax of O categories ( → B # a × O ).
As a result, the output attends to # a objects in total.
We utilize the attentions in unique operations named BIND and UNBIND .
Recall that the predicates in the effects can refer only to the speciﬁed action parameters.
Therefore, we limit the dynamics to the objects attended by x (Fig. 3).
We iteratively extracts the sub-axes of z /n attended by x using matrix operations (with an abuse of notation)
z † /n =
( x ) n z /n
( ( B # a × O ) n B O n ...O × P/n → B # a n ... # a × P/n ).
We call it BIND ( z /n, x ) , as it binds the parameters X to the values in objects O , similar to the function call in a typical programming language.
It is also similar to applying numpy.take in all dimensions.
Binding z /n for all arities results in BIND ( z , x ) =
( BIND ( z / 1 , x ) , · · · BIND ( z /N, x )) .
We also deﬁne UNBIND ( z † /n, x ) =
( x (cid:62) )
n z † /n ∈
B O n ...O × P/n which restores the original shape, but ﬁlls the cells with zeros for the objects not attended by x .
Example 1.
For a moment let’s ignore that we use a boolean tensor and let z /=
[[1 , 2 , 3] , [4 , 5 , 6] , [7 , 8 , 9]] with O = { 1 , 2 , 3 } which represents a two-arg function f (2 , 1) =
z / 2 2 , 1 = 4 etc.
When we bind z / 2 to (2 , 1) by x = [[ . 01 , .
01]] , we obtain z † /=
x ( z / 2) x (cid:62) ≈
≈ [[5 , 4] , [2 , 1]] , approximating the subspace extraction.
To unbind it, x (cid:62) ( z † / 2) x
≈ [[1 . 1 , 2 . 1 , .
≈ [[1 , 2 , 0] , [4 , 5 , 0] , [0 , 0 , 0]] .
After the subspace extraction with BIND , we use APPLY and ACTION to ﬁnd the dynamics / action model inside the subspace.
Notice that ACTION and APPLY no longer directly refers to each object o i by the index i because BIND already resolves the index reference.
They take ﬂattened binary vectors of size (cid:80) n ( # a ) n P/n as the input / output.
In order to reﬂect the changes applied to the bounded representation to the whole representation, we use UNBIND :
z ∼ i, 1 =
z i, 0 − UNBIND ( z i, 0 † )+ UNBIND ( z ∼
i, 1 † ) .
UNBIND is unique in that the unattended axes have near-0 values (0 at the limit of Gumbel-Softmax annealing in PARAMS ).
Therefore, the propositions not bound by x retain their values from z i, 0 .
Finally, we replace SimpleFOSAE’s MLP decoder with NLM layers.
To match the output shape with the input o ∈ R
O × F , the ﬁnal layer has F features and we use only the unary part of the result tuple, NLM F,σ ( · ) / 1 = COMPOSE ( · , 1 , F, σ ) .
The total loss is the sum of (cid:96) ( o i, 0 , o ∼ i, 0 ) , (cid:96) ( o i, 1 , o ∼ i, 1 ) , (cid:96) ( o i, 1 , o ∼∼ i, 1 ) , (cid:96) ( z i, 1 , z ∼ i, 1 ) , and (cid:96) ( z i, 1 † , z ∼ i, 1 † ) , plus the variational losses for discrete VAEs.
Hyperparameters, choice of loss functions and the tuning process are detailed in the appendix Sec.
C.
The number of parameters in FOSAE++ is not affected by the number of objects, since essentially each NLM layer performs a 1D pointwise convolution over tuples of objects.
[Figure]
E XPERIMENTS
We prepared 3 artiﬁcial environments: Photo-realistic Blocksworld (Asai, 2018) renders an RGB
Blocksworld image with Blender 3D engine and extracts the bounding boxes and the image patches of each object.
MNIST 8-Puzzle (Asai & Fukunaga, 2018) is a 42x42 pixel, monochrome imagebased version of the 8-Puzzle.
Tiles contain hand-written digits (0-9) from the MNIST database (LeCun et al., 1998) and valid moves swap the “0” tile with a neighboring tile, i e., the “0” serves as the “blank” tile in the classic 8-Puzzle.
Sokoban (Culberson, 1998) is a PSPACE-hard puzzle domain whose visualization was obtained from the PDDLGym library (Silver & Chitnis, 2020).
In all datasets, segmentation is already provided by a domain-speciﬁc code, but in principle it can be replaced by the output of object-detectors such as YOLO (Redmon & Farhadi, 2018).
The detail of the data preparation is available in the appendix Sec. D. Fig.
4 shows some comparisons between the input o i, 0 and the reconstruction o ∼ i, 0 .
In Sokoban, FOSAE++ generates up to 63 objects in a single scene.
4. I NTERPOLATION /E XTRAPOLATION TO V ARYING , U NSEEN N UMBER OF O BJECTS
The key characteristics of the FOSAE++ is that once trained, its network structurally generalizes to the different number of objects.
We demonstrate this performance in interpolation/extrapolation tasks, where in the former we randomly drop a certain number of objects from the input, and in the latter, we use an environment with a different distribution or more objects.
In both tasks, FOSAE++ shows an excellent reconstruction for a varying, untrained number of objects (Fig. 5).
4. P LANNING E XPERIMENTS
Finally, we ran Fast Downward Helmert (2006) planning system with LMcut Helmert & Domshlak (2009) heuristics and A ∗ search conﬁguration on the PDDL domains generated by FOSAE++.
For the planning to succeed, FOSAE++ must be accurate not only about the direct reconstruction o ∼ i, 0 , o ∼ i, 1 but also about the dynamics that predicts the successor state
z ∼ i, 1 and its reconstruction o ∼∼ i, 1 .
Due to the time constraint, we only performed the experiments for 8-Puzzle as of now.
From the ﬁxed goal state (the solved state of the puzzle), we performed a domain-speciﬁc Dijkstra search and sampled the initial states optimally L -steps away.
Each initial/goal states are rendered, cropped into object-based input, then encoded by FOSAE++ to produce the symbolic initial/goal states.
The planning results are visualized and manually inspected/validated by us.
The symbolic classical planner correctly produced a solution which can be visualized into correct state transitions.
See
Sec.
E.1 for the visualization of all 20 instances.
[Table]
R ELATED W ORK
James et al.
(2020a;b) build on existing work (Konidaris et al., 2014; 2015; 2018) to ﬁnd an egocentric logical representation that is invariant to the state of the observer in a complex environment, or an object-centric , property-based logical representation.
Both representations are special cases of First-Order Logic representation where all predicates are unary.
These approaches also require a training input that already contains action symbols (e.g., toggle-door ).
Andersen & Konidaris (2017) obtains an MDP-based PPDDL (Probabilistic PDDL) model using Active Learning.
Incorporating
Active Learning in FOSAE++ for self data collection is future work.
Huang et al.
(2019) reported that direct discretization approach performed worse than an approach that plans in the probabilistic representation obtained by neural networks.
However, the question of continuous versus discrete is not conclusive.
They used a naive rounding-based discretization which may cause a signiﬁcant amount of information loss compared to the state-of-the-art discrete variational autoencoders.
Furthermore, they rely on mappings from raw inputs to hand-coded symbolic representations that require supervised training.
Object-based input we used can be obtained from state-of-the-art object-recognition methods such as YOLO (Redmon & Farhadi, 2018).
More recent systems (Greff et al., 2019; Engelcke et al., 2020) can handle shapes not limited to rectangles, and can be trained unsupervised.
Connecting our network with these systems is an exciting avenue of future work.
Eslami et al.
(2016) and Xu et al.
(2019) proposed an attention-based and a Bayesian approach to the problem of variable-sized object reconstruction.
Their work do not address the state dynamics and is orthogonal to our work.
The primary difference from our approach is that they studied on sequentially storing and retrieving a variable amount of information into/from a ﬁxed-sized array, while we store them in a latent representation of the corresponding size.
C ONCLUSION
We proposed a ﬁrst fully neural First-Order Logic Action Model Acquisition system that successfully produces a lifted PDDL planning model from noisy, segmented images alone.
The system generates three types of symbols (action, predicate, and propositional) without manual labeling, and provides STRIPS-compatible explanations of actions.
The learned results are generalized over the objects, thus can extrapolate to different and dynamically changing environments.
We demonstrated that the state-of-the-art classical planner can produce correct solutions (as theoretically guaranteed) only with the interactions with the learned model.
Note that the classical planner we used has no learning components.
Our results partially support the Physical Symbol Systems Hypothesis by
Newell & Simon (1976):
In our word, a formal high-level representation learned from the raw inputs is sufﬁcient for intelligent actions .
Future work includes extensions to the formalisms with higher expressivity, including Probabilistic PDDL (Younes & Littman, 2004), Axioms (Thi´ebaux et al., 2005), ADL (Pednault, 1987), numeric planning (Fox & Long, 2006), or
Hierarchical Task
Networks (Erol et al., 1994; Nau et al., 2003; Nejati et al., 2006).
A E XTENDED B ACKGROUNDS A. D ISCRETE V ARIATIONAL A UTOENCODERS
Variational AutoEncoder (VAE) is a framework for reconstructing an observation x from a compact latent representation z that follows a certain prior distribution.
Training is performed by maximizing the sum of the reconstruction loss and the KL divergence between the latent random distribution q ( z | x ) and the target distribution p ( z ) , which gives a lower bound for the likelihood p ( x ) (Kingma &
Welling, 2013).
While typically p ( z ) is a Normal distribution N (0 , 1) , Gumbel-Softmax (GS) VAE (Jang et al., 2017) and Binary-Concrete (BC) VAE (Maddison et al., 2017) use a discrete, uniform categorical/Bernoulli(0.5) distribution, and further approximate it with a continuous relaxation by introducing a temperature parameter τ that is annealed down to 0.
We denote corresponding activation function in the latent space as GS ( l ) and BC ( l ) , where l and l each represents a logit vector and scalar.
A latent vector z ∈ [0 , 1] C of GS-VAE is computed from a logit vector l ∈ R C by z =
GS ( l ) =
S OFTMAX (( l + G UMBEL (0 , 1)) /τ ) , where C is the number of categories, G UMBEL (0 , 1) =
− log( − log u ) and u ∼ U NIFORM (0 , 1) ∈
[0 , 1] C .
A latent scalar z of BC-VAE is computed from a logit scalar l ∈ R by z = BC ( l ) =
S IGMOID (( l +
L OGISTIC (0 , 1)) /τ ) , where L OGISTIC (0 , 1) = log u − log(1 − u ) and u ∼ U NIFORM (0 , 1) ∈ R .
Both functions converge to discrete functions at the limit τ → 0 : GS ( l ) → arg max( l ) (in one-hot representation) and BC ( l ) → STEP ( l ) =
( l < 0) ? 0 : 1 .
Typically, GS-VAE and BC-VAE contains multiple latent vectors / latent scalars to model the complex input.
There are more complex variations such as VQVAE van den Oord et al.
(2017), DVAE++Vahdat et al.
(2018b),
DVAE# Vahdat et al.
(2018a).
They may contribute to the stable performance, but we leave the task of faster / easier training to the future work.
A. N EURAL L OGIC M ACHINE
The NLM (Dong et al., 2019) is a neural Inductive Logic Programming (ILP) (Muggleton, 1991) system based on First-Order Logic and the Closed-World Assumption (Reiter, 1981).
Given a set of base predicates grounded on a set of objects, NLMs sequentially apply horn rules to draw further conclusions such as a property of or a relationship between objects.
For example, in Blocksworld, based on a premise such as (on a b) for blocks a , b , NLMs can infer (not (clear b)) .
NLM has two unique features: (1) The ability to combine the predicates of different arities, and (2) size invariance & permutation equivaliance on input objects, which is achieved by enumerating the permutations of input arguments.
NLM is designed to work on hand-crafted binary tensors representing propositional groundings of FOL statements.
The format of these binary tensors are the multi-arity representation ( z / 1 , . . .
, z /N )
identical to the latent space of SimpleFOSAE.
NLM is designed for a subset of First Order Logic where every statement is a horn rule, contains no function terms (such as a function that returns an object), and all rules are applied between neighboring arities.
With these assumptions, the statements fall in one of the three types below:
[Equation]
Here, p, p, p, q ∈ P, P, P, Q (respectively) are predicates, X = ( x 1 , . . . )
is a sequence of parameters, F ( T ) is a formula consisting of {∧ , ∨ , ¬ , (cid:62) , ⊥} and T , ( P ∪ P ∪ P ) /
# q is a set of predicates of the same arity as q , and π ( X ) is a permutation of X , which is used for composing the predicates with the different argument orders.
All three rules can be implemented as tensor operations (Fig. 6).
Given a binary tensor z /n of shape
O n . .
.O
× P/n , expand copies the n -th axis to n + 1 -th axis resulting in a shape O n +1 . . .
O × P/n , and reduce takes the max of n -th axis resulting in a shape O n − 1 . . .
O × P/n .
While the original paper aa ab ac ba bb bc cc z/ z/ z/ expand { { { reduce (horizontal copying) a b c a b c a b c (vertical max) a b c a b c aa ab ac ba bb bc ca cb cc aa ab ac ba bb bc ca cb cc
Permute aa ac ca cc 3x3x
(O=3,|P/2|=4) 3x3x(!2 ・ 4) fc x 9 pointwise1D conv output (!
[Figure]
) ab proposed both min and max versions to represent ∀ and ∃ , with enough number of layers only one of them is necessary because ∀ x ; p ( · , x ) =
¬∃ x ; ¬ p ( · , x ) .
Finally the COMPOSE combines the information between the neiboring tensors z /n, z / n − 1 , z / n +1 .
In order to use the information in the neighboring arities ( P , P and P ), the input concatenates z /n with EXPAND ( z / n − 1 ) and REDUCE ( z / n +1 ) ( → O n . .
.O
× C where C = P/n + P/ n − 1 + P/ n +1 ).
Next, a P ERMUTE function enumerates and concatenates the results of swapping the ﬁrst n axes in the tensor ( → O n . .
.O
× (! n · C ) ).
It then applies a n -D pointwise convolutional ﬁlter f n with Q output features ( → O n . . .O
× Q ).
In the actual implementation, this n -D pointwise ﬁlter is merely a 1D convolution performed on a matrix reshaped into O n × (! n · C ) .
It is activated by any nonlinearity σ to obtain the ﬁnal result, which we denote as COMPOSE ( z , n, Q, σ ) .
Formally, ∀ j ∈ 1 ..n, ∀ o j ∈ 1 ..
O ,
COMPOSE ( z , n, Q, σ ) o 1 ··· o n = σ ( f n (Π o 1 ··· o n )) ∈ R Q where Π = P ERMUTE (cid:16) EXPAND ( z / n − 1 ); z /n ; REDUCE ( z / n +1 ) (cid:17) ∈
B O n ...O × (! n · C )
An NLM contains N (the maximum arity) COMPOSE operation for the neighboring arities, with appropriately omitting both ends ( 0 and N + 1 ) from the concatenation.
We denote the result as NLM Q,σ ( z ) =
( COMPOSE ( z , 1 , Q, σ ) , · · · , COMPOSE ( z , N, Q, σ )) .
This horizontal arity-wise compositions can be layered vertically, allowing the composition of predicates whose arities differ more than 1 (e.g., 2 layers of NLM can combine unary and quaternary predicates).
Two minor modiﬁcation was made from the original paper.
First, we use a slightly different tensor shapes:
For the notational conveniency, we use hypercube-dimensional tensors of shape B O n ...O × P/n , instead of the original formulation B O × O − 1 × ...
O − n − 1 × P/n which tries to reduce the size by disallowing the duplicates in the parameters.
Our modiﬁcation does not signiﬁcantly affect the complexity of the representation because the original representation also has O ( O n ) complexity.
Second, we do not use the nullary predicates z / 0 in order to disallow VAEs from encoding environment-speciﬁc information in it.
A. L EARNING D ISCRETE L ATENT D YNAMICS USING B ACK -T O -L OGIT
APPLY ( a i , z i, 0 ) is an arbitrary MLP, i.e., a neural black-box function that does not directly translates to a STRIPS action model, preventing efﬁcient search with state-of-the-art classical planner.
CubeSpace AE (Asai & Muise, 2020) addresses this issue by Back-To-Logit technique which replaces the
MLP.
Back-to-Logit places a so-called cube-like graph prior on the binary latent space / transitions.
To understand the prior, the background of cube-like graph is necessary.
BinConcrete
Back-to-Logit (BTL)
[Figure]
a effect =
BatchNormalization apply
[Figure]
discreterepr
[Figure]
discreterepr
[Figure]
A.3. C UBE -L IKE G RAPHS AND ITS E QUIVALENCE TO STRIPS
Asai & Muise (2020) identiﬁed that state transition graphs of STRIPS planning problems form a graph class called directed cube-like graph (Payan, 1992).
A cube-like graph is a simple 2 undirected graph G ( S, D ) =
( V, E ) deﬁned by sets S and D .
Each node v ∈ V is a ﬁnite subset of S , i.e., v ⊆ S .
The set D is a family of subsets of S , and for every edge e = ( v, w ) ∈ E , the symmetric difference
d = v ⊕ w = ( v \ w ) ∪ ( w \ v ) must belong to D .
For example, a unit cube is a cube-like graph because S = { x, y, z } , V = {∅ , { x } , . . .
{ x, y, z }} , E = { ( ∅ , { x } ) , . . .
( { y, z } , { x, y, z } ) } , D = {{ x } , { y } , { z }} .
The set-based representation can be alternatively represented as a bit-vector, e.g., V (cid:48)
=
We denote a one-to-one | S | -bit vector assignment as f : V → V (cid:48)
= { 0 , 1 } | S | .
In STRIPS modeling, we use a directed version of this graph class.
For every edge e = ( v, w ) ∈ E , there is a pair of sets d = ( d + , d − ) =
( w \ v, v \ w ) ∈ D which satisﬁes the asymmetric difference w
= ( v \ d − ) ∪ d + .
It is immediately obvious that this graph class corresponds to the relationship between binary states and action effects in STRIPS, s (cid:48)
=
( s \ DEL (() a )) ∪ ADD (() a ) .
As a special case, in an undirected STRIPS planning problem where all actions are reversible (i.e., for any action a such that t = a ( s ) , there is an action a − 1 that is able to reverse the effect s = a − 1 ( t ) ), the state space graph is equivalent to an undirected cube-like graph.
A.3. E DGE C HROMATIC N UMBER OF C UBE -L IKE G RAPHS AND THE N UMBER OF A CTION S CHEMA IN STRIPS
Edge coloring of cube-like graphs provides an intuitive understanding of the action schema expressible in STRIPS formalism.
Consider coloring a graph which forms a unit cube (Fig.
8) and whose node embeddings correspond to the latent space of some images.
A cube-like graph on the left can be efﬁciently colored (i.e., by fewer colors) by the difference between the neighboring embeddings.
Edges can be categorized into 3 labels (0 , 0 , ⊕ 1) , (0 , ⊕ 1 , 0) and ( ⊕ 1 , 0 , 0) (6 labels if directed), where each label is assigned to 4 edges which share the same node embedding differences, as depicted by the upward arrows with the common node difference (0 , 0 , ⊕ 1) in the ﬁgure.
The set of node embedding differences corresponds to the set D , and each element of D represents an action, e.g., when the node embeddings are decoded into images, moving toward positive direction of xaxis may result in the “0” tile in the image moving to the right, and moving toward positive direction of z -axis may result in the “2” tile in the image moving to the right — effectively making actions compositional.
In contrast, the graph on the right has node embeddings that are randomly shufﬂed.
Despite having the same topology and the same embedding size, this graph lacks the common patterns in the embedding differences like we saw on the left, thus cannot be efﬁciently colored by the node differences.
(x,y,z)=(0,0,0) (0,0,1) (1,1,0) (1,0,0) (0,1,1) (1,1,1) x y z (0,0,+1) (0,0,+1)
(x,y,z)=(0,0,1) (0,0,0) (1,1,0) (0,1,0) (0,1,1) (1,1,1) (1,0,1) (1,0,0) decode
As such, cube-like graphs can be characterized by the edge chromatic number (minimum edge coloring) according to the node differences.
We now discuss some theoretical properties of the graph coloring with and without the assumptions on the colors and the node differences.
Theorem 1 (Edge chromatic number Vizing (1965)) .
Let the edge chromatic number c ( G ) of an undirected graph G be the number of colors in a minimum edge coloring.
Then c ( G ) is either ∆ or ∆ + 1 , where ∆ is the maximum degree of the nodes.
Theorem 2.
Mapping E
→ D provides an edge coloring and thus c ( G ) ≤ min f | D | .
Proof.
f is one-to-one: w (cid:54) =
w (cid:48)
⇐⇒ f
( w ) (cid:54) =
f ( w (cid:48) ) .
For any set X , f ( w ) (cid:54) =
f ( w (cid:48) )
⇐⇒
X ⊕ f ( w ) (cid:54) =
X ⊕ f ( w (cid:48) ) .
For any adjacent edges ( v, w ) and ( v, w (cid:48) ) , w (cid:54) = w (cid:48) because G is simple (at most one edge between any nodes), thus f ( v ) ⊕ f ( w ) (cid:54) =
f ( v ) ⊕ f ( w (cid:48) ) .
The remainder follows from the deﬁnition.
(cid:3)
For Thm. 2, equality holds for hypercubes.
For a certain embedding dimension | S | , there are graph instances where c ( G ) <
min f | D | (Fig. 9, left).
We “proved” this with an Answer Set Programming solver Potassco Gebser et al.
(:action c1 :precondition (and) :effect (and (when (and ( z0 ) ( not ( z1 )) ( z2 ))
(and ( z0 ) ( z1 ) ( not ( z2 )))) ...
[Figure]
| S | =
no assignments satisfy
Based on these results, we next consider the minimum number of actions required to model an undirected cube-like graph G ( S, D ) as a planning model.
We restrict ourselves to a preconditionfree planning domains in order to focus on the action effects.
We ﬁrst need a lemma in order to focus on undirected graphs.
Lemma 1.
Let P be a precondition-free grounded STRIPS planning problem which contains irreversible actions.
There is a corresponding planning problem P (cid:48) whose state transition graph is identical to that of P and whose precondition relaxation P (cid:48)(cid:48) is reversible.
Proof.
For any irreversible action a in P , add a set of actions A − 1 ( a ) whose size is | A − 1 ( a ) | = 2 | ADD (() a ) ∪ DEL (() a ) | .
Each action a (cid:48) ∈
A − 1 ( a ) contains those effects which encode one of the possible non-deterministic outcomes of reversing a , and contains an unsatisﬁable precondition (e.g., adding a new proposition whose value is constantly false).
Then the state transition graphs of P and P (cid:48) are identical and P (cid:48)(cid:48) is reversible.
(cid:3)
Since we assume precondition-free domains, for any P we could instead consider P (cid:48)(cid:48) to discuss the effects of reversible planning domains.
Let G be an undirected cube-like graph which is isomorphic to the state transition graph of a precondition-free reversible planning model P .
Theorem 3.
Let P c be another planning problem deﬁnition which models G .
The action effects in
P c are allowed to use conditional effects.
Then the minimum number of actions in P c required to model G is c ( G ) .
Proof.
For each color c ∈ 1 ..c ( G ) and for each edge ( v, w ) colored as c , and for each propositional value f ( w ) i ∈ { 0 , 1 } , we add a conditional effect to c -th action.
If f ( w ) i = 0 , add a delete-effect, and f ( w ) i = 1 , add an add-effect.
The effect is conditioned by the full conjunction of f ( v ) using negative preconditions.
See Fig.
9 (right), where all effects are put in one conditional effect.
(cid:3)
Theorem 4.
The minimum number of actions required to model G without conditional effects (i.e., by STRIPS effects) is min f 2 | D | .
Proof.
Each d ∈ D needs 2 actions for forward/backward directions.
(cid:3)
Thm. 3 indicates that conditional effects can compact as many edges as possible into just A = ∆ or ∆ + 1 actions regardless of the nature of the transitions, while Thm. 2 and Thm.
4 indicate that
STRIPS effects require a larger number of actions.
Therefore, merely assigning action symbols to state transitions (which is equivalent to edge coloring) does not result in a compact STRIPS model.
Notice that the vanilla MLP AAE binds an unrestricted edge chromatic number c ( G ) =
∆ or ∆ +by the maximum number of action labels A (a hyperparameter of the network), but does not bind 2 | D | , the edge chromatic number in terms of neighboring node embedding differences.
In order to ﬁnd a compact STRIPS action model, we should instead bind 2 | D | by A and restrict latent state transitions to follow a STRIPS transition rule.
In order to constrain the learned latent state space of the environment to a cube-like graph, we propose Cube-Space AutoEncoder .
We ﬁrst explain a vanilla Space AutoEncoder , an architecture that jointly learns the state and the action model by combining the SAE and the AAE into a single network.
We then modify the APPLY progression to restrict state transitions.
Due to the ﬂexibility of neural networks, the loss enhanced by the restriction automatically propagates to the state representation, i.e., it modiﬁes a state representation in order to reduce the loss produced by the restricted action model.
A.3. V ANILLA S PACE A UTO E NCODER
The vanilla Space AutoEncoder (Fig. 10, right) connects the SAE and AAE subnetworks.
The necessary modiﬁcation, apart from connecting them, is the change in loss function.
The original AAE was trained to optimize the distance between the binary vectors using binary crossentropy (BCE), which is assymmetric in deﬁnition: BCE ( x, y )
= x log y + (1 − x ) log(1 − y ) .
While this was not problematic in the AAE which uses the ﬁxed state representation x and the successor prediction y , it is more natural for Vanilla Space AE to use a symmetric loss that equally affects x and y .
In addition to the loss for the successor prediction in the latent space, we also ensure that the predicted successor state can be decoded back to the correct image o i, 1 .
Thus, the total loss is a sum of the losses for: (1) the main reconstructions (cid:96) ( o i, 0 , o ∼ i, 0 ) and (cid:96) ( o i, 1 , o ∼ i, 1 ) , (2) the successor latent state reconstruction (cid:96) ( z i, 1 , z ∼ i, 1 ) , (3) the image reconstruction from the predicted successor (cid:96) ( o i, 1 , o ∼∼ i, 1 ) , and (4) the KL regularization.
We call the second term as direct loss .
We next formally analyze this training objective.
Given an observed transition ( o 0 , o 1 ) , we assume that o 0 follows N ( o ∼ 0 , σ 0 ) and o 1 follows N ( o ∼∼ 0 , σ 1 ) (See Sec.
2.2 for explanation).
The maximization objective is a log-likelihood of observing a pair of states o 0 , o 1 .
We iteratively derive the lower bounds (ELBO) by inserting several latent variables.
We ﬁrst introduce z 0 , z 1 : a action apply o i, o i, o i, o i, o i, ~ ~ ~~ z i,1 ~
Space AE z i, loss
BinConcrete
Back-to-Logit (BTL) z i,1 ~
z i,
BN + BC a effect =
BatchNormalization apply
MLP (Input
[Figure]
Object feature vectors
Reconstruction @ (1,2) @ (2,0)
Raw observation @ (1,2) @ (0,0) @ (2,0) @ (0,0) o
[Figure]
z/ z/ d
[Figure]
Simplified
First Order State AE (SimpleFOSAE
) = continuous repr.
discreterepr. discreterepr.
= sum
[Figure]
o ~
[Equation]
The ﬁrst term of Eq.
1 is the KL divergence for z 0 , z 1 .
Since all latent variables are assumed to be independent (mean-ﬁeld assumption), p ( z 0 , z 1 ) =
p ( z 0 ) p ( z 1 ) , where p ( z 0 ) , p ( z 1 ) are respectively the prior distributions for the binary latent variable z 0 and z 1 that we discussed in Sec. ?? .
q ( z 0 , z 1 | o 0 , o 1 ) can also be decomposed because, in the encoder modeled by q , z 0 depends only on o 0 and zdepends only on o 1 .
Therefore, the entire KL divergence is divided into individual KL divergence:
[Equation]
We next decompose the second term in Eq.
1 as shown in Eq.
This derivation is possible because o ∼ 0 and o ∼∼ 1 are generated independently in the decoder (the network that produces o ∼ 0 , o ∼∼ 1 from z 0 , z 1 ) modeled by p .
The ﬁrst term in Eq.
3 corresponds to a reconstruction loss for o ∼ 0 (MSE due to p ( o
| z 0 )
=
N ( o ∼ 0 , σ 0 ) ).
[Equation]
Next, we derive the lower bound of the second term in Eq.
3 by introducing a one-hot categorical latent variable a for action labels, and its prior distribution p ( a ) =
Cat ( 1 /A ) (uniform categorical distribution of A categories).
[Equation]
Finally, to complete the Vanilla Space AE network, we should model the second term in Eq.
4 using another latent variable z ∼ 1 .
We further derive the lower bound using the same reformulation:
[Equation]
The second term of Eq.
5 is a reconstruction loss for o ∼∼ 1 which can be computed by assuming p ( o 1 | z ∼ 1 , a , z 0 , z 1 ) =
N ( o ∼∼ 1 , σ 1 ) .
However, what should we use as a prior distribution p ( z ∼ 1 | o 1 , a , z 0 , z 1 ) in the ﬁrst term (KL divergence)?
We set it to be q ( z 1 | o 1 ) , because the choice of a prior is arbitrary and we want that the distribution of the predicted successor state z ∼ 1 to be identical to the distribution of the successor state z 1 directly encoded from the input.
As a result, the total maximization objective is derived as follows:
[Equation]
Interpretation
[Equation]
KL divergence for z 0 in Eq.
[Equation]
KL divergence for z 1 in Eq.
[Equation]
Reconstruction loss (cid:96) ( o 0 , o ∼
[Equation]
in Eq.
[Equation]
KL divergence for a in Eq.
[Equation]
KL divergence between z 1 and z ∼ 1 in Eq.
[Equation]
direct loss
[Equation]
, z 0 , z 1 ) .
Reconstruction loss (cid:96) ( o 1 , o ∼∼ 1 ) in Eq.
[Equation]
Direct loss (cid:96) ( z 1 , z ∼ 1 ) can be computed by the same method introduced in (Sec. ?? ).
We assume q ( z ∼ 1 | o 1 , z 0 , a ) =
Bernoulli( q ∼ 1 ) and q ( z 1 | o 1 ) =
Bernoulli( q 1 ) where q ∼ 1 = σ ( l ∼ 1 ) , q 1 = σ ( l 1 ) , and l ∼ 1 , l 1 are the inputs to the corresponding Binary Concrete activation.
Then the KL divergence is computed as
[Equation]
Coincidentally, this is D KL ( q ( z ∼ 1 | o 1 , z 0 , a )
|| Bernoulli(0 . 5))
+ log 2 + BCE ( q ∼ 1 , q 1 ) , whose last binary cross entropy term similar to the loss function of AAE.
Implementation note: If we instead optimize D KL ( q ( z ∼ 1 | o 1 , z 0 , a ) || Bernoulli(0 . 5))
+ log 2 +
BCE ( z ∼ 1 , z 1 ) , the training becomes slower due to the unnecessary logistic noise inside BC of z ∼ 1 =
BC ( l ∼ 1 ) and z 1 = BC ( l 1 ) .
Implementation note: It is important to bootstrap various terms in the optimization objective.
In our experience, delaying the application of the direct loss until a certain epoch seems crucial, and we suspect the reason is due to the relatively short network distance between the two latent spaces z i, 1 / z ∼ i, 1 compared to o i, 1 / o ∼∼ i, 1 .
If we enable the direct loss from the beginning, the total loss does not converge because z ∼ i, 1 prematurely converges to z i, 1 causing a mode-collapse (e.g. all 0), before the image en/decoder learns a meaningful latent representation.
A.3. C UBE -S PACE AE
Cube-Space AE modiﬁes the APPLY network so that it directly predicts the effects without taking the current state as the input, and logically computes the successor state based on the predicted effect and the current state.
A naive implementation of such a network is shown in Fig.
11 (left).
The EFFECT network predicts a binary tensor of F × 3 using F -way Gumbel-Softmax of 3 categories.
Each Gumbel Softmax corresponds to one bit in the F -bit latent space and 3 classes correspond to the add effect, delete effect and NOP, only one of which is selected by the one-hot vector.
The effects are applied to the
a
Naive method
[Equation]
BinConcrete "Back-to-Logit" effect
[Equation]
a Add
[Equation]
Del
BN effect (ignored) =
Batch Normalization = min 1-x current state either by a max/min operation or its smooth variants (smooth min/max).
Formally, the naive Cube-Space AE is formulated as follows:
B F (cid:51)
z ∼ i, 1 = APPLY ( z i, 0 , a i ) =
min(max( z i, 0 , ADD (() a i )) , 1 − DEL (() a i )) or (8) smin(smax( z i, 0 , ADD (() a i )) , 1 − DEL (() a i )) , where (9)
EFFECT (() a i ) =
GS ( MLP ( a i ))
∈ B F × 3 ,
ADD (() a i ) =
EFFECT (() a i ) 0 ,
DEL (() a i ) =
EFFECT (() a i ) 1 ,
NOP ( a i ) =
EFFECT (() a i ) 2 , smax( x, y ) =
log( e x + e y ) , smin( x, y ) =
[Equation]
log( e − x
[Equation]
e − y ) .
While being intuitive, we found these naive implementations extremely difﬁcult to train.
Our contribution to the architecture is Back-to-Logit (BtL, Fig.
11, right), a generic approach that computes a logical operation in the continuous logit space .
We re-encode a logical, binary vector back to a continuous, logit representation by an monotonic function m .
This monotonicity preserves the order between true (1) and false (0) even after transformed into real numbers.
We then apply effects by adding the continuous state vector with a continuous effect vector.
The effect vector is produced by applying an MLP named EFFECT to the action vector a i .
After adding the continuous vectors, we re-activate the result with a discrete activation (Binary Concrete).
Formally,
[Equation]
We found that an easy and successful way to implement m is Batch Normalization Ioffe & Szegedy (2015), a method that was originally developed for addressing the covariate shift in the deep neural networks.
Additional background: For simplicity, we consider a scalar operation, which can be applied to vectors element-wise.
During the batch training of the neural network, Batch Normalization layer
BN ( x ) takes a minibatch input B = (cid:8) x 1 . . .
x | B | (cid:9) , computes the mean µ and variance σ 2 , then shift and scale each x
i so that the result has a mean of 0 and a variance of 1.
It then shifts and scales the results by two trainable coefﬁcients γ and β .
Formally,
[Equation]
While rescaling the normalized result by γ and β seem to negate the original purpose of normalization, the presense of normalization to mean 0 / variance 1 is crucial.
Notice that the ﬁrst scaling depends on other training examples in the same minibatch, while γ and β is not dynamically adjusted for the minibatch.
For example, imagine two minibatches B 1 and B 2 , where B 1 accidentally tends to contain larger values than B 2 but the variance within B 1 and B 2 are the same.
The bias is canceled by the normalization, and thus the outputs rescaled by γ and β are computed based on a relative scale inside the corresponding minibatch.
Implementation note: Since a i is a probability vector over A action ids and a i eventually converges to a one-hot vector due to Gumbel-Softmax annealing, the additional MLP can be merely a linear embedding, i.e., EFFECT (() a i ) =
Ea i , where E ∈ R F × A and z ∈ B F .
It also helps the training if we apply BatchNorm on the effect vector.
Therefore, a recommended implementation is
[Equation]
where EFFECT (() a i ) =
BN ( Ea i ) .
A.3.
B ACK - TO -L OGIT AND ITS E QUIVALENCE TO STRIPS
States learned by BTL has the following property:
Theorem 5.
(Asai & Muise, 2020)
Under the same action a , state transitions are bitwise monotonic, deterministic, and restricted to three mutually exclusive modes, i.e., for each bit j :
[Equation]
This theorem guarantees that each action deterministically sets a certain bit on and off in the binary latent space.
Therefore, the actions and the transitions satisfy the STRIPS state transition rule s (cid:48) =
( s \ DEL (() a )) ∪ ADD (() a ) , thus enabling a direct translation from neural network weights to PDDL modeling language.
The proof is straightforward from the monotonicity of the BatchNorm and Binary Concrete.
Note that we assume BatchNorm’s additional scale parameter γ is kept positive or disabled.
Proof.
For readability, we omit j and assumes a 1-dimensional case.
Let e =
EFFECT
(() a ) ∈ R , which is a constant for the ﬁxed input a .
At the limit of annealing, Binary Concrete BC becomes a
STEP function, which is also monotonic.
BN is monotonic because we assumed the scale parameter γ of BN is positive, and the main feature of BN also only scales the variance of the batch, which is always positive.
Then we have
[Equation]
The possible values a pair ( z i, 0 , z i, 1 ) can have is (0 , 0) , (0 , 1) , (1 , 0) , (1 , 1) .
Since both STEP and
BN are deterministic at the testing time (See Ioffe & Szegedy (2015)), we consider the deterministic mapping from z i, 0 to z i, 1 .
There are only 4 deterministic mappings from { 0 , 1 } to { 0 , 1 } : { (0 , 1) , (1 , 1) } , { (1 , 0) , (0 , 0) } , { (0 , 0) , (1 , 1) } , { (0 , 1) , (1 , 0) } .
Thus our goal is to show that the last mapping { (0 , 1) , (1 , 0) } is impossible in the latent space (cid:8) . . .
( z i, 0 , z i, 1 ) . . .
(cid:9) produced by BTL.
To prove this, ﬁrst, assume ( z i, 0 , z i, 1 ) =
(0 , 1) for some index i .
Then
[Equation] [Equation]
The second step is due to the monotonicity BN (0) <
BN (1) .
This shows z i, 1 is constantly 1 regardless of z i, 0 , therefore it proves that
( z i, 0 , z i, 1 ) =
(1 , 0) cannot happen in any i .
Likewise, if ( z i, 0 , z i, 1 ) =
(1 , 0) for some index
i
[Equation]
Therefore, z i, 1 = 0 regardless of z i, 0 , and thus ( z i, 0 , z i, 1 ) = (0 , 1) cannot happen in any i .
Finally, if the data points do not contain (0 , 1) or (1 , 0) , then by assumption they do not coexist.
Therefore, the embedding learned by BTL cannot contain (0 , 1) and (1 , 0) at the same time.
(cid:3)
A. E FFECT R ULE E XTRACTION
To extract the effects of an action a from Cube-Space AE, we compute ADD ( a ) =
APPLY ( a , 0 ) and
DEL ( a ) = 1 − APPLY ( a , 1 ) for each action a , where 0 , 1 are vectors ﬁlled by zeros/ones and has the same size as the binary embedding.
Since APPLY deterministically sets values to 0 or 1, feeding these vectors is sufﬁcient to see which bit it turns on and off.
For each j -th bit that is 1 in each result, a corresponding proposition is added to the add/delete-effect, respectively.
In FOSAE++, we extracts the effects from parameter-bounded subspace z i, 0 † , z i, 1 † .
The representation is a tuple z † =
( z † / 1 · · · z † /N ) ,
where z † /n ∈ B # a × n ... × # a × P/n .
BTL then operates on ﬂattened and concatenated binary vectors of size (cid:80) n # a n P/n : The input, the output, and the effect share this shape.
We extract the effects from this BTL vector in the same manner as noted above.
After the extraction, however, each bit is converted to a lifted predicate according to the position.
For example, when a bit that corresponds to z † / 2 1 , 2 , 5 has turned from 0 to 1, then the add-effect contains p5 ( ?arg1 , ?
arg2 ) ,
where ?
arg1 is a parameter used in the lifted PDDL encoding.
We show an example of such a learned PDDL model below, obtained from 8-Puzzle with P/ 1 =
P/ 2 = 333 (reformatted for readability).
Note that we disabled the nullary predicates P/ 0 and z / 0 , which consumes the ﬁrst 333 dimensions in the ﬂattened vector.
Another note is that we also count the number of appearances of each action in the training dataset.
If an action label is never used in the dataset, it is not exported in the resulting PDDL output.
Thus, the index for the action starts from 7 in the example.
(define (domain latent) (:requirements :strips :negative-preconditions) (:predicates (p333?x
[Equation] [Equation]
(p665 ?x
[Equation]
(p666 ?
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
action a7 :parameters (?x0) :precondition (and (p339 ?x0) (p388 ?x0) (p391 ?x0) (p398 ?x0) (p402 ?x0) (p420 ?x0) (p421 ?x0) (p446 ?x0) (p447 ?x0) (p473 ?x0) (p475 ?x0) (p489 ?
x0)
(p491 ?x0) (p502 ?x0) (p516 ?x0) (p559 ?x0) (p588 ?
x0) (p615 ?x0) (p641 ?x0) (p648 ?x0) (p831 ?x0 ?x0)
(p950 ?x0 ?
x0) (not (p333 ?x0))
(not (p371 ?x0))
(not (p375 ?x0))
(not (p388 ?x0)) (not (p402 ?x0)) (not (p406 ?x0))
(not (p421 ?x0))
(not (p447 ?x0))
(not (p454 ?x0)) (not (p504 ?x0)) (not (p508 ?x0)) (not (p519 ?x0))
(not (p524 ?x0))
(not (p526 ?x0)) (not (p562 ?x0))
(not (p584 ?x0))
(not (p593 ?x0))
(not (p617 ?x0))
(not (p640 ?x0))
(not (p652 ?x0))
(not (p824 ?
x0 ?x0))
(not (p892 ?
x0 ?x0))
(not (p926 ?
x0 ?x0))
(not (p975 ?
x0 ?x0))
(not (p994 ?
x0 ?x0))) :effect (and(p349 ?x0)
(p361 ?x0) (p366 ?x0) (p370 ?x0) (p371 ?x0) (p378 ?x0)
(p381 ?x0) (p385 ?x0) (p388 ?x0) (p401 ?x0) (p408 ?x0)
(p421 ?x0) (p432 ?x0) (p445 ?x0) (p454 ?x0) (p475 ?x0) (p491 ?x0) (p496 ?
x0)
(p502 ?x0)
(p503 ?x0) (p504 ?x0) (p507 ?x0) (p517 ?x0) (p526 ?x0) (p550 ?x0) (p562 ?x0)
(p563 ?x0) (p575 ?x0) (p584 ?x0) (p588 ?
x0)
(p599 ?x0) (p601 ?x0) (p607 ?x0) (p612 ?x0)
(p617 ?x0) (p631 ?
x0)
(p640 ?x0) (p641 ?x0) (p647 ?x0) (p656 ?x0)
(p663 ?x0) (p724 ?
x0 ?x0) (p768 ?
x0 ?x0) (p831 ?x0 ?x0) (p902 ?x0 ?
x0)
(p911 ?x0 ?x0)
(p993 ?
x0 ?x0) (not (p339 ?x0))
(not (p355 ?x0))
(not (p365 ?x0))
(not (p391 ?x0))
(not (p397 ?
x0))
(not (p398 ?x0))
(not (p402 ?x0))
(not (p406 ?x0))
(not (p422 ?x0))
(not (p446 ?x
not
p447 ?x0))
(not (p448 ?x0))
(not (p451 ?x0))
not (p456 ?x0))
(not (p472 ?x0))
(not (p473 ?x0))
not
p478 ?x0))
(not (p489 ?x0))
(not (p490 ?x0))
not
p495 ?x0))
(not (p516 ?x0))
(not (p518 ?x0)) (not
p524 ?x0)) (not (p525 ?x0))
(not (p527 ?x0))
(not (p534 ?x0)) (not (p544 ?x0))
(not (p559 ?x0))
[Table]
not
[Table]
[Table]
not
[Table]
x0))
[Table]
not
[Table]
x0))
[Table]
not
[Table]
x0)) [Table] not
[Table]
[Table]
not
[Table]
)) (not
[Table] [Table]
not
[Table] [Table]
not
[Table]
x0))
[Table]
not
[Table] [Table]
not
[Table] [Table]
not
[Table] [Table]
not
[Table] [Table]
not
[Table]
not
[Table]
(not
[Table] [Table]
A. P RECONDITION L EARNING WITH D YNAMICS R EVERSED IN T IME
In the main text, we simpliﬁed the model by showing only the forward dynamics, i.e., the dynamics in the same direction as the time.
This forward dynamics can model the effects (add/delete) of the actions.
However, the forward dynamics is insufﬁcient for learning the preconditions of the actions.
The original CSAE paper (Asai & Muise, 2020) used an add-hoc method that extracts common bits of the current states.
In contrast, we added a network that uses the same BTL mechanism that is applied backward in time, i.e., predict the current state
z
i, 0 from a successor state z
i, 1 and a one-hot action vector a i .
We named the network REGRESS
( z i, 1 , a i ) , alluding to the regression planning (Alc´azar et al., 2013) literature.
In REGRESS , add-effects and delete-effects now correspond to positive preconditions and negative preconditions .
A positive precondition (normal precondition) requires that a proposition is (cid:62) prior to using an action.
In contrast, a negative precondition requires that a proposition is ⊥ prior to using an action.
While negative preconditions are (strictly speaking) out of STRIPS formalism, it is commonly supported by the modern classical planners that participated in the recent competitions.
To extract the preconditions from the network, we can use the same method used for extracting the effects from the progressive/forward dynamics.
The entire model thus looks as follows.
[Equation]
(bounded forward dynamics)
[Equation]
(bounded backward dynamics)
[Equation]
(reﬂection to global forward dynamics)
[Equation]
(reﬂection to global backward dynamics)
[Equation]
(reconstruction based on forward dynamics)
[Equation]
(reconstruction based on backward dynamics)
[Equation]
The total loss is (cid:96) ( o i, 0 , o ∼ i, 0 )+ (cid:96) ( o i, 1 , o ∼ i, 1 )+ (cid:96) ( o i, 1 , o ∼∼ i, 1 )+ (cid:96) ( o i, 0 , o ∼∼ i, 0 )+ (cid:96) ( z i, 1 , z ∼ i, 1 )+ (cid:96) ( z i, 1 † , z ∼ i, 1 † )+ (cid:96) ( z i, 0 , z ∼ i, 0 ) + (cid:96) ( z i, 0 † , z ∼ i, 0 † ) + Reg .
B I MPLEMENTATION D ETAIL
We based our code on a publicly-available Latplan source code repository ( https://github. com/guicho271828/latplan/ ), which is based on Keras Deep Learning library.
The repository hosts its own Genetic Algorithm based hyperparameter tuner which we mention several times later.
B. I NPUT D ATA F ORMAT AND L OSS F UNCTIONS
As we discussed in the earlier appendix section, our ﬁnal loss function consists of 9 terms: (cid:96) ( o i, 0 , o ∼ i, 0 )+ (cid:96) ( o i, 1 , o ∼ i, 1 )+ (cid:96) ( o i, 1 , o ∼∼ i, 1 )+ (cid:96) ( o i, 0 , o ∼∼ i, 0 )+ (cid:96) ( z i, 1 , z ∼ i, 1 )+ (cid:96) ( z i, 1 † , z ∼ i, 1 † )+ (cid:96) ( z i, 0 , z ∼ i, 0 )+ (cid:96) ( z i, 0 † , z ∼ i, 0 † ) + Reg .
We ﬁrst describe the input data format that is shared among different domains, then the loss functions deﬁned on it.
Our input/output format o ∈ R
O × F consists of O objects (environment-dependent) each having F features.
F features consists of image-based features and the coordinate / dimension-based features (Fig. 12).
All image patches extracted from the observations are resized into a ﬁxed height H , width
W and color channel C = 3 .
Each ﬂattened object vector has the size
F = H × W × C
+ 4 .
The last 4 dimensions contain the center coordinate and the actual height/width before the resizing.
Out of the 9 terms in the total loss, the terms that apply to the object vectors of this form are (cid:96) ( o i, 0 , o ∼ i, 0 ) , (cid:96) ( o i, 1 , o ∼ i, 1 ) , (cid:96) ( o i, 1 , o ∼∼ i, 1 ) , (cid:96) ( o i, 0 , o ∼∼ i, 0 ) .
central coordinate dimensions cx cy h w } }
[Figure]
For this data format, the loss function consists of the mean square value of the image part, and the square sum of the coordinate / dimension parts.
We do not average the losses for the coordinates/dimensions to avoid making the gradient minuscule.
To further enhance this direction, we additionally have the coordinate loss ampliﬁer λ tuned by GA, as it is often the case that the object location has more visual impact on the reconstruction.
Note that, for the tuning with the validation set and the evaluation with the test set, we set λ
= 1 in order to obtain the consistent, comparable measuring.
λ is altered only during the training.
Formally, for the i -th objects in the input o and the reconstruction o ∼ , we deﬁne the loss as follows.
These losses are averaged over the objects and the batch dimensions.
[Equation]
We call the remaining losses except the regularization terms as “latent dynamics loss”.
They operate on the binary latent data activated by BinaryConcrete: (cid:96) ( z i, 1 , z ∼ i, 1 ) , (cid:96) ( z i, 1 † , z ∼ i, 1 † ) , (cid:96) ( z i, 0 , z ∼ i, 0 ) , (cid:96) ( z i, 0 † , z ∼ i, 0 † ) .
However, note that during the training, all values used for the loss calculation are still continuous due to BinaryConcrete’s annealing.
This means we can’t use Binary Cross Entropy BCE , the standard loss function for binary classiﬁcations, because the “training data” is also a noisy probability value.
It is also in fact symmetric — as we discussed in the previous appendix sections, the role of these losses is not only just to obtain the accurate dynamics, but also to shape the state representation toward a cube-like graph .
While there are several candidate loss functions that can be considered, we adapted Symmetric Cross Entropy (Wang et al., 2019) designed for noisy labels, which simply applies BCE in both ways.
Formally, given BCE ( z , z ∼ ) =
− (cid:80) i
( z i log z ∼ i +
[Equation]
z
[Equation]
log
[Equation]
[Equation]
The remaining regularization losses include KL divergence for Discrete VAEs, as well as the L regularization for the latent vectors z i, 0 ,
z i, 1 which were proven useful in Asai & Kajino (2019).
Finally, we deﬁne the magnitude and the warmup of each loss.
The magnitude multiplies each loss and is tuned by the GA tuner.
Similar to λ , the values are set to 1 during the evaluation.
We have α for the L1 regularization, β for KL divergence, and γ for the latent dynamics loss.
The warmup mechanism works by setting these values to 0 until the training reaches a certain epoch deﬁned by the ratio r relative to the total number of epochs.
We used the warmup r α , r γ for α , γ , as well as r rec and the r dyn for the main reconstruction loss (cid:96) ( o i, 0 , o ∼ i, 0 )
+ (cid:96) ( o i, 1 , o ∼ i, 1 ) and the dynamics-based reconstruction loss (cid:96) ( o i, 1 , o ∼∼ i, 1 )
+ (cid:96) ( o i, 0 , o ∼∼ i, 0 ) .
The motivation behind these magnitudes and warmups is to balance the speed of convergence of the various parts of the networks.
Depending on the hyperparameters (depth, width of the layers), occasionally the network completely ignores the dynamics, falling into something similar to (but the mechanism will be very different from) a mode collapse where the effect caused by the dynamics is empty, e.g., the forward dynamics produces the same state as the current state.
Another failure mode of the network is that the dynamics loss is too strong, due to its BCE which could become too large compared to the reconstruction loss.
As a result, the network learns a perfect but meaningless latent space dynamics which does not produce correct reconstructions.
Tuning the warmup and balancing the losses addressed these issues.
B. N ETWORK D ETAIL
FOSAE++ consists of 6 networks: ENCODE , PARAMS , ACTION , APPLY , REGRESS , DECODE .
Note that BIND and UNBIND are weight-less operations
. B.2. E NCODER
ENCODE can be divided into trivial continuous feature extraction phase (pre-encoder) and the actual
FOSAE encoder.
The feature extractor is a simple 1D pointwise convolution over the objects, i.e., same as applying the same Dense/Fully-Connected(FC) layer on each individual object.
Its depth and the hidden layer width is tuned by the GA tuner.
All activations except the last Binary Concrete are Rectiﬁed Linear Unit (Fukushima, 1980; Nair & Hinton, 2010).
The architecture is illustrated in
Fig.
We should note that we assign the same number of predicates to different arities:
P/ 1 = P/ 2 . . .
=
P/N .
In this Network Detail section, we denote P =
P/ 1 . . .
= P/N as a hyperparameter that speciﬁes the number, although P was already used in the main text as the total number of predicates (i.e., (cid:80) n P/n ).
[Figure]
1D
[Figure]
convolution
[Figure]
B.2. P ARAMS
PARAMS consists of multiple NLM layers, as depicted in Fig.
All hidden activations are ReLU, and the width Q of PARAMS is tuned by the GA tuner.
[Figure] [Figure]
Since ACTION takes parameter-bounded representations ( z i, 0 † , z i, 1 † ) whose objects are already selected and appropriately reordered by BIND , we simply apply an MLP whose last layer has an output size A , and is activated by Gumbel-Softmax.
This results in selecting one action, represented by a one-hot vector.
Before applying the MLP, we should ﬂatten and concatenate the tuple of vectors, each of size # a × n . . .
× # a × P .
The depth and the hidden layer width of the MLP is automatically tuned.
For APPLY and REGRESS , we similarly ﬂatten the input and directly apply the BTL structure described in the earlier sections.
The EFFECT network in z ∼
i, 1 = BC ( BN ( z i, 0 )
+ EFFECT ( a i )) is a single linear layer without bias combined with a batch normalization.
In other words, with the weight matrix W ,
[Equation]
B.2. D ECODER
The decoder consists of NLMs followed by a post-decoder that shares the same width and depth as the pre-encoder.
In Blocksworld, we additionally made the decoder a Bayesian Neural Network to absorb the uncertainty in the reconstruction.
Details are available in (Sec. D.2).
B. H YPERPARAMETER T UNER
The tuning system assumes that the hyperparameter conﬁguration is a vector/dict of categorical/ﬁnite candidates.
The tuner is a textbook integer GA with uniform crossover and point mutation.
Assume that a hyperparameter conﬁguration is represented by H values.
A new conﬁguration p = { p 1 , · · · , p H } is created from two parents q = { q 1 , · · · , q H } and r = { r 1 , · · · , r H } by ∀ i ; p i =
RandomChoice
( q i , r i ) , and then a single value is randomly mutated, i.e., for m =
RandomChoice (1 ..H ) , p m ← RandomChoice ( ValidValuesOf ( p m ) \ { p m } ) .
It stores all evaluated conﬁgurations and never re-evaluates the same conﬁguration.
In the beginning of the tuning process, it bootstraps the initial population with a certain number of conﬁgurations.
New conﬁgurations are evaluated by the validation loss, then pushed in a sorted list.
A certain number of best-performing conﬁgurations in the list are considered as a “live” population.
In each iteration, it selects two parents from the live population by inverse weighted random sampling of the score, preferring the smaller validation losses but also occasionally selecting a second-tier parent.
Non-performing conﬁgurations will “die” by being pushed away from the top as the algorithm ﬁnds better conﬁgurations.
The evaluation and insertion to the queue is asynchronous and all processes can run in parallel.
C T RAINING D ETAILS AND H YPERPARAMETERS
We trained multiple network conﬁgurations on a distributed compute cluster equipped with Tesla
K80 and Xeon E5-2600 v4, which is a rather old hardware.
The list of hyperparameters are shown in Table 3.
In all experiments, we used the total of 5000 state transitions (10000 states) from the training environments.
The details of data collection for each domain is available in the later sections.
This dataset is divided into training/validation/test sets (90%:5%:5%).
The Genetic Algorithm hyperparameter tuner uses the validation loss as the evaluation metric.
We set a limit of 1500 total runs for each environment, with 100 initial population, and ran maximum 100 processes in parallel for each environment.
As an additional trick, to avoid testing unpromising candidates (e.g., those with diverging loss), the epoch parameter is forced to be 50 in the ﬁrstconﬁgurations and the runs ﬁnish quickly.
The rest of the runs use these initial populations as the parents, but replaces the epoch with an appropriate value selected from the candidates.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Decoder
[Table]
Params/Action/Apply/Regress hidden
[Table]
width
[Table]
Params
[Table]
Action
[Table]
Apply
[Table]
Regress depth
[Table]
Params
[Table]
Action
[Table]
Regress
[Table]
Early
[Table]
Explosion detection
[Table]
C. R EPRODUCIBILITY , T RAINING C URVES ON 3 R UNS
In Fig.
[Table]
we show 3 training runs on each environment with the same hyperparameter conﬁguration found by our tuner.
It reproduces an almost identical, stable curves across 3 runs.
( ) binding for move(?from, ?
to) where (?from, ?
to)
=
(c,a) ?
from ?to product with abcd bind( z /3,(c,a)) truth cube z /3 (boolean tensor) of a ternary predicate p(?x,?y,?z)
[Equation]
Note: axis ordering has changed.
[Figure]
We generated 10000 transitions of each training problem using Dijkstra search from the initial state.
We shufﬂed the 50000 transitions, subsampled 5000 transitions out of 50000, then stored them in a single archive.
The rendering and other data are obtained from PDDLGym library Silver & Chitnis (2020).
Each tile is resized into 16x16 pixels, and the tile ordering is also shufﬂed.
In order to make the training data dimension consistent and make it convenient for GPU-based training, we performed a so-called Random Object Masking which removes a certain number of objects from each state via random selection.
The idea is similar to masking-based image augmentation commonly used in the computer vision literature, but the purpose is different from those: Ours has more emphasis on having the consistent number of objects in each state.
For example, Sokoban training problem
(leftmost in Fig.
16) has 49 tiles, while the problem 1 (the second picture) hastiles.
In the combined archive, the number of objects is set to the smallest among the problems.
We also removed certain tiles that cannot be reached by the player.
For example, in problem 0, the three ﬂoors on the top left corner cannot be reached by the player.
Similarly in problem 1, the bottom right corner is not reachable by the player.
We performed a simple custom reachability analysis using the meta-information obtained from PDDLGym library.
This helped reducing the dataset size and thus the training.
Finally, during the dataset merging, we accounted for the potential location bias caused by the map size difference.
For example, if we preserved the original x, y locations, the tiles tend to be biased around 0 , 0 and the location around ( x, y ) =
(12 , 12) (by tiles) is never occupied by any tile.
To address the issue, for each state pair, we relocated the entire environment by a value selected uniformly randomly from a certain width.
The width is decided from the maximum dimension of all training problems, i.e., 12x12.
For example, a state pair in problem 4 (which has a 10x9 map dimension) will be shifted by a random value between 0 ..
(12 − 10) in x -axis, and 0 ..
(12 − 9) in y -axis.
D. B LOCKSWORLD
We generated 5000 random state transitions using Photorealistic-Blocksworld dataset Asai (2018), which in turn is based on CLEVR Johnson et al.
(2017) dataset generator.
It uses Blender 3D rendering engine to produce realistic effects such as metallic reﬂection, surface materials and shadows.
The objects are extracted from the information available from the generator metadata.
We cropped the image region suggested by the metadata, resized them into 32x32x3 image patches and stored them in an archive.
The size of the objects reported by the metadata may vary and is noisy due to the camera jitter, object location, and the heuristics used for extracting the objects.
This is a case even for the objects that are not moved by the random actions.
Each transition is generated by sampling the current state and then randomly moving a block.
To sample a state, we ﬁrst generate a set of block conﬁgurations (color, material, shape, size), then placing them randomly on a straight line in the 3D environment without collisions.
When we move a block, we select a set of blocks of which nothing else is on top, choose one randomly, pick a new horizontal location and place it at the lowest non-colliding height.
We ensure that the block is always moved, i.e., not stacked on top of the same block, and not on the ﬂoor if it is already on the ﬂoor.
In Blocksworld, we noticed that a same conceptual symbolic action (such as move ) may have a nondeterministic outcome in the image space while each individual concept is discrete and deterministic.
For example, move may relocate a block onto a ﬂoor, but the resulting position is chosen randomly during the dataset generation, i.e., it can be placed anywhere on the ﬂoor.
To model this uncertainty in our framework, we used a Bayesian Neural Network layers in the decoder for Blocksworld domain.
The ﬁnal output o ∈ R
O × F is produced by two NLMs of output feature size F , each producing the mean and the variance vectors µ,
(cid:15) ∈ R O × F , and the output reconstruction is generated by the random sampling:
o = µ + σ · (cid:15) , where (cid:15) is a random noise vector following normal distribution N (0 , 1) .
During the testing (e.g., visualization), the random sampling is disabled and we use the value µ for rendering.
D. 8-P UZZLE
The 8-puzzle domain generator is directly included in the Latplan code base.
It uses MNIST dataset for its tiles and the tile size is set to 16.
Similarly, we generated 5000 random state transitions using the generator.
E PDDL GENERATION AND P LANNING E XPERIMENTS E. E XAMPLE OUTPUT : 8-P UZZLE