I NTEGRATING S YMMETRY INTO D IFFERENTIABLE P LANNING WITH S TEERABLE C ONVOLUTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
We study how group symmetry helps improve data efficiency and generalization for end-to-end differentiable planning algorithms, when symmetry appears in decision-making tasks.
Motivated by equivariant convolution networks, we treat the path planning problem as signals over grids.
We show that value iteration in this case is a linear equivariant operator , which is a (steerable) convolution .
This extends Value Iteration Networks (VINs) on using convolutional networks for path planning with additional rotation and reflection symmetry.
Our implementation is based on VINs and uses steerable convolution networks to incorporate symmetry.
The experiments are performed on four tasks: 2D navigation, visual navigation, 2 degrees of freedom (2DOFs) configuration space and workspace manipulation.
Our symmetric planning algorithms improve training efficiency and generalization by large margins compared to non-equivariant counterparts, VIN and GPPN.
I NTRODUCTION
solutions
Model-based planning usually struggles in complex problems, where a solution is to apply planning in more structured and reduced space (Sutton and Barto, 2018; Li et al., 2006; Ravindran and Barto, 2004; Fox and Long, 2002).
When symmetry exists in a task, it could be used for planning by effectively reducing search space.
However, to use symmetry, existing planning algorithms assumes perfect dynamics knowledge and requires explicitly building equivalence classes (Fox and Long, 1999; 2002; Pochter et al., 2011; Zinkevich and Balch, 2001; Narayanamurthy and Ravindran, 2008), while specific task structure can potentially alleviate these requirements.
We use the path planning problem as an example of symmetry in a task, shown in Figure 1.
Given a map M (top row), the objective is to find optimal actions A
=
SymPlan ( M ) (bottom row) to a given position (red dots).
If we rotated the map g.M (top right), its solution g.A (shortest path) can also be connected by a rotation with the original solution A .
Specifically, we say the task has symmetry since the solutions SymPlan ( g.M ) =
g. SymPlan ( M ) are related by a ⟲ 90 ◦ rotation.
As a more concrete example, the action in the NW corner of A is the same as the action in the SW corner of g.A , after also rotating the arrow ⟲ 90 ◦ .
This is an example of symmetry appeared in a specific task, which can be observed before solving the task or assuming other domain knowledge.
If we can use the rotation (and reflection) symmetry in this task, we effectively reduce the search space by | C 4 | = 4 (or | D 4 | = 8 ) times.
Instead, classic planning algorithms like A* would require searching symmetric states (NP-hard) with known dynamics (Pochter et al., 2011).
Recently, symmetry in model-free deep reinforcement learning (RL) has also been studied (van der
Pol et al., 2020a; Wang et al., 2021).
A core benefit of model-free RL that enables great asymptotic performance is its end-to-end differentiability.
However, it can only effectively handle pixel-level “element-wise” symmetry, such as flipping or rotating state and action together.
This motivates us to combine the spirit of both: is it possible to enable end-to-end differentiable planning algorithms to make use of symmetry in environments?
In this work, we propose a framework, named Symmetric Planning (SymPlan), that allows to (1) avoid explicitly building equivalence classes for symmetric states while (2) realize planning in an end-to-end differentiable manner.
We are motivated by work in the equivariant network and geometric deep learning community (Bronstein et al., 2021a;
Cohen et al., 2020; Kondor and Trivedi, 2018; Cohen and Welling, 2016a;b; Weiler and Cesa, 2021): view geometric data as signals over a base space.
For example, an RGB image is a signal, written as mapping Z 2 → R 3 .
The theory in equivariant networks allows to inject symmetry into operations between signals by equivariant operations, such as convolutions.
It satisfies our key desiderata: equivariant networks on images do not need to explicitly consider “symmetric pixels” while guarantee symmetry properties.
This avoids searching symmetric states.
We use the intuition to study a straightforward but general task: path planning.
We focus on 2D grid and prove that value iteration (VI) for 2D path planning is equivariant under translations, rotations, and reflections (isometries of Z 2 ), and further show that VI for path planning is an instance of steerable convolution network (Cohen and Welling, 2016a).
In practice, we use Value Iteration
Network (VIN, (Tamar et al., 2016a)) and its variants, since they only need operations between signals.
We implement the equivariant steerable version of VIN, named SymVIN, and use a variant,
GPPN (Lee et al., 2018), to build SymGPPN.
Both SymPlan methods achieve great improvement on training efficiency and generalization performance to unseen random maps, which showcases the advantage of exploiting symmetry from environments for planning.
Our contributions include:
• We propose a framework to incorporate symmetry into planning for path planning problems (on 2D grids).
We also provide the derivation in detail in appendix.
• Since the framework proves that value iteration for path planning is a steerable CNN, we implement SymVIN by replacing the 2D convolution with steerable convolution.
• Show significant improvement in training and generalization on 2D navigation and manipulation.
R ELATED WORK
Planning with symmetries (Symmetric Planning).
Symmetries widely exist in various domains, and have been exploited in classic planning algorithms as well as model checking (Fox and Long, 1999; 2002;
Pochter et al., 2011; Domshlak et al.; Shleyfman; Shleyfman et al., 2015; Sievers et al.
Wehrle et al.
; Abdulaziz et al.
; Sievers et al., 2015; Sievers; Winterer et al.
; Röger et al., 2018; Sievers et al., 2019; Fišer et al., 2019).
Zinkevich and Balch (2001) show the invariance of value function for an MDP with symmetry.
Narayanamurthy and Ravindran (2008) prove that finding exact symmetry in MDPs is graph isomorphism complete.
However, they are based on classic planning algorithms, such as A*, and have a fundamental issue with exploitation of symmetries: they explicitly construct equivalence classes of symmetric states, which explicitly represents states and introduces symmetry breaking.
Therefore, they are intractable (NP-hard) in maintaining symmetries in trajectory rollout and forward search (for large state space and symmetry group) and incompatible with differentiable pipelines for representation learning, hindering it from wider applications in RL and robotics.
State abstraction for detecting symmetries.
Coarsest state abstraction aggregates all symmetric states into equivalence classes, studied in MDP homomorphisms and bisimulation (Ravindran and
Barto, 2004; Ferns et al., 2004; Li et al., 2006).
However, they usually require perfect MDP dynamics knowledge and do not scale up well, because of the complexity in maintaining abstraction mappings (homomorphisms) and abstracted MDPs.
van der Pol et al.
(2020b) integrate symmetry into model-free RL based on MDP homomorphisms (Ravindran and Barto, 2004), which avoids the challenges in handling symmetry in forward search.
Park et al.
(2022) learn equivariant transition models, but do not consider planning.
Additionally, the formulation in commonly defined symmetric MDPs (Ravindran and Barto, 2004; van der Pol et al., 2020a; Pochter et al., 2011; Zinkevich and Balch, 2001) is different from our symmetry formulation for path planning, since they study "element-wise" symmetry for every state-action pairs and require reward to be symmetric.
Our reward is not symmetric and we mainly study symmetry of the underlying domain (2D grid), as further discussed in Section C.2.
Symmetries and equivariance in deep learning.
Equivariant neural networks are used to incorporate symmetry in supervised learning for different domains (e.g. grid and sphere), symmetry groups (e.g. translations and rotations), and group representations (Bronstein et al., 2021b).
Cohen and Welling (2016b) introduce G-CNNs, followed by Steerable CNNs (Cohen and Welling, 2016a) which generalizes from scalar feature fields to vector fields with induced representations.
Kondor and Trivedi (2018); Cohen et al.
(2020) study theory on equivariant maps and convolutions.
Weiler and Cesa (2021) propose to solve kernel constraints under arbitrary representations for E (2) and its subgroups by decomposing into irreducible representations, named E (2) -CNN.
Differentiable planning.
Our pipeline is based on learning to plan in a neural network in a differentiable manner.
Value iteration network (VIN) (Tamar et al., 2016b) is a representative work that performs value iteration using convolution on lattice grids, and has been further extended (Niu et al., 2017; Lee et al., 2018;
Chaplot et al., 2021; Deac et al., 2021).
Other than using convolution network, works on integrating learning and planning into differentiable networks include (Oh et al., 2017;
Karkus et al., 2017;
Weber et al., 2018; Srinivas et al., 2018; Schrittwieser et al., 2019; Amos and Yarats, 2019; Wang and Ba, 2019; Guez et al., 2019; Hafner et al., 2020; Pong et al., 2018;
Clavera et al., 2020).
In the theoretical side, Grimm et al.
(2020; 2021) propose to understand the differentiable planning algorithms from value equivalence perspective.
B ACKGROUND
[Figure]
Markov decision processes.
We model the path planning problems as Markov decision processes (MDP) (Sutton and Barto, 2018).
An
MDP is a 5 -tuple M = ⟨S , A , P, R, γ ⟩ , with state space S , action space A , transition probability function P : S ×
A × S → R + , reward function R : S ×
A → R , and discount factor γ ∈ [0 , 1] .
Value functions V : S → R and Q : S ×
A → R represent expected future returns.
The core component behind dynamic programming (DP) based algorithms in reinforcement learning is Bellman (optimality) equation (Sutton and Barto, 2018):
V ( s ) = max a R ( s, a ) + γ (cid:80) s ′ P ( s ′ | s, a ) V ( s ′ ) .
Value iteration is an instance of a dynamic programming (DP) method to solve MDPs, which iteratively applies the Bellman (optimality) operator until convergence.
Path planning.
The objective of the path planning problem is to find optimal actions for every location that navigates to the target in shortest time.
However, the original path planning problem is not equivariant under translation due to obstacles, while VINs (Tamar et al., 2016a) implicitly convert it to an equivalent problem, which has equivariant transition function , thus CNNs can be used to inject translation equivariance.
We visualize the construction of an equivalent “spatial MDP” in Figure 2 (Left), where the key idea is to encode obstacle information in the transition function from map (top left) into the reward function in the constructed spatial MDP (bottom right) as “trap” with −∞ reward.
Further details about construction are in Section F.1 and F.3.
In Figure 2 (Right), we provide a visualization of the representation π ( r ) of a rotation r of ⟲ 90 ◦ , and how an action (arrow) is rotated ⟲ 90 ◦ accordingly.
Value Iteration Network.
Tamar et al.
(2016a)
proposed Value Iteration Networks (VINs) that use a convolution network to parameterize value iteration.
It jointly learns in a latent MDP on 2D grid, which has the latent reward function ¯ R :
Z 2 → R |A| and value function ¯ V : Z 2 → R , and applies value iteration on that MDP:
[Equation]
The first equation can be written as: ¯ Q ( k ) =
¯ R a + Conv2D ( ¯ V ( k − 1) ; W V ¯ a ) , where the 2D convolution layer Conv2D has parameter W V .
Our final goal is to use VIN to demonstrate a principled method for incorporating symmetry in differentiable planning.
We intentionally omit equivariant network details and rather focus on the core idea of integrating symmetry with equivariant networks.
We present the necessary group theory background in Section D and full framework and theory in Section E and F.
M ETHOD : I NTEGRATING S YMMETRY INTO P LANNING BY C ONVOLUTION
In this work, we aim to exploit the inherent symmetry in a broadly existed problem: path planning.
As visualized in Figure 1, the equivariance property unveils the inherent symmetry of the path planning problem on the 2D grid that we could exploit.
We provide a rigorous algorithmic framework that can provably make use of symmetry in an efficient manner.
To keep approachable, we first introduce how to use VIN as the foundation to build our algorithm: Symmetric VIN.
In the next section, we provide the explanation on why we make this choice and introduce further theoretical guarantees on how to exploit symmetry.
How to inject symmetry?
VIN uses a regular 2D convolutional network (Equation 1), which has translation equivariance (Cohen and Welling, 2016b; Kondor and Trivedi, 2018).
More concretely, a
VIN will output the same value function for the same map patches that up to 2D translation.
We omit how to characterize translation equivariance here, since it requires a different mechanism to handle and does not decrease the search space nor reduce a path planning MDP to an easier problem.
Beyond translation, we are more interested in rotation and reflection symmetries.
Intuitively, as in Figure 1, if we find the optimal solution to a map, it automatically generalizes the solution to all 8 transformed maps ( 4 rotations times 2 reflections, including identity transformation).
This can be characterized by equivariance of a planning algorithm Plan , such as value iteration VI : g. Plan ( M ) =
Plan ( g.M ) , where M is a maze map, and g is the symmetry group D 4 under which 2D grid is invariant.
More importantly, symmetry also helps training of differentiable planning.
Intuitively, symmetry in path planning poses additional constraints to its search space: if the goal is in the north, go up; if in the east, go right.
In other words, the knowledge can be shared between symmetric cases, or the path planning is effectively reduced by symmetry to a smaller one.
This property can also be depicted by equivariance of Bellman operators T , or a step of value iteration: g. T [ V 0 ]
= T [ g.V 0 ] .
If we use VI ( M ) to denote applying Bellman operators on arbitrary initialization until convergence
T ∞ [ V 0 ] , value iteration is also equivariant:
[Equation]
We formally prove the equivariance in Theorem 5.1 in next section.
In Theorem 5.2, we theoretical show that value iteration in path planning is a specific type of convolution: steerable convolution (Cohen and Welling, 2016a).
Before that, we take the conclusion and first present the pipeline on how to use Steerable CNNs (Cohen and Welling, 2016a) to integrate symmetry.
Pipeline: SymVIN.
We have shown that VI is equivariant given symmetry in path planning.
We introduce our method Symmetric Value Iteration Network (SymVIN), that realizes equivariant VI by integrating equivariance into VIN w.r.t.
rotation and reflection , in addition to translation .
We use an instance of Steerable CNN: E (2) -Steerable CNNs (Weiler and Cesa, 2021) and their package e2cnn for implementation, which is equivariant under D 4 rotation and reflection, and also Ztranslation on the 2D grid Z 2 .
In practice, to inject symmetry into VIN, we mainly need to replace the translation-equivariant Conv2D in Eq.
1 with SteerableConv :
[Equation]
We visualize the full pipeline in Figure 3.
The map and goal are represented as signal M : Z 2 → { 0 , 1 } 2 .
It will be processed by another layer and output to the core value iteration loop.
After some iterations, the final output will be used to predict the actions and compute cross-entropy loss.
It highlights the injected equivariance property: if we rotate the map (from M to g.M ), to guarantee the final policy function to also be equivalently rotated (from A to g.A ), we shall guarantee every transformation (e.g., Q k (cid:55)→ V k and V k
(cid:55)→ Q k +1 ) in value iteration to also be equivariant , for every pair of columns .
We formally justify our design in the section below and provide more technical details in Section F.
Extension: Symmetric GPPN.
Based on same spirit, we also implement a symmetric version of
Gated path planning network (GPPN (Lee et al., 2018)).
It proposes to use LSTM to alleviate the issue of unstable gradient in VINs.
Although it does not strictly follow value iteration, it still follows the spirit of steerable planning.
Thus, we first obtained a fully convolutional variant of GPPN from [Redacted for anonymous review], called ConvGPPN, which incorporates translational equivariance.
It replaces the MLPs in the original LSTM cell with convolutional layers, and then replaces convolutions with equivariant steerable convolutions, resulting in SymGPPN that is equivariant to translations, rotations, and reflections.
See Appendix H.1 for details.
Why do we choose VIN-based planners?
There are two reasons behind the choice.
The expected value operator in value iteration (cid:80) s ′ P ( s ′ |
s, a ) V ( s ′ ) is (1) linear in value function and (2) equivariant (shown in Theorem 5.1).
Cohen et al.
(2020) show that any linear equivariant operator (on homogeneous spaces 2D grid) is a (group) convolution operator
Value iteration, or Bellman (optimality) operator, consists of only maps between fields/signals over Z (e.g., value map and transition function map).
This enables to inject symmetry by enforcing equivariance to those maps.
Take Figure 1 as example, the 4 corner states are symmetric under transformations in D 4 .
Equivariance enforces those 4 states to have the same value if we rotate or flip the map.
This avoids the need to find if a new state is symmetric to any existing state, which is shown to be NP-hard (Narayanamurthy and Ravindran, 2008).
In summary, VIN satisfies both desiderata: (1) it uses convolution as the backbone, and (2) it operates on fields.
Furthermore, we find VIN is empirically and conceptually the simplest differentiable planning algorithm that satisfies them, which leads to our decision.
T HEORY : V ALUE I TERATION IS S TEERABLE C ONVOLUTION
In the last section, we show how to exploit symmetry in path planning by equivariance from convolution via intuition.
The goal of this section is to (1) connect the theoretical justification with the algorithmic design, and (2) provide intuition for the justification.
Even through we focus on a specific task, we hope that the underlying guidelines on integrating symmetry into planning are useful for broader planning algorithms and problems as well.
The complete version is in Section F.
Overview.
There are numerous types of symmetry in various planning tasks.
We study symmetry in path planning as an example, because it is a straightforward planning problem, and its solutions have been intensively studied in robotics and artificial intelligence (LaValle, 2006; Sutton and
Barto, 2018).
However, even for this problem, the symmetry has not been effectively exploited in its planning algorithms, such as Dijkstra’s algorithm, A
*, or RRT, because of NP-hard orbit finding (Narayanamurthy and Ravindran, 2008).
Additionally, we focus on value iteration because it is both widely use and connects closely with convolution (Cohen and Welling, 2016a).
Symmetry from tasks.
If we want to exploit inherent symmetry in a task to improve planning, there are two major steps: (1) characterize the symmetry in the task, and (2) incorporate corresponding symmetry into the planning algorithm.
The theoretical results in Section F.2 mainly characterize the symmetry and direct us to a feasible planning algorithm.
The symmetry in tasks or MDPs can be specified by the equivariance property of the transition and reward function, studied in Ravindran and Barto (2004); van der Pol et al.
(2020b):
[Equation]
Note that how the group G acts on states and actions is called group representation , and is decided by the space S or A , which has been discussed in Equation 19 in Section F.2.
We emphasize that the equivariance property of the reward function is different from prior work (Ravindran and Barto, 2004; van der Pol et al., 2020b): in our case, the reward function encodes obstacles as well, and thus depends on map input M .
Intuitively, using Figure 1 as an example, if a position s is rotated g.s , to find the correct original reward R before rotation, the input map M must also be rotated g.M .
More details in Section F.
Symmetry into planning.
As for exploiting the symmetry in planning algorithms , we focus on value iteration and the VIN algorithm.
We first prove in Theorem 5.1 that value iteration for path planning respects the equivariance property, motivating us to incorporate symmetry with equivariance.
Theorem 5.1 (informal) .
If transition is G -invariant, expected value operator (cid:80) s ′ P ( s ′ | s, a ) V ( s ′ ) and value iteration are equivariant under translation, rotation, reflection on the 2D grid.
[Figure]
We visualize the equivariance of the central value update step R + γP ⋆ V k in Figure 4.
The upper row is a value field V k and its rotated version g.V k and the lower row is for Q -value fields Q k and g.Q k (each ).
The diagram shows that, if we input a rotated value g.V k , the output R + γP ⋆ g.V k is guaranteed to be equal to rotated Q -field g.Q k .
Additionally, rotating Q -field g.Q k has two components: (1) spatially rotating each grid (a feature channel for an action Q ( · , a ) ) and (2) cyclically permuting the channels black red how a specific grid of a Q -value grid Q k ( · , South ) got rotated and permuted.
We discuss the theoretical guarantees in Theorem 5.1 and provide full proofs in the appendix.
However, this theorem provides intuition but is inadequate since we do not know: how to implement it like CNNs with multiple feature channels as in
VINs, since the first theorem only shows for scalarvalued transition probability and value function.
The next result in Theorem 5.2 further proves that value iteration is a general form of convolution ( steerable convolution ), motivating the use of steerable CNNs by Cohen and Welling (2016a) to replace regular CNNs in VIN.
Cohen et al.
(2020) prove that steerable convolution is the most general linear equivariant map under some conditions, which value iteration satisfies.
Theorem 5.2 (informal) .
If transition is G -invariant, the expected value operator is expressible as a steerable convolution ⋆ , which is equivariant under translation, rotation, and reflection on 2D grid.
The value iteration (with max , + , × ) then forms a deep steerable CNN (Cohen and Welling, 2016a).
We provide a complete version of the framework in Section F and the proofs in Section G.
This justifies why we should use Steerable CNN (Cohen and Welling, 2016a) in implementation, since the VI itself is composed of steerable convolution and additional operations ( max , + , × ).
Summary.
We study how to inject symmetry into VIN for (2D) path planning, and expect the taskspecific technical details are useful for two types of readers.
(i) Using VIN.
If one uses VIN for differentiable planning, the resulting algorithms SymVIN or SymGPPN can be a plug-in alternative, as a part in a larger end-to-end system.
Our framework generalizes the idea behind VINs and enables us to understand its applicability and restrictions.
(ii) Studying path planning.
The proposed framework characterizes the symmetry in path planning, so it is possible to apply the underlying ideas to other domains.
For example, it is possible to extend to even higher-dimensional continuous
Euclidean spaces or spatial graphs (Weiler et al., 2018; Brandstetter et al., 2021).
Additionally, we emphasize that the symmetry in spatial MDPs is different from symmetric MDPs (Zinkevich and
Balch, 2001; Ravindran and Barto, 2004; van der Pol et al., 2020a), since our reward function is not
G -invariant (if not conditioning on obstacles).
We further discuss this in Section C.2 and F.4.
E XPERIMENTS
We experiment VIN, GPPN and our SymPlan methods on four path planning tasks, including using given or learned maps.
The additional experiments and ablation studies are in Appendix I.
Environments and datasets.
We demonstrate the idea in four path planning tasks: (1) 2D navigation , (2) visual navigation , (3) 2 degrees of freedom (2DOFs) configuration space manipulation , and (4) 2DoFs workspace manipulation .
We focus on the 2D regular grid setting for path planning, as adopted in prior work (Tamar et al., 2016a; Lee et al., 2018; Chaplot et al., 2021).
For each task, we consider using either given (2D navigation and 2-DOF configuration-space manipulation) or learned maps (visual navigation and 2-DOF workspace manipulation).
In the latter case, the planner needs to jointly learn a mapper that converts egocentric panoramic images (visual navigation) or workspace states (workspace manipulation) into a map that the planners can operate on, as in (Lee et al., 2018; Chaplot et al., 2021).
In both cases, we randomly generate training, validation and test data of 10 K/ 2 K/ 2 K maps for all map sizes, to demonstrate data efficiency and generalization ability of symmetric planning.
Note that the test maps are unlikely to be symmetric to the training maps by any transformation from the symmetry groups G .
For all environments, the planning domain is the 2D regular grid S
=
Ω
=
Z 2 , and the action space is to move in 4 ⟲ directions 1 :
A = (north, west, south, east) .
Methods: planner networks.
We compare five planner methods, where two are our SymPlan methods.
Our two equivariant methods is based on Value Iteration Networks ( VIN , (Tamar et al., 2016a)) and Gated Path Planning Networks ( GPPN , (Lee et al., 2018)).
Our equivariant version of VIN is named SymVIN .
For GPPN, we first obtained a fully convolutional version, named ConvGPPN [Redacted for anonymous review], and furthermore SymGPPN with steerable CNNs.
All methods use (equivariant) convolutions with circular padding in planning in configuration spaces for the manipulation tasks, except GPPN that is not fully convolutional.
Chaplot et al.
(2021) propose SPT based on Transformers, while integrating symmetry to Transformers is beyond steerable convolutions, thus we do not consider it but still adopt some useful setup.
Training and evaluation.
We report success rate and training curves over 3 seeds.
The training process (on given maps) follows (Tamar et al., 2016a; Lee et al., 2018), where we train 30 epochs with batch size 32 , and use kernel size F = 3 by default.
The gradient clip threshold is set to 5 .
The default batch size is 32 , while we need to reduce for some GPPN variants, since LSTM consumes much more memory.
6. P LANNING ON GIVEN MAPS
Environmental setup.
In the 2D navigation task, the map and goal are randomly generated, where the map size is { 15 , 28 , 50 } .
In 2-DOF manipulation in configuration space, we adopt the setting in (Chaplot et al., 2021) and train networks to take as input of configuration space, represented by
[Table]
two joints.
We randomly generate 0 to 5 obstacles in the manipulator workspace.
Then the 2 degreeof-freedom (DOF) configuration space is constructed from workspace and discretized into 2D grid with sizes { 18 , 36 } , corresponding to bins of 20 ◦ and 10 ◦ , respectively.
All methods are trained using the same network size, where for equivariant versions, we use regular representations for all layers, which has size | D 4 | = 8 .
We keep the same parameters for all methods, so all equivariant convolution layers with regular representations will have higher embedding sizes.
Due to memory constraint, we use K = 30 iterations for 2D maze navigation, and K = 27 for manipulation.
We use kernel sizes F = { 3 , 5 , 5 } for m = { 15 , 28 , 50 } navigation, and F = { 3 , 5 } for m = { 18 , 36 } manipulation.
[Figure]
Results.
We show the averaged test results for both 2D navigation and C-space manipulation tasks on generalizing to unseen maps (Table 3) and the training curves for 2D navigation (Figure 6).
For VIN series, our SymVIN is much better than the vanilla VIN in terms of generalization and training performance in both environments, which learns much faster and achieves almost perfect asymptotic performance.
As for GPPN, we found the fully convolutional variant ConvGPPN actually works better than the original one in (Lee et al., 2018), especially in learning speed.
However, SymVIN does fluctuate in some runs, which seems to come from initialization and label, further studied in Appendix.
SymGPPN further boosts ConvGPPN and outperforms all other methods.
One exception is GPPN learns poorly in
C-space manipulation.
For GPPN, the added circular padding in the convolution encoder leads to gradient vanishing problem.
Additionally, we found using regular representations (for D 4 or C 4 ) for state value V : Z 2 → R C V (and for Q -value) works better than trivial representations.
This is counterintuitive since we expect the V value to be scalar Z 2 → R .
One reason is that switching between regular (for Q ) and trivial (for V ) representation introduces unnecessary bottleneck.
Depending on the choice of representations, we implement different max-pooling, with details in Appendix H.2.
We also empirically found using FC only in the final layer Q K (cid:55)→ A helps stabilize the training.
The ablation study on this and more are in Appendix I.
Remark.
Two symmetric planners are both significantly better than their counterparts.
Notably, we did not include any symmetric maps to the test data that symmetric planners would perform much better.
There are several potential sources of advantages: (1) SymPlan allows parameter sharing across positions and maps and implicitly enables planning in a reduced space: every ( s, a, s ′ ) seamlessly generalizes to ( g.s, g.a, g.s ′ ) for any g ∈ G , (2) thus it uses training data more efficiently, (3) it reduces the space of hypothesis class and facilitate generalization to unseen maps.
6. P LANNING ON LEARNED MAPS : SIMULTANEOUSLY PLANNING AND MAPPING
Environmental setup.
For visual navigation , we randomly generate maps using the same strategy as before, and then render four egocentric panoramic views for each location from produced 3D environments with Gym-MiniWorld (Chevalier-Boisvert, 2018), since it allows to generate 3D mazes with any layout.
For m × m maps, all egocentric views for a map is represented by m × m ×
RGB images.
For workspace manipulation , we randomly generate 0 to 5 obstacles in workspace as before.
We use a mapper network to convert the 96 × 96 workspace (image of obstacles) to the m × m 2 degree-of-freedom (DOF) configuration space (2D occupancy grid).
In both environments, the setup is similar to Section 6.1, while we only use m = 15 maps but longer 100 epochs for visual navigation and m = 18 maps still with 30 epochs for workspace manipulation.
Methods: mapper networks and setup.
For visual navigation , we implemented equivariant mapper network based on (Lee et al., 2018).
The mapper network converts every image into a 256dimensional embedding m × m × 4 × 256 and then predicts map layout m × m × 1 .
For workspace manipulation , we use U-net (Ronneberger et al., 2015) with residual-connection (He et al., 2015) as a mapper.
For more training details, see Section I.
Results.
The results are also shown in Table 3, denoted as Visual (navigation, 15 × 15 ) and
Workspace (manipulation, 18 × 18 ).
In visual navigation, the trends are similar to 2D case: two symmetric planners both train much faster.
Besides vanilla VIN, all approaches finally converge to near-optimal successful rate (around 95% ), while the validation and test results show large gaps.
SymGPPN has almost no generalization gap, while VIN does not generalize well to new 3D visual navigation environments.
Our SymVIN improves test successful rate from less than 50% to 90% and is comparable with GPPN.
Since the input is raw images and a mapper is used to learn end-toend, it potentially causes one major source of generalization gap for some approaches.
In workspace manipulation, the results are also analogous to C-space, while ours advantages over baselines are smaller.
In our inspection, we found the mapper network is the bottleneck, since the mapping for obstacles from workspace to C-space is nontrivial to learn.
6. R ESULTS ON GENERALIZATION TO LARGER MAPS
To demonstrate the generalization advantage of ours methods, all methods are trained in small map and tested in larger maps.
All methods are trained on 15 × 15 with K = 30 .
Then we test all methods on map size 15 × 15 through 99 × 99 , averaging over 3 seeds (3 model checkpoints) for each method and 1000 maps for each map size.
Iterations K is set to √ 2 · M , where M is the testing map size (x-axis).
The results are shown in Figure 7.
[Figure]
SymVIN generalizes better than VIN, although the variance is greater.
GPPN diverges for larger variable K since it is even worse than fixed K = 30 in all map sizes.
ConvGPPN converges, while it fluctuates for different seeds.
SymGPPN shows the best generalization and has small variance.
In conclusion, SymVIN and SymGPPN generalize better to different map sizes, compare to all non-equivariant baselines.
Remark.
The SymPlan models demonstrate end-toend planning and learning ability, potentially enabling further applications to other tasks as a differentiable component for planning.
The additional results and ablation studies are provided in Appendix I.
D ISCUSSION
In this work, we study the symmetry in 2D path planning problem, and build a framework using the theory of steerable CNNs to prove that value iteration in path planning is actually a form of steerable
CNN (on 2D grids).
Although we focus on Z 2 , we can generalize to path planning on higherdimensional or even continuous Euclidean spaces (Weiler et al., 2018; Brandstetter et al., 2021), and use equivariant operations on steerable feature fields (such as steerable convolutions, pooling, and point-wise non-linearities) from steerable CNNs.
We practically show that the SymPlan algorithms exactly motivated by the theory provide great improvement.
We hope the framework along with the design of practical algorithms can provide a new pathway to exploiting symmetry structure in differentiable planning.
R EPRODUCIBILITY S TATEMENT
We provide additional details in the appendix.
We also plan to open source the codebase.
We briefly outline the appendix below.
Additional Discussion 2.
Background: Technical background and concepts on steerable CNNs and group CNNs 3.
Method: we provide full details on how to reproduce it 4.
Theory/Framework: we provide the complete version of the theory statements 5.
Proofs: this includes all proofs 6.
Experiment / Environment / Implementation details: useful details for reproducibility 7.
Additional results
C ONTENTS A Outline
B Additional Results
B.1 SPL metric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C Additional Discussion
C.1 Limitations and Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.2 The Considered Symmetry and Difference to Existing Work . . . . . . . . . . . .
D Background: Equivariant Networks
D.1 Basics: Groups and Group Representations . . . . . . . . . . . . . . . . . . . . .
D.2 Group Representations: Visual Understanding . . . . . . . . . . . . . . . . . . . .
D.3 Geometric Deep Learning . .
D.4 Steerable CNNs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E Symmetric Planning in Practice
E.1 Building Symmetric VIN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.2 PyTorch-style pseudocode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F Symmetric Planning Framework
F. Steerable Planning: Planning On Steerable Feature Fields . . . . . . . . . . . . . .
F.
Symmetric Planning: Integrating Symmetry by Convolution . . . . . . . . . . . .
F.
Details: Constructing Path planning in Neural Networks . . . . . . . . . . . . . .
F.
Details: Understanding Symmetric Planning by Abstraction . . . . . . . . . . . . .
F.
Note: Augmented State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
G Symmetric Planning Framework: Proofs
G.1 Proof: equivariance of scalar-valued expected value operation . . . . . . . . . . . .
G.2 Proof: expected value operator as steerable convolution . . . . . . . . . . . . . . . G.3 Proof: equivariance of expected future value . . . . . . . . . . . . . . . . . . . . .
G.4 Proof: equivariance of steerable value iteration . . . . . . . . . . . . . . . . . . .
H Implementation Details
H.1 Implementation of SymGPPN . . . . . . . . . . . . . . . . . . . . . . . . . . . .
H.2 Implementation of max operation . . . . . . . . . . . . . . . . . . . . . . . . . .
I Experiment Details and Ablation Study
I. Environment Setup . . I.
Building Mapper Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I.
Training Setup . . I.
Visualization of Learned Models . . . . . . . . . . . . . . . . . . . . . . . . . . .
I. Ablation Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A O UTLINE
We provide a table of content above.
We omit technical details on symmetry and equivariant networks in the main paper and delay them here.
Specifically, for readers interested in additional details on how to use equivariant networks for symmetric planning, we recommend an order as follows: (1) Basics on group representations and equivariant networks in Section D.1.
(2) Practice on building SymVIN in Section E.1.
Detailed formulation on SymPlan in Section F.1 and Section F.2.
The rest technical sections provide additional reading materials for the readers interested in more in-depth account on studying symmetry in reinforcement learning and planning.
B A DDITIONAL R ESULTS B. SPL METRIC
We add SPL metric as requested by a reviewer.
We rerun all the 2D navigation experiments, so the exact numbers are not the same as the original experiments, but the trends should be similar.
[Table]
[Table]
[Table]
[Table]
[Table]
The performance under SPL metric is pretty similar to success rate.
This indicates that most paths are pretty close to the optimal length from all planners.
Thus, we decided to not include this to avoid extra concept in experiment section.
C A DDITIONAL D ISCUSSION C. L IMITATIONS AND E XTENSIONS
Assumption on known domain structure.
As in VIN, although the framework of steerable planning can potentially handle different domains, one important hidden assumption is that the underlying domain Ω (state space), is known.
In other words, we fix the structure of learned transition kernels p ( s ′ | s, a ) and estimate coefficients of it.
One potential method is to use Transformers that learn attention weights to all states in S , which has been partially explored in SPT (Chaplot et al., 2021).
Additionally, it is also possible to treat unknown MDPs as learned transition graphs, as explored in XLVIN (Deac et al., 2021).
We leave the consideration of symmetry in unknown underlying domains for future work.
The curse of dimensionality.
The paradigm of steerable planning still requires full expansion in computing value iteration (opposite to sampling-based ), since we realize the symmetric planner using group equivariant convolutions (essentially summation or integral).
Convolutions on highdimensional space could suffer from the curse of dimensionality for higher dimensional domains, and are vastly under-explored.
This is a primary reason why we need sampling-based planning algorithms.
If the domain (state-action transition graph) is sparsely connected, value iteration can still scale up to higher dimensions.
It is also unclear either when steerable planning would fail, or how sampling-based algorithms could be integrated with the symmetric planning paradigm.
C. T HE C ONSIDERED S YMMETRY AND D IFFERENCE TO E XISTING W ORK
We need to differentiate between two types of symmetry in MDPs.
Let’s take spatial graph as illustrative example to understand the potential symmetry from a higher level, which means that the nodes V in the graph have spatial coordinates Z n or R n .
Our 2D path planning is a special case of spatial graph, where the actions can only move to adjacent spatial nodes.
Let the graph denoted as G = ⟨V , E⟩ .
E is the set of edges connecting two states with an action.
One type of symmetry is the symmetry of the graph itself.
For the grid case, it means that after Drotation or reflection, the map is unchanged.
Another type of symmetry comes from the isometries of the space.
For a spatial graph, we can rotate it freely in a space, while the relative positions are unchanged.
For our grid case, it is shown in the
Figure 1 that rotating a map resulting in the rotated policy.
However, the map or policy itself can never be equal under any transformation in D 4 .
In other words, the first type is symmetry within a MDP (rely on the property of the MDP itself
M , or Aut( M ) ), and the second type is symmetry between MDPs (only rely on the property of the underlying spatial space Z 2 , or Aut( Z 2 ) ).
Nevertheless, we could input map M and somehow treat symmetric states between MDPs as one state.
See the proofs section for more details.
D B ACKGROUND : E QUIVARIANT N ETWORKS
We omit technical details in the main paper and delay them here.
This section introduces the background on equivariant networks and representation theory.
The first subsection covers necessary basics, while the rest subsections provide additional reading materials for the readers interested in more in-depth account on the preliminaries on studying symmetry in reinforcement learning and planning.
D. B ASICS : G ROUPS AND G ROUP R EPRESENTATIONS
Symmetry groups and equivarance.
A symmetry group is defined as a set G together with a binary composition map satisfying the axioms of associativity, identity, and inverse.
A (left) group action of G on a set X is defined as the mapping ( g, x ) (cid:55)→ g.x which is compatible with composition.
Given a function f : X → Y and G acting on X and Y , then f is G -equivariant if it commutes with group actions: g.f ( x ) =
f ( g.x ) , ∀ g ∈ G, ∀ x ∈ X .
In the special case the action on Y is trivial g.y = y , then f ( x ) =
f ( g.x ) holds, and we say f is G -invariant .
Group representations.
We mainly use two groups: dihedral group D 4 and cyclic group C 4 .
The cyclic group of 4 elements is C 4 = ⟨ r | r 4 = 1 ⟩ , a symmetry group of rotating a square.
The dihedral group D 4 = ⟨ r, s | r 4 =
s 2 = ( sr ) 2 = 1 ⟩ includes both rotations r and reflections s , and has size | D 4 | = 8 .
A group representation defines how a group action transforms a vector space
G × S → S .
These groups have three types of representations of our interest: trivial , regular , and quotient representations, see (Weiler and Cesa, 2021).
The trivial representation ρ triv maps each g ∈ G to 1 and hence fixes all s ∈ S .
The regular representation ρ reg of C 4 group sends each g ∈ C 4 to a 4 × 4 permutation matrix that cyclically permutes a 4 -element vector, such as a one-hot 4direction action.
The regular representation of D 4 maps each element to an 8 × 8 permutation matrix which does not act on 4-direction actions, which requires the quotient representations (quotienting out sr 2 reflection part) and forming a 4 × 4 permutation matrix.
It is worth mentioning the standard [Figure] representation of the cyclic groups, which are 2 × 2 rotation matrices, only used for visualization (Figure 8 middle).
Steerable feature fields and Steerable CNNs.
The concept of feature fields is used in (equivariant)
CNNs (Bronstein et al., 2021a; Cohen et al., 2020; Kondor and Trivedi, 2018; Cohen and Welling, 2016a;b; Weiler and Cesa, 2021).
The pixels of an 2D RGB image x : Z 2 → R 3 on a domain Ω =
Z 2 is a feature field.
In steerable CNNs for 2D grid, features are formed as steerable feature fields f : Z 2 → R C that associate a C -dimensional feature vector f ( x ) ∈ R C to each element on a base space, such as Z 2 .
Defined like this, we know how to transform a steerable feature field and also the feature field after applying CNN on it, using some group (Cohen and Welling, 2016a).
The type of CNNs that operates on steerable feature fields is called Steerable CNN (Cohen and Welling, 2016a), which is equivariant to groups including translations as subgroup ( Z 2 , +) , extending (Cohen and Welling, 2016b).
It needs to satisfy a kernel steerability constraint, where the R 2 and Z 2 cases are considered in (Weiler and Cesa, 2021).
We consider the 2D grid as our domain Ω = S =
Zand use G = p 4 m group as the running example.
The group p 4 m =
( Z 2 , +) ⋊ D 4 (wallpaper group) is semi-direct product of discrete translation group Z 2 and dihedral group D 4 , see (Cohen and Welling, 2016b;a).
We visualize the transformation law of p 4 m on a feature field on Ω =
Z 2 in
Figure 8 (Middle) , usually referred as induced representation (Cohen and Welling, 2016a; Weiler and Cesa, 2021).
D. G ROUP R EPRESENTATIONS : V ISUAL U NDERSTANDING
A group representation is a (linear) group action that defines how a group acts on some space.
Cohen and Welling (2016b;a); Weiler and Cesa (2021) provide more formal introduction to them in the context of equivariant neural networks.
We provide visual understanding and refer the readers to them for comprehensive account.
To visually understand how the group D 4 acts on some vector space, we visualize the trivial, regular, and quotient (quotienting out reflections sr 2 ) representations, which are permutation matrices .
If we apply such a representation ρ ( g )( g ∈ D 4 ) to a vector, the elements get cyclically permuted .
See
Figure 9.
The quotient representation that quotients out reflections and has dimension 4 × 4 is what we need to use on the 4 -direction action space.
D. G EOMETRIC D EEP L EARNING
We review another set of important concepts that motivate our formulation of steerable planning: geometric deep learning and the theories on connecting equivariance and convolution (Bronstein et al., 2021a; Cohen et al., 2020; Kondor and Trivedi, 2018).
Bronstein et al.
(2021a)
use x for feature fields while Cohen and Welling (2016a)
; Cohen et al.
; Weiler and Cesa (2021) use f .
Convolutional feature fields.
The signals are taken from set C =
R D on some structured domain Ω , and all mappings from the domain to signals forms the space of C -valued signals X (Ω , C ) =
{ f : Ω → C} , or X (Ω) for abbreviation.
For instance, for RGB images, the domain is the 2D n × n grid Ω =
Z n × Z n , and every pixel can take RGB values
C
= R 3 at each point in the domain u ∈ Ω , represented by a mapping x : Z n × Z n → R 3 .
A function on images thus operates on 3 n 2 -dimensional inputs.
It is argued that the underlying geometric structure of domains Ω plays key role in alleviating the curse of dimensionality, such as convolution networks in computer vision, and this framework is named Geometric Deep Learning .
We refer the readers to Geometric Deep Learning (Bronstein et al., 2021a) for more details, and to more rigorous theories on the relation between equivariant maps and convolutions in (Cohen et al., 2020)
(vector fields through induced representations) and (Kondor and Trivedi, 2018) (scalar fields through trivial representations).
Group convolution.
Convolutions are shift-equivariant operations, and vice versa.
This is the special case for Ω = R , which can be generalized to any group G (that we can integrate or sum over).
The group convolution for signals on Ω is then defined 2 as
[Equation]
where ψ ( u ) is shifted copies of a filter, usually locally supported on a subset of Ω and padded outside.
Note that although x takes u ∈ Ω , the feature map ( x ⋆ ψ ) takes as input the elements g ∈ G instead of points on the domain u ∈ Ω .
All following group convolution layers take G :
X ( G ) →
X ( G ) .
In the grid case, the domain Ω is homogeneous space of the group G , i.e. the group
G acts transitively: for any two points u, v ∈ Ω there exists a symmetry g ∈ G to reach u = gv .
Analogous to classic shift-equivariant convolutions, the generalized group convolution is Gequivariant (Cohen et al., 2020).
It is observed that ⟨ x, ρ ( g ) θ ⟩ = ⟨ ρ ( g − 1 ) x, θ ⟩ , and from the defining property of group representations ρ ( h − 1 ) ρ ( g )
=
ρ ( h − 1 g ) , the G -equivariance of group convolution follows (Bronstein et al., 2021a): [Equation]
[Equation]
Steerable convolution kernels.
Steerable convolutions extend group convolutions to more general setup and decouple the computation cost with the group size (Cohen and Welling, 2016a; Cohen, 2021).
For example, E (2) -steerable CNNs (Weiler and Cesa, 2021) apply it for E (2) group, which is semi-direct product of translations R 2 and a fiber group H , where H is a group of transformations that fixes the origin and is O (2) or its subgroups.
The representation on the signals/fields is induced from a representation of the fiber group H .
Use R 2 as example, a steerable kernel only needs to be
H -equivariant by satisfying the following constraint (Weiler and Cesa, 2021):
[Equation]
D. S TEERABLE CNN S
We still use the running example on Z 2 and group p 4 m =
Z 2 ⋊ D 4 .
Induced representations.
We follow (Cohen and Welling, 2016a; Cohen et al., 2020) to use π for induced representations.
We still use feature fields over Z 2 as example.
As shown in Figure 8 middle , to transform a feature field f : Z 2 → R C on base Z 2 with group p 4 m =
Z 2 ⋊ D 4 , we need the induced representation (Cohen and Welling, 2016a;
Cohen et al., 2020).
The induced representation in this case is denoted as π ( g ) ≜ ind Z 2 ⋊ D 4 Dρ ( g ) (for all g ), which means how the group action of D 4 transforms a feature field on Z 2 ⋊ D 4 .
It acts on the feature field with two parts: (1) on the base space Z 2 and (2) on the fibers (feature channels R C ) by fiber group H =
D 4 (Cohen and Welling, 2016a; Weiler and Cesa, 2021).
More specifically, applying a translation t ∈
Z 2 and a transformation r ∈ D 4 to some field f , we get π ( tr ) f (Cohen and Welling, 2016a;
Weiler and Cesa, 2021):
[Equation]
ρ ( r ) is the fiber representation that transforms the fibers R C , and ( tr ) − 1 x finds the element before group action (or equivalently transforming the base space Z 2 ).
Thus, π only depends on the fiber representation ρ but not the latter part, thus named induced representation by ρ .
Steerable convolution vs. group convolution.
The steerable convolution on Z 2 The understanding of this point helps to understand how a group acts on various feature fields and the design of state space for path planning problems.
We use the discrete group p 4 =
Z 2 ⋊ C 4 as example, which consists of Z 2 translations and 90 ◦ rotations.
The only difference with p 4 m is p 4 does not have reflections.
The group convolution with filter ψ and signal x on grid (or p ∈
Z 2 ), which outputs signals (a function) on group p
[Equation]
A group G has a natural action on the functions over its elements; if x : G → R and g ∈ G , the function g.x is defined as [ g.x ]( h ) :
= x ( g − 1 · h ) .
For example: The group action of a rotation r ∈ C 4 on the space of functions over p 4 is
[Equation]
11) where r − 1 p spatially rotates the pixels, r − 1 s cyclically permutes the 4 channels.
The G-space (functions over p 4) with a natural action of p 4 on it:
[Equation]
The group convolution in discrete case is defined as
[Equation]
The group convolution with filter ψ and signal x on p 4 group is given by:
[Equation]
Using the fact
[Equation]
) the convolution can be equivalently written into
[Equation]
So (cid:16)(cid:80)
p ∈
Z 2 [ r.ψ ]( p − t , s ) x ( p , s ) (cid:17) can be implemented in usual shift-equivariant convolution
CONV 2 D .
The inner sum (cid:80) p
∈
Z 2 is equivalently for the sum in steerable convolution, and the outer sum (cid:80) s ∈ Cimplement rotation-equivariant convolution that satisfies H -steerability kernel constraint.
Here, the outer sum is essentially using the regular fiber representation of C 4 .
In other words, group convolution on p 4 =
Z 2 ⋊ C 4 group is equivalent to steerable convolution on base space Z 2 with the fiber group of C 4 with regular representation.
Stack of feature fields.
Analogous to ordinary CNNs, a feature space in steerable CNNs can consist of multiple feature fields f
i : Z 2 → R c i .
The feature fields are stacked f = (cid:76) i f i together by concatenating the individual feature fields f i (along the fiber channel), which transforms under the directly sum ρ = (cid:76) i ρ i of individual (fiber) representations.
Every layer will be equivariant between input and output field f in , f out under induced representations π in , π out .
For a steerable convolution between more than one-dimensional feature fields, the kernel is matrix-valued (Cohen et al., 2020; Weiler and Cesa, 2021).
E S YMMETRIC P LANNING IN P RACTICE E. B UILDING S YMMETRIC VIN
In this section, we discuss how to achieve Symmetric Planning on 2D grids with E(2) -steerable
CNNs (Weiler and Cesa, 2021).
We focus on implementing symmetric version of value iteration,
SymVIN, and generalize the methodology to make a symmetric version of a popular follow-up of
VIN, GPPN (Lee et al., 2018).
Steerable value iteration.
We have showed that, value iteration for path planning problems on Zconsists of equivariant maps between steerable feature fields.
It can be implemented as an equivariant steerable CNN, with recursively applying two alternating (equivariant) layers:
[Equation]
where k ∈ [ K ] indexes iteration, V k , Q ak , R am are steerable feature fields over Z 2 output by equivariant layers, P aθ is a learned kernel in neural network, and + , × are element-wise operations.
Implementation of pipeline.
We follow the pipeline in VIN (Tamar et al., 2016a).
The commutative diagram for the full pipeline is shown in Figure 10.
The path planning task is given by a m × m spatial binary obstacle occupancy map and one-hot goal map, represented as a feature field M : Z 2 → { 0 , 1 } 2 .
For the iterative process Q ak (cid:55)→ V k
(cid:55)→ Q ak +1 , the reward field R M is predicted from map M (by a 1 × 1 convolution layer) and the value field V 0 is initialized as zeros.
The network output is (logits of) planned actions for all locations 3 , represented as A : Z 2 → R |A| , predicted from the final Q-value field Q K (by another 1 × 1 convolution layer).
The number of iterations K and the convolutional kernel size F of P aθ are set based on map size M , and the spatial dimension m × m is kept consistent.
Building Symmetric Value Iteration Networks.
Given the pipeline of VIN fully on steerable feature fields, we are ready to build equivariant version with E(2) -steerable CNNs (Weiler and Cesa, 2021).
The idea is to replace every Conv2d with a steerable convolution layer between steerable feature fields, and associate the fields with proper fiber representations ρ ( h ) .
VINs use ordinary CNNs and can choose the size of intermediate feature maps.
The design choices in steerable CNNs is the feature fields and fiber representations (or type ) for every layer (Cohen and
Welling, 2016a; Weiler and Cesa, 2021).
The main difference 4 in steerable CNNs is that we also need to tell the network how to transform every feature field , by specifying fiber representations , as shown in Figure 10.
Specification of input map and output action.
We first specify fiber representations for the input and output field of the network: map M and action A .
For input occupancy map and goal M : Z→ { 0 , 1 } 2 , it does not D 4 to act on the 2 channels, so we use two copies of trivial representations ρ M = ρ triv ⊕ ρ triv .
For action , the final action output A : Z 2 → R |A| is for logits of four actions
A = (north, west, south, east) for every location.
If we use H =
C 4 , it naturally acts on the four actions (ordered ⟲ ) by cyclically ⟲ permuting the R 4 channels.
However, since the D 4 group has 8 elements, we need a quotient representation , see (Weiler and Cesa, 2021) and
Appendix H.
Specification of intermediate fields: value and reward.
Then, for the intermediate feature fields:
Q-values Q k , state value V k , and reward R m
, we are free to choose fiber representations, as well as the width (number of copies).
For example, if we want 2 copies of regular representation of D 4 , the feature field has 2 × 8 = 16 channels and the stacked representation is 16 × 16 (by direct-sum).
For the Q -value field Q ak ( s ) , we use representation ρ Q and its size as C Q .
We need at least C A ≥ |A| channels for all actions of Q ( s, a ) as in VIN and GPPN, then stacked together and denoted as
Q k ≜ (cid:76) a Q ak with dimension Q k : Z 2 → R C Q ∗ C A .
Therefore, the representation is direct-sum (cid:76) ρ Q for C A copies.
The reward is implemented similarly as R M ≜ (cid:76) a R aM and must have same dimension and representation to add element-wisely.
For state value field, we denote the choose as fiber representation as ρ V and its size C V .
It has size V k :
Z 2 → R C V
Thus, the steerable kernel is matrix-valued with dimension P θ : Z 2 → R ( C Q ∗ C A ) × C V .
In practice, we found using regular representations for all three works the best.
It can be viewed as "augmented" state and is related to group convolution, detailed in Appendix H.
Other operations.
We now visit the remained (equivariant) operations.
(1) The max operation in
Q k (cid:55)→ V k +1 .
While we have showed the max operation in V k +1 ( s ) = max a Q ak ( s ) is equivariant in Theorem F.3, we need to apply max(-pooling) for all actions along the "representation channel" from stacked representations C A ∗ C Q to one C Q .
More details are in Appendix H.2.
The final output layer Q K (cid:55)→ A .
the iteration
Q value Q k with convolution to convert the Z R |
| . Extended method: Symmetric GPPN.
Gated path planning network (GPPN (Lee et al., 2018)) proposes to use LSTM to alleviate the issue of unstable gradient in VINs.
Although it does not strictly follow value iteration, it still follows the spirit of steerable planning.
Thus, we first obtained a fully convolutional variant of GPPN from [Redacted for anonymous review], called ConvGPPN.
It replaces the MLPs in the original LSTM cell with convolutional layers, and then replaces convolutions with equivariant steerable convolutions, resulting in a fully equivariant SymGPPN.
See
Appendix H.1 for details.
Extended tasks: planning on learned maps with mapper networks.
We consider two planning tasks on 2D grids: 2D navigation and 2-DOF manipulation.
To demonstrate the ability of handling symmetry in differentiable planning, we consider more complicated state space input: visual navigation and workspace manipulation, and discuss how to use mapper networks to convert the state input and use end-to-end learned maps, as in (Lee et al., 2018; Chaplot et al., 2021).
See Appendix I.2 for details.
E. P Y T ORCH - STYLE PSEUDOCODE
Here, we write a section on explaining the SymVIN method with PyTorch-style pseudocode, since it directly corresponds to what we propose in the method section.
We try to relate (1) existing concepts with VIN, (2) what we propose in Section 4 and 5 for SymVIN, and (3) actual PyTorch implementation of VIN and SymVIN aligned line-by-line based on semantic correspondence.
We provide the key Python code snippets to demonstrate how easy it is to implement SymVIN, our symmetric version of VIN (Tamar et al., 2016a).
In the current Section 5 (SymPlan practice), we heavily use the concepts from Steerable CNNs.
Thanks to the equivariant network community and the e2cnn package, the actual implementation is compact and closely corresponds to their non-equivariant counterpart, VIN, line-by-line.
Thus, the ultimate goal here is to illustrate that, whatever concepts we have in regular CNNs (e.g., have whatever channels we want), we can can use steerable CNNs that incorporate desired extra symmetry (of D 4 rotation+reflection or C 4 rotation).
We highlight the implementation of the value iteration procedure in VIN and SymVIN:
[Equation]
Note that we use actual code snippets to avoid hiding any details.
Defining (steerable) convolution layer.
First, we show the definition of the key convolution layer for a key operation in VIN and SymVIN: expected value operator, in Listing 1 and 2.
As proved in Theorem F.2, the expected value operator can be executed by a steerable convolution layer for (2D) path planning.
This serves as the theoretical foundation on how we should use a steerable layer here.
For the left side, a regular 2D convolution is defined for VIN.
The right side defines a steerable convolution layer, using the library e2cnn from (Weiler and Cesa, 2021).
It provides high-level abstraction for building equivariant 2D steerable convolution networks.
As a user, we only need to specify how the feature fields transform (as shown in Figure 10), and it will solve the G -steerability constraints, process what needs to be trained for equivariant layers, etc.
We use name q_r2conv to highlight the difference.
Value iteration procedure.
Second, we compare the for loop for value iteration updates in VIN and SymVIN, where the former one has regular 2D convolution Conv2D (Listing 3), and the latter one uses steerable convolution (Weiler and Cesa, 2021)
(Listing 4).
The lines are aligned based on semantic correspondence.
The e2cnn layers, including steerable convolution layers, operate on its GeometricTensor data structure, which is to wrap a PyTorch tensor.
We denote them with _geo suffix.
It only additionally needs to specify how this tensor (feature field) transforms under a group (e [Figure] Listing 1: Define ‘expected value‘ convolution layer for VIN. [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Listing 2: Define ‘expected value‘ (steerable) convolution layer for SymVIN.
[Figure]
[Figure]
[Table]
Listing 3: The central value iteration procedure for VIN.
Some variable names are adjusted accordingly for readability.
W and H are width and height for 2D map.
[Table]
[Table]
Listing 4:
The equivariant steerable value iteration procedure for SymVIN.
Lines are aligned by semantic correspondence.
Definition of other field types are similar and thus omitted.
tensor_directsum is used to concatenate two GeometricTensor ’s (feature fields) and compute their associated representations (by direct-sum).
Thus, the e2cnn steerable convolution layer on the right side q_r2conv can be used as a regular
PyTorch layer, while the input and output are GeometricTensor .
We also define the max operation as a customized max-pooling layer, named q_max_pool .
The implementation is similar to the left side of VIN and needs to additionally guarantee equivariance, and the detail is omitted.
Note that for readability, we assume we use regular representations for the Q-value field Q and the state-value field V .
They are empirically found to work the best.
This corresponds to the definition in field_type_q_in in line 9 in the SymVIN definition listing and the comments in line 16in the steerable VI procedure listing for SymVIN.
Other components are omitted.
F S YMMETRIC P LANNING F RAMEWORK
This section formulates the notion of Symmetric Planning (SymPlan).
We expand the understanding of path planning in neural networks by planning as convolution on steerable feature fields ( steerable planning ).
We use that to build steerable value iteration and show it is equivariant.
F. S TEERABLE P LANNING : P LANNING O N S TEERABLE F EATURE F IELDS
We start the discussion based on Value Iteration Networks (VINs, (Tamar et al., 2016a)) and use a running example of planning on the 2D grid Z 2 .
We aim to understand (1) how VIN-style networks embed planning and how its idea generalizes, (2) how is symmetry structure defined in path planning and how could it be injected into such planning networks.
Constructing G -invariant transition: spatial MDP.
Intuitively, the embedded MDP in a VIN is different from the original path planning problem, since (planar) convolutions are translation equivariant but there are different obstacles in different regions.
We found the key insight in VINs is that it implicitly uses an MDP that has translation equivariance.
The core idea behind the construction is that it converts obstacles (encoded in transition probability
P , by blocking ) into “ traps ” (encoded in reward ¯ R , by −∞ reward ).
This allows to use planar convolutions with translation equivariance, and also enables use to further use steerable convolutions.
The demonstration of the idea is shown in Figure 8 (Left) .
We call it spatial MDP , with different transition and reward function ¯ M = ⟨S , A , ¯ P, ¯ R m , γ ⟩ , which converts the “complexity” in the transition function P in M to the reward function ¯ R m in ¯ M .
The state and action space are kept the same: state S =
Z 2 and action A ⊂ Z 2 to move ∆ s in four directions in a 2D grid.
We provide the detailed construction of the spatial MDP in Section F.3.
Steerable features fields.
We generalize the idea from VIN, by viewing functions (in RL and planning) as steerable feature fields , motivated by (Bronstein et al., 2021a; Cohen et al., 2020;
Cohen and Welling, 2016a).
This is analogous to pixels on images Ω
→
[255] 3 , and would allow us to apply convolution on it.
The state value function is expressed as a field V : S → R , while the Q -value function needs a field with |A| channels: Q : S → R |A| .
Similarly, a policy fieldhas probability logits of selecting |A| actions.
For the transition probability P ( s ′ | s, a ) , we can use action to index it as P a ( s ′ | s ) , similarly for reward R a ( s ) .
The next section will show that we can convert the transition function to field and even convolutional filter.
F. S YMMETRIC P LANNING : I NTEGRATING S YMMETRY BY C ONVOLUTION
The seemingly slight change in the construction of spatial MDPs brings important symmetry structure.
The general idea in exploiting symmetry in path planning is to use equivariance to avoid explicitly constructing equivalence classes of symmetric states.
To this end, we construct value iteration over steerable feature fields, and show it is equivariant for path planning.
In VIN, the convolution is over 2D grid Z 2 , which is symmetric under D 4 (rotations and reflections).
However, we also know that VIN is already equivariant under translations.
To consider all symmetries, as in (Cohen and Welling, 2016a; Weiler and Cesa, 2021), we understand the group p 4 m =
G = B
⋊ H as constructed by a base space B = G/H =
( Z 2 , +) and a fiber group
H =
D 4 , which is a stabilizer subgroup that fixes the origin
0 ∈
Z 2 .
We could then formally study such symmetry in the spatial MDP, since we construct it to ensure that the transition probability function in ¯ M is G -invariant.
Specifically, we can uniquely decompose any g ∈ Z 2 ⋊ D 4 as t ∈ Zand r ∈ D 4 (and translations act "trivially" on action), so ¯ P ( s ′ | s
[Equation]
Expected value operator as steerable convolution.
The equivariance property can be shown stepby-step: (1) expected value operation , (2) Bellman operator , and (3) full value iteration .
First, we use G -invariance to prove that the expected value operator (cid:80) s ′ P ( s ′ | s, a ) V ( s ′ ) is equivariant.
Theorem F.1.
If transition is G -invariant, the expected value operator E over Z 2 is G -equivariant.
The proof is in Section G.1 and visual understanding is in Figure 8 middle.
However, this provides intuition but is inadequate since we do not know: (1) how to implement it with CNNs, (2) how to use multiple feature channels like VINs, since it shows for scalar-valued transition probability and value function (corresponding to trivial representation).
To this end, we next prove that we can implement value iteration using steerable convolution with general steerable kernels.
Theorem F.2.
If transition is G -invariant, there exists a (one-argument, isotropic) matrix-valued steerable kernel P a ( s − s ′ )
(for every action), such that the expected value operator can be written as a steerable convolution and is G -equivariant:
[Equation]
The full derivation is provided in Section G.
We write the transition probability as P a ( s, s ′ ) , and we show it only depends on state difference P a ( s − s ′ )
(or one-argument kernel (Cohen et al., 2020)) using G -invariance, which is the key step to show it is some convolution .
Note that we use one kernel P a for each action (four directions), and when the group acts on E , it also acts on the action
P g.a (and state, so technically acting on S × A ).
Additionally, if the steerable kernel also satisfies the D 4 -steerability constraint (Weiler and Cesa, 2021; Weiler et al., 2018), the steerable convolution is equivariant under p 4 m =
Z 2 ⋊ D 4 .
We can then extend VINs from Z 2 translation equivariance to p 4 m -equivariance (translations, rotations, reflections).
The derivation follows the existing work on steerable CNNs (Cohen and Welling, 2016b;a; Weiler and Cesa, 2021;
Cohen et al., 2020), while this is our goal: to justify the close connection between path planning and steerable convolutions.
Steerable Bellman operator and value iteration.
We can now represent all operations in Bellman (optimality) operator on steerable feature fields over Z 2 (or steerable Bellman operator ) as follows:
[Equation]
21) where V, R a , ¯ P a are steerable feature fields over Z 2 .
As for the operations, max a is (max) pooling (over group channel), + , × are point-wise operations, and ⋆ is convolution.
As the second step, the main idea is to prove every operation in Bellman (optimality) operator on steerable fields is equivariant, including the nonlinear max a operator
and + , × .
Then, iteratively applying Bellman operator forms value iteration and is also equivariant, as shown below and proved in Appendix G.4.
Proposition F.3.
For a spatial MDP with G -invariant transition, the optimal value function can be found through G-steerable value iteration.
Remark.
Our framework generalizes the idea behind VINs and enables us to understand its applicability and restrictions.
More importantly, this allows us to integrate symmetry but avoid explicitly building equivalence classes and enables planning with symmetry in end-to-end fashion.
We emphasize that the symmetry in spatial MDPs is different from symmetric MDPs (Zinkevich and Balch, 2001; Ravindran and Barto, 2004; van der Pol et al., 2020a), since our reward function is not Ginvariant (if not conditioning on reward).
Although we focus on Z 2 , we can generalize to path planning on higher-dimensional or even continuous Euclidean spaces (like R 3 space (Weiler et al., 2018) or spatial graphs in R 3 (Brandstetter et al., 2021)), and use equivariant operations on steerable feature fields (such as steerable convolutions, pooling, and point-wise non-linearities) from steerable
CNNs.
We refer the readers to (Cohen and Welling, 2016b;a; Cohen, 2021; Weiler and Cesa, 2021) for more details.
F. D ETAILS : C ONSTRUCTING P ATH PLANNING IN N EURAL N ETWORKS
We provide the detailed construction of doing path planning in neural networks in the Section F.
This further explains the visualization in Figure 8 left.
We use the running example of planning on the 2D grid Z 2 .
We aim to understand (1) how VINstyle networks embed planning and how its idea generalizes, (2) how is symmetry structure defined in path planning and how could it be injected into such planning networks.
Recall that we aim to understand (1) how VIN-style networks embed planning and how its idea generalizes, (2) how is symmetry structure defined in path planning and how could it be injected into such planning networks.
Path planning as MDPs.
To answer the above two questions, we first need to understand how a VIN embeds a path planning problem into a convolutional network as some embedded MDP.
Intuitively, the embedded MDP in a VIN is different from the original path planning problem, since (planar) convolutions are translation equivariant but there are different obstacles in different regions.
For path planning on the 2D grid S =
Z 2 , the objective is to avoid some obstacle region C obs ⊂
Zand navigate to the goal region C goal through free space C\C obs .
An action a = ∆ s ∈
A is to move from the current state s to a next free state s ′ =
s + ∆ s , where for now we limit it to be in four directions:
A = .
Assuming deterministic transition, the agent moves to s ′ with probability 1 if s + ∆ s ∈ C\C obs .
If it hits an obstacle, it stays at s if s + ∆ s ∈ C obs : P ( s + ∆ s | s, ∆ s ) = 0 and
P ( s | s, ∆ s ) = 1 .
Every move has a constant negative reward R ( s, a )
= − 1 to encourage shortest path.
We call this ground path planning MDP, a 5 -tuple M = ⟨S , A , P, R, γ ⟩ .
Constructing embedded MDPs.
However, such transition function is not translation-invariant, i.e. at different position, the transition probabilities are not related by any symmetry: P ( s ′ | s, a ) ̸ =
P ( g.s ′ | g.s, g.a ) .
Instead, we could always construct a "symmetric" MDP that has equivalent optimal value and policy for path planning problems, which is implicitly realized in VINs.
The idea is to move the information of obstacles from transition function to reward function: when we hit some action s + ∆ s ∈ C obs , we instead allow transition ¯ P ( s + ∆ s | s, ∆ s )
=
1 (with all other s ′ asprobability) while set a "trap" with negative infinity reward ¯ R m ( s, ∆ s ) =
−∞ .
The reward function needs the information from the occupancy map M , indicating obstacles C obs and free space.
For the free region, the reward is still a constant ¯ R M ( s, ∆ s )
= − 1 , indicating the cost of movement.
We call it the embedded MDP, with different transition and reward function ¯ M = ⟨S , A , ¯ P, ¯ R M , γ ⟩ , which converts the “complexity” in the transition function P in M to the reward function ¯ R m in ¯
M .
Here, map M shall also be treated as an “input”, thus later we will derive how the group acts on the map g.M .
It has the same optimal policy and value as the ground MDP M , since the optimal policies in both MDPs will avoid obstacles in M or trap cells in ¯ M .
It could be easily verified by simulating value iteration backward in time from the goal position.
The transition probability ¯ P of the embedded MDP ¯ M is for an “empty” maze and thus translationinvariant.
Note that the reward function ¯ R is not not necessarily invariant.
This construction is not limited to 2D grid and generalizes to continuous state space or even higher dimensional space, such as R 6 configuration space for 6-DOF manipulation.
Note, all of this is what we use to conceptually understand how a VIN is possible to learn.
The reward cannot be negative infinity, but the network will learn it to be smaller than all desired Qvalues.
F. D ETAILS : U NDERSTANDING S YMMETRIC P LANNING BY A BSTRACTION
How do we deal with potential symmetry in path planning?
How do we characterize it?
We try to understand symmetric planning (steerable planning after integrating symmetry with equivariance) and how it is difference classic planning algorithms, such as A*, for planning under symmetry .
Steerable planning.
Recall that we generalize the idea of VIN by considering it as a planning network that composes of mappings between steerable feature fields.
The critical point is that, convolutions directly operate on local patches of pixels and never directly touch coordinates of pixels.
In analogy, this avoids a critical drawback in other explicit planning algorithms: in sampling-based planning, a trajectory ( s 1 , a 1 , s 2 , a 2 , . . . ) is sampled and inevitable represented by states Ω = S .
However, to find another symmetric state g.s , we potentially need to compare it against all known states S ′
⊂ S with all symmetries g ∈ G .
On high level, an implicit planner can avoid such symmetry breaking and is more easily compatible with symmetry by using equivariant constraints.
We can use MDP homomorphism to understand this (Ravindran and Barto, 2004; van der Pol et al., 2020b).
MDP homomorphisms.
An MDP homomorphism h :
M → M is a mapping from one MDP
M = ⟨S , A , P, R, γ ⟩ to another M = ⟨S , A , P, R, γ ⟩ (Ravindran and Barto, 2004; van der Pol et al., 2020b).
h consists of a tuple of surjective maps h = ⟨ ϕ, { α s | s ∈ S}⟩ , where ϕ : S → S is the state mapping and α s :
A → A is the state-dependent action mapping.
The mappings are constructed to satisfy the following conditions:
[Equation]
for all s, s ′ ∈ S and for all a ∈ A .
We call the reduced MDP M the homomorphic image of M under h .
If h = ⟨ ϕ, { α s | s ∈ S}⟩ has bijective maps ϕ
and { α s } , we call h an MDP isomorphism .
Given MDP homomorphism h , ( s, a ) and ( s ′ , a ′ ) are said to be h -equivariant if σ ( s ) =
σ ( s ′ )
and α s ( a ) =
α s ′ ( a ′ ) .
Symmetry-induced MDP homomorphisms.
Given group G , an MDP homomorphism h is said to be group structured if any state-action pair ( s, a ) and its transformed counterpart g. ( s, a ) are mapped to the same abstract state-action pair: ( ϕ ( s ) , α s ( a )) =
( ϕ ( g.s ) , α g.s ( g.a )) , for all s ∈
S , a ∈ A , g ∈ G .
For convenience, we denote g. ( s, a ) as ( g.s, g.a ) , where g.a implicitly 6 depends on state s .
Applied to the transition and reward functions, the transition function P is G -invariant if P satisfies P ( g.s ′ | g.s, g.a )
= P ( s ′ | s, a ) , and reward function R is G -invariant if R ( g.s, g.a ) =
R ( s, a ) , for all s ∈ S , a ∈ A , g ∈ G .
However, this only fits the type of symmetry in (van der Pol et al., 2020a; Wang et al., 2021).
And also, they cannot handle invariance to translation Z 2 .
In our case, we need to augment the reward function with map M input:
[Equation]
for all s ∈ S , a ∈ A , g ∈ G = p 4 m .
This means that, at least for rotations and reflections D 4 , the MDPs constructed from transformed maps { g.M } are MDP isomorphic to each other.
F. N OTE : A UGMENTED S TATE
We derive the relationship between group convolution and steerable convolution in Section D.4.
The augmented state Z 2 ⋊ D 4 → R can be similarly treated on the group p 4 m =
Z 2 ⋊ D 4 .
It is equivalent to using regular representation on the base space Z 2 as Z 2 → R 8 .
G S YMMETRIC P LANNING F RAMEWORK : P ROOFS
We show the derivation and proofs for all theoretical results in this section.
We follow the notation in (Cohen et al., 2020) to use ⋆ for (one-argument) convolution and · for (two-argument) multiplication:
[Equation]
G. P ROOF : EQUIVARIANCE OF SCALAR - VALUED EXPECTED VALUE OPERATION
We present the Theorem F.1 here and its formal definition.
Theorem G.1.
If transition is G -invariant, the expected value operator E over Z 2 is G -equivariant:
[Equation]
Proof.
E is the expected value operator.
We also write the transition probability as
Recall the G -invariance condition of transition probability, the group element g acts on s, a, s ′ :
[Equation]
where we can uniquely decompose any g ∈ Z 2 ⋊ D 4 as t ∈
Z 2 and r ∈ D 4 (Cohen and Welling, 2016a).
Note that, since the action is the difference between states a = ∆ s = s ′ − s , the translation part t acts trivially on it, so g.a = ( tr ) .a
= r.a for all r ∈ D 4 .
We transform the feature field and show its equivariance:
[Equation]
We use the trivial representation ρ triv ( g ) =
Id 1 × 1 = 1 to emphasize that (1) the group element g acts on feature fields P a
and V , and (2) both feature fields P a
and V are scalar-valued and correspond to the one-dimensional trivial representation of r ∈ D 4 .
In the third line, we use the G -invariance of transition probability.
The fourth line uses substitution ˜ s ′ ≜ ( tr ) .s
′ , for all s ′ ∈
Z 2 and tr ∈ Z 2 ⋊
D 4 .
This is an one-to-one mapping and the summation does does not change.
G. P ROOF : expected value operator AS STEERABLE CONVOLUTION
In this section, we derive how to cast expected value operator as steerable convolution.
The equivariance proof is in the next section.
In Theorem F.1, we show equivariance of value iteration in 2D path planning, while it is only for the case that feature fields P a
and V are scalar-valued and correspond to one-dimensional trivial representation of r ∈ D 4 .
Here, we provide the derivation for Theorem F.2 show that steerable CNNs (Cohen and Welling, 2016a) can achieve value iteration since we could construct the G-invariant transition probability as a steerable convolutional kernel.
This generalizes Theorem F.1 from scalar-valued kernel (for transition probability) with trivial representation to matrix-valued kernel with any combination of representations, enabling using stack (direct-sum) of feature fields and representations.
We state Theorem F.2 here for completeness:
Theorem G.2.
If transition is G -invariant, there exists a (one-argument, isotropic) matrix-valued steerable kernel P a ( s − s ′ )
(for every action), such that the expected value operator can be written as a steerable convolution and is G -equivariant:
[Equation]
Steerable kernels.
In our earlier definition, ψ a and f in are transition probability and value function, which are both real-valued ψ a : Z 2 → R , f in : Z 2 → R .
However, this is a special case which corresponds to use one-dimensional trivial representation of the fiber group D 4 .
In the general case in steerable CNNs (Cohen and Welling, 2016a; Weiler and Cesa, 2021), we can choose the feature fields ψ a : Z 2 → R C out × C in and f in : Z 2 → R C in and their fiber representations, which we will introduce the group representations of D 4 and how to choose in practice in the next section.
Weiler et al.
(2018) show that convolutions with steerable kernels ψ a : Z 2 → R C out × C in is the most general equivariant linear map between steerable feature space, transforming under ρ in and ρ out .
In analogy to the continuous version 7 in (Weiler and Cesa, 2021), the convolution is equivariant iff the kernel satisfies a H -steerability kernel constraint:
[Equation]
Expected value operation as steerable convolution.
The foremost step is to show that the expected value operation is a form of convolution and is also G -equivariant.
By definition, if we want to write a (linear) operator as a form of convolution, we need one-argument kernel.
Cohen et al.
(2020) show that every linear equivariant operator is some convolution and provide more details.
For our case, this is formally shown as follows.
Proposition G.3.
If the transition probability is G -invariant, it can be expressed as an (oneargument) kernel P a ( s ′ | s )
=
P a ( s ′ − s ) that only depends on the difference s ′ − s .
Proof.
The form of our proof is similar to (Cohen et al., 2020), while its direction is different from us.
We construct a MDP such that the transition probability kernel is G -invariant, while Cohen et al.
(2020) assume the linear operator ψ · f is linear equivariant operator on a homogeneous space, and then derive that the kernel is G -invariant and expressible as one-argument kernel.
Additionally, our kernel ψ a ( s, s ′ ) and ψ a ( s − s ′ ) both live on the base space B = Z 2 but not on the group G = Z 2 ⋊ D 4 .
We show that the transition probability only depends on the difference ∆ s
= s ′ − s , so we can define the two-argument kernel P a ( s ′ | s ) on S × S by an one-argument kernel P
a ( s ′ − s ) (for every action a ) on S =
Z 2 , without loss of generality:
[Equation]
where the second step uses G -invariance with g = sr , understood as the composition of a translation s ∈ Z 2 and a transformation in r ∈ D 4 .
Additionally, we can also derive that, for the one-argument kernel, if we rotate state difference r. ( s ′ − s ) , the probability is the same for rotated action r.a .
[Equation]
[Equation]
The expected value operator with two-argument kernel can be then written as
[Equation]
Note that we do not differentiate between cross-correlation ( s ′ − s ) and convolution (
G. P ROOF : EQUIVARIANCE OF expected future value
Our derivation follows the existing work on group convolution and steerable convolution networks (Cohen and Welling, 2016b;a; Weiler and Cesa, 2021; Cohen et al., 2020).
However, the goal of providing the proof is not just for completeness, but instead to emphasize the close connection between how we formulate our planning problem and the literature of steerable CNNs, which explains and justifies our formulation.
Additionally, there are several subtle differences worth to mention.
(1) Throughout the paper, we do not discuss kernels or fields that live on a group G to make it more approachable.
Nevertheless, group convolutions are a special case of steerable convolutions with fiber representation ρ as regular representation.
(2) We use Z 2 as running example.
Some prior work uses R 2 or Z 2 , but they are merely just differ in integral and summation.
The definition of convolution and cross-correlation might be defined and used interchangeably in the literature of (equivariant) CNNs.
Notation.
To keep notation clear and consistent with the literature (Cohen and Welling, 2016a;
Cohen et al., 2020; Weiler and Cesa, 2021), we denote the transition probability ¯ P ( s ′ | s, a ) ≜ ψ a ( s, s ′ ) ∈ R (one kernel for an action) and value function as V ( s ′ ) ≜
f in ( s ′ ) ∈ R , and the resulting expected value as f a out ( s ) =
(cid:80) s ′ ψ a ( s, s ′ )
f in ( s ′ )
(given a specific action a ).
Transformation laws: induced representation.
For some group acting on the base space Z 2 , the signals f : Z 2 → R c are transformed like Cohen and Welling (2016a):
[Equation]
Apply a translation t and a transformation r ∈ D 4 to f , we get π ( tr ) f .
The transformation law on the input space f in is (Cohen and Welling, 2016a; Weiler and Cesa, 2021):
[Equation]
The transformation law of the output space after applying π in on input f in is given by Cohen and
Welling (2016a):
[Equation]
In our case, the output space is f a out : Z 2 → R C out and the input space is f in : Z 2 → R C in .
Intuitively, if we rotate a vector field (fibers represent arrows) by the induced representation π ( tr ) of f , we also need to rotate the direction of arrows by ρ ( r ) , r ∈ D 4 .
Equivariance.
Now we prove the steerable convolution is equivariant:
[Equation]
The induced representation of input field f in is induced by the fiber representation ρ in , expressed by π in ≜ ind GH ρ in = ind Z 2 ⋊
D 4 Dρ in , where ρ in is the fiber representation of group H
= D 4 .
The induced representation of output field π out is analogously from ρ out .
Weiler and Cesa (2021) proved equivariance of steerable convolutions for R 2 case, while we include the proof under our setup for completeness.
The definition in (Weiler and Cesa, 2021) uses a form of cross-correlation and we use convolution , while it is usually referred to interchangeably in the literature and is equivalent.
Cohen and Welling (2016a); Weiler et al.
; Weiler and Cesa (2021); Cohen et al.
Cohen (2021) provide more details and we refer the readers to them for more comprehensive account.
The convolution on discrete grids Z 2 with input field f in transformed by the induced representation π in gives:
[Equation]
where s ′ ∈ S =
Z
[Equation]
and thus satisfies the equivariance condition:
[Equation]
Definition of ⋆ 2.
Transformation law of the induced representation π in (Cohen and Welling, 2016a; Weiler and Cesa, 2021) 3.
Kernel steerability ψ a ( s )
=
ρ out ( h ) ψ a ( h − 1 s ) ρ in ( h − 1 ) (Weiler and Cesa, 2021) 4.
Move and cancel 5.
Substitutes ˜ s
= r − 1 ( s ′ − t ) , r − 1 s ′ =
r − 1 t + ˜ s , so r − 1 ( s − s ′ ) =
r − 1 ( s − t ) − ˜ s .
Since r ∈ D 4 and s − s ′ ∈
Z 2 , the result is still in p 4 m , it is one-to-one correspondence p 4 m × Z 2 → Z 2 , and the summation does not change.
Weiler and Cesa (2021) analogously considers the continuous case, where D 4 is orthogonal transformations so the Jacobian is always 1 . 6.
Definition of ⋆ 7.
Transform law of the induced representation π out
G. P ROOF : EQUIVARIANCE OF STEERABLE VALUE ITERATION
As the third and final step, we would like to show that the full steerable value iteration pipeline is equivariant under G
=
Z 2 ⋊ D 4 .
We need to show that every operation in the steerable value iteration is equivariant.
The key is to prove that max a is an equivariant non-linearity over feature fields, which follows
Section D.2 in (Weiler and Cesa, 2021).
Step 1: V (cid:55)→ Q .
Here [Equation] we prove the equivariance of Q
[Equation]
s ) .
First, let the group acts on both sides:
[Equation]
The the last step we substitute ˜ s = g − 1 s and ˜ a = g.a .
M : Z 2 → { 0 , 1 } 2 is the concatenation of maze occupancy map and goal map, which also lives on Z.
We use two copies of trivial representations as fiber representation ρ M , and denote the induced representation of the field M as π M .
Then, we prove the equivariance: if we transform the occupancy map (and goal map), the value iteration should have both input V and output Q transformed.
Since this is an iterative process, the only input to the value iteration is actually the occupancy map M : Z 2 → { 0 , 1 } 2 .
Before that, we observe that the reward also has G -invariance when we have map as input:
[Equation]
Additionally, since the reward ¯ R aM ( s ) means the reward at given position in map M after executing action a , when we transform the map, we also need to transform the action: ¯ R g.ag.
M ( s ) .
Since it is iterative process, let the Q -map being transformed by g :
[Equation]
[Equation] [Equation] [Equation] The second last step uses the G -invariance condition ¯ R aM ( s ) =
¯ R g.ag.
M ( g.s ) .
The last step uses the equivariance of steerable convolution.
It should be understood as: (1) transforming map g.M and action g.a , is always equal to (2) transforming values [ g.Q ak ] and [ g.V k ] .
This proves the equivariance visually shown in Figure 11.
Step 2: Q (cid:55)→ V .
The second step is to show for V k +1 ( s ) = max a Q ak ( s ) .
Intuitively, we sum over every channel of each representation.
For example, if we have N copies of the regular representation with size | D 4 | = 8 , we transform the tensor ( N × 8) × m × m to (1 × 8) × m × m along the N channel.
Thus, how we use the 8 × 8 regular representation to transform the N × 8 channels still holds for 1 × 8 , which implies equivariance.
The m × m spatial map channels form the base space Z 2 and are transformed as usual (spatially rotated).
Weiler and Cesa (2021) provide detailed illustration and proofs for equivariance of different types of non-linearities.
Step 3: multiple iterations.
Since each layer is equivariant (under induced representations), Cohen and Welling (2016b)
; Kondor and Trivedi (2018); Cohen et al.
(2020) show that stacking multiple equivariant layers is also equivariant.
Thus, we know iteratively applying step 1 and 2 ( equivariant steerable Bellman operator ) is also equivariant ( steerable value iteration ).
H I MPLEMENTATION D ETAILS H. I MPLEMENTATION OF S YM GPPN
ConvGPPN [Redacted for anonymous review] is inspired by VIN and GPPN.
To avoid the training issues in VIN, GPPN proposes to use LSTM to alleviate them.
In particular, it does not use max pooling in the VIN.
Instead, it uses a CNN and LSTM to mimic the value iteration process.
ConvGPPN, on the other hand, integrates CNN into LSTM, resulting in a single component convLSTM for value iteration.
We found that ConvGPPN performs better than GPPN in most cases.
Based on
ConvGPPN, SymGPPN replaces each convolutional layer with steerable convolutional layer.
H. I MPLEMENTATION OF max OPERATION
Here, we consider how to implement the max operation in V k +1 ( s ) = max a Q ak ( s ) .
The max is taken over every state, so the computation mainly depends on our choice of fiber representation.
For example, if we use trivial representations for both input and output, the input would be Q k : Z 2 → R 1 ∗ C A and the output is state-value V k : Z 2 → R .
This recovers the default value iteration since we take max over R C A vector.
In steerable CNNs, we can use stack of fiber representations.
We can choose from regular-regular, trivial-trivial, and regular-trivial (trivial-regular is not considered).
We already covered trivial representations for both input and output, they would be Q k : Z 2 → R C Q ∗ C A and V k : Z 2 → R C V with C Q = C V
= 1 , since every channel would need a trivial representation.
If we use regular representation for Q and trivial for V , they are Q k : Z 2 → R C Q ∗ C A and V k : Z→ R C V with C Q = | D 4 | = 8 and C V = 1 . It degenerates that we just take max over all C Q ∗ C A channels.
For both using regular representations, we need to make sure they use the same fiber group (such as
D 4 or C 4 ), so C Q = C V .
If using D 4 , we have Q k : Z 2 → R 8 ∗ C A and V k : Z 2 → R 8 , and we take max over every C A channels (for every location) and have 8 channels left, which are used as Z 2 → R 8 .
Empirically, we found using regular representations for both works the best overall.
I E XPERIMENT D ETAILS AND A BLATION S TUDY I. E NVIRONMENT S ETUP
Action space.
Note that the MDP action space
A needs to be compatible with the group action
G × A → A .
Since the E2CNN package (Weiler and Cesa, 2021) uses counterclockwise rotations as generators for rotation groups C n , the action space needs to be counterclockwise .
We show the figures for Configuration-space and Workspace manipulation in Figure 12, and the figures for 2D and Visual Navigation in Figure 13.
Manipulation.
For planning in configuration space, the configuration space of the 2 DoFs manipulator has no constraints in the { 0 , π } boundaries, i.e., no joint limits.
To reflect this nature of the configuration space in manipulation tasks, we use circular padding before convolution operation.
The circular padding is applied to convolution layers in VIN, SymVIN, ConvGPPN, and
SymGPPN.
Moreover, in GPPN, there is a convolution encoder before the LSTM layer.
We add the circular padding in the convolution layers in GPPN as well.
In 2-DOF manipulation in configuration space, we adopt the setting in (Chaplot et al., 2021) and train networks to take as input of configuration space, represented by two joints.
We randomly generate 0 to 5 obstacles in the manipulator workspace.
Then the 2 degree-of-freedom (DOF) configuration space is constructed from workspace and discretized into 2D grid with sizes { 18 , 36 } , corresponding to bins of 20 ◦ and 10 ◦ , respectively.
We allow each joint to rotate over 2 π , so the configuration space of 2-DOF manipulation forms a torus T 2 .
Thus, the both boundaries need to be connected when generating action demonstrations, and (equivariant) convolutions need to be circular (with padding mode) to wrap around for all methods.
We allow each joint to rotate over 2 π , so the both boundaries in configuration space need to be connected when generating action demonstrations, and (equivariant) convolutions need to be circular (with padding mode) to wrap around for all methods.
I. B UILDING M APPER N ETWORKS
For visual navigation.
For navigation, we follow the setting in GPPN (Lee et al., 2018).
The input is m × m panoramic egocentric RGB images in 4 directions of resolution 32 × 32 × 3 , which forms a tensor of m × m × 4 × 32 × 32 × 3 .
A mapper network converts every image into a 256 -dimensional embedding and results in a tensor in shape m × m × 4 × 256 and then predicts map layout m × m × 1 .
For the first image encoding part, we use a CNN with first layer of 32 filters of size 8 × 8 and stride of 4 × 4 , and second layer with 64 filters of size 4 × 4 and stride of 2 × 2 , with a final linear layer of size 256 .
The second obstacle prediction part, the first layer has 64 filters and the second layer has 1 filter, all with filter size 3 × 3 and stride 1 × 1 .
For workspace manipulation.
For workspace manipulation , we use U-net Ronneberger et al.
(2015) with residual-connection He et al.
(2015) as a mapper, see Figure.14.
The input is 96 × 96 top-down occupancy grid of the workspace with obstacles, and the target is to output 18 ×configuration space as the maps for planning.
During training, we pre-train the mapper and the planner separately for 15 epochs.
Where the mapper takes manipulator workspace and outputs configuration space.
The mapper is trained to minimize the binary cross entropy between output and ground truth configurations space.
The planner is trained in the same way as described in Section 6.1.
After pre-training, we switch the input to the planner from ground truth configuration space to the one from the mapper.
During testing, we follow the pipeline in Chaplot et al.
(2021) that the mapper-planner only have access to the manipulator workspace. [Figure]
T RAINING S ETUP
We try to mimic the setup in VIN and GPPN (Lee et al., 2018).
For non-SymPlan related parameters, we use learning rate of 10 − 3 , batch size of 32 if possible (GPPN variants need smaller), RMSprop optimizer.
For SymPlan parameters, we use 150 hidden channels (or 150 trivial representations for SymPlan methods) to process the input map.
We use 100 hidden channels for Q-value for VIN (or 100 regular representations for SymVIN), and use 40 hidden channels for Q-value for GPPN and ConvGPPN (or 40 regular representations for SymGPPN on 15 × 15 , and 20 for larger maps because of memory constraint).
I. V ISUALIZATION OF L EARNED M ODELS
We visualize a trained VIN and a SymVIN, evaluated on a 15 × 15 map and its rotated version.
For non-symmetric VIN in Figure 15, the learned policy is obviously not equivariant under rotation.
We also visualize SymVIN on larger map sizes: 28 × 28 and 50 × 50 , to demonstrate its performance and equivariance.
I. A BLATION S TUDY
Additional training curves.
We also provide other training curves that we only show test numbers in the main text.
Training efficiency with less data.
Since the supervision is still dense, we experiment on training with even smaller dataset to experiment in more extreme setup.
We experiment how symmetry may affect the training efficiency of Symmetric Planners by further reducing the size of training dataset.
We compare on two environments: 2D navigation and visual navigation, with training/validation/test size of 1K/200/200, for all methods. [Figure] T e s t / A cc u r a c y [Figure]
Choose of symmetry groups for navigation.
One important benefit of partially equivariant network is that, we do not need to design the group representation of MDP action space ρ A ( g ) for different group or action space.
Thus, we experiment several G -equivariant variants with different group equivariance: (discrete rotation group) C 2 , C 4 , C 8 , C 16 , and (dihedral group) D 2 , D 4 , D 8 , all based on E (2)
-steerable CNN (Weiler and Cesa, 2021).
For all intermediate layers, we use regular representations ρ reg ( g ) of each group, followed by a final policy layer with non-equivariant 1 ×convolution.
The results are reported in the Figure 19 (left).
We only compare VIN (denoted as "none" symmetry) against our E(2) -VIN
(other symmetry group option) on 2D navigation with 15 × 15 maps. [Figure] [Figure] [Figure]
[Figure]
In general
[Figure] the planners equipped with any G group equivariance outperform the vanilla nonequivariant VIN [Figure] and D 4 -equivariant steerable CNN performs the best on most map sizes.
Additionally, since the environment has actions in 8 directions (4 diagonals), C 8 or D 8 groups seem to take advantage of that and have slightly higher accuracy on some map sizes, while C 16 is overconstrained compared to the true symmetry G
= D 4 and be detrimental to performance.
The non-equivariant VIN also experiences higher variance on large maps
[Figure]
Choosing fiber representations.
As we use steerable convolutions (Weiler and Cesa, 2021) to build symmetric planners [Figure] we are free to choose the representations for feature fields, where interme- [Table]
diate equivariant convolutional layers will be equivariant between them f ( ρ in ( g ) x )
=
ρ out ( g ) f
( x ) .
We found representations for some feature fields are critical to the performance: mainly V : S → R and Q : S → R | A | .
We use the best setting as default, and ablate every option.
As shown in Table 4, changing ρ V or ρ Q to trivial representation would result in much worse results.
Fully vs. Partially equivariance for symmetric planners.
One seemingly minor but critical design choice in our SymPlan networks is the choice of the final policy layer, which maps Q-values
S → R |A| to policy logits S
→ R |A| .
Fully equivariant is expected to perform better, but it has some points worth to mention.
(1) We experience unstable training at the beginning, where the loss can go up to 10 6 in the first epoch, while we did not observe it in non-equivariant or partially equivariant counterparts.
However, this only slightly affects training.
In summary, we found even though fully equivariant version can perform slightly better in the best tuned setting, on average setting, partially equivariant version is more robust and the gap is much larger, as shown in the follow table, which an example of averaging over three choices of representations introduced in the last paragraph.
On average partially equivariant version is much better.
In our experiments, partially equivariant version also is easier to tune.
[Table]
Generalization additional experiment for fixed K .
For fixed K setup in Figure 24 (left), we keep number of iterations to be K = 30 and kernel size F = 3 for all methods.
[Figure]
For SymVIN, it far surpasses VIN for all sizes and preserves the gap throughout the evaluation.
Additionally, SymVIN has slightly higher variance across three random seeds (three separately trained models).
Among GPPN and its variants, SymGPPN significantly outperforms both GPPN and ConvGPPN.
Interestingly, ConvGPPN has sharper drop with map size than both SymGPPN and ConvGPPN and thus has increasingly larger gap with SymGPPN and finally even got surpassed by GPPN.
Across random seeds, the three trained models of ConvGPPN give unexpectedly high variance compared to
GPPN and SymGPPN.