Neural Algorithms for Graph Navigation
Abstract
The application of deep reinforcement learning (RL) to graph learning and metalearning admits challenges from both topics.
We consider the task of one-shot, partially observed graph navigation, acknowledging and addressing the difﬁculties of partially observed graph environments.
In this work, we present a framework for graph meta-learning and we propose an agent equipped with external memory and local action priors adapted to the underlying graphs.
We demonstrate the efﬁcacy of our framework through partially-observed navigation on synthetic graphs, as well as application to partially-observed navigation on 3D meshes, showing substantially improvement in one-shot performance over baseline agents
Introduction
Learning a general exploration strategy is a fundamental problem in AI, which requires the framework of Reinforcement Learning (RL) [29, 24, 12].
In this paper, we focus on coverage as a graph exploration problem, such that given a possibly unseen graph environment, our goal is to efﬁciently visit (i.e., cover) as many new states (i.e., vertices) as possible within a time budget.
We formulate the state-coverage problem as a meta-learning task [31], where the goal is to learn a policy from a distribution of input environments that can efﬁciently explore and generalize to unseen environments.
There has been much work on meta reinforcement learning [10, 8], as well as using memory for partially observed tasks [23, 11, 6], but learning on graph environments requires speciﬁc considerations.
The enormous success of Graph Neural Networks (GNN) [26, 3, 17, 7] in learning node embeddings can be attributed in part to their explicit permutation invariance.
However, we observe that navigation objectives on graphs with indistinguishable nodes cannot necessarily be optimally solved.
Including canonical node labels guarantees that the nodes are distinct, but with canonical labels GNNs no longer guarantee permutation invariance [22].
Furthermore, although there exist networks with multiscale dependence on the graph structure [4], partial observability enforces a bound on the GNN spatial width and stymies direct dependence on non-local features.
To address these issues, we summarize the main contributions:
• We propose memory-augmented meta-RL for navigation on unseen graphs.
• We introduce an external memory adapted to the graph environment, which enables a recurrent model with greater representational power for the task of meta-RL graph navigation.
• We introduce a local entropy prior for more efﬁcient exploration on graphs during training.
• We perform extensive experiments and an ablation study on synthetic and real-world graph datasets demonstrating the efﬁcacy of our proposed model.
Related Work
Reinforcement Learning over Graphs.
Reinforcement learning on graph environments has received more attention recently, particularly in the setting of combinatorial optimization [2, 21]
Recent works in this setting span numerous computational challenging problems including Traveling
Salesman [14], MAXCUT [1, 35], and community detection [33] among others.
Our setting departs from most combinatorial optimization problems in the assumptions of partial observability and locally determined action space.
Most similar to our work is a study of transferable graph exploration [6], although their work mainly concerns learning with an action space independent of the graph structure, and therefore the memory of a regular LSTM model sufﬁces in their environments.
Memory-Augmented Neural Networks.
The incorporation of deep learning with an external memory has been well-studied [11].
In reinforcement learning, recurrent parameterizations are well-suited for partially observed MDPs [13].
Our work considers two separate notions of memory, namely spatially adapted memory for reinforcement [23], and memory empowering graph neural networks [15].
Our proposed memory generalizes both notions, deﬁning a time-dependent embedding over arbitrarily structured graphs.
Proposed Framework 3.1 Graph Reinforcement Environment
Our setting concerns meta-learning a policy to effectively explore/navigate unseen graphs , where the full structure of the graph is not observed and rather needs to be explored by the agent.
Formally, we deﬁne the POMDP as follows: for a given graph G =
( V, E ) , the state space and action spare are given by S =
A = V , with deterministic dynamics that only allow for transitions between adjacent vertices.
The reward function introduces partial observability into the problem, with positive reward for visiting a node for the ﬁrst time in an episode, and small negative reward for each timestep.
For our experiments we use +1 and − 0 .
1 , respectively.
This setup characterizes the graph environment for a single graph, while our ultimate goal is to train an agent that can effectively one-shot navigate an unseen graph.
Formally, for an input distribution of graph environments G , we are interested in the objective max π E G ∼G ,τ ∼ π,P G (cid:104)(cid:80)
Ti =
R G ( s i , a i , s i +1 ) (cid:105) where R G and P G denote the reward function and dynamics of G , and we use τ ∼ π, P G to indicate the dependence of the trajectory on the policy and the dynamics of the sampled graph.
This objective doesn’t assume “rapid learning" as in MAML [10], but only that the learned policy will be run on unseen test graphs without further conditioning.
3.2 External Graph Memory
Neural Network Architecture.
We ﬁrst characterize the memoryless neural network that parameterizes our policy π and value function V .
For compactness, we assume bias terms are subsumed in the respective weight matrix.
Consider a graph environment given by G with n = | V | vertices.
By abuse of notation, we will also use s to denote the initial state embedding given by s ∈ R n × k where each node possesses k features (one of which indicates the agent’s current location).
Then the intermediate embedding is deﬁned through a graph neural network (GNN): φ ( s ) =
GNN (cid:0) A, GNN ( A, s ) (cid:1) .
We point out this embedding is also a matrix of node embeddings, namely φ ( s ) ∈ R n × h 1 , where h 1 is a hidden dimension.
The graph neural network architecture is given by a multi-headed graph attention network [30] with skip connections from the input.
Given this shared embedding, we further choose weights W 1 ∈ R h
2 × h 1 , w 2 ∈ R h 2 , W 3 ∈ R h 3 × h 1 , W 4 ∈ R h 4 × h 3 , w 5 ∈ R h 4 .
Now, for an activation function σ and self-attention pooling operator POOL [19] over node embeddings, we deﬁne,
[Equation]
where for this model φ π = φ V = φ .
Furthermore, for simplicity we explicitly enforce that π ( ·| s ) is only supported on the set N ( s ) , as all other actions do not move the agent.
With these parameterizations, we train using the PPO algorithm [28].
We now characterize the model using external graph memory.
The recurrent model conditions the state embedding φ on an external memory embedding M ∈ R n × h M .
Therefore φ now depends on the entire trajectory: we characterize the time-dependent update as M 0 = 0 and
[Equation]
In other words, after reading global information into a matrix r , we apply a Gated Recurrent Unit [5] to locally update our memory embedding, i.e. we only update the rows of M t +1 corresponding to vertices in a 1-hop neighborhood of s .
For simplicity the GRU cells above share weights.
We also mention that for the recurrent model we choose φ π = ψ but keep φ V = φ , as we observed more instability when the value function was allowed to depend on the memory embeddings.
The principle of using an external and shaped memory was explored in the work of [23].
However, their method was restricted to 2D and 3D grids, requiring a notion of global coordinates which isn’t present for experimental settings, and isn’t present for graphs in general.
Entropy Regularization.
In order to ensure adequate exploration during training, the PPO objective is typically augmented with an entropy regularizer.
We incorporate this term by learning a stochastic policy π and adding − λE τ ∼ π,s ∼ τ [ H ( π ( ·| s ))] to our loss, where H ( p ) =
− (cid:80) ni
=1 p i log p i .
However, if our graph has bounded maximum degree, then the set of legal actions at any state will be signiﬁcantly smaller than the set of all actions.
We address this mismatch by instead minimizing the KL divergence between π ( ·| s ) and an appropriate prior distribution Q s .
Of course, for the choice Q s ( a ) ∝ δ a ∈ N ( s ) where N ( s ) is the neighborhood of s , this objective is equivalent to maximizing entropy up to a constant term, but we may consider other priors which encourage more efﬁcient exploration of the environment.
For example, in continuous domains, the work in [20] uses an Ornstein-Uhlenbeck process to encourage exploration with a consistent direction over time.
As a simple analogue on graphs, we consider the prior Q s ( a ) ∝ δ a ∈ N ( s ) + αδ a =˜ s , where ˜ s is the state immediately previous in the current trajectory and α is a hyperparameter.
Choosing negative α induces a prior that discourages backtracking, which will naturally encourage faster mixing.
Experiments
Synthetic Datset.
Our experiments seek to understand the impact of our contributions and the various choices in the proposed model, namely an external graph memory and a local relative entropy prior.
We include node level features indicating the agent’s current node, the nodes visited during the episode, and a normalized timestep.
We will nevertheless refer to the model equipped with these features, but without the graph-adapted memory parameterization as “memoryless", as it only possesses an unlearned, binary representation of the past visitations.
The modeling choices of memory and entropy prior yield four possible models, all of which we train and test under the same meta-learning conditions.
Our implementation of PPO is based on the code given in [18].
We generate synthetic graphs using different random graph families (Watts-Strogatz, Erdos Renyi, Tree graphs), see Appendix A for details of the experimental setup and the synthetic graph generation.
ModelNet Dataset.
For real-world applications, the coverage reward function may be too simplistic, since it assigns equal value to visiting each node.
In practice, the value of a node may depend on its underlying features.
To that end, we consider a real-world dataset (ModelNet10) equipped with a reward function (given explicitly in the appendix) that provides an explicit incentive to the agent to prioritize visiting nodes near several so-called “anchors" points rather than all nodes uniformly.
For our experiments we sample three anchor points.
[Figure]
We train on 100 graphs from ModelNet, decorated with features using a process given in the Appendix, and test on 100 unseen graphs from the same distribution, with the episode length capped atsteps.
The results are plotted in Figure 2a.
In Figure 2b we visualize the feature-aware reward: anchor points are white, densities are red, unvisited nodes are black, and visited nodes are yellow.
Conclusion
In this work, we empirically demonstrated the merit of graph-adapted memory and non-uniform entropy priors for the task of meta-graph navigation.
Our work offers a ﬁrst step in understanding the impact of RL and meta-RL in the context of graph environments, under the implicit constraints induced by using GNNs for parameterization.
For future work, the question of how to scale RL based on graph trajectories to substantially larger graphs remains challenging, as backpropagating through memory over the entire trajectory becomes increasingly difﬁcult with increased length.
Additionally, there may exist more diverse choices of local relative prior beyond controlling the degree of backtracking, which may improve performance further.
A Appendix A.1 Experiment Discussion
Discussion.
For the synthetic graphs, the results divide according to the sparsity of the underlying graphs.
For the sparse Watts-Strogatz and tree graphs, the model with both proposed methods perform best, with a marked difference between the models with graph-adapted memory versus the memoryless models.
Notably, the introduction of the relative entropy prior that biases against backtracking stabilizes the performance across different random initializations.
As corroborated by the median reward in Table 2, the recurrent model with regular entropy is capable of performing similarly to the recurrent model with relative entropy.
But the random trajectories encouraged by a uniform action prior can result in instability and poor performance, dramatically increasing the normalized standard deviation of the former model.
For the dense Watts-Strogatz model and the Erdos-Renyi model, we observe that the choice of entropy prior has signiﬁcantly less impact on the performance.
This is unsurprising, as increasing the graph density implies higher average degree, and the distributional distance between the uniform action prior and the action prior biased against backtracking decreases as the support of the action distribution increases.
However, the inclusion of graph-adapted memory still beneﬁts the model signiﬁcantly, indicating that even in the denser regime, saving memory of the past trajectory is crucial for efﬁcient exploration.
Discussion.
On ModelNet, even after downsampling the mesh graphs retain high density and low diameter.
Nevertheless, we see different behavior than on the dense synthetic families as a result of the feature-dependent reward function.
The choice of non-backtracking prior still improves performance, but the distinction between memoryless and recurrent model becomes smaller.
We attribute this change to the nature of the reward function; once the agent enters the support of a truncated Gaussian, it may greedily choose actions that most increase the density for that feature to locate the anchor point, and therefore the region of highest reward.
In Figure 2b, we compare the visited nodes under the memoryless, uniform entropy prior model against our proposed model.
Indicated by the yellow highlighted points, the former (fourth column) locates and effectively covers the two lower anchor points but struggles to sufﬁciently explore and cover the third before the time budget is exhausted.
Meanwhile, the latter (ﬁfth column) effectively covers all the anchors, as well as most of the graph vertices, which will possess a small positive reward if inside the support of one of the truncated Gaussians.
The agent neglects to visit nodes in the top left which have zero density and therefore zero positive reward.
A.2 Experimental Details
[Figure]
Synthetic Graphs Experimental Setup.
To test our proposed framework, we consider several random graph families from which we sample training and test sets.
Speciﬁcally, we considervertex graphs drawn from the following families:
• Sparse Watts-Strogatz graphs (Sparse WS) [32] graphs conditioned on connectedness.
• Dense Watts-Strogatz graphs (Dense WS)
[32] graphs conditioned on connectedness.
• Uniformly random tree graphs [25].
• Erdos-Renyi graphs conditioned on connectedness [9].
The Watts-Strogatz graph distribution is characterized by two parameters [32], the average node degree k and rewiring probability p , where the former primarily controls graph density and diameter.
For the sparse family we choose k = 2 , p = 1 and for the dense family we choose k = 4 , p = 0 . 1 .
For the Erdos-Renyi graph we sample each edge with a uniform probability p = 0 .
Average statistics of the four graph families, indicating their level of density, are given in Table 1.
Further characterization of this synthetic experiment, and other details of the experimental setup are given in the supplementary materials.
[Table]
[Table]
For each graph family, we sample 500 training graphs and 100 testing graphs, using the coverage reward function as stated above, with the number of steps per episode capped at 100 .
The results of the synthetic experiment over the four chosen graph families are given in Figure 1.
We plot how the test mean reward evolves during training, and test median reward for the graph families are given in
Table 2.
ModelNet Graph Experimental Setup.
The ModelNet10 dataset [34] consists of meshes in 3D space from 10 categories, equipped a simplex structure of vertices, edges, and faces.
In order to render meshes as graph environments, we apply decimation [27] to downsample while preserving overall shape, until the underlying graph has between 50 and 100 vertices.
In order to place meaningful features on our induced graphs, we consider the following process on each graph:
• We sample several distinct “anchor" points from among the vertices.
• We deﬁne a truncated, isotropic Gaussian in coordinate space after normalizing the pairwise distances, centered on each anchor point.
• For each vertex, we take as features the density under the respective Gaussian distributions.
• We further restrict the initial distribution to be uniform over all vertices except the “anchor" points.
Assume this process induces a feature vector x v for each v ∈ V .
Then we can deﬁne the feature-aware reward function as,
[Equation]
otherwise
[Equation]
The ModelNet meshes are explicitly processed as follows:
• Discard meshes with over 1000 points • Apply a decimation algorithm [27] to reduce to approximately 100 vertices, discarding meshes with vertex counts outside the range [50 , 100] • Extract the graph induced by the largest connected component of the decimated mesh •
On three random, distinct vertices, deﬁne a standard Gaussian distribution on normalized 3D coordinates centered at each sampled point • Deﬁne features on all vertices, given by their densities under the three Gaussians, clipping densities below 10 − 3 to
Architecture and Hyperparameters.
We characterize the critic, policy, and memory networks.
Given the parameterization in Section 3, we choose the attention networks in the GAT operator and the attention POOL operator to be two layer networks, with 4 attention heads for the former.
All activations are chosen as ReLUs, and all hidden sizes are 32, excluding h 1 = 64 and h M = 3 .
For training, we optimize with the Adam optimizer [16], using an initial learning rate of 7 ∗ 10 − 4 for the synthetic experiments and 2 ∗ 10 − 4 for the ModelNet experiments.
The PPO clipping parameter is 0.2, and gradients are scaled to bound their norms by 0.5.
The batch size is 400, with 4 gradient updates on each batch.
Finally, α is chosen as − 2 .
for the relative entropy prior model.
A.3 Additional Experiments
Comparison to Single Vector Memory Embedding.
In order to compare against a simpler choice of recurrent model, we also consider a reinforcement learning agent with a memory embedding given as a single vector.
The policy and value function are parameterized as before, but the recurrent update utilizes a single memory embedding m t as follows:
[Equation]
In other words, the single memory embedding is appended to every node embedding before applying the graph convolutions, and the new memory vector is pooled from its neighbors after an update through a GRU.
We demonstrate that this simpler implementation of memory is outperformed by the graph-adapted memory in Figure 4.
The comparison to the single memory embedding baseline in Figure 4 on the sparser graph families indicates the speciﬁc need for structurally determined memory.
The performance of the single vector memory model is comparable to the memoryless model, suggesting that in the absence of meaningful features to distinguish vertices, the past trajectory cannot be effectively captured by a single vector.
is
Additional Visualizations.
We include another visualization of the ModelNet mesh visitations, further separating the base model from our proposed model as shown in Figure 5.
We also include a visualization of the performance of several considered models in the synthetic experiment setting in
Figure 3.
We characterize not just the visitation of each node but the total number of visits, noting that repeatedly visiting a node suggests dithering behavior of the agent.
We observe that dithering is indeed a component of the random and memoryless models, leading to suboptimal navigation, while our proposed model effectively covers nearly the entire graph.