PALMER: Perception-Action Loop with Memory Retrieval
Abstract
To achieve full autonomy in a priori unknown real-world scenarios, agents should be able to operate without assuming auxiliary instrumentation in the environment, act from high-dimensional sensory observations (e.g., images), learn from past experience to adapt and improve, and be capable of long horizon planning.
Classical search algorithms are proficient at addressing the last requirement, while deep learning methods can provide the necessary flexibility to address the others.
Therefore, recent work has proposed combining learning based local-policies (i.e., skills) with classical planning algorithms for subgoal (i.e., subtask) generation.
Following this line of work, we introduce a general-purpose planning algorithm called PALMER that creates a tight feedback loop between representation learning, a non-parametric memory, reinforcement learning, and sampling-based motion planning.
We introduce two main ideas: i) learning a perception function that projects high-dimensional observations into a representation space where latent distances correlate with reachability between states, ii) using these perceptual representations as a way to remember transitions stored in a replay buffer, to explicitly search and recompose past experiences.
We discuss how this joint consideration of perception , action , and memory can address two of the main bottlenecks in reinforcement learning, namely visitation-counts and long-horizon planning.
Introduction
Animals and humans operate on high-dimensional stimuli to achieve diverse and ever-changing goals necessary to fulfill the ecological requirements of survival [1, 2, 3, 4, 5].
Experiential learning through trial-and-error plays a fundamental role in this [6, 7, 8, 9, 10, 5].
Even in simplest environments a brute-force approach to trial-and-error based search is intractable.
This in turn motivates memorybased mechanisms for retrospective reasoning and compositional thinking, such as: i) remembering relevant segments of past experience, ii) recomposing them in new counterfactual ways to form hypotheses and iii) executing such hypotheses as part of a targeted search strategy.
Such memorybased mechanisms can significantly accelerate the trial-and-error process compared to a uniform search method.
This is because the same behavior can remain valid for different goals and in different contexts due to the compositional structure of real-world environments.
What principles can allow for memory mechanisms to remember and recompose bits of experience?
The concept of dynamic programming (DP) is directly related to this discussion, as it greatly reduces the computational cost of trial-and-error by employing the principle of optimality [11].
This principle can be colloquially stated as treating new and complex problems as a recomposition of old and simpler sub-problems that were already solved before.
Recent work [12, 13, 14] employs this perspective to build hierarchical reinforcement learning (RL) algorithms for goal-reaching tasks.
Such methods set edges between states using a distance regression model to build a planning graph, perform shortest path computations over it using DP-based graph search, and follow the resulting shortest paths with a learning-based local policy.
Our paper builds upon this line of work.
Contribution: We describe a method for an RL policy to explicitly remember and retrieve previously observed sequences of state transitions from its replay buffer using perceptual representations .
This is achieved through a two-step process.
First , we learn a latent space where the distance between two states captures how much time it takes for the optimal policy to go from one to the other.
Second , given a query state and a replay buffer, we use this distance metric to retrieve nearest neighbor states from the replay buffer that are easily reachable from the query state.
We show that this perception enabled memory mechanism can have two important uses.
First one addresses the exploration problem, by defining an approximate visitation-count for any given state (i.e., discrete or continuous) based on its number of neigboring states in the replay buffer.
Second one addresses the long-horizon planning problem, by defining sampling-based planning algorithms that search over the replay buffer [12] to stitch together trajectories whose endpoints are neighboring states.
This trajectory stitching approach allows for creating robust roadmaps to connect any two distant states (as depicted in Fig.1).
Our approach operates entirely on offline, unlabeled data [15, 16, 17], and can therefore be flexibly combined with any auxiliary data collection method to populate the replay buffer.
Our experiments present a proof-of-concept by implementing an image-based navigation policy in simulation, using an offline replay buffer populated with uniform random-walk exploration data.
Perception-Action Loop with Memory Retrieval
Nomenclature: An environment is represented as a tuple ⟨S , A , p env ⟩ , where S and A are the state and action spaces, and p env ( s ′ | s, a ) is the markovian transition dynamics.
A trajectory τ ∈ T is any sequence of states and actions.
τ i,s, 0 ; τ i,s, − 1 ; τ i,s,j denote the first, last, and j ’th states in τ i respectively.
The length of a trajectory in terms of timesteps is denoted as len ( τ ) , and concatenation of two trajectories is denoted as τ cat = τ 1 ◦ τ 2 .
We assume an additive reward function R : T → R where R ( τ ) =
(cid:80) ( s,a ) ∈
τ r ( s, a ) .
We call a finite set of trajectories M = { τ i } a replay buffer.
2.1 Perceptual Representations that Capture Local Reachability
A key component of our framework is a perceptual encoder f ϕ ( s ) :
S → R d that maps states into a representation space where L2 distance d ϕ ( s t , s g ) :
= ∥ f ϕ ( s t ) − f ϕ ( s g ) ∥ captures local reachability.
To discuss what local reachability means more rigorously, we follow the work of [18, 12] and define a goal-conditoned reward function r ( s t , a, s t +1 , s g )
= − 1 s t +1 ̸
= s g that returns − 1 for all timesteps before reaching a goal (i.e., and 0 afterwards).
This means goal-conditioned Q-values [18, 19] for the optimal policy correspond to negative shortest-path distances (i.e., max a Q ( s t , a, s g )
=
V ( s i , s j )
= − len ( τ sp ) ).
We can then define a symmetric distance metric between states as d Q ( s c , s g ) :
= max ( − V ( s c , s g ) ,
− V ( s g , s c ) ) .
This in turn corresponds to the two-way consistency criterion proposed in [13].
What we want from f ϕ ( s ) is for d ϕ ( s c , s g ) and d Q ( s c , s g ) to roughly correlate.
2.2 Combining Representation Learning with Goal-conditioned Reinforcement Learning
Any perceptual encoder f ϕ whose latent representations satisfy the local reachability property defined in Sec.2.1 can be used to implement the nearest neighbor retrieval and trajectory stitching mechanisms from upcoming sections 2.3 and 2.4.
This section discusses one possible way to obtain such a perceptual encoder, by using goal-conditioned Q-values for contrastive representation learning.
We propose a model (depicted in Fig.2) that combines the following standard components from the literature: i) z
= f ϕ ( s ) , projecting a state into a latent representation; ii) p fwd ( z ′ t +1 | z t , a t ) , modelling the transition distribution induced by p env ( s ′ | s, a ) over the latent space z
= f ϕ ( s ) , as discussed in [20, 21]; iii) π inv ( a ′ t | z t , z g ) , defining a distribution of actions to reach a goal state, as discussed in [20, 21, 14]; iv) p t ( T ′ |
z t , z g ) , modelling the distribution of timesteps necessary to reach a goal state, as discussed in [22]; v) Q ( s t , a t , s g ) , a Q-value function that provides local reachability estimates between pairs of states, as discussed in [12, 18].
Following [12, 18], we train Q ( s t , a t , s g ) over an offline replay buffer M , using hindsight relabelling [23, 18] with a reward function r ( s t , a, s t +1 , s g ) =
− 1 s t +1 ̸
= s g .
After training Q ( s t , a t , s g ) in isolation, we freeze its parameters and use it to define a contrastive loss function [24] L Q as explained below.
We then train the remaining components using the same replay buffer M .
We randomly sample a transition ( s t , a t , s t +1 ) and a time difference T , and set the goal state as s g :=
s t + T , as in hindsight relabelling.
We then minimize the following losses:
• L
[Equation]
where l
hinge is the hinge loss [25].
This contrastive loss dictates that perceptual representations should be close together (i.e., d ϕ ( s t , s g )
≤ d p holds)
if and only if two states are close to each other in terms of reachability (i.e., d Q ( s t , s g ) ≤ c Q holds).
d p and c Q are hyperparameters.
• L T ( T ′ , T ) , L inv ( a ′ t , a t ) , and L fwd ( z ′ t +1 , z t +1 ) are MSE and cross-entropy losses
L T and L inv dictate that perceptual representations should capture enough information to know when and how an agent can reach from one state to another, while L fwd dictates that they should capture only a minimal-sufficient statistic for doing so ([21] presents a more elaborate discussion).
2.3 Perceptual Memory Retrieval (PMR)
Given a perceptual encoder f ϕ that captures local reachability, we go over all states s i ∈ M in the replay buffer and compute their projections z i = f ϕ ( s i ) , which are stored alongside the states themselves.
We then employ z i to implement two retrieval mechanisms from the replay buffer: i) retrieving neighbor states, and ii) retrieving neighbor trajectories.
Given a query state s c and radius d p (i.e., the same one used in the contrastive loss L Q in Sec.2.2), retrieving neighbor states amounts to computing the set N d p ( s c ) = { s n | d ϕ ( s c , s n ) ≤ d p } .
The d ϕ computations and comparison operations involved in this computation can be vectorized.
Furthermore, computing N d p ( s c ) is embarrassingly parallelizable across different query states s c .
The number of neighbors |N
d p ( s c ) | of a query state s c is a good measure of how many times the agent has visited around s c .
This, combined with how cheap and easy it is to compute |N
d p ( s c ) | , makes it a good visitation-count that is applicable to both discrete and continuous state spaces.
Given a starting state s c and a goal state s g , we can also search the replay buffer for the highest reward trajectory τ that starts from a state τ s, 0 in N d p ( s c ) and ends in a state τ s, − 1 in N d p ( s g ) .
This corresponds to the following optimization problem:
[Equation]
To find τ M ( s c ,s g ) , we first select those state pairs ( s i , s j ) ∈ N d p ( s c ) × N d p ( s g ) that come from the same episode.
We then take all sequences of transitions τ ij =
{ s i , a i , s i +1 , ..., s j − 1 , a j − i , s j } that start from s i and end at s j .
We sort them based on R ( τ ij ) , and return the trajectory with the highest reward.
We call this trajectory retrieval process perceptual memory retrieval (PMR).
We note that all operations involved in PMR can be vectorized, and it takes ∼ 8 e − 4 seconds to compute τ M ( s c ,s g ) for each query pair.
Again, PMR is embarrassingly parallelizable across different query pairs ( s c , s g ) .
In following sections, we discuss how PMR can be employed for long-horizon planning.
2.4 ‘Experiential Planning’ Through Trajectory Restitching
Sampling-based planning algorithms such as RRT [26], RRT* [27], and PRM [28] connect points sampled from obstacle-free space with line segments in order to build a planning graph.
We instead reimagine them as memory search mechanisms by altering their subroutines so that whenever an edge is created, a trajectory is retrieved from the replay buffer through PMR (eq.1) and stored in that edge.
Our new definitions for these subroutines directly mirror the original ones given in [27]:
1) Sampling:
Sampling originally returns a point from obstacle free space.
We
instead
return a state s c from the replay buffer M using any distribution (e
g., uniform, or based on visitation-counts).
2) Lines and Their Cost: The equivalent of drawing a line segment in our framework is retrieving a trajectory τ M ( s c ,s g ) , and its length and cost are len ( τ M ( s c ,s g ) ) and −R ( τ M ( s c ,s g ) ) respectively.
Nearest State and Neighborhood Queries:
Given a query point s i , these subroutines return the closest point or a neighborhood of points within a distance, among a set of vertices V
= { s j } .
We preserve these definitions, and only replace the metric from euclidean distance to len ( τ M ( s c ,s g ) ) .
[Equation]
s
[Equation] [Equation]
≤ r } 4) Collision Tests: Collision tests originally prevent the sampling and line drawing subroutines from intersecting obstacles.
Since we are planning in retrospect, any such undesirable event can be handled during PMR by adjusting the reward function (i.e., if τ has such an event, this reflects on R ( τ ) ).
Using these subroutines directly in-place of their originals, we reimplement experiential equivalents of PRM, RRT, and RRT*, which we call R-PRM, R-RRT, R-RRT*.
We denote the resulting planned trajectory as τ M * ( s c ,s g ) .
Algorithms 1, 2 describe R-PRM as an example, and the supplementary contains descriptions for R-RRT, R-RRT*.
We note two things about our proposed ‘experiential planning’ algorithms.
First, they can optimize any general reward function R .
As the number of sampled vertices increases, R ( τ M * ( s c ,s g ) ) gets optimized through dynamic programming (i.e., by minimizing the Bellman error between vertices of the roadmap G ), therefore employing the same mechanism as classical sampling-based planning
1: Input: f ϕ , M 2: V ← { SampleFree i } i =1 ,...,num _ vertices ; E ← ∅ ▷ Initialize vertices and edges 3: for each s i ∈ V do 4:
U ← Near ( V, s i , r ) \ { s i } 5: for each s j ∈ U do ▷ Place PMR trajectories in edges 6:
E ← E ∪ { ( s i , s j ) : τ edge =
τ M ( s i ,s j ) , d edge = −R ( τ M ( s i ,s j ) ) } return G =
( V, E )
1: Input: s c , s g , G =
( V, E ) , R ( τ ) , f ϕ , M 2: for each s i ∈ V do ▷ Insert s c and s g into the PRM graph 3: if len ( τ M ( s c ,s i ) )
≤ r then ▷ Place PMR trajectories in edges 4:
E ← E ∪ { ( s c , s i ) : τ edge =
τ M ( s c ,s i ) , d edge = −R ( τ M ( s c ,s i ) ) } 5: if len ( τ M ( s i ,s g ) ) ≤ r
then 6:
E ← E ∪ { ( s i , s g ) : τ edge =
τ M ( s i ,s g ) , d edge = −R ( τ M ( s i ,s g ) ) } 7: τ stitched ← ∅ 8: { s j } ← ShortestPath ( s c , s g , G, R ( τ ))
▷ Trajectory stitching by dynamic programming 9: for 0 < i < |{ s j }| do ▷ Concatenate PMR trajectories along the shortest path 10: τ stitched ← τ stitched ◦ τ M ( s i − 1 ,s i ) return τ M * ( s c ,s g )
=
τ stitched algorithms [27].
Second, they operate on an offline dataset of unlabeled transitions which solely consists of high-dimensional on-board sensory data (e.g. images), without assuming any auxiliary instrumentation in the environment or oracle information that cannot be sensed by the agent on its own .
They therefore aim to relax the assumptions classical sampling-based planning methods make about what constitutes a model (e.g., replacing a geometric environment model with sensory experience) and what constitutes a state (e.g., enabling search and planning directly over images).
2.5 PALMER: Closing the Perception-Action Loop to Optimize Memory Contents
We iteratively form and execute τ M * ( s c ,s g ) , and whenever execution is successful, we insert the resulting transitions back into M to optimize its contents through trial-and-error.
This creates the following perception-action loop: i) M with optimized contents is used to train a more accurate
Q ( s t , a, s g ) , ii) a more accurate Q ( s t , a, s g ) creates a more accurate distance metric d ϕ , iii) a better d ϕ generates better τ M * ( s c ,s g ) , iv) better τ M * ( s c ,s g ) result in higher frequencies of successful execution to further optimize M , therefore looping back to step i (see the supplementary for an algorithmic description).
This is essentially a generalized policy-iteration process [29], where generating and executing τ M * ( s c ,s g ) is equivalent to policy-evaluation, and putting only successfully executed trajectories back into M and then updating Q and f ϕ is equivalent to policy improvement.
Related work
Self-supervised goal reaching: Our approach is closely related to goal-reaching methods that combine learning-based distance-regression with graph search, particularly Semi-parametric Topological
Memory (SPTM) [14] and Search on the Replay Buffer (SoRB) [12], which we compare to in our experiments.
The key difference of our approach is that when setting the edges of the planning graph, it retrieves transitions that actually happened rather than relying on learned distance regression.
This brings two main benefits.
First is robustness.
Local reachability estimates are susceptible to overestimation when evaluated between pairs of states that are far apart or unreachable.
This is because such states rarely occur together and are therefore out of distribution for the distance regression model.
This creates ‘hallucinated’ shortcuts in the planning graph that corrupt shortest path queries [12, 13].
To address this, [14] employs temporally consistent localization and adaptive waypoint selection, while [12] employs distributional Q-learning and an ensemble of Q-functions.
In our approach, eq.1 naturally adresses this problem, since it requires an actual short trajectory in the dataset approximately connecting two states before marking them as close.
The second benefit of our approach is that it can optimize general reward functions.
This is because it decouples the reachability metric len ( τ ) (used in nearest neigbor queries and as a threshold to create edges) from the downstream task reward R ( τ ) (used to set edge distances), unlike previous work.
Image-Based Navigation: [30, 31, 32] present learning-based navigation systems that incrementally build roadmaps through online operation.
Our approach has two main differences: i) it builds a roadmap entirely using raw offline data, therefore allowing applications like multi-robot learning without additional loop-closure mechanisms to fuse graphs from multiple agents, ii) our approach can optimize general reward functions, therefore it is not limited to navigation.
Robot Motion Planning:
A common approach to motion planning is to first run a sampling-based planning
algorithm
[33, 27], and then refine the result through trajectory optimization [34, 35, 36] to satisfy constraints [37, 38, 39].
An important bottleneck is that sampling-based planning algorithms require a precomputed map of the environment, and our approach extends such algorithms in a way that relaxes this requirement by replacing a precomputed map with raw exploration experience.
SLAM and Geometric Maps: SLAM based methods [40] can autonomously construct high-fidelity geometric maps [41, 42], therefore alleviating the bottleneck of precomputing environment maps.
The downside of such approaches is that they can abstract away useful physical and semantic affordances.
For example, a purely geometric map cannot plan a path through a traversable field of tall-grass, while our approach can learn such affordances as long as they are represented in past experiences.
Experiments
Setup: Our experiments are performed in ViZDoom [43], Habitat [44], and the Maze2D benchmark [15].
The VizDoom environment consists of a clover shaped maze.
States solely consist of four images
I North/East/South/West that form a panorama (i.e., 4 × 3 × 160 × 120 dimensions), and actions move the agent North/South/East/West by a fixed distance ∆ .
The maze contains many long-thin columnlike obstructions (shown as dots in visualizations).
Habitat experiments contain demonstrations on two large-scale scans of real-world apartments: i) Roxboro, with a total area of 62 m2, and ii) Annawan, which has a total-area of 75 m 2 .
States consist of a single 150 FOV image (i.e., 3 × 256 ×dimensions).
There are 3 actions: { turn _ left _ 30 _ deg, turn _ right _ 30 _ deg, move _ forward _ ∆ } .
Maze2D is a continuous control task, where states consist of the 2D position and velocity of a point mass, and actions correspond to 2D accelerations.
In all environments, an offline training dataset is collected by a uniform random walk exploring the environment.
For VizDoom and Habitat, this offline training dataset consists of only 300k and 150k timesteps respectively, while for Maze2D there are 1e6 timesteps.
Supplementary contains further details.
4.1 Experiments in Vizdoom
Validating Perceptual Representations: Fig.3 shows that d ϕ ( s t , s g ) obtained from our model captures a suitable notion of local reachability.
Fig.5 in turn shows that retrieving nearest neighbor states
N d p ( s c ) from M using d ϕ (i.e., NN retrieval) returns physically close states.
Validating Perceptual Memory Retrieval (PMR): Fig.5 shows visualizations of trajectories retrieved with PMR.
We implement a retrieval policy π M that computes τ M ( s t ,s g ) through eq.1 at each timestep t and executes argmax a Q ( s t , a, τ M ( s t ,s g ) ,s, 1 ) , therefore forming a model predictive control (MPC) loop.
We evaluate π
M in an image-based navigation task where start/goal images are sampled randomly to have an euclidean distance n × ∆ in between and a trial is considered successful if the agent can get within ∆ proximity of the goal position within 4 × n time-steps.
We use the local policies from SORB [12] and SPTM [14] as baselines.
Fig.4 shows the results.
The main mode of failure for both SPTM and SORB local policies is that they get stuck in column-like structures.
π M avoids this, since eq.1 retrieves collision free
τ M ( s t ,s g ) .
Robust Distances: PMR also helps avoid hallucinations in local distance regression.
Fig.5 illustrates this point by setting edges between sampled states by thresholding len ( τ M ( s t ,s g ) ) , where methods of [12, 22] are used as baselines.
It can be seen that edges set by len ( τ M ( s c ,s g ) ) are more robust.
Experiential Planning Algorithms: Fig.6 shows visualizations of planning graphs and τ M * ( s c ,s g ) pro- duced by R-PRM, R-RRT, and R-RRT*.
It can be seen that R-PRM doesn’t contain any hallucinated edges, while R-RRT and R-RRT* maintain the visual characteristics of their classical counterparts (i.e., R-RRT has jagged branches with uniform coverage, while R-RRT* has straight branches shooting out from the root).
We implement an MPC policy π M * that replans at each timestep t using Algorithm 2 to return τ M * ( s t ,s g ) , and executes argmax a Q ( s t , a, τ M * ( s t ,s g ) ,s, 1 ) .
We again use SORB [12] and SPTM [14] as baselines.
3 Fig.4 shows the results.
In addition to the local policy getting stuck, a new mode of failure for both baselines is that false distance estimates throw-off graph search by setting hallucinated shortcuts.
A new baseline is π mpc , which extends the SPTM local policy by using p fwd and p t from Sec.2.2 to implement an MPC loop with n-step look-ahead.
π mpc avoids getting stuck in columns thanks to n-step lookahead but still isn’t sufficient for global navigation as the accuracy of simulated rollouts from p fwd decreases with the number of timesteps.
Closing the Loop with PALMER:
We optimize the contents of M by iteratively generating and executing τ M * ( s c ,s g ) .
We then retrain all model components only on the resulting new data that is equal in size to the initial unoptimized M .
Fig.7 shows the results.
When π M , and argmax a Q ( a t , a, s g ) are used as policies, their success ratio increases significantly if they are trained on the optimized
M .
Q-value estimates trained on the optimized M also propagate better to goals further away.
The scaling of len ( τ M ( s c ,s g ) ) with goal-distance changes from an exponential trend to an approximately linear one, due to the inclusion of transitions from successfully executed τ M * ( s c ,s g ) .
4.2 Experiments in Habitat
As shown in Fig.8, we find that our method allows image-based navigation in this new domain with significantly different visuals and layouts (i.e., real-world apartments), action space (i.e., turn-left, turnright, go-forward), and state space (i.e., single 256 × 256 RGB images with 150 FOV).
Perhaps more surprisingly, we find that training f ϕ only on exploration data from a single apartment generalizes substantially well to any unseen apartment, which directly allows perceptual memory retrieval and experiential planning when provided with a corresponding replay buffer.
For a quantitative evaluation, we randomly pick two apartments, named Roxbox and Annawan.
In both apartments, we collect an exploration dataset using a uniform random walk sequence of only 150k timesteps.
We train the model components solely on data from Roxbox.
We then use them to implement our π M ∗ policy from
Sec.4.1, which we then evaluate on both apartments.
For n ∈ { 8 , 16 , 24 , 32 , 36 , 44 } , we randomly sample 100 pairs of start and goal-states in a way that the geodesic distance between them lies within n × ∆ and ( n + 8) × ∆ through rejection sampling.
A policy is considered successful if it can get within 2 × ∆ proximity of the goal-state.
We do not include SPTM and SORB as baselines, because we found that the models π inv ( a | s t , s g ) and argmax a Q ( s t , a, s g ) that they use as local navigation policies achieved almost zero percent success rate in reaching local goals beyond ∼ 2 × ∆ distance.
We empirically observed that most of the time these policies get stuck in repetitive rotational motions without moving forward.
This is most likely due to the difficulty of offline RL training with hindsight relabelling over random-walk data obtained with a much more challenging non-cartesian action space { turn _ left _ 30 _ deg, turn _ right _ 30 _ deg, move _ forward _ ∆ } .
4.3 Experiments in Maze2D
[Table]
To test our method on a continuous control task
[Table]
we perform additional experiments on the Maze2D benchmark.
As shown in Table.1, we find that the same π M ∗ policy from sections 4.1 and 4.
achieves strong performance, and can solve mazes of all three complexities.
Discussion and Future Directions
Is PALMER less expressive than standard deep Q-learning: Two important premises of deep Qlearning [47, 38] are: i) minimizing Bellman error through temporal-difference (TD) updates can restitch observed transitions in new optimal ways [15, 48], ii) a neural network can learn to extrapolate
Q-values to unobserved but close-by states in high-dimensional spaces (e.g. images) [49].
Both arguments are equally valid for our approach, since it can: i) restitch transitions at arbitrary resolutions (i.e., anywhere from one-step transitions to multi-step trajectories) by virtue of sampling-based planning, ii) group together close-by states through d ϕ .
Therefore, PALMER is an RL algorithm that: i) optimizes Bellman error through sampling-based optimal planning rather than gradient-based
TD-updates [49], ii) performs extrapolation between states using a perceptual-backbone f ϕ rather than a deep Q-network, and iii) replaces the greedy-policy argmax a Q ( s t , a, s g ) and value estimate max a Q ( s t , a, s g ) with argmax a Q ( s t , a, τ M * ( s t ,s g ) ,s, 1 ) and R ( τ M * ( s c ,s g ) ) respectively.
The key benefits of these alterations come into play when s t and s g are far apart, and these benefits are: i) the PMR mechanism in eq.1 that prevents hallucinations in Q ( s t , a, s g ) , ii) global propagation of value estimates by virtue of employing sampling-based planning methods, which are known to be particularly proficient at searching high-dimensional state spaces across long-horizons [39, 33].
Combining PALMER with standard deep Q-learning: Our approach can also be flexibly combined with any traditional Q-learning method [49, 50, 51], by using our experiential planning algorithms (Sec.2.4) as experience replay methods [52].
This alternative approach stitches together τ M * ( s c ,s g ) during training, and perform backwards TD-updates over this trajectory starting from s g = τ M * ( s c ,s g ) ,s, − 1 and ending at s c = τ M * ( s c ,s g ) ,s, 0 .
As suggested by Fig.7, this can allow value estimates Q ( s t , a, s g ) to propagate more globally.
Our proof-of-concept experiments identify this as a promising direction, and we leave a further extensive evaluation to future work.
Connections to contingency learning: Contingency learning refers to the acquisition of knowledge of statistical correlations between percepts [53, 3, 54].
Following this definition, PALMER can be interpreted as a contingency learning framework, as the latent distance metric d ϕ captures statistically how likely two states are to be observed in close temporal proximity .
The knowledge of these statistical contingencies between states is then used for long-horizon decision making through the proposed perceptual memory retrieval and experiential planning mechanisms.
Conclusion and Limitations
We presented PALMER, which uses perceptual representations that correlate with local reachability as a way to explicitly remember previously visited states and trajectories in the replay buffer .
We have shown how this memory retrieval mechanism can be employed to implement ‘experiential planning’ algorithms, which sit as an interface between any given local controller π and any given offline dataset of state transitions M .
Given a goal state s g and reward function R , these algorithms search the contents of M to stitch together a sequence of transitions τ M * ( s c ,s g ) = { s 1 , a 1 , s 2 , ... } that reaches s g while maximizing R .
This sequence is then communicated to π to be executed, which communicates back the resulting real transitions to update the contents of M .
Our experiments show that PALMER can solve long-horizon planning tasks from continuous highdimensional inputs.
In particular, we show that given an offline dataset of only 150k transitions (i.e., compared to sample complexities around the orders of magnitude 1e6-1e7 common in RL) obtained from an entirely uniform random-walk (i.e., which is significantly less structured compared to on-policy rollouts), memory-based experiential planning allows image-based navigation between any two points in large-scale scans of real-world apartments.
The biggest limitation of our current exposition as well as the most immediate direction for future work is to extend our experiments to more complex planning problems and real-world robotic systems [55, 56].
We believe that our memory-based experiential planning perspective highlights a number of interesting questions for future research.
First, which transitions should be kept in the replay buffer
M , and which ones should be discarded?
M cannot be infinitely expanded after deployment, and it is critical to distill away redundancies between stored experiences.
Second, when the environment undergoes a change, which transitions in the replay buffer remain valid and can still be used for planning, and which ones become invalid?
A mechanism that can answer this question can allow quick and sample-efficient adaptation to environmental changes.
Third, how can we extend f ϕ to allow more abstract associations and functional equivariances between states?
This can improve generalization by defining a more flexible notion of experience retrieval that can recycle past behavior in new contexts and for new tasks.
We leave these questions to future work.
Supplementary Material
This document presents additional implementation details, visualizations, and conceptual discussions that were excluded or briefed due to space limitations in the main paper.
The organization mirrors the sections from the main paper (with the exact same order and numbering), and the exposition generally maintains a question-answer format.
It aims to provide significantly more details, at a degree of clarity sufficient for re-implementation.
As such, we recommend referring to this document whenever any section in the main paper can benefit from further elaboration.
The supplementary material consists of:
• vizdoom_nav — navigation videos in VizDoom.
• habitat_roxbox_nav — navigation videos in the training apartment Roxbox in Habitat.
• habitat_annawan_nav — navigation videos in the test apartment Annawan in Habitat.
• maze2d_experiments — goal reaching videos in Maze2D. • planning_vis — videos visualizing the execution of R-RRT and R-RRT*.
• code — source code for PALMER.
Table of Contents 1. Introduction
• Why learn a latent distance metric for nearest neighbor retrieval?
• What does stitching transitions together mean?
Perception-Action Loop with Memory Reorganization 2.1.
Perceptual Representations that Capture Local Reachability and Functional Similarity • What does local reachability mean?
Combining Representation Learning with Goal-conditioned Reinforcement Learning • How are the model components trained, and what are their exact inputs and outputs?
• What do the terms in the contrastive loss-function L Q mean?
Combining Representation Learning with Goal-conditioned Reinforcement Learning •
How do we solve the optimization problem in equation 1 from the main paper?
• What do the retrieved trajectories τ M ( s c ,s g ) look like? 2.4.
Trajectory Restitching Through ‘experiential planning’ •
How does R-PRM work in detail?
• How does R-RRT work in detail? • How does R-RRT* work in detail?
• What does optimizing the Bellman error on the roadmap mean?
• What does restitching transitions at arbitrary resolution mean?
PALMER:
Closing the Perception-Action Loop to Optimize Memory Contents • What does optimizing memory contents mean?
• How does the perception-action loop in PALMER work in detail?
3. Related Work
• What is the main reason why memory-based reasoning over actually observed transitions is necessary?
Why are methods like SPTM or SoRB that solely rely on learning-based distance estimates are inherently prone to false predictions?
• What are the details for our implementations of SoRB and SPTM?
4. Experiments
Setup
• What are the details for the experimental setup in VizDoom?
Validating Perceptual Memory Retrieval (PMR) • What is the exact evaluation process that produced Fig.4 in the main paper?
Robust Distances
• What is the exact evaluation process for the right panel of Fig.5 in the main paper?
experiential planning Algorithms
• Why does the policy π M ∗ use R-PRM for experiential planning?
• What are the details for the π mpc baseline?
Experiments in Habitat
• What are the details for the experimental setup in Habitat?
• Why does the agent occasionally take random-looking actions in the habitat navigation trials?
5. Discussion and Future Directions
• How is PALMER related to the "Options Framework (Sutton et al.)
" and "Skill-Chaining (Konidaris et al.)"?
• How is PALMER related to "LQR-Trees (Tedrake et al.)?
Introduction
Why learn a latent distance metric for nearest neighbor retrieval: In a low-dimensional state space such as 3D positions, L2 distance (i.e., euclidean distance) directly correlates with local physicalreachability (i.e., we emphasize local, because euclidian distances still do not match geodesic distances globally).
Therefore in such state-spaces, grouping together two nearby states and treating them as the same single state for downstream global planning should still result in a feasible planned trajectory.
By feasible, we mean that the gaps and approximations introduced by state grouping are functionally inconsequential and can be handled reasonably well by a local policy tracking the global planned trajectory.
This property doesn’t hold in high-dimensional state spaces such as images, since the L2 distance doesn’t correlate with physical reachability.
The main purpose of f ϕ is to project such high-dimensional spaces into a low-dimensional representation space where this property holds, so that nearby states can be fused together to make sampling-based planning computationally tractable.
What does stitching transitions together mean: As shown in Fig9, if there are two separate sequences of transitions in an offline memory buffer that traverse segments ( A, B ) and ( C, D ) , an agent should be capable of going from s c to s g through the segments ( A, D ) event if such a direct path of transitions was never actually observed.
Traditional deep Q-learning methods achieve this by combining and propagating value estimates through TD-updates (i.e., argmax a Q ( s 1 , a, s g ) points to segment D after TD updates over the path ( C, D ) , therefore the argmax policy would follow the path ( A, D ) when going from s c to s g .
[15] provides a further discussion).
In our approach, this is achieved by setting edge distances for ( A, B, C, D ) in a planning graph through perceptual memory retrieval, and then performing a shortest path computation to retrieve the path ( A, D ) .
Perception-Action Loop with Memory Reorganization 2. Perceptual Representations that Capture Local Reachability
What does local reachability mean: At a high-level (and for the special case of image-based navigation) what the term ‘local reachability’ intends to convey is that if two images I
and I 2 are from physically close positions, d ϕ = |
f ϕ ( I 1 ) − f ϕ
( I 2 ) | should be small.
This in turn provides a metric for grouping together states that is better than the L2 distance in image space, which has no correlation with physical reachability.
Such a metric is necessary to make search and sampling-based planning planning over the memory buffer computationally tractable.
This learned metric d ϕ serves the exact same purpose as the hand-crafted image compression criterion employed in [57, 58] to initialize cells from states.
2. Combining Representation Learning with Goal-conditioned Reinforcement Learning
How are the model components trained, and what are their exact inputs and outputs: More detailed descriptions for all model components are given below (Fig.10 presents a visualization):
• The architecture and training of Q ( s t , a t , s g ) is completely decoupled from the other components.
It consists of cascaded convolutional and fully-connected layers with batch normalization and ReLU activations between each layer.
It takes as input the concatenated images for current and goal states (i.e., shape B × C × H × W ), and outputs a vector of Q-values for each action (i.e., shape B × num _ actions ).
It is trained through offline DDQN [45] with hindsight goal-relabelling [23].
We first sample t ∼ Uniform (0 , dataset _ size ) and T ∼ Geom ( p ) , and then retrieve from the replay buffer a transition and a goal state as ( s t , a t , s t +1 , s g :=
s t + T ) .
We then minimize the TD error [ Q θ ( s t , a t , s g ) − ( 1 s t +
=
s g
+ γ
s t +1 ̸
= s g max a Q θ − ( s t +1 , a, s g )) ] 2 , as in [59].
The perceptual backbone f ϕ ( s ) uses a standard Resnet-18 architecture.
It takes as input the images for a given state (i.e., B × C × H × W ), and outputs a low-dimensional representation vector z
= f ϕ ( s ) (i.e., shape B × D ).
All other components take as input these low-dimensional representations, rather than operating over images.
• p fwd ( z ′ t +1 | z t , a t ) , π inv ( a ′ t | z t , z g ) , and p t ( T ′ | z t , z g ) all consist of fully-connected layers with ReLU activations.
To train them, we first sample t ∼ Uniform (0 , dataset _ size ) .
We then sample T according to T ∼ Uniform (0 , T max ) or T ∼ Uniform ( T max , dataset _ size − t ) , half the time from the former distribution, half the time from the latter.
We retrieve from the replay buffer a transition and a goal state as ( s t , a t , s t +1 , s g :=
s t + T ) , and project them into low dimensional representations ( z t , a t , z t +1 , z g ) using f ϕ .
These are concatenated and passed to models p fwd ( z ′ t +1 | z t , a t ) , π inv ( a ′ t | z t , z g ) , p t ( T ′ | z t , z g ) in a way compatible with their arguments.
p fwd outputs the mean for the predicted next state distribution (i.e., shape B × D ), and is trained using the MSE loss L fwd with z t +1 as the target.
π inv outputs a vector of probabilities over actions (i.e., shape B × num _ actions ), and is trained with the cross-entropy loss L inv with a t as the target.
p t also outputs a discrete probability distribution over [0 , T max ] that predicts the distribution of time-steps to reach the goal, where the last bin T max serves as a catch-all for all values above it.
It is trained using the cross entropy loss L T , with T as the target.
As mentioned in the main paper, all components f ϕ , p fwd ( z ′ t +1 | z t , a t ) , π inv ( a ′ t | z t , z g ) , p t ( T ′ | z t , z g ) are trained jointly, with an additional loss function L Q that regularizes f ϕ .
What do the terms in the contrastive loss-function L Q mean: The loss function L Q ( s t , s g ) =
l hinge ( d ϕ ( s t , s g ) − d p ) 1 d Q ( s t ,s g ) ≤ c Q + l hinge ( d p − d ϕ ( s t , s g ))
1 d Q ( s t ,s g ) ≥ c Q consists of two penalty terms l hinge ( d ϕ ( s t , s g ) − d p ) (i.e., only active when d ϕ ≥ d p ) and l hinge ( d p − d ϕ ( s t , s g ))
(i.e., only active when d ϕ ≤ d p ).
These penalty terms are gated through two complementary indicator functionsd Q ( s t ,s g ) ≤ c Q and 1 d Q ( s t ,s g ) ≥ c Q .
This essentially means that for L Q to be zero, d ϕ ≤ d p should hold (i.e., perceptual representations are close) if and only if d Q ( s t , s g ) ≤ c Q holds (i.e., states are physically close).
The reason for employing such a conservative switching mechanism with a hinge loss in L Q (i.e., rather than a continuous penalty term as in [24, 60]) is because Q-value estimates are quite inaccurate (especially when s c and s g are far apart), and their exact value is generally unreliable (i.e., they can indicate whether two-states are close sufficiently well, but cannot robustly answer how close they are).
To pick the hyperparameter c Q , we compute the average Q-value between states in the replay buffer that were observed to be within one-step proximity, and use a fraction of this value to as a conservative estimate.
While conceptually the choice for the hyperparameter d p is arbitrary, we heuristically pick it by examining the average d ϕ distance between subsequent states in the replay buffer, obtained from a preliminary f ϕ backbone trained without L Q .
2. Perceptual Memory Retrieval (PMR)
How do we solve the optimization problem in equation 1 from the main paper: Our experiments use −R ( τ ) =
len ( τ ) .
We first compute the perceptual representations z for all states in the replay buffer, and stack them into a tensor block of shape ( dataset _ size, D ) .
Given s c and s g , we search this tensor with vectorized masking operations to retrieve a set of neighboring states N ( s c , d p ) and
N ( s g , d p ) (i.e., sets of states within a perceptual distance threshold d p of s c and s g ), to address cons.4.
We sort the resulting pairs of states ( s i , s j ) ∈
N ( s c , d p ) ×
N ( s g , d p ) in terms of j − i , and filter these pairs using cons.5.
We then pick the first (i.e., closest) pair, and return all the states with indices between i, j from the replay buffer as the resulting trajectory τ M ( s c ,s g ) .
The important thing to emphasize about all of these operations is that they can be trivially vectorized, and therefore the optimization problem in eq.3-5 can be solved in less time than a forward pass of f ϕ .
What do the retrieved trajectories τ M ( s c ,s g ) look like: As the perceptual distance threshold for d ϕ increases, the physical radii spanned by the nearest neighbor sets N ( s c , d p ) and N ( s g , d p ) increase, as shown in Fig.11.
As a result, constraint 4 in the PMR equation gets looser, more trajectories satisfy constraints 4 and 5 (because their start and end points are allowed to deviate further from the query pair s c , s g ), and therefore the optimization in equation 3 returns a shorter trajectory
τ M ( s c ,s g ) .
1: V ← { SampleFree i } i =1 ,...,num _ vertices ; E ← ∅ ▷ Initialize vertices and edges 2: for each s i ∈ V do 3:
U ← Near ( V, s i , r ) \ { s i } 4: for each s j ∈ U do ▷ Draw lines as edges 5: if CollisionFree ( s i , s j ) then E ← E ∪ { ( s i , s j ) , ( s j , s i ) } return G =
( V, E )
1: Input: s c , s g , G =
( V, E ) , R ( τ ) , f ϕ M 2: for each s i ∈ V do ▷ Insert s c and s g into the PRM graph 3: if CollisionFree ( s c , s i ) then 4:
E ← E ∪ { ( s c , s i ) , ( s i , s c ) } 5: if CollisionFree ( s i , s g ) then 6:
E ← E ∪ { ( s i , s g ) , ( s g , s i ) } return { s j } ← ShortestPath ( s c , s g , G ) ▷ Shortest path through graph search
1: Input: f ϕ , M 2: V ← { SampleFree i } i =1 ,...,num _ vertices ; E ← ∅ ▷ Initialize vertices and edges 3: for each s i ∈ V do 4:
U ← Near ( V, s i , r ) \ { s i } 5: for each s j ∈ U do ▷ Place PMR trajectories in edges 6:
E ← E ∪ { ( s i , s j ) : τ edge =
τ M ( s i ,s j ) , d edge = −R ( τ M ( s i ,s j ) ) } return G =
( V, E )
How does R-PRM work in detail: Alg.3, 4 give step-by-step descriptions for the classical PRM algorithm (adapted from [27]), while Alg.5, 6 describe our new definitions for R-PRM.
It can be seen that there are two main differences:
• In R-PRM, whenever an edge is created, a trajectory τ M ( s c ,s i ) is retrieved through perceptual memory retrieval and stored in a field τ edge , while its reward −R ( τ M ( s c ,s i ) ) is stored in a different field d edge .
In PRM the length and cost of a line segment are the same (i
e., euclidian distance), whereas in R-PRM the length of a trajectory len ( τ M ( s c ,s i ) ) and its reward −R ( τ M ( s c ,s i ) ) are decoupled.
This means that a shortest path query in R-PRM returns a sequence of nodes and edges that optimize the reward function R .
An additional step in R-PRM is that at the end of the shortest-path query, all trajectories τ edge =
τ M ( s i − 1 ,s i ) stored in the returned edges are concatenated into a single trajectory τ M * ( s c ,s g ) =
τ stitched .
How does R-RRT work in detail: Alg.7 gives a step-by-step description for the classical RRT algorithm (adapted from [27]), while Alg.8 describes our new definition for R-RRT.
In addition to the two previous differences between PRM and R-PRM, there is one additional difference between RRT and R-RRT.
In RRT, there is a steering sub-routine that draws a line segment of length r starting from s nearest and extending towards s rand , to create a new vertex s new .
In R-RRT, this is replaced by retrieving a trajectory τ M ( s nearest ,s rand ) starting from s nearest and ending at s rand , and its r ’th state is used to create the new vertex s new . How does R-RRT* work in detail: Alg.9 gives a step-by-step description for the classical RRT* algorithm (adapted from [27]), while Alg.10 describes our new definition for R-RRT*. R-RRT* almost exactly maintains the tree rewiring machinery employed in RRT*, the only difference being
1: Input: s c , s g , G =
( V, E ) , R ( τ ) , f ϕ , M 2: for each s i ∈ V do ▷ Insert s c and s g into the PRM graph 3: if len ( τ M ( s c ,s i ) )
≤ r then ▷ Place PMR trajectories in edges 4:
E ← E ∪ { ( s c , s i ) : τ edge =
τ M ( s c ,s i ) , d edge = −R ( τ M ( s c ,s i ) ) } 5: if len ( τ M ( s i ,s g ) ) ≤ r
then 6:
E ← E ∪ { ( s i , s g ) : τ edge =
τ M ( s i ,s g ) , d edge = −R ( τ M ( s i ,s g ) ) } 7: τ stitched ← ∅ 8: { s j } ← ShortestPath ( s c , s g , G, R ( τ ))
▷ Trajectory stitching by dynamic programming 9: for 0 < i < |{ s j }| do ▷ Concatenate PMR trajectories along the shortest path 10: τ stitched ← τ stitched ◦ τ M ( s i − 1 ,s i ) return τ M * ( s c ,s g )
=
τ stitched
1: V ← { s init } ; E ← ∅ ▷ Initialize vertices and edges 2: for i = 1 , ..., n do
: s rand ← SampleFree i ▷ Sample random vertex 4: s nearest ← Nearest ( V, s rand )
▷ Find the nearest vertex in V 5: s new ← Steer ( s nearest , s rand , r )
▷ Draw a line segment of length r 6:
if CollisionFree ( s nearest , s new ) then 7:
V ← V ∪ { s new } ; E ← E ∪ { ( s nearest , s new ) , ( s new , s nearest ) } return G =
( V, E ) that the line costs (i.e., euclidian distance) are replaced with −R ( τ M ( s i ,s j ) )
(i.e., in addition to the previous three differences from R-PRM and R-RRT).
What does optimizing the Bellman error on the roadmap mean: Dynamic programming based graphsearch algorithms like Dijkstra or (discrete) value-iteration generally employ a cost caching mechanism to iteratively update cost-to-come values, and these updates reduce Bellman error (i.e., the difference between the cost-to-come values before and after the update) during forward-search [33].
R-PRM uses Dijkstra for shortest-path search, while R-RRT* employs the same dynamic programming based tree-rewiring mechanism as the original RRT*, therefore both algorithms are optimizing the Bellman error between the vertices of their graphs through dynamic programming.
What does restitching transitions at arbitrary resolution mean: The retrospective-planning algorithms R-PRM, R-RRT, R-RRT* are sampling based.
This means that if two consequent states s t and s t +1 are retrieved during their SampleFree i routines, then these algorithms will set an edge using τ M ( s t ,s t +1 ) , which is simply the single transition ( s t , a t , s t +1 ) .
Therefore, given enough samples, these algorithms can restitch trajectories down to the level of individual transitions.
2. PALMER: Closing the Perception-Action Loop to Optimize Memory Contents
What does optimizing memory contents mean: A replay buffer M is a collection of trajectories.
Optimizing its contents means adding trajectories to M that achieve higher total reward.
How does the perception-action loop in PALMER work in detail: Alg.11 gives a step-by-step description of the overall perception-action loop implemented in PALMER (where experiential planning refers to any of the algorithms R-PRM, R-RRT, R-RRT*).
What PALMER does is essentially bridging any auxiliary exploration method with any auxiliary exploitation method , by reorganizing exploration experience in M into τ M * ( s c ,s g ) that can be used for exploitation.
The particular way in which the trajectories τ
M * ( s c ,s g ) can be executed for exploitation has a great deal of flexibility, for example: all actions in τ M * ( s c ,s g ) can be executed sequentially in an open-loop manner, the first actions of τ
M * ( s c ,s g ) generated at each timestep can be executed in a model predictive control (MPC) manner, states in τ
M * ( s c ,s g ) can be tracked by an auxiliary local feedback controller, or the entirety of τ
M * ( s c ,s g ) can be used to initialize a separate trajectory optimization method.
Algorithm
1: V ← { s init } ; E ← ∅ ▷ Initialize vertices and edges 2: for i = 1 , ..., n do
: s rand ← SampleFree i ▷ Sample random vertex 4: s nearest ← Nearest ( V, s rand )
▷ Find the nearest vertex in V 5: s new ← τ M ( s nearest ,s rand ) ,r ▷ Get the r ’th state in τ M ( s nearest ,s rand ) 6: if len ( τ M ( s nearest ,s new ) )
≤ r then 7:
V ← V ∪ { s new } 8:
E ← E ∪ { ( s nearest , s new ) : τ edge =
τ M ( s nearest ,s new ) , 9: d edge = −R ( τ M ( s nearest ,s new ) ) } return G =
( V, E )
1: V ← { s } ; E ∅ ▷ Initialize vertices and edges 2: for i = 1 , ..., n do
: s rand ← SampleFree i ▷ Sample random vertex 4: s nearest ← Nearest ( V, s rand )
▷ Find the nearest vertex in V 5: s new ← Steer ( s nearest , s rand , r )
▷ Draw a line segment of length r 6: 7: if CollisionFree ( s nearest , s new ) then 8:
S near ← Near ( V, s new , r ) 9:
V ← V ∪ { s new } 10: s min ← s nearest 11: c min ← Cost ( s nearest )
+ Cost ( Line ( s nearest , s new ))
12: 13: for each s near ∈ X near do ▷ Connect along a minimum-cost path 14: c near ← Cost ( s near ) + Cost ( Line ( s near , s new )) 15: if CollisionFree ( s near , s new ) and c near ≤ c min then 16: s min
← s near ; c min ← c near 17:
E ← E ∪ { ( s min , s new ) } 18: 19: for each s near ∈ X near do ▷ Rewire the tree 20: c near,new ← Cost ( s new ) + Cost ( Line ( s new , s near )) 21:
if CollisionFree ( s new , s near ) and c near,new ≤ c near then 22: s parent ← Parent ( s near ) 23:
E ← ( E \{ ( s parent , s near ) } ∪ { s new , s near } ) return G =
( V, E )
Related Work
What is the main reason why memory-based reasoning over actually-observed transitions is necessary?
Why are methods like SPTM or SoRB that solely rely on learning-based distance estimates are inherently prone to false predictions: By definition, states that are far apart from each other in-terms of physical reachability are rarely observed together in an experiential learning framework (e.g., within the same RL episode, or within close by time-steps during random exploration).
Therefore, for any learning-based prediction model that is conditioned on current-goal state pairs (e.g.,
Q ( s t , a, s g ) , π inv ( a ′ | s t , s g ) , p t ( T ′ | s t , s g ) ), if it is trained solely on the observed distribution of experiential data, far apart states will be out of distribution (i.e., they have a low probability of being sampled from the experiential data distribution, therefore they are underrepresented in the replay buffer).
This means that predictions for such far apart states will inevitably be inaccurate, unless a hard-negative sampling mechanism is implemented to explicitly push their reachability-estimates lower.
The problem with this is that there is no inherent signal solely contained in perceptual input (e.g., images) that can guide the resampling process in a way that is generally applicable to all tasks.
1: V ← { s init } ; E ← ∅ ▷ Initialize vertices and edges 2: for i = 1 , ..., n do
: s rand ← SampleFree i ▷ Sample random vertex 4: s nearest ← Nearest ( V, s rand )
▷ Find the nearest vertex in V 5: s new ← τ M ( s nearest ,s rand ) ,r ▷ Get the r ’th state in τ M ( s nearest ,s rand ) 6: 7: if len ( τ M ( s nearest ,s new ) )
≤ r then 8:
S near ← Near ( V, s new , r ) 9:
V ← V ∪ { s new } 10: s min ← s nearest 11: c min ← Cost ( s nearest ) + −R ( τ M ( s nearest ,s new ) ) 12: 13: for each s near ∈ X near do ▷ Connect along a minimum-cost path 14: c near ← Cost ( s near ) + −R ( τ M ( s near ,s new ) ) 15: if len ( τ M ( s near ,s new ) ) ≤ r and c near ≤ c min then 16: s min
← s near ; c min ← c near 17:
E ← E ∪ { ( s min , s new ) : τ edge =
τ M ( s min ,s new ) , d edge = −R ( τ M ( s min ,s new ) ) } 18: 19: for each s near ∈ X near do ▷ Rewire the tree 20: c near,new ← Cost ( s new ) + −R ( τ M ( s new ,s near ) ) 21: if len ( τ M ( s new ,s near ) ) ≤ r and c near,new ≤ c near then 22: s parent ← Parent ( s near ) 23:
E ← ( E \{ ( s parent , s near ) } ∪ { s new , s near } ) return G =
( V, E )
For example, [59] employs a hard-negative sampling mechanism for a manipulation task using joint pose labels for guidance, but such a mechanism has two bottlenecks: i) it is specific to their particular manipulation task, ii) it assumes auxiliary labels.
Similarly, the temporally consistent localization mechanism used in SPTM is essentially a heuristic fix specific to navigation.
For the case of SoRB, there is no inherent mechanism in distributional RL that addresses this out-of-distribution issue either.
While employing an ensemble of Q-functions could potentially capture the epistemic uncertainty for out-of-distribution pairs to directly address this problem, we used an ensemble of Q-functions in our implementation of SoRB and empirically observed that it was insufficient.
A similar conclusion can be drawn from the Fig.8 of the original SoRB paper [12], as the bulk of the performance increase appears to be due to distributional RL, and ensembles only provide a moderate benefit.
All of these considerations highlight the importance of memory as a robustification mechanism.
To summarize the discussions from above, there are two main reasons that cause false reachability predictions: i) there is no robust and general signal solely contained in the isolated instances of perceptual input ( s c , s g )
(i.e., without the trajectory of states in between that connect them) that can identify whether two states are physically far apart, ii) both physically close and far apart states can occur with a long temporal distance in between.
Therefore, an agent needs to rely on memory: in order to identify whether two states are close or not, it should try to remember if it ever actually observed those two states close-by in a segment of past experience.
What are the details for our implementations of SoRB and SPTM:
Tha main difference of our SPTM implementation is that it doesn’t employ temporally consistent localization and adaptive waypoint selection.
The main differences of our SoRB implementation are: i) we use an ensemble of Qfunctions, but they are trained with DDQN rather than distributional RL, ii) we train the Q-function on offlline random-walk data, rather than an online episodic training setup with resets and a reward oracle as employed in the original paper.
We acknowledge and emphasize that for SoRB, these differences are the most likely reason for the lower performance level we observed in our evaluations compared to the original paper, as they inevitably reduce the accuracy of Q-values.
We however note that our method also employs the same Q-values, and generally these implementation differences in the baselines were chosen to facilitate a clear understanding of our approach without confounders,
1: Input: R ( τ ) 2: f ϕ .init () , Q.init () , M ← ∅ ▷ Initialize policy parameters 3: while t ≤ max _ timestep do 4: while i ≤ num _ exploration _ steps do ▷ Exploration 5: i) Using [ any suitable method ]: explore the environment to obtain an exploration trajectory τ new 6: ii) Using [ τ new ]: update M ▷ Memory expansion 7: while i ≤ num _ updates do 8:
Using [ M ]: update Q ( s t , a t , s g )
▷ Train value function 9:
Using [ M and Q ( s t , a t , s g ) ]: update f ϕ ( s t , s g )
▷ Train perception model 10: while i ≤ num _ exploitation _ steps do ▷ Exploitation 11: i) Using [ M ]: sample a random goal s g ∼ M 12: ii) Using [ f ϕ ( s c , s g ) , R ( τ ) , and experiential planning ]: generate τ M * ( s c ,s g ) 13: iii) Using [ any suitable method ]: execute τ M * ( s c ,s g ) to obtain a real trajectory τ real 14: iv) Using [ τ real ]: update M ▷ Memory optimization because: i) they do not directly address the root cause of the false prediction problem (as discussed above), ii) one of the main benefits of our method is that it operates over arbitrary offline data without any resets or reward oracles (and it uses DDQN to train the related Q-function).
Experiments
[Figure]
What are the details for the experimental setup in VizDoom: As shown in Fig.12, states solely consist of four images I North/East/South/West that form a panorama (i.e., 4 × 3 × 160 × 120 dimensions), and actions move the agent North/South/East/West by a fixed distance ∆ .
The geodesic distances scale approximately by a factor of × 3 compared to euclidian distances (e.g., If a goal has an euclidian distance of 14∆ , it takes approximately 42 timesteps for an optimal policy to reach it).
The map contains many long-thin column-like obstructions (e.g., torches, pillars, trees), as we found that image-based navigation policies are prone to getting stuck in such obstacles.
These obstacles have dynamically changing appearances (e.g., flickering flames on torches, glowing lights on pillars), and can completely block the field of view of the agent after a collision (as shown in I East in Fig.12).
The replay buffer M consists of 300k images obtained from a uniform random walk exploring the map in a single continuous sequence of actions, without resets and rewards.
Validating Perceptual Memory Retrieval
(PMR)
What is the exact evaluation process that produced Fig.4 in the main paper:
For every integer value n ∈ [0 , 14] , we randomly sample 1000 pairs of start and goal-states in a way that the euclidian distance between them lies within n × ∆ and ( n + 1) × ∆ through rejection sampling, and a policy is considered successful if it can get within ∆ proximity of the goal-state.
Robust Distances
What is the exact evaluation process for the right panel of Fig.5 in the main paper: To produce the roadmap visualizaitons, we randomly sample 250 states from the replay buffer and set the edges between them by thresholding the distance estimates from all methods.
Thresholds were calibrated individually and by hand for each baseline, by picking the threshold with the lowest number of false edges until a further reduction in the threshold resulted in splitting the roadmap into a large number of isolated subgraphs (i.e., therefore making it impossible to use it for global planning).
Experiential Planning Algorithms
Why does the policy π M ∗ use R-PRM for experiential planning: PRM and R-PRM are multi-query methods [27], meaning that the full roadmap only needs to be constructed once.
For any query pair of current-goal states ( s c , s g ) , the same roadmap can be reutilized by inserting ( s c , s g ) in the roadmap and performing a shortest path query.
In contrast, RRT and RRT* require recreating a full roadmap for every new query pair.
Since π M ∗ replans at each timestep with a different current state s t , PRM based approaches are computationally much cheaper.
We also note that planning graphs for all methods in this experiment contain 500 vertices.
What are the details for the π mpc baseline:
The π mpc policy uses the p fwd model to obtain simulated rollouts, and uses the p t model to rank those rollouts in terms of how close they get to the goal.
This allows an MPC optimization loop that picks and implements the first action from the most successful simulated rollout.
As previously discussed above, the main bottleneck for SPTM and
SoRB is the difficulty of estimating reachability metrics solely using the two states s c , s g without any consideration of the states in between.
Simulated rollouts in π mpc naturally address this problem by generating and evaluating entire trajectories.
The main bottleneck for π mpc is that the accuracy of state predictions in simulated rollouts from p fwd degrade with the rollout length.
Experiments in Habitat
What are the details for the experimental setup in Habitat: As shown in Fig.13, states solely consist of a single 150 FOV image (i.e., 3 × 256 × 256 dimensions).
There are 3 actions: { turn _ left _ 30 _ deg, turn _ right _ 30 _ deg, move _ forward _ ∆ } .
We run evaluations on two randomly picked apartments: Roxbox, and Annawan.
In both apartments, we collect a replay buffer M that consists of 150k images obtained from a uniform random walk exploring the map in a single continuous sequence of actions, without resets and rewards.
We use only the memory buffer from
Roxbox to train the perception model f ϕ , and use this same model to do perceptual memory retrieval and experiential planning on replay buffers from both apartments.
We have observed that the latent distances from f ϕ generalize well, and can directly allow perceptual memory retrieval and experiential planning without any fine-tuning on the images from the new test apartment.
Why does the agent occasionally take random-looking actions in the habitat navigation trials: This is due to a combination of two main factors.
First, our MPC loop replans from scratch at each timestep using Algorithm.6.
This frequent replanning has a destabilizing effect on the control loop, similar to employing a large derivative action in a PID controller (i.e., a strong anticipatory term causes frequent switches in the actions).
This first factor is exacerbated by the second main factor: the poor performance of argmax a Q ( s t , a, s g ) .
This is most likely due to the difficulty of offline RL training with hindsight relabelling over random-walk data of only 150k timesteps obtained with a much more challenging non-cartesian action space { turn _ left _ 30 _ deg, turn _ right _ 30 _ deg, move _ forward _ ∆ } .
The restitched trajectories τ M * ( s t ,s g ) ,s, 1 ) produced by R-PRM at each time-step are converted to actions by following their first state τ M * ( s t ,s g ) ,s, 1 using argmax a Q ( s t , a, τ M * ( s t ,s g ) ,s, 1 ) , and inac- curate Q-values occasionally cause random-looking actions.
There are multiple ways to counter this.
The most direct way is to train a better Q-function.
Our current Q-function is trained in a particularly challenging setting, as: i) it is trained on entirely offline data with hindsight goal relabelling, ii) this data is collected through uniformly random actions, and iii) it consists of only 150k environment steps.
A second way is to instead counter the large derivative action by reducing the replanning rate and introducing a momentum mechanism to the controller.
For example, we can replan through R-PRM only every n ’th timestep (i.e., hence reducing the replanning rate), and act according to argmax a Q ( s t , a, τ M * ( s t ,s g ) ,s,n ) for the timesteps inbetween (i.e., hence introducing momentum to the controls).
We couldn’t get this fix to work well, because the Q-values are only accurate up to states that are ∼ 2 × ∆ distance away (hence acting according to argmax a Q ( s t , a, τ M * ( s t ,s g ) ,s,n ) isn’t possible for n ≥ 2 in our case).
We note however that our method can still navigate from any point to any point in a challenging 3D reconstruction of a real-world apartment in Habitat using poor Q-value estimates, highlighting the robustness introduced by memory-based planning.
Discussion and Future Directions
How is PALMER related to the "Options Framework (Sutton et al.)
" and "Skill-Chaining (Konidaris et al.)
": The idea of restitching (i.e., chaining) transition sequences from a replay buffer has direct connections to the options framework [61] and skill-chaining [62].
Essentially, PALMER can be thought of as a framework for converting every possible sequence of transitions τ ∈ M in memory into an option o = { π o , I o , β o } , where the option policy π o is implemented by simply executing all the actions in τ in an open-loop manner, the initiation set is I o = { s ∈ S :
d ϕ ( s, τ s, 0 ) ≤ d p ) } , and the termination condition is β o = { s ∈ S :
d ϕ ( s, τ s, − 1 ) ≤ d p ) } .
Therefore, PALMER can be interpreted as a skill-chaining algorithm that converts unstructured transitions in a replay buffer into a set of executable options to be chained.
How is PALMER related to "LQR-Trees (Tedrake et al.)
": A central idea in PALMER is repurposing the edge creation subroutines of sampling-based planning algorithms so that whenever an edge is created some additional processing is done to connect the endpoints (i.e., particularly, perceptual memory retrieval in our case).
This approach is directly inspired by the method of LQR-Trees [63], which instead creates a trajectory stabilizing LQR controller to connect the endpoints of each edge.
This results in a roadmap of local controllers, rather then a roadmap of memories as in PALMER.