H YPERBOLIC D EEP R EINFORCEMENT L EARNING
Twitter
A BSTRACT
We propose a new class of deep reinforcement learning (RL) algorithms that model latent representations in hyperbolic space.
Sequential decision-making requires reasoning about the possible future consequences of current behavior.
Consequently, capturing the relationship between key evolving features for a given task is conducive to recovering effective policies.
To this end, hyperbolic geometry provides deep RL models with a natural basis to precisely encode this inherently hierarchical information.
However, applying existing methodologies from the hyperbolic deep learning literature leads to fatal optimization instabilities due to the non-stationarity and variance characterizing RL gradient estimators.
Hence, we design a new general method that counteracts such optimization challenges and enables stable end-to-end learning with deep hyperbolic representations.
We empirically validate our framework by applying it to popular on-policy and offpolicy RL algorithms on the Procgen and Atari 100K benchmarks, attaining near universal performance and generalization beneﬁts.
Given its natural ﬁt, we hope future RL research will consider hyperbolic representations as a standard tool.
I NTRODUCTION
Reinforcement Learning (RL) achieved notable milestones in several game-playing and robotics applications (Mnih et al., 2013; Vinyals et al., 2019;
Kalashnikov et al., 2018; OpenAI et al., 2019; Lee et al., 2021).
However, all these recent advances relied on large amounts of data and domain-speciﬁc practices, restricting their applicability in many important real-world contexts (Dulac-Arnold et al., 2019).
We argue that these challenges are symptomatic of current deep RL models lacking a proper prior to efﬁciently learn generalizable features for control (Kirk et al., 2021).
We propose to tackle this issue by introducing hyperbolic geometry to RL, as a new inductive bias for representation learning.
The evolution of the state in a Markov decision process can be conceptualized as a tree, with the policy and dynamics determining the possible branches.
Analogously, the same hierarchical evolution often applies to the most signiﬁcant features required for decision-making (e.g., presence of bricks, location of paddle/ball in Fig. 1).
These relationships tend to hold beyond individual trajectories, making hierarchy a natural basis to encode information for RL (Flet-Berliac, 2019).
Consequently, we hypothesize that deep RL models should prioritize encoding precisely hierarchically-structured features to facilitate learning effective and generalizable policies.
In contrast, we note that nonevolving features, such as the aesthetic properties of elements in the environment, are often linked with spurious correlations, hindering generalization to new states (Song et al., 2019).
Similarly, human cognition also appears to learn representations of actions and elements of the environment by focusing on their underlying hierarchical relationship (Barker & Wright, 1955; Zhou et al., 2018).
Hyperbolic geometry (Beltrami, 1868; Cannon et al., 1997) provides a natural choice to efﬁciently encode hierarchically-structured features.
A deﬁning property of hyperbolic space is exponential volume growth , which enables the embedding of tree-like hierarchical data with low distortion using only a few dimensions (Sarkar, 2011).
In contrast, the volume of Euclidean spaces only grows polynomially, requiring high dimensionality to precisely embed tree structures (Matouˇsek, 1990), potentially leading to higher complexity, more parameters, and overﬁtting.
We analyze the properties of learned RL representations using a measure based on the  -hyperbolicity (Gromov, 1987), quantifying how close an arbitrary metric space is to a hyperbolic one.
In line with our intuition, we show that performance improvements of RL algorithms correlate with the increasing hyperbolicity of the discrete space spanned by their latent representations.
This result validates the importance of appropriately encoding hierarchical information, suggesting that the inductive bias provided by employing hyperbolic representations would facilitate recovering effective solutions.
Hyperbolic geometry has recently been exploited in other areas of machine learning showing substantial performance and efﬁciency beneﬁts for learning representations of hierarchical and graph data (Nickel & Kiela, 2017; Chamberlain et al., 2017).
Recent contributions further extended tools from modern deep learning to work in hyperbolic space (Ganea et al., 2018;
Shimizu et al., 2020), validating their effectiveness in both supervised and unsupervised learning tasks (Khrulkov et al., 2020; Nagano et al., 2019; Mathieu et al., 2019).
However, most of these approaches showed clear improvements on smaller-scale problems that failed to hold when scaling to higher-dimensional data and representations.
Many of these shortcomings are tied to the practical challenges of optimizing hyperbolic and Euclidean parameters end-to-end (Guo et al., 2022).
In RL, We show the nonstationarity and high-variance characterizing common gradient estimators exacerbates these issues, making a naive incorporation of existing hyperbolic layers yield underwhelming results.
In this work, we overcome the aforementioned challenges and effectively train deep RL algorithms with latent hyperbolic representations end-to-end.
In particular, we design spectrally-regularized hyperbolic mappings (S-RYM), a simple recipe combining scaling and spectral normalization (Miyato et al., 2018) that stabilizes the learned hyperbolic representations and enables their seamless integration with deep RL.
We use S-RYM to build hyperbolic versions of both on-policy (Schulman et al., 2017) and off-policy algorithms (Hessel et al., 2018), and evaluate on both Procgen (Cobbe et al., 2020) and Atari 100K benchmarks (Bellemare et al., 2013).
We show that our framework attains near universal performance and generalization improvements over established Euclidean baselines, making even general algorithms competitive with highly-tuned SotA baselines.
We hope our work will set a new standard and be the ﬁrst of many incorporating hyperbolic representations with RL.
To this end, we share our implementation at sites.google.com/view/hyperbolic-rl .
P RELIMINARIES
In this section, we introduce the main deﬁnitions required for the remainder of the paper.
We refer to App.
A and (Cannon et al., 1997) for further details about RL and hyperbolic space, respectively.
2. R EINFORCEMENT LEARNING
The RL problem setting is traditionally described as a Markov Decision Process (MDP), deﬁned by the tuple ( S, A, P, p 0 , r,  ) .
At each timestep t , an agent interacts with the environment, observing some state from the state space s 2 S , executing some action from its action space a 2 A , and receiving some reward according to its reward function r : S ⇥
A 7! R .
The transition dynamics P : S ⇥
A ⇥ S 7!
R and initial state distribution p 0 : S 7!
R determine the evolution of the environment’s state while the discount factor  2 [0 , 1) quantiﬁes the agent’s preference for earlier rewards.
Agent behavior in RL can be represented by a parameterized distribution function ⇡ ✓ , whose sequential interaction with the environment yields some trajectory ⌧ = ( s 0 , a 0 , s 1 , a 1 , ..., s T , a T ) .
The agent’s objective is to learn a policy maximizing its expected discounted sum of rewards over trajectories:
[Equation]
We differentiate two main classes of RL algorithms with very different optimization procedures based on their different usage of the collected data.
On-policy algorithms collect a new set of trajectories with the latest policy for each training iteration, discarding old data.
In contrast, off-policy algorithms maintain a large replay buffer of past experiences and use it for learning useful quantities about the environment, such as world models and value functions.
Two notable instances from each class are Proximal Policy Optimization (PPO) (Schulman et al., 2017) and Rainbow DQN (Hessel et al., 2018), upon which many recent advances have been built upon.
2. M ACHINE L EARNING IN H YPERBOLIC S PACES
A hyperbolic space H n is an n -dimensional Riemannian manifold with constant negative sectional curvature  c .
Beltrami (1868) showed the equiconsistency of hyperbolic and Euclidean geometry using a model named after its re-discoverer, the Poincar´e ball model .
This model equips an ndimensional open ball B n = { x 2 R n : c k x k < 1 } of radius 1 / p c with a conformal metric of the form G x =  2 x I , where  x = 2 1  c k x k 2 is the conformal factor (we will omit the dependence on the curvature  c in our deﬁnitions for notation brevity).
The geodesic (shortest path) between two points in this metric is a circular arc perpendicular to the boundary with the length given by:
[Equation]
From these characteristics, hyperbolic spaces can be viewed as a continuous analog of trees.
In particular, the volume of a ball on H n grows exponentially w.r.t.
its radius.
This property mirrors the exponential node growth in trees with constant branching factors.
Visually, this makes geodesics between distinct points pass through some midpoint with lower magnitude, analogously to how tree geodesics between nodes (deﬁned as the shortest path in their graph) must cross their closest shared parent (Fig. 2).
Key operations for learning.
On a Riemannian manifold, the exponential map exp x ( v ) outputs a unit step along a geodesic starting from point x in the direction of an input velocity v .
It thus allows locally treating H n as Euclidean space.
We use the exponential map from the origin of the
Poincar´e ball to map Euclidean input vectors v into H n ,
[Equation]
Following Ganea et al.
, we consider the framework of gyrovector spaces (Ungar, 2008) to extend common vector operations to non-Euclidean geometries, and in particular H n .
The most basic such generalized operation is the Mobius addition  of two vectors,
[Equation]
Next, consider a Euclidean afﬁne transformation f ( x ) =
h x , w i + b used in typical neural network layers.
We can rewrite this transformation as f ( x ) =
h x  p , w i and interpret w , p 2 R d as the normal and shift parameters of a hyperplane H = { y 2 R d : h y  p , w i = 0 } (Lebanon & Lafferty, 2004).
This allows us to further rewrite f ( x ) in terms of the signed distance to the hyperplane H , effectively acting as a weighted ‘decision boundary’:
[Equation]
This formulation allows to extend afﬁne transformations to the Poincar ´ e ball by considering the signed distance from a gyroplane in B d (generalized hyperplane) H = { y 2 B [Equation] : h y  p , w
[Equation]
=
[Equation] [Equation]
In line with recent use of hyperbolic geometry in supervised (Khrulkov et al., 2020; Guo et al., 2022) and unsupervised (Nagano et al., 2019; Mathieu et al., 2019) deep learning, we employ these operations to parameterize hybrid neural networks: we ﬁrst process the input data x with standard layers to produce some Euclidean velocity vectors x E = f E ( x ) .
Then, we obtain our hyperbolic representations by applying the exponential map x H = exp 0 ( x E ) .
Finally, we employ afﬁne transformations { f i } of the form 6 to output the set of policy and value scalars with f H ( x H ) =
{ f i ( x H ) } .
H YPERBOLIC REPRESENTATIONS FOR REINFORCEMENT LEARNING
In this section, we base our empirical RL analysis on Procgen (Cobbe et al., 2020).
This benchmark consists of 16 visual environments, with procedurally-generated random levels.
Following common practice, we train agents using exclusively the ﬁrst 200 levels of each environment and evaluate on the full distribution of levels to assess agent performance and generalization.
3. T HE INHERENT HYPERBOLICITY OF DEEP RL
Key quantities for each state, such as the value and the policy, are naturally related to its possible successors.
In contrast, other ﬁxed, non-hierarchical information about the environment such as its general appearance, can often be safely ignored.
This divide becomes particularly relevant when considering the problem of RL generalization.
For instance, Raileanu & Fergus (2021) found that agents’ can overﬁt to spurious correlations between the value and non-hierarchical features (e.g., background color) in the observed states.
Hence, we hypothesize that effective representations should encode features directly related to the hierarchical state relationships of MDPs.
-hyperbolicity.
We analyze the representation spaces learned by RL agents, testing whether they preserve and reﬂect this hierarchical structure.
We use the  -hyperbolicity of a metric space ( X, d ) (Gromov, 1987; Bonk & Schramm, 2011), which we formally describe in App.
A.2.
For our usecase, X is  -hyperbolic if every possible geodesic triangle 4 xyz 2 X is  -slim.
This means that for every point on any side of 4 xyz there exists some point on one of the other sides whose distance is at most  .
In trees, every point belongs to at least two of its sides yielding  =
(Figure 3).
Thus, we can interpret  -hyperbolicity as measuring the deviation of a given metric from an exact tree metric.
The representations learned by an RL agent from encoding the collected states span some ﬁnite subset of Euclidean space x
E 2 X E ⇢ R n , yielding a discrete metric space X E .
To test our hypothesis, we compute the  -hyperbolicity of X E and analyze how it relates to agent performance .
Similarly to (Khrulkov et al., 2020), we compute  using the efﬁcient algorithm proposed by Fournier et al. (2015).
To account for the scale of the representations, we normalize  by diam ( X E ) , yielding a relative hyperbolicity measure  rel = 2  / diam ( X E ) (Borassi et al., 2015), which can span values between 0 (hyperbolic hierarchical tree-like structure) and 1 (perfectly non-hyperbolic spaces).
Results.
We train an agent with PPO (Schulman et al., 2017) on four Procgen environments, encoding states from the latest rollouts using the representations before the ﬁnal linear policy and value heads, x E = f E ( s ) .
Hence, we estimate  rel from the space spanned by these latent encodings as training progresses.
As shown in Figure 4,  rel quickly drops to low values ( 0 .
22  0 . 28 ) in the ﬁrst training iterations, reﬂecting the largest relative improvements in agent performance.
Subsequently, in the fruitbot and starpilot environments,  rel further decreases throughout training as the agent recovers high performance with a low generalization gap between the training and test distribution of levels.
Instead, in bigﬁsh and dodgeball ,  rel begins to increase again after the initial drop, suggesting that the latent representation space starts losing its hierarchical structure.
Correspondingly, the agent starts overﬁtting as test levels performance stagnates while the generalization gap with the training levels performance keeps increasing.
These results validate our hypothesis, empirically showing the importance of encoding hierarchical features for recovering effective solutions.
Fur- thermore, they suggest that PPO’s poor generalization in some environments is due to the observed tendency of the Euclidean latent space to encode spurious features that hinder its hyperbolicity.
Motivated by our ﬁndings, we propose employing hyperbolic geometry to model the latent representations of deep RL models.
Representing tree-metrics in Euclidean spaces incurs non-trivial worse-case distortions, growing with the number of nodes at a rate dependent on the dimensionality (Matouˇsek, 1990).
This property suggests that it is not possible to encode complex hierarchies in Euclidean space both efﬁciently and accurately, explaining why some solutions learned by PPO could not maintain their hyperbolicity throughout training.
In contrast, mapping the latent representations to hyperbolic spaces of any dimensionality enables encoding features exhibiting a tree-structured relation over the data with arbitrarily low distortion (Sarkar, 2011).
Hence, hyperbolic latent representations introduce a different inductive bias for modeling the policy and value function, stemming from this inherent efﬁciency of speciﬁcally encoding hierarchical information (Tifrea et al., 2018).
3. O PTIMIZATION CHALLENGES
Naive integration.
We test a simple extension to PPO, mapping the latent representations of states s 2 S before the ﬁnal linear policy and value heads x E = f E ( s ) to the Poincar´e ball with unitary curvature.
As described in Section 2, we perform this with an exponential map to produce x H = exp 10 ( x E ) , replacing the ﬁnal ReLU.
To output the value and policy logits, we then ﬁnally perform a set of afﬁne transformations in hyperbolic space, ⇡ ( s ) , V ( s )
=
f H ( x H ) =
{ f 1 i ( x H ) } | A | i =0 .
We also consider a clipped version of this integration, following the recent stabilization practice from Guo et al.
(2022), which entails clipping the magnitude of the latent representations to not exceed unit norm.
We initialize the weights of the last two linear layers in both implementations to 100 ⇥ smaller values to start training with low magnitude latent representations, which facilitates the network ﬁrst learning appropriate angular layouts (Nickel & Kiela, 2017; Ganea et al., 2018).
Results.
We analyze this naive hyperbolic PPO implementation in Figure 6.
As shown in part (A.1), performance is generally underwhelming, lagging considerably behind the performance of standard PPO.
While applying the clipping strategy yields some improvements, its results are still considerably inferior on the tasks where Euclidean embeddings appear to already recover effective representations (e.g. starpilot ).
In part (A.2) we visualize the negated entropy of the different PPO agents.
PPO’s policy optimization objective includes both a reward maximization term, which requires an auxiliary estimator, and an entropy bonus term that can instead be differentiated exactly and optimized end-to-end.
Its purpose is to push PPO agents to explore if they struggle to optimize performance with the current data.
We note that the Hyperbolic PPO agents take signiﬁcantly longer to reach higher levels of entropy in the initial training phases and are also much slower to reduce their entropy as their performance improves.
These results appear to indicate the presence of optimization challenges stemming from end-to-end RL training with hyperbolic representations.
Therefore, we turn our attention to analyzing the gradients in our hyperbolic models.
In part (B.1), we visualize the magnitude of the gradients both as backpropagated from the ﬁnal representations and to the convolutional encoder.
In part (B.2), we also visualize the variance of the same gradients with respect to the different input states in a minibatch.
We ﬁnd that hyperbolic PPO suffers from a severe exploding gradients problem, with both magnitudes and variances being several orders of magnitude larger than the Euclidean baseline.
Similar instabilities have been documented by much recent literature, as described in App.
B. Yet, in the RL case, common stabilization techniques such as careful initialization and clipping are visibly insufﬁcient, resulting in ineffective learning and inferior agent performance.
3. S TABILIZING HYPERBOLIC REPRESENTATIONS
We attribute the observed optimization challenges from our naive hyperbolic PPO implementation to the high variance and non-stationarity characterizing RL.
Initialization and clipping have been designed for stationary ML applications with ﬁxed dataset and targets.
In these settings, regularizing the initial learning iterations enables the model to ﬁnd appropriate angular layouts of the representations for the underlying ﬁxed loss landscape.
Without appropriate angular layouts, useful representations become very hard to recover due to the highly non-convex spectrum of hyperbolic neural networks, often resulting in failure modes with low performance (Ganea et al., 2018; L´opez & Strube, 2020).
We can intuitively see why this reliance is incompatible with the RL setting, where the trajectory data and loss landscape can change signiﬁcantly throughout training, making early angular layouts inevitably suboptimal.
This is further exacerbated by the high variance gradients already characterizing policy gradient optimization (Sutton & Barto, 2018; Wu et al., 2018) which facilitate entering unstable learning regimes that can lead to our observed failure modes.
Spectral norm.
Another sub-ﬁeld of ML dealing with non-stationarity and brittle optimization is generative modeling with adversarial networks (GANs) (Goodfellow et al., 2014).
In GAN training, the generated data and discriminator’s parameters constantly evolve, making the loss landscape highly non-stationary as in the RL setting.
Furthermore, the adversarial nature of the optimization makes it very brittle to exploding and vanishing gradients instabilities which lead to common failure modes (Arjovsky & Bottou, 2017; Brock et al., 2018).
In this parallel literature, spectral normalization (SN) (Miyato et al., 2018) is a popular stabilization practice whose success made it ubiquitous in modern GAN implementations.
Recent work (Lin et al., 2021) showed that a reason for its surprising effectiveness comes from regulating both the magnitude of the activations and their respective gradients very similarly to LeCun initialization (LeCun et al., 2012).
Furthermore, when applied to the discriminator model, SN’s effects appear to persist throughout training , while initialization strategies tend to only affect the initial iterations.
In fact, they also show that ablating SN from GAN training empirically results in exploding gradients and degraded performance, closely resembling our same observed instabilities.
We provide details about GANs and SN in App.
A.3.
[Table]
[Figure]
S-RYM.
Inspired by these connections, we propose to counteract the optimization challenges in RL and hyperbolic representations with SN.
We make two main changes from its usual application for
GAN regularization.
First, we apply SN only in the Euclidean encoder sub-network ( f E ), leaving the ﬁnal linear transformation in hyperbolic space ( f H ) unregularized since our instabilities appear to occur in the gradients from the hyperbolic representations.
Furthermore, we add a scaling term to preserve stability for different latent representation sizes.
In particular, modeling x E 2 R n by an independent Gaussian, the magnitude of the representations follows some scaled Chi distribution k x E k ⇠  n , which we can reasonably approximate with E [ k x E k ] =
E [  n ] ⇡ p n .
Therefore, we propose to rescale the output of f E by 1 / p n , such that modifying the dimensionality of the representations should not signiﬁcantly affect their magnitude before mapping them H n .
We call this general stabilization recipe spectrally-regularized hyperbolic mappings (S-RYM).
Results.
As shown in Figure 7, integrating S-RYM with our hyperbolic RL agents appears to resolve their optimization challenges and considerably improve the Euclidean baseline’s performance (A).
To validate that these performance beneﬁts are due to the hyperbolic geometry of the latent space, we also compare with another Euclidean ablation making use of SN, which fails to attain any improvement.
Furthermore, S-RYM maintains low gradient magnitudes (B), conﬁrming its effectiveness to stabilize training.
In App.
E.1, we also show that SN and rescaling are both crucial for S-RYM.
Thus, in the next section we evaluate our hyperbolic deep RL framework on a large-scale, analyzing its efﬁcacy and behavior across different benchmarks, RL algorithms, and training conditions.
E XTENSIONS AND EVALUATION
To test the generality of our hyperbolic deep RL framework, in addition to the on-policy PPO we also integrate it with the off-policy Rainbow DQN algorithm (Hessel et al., 2018).
Our implementations use the same parameters and models speciﬁed in prior traditional RL literature, without any additional tuning.
Furthermore, in addition to the full Procgen benchmark (16 envs.)
we also evaluate on the popular Atari 100K benchmark (Bellemare et al., 2013; Kaiser et al., 2020)
(26 envs.), repeating for 5 random seeds.
We provide all details about benchmarks and implementations in App. C.
Generalization on Procgen.
Given the documented representation efﬁciency of hyperbolic space, we evaluate our hyperbolic PPO implementation also reducing the dimensionality of the ﬁnal representation to 32 (see App.
E.2), with relative compute and parameter efﬁciency beneﬁts.
We compare our regularized hyperbolic PPO with using data augmentations, a more traditional way of encoding inductive biases from inducing invariances.
We consider random crop augmentations from their popularity and success in modern RL.
As shown in Table 1, our hyperbolic PPO implementation with S-RYM appears to yield conspicuous performance gains on most of the environments.
At the same time, reducing the size of the representations provides even further beneﬁts with signiﬁcant improvements in 13/16 tasks.
In contrast, applying data augmentations yields much lower and inconsistent gains, even hurting on some tasks where hyperbolic RL provides considerable improvements (e.g. bossﬁght ).
We also ﬁnd that test performance gains do not always correlate with gains on the speciﬁc 200 training levels, yielding a signiﬁcantly reduced generalization gap for the hyperbolic agents.
We perform the same experiment but apply our hyperbolic deep RL framework to Rainbow
DQN with similar results, also obtaining signiﬁcant gains in 13/16 tasks, as reported in App.
D.1.
We also evaluate the robustness of our PPO agents to encoding spurious features, only relevant for the training levels.
In particular, we examine tasks where PPO tends to perform well and consider lowering the training levels from 200 to 100, 50, and 25.
As shown in Figure 8, the performance of
PPO visibly drops at each step halving the number of training levels, suggesting that the Euclidean representations overﬁt and lose their original efﬁcacy.
In contrast, hyperbolic PPO appears much more robust, still surpassing the original PPO results with only 100 training levels in fruitbot and 50 in starpilot .
While also applying data augmentation attenuates the performance drops, its effects appear more limited and inconsistent, providing almost null improvements for starpilot .
[Table]
Sample-efﬁciency on Atari 100K. We focus on the performance of our hyperbolic Rainbow DQN implementation, as the severe data limitations of this benchmark make PPO and other on-policy algorithms impractical.
We show the absolute and relative per-environment performance changes from our hyperbolic RL framework in Figure 9, and provide aggregate statistics in Table 2.
Also on this benchmark, the exact same hyperbolic deep RL framework provides consistent and signiﬁcant beneﬁts.
In particular, we record improvements on 22/
Atari environments over the Euclidean baseline, almost doubling the ﬁnal human normalized score.
Considerations and comparisons.
Our results empirically validate that introducing hyperbolic representations to shape the prior of deep RL models is both remarkably general and effective.
We record almost universal improvements on two fundamentally different RL algorithms, considering both generalizations to new levels from millions of frames (Procgen) and to new experiences from only 2hrs of total play time (Atari 100K).
Furthermore, our hyperbolic RL agents outperform the scores reported in most other recent advances, coming very close to the current SotA algorithms which incorporate different expensive and domain-specialized auxiliary practices (see App. D.2D.3).
Our approach is also orthogonal to many of these advances and appears to provide compatible and complementary beneﬁts (see App. E.3).
Taken together, we believe these factors show the great potential of our hyperbolic framework to become a standard way of parameterizing deep RL models.
Representations interpretation.
We train our hyperbolic PPO agent with only 2-dimensional representations, which still remarkably provide concrete generalization beneﬁts over Euclidean PPO (App. D.4).
Then, we analyze how these representations evolve within trajectories, mapping them on the Poincar´e disk and visualizing the corresponding states.
We observe a recurring cyclical behavior, where the magnitude of the representations monotonically increases within subsets of the trajectory as more obstacles/enemies appear.
We show this in Fig.
10 and Fig.
12, comparing the representations of on-policy states sampled at constant intervals with trajectory deviations from executing random behavior.
We observe the representations form tree-like structures, with the magnitudes in the on-policy states growing in the direction of the Value function’s gyroplane ’s normal.
This intuitively reﬂects that as new elements appear the agent recognizes a larger opportunity for rewards, yet, requiring a ﬁner level of control as distances to the policy gyroplanes will also grow exponentially, reducing entropy.
Instead, following random deviations, magnitudes grow in directions orthogonal to the Value gyroplane’s normal.
This still reﬂects the higher precision required for optimal decision-making, but also the higher uncertainty to obtain future rewards from worse states.
R ELATED WORK
Generalization is a key open problem in RL (Kirk et al., 2021).
End-to-end training of deep models with RL objectives appears has been shown prone to overﬁtting from spurious features only relevant in the observed transitions (Song et al., 2019; Bertran et al., 2020).
To address this, prior work considered different data augmentation strategies (Laskin et al., 2020b; Yarats et al., 2021a; Cobbe et al., 2019), and online adaption methods on top to alleviate engineering burdens (Zhang & Guo, 2021;
Raileanu et al., 2020).
Alternative approaches have been considering problem-speciﬁc properties of the environment (Zhang et al., 2020; Raileanu & Fergus, 2021), auxiliary losses (Laskin et al., 2020a; Schwarzer et al., 2020), and frozen pre-trained layers (Yarats et al., 2021b; Stooke et al., 2021).
Instead, we propose to encode a new inductive bias making use of the geometric properties of hyperbolic space, something orthogonal and likely compatible with most such prior methods.
While hyperbolic representations found recent popularity in machine learning, there have not been notable extensions for deep RL (Peng et al., 2021).
Most relatedly, Tiwari & Prannoy (2018) proposed to produce hyperbolic embeddings of the state space of tabular MDPs to recover options (Sutton et al., 1999).
Yet, they did not use RL for learning, but ﬁxed data and a supervised loss based on the co-occurrence of states, similarly to the original method by Nickel & Kiela (2017).
D ISCUSSION AND FUTURE WORK
In this work, we introduce hyperbolic geometry to deep RL.
We analyze training agents using latent hyperbolic representations and propose spectrally-regularized hyperbolic mappings , a new stabilization strategy that overcomes the observed optimization instabilities.
Hence, we apply our framework to obtain hyperbolic versions of established on-policy and off-policy RL algorithms, which we show substantially outperform their Euclidean counterparts in two popular benchmarks.
We provide numerous results validating that hyperbolic representations provide deep models with a more suitable prior for control, with considerable beneﬁts for generalization and sample-efﬁciency.
We share our implementation to facilitate future RL advances considering hyperbolic space as a new, general tool.