Modern Hopﬁeld Networks for Return Decomposition for Delayed Rewards
Abstract
Delayed rewards, which are separated from their causative actions by irrelevant actions, hamper learning in reinforcement learning (RL).
Especially real world problems often contain such delayed and sparse rewards.
Recently, return decomposition for delayed rewards (RUDDER) employed pattern recognition to remove or reduce delay in rewards, which dramatically simpliﬁes the learning task of the underlying RL method.
RUDDER was realized using a long short-term memory (LSTM).
The LSTM was trained to identify important state-action pair patterns, responsible for the return.
Reward was then redistributed to these important stateaction pairs.
However, training the LSTM is often difﬁcult and requires a large number of episodes.
In this work, we replace the LSTM with the recently proposed continuous modern Hopﬁeld networks (MHN) and introduce Hopﬁeld-RUDDER.
MHN are powerful trainable associative memories with large storage capacity.
They require only few training samples and excel at identifying and recognizing patterns.
We use this property of MHN to identify important state-action pairs that are associated with low or high return episodes and directly redistribute reward to them.
However, in partially observable environments, Hopﬁeld-RUDDER requires additional information about the history of state-action pairs.
Therefore, we evaluate several methods for compressing history and introduce reset-max history , a lightweight history compression using the max-operator in combination with a reset gate.
We experimentally show that Hopﬁeld-RUDDER is able to outperform LSTM-based RUDDER on various 1D environments with small numbers of episodes.
Finally, we show in preliminary experiments that Hopﬁeld-RUDDER scales to highly complex environments with the Minecraft ObtainDiamond task from the MineRL NeurIPS challenge.
Introduction
Recent advances in reinforcement learning (RL) have resulted in impressive models that are capable of surpassing humans in games [30, 18, 27].
However, RL is still waiting for its breakthrough in real world applications, which are often characterized by delayed and sparse rewards [7].
Delayed rewards are given later than their causative action, separated by irrelevant state-action pairs [28].
This delay hampers and slows down learning [2, 21, 17].
Recently, [2] propose return decomposition for delayed rewards (RUDDER) to use pattern recognition to detect the causative actions, which can be used to remove or alleviate these delays.
As a consequence, RUDDER dramatically simpliﬁes and speeds up learning of the underlying RL algorithms.
RUDDER can be realized in different ways, including a long short-term memory (LSTM) [10] in [2] or sequence alignment in [19].
But these realizations of RUDDER come with drawbacks.
LSTMs require a large number of samples, while sequence alignment is difﬁcult to scale to large state-action spaces.
Thus, there is a need for scalable and sample efﬁcient realization of RUDDER.
A key idea of RUDDER is to identify important patterns in the state-action sequence.
This is a task that associative memories, such as the recently introduced continuous modern Hopﬁeld networks (MHN) [23], excel at.
In this work, we introduce Hopﬁeld-RUDDER, a realization of RUDDER based on MHN to identify important state-action pairs.
We show that Hopﬁeld-RUDDER is scalable and can reduce the reward delay even with a small number of samples.
In related work, adapted Transformer [29] architectures, which share characteristics of MHN, for
RL have been proposed.
In [6], RL is reformulated as a conditional sequence modelling task for ofﬂine RL that can be learned by a Decision Transformer .
[14] uses a linear Transformer [15] as an outer-product-based fast-weight programmer [24, 25].
However, in contrast to Hopﬁeld-RUDDER, these approaches require large amounts of data, as they are based on Transformer architectures.
Furthermore, Hopﬁeld-RUDDER can speed up any learning algorithm that relies on a delayed reward signal.
Our main contributions are: We (i) introduce modern Hopﬁeld networks for return decomposition for delayed rewards (Hopﬁeld-RUDDER), (ii) propose and compare different methods, such as reset-max history , for history compression for Hopﬁeld-RUDDER in POMDP environments, (iii) show that Hopﬁeld-RUDDER outperforms LSTM-based RUDDER on 1D environments with small numbers of samples, and (iv) perform preliminary experiments in the MineRL Minecraft environment indicating that Hopﬁeld-RUDDER scales to large and complex environments.
Review
We deﬁne our setting as a ﬁnite Markov decision process (MDP) or a ﬁnite partially observable Markov decision process (POMDP) to be a 4 -tuple of ( S , A , R , p ) of ﬁnite sets S with states s (random variable
S t at time t ),
A with actions a (random variable A t at time t ), R with rewards r (random variable R t at time t ), and state-reward transition distribution p ( S t +1 =
s t
+1 , R t +1 =
r t | S t =
s t , A t = a t ) .
The return of a sequence of length T at time t = { 1 , . .
, T } is deﬁned as g t = (cid:80) T − t k =
r t + k .
RUDDER.
Complex tasks are often hierarchically composed of sub-tasks.
Hence, the Q -function of an optimal policy often resembles a step function [2].
Such steps indicate patterns like achievements, failures, accomplished sub-tasks, or changes of the environment.
RUDDER reduces the delay in the rewards by identifying these patterns and moving (redistributing) the rewards to the causative state-action patterns.
[2] propose an LSTM as realization of RUDDER.
The LSTM predicts the return at the end of an episode as early as possible.
These predictions (cid:98) g are then used to redistribute the reward: r t = (cid:98) g t − (cid:98) g
t − 1 .
The redistributed reward serves as reward for a subsequent learning method and can be used to optimize a policy.
However, training an LSTM network requires a large amount of episodes, which are often difﬁcult or expensive to obtain.
Modern Hopﬁeld networks. Hopﬁeld networks are energy-based, binary associative memories, which popularized artiﬁcial neural networks in the 1980s [11, 12]. Associative memory networks have been designed to store and retrieve samples.
Their storage capacity can be considerably increased by polynomial terms in the energy function [5, 20, 3, 8, 1, 13, 4, 16]. In contrast to these binary memory networks, we use continuous associative memory networks with very high storage capacity. These The associative memory of modern Hopﬁeld networks associates 2 time steps (raw state patterns)
U of a new episode with 7 time steps (stored patterns) Y of previously observed episodes and their corresponding observed returns G . Thereby, the state patterns U are associated with estimated returns (cid:98) G . U and Y are transformed via m state and m stored before association, respectively, and can include information about previous state-action pairs, such as via reset-max history. Dotted rectangles show a single raw and transformed stored pattern with its observed return.
modern Hopﬁeld networks for deep learning architectures have an energy function with continuous states and can retrieve samples with only one update [23, 22].
Modern Hopﬁeld Networks have already been successfully applied to immune repertoire classiﬁcation [31] and chemical reaction prediction [26].
There they serve as powerful associative memories with large memory capacity and excel at pattern-recognition as they associate inputs with similar patterns in their memory.
Hopﬁeld-RUDDER
The application of MHN for RUDDER naturally follows from the idea of identifying patterns:
Patterns, e.g. state-action pairs, of observed episodes with known returns are stored in the memory of the MHN.
A new pattern can then be used to query this memory for similar patterns and their returns.
This yields an estimated return of the new pattern, which can be used to redistribute the rewards.
Fig.
1 illustrates this in detail: The MHN associates N stored patterns Y =
{ y i } Ni =1 from previously observed episodes and their corresponding observed returns G = { g i } Ni =1 with M new state patterns
U = { u i } Mi =1 .
As such, it directly associates the unknown state patterns U with estimated returns (cid:98) G = { (cid:98) g i } Mi =1 .
β is a hyper-parameter controlling the temperature of the softmax function.
The raw state patterns U and stored patterns Y may be transformed by functions m state and m stored , for example with linear mappings or multiple hidden layers, as in [23, 31].
This results in a return decomposition function ψ in the form of
[Equation]
The redistributed reward r t at time t can then be obtained by computing the differences of the estimated return values as
[Equation]
For state patterns u t and u t − 1 at times t and t − 1 this results in
[Equation]
where Y contains a representation of the stored state-action pairs { ( s i , a i ) } N i =1 with their known corresponding returns G and u t represents a state-action pair ( s t , a t ) without known return.
In the simplest case, a raw state pattern u or stored pattern y is a vector containing the observation s t and action a t at time t in an episode.
The estimated return for a single state pattern u t at time t in this case is (cid:98) g t = ψ ( Y , G , s t , a t ; β ) with the redistributed reward r t computed from state patterns u t − 1 and u t as r t = (cid:98) g t − (cid:98) g
t − 1 = ψ
( Y , G , s t , a t ; β ) − ψ
( Y , G , s t − 1 , a t − 1 ; β ) .
History compression and reset-max history. Assume a key event at time t 1 , e.g. the collection of a key, that can result in a reward at time t 2 , e.g. the opening of a chest.
In a POMDP, this key event might only be observable at t 1 but not at t 2 , e.g. the observation space contains no information whether a key has been collected already.
Assume further that observation s t and action a t are used as state or stored pattern: The association of the pattern at t 2 with a return, which depends on whether a key was collected at t 1 , may severely suffer from the missing information.
To address this problem, the state and stored patterns can be augmented by the history of the previous state-action pairs.
However, retaining the complete history of a state-action pair and including it in the state and stored patterns would not be feasible for environments with large observation spaces or long episode sequences.
As the purpose of this history is to enable the detection and storage of key events, we propose a fast and light-weight history compression method: the reset-max history .
Assuming a vector v ot ∈ R K ×with K features that contains a representation of the state-action pair ( s t , a t ) at time t = { 1 , . .
, T } , the reset-max history features v ht ∈ R J × 1 with J features, and the ﬁnal state or stored pattern v t ∈ R ( J + K ) × 1 is computed as follows:
[Equation]
(4) with v h 0 initialized with −∞ and f
[Equation]
, (5) where [; ] is the operator for vertical concatenation, the max operator is used to store the previously observed maximum feature values, and the reset gate f reset utilizes the sigmoid activation function σ and learned weight matrix W ∈ R J × ( J + K ) to reset the stored maximum feature values.
Notably, this changes the return decomposition function ψ for a state pattern u t to be dependent on all previous and current state-action pairs { ( s i , a i ) } ti =1 in the episode: (cid:98) g t = ψ ( Y , G , s 1 ,...,t , a 1 ,...,t ; β, W ) .
Experiments
In this section, we ﬁrst compare different history compression methods on multiple 1D toy environments.
Subsequently, we compare the performance of Hopﬁeld-RUDDER to the originally proposed
LSTM-RUDDER.
Finally, we visually analyse the reward redistribution of Hopﬁeld-RUDDER on the difﬁcult and complex task of collecting a diamond in the MineRL Minecraft environment. 1D key-chest environment.
We designed 16 different versions of a 1D environment, the 1D keychest environment , in order to evaluate the performance of Hopﬁeld-RUDDER, different history compression methods, and to compare Hopﬁeld-RUDDER to LSTM-RUDDER.
We use a reward redistribution score rr _ score , which is based on the known optimal policy, for this evaluation and comparison.
For more details see App. A1.
Environment details.
The 1D key-chest environment is illustrated in Fig.
The agent starts at position s in the middle.
At each position it can move either one position to the right or to the left, except for the left-most and right-most position, where further movement to the left and right, respectively, is ignored.
If the agent visits position k , it collects 1 key .
If the agent visits position s , it loses all collected keys with a probability of p l .
If the agent visits position c while holding n k keys , it will receive 1 reward at the end of the episode.
All episodes have the same ﬁxed length.
A reward of 0 or 1 is given at the end of each episode, depending on whether the chest was opened or not.
We evaluate the performance of RUDDER on different versions of this environment with p l = { 0 , 0 . 5 } and n k = { 1 , 3 } .
We consider fully observable MDP and partially observable POMDP versions of the environment.
Furthermore, we add n rnd =
{ 0 , n obs } features that contain random values { 0 , 1 } to the observation space, where n obs is the number of features of the original observation space.
In total, we created 16 different versions of this environment (see Tab. 1 and App. A1).
To redistribute reward in this environment successfully, RUDDER has to identify that position k needs to be visited n k times before visiting position c .
Furthermore, RUDDER has to identify that the key -collection process has to be restarted if the keys are lost at s .
Reward redistribution score By design, the optimal of the two available actions at any state s in the 1D key-chest environment is known.
In particular, at each state s t at time t in an episode, the agent can either take a left or right action a t ∈ { a l , a r } , which can be a correct, incorrect, or irrelevant action.
We use this knowledge to compute a score rr _ score for the quality of the reward redistribution ψ (:) for an episode with time t = { 1 , . . .
, T } as follows: [Equation] where scr (:) assigns a score {− 1 , 0 , 1 } for the redistributed reward r t at each time t . In detail, action left a l is correct if q ∗ ( s t , a l ) > q ∗ ( s t , a r ) .
Action right a r is correct if q ∗ ( s t , a r ) > q ∗ ( s t , a l ) .
An action is irrelevant if q ∗ ( s t , a l ) =
= q ∗ ( s t , a r ) .
Here, q ∗ ( s t , a t ) is the Q-function of the optimal policy π ∗ .
The redistributed reward r t = ψ (:) is correct if r t for the correct action is higher than for the incorrect action.
If the redistributed reward is correct, scr (:) assigns a score of 1, if it is incorrect a score of -1.
If the action is irrelevant, the reward redistribution receives a score of 0.
Experimental setup. We sample a large test set of 1 , 000 episodes and small training sets of { 8 , 16 , 32 , 64 , 128 , 256 , 512 } episodes, using a random agent.
Half of the episodes have a return ofand 1 in each set, with ﬁxed sequence lengths between 32 and 148 times per episode (see App. A1).
Reset-max history outperforms other history compression methods.
We ﬁrst analyze the performance of Hopﬁeld-RUDDER, where the training set is used as stored patterns.
For this, we evaluate Hopﬁeld-RUDDER without history information and with 5 different versions of history compression: (i) feature-wise max-pooling, (ii) feature-wise sum-pooling, (iii) fully-connected LSTM (LSTMf), (iv) sparsely-connected LSTM (LSTMs), and (v) reset-max history, as detailed in App. A1.2.
Furthermore, we evaluate each setting with a linear mapping m state =
m stored and without any mapping.
As shown in Tab. 1 and A2, reset-max history consistently outperforms all other Hopﬁeld-RUDDER versions, except for one POMDP setting and two MDP settings, in which it is the runner-up method.
Hopﬁeld-RUDDER with reset-max history consistently outperforms LSTM-RUDDER.
Subsequently, we analyze the performance of LSTM-RUDDER with reset-max history and without history information.
As shown in Tab. 1, Hopﬁeld-RUDDER with reset-max history consistently outperforms all LSTM-RUDDER versions on all environment versions.
More details on results, history compression methods, and training can be found in App. A1.
Minecraft.
In order to show that Hopﬁeld-RUDDER can scale to complex problems, we perform preliminary experiments within the challenging MineRL Minecraft environment [9].
Speciﬁcally, we use Hopﬁeld-RUDDER to redistribute rewards for the task ObtainDiamond .
As the high complexity of ObtainDiamond makes random exploration unfeasible, [9] provide the Minecraft environment with demonstrations of human players solving this task.
Players and agents are randomly placed in a procedurally generated 3D environment without any items in their possession.
The objective is to gather resources and build the tools required to obtain a diamond.
For ObtainDiamond , auxiliary reward is given the ﬁrst time the player obtains an item, even if multiple copies of this item are required.
For example, agents have to gather multiple logs in order [Table] to build all necessary tools but receive reward only for the ﬁrst log they acquire.
However, we omit these auxiliary rewards and only use episodic return by giving a return of 1 for demonstrations that obtain a diamond and a return of 0 for those that do not manage to obtain it.
In order to show that Hopﬁeld-RUDDER is able to redistribute reward to relevant state-action pairs in
ObtainDiamond , we train a simple Hopﬁeld-RUDDER model using the reset-max history history compression.
We only use the sequence of inventory states and associated actions as input to HopﬁeldRUDDER.
Following [19], we use 10 successful and 10 unsuccessful demonstration episodes as training set.
The state-action pairs from these 20 episodes are used as stored patterns Y .
We use the obfuscated version of the environment and demonstrations, therefore the observations s and actions a are the hidden representation of an unknown auto encoder (for further details see App. A2.3).
We use a neural network with 2 hidden layers and ReLU activation as mapping functions m state
= m stored .
Training Hopﬁeld-RUDDER does not require massive compute resources.
On the contrary, training a model for 100 episodes takes only minutes on an Nvidia A40 GPU.
A small hyper-parameter search showed that the model is not overly sensitive to hyper-parameter changes and most models produce a similar reward redistribution.
3 shows the redistribution for a demonstration episode, which was not used for training the model.
We use the non-obfuscated inventory states for visualization.
It visually seems that HopﬁeldRUDDER is able to redistribute the reward to important state-action pairs.
Furthermore, HopﬁeldRUDDER appears to not redistribute rewards to irrelevant state-action pairs, resulting in a reward redistribution with little noise.
Conclusion
We have introduced Hopﬁeld-RUDDER, a novel realization of RUDDER.
Hopﬁeld-RUDDER replaces the LSTM in the original LSTM-RUDDER with a powerful auto-associative memory, the modern Hopﬁeld network.
In contrast to LSTM-RUDDER, Hopﬁeld-RUDDER can be trained on a drastically lower number of samples. Acknowledgments and Disclosure of Funding
The ELLIS Unit Linz, the LIT AI Lab, the Institute for Machine Learning, are supported by the Federal
State Upper Austria.
IARAI is supported by Here Technologies.
We thank the projects AI-MOTION (LIT-2018-6-YOU-212), DeepToxGen (LIT-2017-3-YOU-003), AI-SNN (LIT-2018-6-YOU-214),
DeepFlood (LIT-2019-8-YOU-213), Medical Cognitive Computing Center (MC3), INCONTROL-RL (FFG-881064), PRIMAL (FFG-873979), S3AI (FFG-872172), DL for GranularFlow (FFG-871302),
AIRI FG 9-N (FWF-36284, FWF-36235), ELISE (H2020-ICT-2019-3 ID: 951847), AIDD (MSCAITN-2020 ID: 956832).
We thank Janssen Pharmaceutica (MaDeSMart, HBC.2018.2287), Audi.
JKU
Deep Learning Center, TGW LOGISTICS GROUP GMBH, Silicon Austria Labs (SAL), FILL
Gesellschaft mbH, Anyline GmbH, Google, ZF Friedrichshafen