Original: Quantile Regression Reinforcement Learning with State Aligned Vector Rewards
Revised: Quantile Regression Reinforcement Learning with State Aligned Vector Rewards

Original: Abstract
Revised: Abstract

Original: The sample efﬁciency of deep reinforcement learning (DRL) algorithms is limitedby the weak scalar training signal.
Revised: The sample efﬁciency of deep reinforcement learning (DRL) algorithms is limitedby the weak scalar training signal.

Original: We propose to use state aligned vector rewardsto capitalize on the spatial nature of reaching problems and show that a state changedistribution can be learned given an action distribution.
Revised: We propose to use state aligned vector rewardsto capitalize on the spatial nature of reaching problems and show that a state changedistribution can be learned given an action distribution.

Original: Our agent, trained with anew DRL method inspired by quantile regression, is able to learn multiple timesfaster in high dimensional state spaces than a classical DRL algorithm.
Revised: Our agent, trained with anew DRL method inspired by quantile regression, is able to learn multiple timesfaster in high dimensional state spaces than a classical DRL algorithm.

Original:  Introduction
Revised:  Introduction

Original: While neural networks are powerful function approximators, they require large amounts of trainingdata to converge.
Revised: While neural networks are powerful function approximators, they require large amounts of trainingdata to converge.

Original: In reinforcement learning (RL) this means many costly interactions with theenvironment.
Revised: In reinforcement learning (RL) this means many costly interactions with theenvironment.

Original: The problem is aggravated by the weak training signal of RL – a scalar reward – whichwas originally inspired by the general “rewarding” role of dopamine activity in mammal brains.
Revised: The problem is aggravated by the weak training signal of RL – a scalar reward – whichwas originally inspired by the general “rewarding” role of dopamine activity in mammal brains.

Original: However, Pinto and Lammel [17] point out that the diversity of dopamine circuits in the mid brain isbetter modeled by viral vector strategies.
Revised: However, Pinto and Lammel [17] point out that the diversity of dopamine circuits in the mid brain isbetter modeled by viral vector strategies.

Original: Gershman et al.
Revised: Gershman et al.

Original: [7] also show that human RL incorporateseffector speciﬁc value estimations to cope with the high dimensional action space.
Revised: [7] also show that human RL incorporateseffector speciﬁc value estimations to cope with the high dimensional action space.

Original: Inspired by thesenew biological insights, we improve the sample efﬁciency of a deep RL algorithm by modeling ad -dimensional vector reward.
Revised: Inspired by thesenew biological insights, we improve the sample efﬁciency of a deep RL algorithm by modeling ad -dimensional vector reward.

Original: A vector reward can in metric spaces easily be deﬁned in alignmentwith the state space.
Revised: A vector reward can in metric spaces easily be deﬁned in alignmentwith the state space.

Original: We further show that if state and action space are not aligned, a mapping fromaction distribution to state change distribution can be learned to allow vector reward training.
Revised: We further show that if state and action space are not aligned, a mapping fromaction distribution to state change distribution can be learned to allow vector reward training.

Original: As a motivating example, consider the agent in Figure 1(a) trying to reach the goal (marked by theblue dot).
Revised: As a motivating example, consider the agent in Figure 1(a) trying to reach the goal (marked by theblue dot).

Original: If we take p as the position vector of the agent relative to the goal, a sensible reward toguide the agent to the goal is r = ||
Revised: If we take p as the position vector of the agent relative to the goal, a sensible reward toguide the agent to the goal is r = ||

Original: p
Revised: p

Original: ||
Revised: ||

Original: − || p + a || where || · || can be any norm in the vector space ofthe environment.
Revised: − || p + a || where || · || can be any norm in the vector space ofthe environment.

Original: We will focus on the L 1 norm in this paper.
Revised: We will focus on the L 1 norm in this paper.

Original: During training the agent might tryaction a which moves it closer to the goal in x direction, but a bit further away from the goal in ydirection.
Revised: During training the agent might tryaction a which moves it closer to the goal in x direction, but a bit further away from the goal in ydirection.

Original: The scalar reward conveys the information that the action was rather positive (since theagent got closer to the goal) but misses out on the distinction that the action was good in x -directionbut bad in y -direction.
Revised: The scalar reward conveys the information that the action was rather positive (since theagent got closer to the goal) but misses out on the distinction that the action was good in x -directionbut bad in y -direction.

Original: To give this distinction, a more informative reward would keep the dimensionsseparate and therefore be a vector itself: r = | p | − | p + a | where | · | denotes the element-wiseabsolute value here.
Revised: To give this distinction, a more informative reward would keep the dimensionsseparate and therefore be a vector itself: r = | p | − | p + a | where | · | denotes the element-wiseabsolute value here.

Original: Note that this reward is dimension wise aligned with the position p , the state , ofthe agent.
Revised: Note that this reward is dimension wise aligned with the position p , the state , ofthe agent.

Original: (Since we focus on reaching problems in this work, we will use the terms “position” and“state” interchangeably.)
Revised: (Since we focus on reaching problems in this work, we will use the terms “position” and“state” interchangeably.)

Original: The problem with a state aligned vector reward is that the action space isin most cases not state aligned.
Revised: The problem with a state aligned vector reward is that the action space isin most cases not state aligned.

Original: To see this, consider the schematic robot arm in Figure 1(b): Theaction dimensions a 1 and a 2 correspond to the torques and do not directly translate to a shift in xand y dimension, respectively.
Revised: To see this, consider the schematic robot arm in Figure 1(b): Theaction dimensions a 1 and a 2 correspond to the torques and do not directly translate to a shift in xand y dimension, respectively.

Original: To address this issue we use quantile function networks [5, 4] toestimate the position change given the current observation and quantile target.
Revised: To address this issue we use quantile function networks [5, 4] toestimate the position change given the current observation and quantile target.

Original: Additionally, we givea parameterization of the action probability distribution as input to this position change predictionnetwork (short PCPN).
Revised: Additionally, we givea parameterization of the action probability distribution as input to this position change predictionnetwork (short PCPN).

Original: We then train the agent, parameterized by another neural network which maps
Revised: We then train the agent, parameterized by another neural network which maps

Original: Goal
Revised: Goal

Original: Agent a r x r y
Revised: Agent a r x r y

Original: Goal
Revised: Goal

Original: Position
Revised: Position

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: from observations to action probability distributions, through a new RL method we call quantileregression reinforcement learning (short QRRL).
Revised: from observations to action probability distributions, through a new RL method we call quantileregression reinforcement learning (short QRRL).

Original: A schematic overview can be seen in Figure 2.
Revised: A schematic overview can be seen in Figure 2.

Original:  Quantile Regression and Implicit Quantile Networks
Revised:  Quantile Regression and Implicit Quantile Networks

Original: Quantile regression [11] discusses approximation techniques for the inverse cumulative distributionfunction F − 1 Y , i.e., the quantile function , of some probability distribution Y .
Revised: Quantile regression [11] discusses approximation techniques for the inverse cumulative distributionfunction F − 1 Y , i.e., the quantile function , of some probability distribution Y .

Original: Recent work [4, 15]shows that a neural network can learn to approximate the quantile function by mapping a uniformlysampled quantile target τ ∼ U
Revised: Recent work [4, 15]shows that a neural network can learn to approximate the quantile function by mapping a uniformlysampled quantile target τ ∼ U

Original: ([0 , 1]) to its corresponding quantile function value F − 1 Y ( τ ) ∈ R .
Revised: ([0 , 1]) to its corresponding quantile function value F − 1 Y ( τ ) ∈ R .

Original: Thereby the trained neural network implicitly models the full probability distribution Y .
Revised: Thereby the trained neural network implicitly models the full probability distribution Y .

Original: Moreformally, Ostrovski et al.
Revised: Moreformally, Ostrovski et al.

Original: [15] show that the expected quantile loss
Revised: [15] show that the expected quantile loss

Original: [Equation]
Revised: [Equation]

Original: ( z
Revised: ( z

Original: [Equation]
Revised: [Equation]

Original: ˆ Q
Revised: ˆ Q

Original: [Equation]
Revised: [Equation]

Original: )) (cid:105)(cid:105) with ρ τ ( δ ) = (
Revised: )) (cid:105)(cid:105) with ρ τ ( δ ) = (

Original: [Equation]
Revised: [Equation]

Original: δ (1) of a parameterized quantile function ˆ Q θ aproximating the quantile function F − 1 Z of some distribution
Revised: δ (1) of a parameterized quantile function ˆ Q θ aproximating the quantile function F − 1 Z of some distribution

Original: Z is equal to the quantile divergence q ( Z, Q θ ) (see [15]) plus some constant not depending on theparameters θ .
Revised: Z is equal to the quantile divergence q ( Z, Q θ ) (see [15]) plus some constant not depending on theparameters θ .

Original: Here, Q θ is the distribution implicitly deﬁned by ˆ Q θ .
Revised: Here, Q θ is the distribution implicitly deﬁned by ˆ Q θ .

Original: Therefore, training a neuralnetwork ˆ Q θ ( τ ) to minimize ρ τ ( z − ˆ Q θ ( τ )) with z sampled from the target probability distribution
Revised: Therefore, training a neuralnetwork ˆ Q θ ( τ ) to minimize ρ τ ( z − ˆ Q θ ( τ )) with z sampled from the target probability distribution

Original: Z effectively models an approximate distribution Q θ of Z implicitly in the network parameters θ ofthe neural network ˆ Q θ ( τ ) .
Revised: Z effectively models an approximate distribution Q θ of Z implicitly in the network parameters θ ofthe neural network ˆ Q θ ( τ ) .

Original: In our setup, we train the PCPN with the quantile regression loss (1) toapproximate a position change quantile function per position dimension.
Revised: In our setup, we train the PCPN with the quantile regression loss (1) toapproximate a position change quantile function per position dimension.

Original: Aside from a target quantileτ ∼ U ([0 , 1]) per position dimension, the network input consists of an observation o of the agent andan action probability distribution parameterized by the output of the agent network.
Revised: Aside from a target quantileτ ∼ U ([0 , 1]) per position dimension, the network input consists of an observation o of the agent andan action probability distribution parameterized by the output of the agent network.

Original:  Quantile Regression Reinforcement Learning and SAVER
Revised:  Quantile Regression Reinforcement Learning and SAVER

Original: Our new RL technique described in the beginning of this section is generally applicable to RLproblems.
Revised: Our new RL technique described in the beginning of this section is generally applicable to RLproblems.

Original: We therefore use the terms “action” and “policy” in the common RL meaning here, which is distinct from the meaning of “action” and “policy” in the rest of the paper.
Revised: We therefore use the terms “action” and “policy” in the common RL meaning here, which is distinct from the meaning of “action” and “policy” in the rest of the paper.

Original: In the end of this sectionwe then link the position change estimation of the PCPN to this new meaning of “action”.
Revised: In the end of this sectionwe then link the position change estimation of the PCPN to this new meaning of “action”.

Original: The idea of QRRL is to model the policy implicitly in the network parameters, therefore allowingfor complex stochastic policies.
Revised: The idea of QRRL is to model the policy implicitly in the network parameters, therefore allowingfor complex stochastic policies.

Original: For this we model for each action dimension the quantile functionˆ Q
Revised: For this we model for each action dimension the quantile functionˆ Q

Original: θ ( τ, o ) =
Revised: θ ( τ, o ) =

Original: F − 1 Q θ ( τ, o ) of the implicitly deﬁned action distribution Q θ ( o ) by a neural network withan observation o and a target quantile τ ∈ [0 , 1] as input.
Revised: F − 1 Q θ ( τ, o ) of the implicitly deﬁned action distribution Q θ ( o ) by a neural network withan observation o and a target quantile τ ∈ [0 , 1] as input.

Original: An action a τ ∈
Revised: An action a τ ∈

Original: A ≡ R d , d being thenumber of action dimensions, can then be sampled by sampling τ ∼ U ([0 , 1] d ) and taking thenetwork outputs as action.
Revised: A ≡ R d , d being thenumber of action dimensions, can then be sampled by sampling τ ∼ U ([0 , 1] d ) and taking thenetwork outputs as action.

Original: To train the networks quantile regression comes in handy.
Revised: To train the networks quantile regression comes in handy.

Original: Informallyput, quantile regression is linked to the Wasserstein metric [5] which is also sometimes referred toas earth movers distance .
Revised: Informallyput, quantile regression is linked to the Wasserstein metric [5] which is also sometimes referred toas earth movers distance .

Original: Imagine a pile of earth representing probability mass.
Revised: Imagine a pile of earth representing probability mass.

Original: In RL we want tomove probability mass towards actions that were good and away from actions that were bad, where“good” and “bad” are measured by accumulative reward achieved.
Revised: In RL we want tomove probability mass towards actions that were good and away from actions that were bad, where“good” and “bad” are measured by accumulative reward achieved.

Original: Quantile regression can achievethis neatly by shaping the pile of earth according to an advantage estimation and the constraint ofmonotonicity (a core property of quantile functions) 1 .
Revised: Quantile regression can achievethis neatly by shaping the pile of earth according to an advantage estimation and the constraint ofmonotonicity (a core property of quantile functions) 1 .

Original: This leads us to the QRRL actor objective
Revised: This leads us to the QRRL actor objective

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  θ is monotonically increasing with τ (cid:48) where ( R t,n − V ψ ( o t )) is an advantage estimation [14] with R t,n = γ n V ψ ( o t + n ) + (cid:80) t + n t
Revised: ˆ Q θ is monotonically increasing with τ (cid:48) where ( R t,n − V ψ ( o t )) is an advantage estimation [14] with R t,n = γ n V ψ ( o t + n ) + (cid:80) t + n t

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Revised: [Equation]

Original: t γ t
Revised: γ t

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: being the n -step estimate of the discounted future reward and V ψ ( · ) being the output of a criticnetwork with parameters ψ , which is trained to approximate R t,n .
Revised: being the n -step estimate of the discounted future reward and V ψ ( · ) being the output of a criticnetwork with parameters ψ , which is trained to approximate R t,n .

Original: Note that QRRL is a constraintoptimization.
Revised: Note that QRRL is a constraintoptimization.

Original: We found it however sufﬁcient to add this constraint as an additional loss term
Revised: We found it however sufﬁcient to add this constraint as an additional loss term

Original: L

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where we use the Huber loss L Huber [9] for better convergence.
Revised: where we use the Huber loss L Huber [9] for better convergence.

Original: We weight this additional loss termwith a constant Lagrange multiplier λ mon in the full loss term
Revised: We weight this additional loss termwith a constant Lagrange multiplier λ mon in the full loss term

Original: [Equation]
Revised: [Equation]

Original: with L a ( a τ , o t , τ (cid:48) , θ )
Revised: with L a ( a τ , o t , τ (cid:48) , θ )

Original: =
Revised: =

Original: ρ τ (cid:48)
Revised: ρ τ (cid:48)

Original: ( a τ − ˆ Q θ ( τ (cid:48) , o )) · ( R t,n − V ψ ( o t )) and L c ( o t , ψ ) =
Revised: ( a τ − ˆ Q θ ( τ (cid:48) , o )) · ( R t,n − V ψ ( o t )) and L c ( o t , ψ ) =

Original: ( R t,n − V ψ ( o t ))
Revised: ( R t,n − V ψ ( o t ))

Original: 2 .λ c is another constant Lagrange multiplier to weight the critic loss L c against the actor loss L a .
Revised: 2 .λ c is another constant Lagrange multiplier to weight the critic loss L c against the actor loss L a .

Original: In our State Aligned VEctor Reward (SAVER) agent, we use QRRL to train the agent network AN η through the PCPN.
Revised: In our State Aligned VEctor Reward (SAVER) agent, we use QRRL to train the agent network AN ηthrough the PCPN.

Original: For this we feed the action probability distribution output µ , σ of theagent network to a pretrained PCPN and train on the QRRL loss (2) with respect to the agentnetwork parameters η , where we take the actual position change ∆ p introduced by a sampled actiona ∼ N ( µ , Iσ ) as QRRL action target a τ = ∆ p and compare it to K sampled position changeestimations ∆ˆ p ( τ ( i ) ) , τ ( i ) ∼ U ([0 , 1] d ) for i ∈ { 1 , ..., K } .
Revised: For this we feed the action probability distribution output A of the agent networkto a pretrained PCPN and train on the QRRL loss (2) with respect to the agent network parametersη , where we take the actual position change ∆ p introduced by a sampled action a ∼ A as QRRLaction target a τ = ∆ p and compare it to K sampled position change estimations ∆ˆ p ( τ ( i ) ) , τ ( i ) ∼ U ([0 , 1] d ) for i ∈ { 1 , ..., K } .

Original: Note that we pretrain the PCPN in oursetup with random observations and action probability distributions and freeze the PCPN weightsduring agent training.
Revised: Note that we pretrain the PCPN in our setup with random observationsand action probability distributions and freeze the PCPN weights during agent training.

Original: Since the output of the PCPN can be seen as state aligned action, trainingon vector rewards is straight forward.
Revised: Since theoutput of the PCPN can be seen as state aligned action, training on vector rewards is straight forward.

Original: Instead of having a scalar critic estimating the value function V ( · ) ∈ R we estimate a value per action dimension, i.e., V ( · ) ∈ R d .
Revised: Instead of having a scalar critic estimating the value function V ( · ) ∈ R we estimate a value peraction dimension, i.e., V ( · ) ∈ R d .

Original: Similarly, the vector rewards canbe summed to a vector n -step discounted reward estimation R t,n = γ n V ( o t + n ) + (cid:80) t + n t (cid:48)
Revised: Similarly, the vector rewards can be summed to a vector n -stepdiscounted reward estimation R t,n = γ n V ( o t + n ) + (cid:80) t + n t (cid:48)

Original: =
Revised: =

Original: t γ t (cid:48) − t r t (cid:48) ,which leads us to a vector advantage R t,n − V ( o t ) at timestep t .
Revised: t γ t (cid:48) − t r t (cid:48) , which leads us to a vectoradvantage R t,n − V ( o t ) at timestep t .

Original: Therefore we can apply loss (2) toeach position change estimation dimension individually.
Revised: Therefore we can apply loss (2) to each position changeestimation dimension individually.

Original:  Experiments and Results
Revised:  Experiments and Results

Original: To test our ideas, we implement a simple hypercube environment to model high dimensional metricspaces.
Revised: To test our ideas, we implement a simple hypercube environment to model high dimensional metricspaces.

Original: For proof of concept, we compare our SAVER agent against an A2C agent, the synchronousvariant of A3C [14].
Revised: For proof of concept, we compare our SAVER agent against an A2C agent, the synchronousvariant of A3C [14].

Original: We implement a simple feed forward neural architecture architecture and ﬁx allhyperparameters except the initial learning rate to values that work well for SAVER and A2C. Theagent’s action executed in the environment consists of two parts: a softmax distribution from whichthe dimension to be manipulated is chosen (discrete action part) and a scalar Gaussian distributiondeﬁned by network outputs µ and σ from which the step size is sampled (continuous action part).
Revised: We implement a simple feed forward neural architecture architecture and ﬁx allhyperparameters except the initial learning rate to values that work well for SAVER and A2C. Theagent’s action executed in the environment consists of two parts: a softmax distribution from whichthe dimension to be manipulated is chosen (discrete action part) and a scalar Gaussian distributiondeﬁned by network outputs µ and σ from which the step size is sampled (continuous action part).

Original: We pretrain the PCPN with 10,000 batches of 128 transitions each by sampling softmax logits and µ
Revised: We pretrain the PCPN with 10,000 batches of 128 transitions each by sampling softmax logits and µ

Original: [Figure]
Revised: [Figure]

Original: mean averaged across 3 training runs – over the course of training is plotted in Figure 3.
Revised: mean averaged across 3 training runs – over the course of training is plotted in Figure 3.

Original: that the higher dimensional the hypercube is, the more advantageous it is to train with vector rewards
Revised: that the higher dimensional the hypercube is, the more advantageous it is to train with vector rewards

Original:  Related Work
Revised:  Related Work

Original: Klinkhammer [10] discusses multiple problem aligned rewards for better learning, while Brys et al.[3] discuss the effect of correlated rewards on learning performance.
Revised: Klinkhammer [10] discusses multiple problem aligned rewards for better learning, while Brys et al.[3] discuss the effect of correlated rewards on learning performance.

Original: Brys et al.
Revised: Brys et al.

Original: [3] also suggestmultiple reward shapings of the same reward function for faster learning.
Revised: [3] also suggestmultiple reward shapings of the same reward function for faster learning.

Original: Van Seijen et al.
Revised: Van Seijen et al.

Original: [19]decompose the reward function into multiple rewards and train an architecture similar to ours withdeep Q learning, assigning a Q-value output to each reward.
Revised: [19]decompose the reward function into multiple rewards and train an architecture similar to ours withdeep Q learning, assigning a Q-value output to each reward.

Original: While all three approaches come tothe same conclusion as we do, i.e., increased training performance, they do require hand engineeredreward functions, reward shapings and/or reward decompositions.
Revised: While all three approaches come tothe same conclusion as we do, i.e., increased training performance, they do require hand engineeredreward functions, reward shapings and/or reward decompositions.

Original: In contrast, our approach is basedon the fact that many state spaces are metric spaces and therefore allow for a straightforward vectorreward interpretation.
Revised: In contrast, our approach is basedon the fact that many state spaces are metric spaces and therefore allow for a straightforward vectorreward interpretation.

Original: This makes our approach easier to apply to a larger set of tasks.
Revised: This makes our approach easier to apply to a larger set of tasks.

Original: While stateproximity was already used by McCallum [13] for faster backpropagation of rewards in tabular RL,we are unaware of any deep learning algorithm capitalizing on state aligned vector rewards as we do.
Revised: While stateproximity was already used by McCallum [13] for faster backpropagation of rewards in tabular RL,we are unaware of any deep learning algorithm capitalizing on state aligned vector rewards as we do.

Original: Many recent approaches to deep RL learn the environment dynamics to have a richer training signal[6], imagine action consequences [18], improve exploration [16] or dream up solutions [8].
Revised: Many recent approaches to deep RL learn the environment dynamics to have a richer training signal[6], imagine action consequences [18], improve exploration [16] or dream up solutions [8].

Original: Aninteresting line of research [20, 1, 12, 2] in this direction uses successor features to share knowledgebetween tasks in the same environment.
Revised: Aninteresting line of research [20, 1, 12, 2] in this direction uses successor features to share knowledgebetween tasks in the same environment.

Original: In contrast to most of these works, which often only predictone possible next state or successor feature, our PCPN incorporates the full probability distributionof possible state changes.
Revised: In contrast to most of these works, which often only predictone possible next state or successor feature, our PCPN incorporates the full probability distributionof possible state changes.

Original: While Ha and Schmidhuber [8] also predict the full probability distributionof their next state representation by a Gaussian mixture model, our approach is more general in that itis also able to approximate non-Gaussian probability distributions.
Revised: While Ha and Schmidhuber [8] also predict the full probability distributionof their next state representation by a Gaussian mixture model, our approach is more general in that itis also able to approximate non-Gaussian probability distributions.

Original:  Conclusion
Revised:  Conclusion

Original: In this work we present the idea of state aligned vector rewards for faster reinforcement learning.
Revised: In this work we present the idea of state aligned vector rewards for faster reinforcement learning.

Original: Additionally, we also present a new RL technique we call quantile regression reinforcement learning(QRRL).
Revised: Additionally, we also present a new RL technique we call quantile regression reinforcement learning(QRRL).

Original: QRRL allows for complex stochastic policies in continuous action spaces, not limitingagents to Gaussian actions.
Revised: QRRL allows for complex stochastic policies in continuous action spaces, not limitingagents to Gaussian actions.

Original: Combining both, we show that our SAVER agent can be trained througha quantile network pretrained in the environment.
Revised: Combining both, we show that our SAVER agent can be trained througha quantile network pretrained in the environment.

Original: Our SAVER agent is capable of training orders ofmagnitudes faster in high dimensional metric spaces.
Revised: Our SAVER agent is capable of training orders ofmagnitudes faster in high dimensional metric spaces.

Original: While d -dimensional metric spaces are mainlymathematical constructs for d > 3 , we see a lot of potential in SAVER to be applied to problems inmathematics and related ﬁelds, including the ﬁeld of deep (reinforcement) learning itself.
Revised: While d -dimensional metric spaces are mainlymathematical constructs for d > 3 , we see a lot of potential in SAVER to be applied to problems inmathematics and related ﬁelds, including the ﬁeld of deep (reinforcement) learning itself.