Original: Quantile Regression Reinforcement Learning with State Aligned Vector Rewards
Revised: Quantile Regression Reinforcement Learning with State Aligned Vector Rewards

Original: Abstract
Revised: Abstract

Original: The sample efﬁciency of deep reinforcement learning (DRL) algorithms is limitedby the weak scalar training signal.
Revised: The sample efﬁciency of deep reinforcement learning (DRL) algorithms is limited by the weak scalar training signal.

Original: We propose to use state aligned vector rewards tocapitalize on the spatiotemporal nature of reaching problems and show that a statechange distribution can be learned given an action distribution.
Revised: We propose to use state aligned vector rewards to capitalize on the spatiotemporal nature of reaching problems and show that a state change distribution can be learned given an action distribution.

Original: Our agent, trainedwith a new DRL method inspired by quantile regression, is able to learn multipletimes faster in high dimensional state spaces than a classical DRL algorithm.
Revised: Our agent, trained with a new DRL method inspired by quantile regression, is able to learn multiple times faster in high dimensional state spaces than a classical DRL algorithm

Original:  Introduction
Revised:  Introduction

Original: While neural networks are powerful function approximators, they require large amounts of trainingdata to converge.
Revised: While neural networks are powerful function approximators, they require large amounts of training data to converge.

Original: In reinforcement learning (RL) this means many costly interactions with theenvironment.
Revised: In reinforcement learning (RL) this means many costly interactions with the environment.

Original: The problem is aggravated by the weak training signal of RL – a scalar reward – whichwas originally inspired by the general “rewarding” role of dopamine activity in mammal brains.
Revised: The problem is aggravated by the weak training signal of RL – a scalar reward – which was originally inspired by the general “rewarding” role of dopamine activity in mammal brains.

Original: However, Pinto and Lammel [18] point out that the diversity of dopamine circuits in the mid brain isbetter modeled by viral vector strategies.
Revised: However, Pinto and Lammel [18] point out that the diversity of dopamine circuits in the mid brain is better modeled by viral vector strategies.

Original: Gershman et al.
Revised: Gershman et al.

Original: [7] also show that human RL incorporateseffector speciﬁc value estimations to cope with the high dimensional action space.
Revised: [7] also show that human RL incorporates effector speciﬁc value estimations to cope with the high dimensional action space.

Original: Inspired by thesenew biological insights, we improve the sample efﬁciency of a deep RL algorithm by capitalizingon the spatiotemporal nature of reaching problems.
Revised: Inspired by these new biological insights, we improve the sample efﬁciency of a deep RL algorithm by capitalizing on the spatiotemporal nature of reaching problems.

Original: A vector reward can in metric spaces easily bedeﬁned in dimensionwise alignment with the state space.
Revised: A vector reward can in metric spaces easily be deﬁned in dimensionwise alignment with the state space.

Original: We say that two vector spaces are alignedif their dimensions correlate and show that if state and action space are not aligned, a mapping fromaction distribution to state change distribution can be learned to allow vector reward training.
Revised: We say that two vector spaces are aligned if their dimensions correlate and show that if state and action space are not aligned, a mapping from action distribution to state change distribution can be learned to allow vector reward training.

Original: As a motivating example, consider the agent in Figure 1(a) trying to reach the goal (marked by theblue dot).
Revised: As a motivating example, consider the agent in Figure 1(a) trying to reach the goal (marked by the blue dot).

Original: If we take s as the position state vector of the agent relative to the goal, a sensible rewardto guide the agent to the goal is r = ||
Revised: If we take s as the position state vector of the agent relative to the goal, a sensible reward to guide the agent to the goal is r = ||

Original: s ||
Revised: s ||

Original: − || s
Revised: − || s

Original: + a || where || · || can be any norm in the vector spaceof the environment.
Revised: + a || where || · || can be any norm in the vector space of the environment.

Original: We will focus on the L 1 norm in this paper.
Revised: We will focus on the L 1 norm in this paper.

Original: During training the agent might tryaction a which moves it closer to the goal in x direction, but a bit further away from the goal in ydirection.
Revised: During training the agent might try action a which moves it closer to the goal in x direction, but a bit further away from the goal in y direction.

Original: The scalar reward conveys the information that the action was rather positive (since theagent got closer to the goal) but misses out on the distinction that the action was good in x -directionbut bad in y -direction.
Revised: The scalar reward conveys the information that the action was rather positive (since the agent got closer to the goal) but misses out on the distinction that the action was good in x -direction but bad in y -direction.

Original: To give this distinction, a more informative reward would keep the dimensionsseparate and therefore be a vector itself: r = | s | − | s + a | where | · | denotes the element-wiseabsolute value here.
Revised: To give this distinction, a more informative reward would keep the dimensions separate and therefore be a vector itself: r = | s | − | s + a | where | · | denotes the element-wise absolute value here.

Original: Note that this reward is dimensionwise aligned with the position s , the state , ofthe agent.
Revised: Note that this reward is dimensionwise aligned with the position s , the state , of the agent.

Original: (Since we focus on reaching problems in this work, we will use the terms “position” and“state” interchangeably.)
Revised: (Since we focus on reaching problems in this work, we will use the terms “position” and “state” interchangeably.)

Original: The problem with a state aligned vector reward is that the action space isin most cases not state aligned.
Revised: The problem with a state aligned vector reward is that the action space is in most cases not state aligned.

Original: To see this, consider the schematic robot arm in Figure 1(b): Theaction dimensions a 1 and a 2 correspond to the torques and do not directly translate to a shift in xand y dimension, respectively.
Revised: To see this, consider the schematic robot arm in Figure 1(b): The action dimensions a 1 and a 2 correspond to the torques and do not directly translate to a shift in x and y dimension, respectively.

Original: To address this issue we use quantile function networks [5, 4] toestimate the position change given the current observation and quantile target.
Revised: To address this issue we use quantile function networks [5, 4] to estimate the position change given the current observation and quantile target.

Original: Additionally, we givea parameterization of the action probability distribution as input to this position change prediction
Revised: Additionally, we give

Original: Goal
Revised: Goal

Original: Agent a r x r y
Revised: Agent a r x r y

Original: Goal
Revised: Goal

Original: Positiona
Revised: Position

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  network (short PCPN).
Revised: a parameterization of the action probability distribution as input to this position change prediction network (short PCPN).

Original: We then train the agent, parameterized by another neural network which mapsfrom observations to action probability distributions, through a new RL method we call quantileregression reinforcement learning (short QRRL).
Revised: We then train the agent, parameterized by another neural network which maps from observations to action probability distributions, through a new RL method we call quantile regression reinforcement learning (short QRRL).

Original: A schematic overview can be seen in Figure 2.
Revised: A schematic overview can be seen in Figure 2.

Original:  Quantile Regression and Implicit Quantile Networks
Revised:  Quantile Regression and Implicit Quantile Networks

Original: Quantile regression [11] discusses approximation techniques for the inverse cumulative distributionfunction F − 1 Z , i.e., the quantile function G Z :
Revised: Quantile regression [11] discusses approximation techniques for the inverse cumulative distribution function F − 1 Z , i.e., the quantile function G Z :

Original: = F − 1 Z , of some probability distribution Z .
Revised: = F − 1 Z , of some probability distribution Z .

Original: We referto the quantile function as G Z to avoid confusion with the in this work unrelated Q-function of
Revised: We refer to the quantile function as G Z to avoid confusion with the in this work unrelated Q-function of

Original: Q-learning [14, 5, 4].
Revised: Q-learning [14, 5, 4].

Original: Recent work [4, 16] shows that a neural network can learn to approximate thequantile function by mapping a uniformly sampled quantile target τ 
Revised: Recent work [4, 16] shows that a neural network can learn to approximate the quantile function by mapping a uniformly sampled quantile target τ ∼ U

Original: ∼ U ([0 , 1]) to its correspondingquantile function value G Z ( τ ) ∈ R .
Revised: ([0 , 1]) to its corresponding quantile function value G Z ( τ ) ∈ R .

Original: Thereby the trained neural network implicitly models the fullprobability distribution Z .
Revised: Thereby the trained neural network implicitly models the full probability distribution Z .

Original: More formally, Ostrovski et al.
Revised: More formally, Ostrovski et al.

Original: [16] show that the expected quantile loss
Revised: [16] show that the expected quantile loss

Original: [Equation]
Revised: [Equation]

Original: − G ˜

Original: Z θ ( τ )) (cid:3)(cid:3) with ρ τ ( δ ) = ( τ [Equation] δ<
Revised: of a parameterized quantile function

Original: [Equation] ) · δ [Equation] ) of a parameterized quantile function G ˜ Z θ aproximating the quantile function G Z of some distribution
Revised: G ˜ Z θ aproximating the quantile function G Z of some distribution

Original: Z is equal to the quantile divergence q ( Z, ˜
Revised: Z is equal to the quantile divergence q ( Z, ˜

Original: Z θ ) (see [16]) plus some constant not depending on theparameters θ .
Revised: Z θ ) (see [16]) plus some constant not depending on the parameters θ .

Original: Here, ˜ Z θ is the distribution implicitly deﬁned by G ˜
Revised: Here, ˜ Z θ is the distribution implicitly deﬁned by G ˜

Original: Z θ .
Revised: Z θ .

Original: Therefore, training a neuralnetwork G ˜ Z θ ( τ ) to minimize ρ τ ( z − G ˜
Revised: Therefore, training a neural network G ˜ Z θ ( τ ) to minimize ρ τ ( z − G ˜

Original: Z θ ( τ )) with z sampled from the target probability distribution
Revised: Z θ ( τ )) with z sampled from the target probability distribution

Original: Z effectively models an approximate distribution ˜
Revised: Z effectively models an approximate distribution ˜

Original: Z θ of Z implicitly in the network parameters θ ofthe neural network G ˜
Revised: Z θ of Z implicitly in the network parameters θ of the neural network G ˜

Original: Z θ ( τ ) .
Revised: Z θ ( τ ) .

Original: In our setup, we train the PCPN with the quantile regression loss (1) toapproximate a position change quantile function per position dimension.
Revised: In our setup, we train the PCPN with the quantile regression loss (1) to approximate a position change quantile function per position dimension.

Original: Aside from a target quantileτ ∼ U ([0 , 1]) per position dimension, the network input consists of an observation o of the agent andan action probability distribution parameterized by the output of the agent network.
Revised: Aside from a target quantile τ ∼ U ([0 , 1]) per position dimension, the network input consists of an observation o of the agent and an action probability distribution parameterized by the output of the agent network.

Original:  Quantile Regression Reinforcement Learning and SAVER
Revised:  Quantile Regression Reinforcement Learning and SAVER

Original: Our new RL technique described in the beginning of this section is generally applicable to RLproblems.
Revised: Our new RL technique described in the beginning of this section is generally applicable to RL problems.

Original: We therefore use the terms “action” and “policy” in the common RL meaning here, whichis distinct from the meaning of “action” and “policy” in the rest of the paper.
Revised: We therefore use the terms “action” and “policy” in the common RL meaning here, which is distinct from the meaning of “action” and “policy” in the rest of the paper.

Original: In the end of this sectionwe then link the position change estimation of the PCPN to this new meaning of “action”.
Revised: In the end of this section we then link the position change estimation of the PCPN to this new meaning of “action”.

Original: The idea of QRRL is to model the policy implicitly in the network parameters, therefore allowingfor complex stochastic policies.
Revised: The idea of QRRL is to model the policy implicitly in the network parameters, therefore allowing for complex stochastic policies.

Original: For this we model for each action dimension the quantile function
Revised: For this we model for each action dimension the quantile function

Original: G ˜ Z θ ( τ, o ) =
Revised: G ˜ Z θ ( τ, o ) =

Original: F − 1 ˜
Revised: F − 1 ˜

Original: Z θ ( τ, o ) of the implicitly deﬁned action distribution ˜
Revised: Z θ ( τ, o ) of the implicitly deﬁned action distribution ˜

Original: Z θ ( o ) by a neural networkwith an observation o and a target quantile τ ∈ [0 , 1] as input.
Revised: Z θ ( o ) by a neural network with an observation o and a target quantile τ ∈ [0 , 1] as input.

Original: An action a τ ∈
Revised: An action a τ ∈

Original: A ≡ R d , d beingthe number of action dimensions, can then be sampled by sampling τ ∼ U ([0 , 1] d ) and taking thenetwork outputs as action.
Revised: A ≡ R d , d being the number of action dimensions, can then be sampled by sampling τ ∼ U ([0 , 1] d ) and taking the network outputs as action.

Original: To train the networks quantile regression comes in handy.
Revised: To train the networks quantile regression comes in handy.

Original: Informallyput, quantile regression is linked to the Wasserstein metric [5] which is also sometimes referred toas earth movers distance .
Revised: Informally put, quantile regression is linked to the Wasserstein metric [5] which is also sometimes referred to as earth movers distance .

Original: Imagine a pile of earth representing probability mass.
Revised: Imagine a pile of earth representing probability mass.

Original: In RL we want tomove probability mass towards actions that were good and away from actions that were bad, where“good” and “bad” are measured by accumulative reward achieved.
Revised: In RL we want to move probability mass towards actions that were good and away from actions that were bad, where “good” and “bad” are measured by accumulative reward achieved.

Original: Quantile regression can achievethis neatly by shaping the pile of earth according to an advantage estimation and the constraint ofmonotonicity (a core property of quantile functions) 1 .
Revised: Quantile regression can achieve this neatly by shaping the pile of earth according to an advantage estimation and the constraint of monotonicity (a core property of quantile functions) 1 .

Original: This leads us to the QRRL actor objectivemin θ E a τ [ E τ (cid:48)
Revised: This leads us to the QRRL actor objective min θ E a τ [ E τ (cid:48)

Original: [ ρ τ (cid:48)
Revised: [ ρ τ (cid:48)

Original: ( a τ − G ˜
Revised: ( a τ − G ˜

Original: Z θ ( τ (cid:48) , o t )) · ( R t,n − V ψ ( o t )]] s.t. G
Revised: Z θ ( τ (cid:48) , o t )) · ( R t,n − V ψ ( o t )]] s.t. G

Original: ˜
Revised: ˜

Original: Z θ is monotonically increasing with τ (cid:48) where ( R t,n − V ψ ( o t )) is an advantage estimation [15] with R t,
Revised: Z θ is monotonically increasing with τ (cid:48) where ( R t,n − V ψ ( o t )) is an advantage estimation [15] with R t,n = γ n V ψ ( o t + n ) + (cid:80)

Original: [Equation]
Revised: [Equation]

Original: being the n -step estimate of the discounted future reward and V ψ ( · ) being the output of a criticnetwork with parameters ψ , which is trained to approximate R t,n .
Revised: being the n -step estimate of the discounted future reward and V ψ ( · ) being the output of a critic network with parameters ψ , which is trained to approximate R t,n .

Original: Note that QRRL is a constraintoptimization.
Revised: Note that QRRL is a constraint optimization.

Original: We found it however sufﬁcient to add this constraint as an additional loss term
Revised: We found it however sufﬁcient to add this constraint as an additional loss term

Original: L

Original: [Equation]
Revised: [Equation]

Original: where we use the Huber loss L Huber [9] for better convergence.
Revised: where we use the Huber loss L Huber [9] for better convergence.

Original: We weight this additional loss termwith a constant Lagrange multiplier λ mon in the full loss term
Revised: We weight this additional loss term with a constant Lagrange multiplier λ mon in the full loss term

Original: [Equation]
Revised: [Equation]

Original: )  with L a ( a τ , o t , τ (cid:48) , θ ) =
Revised: ) (2) with L a ( a τ , o t , τ (cid:48) , θ ) =

Original: ρ τ (cid:48)
Revised: ρ τ (cid:48)

Original: ( a τ − G ˜
Revised: ( a τ − G ˜

Original: Z θ ( τ (cid:48) , o )) · ( R t,n − V ψ ( o t )) and L c ( o t , ψ ) =
Revised: Z θ ( τ (cid:48) , o )) · ( R t,n − V ψ ( o t )) and L c ( o t , ψ ) =

Original: ( R t,n − V ψ ( o t )) 
Revised: ( R t,n − V ψ ( o t )) 2 .

Original: 2 .λ c is another constant Lagrange multiplier to weight the critic loss L c against the actor loss L a .
Revised: λ c is another constant Lagrange multiplier to weight the critic loss L c against the actor loss L a .

Original: In our State Aligned VEctor Reward (SAVER) agent, we use QRRL to train the agent network AN ηthrough the PCPN.
Revised: In our State Aligned VEctor Reward (SAVER) agent, we use QRRL to train the agent network AN η through the PCPN.

Original: For this we feed the action probability distribution output A of the agent networkto a pretrained PCPN and train on the QRRL loss (2) with respect to the agent network parametersη , where we take the actual position change ∆ p introduced by a sampled action a ∼ A as QRRLaction target a τ = ∆ p and compare it to K sampled position change estimations ∆ˆ p ( τ ( i ) ) , τ ( i ) ∼
Revised: For this we feed the action probability distribution output A of the agent network to a pretrained PCPN and train on the QRRL loss (2) with respect to the agent network parameters η , where we take the actual position change ∆ p introduced by a sampled action a ∼ A as QRRL action target a τ = ∆ p and compare it to K sampled position change estimations ∆ˆ p ( τ ( i ) ) , τ ( i ) ∼

Original: U ([0 , 1] d ) for i ∈ { 1 , ..., K } .
Revised: U ([0 , 1] d ) for i ∈ { 1 , ..., K } .

Original: Note that we pretrain the PCPN in our setup with random observationsand action probability distributions and freeze the PCPN weights during agent training.
Revised: Note that we pretrain the PCPN in our setup with random observations and action probability distributions and freeze the PCPN weights during agent training.

Original: Since theoutput of the PCPN can be seen as state aligned action, training on vector rewards is straight forward.
Revised: Since the output of the PCPN can be seen as state aligned action, training on vector rewards is straight forward.

Original: Instead of having a scalar critic estimating the value function V ( · ) ∈ R we estimate a value peraction dimension, i.e., V ( · ) ∈ R d .
Revised: Instead of having a scalar critic estimating the value function V ( · ) ∈ R we estimate a value per action dimension, i.e., V ( · ) ∈ R d .

Original: Similarly, the vector rewards can be summed to a vector n -stepdiscounted reward estimation R t,n = γ n V ( o t + n ) + (cid:80) t + n t (cid:48)
Revised: Similarly, the vector rewards can be summed to a vector n -step discounted reward estimation R t,n = γ n V ( o t + n ) + (cid:80) t + n t (cid:48)

Original: =
Revised: =

Original: t γ t (cid:48) − t r t (cid:48) , which leads us to a vectoradvantage R t,n − V ( o t ) at timestep t .
Revised: t γ t (cid:48) − t r t (cid:48) , which leads us to a vector advantage R t,n − V ( o t ) at timestep t .

Original: Therefore we can apply loss (2) to each position changeestimation dimension individually.
Revised: Therefore we can apply loss (2) to each position change estimation dimension individually.

Original:  Experiments and Results
Revised:  Experiments and Results

Original: To test our ideas, we implement a hypercube environment to model high dimensional metric spaces.
Revised: To test our ideas, we implement a hypercube environment to model high dimensional metric spaces.

Original: For proof of concept, we compare our SAVER agent against an A2C agent, the synchronous variantof A3C [15].
Revised: For proof of concept, we compare our SAVER agent against an A2C agent, the synchronous variant of A3C [15].

Original: We implement a simple feed forward neural architecture architecture and ﬁx allhyperparameters except the initial learning rate to values that work well for SAVER and A2C.
Revised: We implement a simple feed forward neural architecture architecture and ﬁx all hyperparameters except the initial learning rate to values that work well for SAVER and A2C.

Original: The
Revised: The

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: agent’s action executed in the environment consists of two parts: a softmax distribution from which the dimension to be manipulated is chosen (discrete action part) and a scalar Gaussian distribution deﬁned by network outputs µ and σ from which the step size is sampled (continuous action part).
Revised: agent’s action executed in the environment consists of two parts: a softmax distribution from which the dimension to be manipulated is chosen (discrete action part) and a scalar Gaussian distribution deﬁned by network outputs µ and σ from which the step size is sampled (continuous action part).

Original: The agent’s start position is random and the goal of the agent is to reach the origin of the d-dimensional space.
Revised: The agent’s start position is random and the goal of the agent is to reach the origin of the d-dimensional space.

Original: We pretrain the PCPN with 10,000 batches of 128 transitions each by sampling softmax logits and µ uniformly at random from U ([ − 1 , 1]) and σ from U ([0 , 1]) .
Revised: We pretrain the PCPN with 10,000 batches of 128 transitions each by sampling softmax logits and µ uniformly at random from U ([ − 1 , 1]) and σ from U ([0 , 1]) .

Original: We search for an appropriate learning rate in the 4 dimensional hypercube and settle for 0.01 for SAVER (chosen from { 0 . 01 , 0 . 003 , 0 . 001 } )and 0.0001 for A2C (chosen from { 0 . 001 , 0 . 0003 , 0 . 0001 } ).
Revised: We search for an appropriate learning rate in the 4 dimensional hypercube and settle for 0.01 for SAVER (chosen from { 0 . 01 , 0 . 003 , 0 . 001 } ) and 0.0001 for A2C (chosen from { 0 . 001 , 0 . 0003 , 0 . 0001 } ).

Original: The mean episode length – 100 episode mean averaged across 3 training runs – over the course of agent training is plotted in Figure 3.
Revised: The mean episode length – 100 episode mean averaged across 3 training runs – over the course of agent training is plotted in Figure 3.

Original: Notethat the higher dimensional the hypercube is, the more advantageous it is to train with vector rewards.
Revised: Note that the higher dimensional the hypercube is, the more advantageous it is to train with vector rewards.

Original:  Related Work
Revised:  Related Work

Original: Klinkhammer [10] discusses multiple problem aligned rewards for better learning, while Brys et al.[3] discuss the effect of correlated rewards on learning performance.
Revised: Klinkhammer [10] discusses multiple problem aligned rewards for better learning, while Brys et al. [3] discuss the effect of correlated rewards on learning performance.

Original: Brys et al.
Revised: Brys et al.

Original: [3] also suggestmultiple reward shapings of the same reward function for faster learning.
Revised: [3] also suggest multiple reward shapings of the same reward function for faster learning.

Original: Van Seijen et al.
Revised: Van Seijen et al.

Original: [20]decompose the reward function into multiple rewards and train an architecture similar to ours withdeep Q learning, assigning a Q-value output to each reward.
Revised: [20] decompose the reward function into multiple rewards and train an architecture similar to ours with deep Q learning, assigning a Q-value output to each reward.

Original: While all three approaches come tothe same conclusion as we do, i.e., increased training performance, they do require hand engineeredreward functions, reward shapings and/or reward decompositions.
Revised: While all three approaches come to the same conclusion as we do, i.e., increased training performance, they do require hand engineered reward functions, reward shapings and/or reward decompositions.

Original: In contrast, our approach is basedon the fact that many state spaces are metric spaces and therefore allow for a straightforward vectorreward interpretation.
Revised: In contrast, our approach is based on the fact that many state spaces are metric spaces and therefore allow for a straightforward vector reward interpretation.

Original: This makes our approach easier to apply to a larger set of tasks.
Revised: This makes our approach easier to apply to a larger set of tasks.

Original: While stateproximity was already used by McCallum [13] for faster backpropagation of rewards in tabular RL,we are unaware of any deep learning algorithm capitalizing on state aligned vector rewards as we do.
Revised: While state proximity was already used by McCallum [13] for faster backpropagation of rewards in tabular RL, we are unaware of any deep learning algorithm capitalizing on state aligned vector rewards as we do.

Original: Many recent approaches to deep RL learn the environment dynamics to have a richer training signal[6], imagine action consequences [19], improve exploration [17] or dream up solutions [8].
Revised: Many recent approaches to deep RL learn the environment dynamics to have a richer training signal [6], imagine action consequences [19], improve exploration [17] or dream up solutions [8].

Original: Aninteresting line of research [21, 1, 12, 2] in this direction uses successor features to share knowledgebetween tasks in the same environment.
Revised: An interesting line of research [21, 1, 12, 2] in this direction uses successor features to share knowledge between tasks in the same environment.

Original: In contrast to most of these works, which often only predictone possible next state or successor feature, our PCPN incorporates the full probability distributionof possible state changes.
Revised: In contrast to most of these works, which often only predict one possible next state or successor feature, our PCPN incorporates the full probability distribution of possible state changes.

Original: While Ha and Schmidhuber [8] also predict the full probability distributionof their next state representation by a Gaussian mixture model, our approach is more general in that itis also able to approximate non-Gaussian probability distributions.
Revised: While Ha and Schmidhuber [8] also predict the full probability distribution of their next state representation by a Gaussian mixture model, our approach is more general in that it is also able to approximate non-Gaussian probability distributions.

Original:  Conclusion
Revised:  Conclusion

Original: In this work we present the idea of state aligned vector rewards for faster reinforcement learning.
Revised: In this work we present the idea of state aligned vector rewards for faster reinforcement learning.

Original: Additionally, we also present a new RL technique we call quantile regression reinforcement learning(QRRL).
Revised: Additionally, we also present a new RL technique we call quantile regression reinforcement learning (QRRL).

Original: QRRL allows for complex stochastic policies in continuous action spaces, not limitingagents to Gaussian actions.
Revised: QRRL allows for complex stochastic policies in continuous action spaces, not limiting agents to Gaussian actions.

Original: Combining both, we show that our SAVER agent can be trained througha quantile network pretrained in the environment.
Revised: Combining both, we show that our SAVER agent can be trained through a quantile network pretrained in the environment.

Original: Our SAVER agent is capable of training orders ofmagnitudes faster in high dimensional metric spaces.
Revised: Our SAVER agent is capable of training orders of magnitudes faster in high dimensional metric spaces.