Original: H OW MANY WEIGHTS ARE ENOUGH : CAN TENSOR FACTORIZATION LEARN EFFICIENT POLICIES ?
Revised: H OW MANY WEIGHTS ARE ENOUGH : CAN TENSOR FACTORIZATION LEARN EFFICIENT POLICIES ?

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep reinforcement learning requires a heavy price in terms of sample efﬁciency and overparameterization in the neural networks used for function approximation.
Revised: Deep reinforcement learning requires a heavy price in terms of sample efﬁciency and overparameterization in the neural networks used for function approximation.

Original: In this work, we employ tensor factorization in order to learn more compact representations for reinforcement learning policies.
Revised: In this work, we employ tensor factorization in order to learn more compact representations for reinforcement learning policies.

Original: We show empirically that in the low-data regime, it is possible to learn online policies with 2 to 10 times less total coefﬁcients, with little to no loss of performance.
Revised: We show empirically that in the low-data regime, it is possible to learn online policies with 2 to 10 times less total coefﬁcients, with little to no loss of performance.

Original: We also leverage progress in second order optimization, and use the theory of wavelet scattering to further reduce the number of learned coefﬁcients, by foregoing learning the topmost convolutional layer ﬁlters altogether.
Revised: We also leverage progress in second order optimization, and use the theory of wavelet scattering to further reduce the number of learned coefﬁcients, by foregoing learning the topmost convolutional layer ﬁlters altogether.

Original: We evaluate our results on the Atari suite against recent baseline algorithms that represent the state-of-the-art in data efﬁciency, and get comparable results with an order of magnitude gain in weight parsimony.
Revised: We evaluate our results on the Atari suite against recent baseline algorithms that represent the state-of-the-art in data efﬁciency, and get comparable results with an order of magnitude gain in weight parsimony.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The successes of reinforcement learning (thereafter ’RL’), and speciﬁcally deep RL, come at a heavy computational price.
Revised: The successes of reinforcement learning (thereafter ’RL’), and speciﬁcally deep RL, come at a heavy computational price.

Original: It is well known that achieving human-level performance in domains such as
Revised: It is well known that achieving human-level performance in domains such as

Original: Atari (Sutton & Barto, 2018; Mnih et al., 2013; Hessel et al., 2017) requires hundreds of millions of frames of environment interaction.
Revised: Atari (Sutton & Barto, 2018; Mnih et al., 2013; Hessel et al., 2017) requires hundreds of millions of frames of environment interaction.

Original: As such, the problem of sample efﬁciency in RL is of critical importance.
Revised: As such, the problem of sample efﬁciency in RL is of critical importance.

Original: Several tracks of concurrent research are being investigated, and have reduced by orders of magnitude the number of environment interactions required for good performance beyond the previous benchmark of biologically-inspired episodic control methods (Blundell et al., 2016; Pritzel et al., 2017) to a couple hours of human gameplay time (van Hasselt et al., 2019; Kaiser et al., 2019).
Revised: Several tracks of concurrent research are being investigated, and have reduced by orders of magnitude the number of environment interactions required for good performance beyond the previous benchmark of biologically-inspired episodic control methods (Blundell et al., 2016; Pritzel et al., 2017) to a couple hours of human gameplay time (van Hasselt et al., 2019; Kaiser et al., 2019).

Original: However, while the data-efﬁciency of RL methods has seen recent drastic performance gains, the function approximators they use still require millions of learned weights, potentially still leaving them heavily overparameterized.
Revised: However, while the data-efﬁciency of RL methods has seen recent drastic performance gains, the function approximators they use still require millions of learned weights, potentially still leaving them heavily overparameterized.

Original: Independently motivated by biological facts like the behavioural readiness of newborn animals, several authors (Gaier & Ha, 2019; Cuccu et al., 2018; Wang et al., 2019) have recently looked at doing away with learning so many weights for RL tasks.
Revised: Independently motivated by biological facts like the behavioural readiness of newborn animals, several authors (Gaier & Ha, 2019; Cuccu et al., 2018; Wang et al., 2019) have recently looked at doing away with learning so many weights for RL tasks.

Original: Smaller networks not only train faster, but may yet offer another avenue for gains in the form of better generalization (Zhang et al., 2016).
Revised: Smaller networks not only train faster, but may yet offer another avenue for gains in the form of better generalization (Zhang et al., 2016).

Original: Recent work from Gaier & Ha (2019) studies the effect of inductive bias of neural architectures in RL ; they forego training altogether, but transfer networks that only obtain ’better than chance performance on MNIST’.
Revised: Recent work from Gaier & Ha (2019) studies the effect of inductive bias of neural architectures in RL ; they forego training altogether, but transfer networks that only obtain ’better than chance performance on MNIST’.

Original: In similar fashion, Wang et al.
Revised: In similar fashion, Wang et al.

Original: (2019) investigate the effect of random projections in the restricted setting of imitation learning.
Revised: (2019) investigate the effect of random projections in the restricted setting of imitation learning.

Original: Finally, Cuccu et al.
Revised: Finally, Cuccu et al.

Original: (2018) manage human-level performance on the Atari suite using a separate dictionary learning procedure for their features, bypassing the usual end-to-end learning paradigm.
Revised: (2018) manage human-level performance on the Atari suite using a separate dictionary learning procedure for their features, bypassing the usual end-to-end learning paradigm.

Original: The perspective of neural architecture search applied to RL appears difﬁcult, if not computationally inextricable.
Revised: The perspective of neural architecture search applied to RL appears difﬁcult, if not computationally inextricable.

Original: Concurrently, the study of biologically-inspired models of learning has exhibited two mathematical characterizations that might be critical in explaining how biological learning takes place so efﬁciently.
Revised: Concurrently, the study of biologically-inspired models of learning has exhibited two mathematical characterizations that might be critical in explaining how biological learning takes place so efﬁciently.

Original: First, the low-rank properties of learned perceptual manifolds (Chung et al., 2018; 2016) are giving rise to a rich theory borrowing from statistical physics.
Revised: First, the low-rank properties of learned perceptual manifolds (Chung et al., 2018; 2016) are giving rise to a rich theory borrowing from statistical physics.

Original: Second, another well known line of work has identiﬁed Gabor ﬁlters, and more generally wavelet ﬁlter-like structures, in the actual visual cortex of animals (Jones & Palmer, 1987), and linked those to sparsity-promoting methods and dictionary learning (Olshausen & Field, 1996; 1997; Hyv¨arinen & Hoyer, 2001).
Revised: Second, another well known line of work has identiﬁed Gabor ﬁlters, and more generally wavelet ﬁlter-like structures, in the actual visual cortex of animals (Jones & Palmer, 1987), and linked those to sparsity-promoting methods and dictionary learning (Olshausen & Field, 1996; 1997; Hyv¨arinen & Hoyer, 2001).

Original: But these breakthroughs have not, so far, been reﬂected as inductive priors in the shape of modiﬁcations in deep RL neural networks architectures, which remain fairly ﬁxed on the Atari domain.
Revised: But these breakthroughs have not, so far, been reﬂected as inductive priors in the shape of modiﬁcations in deep RL neural networks architectures, which remain fairly ﬁxed on the Atari domain.

Original: Therefore the following questions remain: how parsimonious do function approximators in RL need to be, in order to maintain good performance?
Revised: Therefore the following questions remain: how parsimonious do function approximators in RL need to be, in order to maintain good performance?

Original: And can we be at once sample-efﬁcient and weightefﬁcient ?
Revised: And can we be at once sample-efﬁcient and weightefﬁcient ?

Original: In this work, we turn to the mathematical theories of tensor factorization (Cichocki et al., 2009), second-order optimization (Amari, 1998; Martens & Grosse, 2015) and wavelet scattering (Mallat, 2011) to answer this question positively and empirically, in a model-free setting.
Revised: In this work, we turn to the mathematical theories of tensor factorization (Cichocki et al., 2009), second-order optimization (Amari, 1998; Martens & Grosse, 2015) and wavelet scattering (Mallat, 2011) to answer this question positively and empirically, in a model-free setting.

Original: We propose to use these methods in order to save weights and therefore favour convergence of policies:
Revised: We propose to use these methods in order to save weights and therefore favour convergence of policies:

Original: • We replace dense, fully-connected layers with tensor regression layers .
Revised: • We replace dense, fully-connected layers with tensor regression layers .

Original: • Optionally, we replace the topmost layer in the convolutional architecture with a scattering layer; the deeper convolutional layers are left untouched.
Revised: • Optionally, we replace the topmost layer in the convolutional architecture with a scattering layer; the deeper convolutional layers are left untouched.

Original: • The (positive) impact of second-order optimization is also evaluated.
Revised: • The (positive) impact of second-order optimization is also evaluated.

Original: To the best of our knowledge, this is the ﬁrst time those ﬁelds have been combined together in this context, and that tensor factorization is applied to deep RL.
Revised: To the best of our knowledge, this is the ﬁrst time those ﬁelds have been combined together in this context, and that tensor factorization is applied to deep RL.

Original:  B ACKGROUND & R ELATED WORK 2. D EEP R EINFORCEMENT L EARNING
Revised:  B ACKGROUND & R ELATED WORK 2. D EEP R EINFORCEMENT L EARNING

Original: We consider the standard Markov Decision Process framework as in Sutton & Barto (2018).
Revised: We consider the standard Markov Decision Process framework as in Sutton & Barto (2018).

Original: This setting is characterised by a tuple (cid:104) S, A, T, R, γ (cid:105) , where S is a set of states, A a set of actions,
Revised: This setting is characterised by a tuple (cid:104) S, A, T, R, γ (cid:105) , where S is a set of states, A a set of actions,

Original: R a reward function that is the immediate, intrinsic desirability of a certain state, T a transition dynamics and γ ∈ [0 , 1] a discount factor.
Revised: R a reward function that is the immediate, intrinsic desirability of a certain state, T a transition dynamics and γ ∈ [0 , 1] a discount factor.

Original: The purpose of the RL problem is to to ﬁnd a policy π , which represents a mapping from states to a probability distribution over actions, that is optimal, i.e., that maximizes the expected cumulative discounted return (cid:80) ∞ k =

Revised: The purpose of the RL problem is to to ﬁnd a policy π , which represents a mapping from states to a probability distribution over actions, that is optimal, i.e., that maximizes the expected cumulative discounted return (cid:80) ∞ k =

Original: γ k R t + k +1 at each state s t ∈ S .
Revised: γ k R t + k +1 at each state s t ∈ S .

Original: In Q-learning, the policy is given implicitly by acting greedily or (cid:15) -greedily with respect to learned action-value functions q π ( s, a ) , that are learned following the Bellman equation.
Revised: In Q-learning, the policy is given implicitly by acting greedily or (cid:15) -greedily with respect to learned action-value functions q π ( s, a ) , that are learned following the Bellman equation.

Original: In deep Q-learning , q θ becomes parameterized by the weights θ of a neural network and one minimizes the expected
Revised: In deep Q-learning , q θ becomes parameterized by the weights θ of a neural network and one minimizes the expected

Original: Bellman loss :
Revised: Bellman loss :

Original: [Equation]
Revised: [Equation]

Original: In practice, this is implemented stochastically via uniform sampling of transitions in an experience replay buffer, as is done in the seminal paper Mnih et al.
Revised: In practice, this is implemented stochastically via uniform sampling of transitions in an experience replay buffer, as is done in the seminal paper Mnih et al.

Original: Several algorithmic reﬁnements to that approach exist.
Revised: Several algorithmic reﬁnements to that approach exist.

Original: First, Double Q-learning (van Hasselt et al., 2015) proposes to decouple learning between two networks in order to alleviate the Q-value overestimation problem.
Revised: First, Double Q-learning (van Hasselt et al., 2015) proposes to decouple learning between two networks in order to alleviate the Q-value overestimation problem.

Original: Second, dueling Q-networks (Wang et al., 2015) explicitly decompose the learning of an action-value function q θ ( s, a ) as the sum of an action-independent state-value, much like what is traditionally done in policy gradient methods (Sutton & Barto, 2018), implemented via a two-headed neural network architecture.
Revised: Second, dueling Q-networks (Wang et al., 2015) explicitly decompose the learning of an action-value function q θ ( s, a ) as the sum of an action-independent state-value, much like what is traditionally done in policy gradient methods (Sutton & Barto, 2018), implemented via a two-headed neural network architecture.

Original: Finally, prioritized RL (Schaul et al., 2015) proposes to replace the uniform sampling of transitions in the experience replay buffer with importance sampling, by prioritizing those transitions that present the most Bellman error (those transitions that are deemed the most ’surprising’ by the agent).
Revised: Finally, prioritized RL (Schaul et al., 2015) proposes to replace the uniform sampling of transitions in the experience replay buffer with importance sampling, by prioritizing those transitions that present the most Bellman error (those transitions that are deemed the most ’surprising’ by the agent).

Original: Fortunato et al.
Revised: Fortunato et al.

Original: (2017) use extra weights to learn the variance of the exploration noise in a granular fashion, while Bellemare et al.
Revised: (2017) use extra weights to learn the variance of the exploration noise in a granular fashion, while Bellemare et al.

Original: (2017) propose to learn a full distribution of action-values for each action and state.
Revised: (2017) propose to learn a full distribution of action-values for each action and state.

Original: Combined, those methods form the basis of the Rainbow algorithm in
Revised: Combined, those methods form the basis of the Rainbow algorithm in

Original: Hessel et al.
Revised: Hessel et al.

Original: 2. T ENSOR FACTORIZATION
Revised: 2. T ENSOR FACTORIZATION

Original: Here we introduce notations and concepts from the tensor factorization literature.
Revised: Here we introduce notations and concepts from the tensor factorization literature.

Original: An intuition is that the two main decompositions below, CP and Tucker decompositions, can be understood as multilinear algebra analogues of SVD or eigendecomposition.
Revised: An intuition is that the two main decompositions below, CP and Tucker decompositions, can be understood as multilinear algebra analogues of SVD or eigendecomposition.

Original: CP decomposition.
Revised: CP decomposition.

Original: A tensor X ∈ R I 

Revised: A tensor X ∈ R I

Original: ×
Revised: ×

Original: I 2 ×···× I N , can be decomposed into a sum of R ranktensors, known as the Canonical-Polyadic decomposition, where R is the rank of the decomposition.
Revised: I 2 ×···× I N , can be decomposed into a sum of R ranktensors, known as the Canonical-Polyadic decomposition, where R is the rank of the decomposition.

Original: Its purpose is to ﬁnd vectors u (1) k , u (2) k , · · · , u ( N ) k , for k =
Revised: Its purpose is to ﬁnd vectors u (1) k , u (2) k , · · · , u ( N ) k , for k =

Original: R ] , as well as a vector of weights λ ∈ R R such that:
Revised: R ] , as well as a vector of weights λ ∈ R R such that:

Original: [Equation]
Revised: [Equation]

Original: Tucker decomposition.
Revised: Tucker decomposition.

Original: A tensor X ∈ R I 

Revised: A tensor X ∈ R I

Original: ×
Revised: ×

Original: I 2 ×···× I N , can be decomposed into a low rank approximation including a core G ∈ R R 1 × R 2 ×···× R N and a set of projection factors (cid:0) U (0) , · · · , U ( N − 1) (cid:1) , with U ( k ) ∈ R R k , ˆ I k , k ∈ (0 , · · · , N − 1) that, when projected along the corresponding dimension of the core, reconstruct the full tensor X .
Revised: I 2 ×···× I N , can be decomposed into a low rank approximation including a core G ∈ R R 1 × R 2 ×···× R N and a set of projection factors (cid:0) U (0) , · · · , U ( N − 1) (cid:1) , with U ( k ) ∈ R R k , ˆ I k , k ∈ (0 , · · · , N − 1) that, when projected along the corresponding dimension of the core, reconstruct the full tensor X .

Original: The tensor in its decomposed form can be written:
Revised: The tensor in its decomposed form can be written:

Original: [Equation]
Revised: [Equation]

Original: Tensor regression layer.
Revised: Tensor regression layer.

Original: For two tensors X 
Revised: For two tensors X ∈

Original: [Equation]
Revised: [Equation]

Original:  Y ∈ R
Revised: and Y ∈ R

Original: [Equation]
Revised: [Equation]

Original: L y , we denote by (cid:104)X , Y(cid:105) N
Revised: L y , we denote by (cid:104)X , Y(cid:105) N

Original: [Equation]
Revised: [Equation]

Original:  the contraction of X by Y along their N last modes; their generalized inner product is
Revised: y the contraction of X by Y along their N last modes; their generalized inner product is

Original: [Equation]
Revised: [Equation]

Original: This makes it possible to deﬁne a tensor regression layer (Kossaiﬁ et al., 2017b) that is differentiable and learnable end-to-end by gradient descent.
Revised: This makes it possible to deﬁne a tensor regression layer (Kossaiﬁ et al., 2017b) that is differentiable and learnable end-to-end by gradient descent.

Original: Let us denote by X ∈ R I 

Revised: Let us denote by X ∈ R I

Original: ×
Revised: ×

Original: I 2 ×···×
Revised: I 2 ×···×

Original: I N the input activation tensor for a sample and y ∈ R
Revised: I N the input activation tensor for a sample and y ∈ R

Original: I N the label vector.
Revised: I N the label vector.

Original: A tensor regression layer estimates the regression weight tensor W ∈ R
Revised: A tensor regression layer estimates the regression weight tensor W ∈ R

Original: I 

Revised: I

Original: ×
Revised: ×

Original: I 2 ×···× I N under a low-rank decomposition.
Revised: I 2 ×···× I N under a low-rank decomposition.

Original: For instance, with a
Revised: In the case of a

Original: Tucker decomposition  with ranks ( R 1 , · · · , R N ) , we have :
Revised: Tucker decomposition (as per our experiments) with ranks ( R 1 , · · · , R N ) , we have :

Original: [Equation]
Revised: [Equation]

Original: W
Revised: G

Original: [Equation]
Revised: [Equation]

Original: G × 1 U

Original: [Equation]

Original: [Equation]

Original: as G ∈ R R 1 ×···× R N ,
Revised: as G ∈ R R 1 ×···× R N ,

Original: U ( k ) ∈
Revised: U ( k ) ∈

Original: R
Revised: R

Original: I k
Revised: I k

Original: × R k for each k in [1 . . .
Revised: × R k for each k in [1 . . .

Original: N ] and U ( N ) ∈ R 1 × R N .
Revised: N ] and U ( N ) ∈ R 1 × R N .

Original: 2. W AVELET SCATTERING
Revised: 2. W AVELET SCATTERING

Original: The wavelet scattering transform was originally introduced by Mallat (2011) and Bruna & Mallat (2012) as a non-linear extension to the classical wavelet ﬁlter bank decomposition.
Revised: The wavelet scattering transform was originally introduced by Mallat (2011) and Bruna & Mallat (2012) as a non-linear extension to the classical wavelet ﬁlter bank decomposition.

Original: Its principle is as follows.
Revised: Its principle is as follows.

Original: Denoting by x (cid:126) y [ n ] the 2-dimensional, circular convolution of two signals x [ n ] and y [ n ] , let us assume that we have pre-deﬁned two wavelet ﬁlter banks available ¶ ψ (1) λ 1 [ n ] © λ 1 ∈ Λ¶ ψ (2) λ 2 [ n ] © λ 2 ∈ Λ 2 , with λ 1 and λ 2 two frequency indices.
Revised: Denoting by x (cid:126) y [ n ] the 2-dimensional, circular convolution of two signals x [ n ] and y [ n ] , let us assume that we have pre-deﬁned two wavelet ﬁlter banks available ¶ ψ (1) λ 1 [ n ] © λ 1 ∈ Λ¶ ψ (2) λ 2 [ n ] © λ 2 ∈ Λ 2 , with λ 1 and λ 2 two frequency indices.

Original: These wavelet ﬁlters correspond to high frequencies, so we also give ourselves the data of a lowpass ﬁlter φ J [ n ] .
Revised: These wavelet ﬁlters correspond to high frequencies, so we also give ourselves the data of a lowpass ﬁlter φ J [ n ] .

Original: Finally, and by opposition to traditional linear wavelet transforms, we also assume a given nonlinearity ρ ( t ) .
Revised: Finally, and by opposition to traditional linear wavelet transforms, we also assume a given nonlinearity ρ ( t ) .

Original: Then the scattering transform is given by coefﬁcients of order 0,1, and 2, respectively :
Revised: Then the scattering transform is given by coefﬁcients of order 0,1, and 2, respectively :

Original: [Equation]
Revised: [Equation]

Original: This can effectively be understood and implemented as a two-layer convolutional neural network whose weights are not learned but rather frozen and given by the coefﬁcients of wavelets ψ and φ (with Gabor ﬁlters as a special case (Mallat, 1998)).
Revised: This can effectively be understood and implemented as a two-layer convolutional neural network whose weights are not learned but rather frozen and given by the coefﬁcients of wavelets ψ and φ (with Gabor ﬁlters as a special case (Mallat, 1998)).

Original: The difference with traditional ﬁlter banks comes from the iterated modulus/nonlinear activation function applied at each stage, much like in traditional deep learning convolutional neural networks.
Revised: The difference with traditional ﬁlter banks comes from the iterated modulus/nonlinear activation function applied at each stage, much like in traditional deep learning convolutional neural networks.

Original: The generic mathematical deﬁnition involves order n iterated scatterings, in the vein of S
Revised: The generic mathematical deﬁnition involves order n iterated scatterings, in the vein of S

Original: i x above, but sometimes restricts nonlinearity ρ to be a modulus function |·| .
Revised: i x above, but sometimes restricts nonlinearity ρ to be a modulus function |·| .

Original: In practice, the potential of scattering transforms to accelerate learning by providing ready-made convolutional layers has been investigated in Oyallon et al.
Revised: In practice, the potential of scattering transforms to accelerate learning by providing ready-made convolutional layers has been investigated in Oyallon et al.

Original: and
Revised: and

Original: Oyallon et al.
Revised: Oyallon et al.

Original: and is a subject of active ongoing research.
Revised: and is a subject of active ongoing research.

Original: Scattering will be the second of our weight-saving methods.
Revised: Scattering will be the second of our weight-saving methods.

Original: 2. S ECOND ORDER OPTIMIZATION WITH K-FAC
Revised: 2. S ECOND ORDER OPTIMIZATION WITH K-FAC

Original: While stochastic gradient descent is usually performed purely from gradient observations derived from auto-differentiation, faster, second order optimization methods ﬁrst multiply the weights’ θ gradient vector ∇ θ by a preconditioning matrix, yielding the weight update θ n +1 ← θ n − ηG − 1 n ∇ θ , with η a step size.
Revised: While stochastic gradient descent is usually performed purely from gradient observations derived from auto-differentiation, faster, second order optimization methods ﬁrst multiply the weights’ θ gradient vector ∇ θ by a preconditioning matrix, yielding the weight update θ n +1 ← θ n − ηG − 1 n ∇ θ , with η a step size.

Original: In the case of second order methods, the matrix G − 1 n is chosen to act as a tractable iterative approximation to the inverse Hessian or Empirical Fisher Information Matrix (Amari, 1998) of the neural network model in question.
Revised: In the case of second order methods, the matrix G − 1 n is chosen to act as a tractable iterative approximation to the inverse Hessian or Empirical Fisher Information Matrix (Amari, 1998) of the neural network model in question.

Original: Kronecker-factored approximate curvature or K-FAC (Martens & Grosse, 2015) enforces a Kronecker decomposition of the type G =
Revised: Kronecker-factored approximate curvature or K-FAC (Martens & Grosse, 2015) enforces a Kronecker decomposition of the type G =

Original: A ⊗ B , with A and B  smaller, architecture-dependent matrices.
Revised: A ⊗ B , with A and B being smaller, architecture-dependent matrices.

Original: Unlike the above methods, K-FAC has been applied as a plug-in in the deep RL literature and shown to promote both anytime convergence properties as well as terminal accuracies (Wu et al., 2017).
Revised: Unlike the above methods, K-FAC has been applied as a plug-in in the deep RL literature and shown to promote both anytime convergence properties as well as terminal accuracies (Wu et al., 2017).

Original:  O BSERVATIONS AND METHODS 3. E XPLORING TRAINED AGENTS
Revised:  O BSERVATIONS AND METHODS 3. E XPLORING TRAINED AGENTS

Original: Stability of trained dense layers eigenvalues.
Revised: Stability of trained dense layers eigenvalues.

Original: In order to assess experimentally if tensor factorization can make sense in RL, we investigate the eigenvalues of the dense layers of a deep RL agent.
Revised: In order to assess experimentally if tensor factorization can make sense in RL, we investigate the eigenvalues of the dense layers of a deep RL agent.

Original: Unlike the traditional supervised learning setting, the input data distribution to RL function approximators shifts as the agent explores its environment; as such, concentration properties of the eigenvalues of the linear layers cannot be guaranteed all the way throughout training.
Revised: Unlike the traditional supervised learning setting, the input data distribution to RL function approximators shifts as the agent explores its environment; as such, concentration properties of the eigenvalues of the linear layers cannot be guaranteed all the way throughout training.

Original: This is all the more pressing that conditioning techniques such as batch normalization (Ioffe & Szegedy, 2015) are rarely used in deep RL.
Revised: Since conditioning techniques such as batch normalization (Ioffe & Szegedy, 2015) are rarely used in deep RL, this is all the more important.

Original: Our experiments (see ﬁgure 1) show that the distribution of eigenvalues does not seem to widen signiﬁcantly, at least during the initial phases of training we care about.
Revised: Our experiments (see ﬁgure 1) show that the distribution of eigenvalues does not seem to widen signiﬁcantly, at least during the initial phases of training we care about.

Original: [Figure]
Revised: [Figure]

Original: Furthermore and interestingly, it does not seem to deviate signiﬁcantly from the one observed at initialization.
Revised: Furthermore and interestingly, it does not seem to deviate signiﬁcantly from the one observed at initialization.

Original: All together, this suggests there might be some merit in learning low-rank policies.
Revised: All together, this suggests there might be some merit in learning low-rank policies.

Original: 3. A RCHITECTURAL MODIFICATIONS
Revised: 3. A RCHITECTURAL MODIFICATIONS

Original: Baseline.
Revised: Baseline.

Original: We then proceed to build upon the well-known Rainbow algorithm (Hessel et al., 2017).
Revised: We then proceed to build upon the well-known Rainbow algorithm (Hessel et al., 2017).

Original: Rainbow uses a fairly standard shallow convolutional architecture like the seminal DQN paper of
Revised: Rainbow uses a fairly standard shallow convolutional architecture like the seminal DQN paper of

Original: Mnih et al.
Revised: Mnih et al.

Original: and is an oft-cited baseline on the Atari suite.
Revised: and is an oft-cited baseline on the Atari suite.

Original: In spite of its performance, Rainbow often requires dozens of millions of a single game’s frames in order to perform well.
Revised: In spite of its performance, Rainbow often requires dozens of millions of a single game’s frames in order to perform well.

Original: Very recently, a ’data-efﬁcient’ efﬁcient version of Rainbow has been proposed by van Hasselt et al.
Revised: Very recently, a ’data-efﬁcient’ efﬁcient version of Rainbow has been proposed by van Hasselt et al.

Original: (2019), with a view to match or beat the latest state-of-the-art results achieved by model-based RL methods.
Revised: (2019), with a view to match or beat the latest state-of-the-art results achieved by model-based RL methods.

Original: This is achieved with no major change in network architecture, but via a selection of mildly handtuned hyperparameters favouring efﬁciency against wall-clock running time (see appendix).
Revised: This is achieved with no major change in network architecture, but via a selection of mildly handtuned hyperparameters favouring efﬁciency against wall-clock running time (see appendix).

Original: We do take this as a baseline.
Revised: We do take this as a baseline.

Original: Changes.
Revised: Changes.

Original: We modify the architecture of the neural network function approximators used, in accordance with the principles described above, combining them to reﬂect inductive biases promoting fewer learnable parameters:
Revised: We modify the architecture of the neural network function approximators used, in accordance with the principles described above, combining them to reﬂect inductive biases promoting fewer learnable parameters:

Original: • We replace the fully-connected, linear layers used in Rainbow and data-efﬁcient Rainbow with tensor regression layers (Kossaiﬁ et al., 2017b)
Revised: • We replace the fully-connected, linear layers used in Rainbow and data-efﬁcient Rainbow with tensor regression layers (Kossaiﬁ et al., 2017b)

Original: in order to learn low-rank policies (ranks in appendix).
Revised: in order to learn low-rank policies (ranks in appendix).

Original: • We use either the K-FAC second order stochastic optimizer, or the standard ADAM optimizer (Kingma & Ba, 2014).
Revised: • We use either the K-FAC second order stochastic optimizer, or the standard ADAM optimizer (Kingma & Ba, 2014).

Original: Optimization with K-FAC yields better results ceteris paribus and therefore works to counter performance loss due to using fewer weights.
Revised: Optimization with K-FAC yields better results ceteris paribus and therefore works to counter performance loss due to using fewer weights.

Original: • We combine the two methods with various rank (and therefore weight compression) ratios, targetting sensible compression ratios guided by deep learning intuition; and evaluate those on the same subset of Atari games as both van Hasselt et al.
Revised: • We combine the two methods with various rank (and therefore weight compression) ratios, targetting sensible compression ratios guided by deep learning intuition; and evaluate those on the same subset of Atari games as both van Hasselt et al.

Original: Kaiser et al.
Revised: Kaiser et al.

Original: • When possible, we replace the ﬁrst convolutional layer in the approximating neural network with a scattering layer for further gains in terms of learnable weights.
Revised: • When possible, we replace the ﬁrst convolutional layer in the approximating neural network with a scattering layer for further gains in terms of learnable weights.

Original: In that way, we investigate the impact of not actually learning one of the convolutional layer weights.
Revised: In that way, we investigate the impact of not actually learning one of the convolutional layer weights.

Original: Deep RL, paradoxically, tends to use shallow network architectures with two or three convolutional layers only, which makes it very ﬁt for scattering methods.
Revised: Deep RL, paradoxically, tends to use shallow network architectures with two or three convolutional layers only, which makes it very ﬁt for scattering methods.

Original: However, one must not forget that the convolutionallayers one learns must be somehow be well adapted to featuresthat discriminate  between high and low rewards; the promise of fully unsupervised scattering layers seems remote, without resorting to further ad-hoc methods like dictionary learning.
Revised: But since learned convolutional features discriminate well between high and low rewards, the promise of fully unsupervised scattering layers seems remote, without resorting to further ad-hoc methods like dictionary learning.

Original: Therefore we limit ourselves to one-step (one single layer) scatterings.
Revised: Therefore we limit ourselves to one-step (one single layer) scatterings.

Original: This is illustrated in ﬁgure 2.
Revised: This is illustrated in ﬁgure 2.

Original: We then proceed to evaluate the merit of these changes.
Revised: We then proceed to evaluate the merit of these changes.

Original:  E XPERIMENTAL RESULTS 4. P RIORITIZED TENSORIZED DQN 
Revised:  E XPERIMENTAL RESULTS 4. P RIORITIZED TENSORIZED DQN Proof

Original: Proof of concept.
Revised: of concept.

Original: We begin with showing proof of concept on the simple Pong Atari game.
Revised: We begin with showing proof of concept on the simple Pong Atari game.

Original: Our experimental setup consists in our own implementation of prioritized double DQN as a baseline (Schaul et al., 2015; van Hasselt et al., 2015).
Revised: Our experimental setup consists in our own implementation of prioritized double DQN as a baseline (Schaul et al., 2015; van Hasselt et al., 2015).

Original: We replaced the densely connected layer of the original DQN architecture with a tensor regression layer implementing Tucker decomposition for different Tucker ranks, yielding different network compression factors.
Revised: We replaced the densely connected layer of the original DQN architecture with a tensor regression layer implementing Tucker decomposition for different Tucker ranks, yielding different network compression factors.

Original: Qualitative behaviour.
Revised: Qualitative behaviour.

Original: First results, both in terms of learning performance and compression factor, can be seen in ﬁgure 3.
Revised: First results, both in terms of learning performance and compression factor, can be seen in ﬁgure 3.

Original: Our two main ﬁndings are that ﬁrst, the ﬁnal performance of the agent remains unaffected by the tensor factorization, even with high compression rates  of ﬁve times.
Revised: Our two main ﬁndings are that ﬁrst, the ﬁnal performance of the agent remains unaffected by the tensor factorization, even with high compression rates - with respect to all network weights - of ﬁve times.

Original: In line with intuition, larger compression rates do however cause more delays in learning.
Revised: In line with intuition, larger compression rates do however cause more delays in learning.

Original: Second, tensor factorization negatively affects stability during training - in tough compression regimes, the plateauing phases of learning curves feature occasional noisy drawdowns, illustrating the increased difﬁculty of learning, as seen in ﬁgure 4.
Revised: Second, tensor factorization negatively affects stability during training - in tough compression regimes, the plateauing phases of learning curves feature occasional noisy drawdowns, illustrating the increased difﬁculty of learning, as seen in ﬁgure 4.

Original: Interestingly, approximation errors incurred by tensor regression noise do sometimes have poor consequences illustrated by those drawdowns, but overall seem to behave as additional exploration noise.
Revised: Interestingly, approximation errors incurred by tensor regression noise do sometimes have poor consequences illustrated by those drawdowns, but overall seem to behave as additional exploration noise.

Original: [Figure]
Revised: [Figure]

Original: 4. D ATA - EFFICIENT R AINBOW ON A TARI
Revised: 4. D ATA - EFFICIENT R AINBOW ON A TARI

Original: Evaluation protocol.
Revised: Evaluation protocol.

Original: For all our Atari experiments, we used OpenAI
Revised: For all our Atari experiments, we used OpenAI

Original: Gym (Brockman et al., 2016), and a combination of PyTorch (Paszke et al., 2017), TensorLy (Kossaiﬁ et al., 2016) and Kymatio (Andreux et al., 2018) for auto-differentiation.
Revised: Gym (Brockman et al., 2016), and a combination of PyTorch (Paszke et al., 2017), TensorLy (Kossaiﬁ et al., 2016) and Kymatio (Andreux et al., 2018) for auto-differentiation.

Original: We evaluated our agents in the low-data regime of 100,000 steps, on half the games, with 3 different random seeds for reproducibility (Henderson et al., 2017), taking the data-efﬁcient Rainbow agent (van Hasselt et al., 2019) as our baseline.
Revised: We evaluated our agents in the low-data regime of 100,000 steps, on half the games, with 3 different random seeds for reproducibility (Henderson et al., 2017), taking the data-efﬁcient Rainbow agent (van Hasselt et al., 2019) as our baseline.

Original: Our speciﬁc hyperparameters are described in appendix.
Revised: Our speciﬁc hyperparameters are described in appendix.

Original: We report our results in tables 1 and
Revised: We report our results in tables 1 and

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Table 1 shows proof of concept of the online learning of low-rank policies, with a loss of ﬁnal performance varying in proportion to the compression in the low-rank linear layers, very much like in the deep learning literature (Kossaiﬁ et al., 2017a;b).
Revised: Table 1 shows proof of concept of the online learning of low-rank policies, with a loss of ﬁnal performance varying in proportion to the compression in the low-rank linear layers, very much like in the deep learning literature (Kossaiﬁ et al., 2017a;b).

Original: The number of coefﬁcients in the original data-efﬁcient Rainbow is of the order of magnitude of 1M and varies depending on the environment and its action-space size.
Revised: The number of coefﬁcients in the original data-efﬁcient Rainbow is of the order of magnitude of 1M and varies depending on the environment and its action-space size.

Original: The corresponding tensor regression layer ranks are in appendix, and chosen to target 400k, 200k and 100k coefﬁcients respectively.
Revised: The corresponding tensor regression layer ranks are in appendix [Table] and chosen to target 400k, 200k and 100k coefﬁcients respectively.

Original: While individual game results tend to decrease monotonously with increasing compression, we observe that they are noisy due to the nature of exploration in RL, and average scores reported correspond to the intuition that performance seems to decrease fast after a certain overparameterization threshold is crossed.
Revised: While individual game results tend to decrease monotonously with increasing compression, we observe that they are noisy due to the nature of exploration in RL [Table] and average scores reported correspond to the intuition that performance seems to decrease fast after a certain overparameterization threshold is crossed.

Original: To take this noisy character into account, we take care to be conservative and report the average of the ﬁnal three episodes of the learned policy after 80k,90k and steps, 
Revised: To take this noisy character into account, we take care to be conservative and report the average of the ﬁnal three episodes of the learned policy after 80k, and steps, respectively.

Original: Denoised baseline.
Revised: Denoised baseline.

Original: So as to not muddy the discussion and provide fair baselines, we do report on the NoisyNet (Fortunato et al., 2017) ablation of Rainbow (’Denoised’ columns), as the NoisyLinear layer doubles up the number of coefﬁcients required and actually performs worse in our experiments.
Revised: So as to not muddy the discussion and provide fair baselines, we do report on the NoisyNet (Fortunato et al., 2017) ablation of Rainbow (’Denoised’ columns), as the NoisyLinear layer doubles up the number of coefﬁcients required and actually performs worse in our experiments.

Original: Its principle is that the variance of exploration noise represents a criticial tradeoff for performance (too little and one stalls, too much and one risks catastrophic updates), so it is sensible to treat it as a parameter to learn.
Revised: Its principle is that the variance of exploration noise represents a criticial tradeoff for performance (too little and one stalls, too much and one risks catastrophic updates), so it is sensible to treat it as a parameter to learn.

Original: However, in order to decouple both factors of overparametrization and exploration in the discussion of deep RL performance, we simply use a ﬁxed exploration schedule.
Revised: However, in order to decouple both factors of overparametrization and exploration in the discussion of deep RL performance, we simply use a ﬁxed exploration schedule.

Original: This denoised exploration baseline is an ablation we can then compare our tensorized methods to.
Revised: This denoised exploration baseline is an ablation we can then compare our tensorized methods to.

Original: Second-order optimization.
Revised: Second-order optimization.

Original: We then proceed to assess the impact of second-order optimization in architecture by substituting ADAM optimization for K-FAC, and introducing scattering, in table 2.
Revised: We then proceed to assess the impact of second-order optimization in architecture by substituting ADAM optimization for K-FAC, and introducing scattering, in table 2.

Original: In spite of our conservative reporting, the efﬁciency boost from using a second order scheme more than makes up for low-rank approximation error (109% performance) with ﬁve times less coefﬁcients than van Hasselt et al.
Revised: In spite of our conservative reporting, the efﬁciency boost from using a second order scheme more than makes up for low-rank approximation error (109% performance) with ﬁve times less coefﬁcients than van Hasselt et al.

Original: [Table]

Original: and learning with a full order of magnitude less coefﬁcients (98% performance) is made possible by the combination of K-FAC and TRL.
Revised: and learning with a full order of magnitude less coefﬁcients (98% performance) is made possible by the combination of K-FAC and TRL.

Original: The results however do show a sharp drop in average performance when scattering is added.
Revised: The results however do show a sharp drop in average performance when scattering is added.

Original: Interestingly enough some games perform relatively very well with that method, simultaneously showing proof of viability and of additional work required
Revised: Interestingly enough some games perform relatively very well with that method, simultaneously showing proof of viability and of additional work required.

Original: [Table]
Revised: [Table]

Original: The Scattering column also includes KFAC optimization and TRL 5x, resulting in around 10x total weights efﬁciency gains.
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: hero

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: pong -19.

Revised: [Table]

Revised: The Scattering column also includes KFAC optimization and TRL 5x, resulting in around 10x total weights efﬁciency gains.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have demonstrated that in the low-data regime, it is possible to leverage biologically plausible characterizations of experience data (namely low-rank properties and wavelet scattering separability) to exhibit architectures that learn policies with an order of magnitude less weights than current state-of-the-art baselines , essentially without loss of performance, and in an online fashion.
Revised: We have demonstrated that in the low-data regime, it is possible to leverage biologically plausible characterizations of experience data (namely low-rank properties and wavelet scattering separability) to exhibit architectures that learn policies with an order of magnitude less weights than current state-of-the-art baselines , essentially without loss of performance, and in an online fashion.

Original: In particular, this provides a compelling alternative to methods like policy distillation (Rusu et al., 2015; Czarnecki et al., 2019).
Revised: In particular, this provides a compelling alternative to methods like policy distillation (Rusu et al., 2015; Czarnecki et al., 2019).

Original: We do hope that this will lead to even further progress towards sample efﬁciency and speedy exploration methods.
Revised: We do hope that this will lead to even further progress towards sample efﬁciency and speedy exploration methods.

Original: Further work will, ﬁrst, focus on thorough evaluation and research of scattering architectures in order to achieve further gains, and second investigate additional, orthogonal biologically-friendly research directions such as promoting sparsity via, for instance, L 1 regularization.
Revised: Further work will, ﬁrst, focus on thorough evaluation and research of scattering architectures in order to achieve further gains, and second investigate additional, orthogonal biologically-friendly research directions such as promoting sparsity via, for instance, L 1 regularization.

Original: Finally, we are excited by the potential of tensor factorization to offer shared core tensors for policies in multi-task and meta-learning.
Revised: Finally, we are excited by the potential of tensor factorization to offer shared core tensors for policies in multi-task and meta-learning.

Original: A A PPENDIX H YPERPARAMETERS AND R EPRODUCIBILITY
Revised: A A PPENDIX H YPERPARAMETERS AND R EPRODUCIBILITY

Original: Hyperparameters are as follows.
Revised: Hyperparameters are as follows.

Original: First, our speciﬁc architecture-modiﬁed hyperparameters:
Revised: First, our speciﬁc architecture-modiﬁed hyperparameters:

Original: [Table] [Table]
Revised: [Table] [Table]

Original: linear
Revised: linear

Original: [Table] [Table]
Revised: [Table] [Table]

Original: linear
Revised: linear

Original: [Table]
Revised: [Table]

Original: 5x compression 48 Hidden linear layer rank constraint
Revised: 5x compression 48 Hidden linear layer rank constraint

Original: [Table]
Revised: [Table]

Original: 10x compression 16 Final linear layer rank constraint
Revised: 10x compression 16 Final linear layer rank constraint

Original: [Table]
Revised: [Table]

Original: 10x compression
Revised: 10x compression

Original: [Table]
Revised: [Table]

Original: Furthermore
Revised: Furthermore

Original: [Table]
Revised: [Table]

Original: we mirror the Data-Efﬁcient Rainbow van Hasselt et al.
Revised: we mirror the Data-Efﬁcient Rainbow van Hasselt et al.

Original: ( baseline
Revised: ( baseline

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]