L EAKY T ILING A CTIVATIONS : A S IMPLE A PPROACH TO L EARNING S PARSE R EPRESENTATIONS O NLINE
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent work has shown that sparse representations—where only a small percentage of units are active—can signiﬁcantly reduce interference.
Those works, however, relied on relatively complex regularization or meta-learning approaches, that have only been used ofﬂine in a pre-training phase.
We design an activation function that naturally produces sparse representations, and so is more amenable to online training.
The idea relies on the simple approach of binning, but overcomes the two key limitations of binning: zero gradients for the ﬂat regions almost everywhere, and lost precision—reduced discrimination—due to coarse aggregation.
We introduce a
Leaky Tiling Activation (LTA) that provides non-negligible gradients and produces overlap between bins that improves discrimination.
We ﬁrst show that LTA is robust under covariate shift in a synthetic online supervised problem, where we can vary the level of correlation and drift.
Then we move to deep reinforcement learning setting and investigate both value-based and policy gradient algorithms that use neural networks with LTAs, in classic discrete control and Mujoco continuous control environments.
We show that algorithms equipped with LTAs are able to learn a stable policy faster without needing target networks on most domains.
I NTRODUCTION
Representation learning in online reinforcement learning (RL) systems can strongly impact learning efﬁciency, both positively due to generalization but also negatively due to interference (Liang et al., 2016; Heravi, 2019; Le et al., 2017; Liu et al., 2019; Chandak et al., 2019; Caselles-Dupré et al., 2018; Madjiheurem & Toni, 2019).
Neural networks particularly suffer from interference—where updates for some inputs degrade accuracy for others—when training on temporally correlated data (McCloskey & Cohen, 1989; French, 1999; Kemker et al., 2018).
Data is naturally temporally correlated in online RL.
The effect might even be worse in RL, because the effect of interference is likely magniﬁed by bootstrapping.
Target networks—a slowly-updating network used to compute the bootstrap target (Mnih et al., 2015)—can mitigate some of this interference and stabilize training (van
Hasselt et al., 2018; Mnih et al., 2015; Yang et al., 2019), but can further slow learning by using outdated estimates of value in the targets (Ghiassian et al., 2020).
Recent work (Liu et al., 2019; Ghiassian et al., 2020; Javed & White, 2019; Rafati & Noelle, 2019;
Hernandez-Garcia & Sutton, 2019), as well as older work (McCloskey & Cohen, 1989; French, 1991), have shown that sparse representation can reduce interference in updates.
A sparse representation is one where only a small number of features are active, for each input.
Each update only impacts a small number of weights and so is less likely to interfere with many state values.
Further, when constrained to learn sparse features, the feature vectors are more likely to be orthogonal (Cover, 1965), which further mitigates interference.
The learned features can still be highly expressive, and even more interpretable, as only a small number of attributes are learned to describe an input.
Learning sparse representations online, however, remains relatively open.
Some previous work has relied on representations pre-trained before learning, either with regularizers that encourage sparsity (Liu et al., 2019) or with meta-learning (Javed & White, 2019).
Other work has trained the sparse-representation neural network online, by using sparsity regularizers online with replay buffers (Hernandez-Garcia & Sutton, 2019) or using a winner-take-all strategy (Rafati & Noelle, 2019).
Hernandez-Garcia & Sutton (2019) found that many of these sparsity regularizers were ineffective for obtaining sparse representations without high levels of dead neurons, though the regularizers did still often improve learning.
The Winner-Take-All approach is non-differentiable, and there are mixed results on it’s efﬁcacy, some positive (Rafati & Noelle, 2019) and some negative (Liu et al., 2019).
Providing a simpler approach to obtain sparse representations, that are easy to train online, would make it easier for RL researchers to adopt sparse representations and further explore their utility.
In this work, we introduce a Leaky Tiling Activation function that makes it easy to control the sparsity level, and is simple to train.
The idea relies on re-expressing a binning operation—where inputs are aggregated into intervals—which we call a Tiling Activation.
Using this functional form, we introduce a smoothing parameter η that ensures gradient information can ﬂow backwards through the activation, while still maintaining the sparsity obtained by binning.
We prove that the LTA guarantees sparsity by construction.
We ﬁrst investigate the properties in an online supervised learning problem, where we can carefully control the level of correlation.
We then empirically show that RL algorithms using LTA learn more quickly and stably compared to both those with standard ReLU activations and several online sparse representation learning approaches.
P ROBLEM F ORMULATION
LTA is a generic activation that can be applied in a variety of settings.
The motivation for the activation, however, is for the online setting, for temporally correlated data streams.
We therefore explicitly formalize the two problems we are motivated by: the online supervised learning problem and the reinforcement learning (RL) problem.
In online supervised learning, the agent observes a temporally correlated stream of data, generated by a stochastic process { ( X t , Y t ) } t ∈ N , where the observations X t depend on the past { X t − i } i ∈ N .
In our supervised setting, X t depends only on X t − 1 , and the target Y t depends only on X t according to a stationary underlying mean function f ( x ) =
E [ Y t | X t = x ] .
On each time step, the agent observes X t , makes a prediction f θ ( X t ) with its parameterized function f θ , receives target Y t and incurs a squared prediction error.
1 The goal of the agent is to minimize prediction error, where f is the ideal predictor.
Learning is complicated by the correlation in the data, unlike standard supervised learning where data is independent and identically distributed (iid).
In the RL setting, we formalize the interaction using Markov decision processes (MDPs).
An MDP consists of ( S , A , P , R, γ ) , where S is the state space, A is the action space, P is the transition probability kernel, R is the reward function, and γ ∈ [0 , 1] is the discount factor.
At each time step t = 1 , 2 , . . . , the agent observes a state s t ∈ S and takes an action a t ∈ A .
Then the environment transits to the next state according to the transition probability distribution, i.e., s t +1 ∼ P ( ·| s t , a t ) , and the agent receives a scalar reward r t +1 ∈ R according to the reward function R : S
× A × S
→ R .
A policy is a mapping from a state to an action (distribution) π : S ×A → [0 , 1] .
For a given state-action pair ( s, a ) , the action-value function under policy π is deﬁned as Q π ( s, a ) =
E [ G t | S t = s, A t = a ; A t +1: ∞ ∼ π ] where G t def = (cid:80)
∞ t =
γ t R ( s t , a t , s t +1 ) is the return of a sequence of transitions s 0 , a 0 , s 1 , a 1 , ... by following the policy π .
The goal of an agent is to ﬁnd an optimal policy π ∗ : the policy that obtains maximal expected return from each state.
The policy is either directly learned, as in policy gradient methods (Sutton et al., 1999; Sutton & Barto, 2018), or the action-values are learned and the policy inferred by acting greedily with respect to the action-values, as in Q-learning (Watkins & Dayan, 1992).
In either setting, we often parameterize these functions as a Neural Network (NN), π θ : S → A , which inputs a given state (or given observation), and outputs an action: A t = π θ ( S t ) .
An action-value Q θ : S × A (cid:55)→ R can also be parameterized using a NN, typically by inputting states into the network and outputting multiple values, one for each possible action.
B INNING WITH N ON - NEGLIGIBLE G RADIENTS
In this section, we develop the Leaky Tiling Activation (LTA), as a new modular component for neural networks that provides sparse layers.
We ﬁrst introduce a new way to compute the binning of an input, using indicator functions.
This activation provides guaranteed sparsity—only one unit is active—but has a gradient of zero almost everywhere.
Then, building on this form, we provide a smoothed version, where activation is leaked into nearby bins resulting in non-negligible gradients that make it compatible with back-propagation algorithms.
We then prove that the leaky version is still guaranteed to have sparse activation, with the level controllable by the parameter η , which controls how much activation leaks to nearby units.
3. T ILING A CTIVATION
The tiling activation inputs a scalar z and outputs a binned vector.
This vector is one-hot, with a 1 in the bin corresponding to the value of z , and zeros elsewhere.
Note that a standard activation typically maps a scalar to a scalar.
However, the tiling activation maps a scalar to a vector, as depicted in Figure 1(a).
This resembles tile coding, which inspires the name Tiling Activation; to see this connection, we include a brief review of tile coding in the Appendix A.1.
In this section, we show how to write the tiling activation compactly, using element-wise max and indicator functions.
[Figure]
Assume we are given a range [ l, u ] for constants l, u ∈ R , where we expect the input z ∈ [ l, u ] .
The goal is to convert the input, to a one-hot encoding, with evenly spaced bins of size δ ∈ R + .
Without loss of generality, we assume that u − l is evenly divisible by δ ; if it is not, the range [ l, u ] could be slightly expanded, evenly on each side, to ensure divisibility.
Deﬁne the k -dimensional tiling vector
[Equation]
where k = ( u − l ) /δ .
The tiling activation is deﬁned as
[Equation]
where I + ( · ) is an indicator function, which returns 1 if the input is positive, and zero otherwise.
The indicator function for vectors is applied element-wise.
In Proposition 1, we prove that φ returns a binned encoding: if c i <
z < c i +1 , then φ ( z ) returns e i the one-hot (standard basis) vector with ain the i -th entry and zero elsewhere.
For values of z that fall on the boundary, z = c i , the encoding returns a vector with ones in both entries c
i − 1 and c i .
We provide the following example for intuition for the tiling activation.
Example .
Assume [ l, u ] =
[0 , 1] and set the tile width to δ = 0 .
Then the tiling vector c has four tiles ( k = 4 ): c
= (0 , 0 . 25 , 0 . 5 , 0 . 75) .
If we apply the tiling activation to z = 0 . 3 , because 0 . 25 < 0 .
3 < 0 .
5 , the output should be (0 , 1 , 0 , 0) .
To see φ ( z ) does in fact return this vector, we compute each max term max( c − z, 0) =
and
[Equation]
The addition of these two is (0 . 05 , 0 , 0 . 2 , 0 . 45) and so 1 − I + (0 . 05 , 0 , 0 . 2 , 0 . 45) = 1 − (1 , 0 , 1 , 1) =
The ﬁrst max extracts which components in c are strictly greater than z , and the second max those which are strictly less than.
The addition gives the bins that are strictly greater and strictly less than the bin for z , leaving only the entry corresponding to that bin as 0, with all others positive.
Going beyond this intuition, we can formally characterize the three possible situations for the LTA: 1) it gives exactly one nonzero entry when the input is greater than any element in c ; 2) it gives two nonzero entries when the input value is equal to one of the element in c except the ﬁrst one; and 3) it gives exactly one nonzero entry when the input is equal to the ﬁrst element in c .
See Appendix A.2.
for a simple proof.
We assume throughout that δ evenly divides the range [ l, u ] ; we state this explicitly in Assumption 1 to highlight this condition.
This is not restrictive and can be easily satisﬁed simply by expanding the range slightly, beyond the actual range of inputs.
3. L EAKY T ILING A CTIVATION (LTA)
The Tiling Activation provides a way to obtain sparse, binary encodings for features learned within a neural network.
As motivated upfront, such encodings can enable faster learning and reduce interference.
Unfortunately, the tiling activation has a zero derivative almost everywhere.
The value switches from 0 to 1, when a threshold is passed (as shown in Figure 1(a)).
In this section, we provide a leaky tiling activation, that has non-zero derivatives and so is amenable to use with backpropagation.
To design the LTA, we deﬁne the leaky indicator function
[Equation]
where η is a small constant for controlling the sparsity.
The ﬁrst term I + ( η − x ) is 1 if x < η , andotherwise.
The second term I + ( x − η ) is 1 if x > η , and 0 otherwise.
If x < η , then I η, + ( x ) = x , and else I η, + ( x ) = 1 .
The original indicator function I + can be acquired by setting η = 0 .
When η > 0 , the derivative is non-zero for x < η , and zero otherwise.
This means the derivative can be propagated backwards through this indicator, for this range of inputs, similar to the ReLU activation.
Using this leaky indicator function, we deﬁne the following Leaky Tiling Activation (LTA)
[Equation]
where again I η, + is applied elementwise.
We depict this activation, with two different η , in Figures 1(b) and 1(c).
For the smaller η , the
LTA extends the activation to the neighbouring bins.
The activation in these neighbouring bins is sloped, resulting in non-zero derivatives.
For this smaller η , however, there are still regions where the derivative is zero (e.g., z = 0 . 3 in Figure 1(b)).
The regions where derivatives are non-zero can be expanded by increasing η as shown in Figure 1(c).
Hence we can adjust η to make trade-off between the sparsity level and how much neural network can change as we demonstrate in Section A.5.
Figure 2 shows a neural network with LTA applied to the second hidden layer and its output y is linear in the sparse representation.
LTA itself does not introduce any new training parameters, just like other activation functions.
For input x , after computing ﬁrst layer h 1 = xW 1 , we apply φ η ( z ) to h 1 W 2 ∈ R d to get the layer
h 2 of size kd .
This layer consists of stacking the k -dimensional sparse encodings, for each element in h 1 W 2 .
3. G UARANTEED S PARSITY FROM THE LTA
We now show that the LTA maintains one of the key properties of the tiling activation: sparsity.
The distinction with many existing approach is that our sparsity is guaranteed by design and hence is not a probabilistic guarantee.
We ﬁrst characterize the vectors produced by the LTA, in Propositionwith proof in Appendix A.2.2.
Then, we provide an upper bound on the proportion of nonzero entries in the generated vector in Theorem 1, with in Appendix A.2.3.
Assumption 1.
δ < u − l , where
kδ =
u − l for k ∈ N .
Theorem 1 (Sparsity guarantee for LTA.) .
For any z ∈ [ l, u ] , η > 0 , φ η ( z ) outputs a vector whose number of nonzero entries (cid:107) φ η ( z ) (cid:107) 0 satisﬁes:
[Equation]
Corollary 1.
Let ρ ∈ [0 , 1) be the desired sparsity level: the maximum proportion of nonzero entries of φ η ( z ) , ∀ z ∈ [ l, u ] .
Assume ρk ≥ 3 , i.e., some inputs have three active indices or more (even with η = 0 , this minimal active number is 2).
Then η should be chosen such that
[Equation]
As an example, for k = 100 , δ = 0 . 05 and a desired sparsity of at least 10% ( ρ = 0 . 1 ), we can use η = 0 .
052 ( (cid:98) 100 × 0 .
1 (cid:99) − 1)
= 0 .
Note that the bound in Theorem 1 is loose in practice as the bound is for any input z .
In Appendix A.2.3 and A.5, we theoretically and empirically show that the actual sparsity is usually lower than the upper bound, and quite consistent across inputs.
E XPERIMENTS IN S UPERVISED L EARNING UNDER C OVARIATE S HIFT
A key property of LTA is that it provides an immediate, deterministic sparsity guarantee.
We hypothesize that this property is suitable for handling highly correlated data, where there is a strong need for interference reduction.
In this section, we focus on testing the hypothesis that LTA provides representations that are more robust to learning online on correlated data. Speciﬁcally, we hypothesize that convergence speed and stability for ReLU networks suffer under strongly correlated training data, whereas comparable LTA networks are nearly unaffected.
We create a synthetic supervised problem with a relative simply target function, and focus the investigation on the impact of a drifting distribution on inputs, which results both in covariate shift and creates temporal correlation during training.
We also report results on two benchmark image classiﬁcation tasks in the Appendix A.6.
The Piecewise Random Walk Problem has Gaussian X t ∼ N ( S t , β 2 ) with ﬁxed variance β 2 and a mean S t that drifts every T [-steps, with the target y t = sin(
 πx 2 t ) where x t is a realization of X t .-]steps.
More precisely, the mean S t stays ﬁxed for T timesteps, then takes a step according to a ﬁrst order autoregressive random walk: S t
= (1 − c )
S t + Z t where c ∈ (0 , 1] and Z t ∼ N (0 , σ 2 ) for ﬁxed variance σ 2 .
If c = 0 , then this process is a standard random walk; otherwise, with c < 1 , it keeps S t in a bounded range with high probability.
{+For x t ∼ X t , the training label y t is deﬁned as y t = sin(
+}
πx 2 t ) . This process is designed to modulate the level of correlation—which we call correlation difﬁculty — without changing the equilibrium distribution over X t .
As the correlation difﬁculty d varies from 0 to 1 , the training data varies from low to high correlation;
d = 0 recovers iid sampling.
All d ∈ [0 , 1) share the same equilibrium distribution in X t .
X t is ergodic and has Gaussian equilibrium distribution N (0 , ξ 2 ) , with variance ξ 2 dependent upon β 2 , σ 2 and c .
In particular, the visitation distribution X t for any training run will converge to the equilibrium distribution.
This ensures that measuring loss with respect to the stationary distribution is a fair comparison, because the visitation distribution of X t is identical across all settings.
We depict sample trajectories with low d and high d in Figure 3(a).
For a rigorous construction of X t and S t , and justiﬁcation for this equilibrium distribution and implementation details, see Appendix A.7.
We measure the mean squared error over the equilibrium distribution in X t , for neural networks using
LTA and ReLU activations across a range of correlation difﬁculty values.
In Figure 3, we can see that LTA outperforms ReLU in two ways.
First, LTA converges to a lower loss with less variance across all correlation difﬁculties.
Second, LTA only marginally suffers under high difﬁculties d > 0 .
9 , whereas the performance of ReLU begins to deteriorate for relatively mild d > 0 . 5 .
Note that the LTA reaches a lower error, even on iid data ( d = 0 ).
We hypothesize this gap arises because the networks are training online, with one sample from each X t being used for each weight update. Figure 20 in the Appendix supports this hypothesis, with the gap vanishing in an identical experiment where 50 samples are drawn from each X t .
[Figure]
E XPERIMENTAL R ESULTS IN RL
In this section, we empirically study the effect of using LTA in RL.
We show overall performance on several benchmark discrete action and continuous action environments.
We then compare our method with other simple strategies to obtain sparse representations.
Appendix A.4 includes details for reproducing experiments and Appendix A.5 has additional RL experiments.
5. A LGORITHMS AND N AMING C ONVENTIONS
All the algorithms use a two-layer neural network, with the primary difference being the activation used on the last layer.
See Appendix A.5 for results using LTA in all the hidden layers.
DQN is used for the discrete action environments, and Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2016) for continuous action environments.
On each step, all algorithms sample a mini-batch size of 64 from an experience replay buffer with maximum size 100 k. Note that we are able to keep the same LTA setting across all experiments: we set [ l, u ] =
[ − 20 , 20] ; we set δ = η = 2 . 0 , c
= {− 20 , − 18 , − 16 , ..., 18 } , and hence k
= 40 / 2 = 20 .
We ﬁrst compare to standard DQN agents, with the same architectures except the last layer.
DQN : DQN with tanh or ReLU on
the last layer
(best
performance reported)
DQN-LTA : DQN with LTA on the last layer.
DQN-Large : DQN, but with the last layer of the same size as DQN-LTA.
If DQN has a last layer of size d , then DQN-LTA has a last layer of size dk .
The number of parameters is actually the same; the
LTA activation simply expands the number of features due to binning.
We include DQN-Large for a comparison to a DQN agent with the same number of (ReLU or tanh) features in the last layer, used to learn the value function.
DQN-Large has k times more parameters in this last layer than DQN or
DQN-LTA, and so we expect that over-parameterization to be detrimental.
We also compare to several simple strategies to obtain local or sparse features.
Radial basis functions (RBFs) have traditionally been used to obtain local features in RL, and recent work has used (cid:96) 2 and (cid:96) 1 regularization directly on activations as a simple baseline (Arpit et al., 2016;
Liu et al., 2019).
All of these strategies have the same sized last layer as the sparse feature dimension of DQN-LTA.
DQN-RBF : DQN using radial basis functions (RBFs) on the last layer, with the centers deﬁned by the same c as LTA: φ ( z ) =
[exp ( − ( z − c 1 ) 2 σ ) , . . .
, exp ( − ( z − c k ) 2 σ )] where σ is the bandwidth parameter.
DQN-L2/L1 : DQN with (cid:96) 2 or (cid:96) 1 regularization on the activation functions of the ﬁnal hidden layer where there is the same number of units as that in DQN-Large.
To the best of our knowledge, no suitable sparse representation approaches exist for RL.
SR-NNs for
RL were only developed for ofﬂine training (Liu et al., 2019).
That work also showed that k-sparse
NNs (Makhzani & Frey, 2013) and Winner-Take-All NNs (Makhzani & Frey, 2015) performed signiﬁcantly worse than (cid:96) 2 regularization, where (cid:96) 2 actually performed quite well in most of their experiments.
One other possible option is to use Tile Coding NNs (Ghiassian et al., 2020), which ﬁrst tile code inputs before feeding them into the neural network.
This approach focuses on using discretization to break, or overcome, incorrect generalization in the inputs; their goal is not to learn sparse representations.
This approach is complementary, in that it could be added to all the agents in this work.
Nonetheless, because it is one of the only papers with a lightweight approach to mitigate interference in online RL, we do compare to it in the Appendix A.5.
5. O VERALL P ERFORMANCE
In this section, we demonstrate the overall performance of using LTAs on both discrete and continuous control environments.
Our goals are to investigate if we can 1) obtain improved performance with
LTA, with ﬁxed parameter choices across different domains; 2) improve stability in learning with
LTA; and 3) see if we can remove the need to use target networks with LTA, including determining that learning can in fact be faster without target networks and so that it is beneﬁcial to use LTA without them.
All experiments are averaged over 20 runs ( 20 different random seeds), with ofﬂine evaluation performed on the policy every 1000 training/environment time steps.
Discrete control.
We compare performance on four discrete-action environments from OpenAI (Brockman et al., 2016): MountainCar, CartPole, Acrobot and LunarLander.
We use 64 ×
[Figure]
[Figure]
ReLU hidden units on all these domains.
Since LTA has k = 20 , this yields 64 × 20 =dimensional sparse features.
Hence, DQN-Large, use two layers ReLU with size 64 × 1280 .
We plot evaluation learning curves, averaged over 20 runs, in Figure 4.
The results show the following.
1) With or without using a target network, DQN with LTA can signiﬁcantly outperform the version without using LTA.
2) LTA has signiﬁcantly lower variability across runs (smaller standard errors) in most of the ﬁgures.
3) DQN-LTA trained without a target network outperforms DQN-LTA trained with a target network, which indicates a potential gain by removing the target network.
Without using LTA, DQN trained without a target network cannot perform well in general, providing further evidence for the utility of sparse feature highlighted in previous works (Liu et al., 2019; Rafati & Noelle, 2019).
5) Simply using a larger neural network does not obtain the same performance improvements, and in some cases signiﬁcantly degrades performance.
Continuous control.
We compare performance on ﬁve continuous-action environments from Mujoco (Todorov et al., 2012): Swimmer, Inverted Pendulum, Double Inverted Pendulum, Hopper, and
Walker.
For these experiments, we use DDPG with exactly the same LTA setting as the above discrete control domain to show its generality.
The results are qualitatively similar to the discrete-action environments, as seen in Figure 5, except in one domain (Swimmer).
In all other domains, DDPG equipped with LTA, without target networks, achieves comparable and sometimes signiﬁcantly better performance to DDPG.
Swimmer highlights that LTA is not always sufﬁcient on its own to overcome instabilities, and could be complemented by strategies such as using mellowmax (Kim et al., 2019).
5. C OMPARISON WITH O THER S PARSE A PPROACHES
LTA provides clear beneﬁts, but it is natural to ask if other simple strategies that provide sparse or local features could have provided similar beneﬁts.
We compare to DQN-RBF, DQN-L2 and
DQN-L1, on the discrete action environments, shown in Figure 6.
We ﬁnd the following.
1) LTA performs consistently well across all environments using a ﬁxed parameter setting; none of the other approaches achieve consistent performance, even though we tuned their parameters per environment.
Both the (cid:96) 1 and (cid:96) 2 approaches have a high variance across different random seeds.
The RBF variant can do better than the (cid:96) 1 and (cid:96) 2 approaches but is worse than our algorithm.
It is a known issue that RBF is sensitive to the bandwidth parameter choice and we observe similar phenomenon.
It is also known that the exponential activation can be problematic in the back-propagation process.
We empirically investigate the overlap and instance sparsities of each algorithm in the Appendix A.5.
[Figure]
5. T ESTING S TABILITY IN A S IMULATED A UTONOMOUS D RIVING D OMAIN
Our results have shown improved stability with LTA.
In this section, we test LTA in an environment focused on stability, namely an autonomous driving task (Leurent, 2018).
In the real world, a stable policy is of vital importance to ensure safety in autonomous driving.
In this simulated task, the goal is not only to obtain high return, but also keep the number of car crashes as low as possible.
Figure 7(a) shows the domain, where the agent—the green car—has to learn to switch lanes, avoid car crashes, and go as fast as possible.
The observations are 25 -dimensional, with vehicle dynamics that follow the Kinematic Bicycle Model.
The action space is discrete.
LTA learns faster, with signiﬁcantly fewer car crashes incurred during the evaluation time steps as shown in Figure 7(c).
Target networks are harmful in this environment, potentially because they slow early learning; the agent to accumulate a signiﬁcant number of crashes before improving.
[Figure]
C ONCLUSION
In this work, we proposed a simple approach to learning sparse representations with neural networks, with an activation that naturally produces sparse layers.
The activation draws on the idea of binning— or aggregation—which produces a one-hot encoding of an input.
This approach ensures sparsity, which can reduce interference when learning online, but suffers from two problems: a loss of precision due to aggregation and zero gradient almost everywhere.
We provide a Leaky Tiling Activation (LTA), that ameliorates both of these issues.
A parameter η within the LTA provides larger regions with non-negligible gradients, with the positive consequence that it also increases overlap, which can improve discrimination.
We show that the LTA still has sparsity guarantees, related to the choice of η .
Though sparse representations have been shown to reduce interference when learning online, it has remained difﬁcult to actually learn these sparse representations online.
The LTA provides sparse representations by construction, and so it is much easier to use when learning online.
We highlight that LTA is robust to high levels of covariate shift, in a synthetic supervised learning problem.
We then show across several discrete-action and continuous-action reinforcement learning environments that using the LTA signiﬁcantly improve learning efﬁciency and stability, and in most cases even removes the need to use target networks.
A A PPENDIX
This appendix includes the following contents:
Section A.1 brieﬂy reviews tile coding which inspires LTA and the naming.
Section A.2 shows the proofs for theorems about sparsity guarantee in this paper.
Section A.3 discusses possible practical issues when using our LTA function.
Section A.4 includes experimental details of Section 5 for reproducible research.
Section A.5 presents additional experiments in reinforcement learning setting.
It includes: 1) empirical sparsity achieved by LTA; 2) comparison with additional baseline Tile Coding NNs Ghiassian et al.
(2020); 3) out of boundary loss in LTA; 4) sensitivity analysis of sparsity control parameter and tile (bin) width; 5) the effect of using LTA in both hidden layers in RL problems.
Section A.6 reports the results of LTA on two popular image classiﬁcation datasets: Mnist LeCun & Cortes (2010) and Mnistfashion Xiao et al.
Section A.7 includes the details of the synthetic supervised learning experiment from Section 4.
A.
T ILE C ODING R EVIEW
We give a brief review of tile coding here, as tile coding inspires our Leaky Tiling Activation (and its naming).
Tile coding is a generalization of state aggregation, that uses multiple tilings (aggregations) to improve discrimination.
For input z ∈ [0 , 1] , state-aggregation would map z to a one-hot vector with a 1 in the corresponding bin (bin can be also called tile), with k the number of bins discretized into intervals of length δ .
In tile coding, multiple such tilings of the input are concatenated, where each tiling is offset by a small amount.
This is depicted in Figure 8, where we show two tilings, one covering from [ − 0 .
05 , 1] and the other from [0 . 0 , 1 . 05] both with k = 4 and δ = 1 .
05 / 4 = 0 .
The resulting feature vector is a concatenation of these two tilings, to produce 8 features.
Two nearby inputs z = 0 . 3 and z (cid:48)
= 0 .
55 would be aggregated together in state-aggregation, making it impossible to produce different values for those inputs.
Tile coding, on the other hand, would cause them to share one feature in the second tiling, but each have a distinct feature that enables them to have different values.
The large bins/tiles allows for some information sharing, for fast generalization; and the overlapping tilings allows for improved discrimination.
[Figure]
Unfortunately, tile coding can scale exponentially in the dimension d of the inputs.
Each dimension is discretized to a granularity of δ ( k regions), with the cross-product between all these regions resulting in k d bins.
One natural approach to make binning more scalable and more practically usable, is to combine it with neural networks (NN).
Our activation function—Leaky Tiling Activation—enables a differentiable binning operation.
A. P ROOFS
We now provide the proof for Proposition 1, Proposition 2, Theorem 1 and Corollary 1 below.
A.2. P ROOF FOR P ROPOSITION
For convenience, we deﬁne the k -dimensional one-hot vector e
i whose i th entry is one and otherwise zero.
Proposition 1.
Under Assumption 1, for any z ∈ [ l, u ] ,
If c i <
z < c i +1 for i ∈
[ k − 1] or c k < z , then φ ( z ) =
e i 2.
If z = c i for i ∈ { 2 , ..., k } , then φ ( z ) =
e i −
+ e i 3.
If z = c 1 , then φ ( z ) =
e
Proof.
In all thee cases, the ﬁrst max operation in φ is
[Equation]
To understand the output of the second max operation, we look at the three cases separately.
Case 1: Because c i <
z < c i +1 , we know that c
i − 1 <
z − δ <
c i .
This implies the second max operation in φ is:
[Equation]
Therefore, the sum of both max operations max( c − z, 0) + max( z − δ − c , 0) has positive elements everywhere except the i th position, which is zero.
Hence I + (max( c − z, 0) + max( z − δ − c , 0)) gives a vector where every entry is 1 except the i th entry which is 0 .
Then 1 − I + (max( c − z, 0) + max( z − δ − c , 0)) =
e i .
Case 2: If z = c i , i ∈ { 2 , ..., k } , then c
i − 1 = z − δ , and
[Equation]
It follows that max( c − z, 0) + max( z − δ − c , 0) has exactly two zero entries, at indices i −
and i .
This gives 1 − I + (max( c − z, 0) + max( z − δ − c , 0))
=
e i −
+ e i , a vector with ones at indices i −
and i .
Case 3: When z = c 1 , max( z − δ − c , 0) is a zero vector and max( c − z, 0) is positive everywhere except the ﬁrst entry, which is zero.
Again this gives 1 − I + (max( c − z, 0) + max( z − δ − c , 0))
=
e 1 .
A.2. P ROOF FOR P ROPOSITION
Proposition 2.
Let I be the set of indices where φ ( z ) is active (from Theorem 1 we know it can only contain one or two indices).
Under Assumption 1, for any z ∈ [ l, u ] , the function φ η ( z ) =
φ ( z ) + ∆ , where ∆ is mostly zero, with a few non-zero entries.
The vector ∆ has the following properties:
∆ is zero at indices i ∈ I , i.e., φ η ( z ) equals φ ( z ) at indices i ∈ I . 2.
∆ is non-zero at indices { j | j / ∈ I , j ∈ [ k ] , 0 < z − δ − c j ≤ η, 0 < c
j − z ≤ η } .
Proof.
Part 1.
Let the i th entry be active in φ ( z ) .
We have one of the three cases hold as stated in
Theorem 1.
Assume c
i <
z < c i +1 .
Note that
[Equation]
taking the sum of the above two equations gives us a vector as following:
[Equation]
Then applying I η, + ( · ) to vector equation 6 gives us a vector retaining elements ≤ η and all other elements become 1 .
Hence the i th position is zero after applying I η, + ( · ) to vector equation 6.
Using one minus this vector would give us a vector with only the i th is one.
But this is exactly φ η ( z ) .
And since φ η ( z ) =
φ ( z ) + ∆ , and the i th entry of φ ( z ) is also one, the i th entry of ∆ must be zero.
Similar reasoning applies to the cases when z =
c i , i ∈ { 2 , ..., k } or z = c 1 .
Part 2.
Note that applying I η, + ( · ) to the vector max( c − z, 0)+max
( z − δ − c , 0) keeps all elements no more than η and making all other elements one.
As a result, 1 − I η, + (max( c − z, 0)+max
( z − δ − c , 0)) would give a vector zero everywhere except those entries in max( c − z, 0) + max( z − δ − c , 0) which are ≤ η .
The set of indices which are ≤ η in the vector max( c − z, 0) + max( z − δ − c , 0) can be written as { j | j ∈ [ k ] , 0 <
z − δ − c j ≤ η, 0 < c j − z ≤ η } , which is also the set of indices where φ η ( z ) is nonzero.
Since φ η ( z ) =
φ ( z ) + ∆ and φ ( z ) has nonzero entries in the indices I and ∆ has zero values at those entries by Part 1, then ∆ must have nonzero entries at { j | j / ∈ I , j ∈ [ k ] , 0 <
z − δ − c j ≤ η, 0 < c j − z ≤ η } .
A.2. P ROOF FOR T HEOREM
Theorem 1.
Sparsity guarantee for LTA.
For any z ∈ [ l, u ] , η > 0 , φ η ( z ) outputs a vector whose number of nonzero entries (cid:107) φ η ( z ) (cid:107) 0 satisﬁes:
[Equation]
Proof.
Similar to the proof of Theorem 1, we divide to three cases to prove the result.
Case 1.
Consider the case that c
i <
z < c i +1 , i ∈
[ k − 1] .
Note that the number of nonzero entries in φ η is equal to the number of entries less than η in the vector equation 6, hence we can count the number of entries less than η in ( z − δ − c 1 , ..., z − δ − c
i − 1 , 0 , c
i +1 − z, c i +2 − z, ..., c k − z ) .
First, we count the number of entries that are less than or equal to η on the left side of the i th position.
Since the i th position is zero, which indicates z − δ − c
i < 0 , hence z − δ − c
i − 1 − δ <
and it follows that 0 <
z − δ − c i − 1 < δ .
Then δ < z − δ − c i − 1 + δ
=
z − δ − c
i − 2 < 2 δ .
Hence ( j − 1)
δ <
z − δ − c
i − j < jδ, j ∈ { 1 , ..., i − 1 } .
Assume there are m entries ≤ η on the the left side of the i th.
Then z − δ − c
i − m ≤ η .
It follows ( m − 1) δ < η and hence m ≤ (cid:98) ηδ (cid:99) + 1 .
Hence the total number of elements ≤ η on the left side of the i th position is at most (cid:98) ηδ (cid:99) + 1 .
Second, count the number of entries ≤ η on the right side of the i th position.
Since c i − z < 0 , 0 <
c
i + δ − z
= c
i +1 − z < δ, 0 <
c i +2 − z < 2 δ, ... .
Hence the possible number of entries ≤ η on the right side of i th position is at most (cid:98) ηδ (cid:99) + 1 .
As a result, together with the i th position which is 0 ≤ η , the possible number of nonzero entries in this case is at most 2 (cid:98) ηδ (cid:99) + 3 .
Case 2.
When z = c
i , i ∈ { 2 , ..., k } , we count the number of entries less than η in the vector
[Equation]
Again, we attempt to count the number of entries less than η in this vector by considering ( i − 1) δ, ..., 2 δ, δ and δ, 2 δ, ..., ( k − i )
δ respectively.
We follow the exactly same argument as above and now we have two zero entries at i − 1 , i th positions.
The difference is that, the number of entries less than η in the vector (( i − 1) δ, ..., 2 δ, δ can be at most (cid:98) ηδ (cid:99) .
As a result, the number of nonzero entries in this case is still at most 2 (cid:98) ηδ (cid:99) + 2 .
Case 3.
When z = c 1 , max( z − δ − c , 0) is a zero vector and max( c − z, 0) + max( z − δ − c , 0) is positive everywhere except the ﬁrst entry, which is zero.
Then we simply count the number of entries ≤ η on the right side of the 0 th position, i.e. j ∈ { 1 , ..., k } .
Similar to the analysis in the above Case 1, the possible number of entries ≤ η on the right side of 0 th position is at most (cid:98) ηδ (cid:99) .
Hence in this case, the number of nonzero entries is at most (cid:98)
η δ (cid:99) + 1 .
In summary, the number of nonzero entries does not exceed 2 (cid:98) ηδ (cid:99) + 3 .
This completes the proof.
Remark.
As we empirically demonstrated in Table 1 and below Figure 10 and Figure 9, the actual sparsity achieved by LTA is lower than the upper bound.
This is because our upper bound is for any possible input z .
Consider that in Case 1 in the above proof, we count the number of entries that are less than or equal to η on the left side and right side of the i th position.
There are (cid:98) ηδ (cid:99) + 1 such entries on both sides only when z is exactly equal to c
i + c
i − 1, which is unlikely to happen in practice.
A.2. P ROOF FOR C OROLLARY
Corollary 1 Let ρ ∈ [0 , 1) be the desired sparsity level: the maximum proportion of nonzero entries of φ η ( z ) , ∀ z ∈ [ l, u ] .
Assume ρk ≥ 3 , i.e., some inputs have three active indices or more (even with η = 0 , this minimal active number is 2).
Then η should be chosen such that
[Equation]
Proof.
Because (cid:107) φ η ( z ) (cid:107) 0 ≤ (cid:98) kρ (cid:99) , from Theorem 1 it is sufﬁcient to pick η such that 2 (cid:98) ηδ (cid:99)
+ 3 ≤ (cid:98) kρ (cid:99)
≤ kρ .
This gives (cid:98) ηδ (cid:99) ≤ ( (cid:98) kρ (cid:99) − 3) / 2 ≤ ( kρ − 3) / 2 .
Additionally, we know ηδ − 1 ≤ (cid:98) ηδ (cid:99) ≤ ( (cid:98) kρ (cid:99) − 3) / 2 , giving the second inequality.
A. M ORE DISCUSSION ABOUT LTA
Our development of the LTA assumed that the inputs are bounded in the range [ l, u ] (recall that c
= [ l, l + δ, l + 2 δ, ..., u ] ).
This is not guaranteed for the standard inputs to activations, namely z = x (cid:62) w for some weight vector w .
The LTA can still be used if z / ∈ [ l, u ] , but then the gradient of the LTA will be zero.
This means that the weights w cannot be adjusted for inputs where z becomes too large or too small.
This issue is usually called gradient vanish , which is common in many popular, existing activation functions such as ReLU, tanh, sigmoid, etc.
Our LTA has two simple ways to avoid this issue: 1) use a squashing activation before handing z to the LTA and 2) regularize (penalize)
z that falls outside the range [ l, u ] .
For example, tanh can be applied ﬁrst to produce z = tanh ( x (cid:62) w ) ∈ [ − 1 , 1] .
Though the simplest strategy, using tanh function can be problematic in deep neural networks due to gradient vanishing problems.
An alternative is to use a penalty, that pushes out-of-boundary z back into the chosen range [ − u, u ]
[Equation]
This penalty is easily added to the loss, giving gradient ∂r ( z ) ∂z =
I ( z > u ) − I ( z < − u ) .
For example, z = max( x (cid:62) w , 0) , which might produce a value greater than u .
If z > u , then the gradient pushes the weights w to decrease z .
It should be noted that the number of nonzero entries can only decrease when z goes out of boundary.
However, in our experiments in our main paper, we found such out of boundary loss is unnecessary on all of our tested domains.
Furthermore, we further verify that
our
LTA performs stably across different weights for the above regularization in Section A.5.
A. R EPRODUCIBLE R ESEARCH
Common settings.
All discrete action domains are from OpenAI Gym Brockman et al.
(2016) with version 0 .
Deep learning implementation is based on tensorﬂow with version 1 . 13 .
Abadi & et.
al (2015).
We use Adam optimizer Kingma & Ba (2015), Xavier initializer Glorot & Bengio (2010), mini-batch size b = 64 , buffer size 100 k, and discount rate γ = 0 .
99 across all experiments.
We evaluate each algorithm every 1 k training/environment time steps.
Algorithmic details.
We use 64 × 64 ReLU units neural network for DQN and 200 × 100 ReLU units neural network for DDPG.
All activation functions are ReLU except: the output layer of the Q value is linear.
The weights in output layers were initialized from a uniform distribution [ − 0 .
Note that we keep the same LTA setting across all experiments: we set [ l, u ] =
[ − 20 , 20] ; we set δ = η = 2 . 0 , c
= {− 20 , − 18 , − 16 , ..., 18 } , and hence k = 40 / 2 = 20 .
This indicates that the
DQN-Large and DDPG-Large versions have 64 × 20 = 1280 and 100 × 20 = 2000 ReLU units in the second hidden layer.
For RBF coding, we set the bandwidth as σ = 2 . 0 , and uses the same tiling (i.e. c vector) as our LTA.
Meta-parameter details.
For DQN, the learning rate is 0 .
and the target network is updated every 1 k steps.
For DDPG, the target network moving rate is 0 .
001 and the actor network learning rate is 0 . 0001 , critic network learning rate is 0 .
For l 1 , l 2 regularization variants, we optimize its regularization weight from { 0 . 1 , 0 . 01 , 0 .
001 , 0 . 0001 } on MountainCar, then we ﬁx the chosen optimal weight 0 .
01 across all domains.
Environmental details on discrete control domains.
We set the episode length limit as 2000 for
MountainCar and keep all other episode limit as default settings.
We use warm-up steps 5000 for populating the experience replay buffer before training.
Exploration noise is 0 . 1 without decaying.
During policy evaluation, we keep a small noise (cid:15) = 0 .
05 when taking action.
Environmental details on continuous control domains.
On Mujoco domains, we use default settings for maximum episodic length.
We use 10 , 000 warm-up time steps to populate the experience replay buffer.
The exploration noise is as suggested in the original paper by Lillicrap et al.
A. A DDITIONAL E XPERIMENTS ON R EINFORCEMENT L EARNING P ROBLEMS
This section shows the following additional empirical results which cannot be put in the main body due to space limitations.
• A.5.1 empirically investigates the representation sparsity generated by LTA.
• A.5.2 shows the comparison with Tile Coding NNs (Ghiassian et al., 2020), which ﬁrst tile code inputs before feeding them into a neural network.
• A.5.3 shows the empirical results of DQN-LTA using linear activation function with regularization weights ∈ { 0 . 0 , 0 . 01 , 1 . 0 } for the out of bound loss in Eq 8.
• A.5.4 shows the sensitivity of LTA to the sparsity control parameter η and tile width parameter δ .
• A.5.5 shows the results when applying LTA to both hidden layers rather than just the second hidden layer in RL problems.
A.5. E MPIRICALLY MEASURING SPARSITY
We also report two sparsity measures of the learned representations averaged across all steps in Table 1.
We compute an estimate of sparsity by sampling a mini-batch of samples from experience replay buffer and taking the average of them.
Instance sparsity corresponds to proportion of nonzero entries in the feature vector for each instance.
Overlap sparsity French (1991) is deﬁned as overlap ( φ, φ (cid:48) )
= (cid:80) i
I ( φ i (cid:54) = 0)
I ( φ (cid:48) i (cid:54) = 0) /
( kd ) , given two kd -dimensional sparse vectors φ, φ (cid:48) .
Low overlap sparsity potentially indicates less feature interference between different input samples.
Theorem 1 guarantees that the instance sparsity, with LTA, should be no more than 12 .
5% for our setting of k = 40 , η = δ ; and should be no more than 25% when k = 20 , η = δ .
In the table, we can see that LTA has lower (better) instance sparsity than the upper bound provided by the theorem.
Further, LTA achieves the lowest instance and overlap sparsity among all baselines in an average sense.
Figure 9 and Figure 10 are corresponding to the learning curves as shown in Figure 6 in Section 5.3.
We show learning curves of instance/overlap sparsity as a function of training steps for LTA, l 1 , lregularization and RBF.
The instance sparsity is computed by randomly sampling a mini-batch of states from replay buffer and count the average number of nonzero entries in that mini-batch divided by feature dimension kd .
The overlap sparsity is computed by randomly sampling two mini-batches of states from the replay buffer and count the average number of simultaneously activated (i.e. nonzero) entries in both mini-batches.
From the sparsity learning curves, one can see that our LTA has very stable sparsity since the beginning of the learning, and the sparsity is almost constant cross domains.
Particularly, the overlap sparsity is very low, which possibly indicates low representation interference.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
[Figure] [Figure]
C OMPARISON WITH TC-NN
We include the ﬁgure of comparing with TC-NN from the work by Ghiassian et al.
(2020), which use a regular tile coding to process the input before feeding into the neural network.
It should be noted that this method itself is not a sparse representation learning technique, and requires the original observation space to be scaled within [0 , 1] .
Figure 11 shows that TC-NN does indeed improve performance for DQN, but not compared to LTA.
DQN-LTA both learns faster and reaches a better, more stable solution.
In our TC-NN implementation, as a correspondent to LTA, we use binning operation to turn each raw input variable to 20 binary variables through binning operation.
We use the same learning rate 0 .
0001 as other baselines.
A.5. I NSENSITIVITY TO OUT OF BOUNDARY LOSS
We demonstrate the effect of using an out of boundary loss as discussed in Section A.3.
This is supplementary to our experiments in the main body, where we did not use an out of boundary loss, i.e. the regularization weight is 0 .
To highlight the issue of out of boundary, we intentionally use a tiling vector c with a small range [ − 1 , 1] with 20 tiles, i.e. δ = 2 / 20 = 0 .
1 and set the sparsity control parameter η = δ = 0 .
In Figure 12, one can see that our algorithm typically does not need such a regularization even if we use a small tiling.
A.5. S ENSITIVITY TO T ILE W IDTH AND S PARSITY
We empirically investigate the sensitivity of LTA to its two parameters: tile width δ and sparsity control parameter η .
We use LunarLander, because it was the most difﬁcult discrete-action environments tested.
We use the same LTA setting as described in the above Section A.5.3.
For both η and δ , we sweep over 0 .
8 / 2 i , i ∈ { 0 , 1 , 2 , ..., 8 } .
Note that this range is extreme ( δ = 0 .
8 gives two bins, and η = 0 .
8 signiﬁcantly increases overlap); we do so to see a clear pattern.
Figure 13 shows the early learning performance of all possible 9 × 9 = 81 combinations of η, δ .
We report the average episodic return (rounded to integers) within the ﬁrst 300 k time steps, for each [Figure] combination.
The results are averaged over 5 runs.
The pattern is as expected.
The algorithms performs best with a reasonably small δ and η . 2) For extremely small η and δ , performance is poor, as expected because the gradient is very small and so learning is slow.
Given a ﬁxed tile width δ , the performance degrades as η becomes smaller than δ , since again the activation has many regions with zero derivative.
If η gets too large, the sparsity level increases and again performance degrades, though performance remained quite good for a broad range between 0 .
025 and 0 .
If δ gets large (big bins), performance degrades more so that with larger η , which matches the intuition that η provides overlap rather than just increasing bin size and so losing precision.
[Figure] [Table] [Figure]
A.5. E MPIRICAL RESULTS WHEN USING LTA TO BOTH HIDDEN LAYERS
As an activation function, LTA can be naturally applied in any hidden layer in a NN.
As a convention, we typically use the same activation functions in a fully connected NN.
In this section, we present such results on the reinforcement learning domains, as a supplement to the results in Section 5.
All setting is the same as those in Section 5, except that we apply LTA to both hidden layers in DQN-LTA and DDPG-LTA.
It should be noted that, in this case, DQN-LTA and DDPG-LTA have the same number of training parameters as their -Large correspondents respectively.
Figure 14 shows the results on the discrete domains, and Figure 15 shows the results on the continuous control domains.
It can be seen that, the LTA versions are better than ReLu versions in the case of not using a target network.
This further validates the utility of our LTA in dealing with nonstationary problems.
[Figure] [Figure]
A. R ESULTS ON IMAGE CLASSIFICATION TASKS
We now report the empirical results on two popular image classiﬁcation tasks:
MNIST (LeCun &
Cortes, 2010) and Mnistfashion (Xiao et al., 2017).
We found that LTA does not present any clear advantage or disadvantage in such a conventional supervised learning setting.
On MNIST, LTA achieves testing error 1 .
22% , and ReLu achieves 1 .
On Mnistfashion, LTA achieves testing error 10 .
67% , and ReLu achieves 10 .
Details.
The NN architecture we use is two convolution layer followed by two fully connected 32 × 32 layers.
The ﬁrst convolutional layer has 6 ﬁlters with size 5 × 5 and is followed by a max pooling operation.
The second convolutional layer has 16 ﬁlters with the same size followed by a max pooling.
LTA is applied to the second hidden layer and the setting is exactly the same as we used in the RL experiment A.4.
To optimize learning rate, we optimize over the range { 0 . 00003 , 0 . 00001 , 0 . 0003 , 0 . 0001 , 0 . 003 , 0 . 001 } .
We use 10 -fold cross validation to choose the best learning rate and the above error rate is reported on testing set by using the optimal learning rate at the end of learning.
The standard deviation of the testing error (randomness comes from NN initialization and random shufﬂing) is sufﬁciently small to get ignored.
A. C ONSTRUCTION OF P IECEWISE R ANDOM W ALK P ROBLEM
In Section 4, recall that we train with data generating process { ( X t , Y t ) } t ∈ N so that interference difﬁculty is controllable, while permitting fair comparison via a ﬁxed equilibrium distribution.
The temporal correlation in high difﬁculty { X t } t ∈ N is designed to mimic state space trajectories through MDPs with a high degree of local state space connectedness.
For example, an agent can only move to adjacent cells in GridWorld, the paddle and ball in Atari Breakout can only move so far in a single frame, and most successor states in Go only differ by a few pieces.
Figure 16 depicts sample trajectories across a range of difﬁculties, alongside the ﬁxed equilibrium distribution.
Here we rigorously construct { X t } t ∈ N and { S t } t ∈ N , and show they have the claimed properties.
Let random variable E t ∼ N ( (cid:15) ; 0 , σ 2 ) be a source of Gaussian noise for the recursive random variable S t
[Equation]
[Figure]
Note in particular the ﬁxed equilibrium distribution (
If c ∈ (0 , 1] , then { S t } t ∈ N is a linear Gaussian ﬁrst order autoregressive process, denoted AR(1).
Then the equilibrium distribution of { S t } t ∈ N is also Gaussian.
If S 0 is sampled from the equilibrium distribution, then the process { S t } t ∈ N is stationary with E [ S t ] = 0 and variance
[Equation]
It also follows that { S t } t ∈ N is ergodic.
See Grunwald et al.
(1995) for reference on AR(1) processes, especially the discussion of equation (3.3) for linear Gaussian AR(1) processes, and section 5.3 for their stationarity and ergodicity.
Let s t be a realization of S t .
We deﬁne X t | S t = s t as a Gaussian r.v.
with mean s t and variance β 2 :
[Equation]
The process { X t } t ∈ N is also a linear Gaussian AR(1) process, so we can again rely on established
AR(1) process theory to conclude that the equilibrium distribution of { X t } t ∈ N is Gaussian.
Moreover, if X 0 is sampled from the equilibrium distribution, then the process { X t } t ∈ N is stationary with
E [ X t ] = 0 and variance ξ
[Equation]
Refer to Theorem 4 in Section A.7.3 for proof that { X t } t ∈ N is a linear Gaussian AR(1) process, and that the resulting equilibrium distribution has variance speciﬁed by equation 12.
Note there are several variance terms in the above construction.
• σ 2 is the noise variance in the random walk,
i
e. the source of jumps in { S t } t ∈ N • ν 2 is the equilibrium distributon variance of { S t } t ∈ N • β 2 is the variance in any particular X t given s t at time step t
• ξ 2 is the equilibrium distribution variance of { X t } t ∈ N
The processes { S t } t ∈ N and { X t } t ∈ N are fully determined by the quantities c, σ 2 , β 2 .
For our experiments we wish to have a single parameter d such that the equilibrium distribution of { X t } t ∈ N is ﬁxed for all d ∈ [0 , 1) , but temporal correlation is variable.
Also, d = 0 should correspond to zero temporal correlation.
The following two theorems specify this behaviour more rigorously.
Theorem 2.
Let difﬁculty parameter d ∈ [0 , 1) and ξ 2 = ( B 2 ) 2 for some B ∈ R + .
Then { X t } t ∈ N will have ﬁxed equilibrium distribution N (0 , ξ 2 ) , invariant to d , if parameters c, σ 2 , β 2 are set as follows
[Equation]
ξ 2 is deﬁned in terms of B ∈ R + simply because B is a more intuitive design parameter.
In particular,
B is a high probability bound with P ( X t ∈ [ − B, B ]) ≤ 0 .
95 for all t , since { X t } t ∈ N is ergodic.
Theorem 3.
d = 0 and S
= 0 induces i.i.d X t from N ( x ; 0 , ξ 2 ) , the equilibrium distribution of { X t } t ∈ N .
We defer proof of both theorems to Section A.7.3.
Correlation difﬁculty d is intuitively characterized as follows
• As d increases, β 2 decreases, so a smaller portion of the overall state space is supported at any given time.
i.e. P ( X t = x | S t = s t ) becomes increasingly narrow, inducing higher temporal correlation.
• As d increases, the noise in random walk { S t } t ∈ N increases in amplitude, so that larger jumps in the mean of any particular X t are more likely.
• At d = 0 , all correlation difﬁculty from { X t } t ∈ N has been removed, in that we recover iid sampling.
• As d → 1 , { X t } t ∈ N converges towards pathological correlation, with X t d −→ δ ( s t ) , where δ ( · ) is the Dirac delta distribution.
i.e. X t becomes constant everywhere except the jumps in realization s t of S t .
Despite the above relationships between d and the other process parameters, the equilibrium distribution N ( x ; 0 , ξ 2 ) is identical for all difﬁculty settings d ∈ [0 , 1) , because the increase (or decrease) in parameters σ 2 , c is tuned speciﬁcally to counteract the decrease (or increase) in β 2 .
Having identical equilibrium distributions means that a hypothetical ideal algorithm, capable of perfectly mitigating interference, would train the identical approximator f θ for any correlation difﬁculty d ∈ [0 , 1) .
Hence, we use approximation loss on the equilibrium distribution as a measure of robustness to interference.
To summarize, correlation difﬁculty d ∈ [0 , 1) controls the likelihood of interference-inducing samples throughout training, but also permits fair comparison between values of d by the (ﬁxed) equilibrium distribution of { X t } t ∈ N .
A.7. P IECEWISE R ANDOM W ALK A DDITIONAL P LOTS
Figure 17 depicts learning curves of loss over training time at different levels of correlation difﬁculty. Figure 18 shows learning rate sensitivity curves for the LTA and ReLU networks for three different difﬁculty settings.
The ADAM optimizer Kingma & Ba (2015) was used in all experiments. For the experiment in Figure 3, where one sample from each X t is used for each weight update, LTA outperforms ReLU even on i.i.d data ( d = 0 ). In order to ﬁnd the conditions under which ReLU and
LTA both perform equally well, we repeat the same experiment, but with 50 samples drawn from each X t . Figure 20 depicts the results, where the performance is indeed equalized for iid sampling ( d = 0 ), but ReLU’s performance still diverges similarly to other experiments as correlation difﬁculty is increased.
The size of the LTA and ReLU neural networks used on the Piecewise Random Walk Problem were chosen based on best performance on iid data, and the best performers do not have the same number of parameters.
(67K vs 5.2K learnable parameters.)
In order to verify that the difference in parameter count was not a signiﬁcant factor in handling high correlation difﬁculty, we additionally include results for a ReLU network with wider hidden layers totalling 81K learnable parameters.
See Figure 19.
Both ReLU networks perform very similarly—in most cases agreeing within p = 0 . 05 —with loss steeply running away as correlation difﬁculty increases.
So we conclude LTA’s robustness to high correlation difﬁculties is not explained simply by parameter count. A.7. P IECEWISE R ANDOM W ALK P ROBLEM H YPERPARAMETER S ELECTION Experimental Parameters Conﬁguration for Piecewise Random Walk problem across all experiments: Iterations per training run 20K
Test batch size per iteration Train batch size per iteration Min, max difﬁculty settings d (0.0, 0.98) Number of difﬁculty settings swept 20 (linear range over (min, max), inclusive)
Training runs per difﬁculty setting 10 for ReLU
Large (Figure 19) 10 for batched run (Figure 20) 30 otherwise
Number of piecewise stationary segments
Target function
[Equation]
High probability bound B ( − 1 ,
[Equation]
Conﬁguration for the neural networks used on the Piecewise Random Walk problem:
LTA
Hidden layers
Layer width w
Layer binning interval bounds ( l, u ) ( − 1 , 1)
Layer bins k
Layer sparsity parameter η
ReLU
Hidden layers
Layer width w
ReLU Large
Hidden layers
Layer width w
Layer widths w , bin count k , and sparsity parameter η were chosen by sweeping a range, and choosing the best performing conﬁguration on the iid setting (i.e. difﬁculty d = 0 ).
In the sweep, the same w, k, η are used for each hidden layer of LTA, and the same w is used for each hidden layer of ReLU.
See the Parameter Selection Details below.
For all three networks, the ADAM optimizer Kingma &
Ba (2015) was used with parameters ( β 1 , β 2 ) =
(0 . 9 , 0 . 999) , and the following learning rates were swept for each difﬁculty setting.
[Table]
The results from the best performing learning rate are reported in Figures 3(c), 17(c), 19(c), and 20.
That is, the performing learning rate is determined separately for each difﬁculty setting, using the mean ﬁnal loss across 30 runs for the LTA and ReLU networks.
10 runs were used for ReLU Large and the batching run.
Results from all learning rates are reported in Figure 18 for three different difﬁculty settings.
Parameter Selection Details
Before sweeping any difﬁculty settings, ﬁrst the network architectures were established for the
LTA and ReLU networks so that loss was minimized on the iid setting ( d = 0 ).
We avoid the overparametrized regime, so that standard bias-variance tradeoff applies here Belkin et al.
(However, the ReLU Large network is overparametrized.)
Learning rate was also swept during the
[Figure] [Figure]
region corresponding p = 0 .
These curves corroborate our ﬁndings that, in general, LTA prefers lower learning rates (but converges more quickly nonetheless
[Figure]
[Figure]
[Figure]
architecture sweep, to ensure that the chosen architecture performed best within the learning rate range chosen for the main experiment depicted in Figure 3(c).
For the 2 layer LTA network, each hidden layer uses the same width w , and sparsity parameter η .
Those parameters, along with learning rate λ were optimized in the following ranges with grid search:
• Layer width w ∈ { 10 , 15 , 20 , 30 , 40 } • Sparsity parameter η = 1 w • Learning rate λ ∈ { 0 . 0005 , 0 . 0001 , 0 . 00005 , 0 .
For the 2 layer ReLU network, fully connected hidden layers were used.
Width w and learning rate λ were found from the following ranges, also with grid search:
• Layer width w ∈ { 5 , 10 , 20 , 30 , 40 , 50 , 60 , 70 , 80 , 100 , 120 } • Learning rate λ ∈ { 0 .
The learning rates are of different magnitude between LTA and ReLU because they perform best in different ranges, as explained in A.7.1.
For the main experiments, where a range of difﬁculties are swept, the range of tested learning rates is shared between LTA and ReLU, and is broad enough to cover the optimal range for both.
Also note that we set η = 1 k , rather than separately sweeping it.
This is because LTA performs consistently well with η in this order of magnitude.
A.7. P ROOFS FOR COVARIATE SHIFT PROPERTIES
Theorem 4.
Let { E t } t ∈ N , { S t } t ∈ N , { X t } t ∈ N be stochastic processes such that
[Equation]
where c ∈ (0 , 1] and s t is a realization of S t .
Then { X t } t ∈ N is a linear Gaussian AR(1) process with equilibrium distribution N ( x ; 0 , β 2 + σ 2 2 c − c 2 ) .
Proof.
Begin by rewriting equation 14 to give X t as a sum of S t and a mean zero Gaussian r.v.
B t
[Equation]
Now substitute according to equation
[Equation]
Let Gaussian r.v.
Θ ∼ N (0 , α 2 ) with some variance
α 2 .
E t − 1 , B t are independent Gaussian, so E t − 1 + B t can be written as a linear combination of B t − 1 , Θ t , since all are independent.
Speciﬁcally, ﬁx α 2 so that the following holds
[Equation]
and substitute into equation
[Equation]
By inspection, { X t } t ∈ N is a linear Gaussian AR(1) process with coefﬁcient (1 − c ) and noise variance α 2 .
Elementary AR process theory gives the equilibrium distribution as
[Equation]
For equation 17 to hold, we need the ﬁrst and second moments of LHS and RHS to be equal.
All terms are mean zero Gaussian, so it sufﬁces to show when the LHS and RHS have equal variance:
[Equation]
Substituting into equation
[Equation]
So { X t } t ∈ N is linear Gaussian AR(1) with the desired equilibrium distribution.
Theorem
Let difﬁculty parameter d ∈ [0 , 1) and ξ 2 = ( B 2 ) 2 for some B ∈ R + .
Then { X t } t ∈ N will have ﬁxed equilibrium distribution N (0 , ξ 2 ) , invariant to d , if parameters c, σ 2 , β 2 are set as follows
[Equation]
Proof.
By the above theorem 4, ξ 2 = β 2 +
[Equation]
= ( B 2 ) 2 .
This completes the proof.
Theoremd = 0 and S
= δ (0) induces iid X t from N ( x ; 0 , ξ 2 ) , the equilibrium distribution of { X t } t ∈ N .
Proof.
[Equation]
Hence, d = 0 implies S t is constant over all time t , and S
= δ (0) gives S t =
δ (0) for all time t .
Let r.v.
B t ∼ N (0 , β 2 ) iid, then X t =
S t + B t = B t when S t is Dirac delta concentrating at zero.
β
= (1 − d )( B 2 ) 2 = ( B 2 ) 2 by the setting from Theorem 2 and setting d = 0 .
Hence, d = 0 induces iid sampling from the equilibrium distribution of X t .
This completes the proof.