A CTIVE T UNING
Anonymous authors
Paper under double-blind review
A BSTRACT
We introduce Active Tuning, a novel paradigm for optimizing the internal dynamics of recurrent neural networks (RNNs) on the ﬂy.
In contrast to the conventional sequence-to-sequence mapping scheme, Active Tuning decouples the RNN’s recurrent neural activities from the input stream, using the unfolding temporal gradient signal to tune the internal dynamics into the data stream.
As a consequence, the model output depends only on its internal hidden dynamics and the closedloop feedback of its own predictions; its hidden state is continuously adapted by means of the temporal gradient resulting from backpropagating the discrepancy between the signal observations and the model outputs through time.
In this way, Active Tuning infers the signal actively but indirectly based on the originally learned temporal patterns, ﬁtting the most plausible hidden state sequence into the observations.
We demonstrate the effectiveness of Active Tuning on several time series prediction benchmarks, including multiple super-imposed sine waves, a chaotic double pendulum, and spatiotemporal wave dynamics.
Active Tuning consistently improves the robustness, accuracy, and generalization abilities of all evaluated models.
Moreover, networks trained for signal prediction and denoising can be successfully applied to a much larger range of noise conditions with the help of Active Tuning.
Thus, given a capable time series predictor, Active Tuning enhances its online signal ﬁltering, denoising, and reconstruction abilities without the need for additional training.
I NTRODUCTION
Recurrent neural networks (RNNs) are inherently only robust against noise to a limited extent and they often generate unsuitable predictions when confronted with corrupted or missing data (cf., e.g., Otte et al., 2015).
To tackle noise, an explicit noise-aware training procedure can be employed, yielding denoising networks, which are targeted to handle particular noise types and levels.
Recurrent oscillators, such as echo state networks (ESNs) (Jaeger, 2001; Koryakin et al., 2012; Otte et al., 2016), when initialized with teacher forcing, however, are highly dependent on a clean and accurate target signal.
Given an overly noisy signal, the system is often not able to tune its neural activities into the desired target dynamics at all.
Here, we present a method that can be seen as an alternative to regular teacher forcing and, moreover, as a general tool for more robustly tuning and thus synchronizing the dynamics of a generative differentiable temporal forward model—such as a standard RNN, ESN, or LSTM-like RNN (Hochreiter & Schmidhuber, 1997; Otte et al., 2014; Chung et al., 2014; Otte et al., 2016)—into the observed data stream.
The proposed method, which we call Active Tuning, uses gradient back-propagation through time (BPTT)
(Werbos, 1990), where the back-propagated gradient signal is used to tune the hidden activities of a neural network instead of adapting its weights.
The way we utilize the temporal gradient signal is related to learning parametric biases (Sugita et al., 2011) and applying dynamic context inference (Butz et al., 2019).
With Active Tuning, two essential aspects apply: First, during signal inference, the model is not driven by the observations directly, but indirectly via prediction error-inducted temporal gradient information, which is used to infer the hidden state activation sequence that best explains the observed signal.
Second, the general stabilization ability of propagating signal hypotheses through the network is exploited, effectively washing out activity components (such as noise) that cannot be modeled with the learned temporal structures within the network.
As a result, the vulnerable internal dynamics are kept within a system-consistent activity milieu, effectively decoupling it from noise or other unknown distortions that are present in the defective actual signal.
In this work we show that Active Tuning elicits enhanced signal ﬁltering abilities, without the need for explicitly training distinct models for exactly such purposes.
Excitingly, this method allows for instance the successful application of an entirely noise-unaware RNN (trained on clean, ideal data) under highly noisy and unknown conditions.
In the following, we ﬁrst detail the Active Tuning algorithm.
We then evaluate the RNN on three time series benchmarks—multiple superimposed sine waves, a chaotic pendulum, and spatiotemporal wave dynamics.
The results conﬁrm that Active Tuning enhances noise robustness in all cases.
The mechanism mostly even beats the performance of networks that were explicitly trained to handle a particular noise level.
It can also cope with missing data when tuning the predictor’s state into the observations. In conclusion, we recommend to employ Active Tuning in all time series prediction cases, when the data is known to be noisy, corrupted, or to contain missing values and the generative differentiable temporal forward model—typically a particular RNN architecture—knows about the potential underlying system dynamics.
The resulting data processing system will be able to handle a larger range of noise and corrupted data, ﬁltering the signal, generating more accurate predictions, and thus identifying the underlying data patterns more accurately and reliably.
A CTIVE T UNING
Starting point for the application of Active Tuning is a trained temporal forward model.
This may be, as mentioned earlier, an RNN, but could also be another type of temporal model.
The prerequisite is, however, a differentiable model that implements dependencies over time, such that BPTT can be used to reversely route gradient information through the computational forward chain.
Without loss of generality, we assume that the model of interest, whose forward function may be referred to as f M , fulﬁlls the following structure:
[Equation]
where σ t is the current latent hidden state of the model (e.g. the hidden outputs of LSTM units, their cell states, or any other latent variable of interest) and x t is the current signal observation.
Based on this information f M generates a prediction for the next input ˜ x t +1 and updates its latent state σ t + accordingly.
Following the conventional inference scheme, we feed a given sequence time step by time step into the network and receive a one-time step ahead prediction after each particular step.
Over time, this effectively synchronizes the network with the observed signal.
Once the network dynamics are initialized, which is typically realized by teacher forcing, the network can generate prediction and its dynamics can be driven further into the future in a closed-loop manner, whereby the network feeds itself with its own predictions.
To realize next time step- and closed-loop predictions, direct contact with the signal is inevitable to drive the teacher forcing process.
In contrast, Active Tuning decouples the network from the direct inﬂuence of the signal.
Instead, the model is permanently kept in closed-loop mode, which initially prevents the network from generating meaningful predictions.
Over a certain time frame, Active Tuning keeps track of the recent signal history, the recent hidden states of the model, as well as its recent predictions.
We call this time frame (retrospective) tuning horizon or tuning length (denoted with R ).
The principle of Active Tuning can best be explained with the help of Figure 1 and Algorithm 1.
The latter gives a more formal perspective onto the principle.
Note that for every invocation of the procedure a previously unrolled forward chain (from the previous invocation or an initial unrolling) is assumed.
L refers to the prediction error of the entire unrolled prediction sequence and the respective observations, whereas L t (cid:48) refers to the local prediction error just for a time step t (cid:48) .
With every new perceived and potentially noise-affected signal observation x t , one or multiple tuning cycles are performed.
Every tuning cycle hereby consists of the following stages:
First, from the currently believed sequence of signal predictions (which is in turn based on a sequence of hidden states) and the actual observed recent inputs, a prediction error is calculated and propagated back into the past reversely along the unfolded forward computation sequence.
The temporal gradient travels to the very left of the tuning horizon and is ﬁnally projected onto the seed hidden state σ t − R , which is then adapted by applying the gradient signal in order to minimize the encountered prediction error.
This adaption can be done using any gradient-based optimizer.
Note that in this paper, we exclusively f M f M ... f M f M σ t 9 R σ t ... ... ... ... ... ...
f M f M ... f M f M σ t 9 R σ t ˜ x t ˜ x t 9 1 ˜ x t 9 2 ...
˜ x t 9 R +2 ˜ x t 9 R +1 ˜ x t 9 R ˜ x t ˜ x t 9 1 ˜ x t 9 2 ...
˜ x t 9 R +2 ˜ x t 9 R +1 ˜ x t 9 R x t x t 9 1 x t 9 R +2 x t 9 R + x t x t 9 1 x t 9 R +2 x t 9 R + f M f M ... f M f M σ t 9 R + σ t + ... ... ... ... ... ...
f M f M ... f M f M σ t 9 R + σ t + ˜ x t +1 ˜ x t
˜ x t 9 1 ...
˜ x t 9 R +3 ˜ x t 9 R +2 ˜ x t 9 R + ˜ x t +1 ˜ x t
˜ x t 9 1 ...
˜ x t 9 R +3 ˜ x t 9 R +2 ˜ x t 9 R + x t +1 x t x t 9 R +3 x t 9 R + x t +1 x t x t 9 R +3 x t 9 R + Input Output f M
Generate do / Perform do / propagate error g ← ∂ L ∂ σ t (cid:48) = ∂ L t (cid:48)
∂ σ t (cid:48)
+   ∂
L ∂ σ t (cid:48)
+ ∂ σ t (cid:48)
+ ∂ σ t (cid:48)
otherwise end do use Adam (Kingma & Ba, 2015), but other optimizers are possible as well.
Second, after the adaption of this seed state (and maybe the seed input as well) the prediction sequence is rolled out from the past into the present again, effectively reﬁning the output sequence towards a better explanation of the recently observed signal.
Each tuning cycle thus updates the current prediction ˜ x t and the current hidden state σ t from which a closed-loop future prediction can be rolled out, if desired.
To transition into the next world time step, one forward step has to be computed.
The formerly leftmost seed states can be discarded and the recorded history is shifted by one time step, making σ t − R + the new seed state that will be tuned within the next world time step.
From then on, the procedure is repeated, yielding the continuous adaptive tuning process.
As a result, the model is predominantly driven by its own imagination, that is, its own top down predictions.
Meanwhile, the predictions themselves are adapted by means of the temporal gradients based on the accumulated prediction error, but not by the signal directly.
In a nutshell, Active Tuning realizes a gradient-based mini-optimization procedure on any of the model’s latent variables within one world time step.
While it needs to be acknowledged that this process draws on additional computational resources, in this paper we investigate the resulting gain in signal processing robustness in the case of noisy data.
Intuitively speaking, Active Tuning tries to ﬁt known temporal patterns, as memorized within the forward model, to the concurrently observed data.
Due to the strong pressure towards consistency maintenance, which is naturally enforced by means of the temporal gradient information in combination with the repeatedly performed forward passes of the hidden state activities, the network will generate adaptations and potential recombinations of patterns that it has learned during training.
Occurrences that cannot be generated from the repertoire of neural dynamics will therefore not appear (or only in signiﬁcantly suppressed form) in the model’s output.
As a consequence, there is a much smaller need to strive for noise robustness during training.
Our results below indeed conﬁrm that the model may be trained on clean, idealized target signals.
However, imprinting a slight denoising tendency during training proves to be useful when facing more noisy data.
Enhanced with our Active Tuning scheme, the model will be able to robustly produce high-quality outputs even under extremely adverse conditions—as long as (some of) the assumed target signals are actually present.
Our scheme is thus a tool that can be highly useful in various application scenarios for signal reconstruction and ﬂexible denoising.
Nevertheless, it should be mentioned that with Active Tuning the computational overhead for inference scales with the number of tuning cycles and tuning length.
E XPERIMENTS
In order to investigate the abilities of Active Tuning we studied its behavior at considering three different types of time series data, namely, one-dimensional linear dynamics, two-dimensional nonlinear dynamics, and distributed spatiotemporal dynamics.
For all three problem domains we used a comparable setup except for the particular recurrent neural network architectures applied.
We trained the networks as one time step ahead predictors whose task is to predict the next input given both the current input and the history of inputs aggregated in the latent hidden state of the models.
The target sequences were generated directly from the clean input sequences by realizing a shift of one time step.
Moreover, we trained networks under six different denoising conditions (normally distributed) per experiment, where we fed a potentially noisy signal into the network and provide the true signal (one time step ahead) as the target value (Lu et al., 2013; Otte et al., 2015; Goodfellow et al., 2016).
These conditions are determined by their relative noise ratios: 0 .
(no noise), 0 .
5 , and 1 . 0 , where the ratios depend on the respective base signal statistics.
For instance, a noise ratio of 0 .
1 means that the noise added to the input has a standard deviation of 0 .
1 times the standard deviation of the base signal.
As a result we obtained predictive denoising experts for each of these conditions.
All models were trained with Adam (Kingma & Ba, 2015) using its default parameters (learning rate η = 0 .
001 , β 1 = 0 .
9 and β 2 = 0 .
over 100 (ﬁrst two experiments) or 200 (third experiment) epochs, respectively.
3. M ULTI -S UPERIMPOSED O SCILLATOR
The ﬁrst experiment is a variant of the multiple superimposed oscillator (MSO) benchmark (Schmidhuber et al., 2007; Koryakin et al., 2012; Otte et al., 2016).
Multiple sine waves with different frequencies, phase-shifts, and amplitudes are superimposed into one signal:
[Equation]
where n gives the number of superimposed waves, f i the frequency, a i the amplitude, and ϕ i the phase-shift of each particular wave, respectively.
Typically, the task on this benchmark is to predict the further progression of the signal given some initial data points (e.g. the ﬁrst 100 time steps) of the sequence.
The resulting dynamics are comparably simple as they can, in principle, be learned with a linear model.
It is, however, surprisingly difﬁcult for BPTT-based models, namely LSTM-like
RNNs, to precisely continue a given sequence for more than a few time steps (Otte et al., 2019).
For this experiment we considered the MSO 5 dynamics with the default frequencies f 1 = 0 .
2 , f 2 = 0 .
311 , f 3 = 0 .
42 , f 4 = 0 .
51 , and f 5 = 0 .
63 (see the ground truth in Figure 3 for an example).
For training, we generated 10 000 examples with 400 time steps each, using random amplitudes a i ∼ [0 , 1] and random phase-shifts ϕ
i ∼ [0 , 2 π ] .
For testing, another 1 000 examples were generated.
As base model, we used an LSTM network with one input, 32 hidden units, one linear output neuron, and no biases.
3. C HAOTIC P ENDULUM
The second experiment is based on the simulation of a chaotic double pendulum.
As illustrated in Figure 2, the double pendulum consists of two joints whose angles are speciﬁed by θ 1 and θand two rods of length l 1 and l 2 .
Besides the length of the rods, the masses m 1 and m 2 affect the behavior of the pendulum.
The pendulum’s end-effector (where m 2 is attached) generates smooth, but highly non-linear trajectories.
More precisely, it exhibits chaotic behavior, meaning that already slight changes of the current system state can quickly cause major changes of the pendulum’s state over time (Korsch et al., 2008; Pathak et al., 2018).
It is thus typically difﬁcult to precisely predict the dynamics of such a system for more than a few time steps into the future, making it a challenging benchmark problem for our purposes.
In the literature, the double pendulum’s dynamics are typically described using the equations of motion, given by Equation 3 and Equation 4, respectively, which are derived from the Lagrangian of the system and the Euler-Lagrange equations; see Korsch et al.
for details.
¨ θ 1 = µg 1 sin( θ 2 ) cos( θ
[Equation]
where
[Equation]
and g = 9 .
81 being the gravitational constant.
For simulating the double pendulum, we applied the fourth-order Runge-Kutta (RK4) (Press, 2007) method to numerically integrate the equations of motion.
All four parameters l 1 , l 2 , m 1 , and m 2 were set to 1 . 0 .
A temporal step size of h = 0 .
01 was chosen for numerical integration.
The initial state of the pendulum is described by its two angles, which were selected randomly for each sample to be within θ 1 ∼ [90 ◦ , 270 ◦ ] and θ 2 ∼ [ θ 1 ± 30 ◦ ] to ensure sufﬁcient energy in the system.
One out of ten sequences was initiated with zero angle momenta, that is ˙ θ 1 , ˙ θ 2 = 0 . 0 .
The number of train and test samples, as well as the sequence lengths were chosen analogously to experiment one.
As base model we used an LSTM network with two inputs, 32 hidden units, two linear output neurons, and again no biases. [Figure]
3. S PATIOTEMPORAL W AVE D YNAMICS
In the third experiment we considered a more complex spatiotemporal wave propagation process, based on the following wave dynamics:
[Equation]
This equation was solved numerically using the method of second order central difference, yielding
[Equation]
with, after solving ∂ 2 u/∂x 2 and analogously ∂ 2 u/∂y 2 via the same method,
[Equation]
Here, x and y correspond to a spatial position in the simulated ﬁeld, while t denotes the time step and c = 3 .
the propagation speed factor of the waves.
The temporal and spatial approximation step sizes were set to h t = 0 .
1 and h x = h y
= 1 . 0 , respectively.
No explicit boundary condition was applied, resulting in the waves being reﬂected at the borders and the overall energy staying constant over time.
We generated sequences for a regular grid of 16 × 16 pixels (see Figure 5 for illustrations of the two-dimensional wave).
In contrast to the previous two experiments, 200 samples with a sequence length of 80 were generated for training, whereas 20 samples over 400 time steps were used for evaluation.
As base network we used a distributed graph-RNN called DISTANA (Karlbauer et al., 2020), which is essentially a mesh of the same RNN module (an LSTM, which consists here of four units only), which is distributed over the spatial dimensions of the problem space (here a two-dimensional grid), where neighboring modules are laterally connected.
We chose this wave benchmark, and this recurrent graph network in particular, to demonstrate the effectiveness of Active Tuning in a setup of higher complexity.
Moreover, to contrast our results with another state-of-theart sequence-to-sequence model, we trained temporal convolution networks (TCNs) (Kalchbrenner et al., 2016; Bai et al., 2018), here with three layers, having one, eight, and one feature maps, respectively, using 3 × 3 spatial kernel sizes and standard dilation rates for the temporal dimension.
R ESULTS AND D ISCUSSION
The quantitative evaluations are based on the root mean square error (RMSE) between the network outputs and the ground truth targets.
All reported values are averages over ten independently randomly initialized and trained models.
In order to elaborate on the applicability of each denoising expert on unseen noise conditions, we evaluated all models using the noise ratios 0 . 0 , 0 . 1 , 0 . 2 , 0 . 5 , and 1 . 0 , resulting in 25 baseline scores for each experiment.
These baselines were compared on all noise ratios against eight Active Tuning setups, which were based on models trained without any noise ( 0 . 0 ) or with only a small portion of input noise ( 0 . 05 ), respectively.
The individual parameters of Active Tuning that were used to produce the results are reported in Table 6, Table 7, and Tableof the appendix.
Note that in all experiments, the latent hidden outputs of the LSTM units (not the cell states) were chosen as optimization target for the Active Tuning algorithm.
Furthermore, these hidden states were initialized normally distributed with standard deviation 0 .
1 in all cases, whereas the cell state were initialized with zero. 4. MSO R ESULTS
The results of the MSO experiments are summarized in Table 1.
Active Tuning improves the results for the weakest model ( 0 . 0 ) in all cases (column 2 vs. column 6), partially almost by an order of magnitude.
Noteworthy, for the inference noise ratio 0 .
1 the noise-unaware model becomes even better than the actual expert when driven with Active Tuning.
Recall that there was no retraining of the model, only the paradigm how the models is applied to the data was changed.
On the other hand, at least in this experiment, it seems that there is no advantage for Active Tuning when the
[Table]
[Table]
[Figure]
base network encountered minimal noise (0.05) during training.
To get an impression of the actual improvement of the output quality, consider Figure 3.
The noise-unaware model (0 . 0) produces very poor predictions when confronted with strong signal noise (1 . 0) .
Yet, when driven with Active
Tuning instead of with regular inference (teacher forcing), the output of the same model is enhanced signiﬁcantly, becomes smooth, and approximates the ground truth reasonably well.
Active Tuning thus helps to catch most of the trend information while mostly ignoring noise.
As an additional evaluation, Table 2 demonstrates the ability of Active Tuning to cope with missing data.
The results are based on the noise-unware model.
While tuning into the signal, particular observation are missing (dropped out) with a certain probability ranging from 0.1 to 0.9. In case of a missing observation, the prediction of the model is used instead. Already with a dropout chance of 20 % the RNN struggles to tune its hidden neural states into the signal, thus generating an error larger than 1. In contrast, exactly the same RNN model remains signiﬁcantly more stable when driven with Active Tuning.
Even with a dropout chance of 50 – 60 % the RNN still produces errors clearly below the reference error of approximately 0.9, which is the RMSE error generated when always predicting zero.
Note that with regular inference, the error decreases slightly with the highest dropout rate.
This is the case because here the network receives so few inputs such that it starts to produce zero outputs for some sequences.
For the pendulum experiment, the potential of Active Tuning becomes even more evident. The results presented in Table 3 indicate that for all noise ratios Active Tuning performs better than all experts, in particular, when applied to the model that was trained on small noise (0.05).
With increasing noise level, the problem becomes progressively impossible to learn. For example, the 1.0-expert-model does not seem to provide any reasonable function at all, indicated by the worse RMSE score compared to other models ( 1 . 0 inference noise row).
In contrast, Active Tuning can [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
still handle these extremely unfavorable conditions surprisingly well.
Figure 4 shows an exemplary case.
The unknown ground truth is plotted against the noisy observations (shown in the left image).
The center image shows the prediction of the reference LSTM (trained with 0.05 noise) when regular inference is applied.
It is clearly difﬁcult to recognize a coherent trajectory reﬂecting the dynamics of the double pendulum.
With Active Tuning, on the other hand, the same network produces a mostly clean prediction that is relatively close to the ground truth sequence. 4. W AVE R ESULTS
The results of the wave experiment reported in Table 4 consistently support the ﬁndings from the pendulum experiments.
When driven with Active Tuning, the considered models produce better results than the explicitly trained denoising experts on all noise levels.
As shown in Figure 5 and in accordance with the previous experiments, the noisy signal observations (1.0) could be ﬁltered effectively and latency-free (exclusively when using Active Tuning) to yield a smooth signal prediction across the entire spatiotemporal sequence.
While the two-dimensional output of the network operating in conventional inference mode is hardly recognizable as a wave, the network output of the same model combined with Active Tuning clearly reveals the two-dimensional wave structure with hardly perceivable deviations from the ground truth.
We furthermore compared performance with a common, non-recurrent, sequence-to-sequence learning architecture. Here we considered a standard TCN architecture (Bai et al., 2018), which we also trained to focus on all considered denoising levels.
The full performance table in shown in the appendix (Table 5).
The ﬁrst result column in Table 4 shows that the best TCN performance is mostly outperformed. Importantly, DISTANA with active tuning outperforms TCN on all noise levels even when the DISTANA model was not trained for denoising at all. Note that we also performed experiments with other noise distributions (e.g. salt-and-pepper noise).
Somewhat surprisingly the quality of the output was only marginally affected by these variations.
Thus, in contrast to what is known about deep convolutional networks (Geirhos et al., 2018), the denoiser RNNs applied here apparently did not overﬁt to the noise type but only to the energy range of the noise. [Table] [Figure]
C ONCLUSION
In this work we augmented RNN architectures with Active Tuning, which decouples the internal dynamics of an RNN from the data stream.
Instead of relying on the input signal to set the internal network states, Active Tuning uses the dynamic loss signal, which is projected onto the internal states, effectively tuning them.
We have shown that RNNs driven with Active Tuning can reliably denoise various types of time series dynamics, mostly yielding higher accuracy than speciﬁcally trained denoising expert RNNs.
In all cases, the augmentation with Active Tuning has beaten the standard RNN with teacher forcing.
Nonetheless, Active Tuning can in principle be mixed with traditional teacher forcing.
We will investigate potential synergies in the future, especially switching teacher forcing on and off in an adaptive manner depending on the current signal conditions.
While we used a tuning length of up to 16 time steps with partially up to 30 tuning cycles, additional future research will aim at reducing these requirements.
Ideally, Active Tuning will work reliably with a single update cycle over a tuning length of a very few time steps, which would allow to perform Active Tuning along with the regular forward pass of the model in a fused computation step.
Additionally, we aim at applying Active Tuning to real-world denoising and forecasting challenges, including speech recognition and weather forecasting.
A A PPENDIX A. T EMPORAL C ONVOLUTION R ESULTS
The performance of the temporal convolution models at predicting and denoising the spatiotemporal wave dynamics are reported in the following table (Table 5) [Table]
The two Active Tuning parameters (tuning length R and number of tuning cycles C ), as well as the parameters for state adaptation with the Adam optimizer (tuning rate η , β 1 and β 2 ) for the application of Active Tuning on the different benchmarks and various noise combinations are reported in distinct tables below.
[Table]
[Table] [Table]