A T EMPORAL K ERNEL A PPROACH FOR D EEP L EARNING WITH C ONTINUOUS - TIME I NFORMATION
A BSTRACT
Sequential deep learning models such as RNN, causal CNN and attention mechanism do not readily consume continuous-time information.
Discretizing the temporal data, as we show, causes inconsistency even for simple continuous-time processes.
Current approaches often handle time in a heuristic manner to be consistent with the existing deep learning architectures and implementations.
In this paper, we provide a principled way to characterize continuous-time systems using deep learning tools.
Notably, the proposed approach applies to all the major deep learning architectures and requires little modiﬁcations to the implementation.
The critical insight is to represent the continuous-time system by composing neural networks with a temporal kernel, where we gain our intuition from the recent advancements in understanding deep learning with Gaussian process and neural tangent kernel.
To represent the temporal kernel, we introduce the random feature approach and convert the kernel learning problem to spectral density estimation under reparameterization.
We further prove the convergence and consistency results even when the temporal kernel is non-stationary, and the spectral density is misspeciﬁed.
The simulations and real-data experiments demonstrate the empirical effectiveness of our temporal kernel approach in a broad range of settings.
I NTRODUCTION
Deep learning models have achieved remarkable performances in sequence learning tasks leveraging the powerful building blocks from recurrent neural networks (RNN) (Mikolov et al., 2010), longshort term memory (LSTM)
(Hochreiter & Schmidhuber, 1997), causal convolution neural network (CausalCNN/WaveNet)
(Oord et al., 2016) and attention mechanism (Bahdanau et al., 2014; Vaswani et al., 2017).
Their applicability to the continuous-time data, on the other hand, is less explored due to the complication of incorporating time when the sequence is irregularly sampled (spaced).
The widely-adopted workaround is to study the discretized counterpart instead, e.g. the temporal data is aggregated into bins and then treated as equally-spaced, with the hope to approximate the temporal signal using the sequence information.
It is perhaps without surprise, as we show in Claim 1, that even for regular temporal sequence the discretization modiﬁes the spectral structure.
The gap can only be ampliﬁed for irregular data, so discretizing the temporal information will almost always introduce intractable noise and perturbations, which emphasizes the importance to characterize the continuous-time information directly.
Previous efforts to incorporate temporal information for deep learning include concatenating the time or timespan to feature vector (Choi et al., 2016;
Lipton et al., 2016; Li et al., 2017b), learning the generative model of time series as missing data problems (Soleimani et al., 2017; Futoma et al., 2017), characterizing the representation of time (Xu et al., 2019; 2020; Du et al., 2016) and using neural point process (Mei & Eisner, 2017; Li et al., 2018).
While they provide different tools to expand neural networks to coupe with time, the underlying continuous-time system and process are involved explicitly or implicitly.
As a consequence, it remains unknown in what way and to what extend are the continuous-time signals interacting with the original deep learning model.
Explicitly characterizing the continuous-time system (via differential equations), on the other hand, is the major pursuit of classical signal processing methods such as smoothing and ﬁltering (Doucet & Johansen, 2009; Särkkä, 2013).
The lack of connections is partly due to the compatibility issues between the signal processing methods and the auto-differential gradient computation framework of modern deep learning.
Generally speaking, for continuous-time systems, model learning and parameter estimation often rely on the more complicated differential equation solvers (Raissi & Karniadakis, 2018; Raissi et al., 2018a).
Although the intersection of neural network and differential equations is gaining popularity in recent years, the combined neural differential methods often require involved modiﬁcations to both the modelling part and implementation detail (Chen et al., 2018;
Baydin et al., 2017).
Inspired by the recent advancement in understanding neural network with Gaussian process and the neural tangent kernel (Yang, 2019; Jacot et al., 2018), we discover a natural connection between the continuous-time system and the neural Gaussian process after composing with a temporal kernel.
The signiﬁcance of the temporal kernel is that it ﬁlls in the gap between signal processing and deep learning: we can explicitly characterize the continuous-time systems while maintaining the usual deep learning architectures and optimization procedures.
While the kernel composition is also known for integrating signals from various domains (Shawe-Taylor et al., 2004), we face the additional complication of characterizing and learning the unknown temporal kernel in a data-adaptive fashion.
Unlike the existing kernel learning methods where at least the parametric form of the kernel is given (Wilson et al., 2016), we have little context on the temporal kernel, and aggressively assuming the parametric form will risk altering the temporal structures implicitly just like discretization.
Instead, we leverage the Bochner’s theorem and its extension (Bochner, 1948; Yaglom, 1987) to ﬁrst covert the kernel learning problem to the more reasonable spectral domain where we can direct characterize the spectral properties with random (Fourier) features.
Representing the temporal kernel by random features is favorable as we show they preserve the existing Gaussian process and NTK properties of neural networks.
This is desired from the deep learning’s perspective since our approach will not violate the current understandings of deep learning.
Then we apply the reparametrization trick (Kingma & Welling, 2013), which is a standard tool for generative models and Bayesian deep learning, to jointly optimize the spectral density estimator.
Furthermore, we provide theoretical guarantees for the random-feature-based kernel learning approach when the temporal kernel is non-stationary, and the spectral density estimator is misspeciﬁed.
These two scenarios are essential for practical usage but have not been studied in the previous literature.
Finally, we conduct simulations and experiments on real-world continuous-time sequence data to show the effectiveness of the temporal kernel approach, which signiﬁcantly improves the performance of both standard neural architectures and complicated domain-speciﬁc models.
We summarize our contributions as follow
• We study a novel connection between the continuous-time system and neural network via the composition with a temporal kernel.
• We propose an efﬁcient kernel learning method based on random feature representation, spectral density estimation and reparameterization, and provide strong theoretical guarantees when the kernel is nonstationary and the spectral density is misspeﬁcied.
• We analyze the empirical performance of our temporal kernel approach for both the standard and domain-speciﬁc deep learning models through real-data simulation and experiments.
N OTATIONS AND B ACKGROUND
We use bold-font letters to denote vectors and matrices.
We use x t and ( x , t ) interchangeably to denote a time-sensitive event occurred at time t , with t ∈ T ≡ [0 , t max ] .
Neural networks are denoted by f ( θ , x ) , where x ∈ X ⊂ R d is the input where diameter ( X ) ≤ l , and the network parameters θ are sampled i.i.d from the standard normal distribution at initialization.
Without loss of generality, we study the standard L-layer feedforward neural network with its output at the h th hidden layer given by f ( h ) ∈ R d h .
We use (cid:15) and (cid:15) ( t ) to denote Gaussian noise and continuous-time Gaussian noise process.
By convention, we use ⊗ and ◦ and to represent the tensor and outer product.
2. U NDERSTANDING THE STANDARD NEURAL NETWORK
We follow the settings from Jacot et al.
; Yang (2019) to brieﬂy illustrate the limiting Gaussian behavior of f ( θ , x ) at initialization, and its training trajectory under weak optimization.
As d 1 , . . .
, d L → ∞ , f ( h ) tend in law to i.i.d Gaussian processes with covariance Σ h ∈ R d h × d h : f ( h ) ∼ N (0 , Σ h ) , which we refer to as the neural network kernel to distinguish from the other kernel notions.
Also, given a training dataset { x i , y i } ni =1 , let f (cid:0) θ ( s ) (cid:1) =
(cid:0) f
( θ ( s ) , x 1 ) , . . .
, f ( θ ( s ) , x n ) (cid:1) be the network outputs at the s th training step and y
=
( y 1 , . . .
, y n ) .
Using the squared loss for example, when training with inﬁnitesimal learning rate, the outputs follows: d f (cid:0) θ ( s ) (cid:1) /
d s = − Θ ( s ) × ( f (cid:0) θ ( s ) (cid:1) − y ) , where Θ ( s ) is the neural tangent kernel (NTK).
The detailed formulation of Σ h and Θ ( s ) are provided in Appendix A.2.
We introduce the two concepts here because: 1. instead of incorporating time to f ( θ , x ) , which is then subject to its speciﬁc structures, can we alternatively consider an universal approach which expands the Σ h to the temporal domain such as by composing
it
with a
time -aware kernel? When jointly optimize the unknown temporal kernel and the model parameters, how can we preserve the results on the training trajectory with NTK?
In our paper, we show that both goals are achieved by representing a temporal kernel via random features.
2. D IFFERENCE BETWEEN CONTINUOUS - TIME AND ITS DISCRETIZATION
We now discuss the gap between continuous-time process and its equally-spaced discretization.
We study the simple univariate continuous-time system f ( t ) :
[Equation]
A discretization with a ﬁxed interval is then given by: f
[ i ] =
f ( i × interval ) for i
= 1 , 2 , . .
Notice that f ( t ) is a second-order auto-regressive process, so both f ( t ) and f
[ i ] are stationary.
Recall that the covariance function for a stationary process is given by k ( t ) :
= cov (cid:0) f
( t 0 ) , f ( t 0 + t ) (cid:1) , and the spectral density function (SDF) is deﬁned as s ( ω ) =
(cid:82) ∞
−∞ exp( − iωt ) k
( t ) dt .
Claim 1.
The spectral density function for f ( t ) and f [ i ] are different.
The proof is relegated to Appendix A.2.2.
The key takeaway from the example is that the spectral density function, which characterizes the signal on the frequency domain, is altered implicitly even by regular discretization in this simple case.
Hence, we should be cautious about the potential impact of the modelling assumption, which eventually motivates us to explicitly model the spectral distribution.
M ETHODOLOGY
We ﬁrst explain our intuition using the above example.
If we take the Fourier transform on (1) and rearrange terms, it becomes: ˜ f
( iω ) =
(cid:16) b( iω {+) 
+}
+ a 0 ( iω ) + a(cid:17) ˜
(cid:15) ( iω ) , where ˜ f ( iω ) and ˜ (cid:15) ( iω ) are the Fourier transform of f ( t ) and (cid:15) ( t ) .
Note that the spectral density of a Gaussian noise process is constant, i.e. | ˜ (cid:15) ( iω ) | 2 = p 0 , so the spectral density of f ( t ) is given by: s θ T ( ω )
= p 0 (cid:12)(cid:12) b 0 / (( iω ) 2 + a 0 ( iω ) + a 1 ) (cid:12)(cid:12) 2 , where we use θ T =
[ a 0 , a 1 , b 0 ] to denote the parameters of the linear dynamic system deﬁned in (1).
The subscript T is added to distinguish from the parameters of the neural network.
The classical Wiener-Khinchin theorem (Wiener et al., 1930) states that the covariance function for f ( t ) , which is a Gaussian process since the linear differential equation is a linear operation on (cid:15) ( t ) , is given by the inverse Fourier transform of the spectral density:
[Equation]
We defer the discussions on the inverse direction, that given a kernel k θ T ( t (cid:48) − t ) we can also construct a continuous-time system, to Appendix A.3.1.
Consequently, there is a correspondence between the parameterization of a stochastic ODE and the kernel of a Gaussian process.
The mapping is not necessarily one-to-one; however, it may lead to a more convenient way to parameterize a continuoustime process alternatively using deep learning models, especially knowing the connections between neural network and Gaussian process (which we highlighted in Section 2.1).
To connect the neural network kernel Σ ( h ) (e.g. for the h th layer of the FFN) to a continuous-time system, the critical step is to understand the interplay between the neural network kernel and the temporal kernel (e.g. the kernel in (2)):
• the neural network kernel characterizes the covariance structures among the hidden representation of data (transformed by the neural network) at any ﬁxed time point; • the temporal kernel, which corresponds to some continuous-time system, tells how each static neural network kernel propagates forward in time.
See Figure 1a for a visual illustration.
Continuing with Example 1, it is straightforward construct the integrated continuous-time system as:
[Equation]
where we use the neural network kernel Σ ( h ) to deﬁne the Gaussian process (cid:15) ( x , t ) on the feature dimension, so the ODE parameters are now functions of the data as well.
To see that (3) generalizes the h th layer of a FFN to the temporal domain, we ﬁrst consider a 2 ( x ) = a 1 ( x ) = 0 and a 0 ( x ) =
b 0 ( x ) .
Then the continuous-time process f ( x , t ) exactly follows f ( h ) at any ﬁxed time point t , and its trajectory on the time axis is simply a Gaussian process.
When a 1 ( x ) , a 2 ( x ) (cid:54) = 0 , f ( x , t ) still matches f ( h ) at the initial point, but its propagation on the time axis becomes nontrivial and is now characterized by the constructed continuous-time system.
We can easily the setting to incorporate higher-order terms:
[Equation]
Keeping the heuristics in mind, an immediate question is what is the structure of the corresponding kernel function after we combine the continuous-time system with the neural network kernel?
Claim 2.
The kernel function for f ( x , t ) in (4) is given by: Σ ( h ) T ( x , t ; x (cid:48) , t (cid:48) )
=
k θ T ( x , t ; x (cid:48) , t (cid:48) ) · Σ ( h ) ( x , x (cid:48) ) , where θ T is the underlying parameterization of (cid:8) a i ( · ) (cid:9) n i =1 and (cid:8) b i ( · ) (cid:9) m i =1 as functions of x .
When (cid:8) a i (cid:9) n i =1 and (cid:8) b i (cid:9) m i =1 are scalars, Σ ( h ) T ( x , t ; x (cid:48) , t (cid:48) )
=
k θ T ( t, t (cid:48) ) · Σ ( h ) ( x , x (cid:48) ) .
We defer the proof and the discussion on the inverse direction from temporal kernel to continuoustime system to Appendix A.3.1.
Claim 2 shows that it is possible to expand any layer of a standard neural network to the temporal domain, as part of a continuous-time system using kernel composition.
The composition is ﬂexible and can happen at any hidden layer.
In particular, given the temporal kernel K T and neural network kernel Σ ( h ) , we obtain the neural-temporal kernel on X × T : Σ ( h ) T = diag (cid:0) Σ ( h ) ⊗ K T (cid:1) , where diag ( · ) is the partial diagonalization operation on X :
[Equation]
The above argument shows that instead of taking care of both the deep learning and continuous-time system, which remains challenging for general architectures, we can convert the problem to ﬁnding a suitable temporal kernel.
We further point out that when using neural networks, we are parameterizing the hidden representation (feature lift) in the feature space rather than the kernel function in the kernel space.
Therefore, to give a consistent characterization, we should also study the feature representation of the temporal kernel and then combine it with the hidden representations of the neural network.
3. T HE RANDOM FEATURE REPRESENTATION FOR TEMPORAL KERNEL
We start by considering the simpler case where the temporal kernel is stationary and independent of features: K T ( t, t (cid:48) )
=
k ( t (cid:48) − t ) , for some properly scaled positive even function k ( · ) .
The classical
Bochner’s theorem (Bochner, 1948) states that:
[Equation]
for some probability density function s on R
[Equation]
(6) where s ( · ) is the spectral density function we highlighted in Section 2.2.
To compute the integral, we may sample ( ω 1 , . . . , ω m ) from s ( ω ) and use the Monte Carlo method: ψ ( t (cid:48) − t ) ≈m
[Equation]
Since e − i
[Equation]
, for the real part, we let:
[Equation]
(7) and it is easy to check
that ψ ( t (cid:48) − t ) ≈ (cid:104) φ ( t ) , φ ( t (cid:48) ) (cid:105) .
Since φ ( t ) is constructed from random samples, we refer to it as the random feature representation of K T .
Random feature has been extensive studied in the kernel machine literature, however, we propose a novel application for random features to parametrize an unknown kernel function.
A straightforward idea is to parametrize the spectral density function s ( ω ) , whose pivotal role has been highlighted in Section 2.2 and Example 1.
Suppose θ T is the distribution parameters for s ( ω ) . Then φ θ T ( t ) is also (implicitly) parameterized by θ T through the samples (cid:8) ω ( θ T ) i (cid:9) m i
=1 from s ( ω ) .
The idea resembles the reparameterization trick for training variational objectives (Kingma & Welling, 2013), which we formalize in the next section.
For now, it remains unknown if we can also obtain the random feature representation for nonstationary kernels where Bochner’s theorem is not applicable.
Note that for a general temporal kernel K T ( x , t ; x (cid:48) , t (cid:48) ) , in practice, it is not reasonable to assume stationarity specially for the feature domain.
In Proposition 1, we provide a rigorous result that generalizes the random feature representation for nonstationary kernels with convergence guarantee.
Proposition 1.
For any (scaled) continuous non-stationary PDS kernel K T on X × T , there exists a joint probability measure with spectral density function s ( ω 1 , ω 2 ) , such that K T (cid:0) ( x , t ) , ( x (cid:48) , t (cid:48) ) (cid:1) = E s [Equation]
ω 1 , [Equation] [Equation] φ ( x , t )
[Equation]
φ ( x (cid:48) , t (cid:48) ) (cid:3) where φ ( x , t ) is given by:
[Equation]
ω
[Equation]
[Equation] Here, (cid:8) ( ω 1 ,i , ω 2 ,i ) (cid:9) m i =1 are the m samples from s ( ω 1 , ω 2 ) . When the sample size m ≥
[Equation]
, with probability at least 1 − δ , for any ε >
[Equation]
where σ 2 p is the second moment of the spectral density function s ( ω 1 , ω 2 ) and C ( d ) is a constant.
We defer the proof to Appendix A.3.2.
It is obvious that the new random feature representation in (8) is a generalization of the stationary setting.
There are two advantages for using the random feature representation:
• the composition in the kernel space suggested by (5) is equivalent to the computationally efﬁcient operation f
( h ) ( x ) ◦ φ ( x , t ) in the feature space (Shawe-Taylor et al., 2004); • we preserve a similar Gaussian process behavior and the neural tangent kernel results that we discussed in Section 2.1, and we defer the discussion and proof to Appendix A.3.3.
In the forward-passing computations, we simply replace the original hidden representation f ( h ) ( x ) by the time-aware representation f ( h ) ( x ) ◦ φ ( x , t ) .
Also, the existing methods and results on analyzing neural networks though Gaussian process and NTK, though not emphasized in this paper, can be directly carried out to the temporal setting as well (see Appendix A.3.3).
3. R EPARAMETERIZATION WITH THE SPECTRAL DENSITY FUNCTION
We now present the gradient computation for the parameters of the spectral distribution using only their samples.
We start from the well-studied case where s ( ω ) is given by a normal distribution
N ( µ, Σ ) with parameters θ T = [ µ, Σ ] .
When computing the gradients of θ T , instead of sampling from the intractable distribution s ( ω ) , we reparameterize each sample ω i via: Σ 1 / 2 (cid:15) + µ , where (cid:15) is sampled from a standard multivariate normal distribution.
The gradient computations that relied on ω is now replaced using the easy-to-sample (cid:15) and θ T = [ µ, Σ ] now become tractable parameters in the model given (cid:15) .
We illustrate reparameterization in our setting in the following example.
Example 1.
Consider a single-dimension homogeneous linear model: f ( θ, x ) =
f (0) ( x ) = θx .
Without loss of generality, we use only a single sample ω 1 from the s ( ω ) which corresponds to the feature-independent temporal kernel k θ ( t, t (cid:48) ) .
Again, we assume s ( ω ) ∼ N ( µ, σ ) .
Then the time-aware hidden representation for this layer for datapoint ( x 1 , t 1 ) is given by:
[Equation]
Using the reparameterization, given a sample (cid:15) 1 from the standard normal distribution, we have:
[Equation]
so the gradients with respect to all the parameters ( θ, µ, σ ) can be computed in the usual way.
Despite the computation advantage, the spectral density is now learnt from the data instead of being given so the convergence result in Proposition 1 does not provide sample-consistency guarantee.
In practice, we may also misspecify the spectral distribution and bring extra intractable factors.
To provide practical guarantees, we ﬁrst introduce several notations: let K T ( S ) be the temporal kernel represented by random features such that K T ( S )
=
E (cid:2) φ (cid:124) φ (cid:3) , where the expectation is taken with respect to the data distribution and the random feature vector φ has its samples { ω i } mi =1 drawn from the spectral distribution S .
Without abuse of notation, we use φ ∼ S to denote the dependency of the random feature vector φ on the spectral distribution S provided in (8).
Given a neural network kernel Σ ( h ) , the neural temporal kernel is then denoted by: Σ ( h ) T ( S ) =
Σ ( h ) ⊗ K T ( S ) .
So the sample version of Σ ( h ) T ( S ) for the dataset (cid:8) ( x i , t i ) (cid:9) n i =1 is given by:
[Equation]
If the spectral distribution S is ﬁxed and given, then using standard techniques and Theorem 1 it is straightforward to show lim n →∞ ˆ Σ ( h ) T ( S )
→ E (cid:2) ˆ Σ ( h ) T ( S ) (cid:3) so the proposed learning schema is sample-consistent.
In our case, the spectral distribution is learnt from the data, so we need some restrictions on the spectral distribution in order to obtain any consistency guarantee.
The intuition is that if S θ T does not diverge from the true S , e.g. d ( S θ T (cid:107) S ) ≤ δ for some divergence measure, the guarantee on S can transfer to S θ T with the rate only suffers a discount that does not depend on n .
Theorem 1.
Consider the f-divergence such that d ( S θ T (cid:107) S )
= (cid:82) c (cid:0)
d S θ T / d S (cid:1) d S , with the generator function c ( x ) =
x k − 1 for any k > 0 .
Given the neural network kernel Σ h , let M = (cid:13)(cid:13) Σ h (cid:13)(cid:13) ∞ , then
Pr (cid:16)
[Equation]
− nε [Equation] 64 max {
[Equation]
M }
[Equation]
1) (cid:17)
[Equation]
C
[Equation]
, (12) where C ( ε ) ∝ (cid:16) 2 l 2 t 2max σ S θ T (cid:15)/ max { 4 ,M } (cid:17)
[Equation]
exp (cid:16) −
[Equation]
that does not depend on δ .
The proof is provided in Appendix A.3.4.
The key takeaway from (12) is that as long as the divergence between the learnt S θ T and the true spectral distribution is bounded, we still achieve sample consistency.
Therefore, instead of specifying a distribution family which is more likely to suffer from misspeciﬁcation, we are motivated to employ some universal distribution approximator such as the invertible neural network (INN) (Ardizzone et al., 2018).
INN consists of a series of invertible operations that transform samples from a known auxiliary distribution (such as normal distribution) to arbitrarily complex distributions.
The Jacobian that characterize the changes of distributions are made invertible by the INN, so the gradient ﬂow is computationally tractable similar to the case in Example 1.
We defer the detailed discussions to Appendix A.3.5.
Remark 1.
It is clear at this point that the temporal kernel approach applies to all the neural networks who have a Gaussian process behavior with a valid neural network kernel, which includes the major architectures such as CNN, RNN and attention mechanism (Yang, 2019).
For implementation, at each forward and backward computation, we ﬁrst sample from the auxiliary distribution to construct the random feature representation φ using reparameterization, and then compose it with the selected hidden layer f ( h ) such as in (10).
We illustrate the computation architecture in Figure 2, where we adapt the vanilla RNN to the proposed framework.
In Algorithm 1, we provide the detailed forward and backward computations, using the L-layer FFN from the previous sections as an example.
R ELATED WORK
The earliest work that discuss training continuous-time neural network dates back to LeCun et al.
(1988); Pearlmutter (1995), but no feasible solution was proposed at that time.
The proposed approach relates to several ﬁelds that are under active research.
ODE and neural network.
Certain neural architecture such as the residual network has been interpreted as approximate ODE solvers (Lu et al., 2018).
More direct approaches have been proposed to learn differential equations from data (Raissi & Karniadakis, 2018; Raissi et al., 2018a;
Long et al., 2018), and signiﬁcant efforts have been spent on developing solvers that combine ODE and the back-propagation framework (Farrell et al., 2013; Carpenter et al., 2015; Chen et al., 2018).
The closest literature to our work is from Raissi et al.
(2018b) who design numerical Gaussian process resulting from temporal discretization of time-dependent partial differential equations.
Random feature and kernel machine learning.
In supervised learning, the kernel trick provides a powerful tool to characterize non-linear data representations (Shawe-Taylor et al., 2004), but the computation complexity is overwhelming for large dataset.
The random (Fourier) feature approach proposed by Rahimi & Recht (2008) provides substantial computation beneﬁts.
The existing literature on analyzing the random feature approach all assume the kernel function is ﬁxed and stationary
(Yang et al., 2012; Sutherland & Schneider, 2015; Sriperumbudur & Szabó, 2015; Avron et al., 2017).
Reparameterization and INN.
Computing the gradient for intractable objectives using samples from auxiliary distribution dates back to the policy gradient method in reinforcement learning (Sutton et al., 2000).
In recent years, the approach gains popularity for training generative models (Kingma &
Welling, 2013), other variational objectives (Blei et al., 2017) and Bayesian neural networks (Snoek et al., 2015).
INN are often employed to parameterize the normalizing ﬂow that transforms a simple distribution into a complex one by applying a sequence of invertible transformation functions (Dinh et al., 2014;
Ardizzone et al., 2018; Kingma & Dhariwal, 2018;
Dinh et al., 2016).
Our approach characterizes the continuous-time ODE via the lens of kernel.
It complements the existing neural ODE methods which are often restricted to speciﬁc architectures, relying on ODE solvers and lacking theoretical understandings.
We also propose a novel deep kernel learning approach by parameterizing the spectral distribution under random feature representation, which is conceptually different from using temporal kernel for time-series classiﬁcation (Li & Marlin, 2015). Our work is a extension of Xu et al. (2019; 2020), which study the case for self-attention mechanism. for each mini-batch do
Sample (cid:8) 1 ,j , 2 ,j m j =1 from the auxilliary distribution P ; Compute the reparameterized samples using the INN g ψ e.g ω
,j ψ
:= g ψ
,j ) ; for sample i in the batch do
Construct
; end
Update the end
It is straightforward from Figure 2 and Algorithm 1 that the proposed approach serves as a plug-in module and do not modify the original network structures of the RNN and FFN.
E XPERIMENTS AND R ESULTS
We focus on revealing the two major advantages of the proposed temporal kernel approach:
• the temporal kernel approach consistently improves the performance of deep learning models, both for the general architectures such as RNN, CausalCNN and attention mechanism as well as the domain-speciﬁc architectures, in the presence of continuous-time information; • the improvement is not at the cost of computation efﬁciency and stability, and we outperform the alternative approaches who also applies to general deep learning models.
We point out that the neural point process and the ODE neural networks have only been shown to work for certain model architectures so we are unable to compare with them for all the settings.
Time series prediction with standard neural networks (real-data and simulation)
We conduct time series prediction task using the vanilla RNN, CausalCNN and self-attention mechanism with our temporal kernel approach (Figure A.1).
We choose the classical Jena weather data for temperature prediction, and the Wikipedia trafﬁc data to predict the number of visits of Wikipedia pages.
Both datasets have vectorized features and are regular-sampled.
To illustrate the advantage of leveraging the temporal information compared with using only sequential information, we ﬁrst conduct the ordinary next-step prediction on the regular observations, which we refer to as Case1 .
To fully illustrate our capability of handling the irregular continuous-time information, we consider the two simulation setting that generate irregular continuous-time sequences for prediction:
Case2 .
we sample irregularly from the history,
i
e. x t 1 , . .
, x t q , q ≤ k , to predict x t k +1 ; Case3 .
we use the full history to predict a dynamic future point, i
e. x t k + q for a random q .
We provide the complete data description, preprocessing, and implementation in Appendix B.
We use the following two widely-adopted time-aware modiﬁcations for neural networks (denote by NN ) as baselines, as well as the classical vectorized autoregression model ( VAR ).
NN+time : we directly concatenate the timespan, e.g. t j − t i , to the feature vector.
NN+trigo : we concatenate the learnable sine and cosine features, e.g. [sin( π 1 t ) , . .
, sin( π k t )] , to the feature vector, where { π i } ki =1 are free model parameters.
We denote our temporal kernel approach by T-NN .
From Figure 3, we see that the temporal kernel outperforms the baselines in all cases when the time series is irregularly sampled ( Case2 and Case3 ), suggesting the effectiveness of the temporal kernel approach in capturing and utilizing the continuous-time signals.
Even for the regular Case reported in Table A.1, the temporal kernel approach gives the best results, which again emphasizes the advantage of directly characterize the temporal information over discretization.
We also show in the ablation studies (Appendix B.5) that INN is necessary for achieving superior performance compared with specifying a distribution family.
To demonstrate the stability and robustness, we provide sensitivity analysis in Appendix B.6 for model selection and INN structures.
Temporal sequence learning with complex domain models
Now we study the performance of our temporal kernel approach for the sequential recommendation task with more complicated domain-speciﬁc two-tower architectures (Appendix B.2).
Temporal information is known to be critical for understanding customer intentions, so we choose the two public e-commerce dataset from Alibaba and Walmart.com , and examine the next-purchase recommendation.
To illustrate our ﬂexibility, we select the GRU-based, CNN-based and attention-based recommendation models from the recommender system domain (Hidasi et al., 2015; Li et al., 2017a) and equip them with the temporal kernel.
The detailed settings, ablation studies and sensitivity analysis are all in
Appendix B.
The results are shown in Table A.2.
We observe that the temporal kernel approach brings various degrees of improvements to the recommendation models by characterizing the continuous-time information.
The positives results from the recommendation task also suggests the potential of our approach for making impact in boarder domains.
D ISCUSSION
In this paper, we discuss the insufﬁciency of existing work on characterizing continuous-time data with deep learning models and describe a principled temporal kernel approach that expands neural networks to characterize continuous-time data.
The proposed learning approach has strong theoretical guarantees, and can be easily adapted to a broad range of applications such as deep spatial-temporal modelling, outlier and burst detection, and generative modelling for time series data.
Scope and limitation .
Although the temporal kernel approach is motivated by the limiting-width
Gaussian behavior of neural networks, in practice, it sufﬁces to use regular widths as we did in our experiments (see Appendix B.2 for the conﬁgurations).
Therefore, there are still gaps between our theoretical understandings and the observed empirical performance, which require more dedicated analysis.
One possible direction is to apply the techniques in Daniely et al.
(2016) to characterize the dual kernel view of ﬁnite-width neural networks.
The technical detail, however, will be more involved.
It is also arguably true that we build the connection between the temporal kernel view and continuoustime system in an indirect fashion, compared with the ODE neural networks.
However, our approach is fully compatible with the deep learning subroutines while the end-to-end ODE neural networks require substantial modiﬁcations to the modelling and implementation.
Nevertheless, ODE neural networks are (in theory) capable of modelling more complex systems where the continuous-time setting is a special case.
Our work, on the other hand, is dedicated to the temporal setting.
A A PPENDIX
We provide the omitted proofs, detailed discussions, extensions and complete numerical results.
A. N UMERICAL RESULTS FOR S ECTION
[Table]
[Table]
A. S UPPLEMENTARY MATERIAL FOR S ECTION
We discuss the detailed background for the Gaussian process behavior of neural network and the training trajectory under neural tangent kernel, as well as the proof for Claim 1.
A.2. G AUSSIAN PROCESS BEHAVIOR AND NEURAL TANGENT KERNEL FOR DEEP LEARNING MODELS
The Gaussian process (GP) view of neural networks at random initialization was originally discussed in (Neal, 2012).
Recently, CNN and other standard neural architectures have all been recognized as functions drawn from GP in the limit of inﬁnite network width (Novak et al., 2018;
Yang, 2019).
When trained by gradient descent under inﬁnitesimal step schedule, the gradient ﬂow of the standard neural architectures can be described by the notion of Neural Tangent Kernel (NTK) whose asymptotic behavior under inﬁnite network width is known (Jacot et al., 2018).
The discovery of NTK has led to several papers studying the training and generalization properties of neural networks (Allen-Zhu et al., 2019; Arora et al., 2019a;b).
For a L-layer FNN f ( θ , x ) = f ( L ) with hidden dimensions { d h } Lh =1 and recursively deﬁned via:
[Equation]
for h = 1 , 2 , . . .
, L − 1 , where σ ( . ) is the activation function and the layer weights W ( L )
∈ R d L − 1 ,
W ( h ) ∈ R d
h − 1 × d h and intercepts are initialized by sampling independently from N (0 , 1) (without loss of generality).
As d 1 , . . .
, d L → ∞ , f ( h ) tend in law to i.i.d Gaussian processes with covariance Σ h deﬁned recursively as shown by Neal [Equation]
[Equation]
We also refer to Σ ( h ) as the neural network kernel to distinguish from the other kernel notions.
Given a training dataset { x i , y i } ni =1 , let f (cid:0) θ ( s ) (cid:1) =
(cid:0) f
( θ ( s ) , x 1 ) , . . .
, f ( θ ( s ) , x n ) (cid:1) be the network outputs at the s th training step and y
=
( y 1 , . . .
, y n ) .
When training the network by minimizing the squared loss (cid:96) ( θ ) with inﬁnitesimal learning rate, i.e. d θ ( s ) ds = −∇ (cid:96) ( θ ( s )) , the network outputs at training step s follows the evolution (Jacot et al., 2018):
[Equation]
The above Θ ( s ) is referred to as the NTK, and recent results shows that when the network widths go to inﬁnity (or sufﬁciently large), Θ ( s ) converges to a ﬁxed Θ 0 almost surely (or with high probability).
For a standard L-layer FFN, the NTK Θ 0 =
Θ ( L )for parameters { W ( h ) , b ( h ) } on the
h th -layer can also be computed recursively:
[Equation]
and Θ ( k
[Equation]
x i ,
[Equation]
k
[Equation]
A number of optimization and generalization properties of neural networks can be studied using
NTK, which we refer the interested readers to (Lee et al., 2019; Allen-Zhu et al., 2019; Arora et al., 2019a;b).
We also point out that the above GP and NTK constructions can be carried out on all standard neural architectures including CNN, RNN and the attention mechanism (Yang, 2019).
A.2. P ROOF FOR C LAIM
In this part, we denote the continuous-time system by X ( t ) in order to introduce the full set notations that are needed for our proof, where the length of the discretized interval is explicitly considered.
Note that we especially construct the example in Section 2 so that the derivations are not too cumbersome.
However, the techniques that we use here can be extended to prove the more complicated settings.
Proof.
Consider X ( t ) to be the second-order continuous-time autoregressive process with covariance function k ( t ) and spectral density function (SDF) s ( ω ) such that s ( ω ) =
(cid:82) ∞
−∞ exp( − iωt ) k
( t ) dt .
The covariance function of the discretization X a [ n ] = X ( na ) with any ﬁxed interval a > 0 is then given by k a [ n ] =
k ( na ) .
According to standard results in time series, the SDF of X ( t ) is given by in the form of:
[Equation]
We assume without loss of generality that b 1 , b 2 are positive numbers.
Note that the kernel function for X a [ i ] can also be given by
[Equation]
which suggests that the SDF for the discretization X a [ n ] can be given by:
[Equation]
where d 2 = b 2 e ab 2 ( e 2 ab 2 − 1) − b 1 e ab 1 ( e 2 ab 2 − 1) .
By the deﬁnition of discrete-time auto-regressive process, X a [ n ] is a second-order AR process only if d 2 = 0 , which happens if and only if: b 2 /b 1 = ( e ab 2 − e − ab 2 ) /
( e ab 1 − e − ab 1 ) .
However, the function g ( x ) = exp( ax ) − exp( − ax ) is concave on [0 , ∞ ) (since the time interval a > 0 ) and g (0)
= 0 , the-above equality hold if b 1 = b 2 .
However, this contradicts with (A.5), since a 1 + a 2 = 0 and a 1 b 22 + a 2 b 21 (cid:54) = 0 suggests a 1 ( b 1 − b 2 ) 2 (cid:54)
= 0 .
Hence, X a [ n ] cannot be a second-order discrete-time auto-regressive process.
A. S UPPLEMENTARY MATERIAL FOR S ECTION
We ﬁrst present the related discussions and proof for Claim 2 on the connection between continuoustime system and temporal kernel.
Then we prove the convergence result in Theorem 1 regarding the random feature representation for non-stationary kernel.
In the sequel, we show the new results for the Gaussian process behavior and neural tangent kernel under random feature representations, and discuss the potential usage of our results.
Finally, we prove the sample-consistency result when the spectral distribution is misspeciﬁed.
A.3. P ROOF AND DISCUSSIONS FOR C LAIM
Proof.
Recall that we study the dynamic system given by:
[Equation]
) where (cid:15) ( x , t = t 0 ) ∼ N (0 , Σ ( h ) , ∀ t 0 ∈ T .
The solution process to the above continuous-time system is also a Gaussian process, since (cid:15) ( x , t ) is a Gaussian process and the solution of a linear different equation is a linear operation on the input.
For the sake of notation, we assume b 0 ( x ) =
and b 1 ( x ) = 0 , . . .
, b m ( x ) = 0 , which does not change the arguments in the proof.
We apply the
Fourier transformation transformation on both sides and solve the for Fourier transform ˜ f
( i ω x , iω ) :
[Equation]
where W ( iω ; ω x ) is the Fourier transform of (cid:15) ( x , t ) .
If we do not make the assumption on { b j ( x ) } mj =1 , they will simply show up on the numeration in the same fashion as { a j ( x ) } nj =1 .
Let
G θ T [Equation] iω [Equation]
= [Equation] and p ( ω x ) = | W ( iω ; ω x ) | 2 be the spectral density of the Gaussian process corresponding to (cid:15) (its spectral density does not depend on ω because (cid:15) is a Gaussian white noise process on the time dimension).
The dependency of G ( · ; · ) on θ T is because we deﬁned θ T to the underlying parameterization of { a j ( · ) } nj =1 in the statement of Claim 2.
Then the spectral density of the process f ( x , t ) is given by
[Equation]
where C is constant that corresponds to the spectral density of the random Gaussian noise on the time dimension.
Notice that the spectral density function obtained this way is regular, since it has the form of p θ T ( ω ; x ) = constant / ( polynomial of ω 2 ) .
Therefore, according to the classical Wiener-Khinchin theorem Brockwell et al.
(1991), the covariance function of the solution process is given by the inverse Fourier transform of the spectral density:
[Equation]
And therefore we reach the conclusion in Claim 2 by taking Σ ( h ) T ( x , t ; x (cid:48) , t (cid:48) )
= ψ ( x − x (cid:48) , t − t (cid:48) ) .
The inverse statement of Claim 2 may not be always true, since not all the neural-temporal kernel can ﬁnd a exact corresponding continuous-time system in the form of (A.8).
However, we may construct the continuous-time system that approximates the kernel (arbitrarily well) in the following way, using the polynomial approximation tools such as the Taylor expansion.
For a neural-temporal kernel Σ ( h ) T , we ﬁrst compute it Fourier transform to obtain the spectral density p ( ω x , ω ) .
Note that p ( ω x , ω ) should be a rational function in the form of ( polynomial in ω 2 ) / ( polynomial in ω 2 ) , or otherwise it does not have stable spectral factorization that leads to a linear dynamic system.
To achieve the goal, we can always apply Taylor expansion or
Pade approximants that recovers the p ( ω x , ω ) arbitrarily well.
Then we conduct a spectral factorization on p ( ω x , ω ) to ﬁnd G ( i ω x , iω t ) and p ( ω x ) such that p [Equation] ω [Equation]
i
[Equation] x , iω [Equation] p
[Equation]
ω [Equation] G ( − i ω x , − iω t ) . Since p ( ω x , ω ) is now in a rational function form of ω 2 , we can ﬁnd G ( i ω x , iω t ) as:
[Equation]
Let α j ( x ) and β j ( x ) be the pseudo-differential operators of a j ( i ω x ) and b j ( i ω x ) deﬁned in terms of their inverse Fourier transforms (Shubin, 1987), then the corresponding continuous-time system is given by:
[Equation]
For a concrete end-to-end example, we consider the simpliﬁed setting where the temporal kernel function is given by:
[Equation]
where B θ 1 ( · ) is the Bessel function so K θ T ( t, t (cid:48) ) belongs to the well-known Matern family.
It is straightforward to show that the spectral density function is given by:
[Equation]
As a consequence, we see that s ( ω ) ∝ (cid:16)(cid:112) 2 θ 1 θ
[Equation]
, so we directly have G θ T ( ω ) =
[Equation]
θθ+ iω (cid:17) − ( θ 1 +1 / 2) instead of having to seek for polynomial approximation.
Now we can easily expand G θ T ( ω ) using the binomial formula to ﬁnd the linear parameters for the continuous-time system.
For instance, when θ 1 = 3 / 2 , we have:
[Equation]
A.3. P ROOF FOR P ROPOSITION
Proof.
We ﬁrst need to show that the random Fourier features for the non-stationary kernel
K T (cid:0) ( x , t ) , ( x (cid:48) , t (cid:48) ) (cid:1) can be given by (11), i.e.
[Equation]
To simplify notations, we let z :=
[ x , t ] ∈ R d +1 and Z
= X × T .
For non-stationary kernels, their corresponding Fourier transform can be characterized by the following lemma.
Assume without loss of generality that K T is differentiable.
Lemma A.1 (Yaglom (1987)) .
A non-stationary kernel k ( z 1 , z 2 ) is positive deﬁnite in R d if and only if after scaling, it has the form:
[Equation]
where µ ( d ω 1 , d ω 2 ) is some positive-semideﬁnite probability measure with bounded variation.
The above lemma can be think of as the extension of the classical Bochner’s theorem underlies the random Fourier feature for stationary kernels.
Notice that when covariance function for the measure µ only has non-zero diagonal elements and ω 1 = ω 2 , then we recover the spectral representation stated in the Bochner’s theorem.
Therefore, we can also approximate (A.12) with the Monte Carlo integral.
However, we need to ensure the positive-semideﬁniteness of the spectral density for µ ( d ω 1 , d ω 2 ) , which we denote by p ( ω 1 , ω 2 ) .
It has been suggested in Remes et al.
(2017) that we consider another density function q ( ω 1 , ω 2 ) and let p be taken on the product space of q and then symmetrise:
[Equation]
(A.13)
Then (A.12) suggests that
[Equation]
Recall that the real part of exp (cid:0) i ( ω (cid:124) 1 z 1 − ω (cid:124) 2 z 2 ) (cid:1) is given by cos( ω (cid:124) 1 z 1 − ω
(cid:124) 2 z 2 ) .
So with the
Trigonometric equalities, it is straightforward to verify that k ( z 1 , z 2 ) =
E q (cid:2) φ ( z ) (cid:124) φ ( z ) (cid:3) .
Hence, the random Fourier features for non-stationary kernel can be given in the form of (11).
Then we show the uniform convergence result as the number of samples goes to inﬁnity when computing E q (cid:2) φ ( z ) (cid:124) φ ( z ) (cid:3) by the Monte Carlo integral.
Let ˜ Z
= Z × Z , so
˜ Z = { ( x , t, x (cid:48) , t, ) (cid:13)(cid:13) x , x (cid:48) ∈
X ; t, t (cid:48) ∈ T } .
Since diam ( X ) = l and T =
[0 , t max ] , we have diam ( ˜ Z ) =
l 2 t 2max .
Let the approximation error be
[Equation]
The strategy is to use a (cid:15) -net covering for the input space ˜ Z , which would require N = (cid:0) 2 l 2 t 2max /r (cid:1)
d +1 balls of radius r .
Let C
= { c i } Ni =1 be the centers for each (cid:15) -ball.
We ﬁrst show the bound for | ∆( c i ) | and the Lipschitz constant L ∆ of the error function ∆ , and then combine them to get the desired result.
Since ∆ is continuous and differentiable w.r.t z , z (cid:48) according to the deﬁnition of φ , we have L ∆ =
(cid:13)(cid:13) ∇ ∆( c ∗ ) (cid:13)(cid:13) , where c ∗ = arg max c ∈C (cid:13)(cid:13) ∇ ∆( c ) (cid:13)(cid:13) .
Let c ∗ =
(˜ z , ˜ z (cid:48) ) .
By checking the regularity conditions for exchanging the integral and differential operation, we verify that E (cid:2) ∇ φ ( z ) (cid:124) φ ( z (cid:48) )
(cid:3)
= ∇ E (cid:2) φ ( z ) (cid:124) φ ( z (cid:48) ) (cid:3) =
∇ E (cid:2) K T ( z , z (cid:48) )
(cid:3) .
We do not present the details here, since it is easy to check the regularity of φ ( z ) (cid:124) φ ( z (cid:48) ) as it consists of the sine and cosine functions who are continuous, bounded and have continuous bounded derivatives.
Hence, we have:
[Equation]
Hence
[Equation]
by the Markov’s inequality
[Equation]
we have
[Equation]
Then we notice that for all c ∈ C , ∆( c ) is the mean of m/ 2 terms bounded by [ − 1 , 1]
[Equation]
and the expectation is 0. So applying a union bound and the Hoeffding’s inequality on bounded random variables
[Equation]
we have:
[Equation]
Combining the above results
[Equation]
we get
[Equation]
where in the second inequality we optimize over r such that r ∗ = (cid:16)
[Equation]
{+(cid:17) 1 / ( d +3) with k 1 = 2(
+}
l 2 t 2max )
d +1 exp( − m(cid:15) 2 / 16) and k 2 = 32 σ 2 p (cid:15) − 2 .
The constant term is given by C ( d ) =
2 7 d +9 d +3 (cid:16)(cid:0) d +12 (cid:1) − d − 1 d +3 + (cid:0) d 2 (cid:1) 2 d +3 (cid:17) .
A.3. T HE G AUSSIAN PROCESS BEHAVIOR AND NEURAL TANGENT KERNEL AFTER COMPOSING WITH TEMPORAL KERNEL WITH THE RANDOM FEATURE REPRESENTATION
This section is dedicated to show the inﬁnite-width Gaussian process behavior and neural tangent kernel properties, similar to what we discussed in Appendix A.2, when composing neural networks in the feature space with the random feature representation of the temporal kernel.
For brevity, we still consider the standard L-layer FFN of (A.1).
Suppose we compose the FFN with the random feature representation φ ( x , t ) at the k th layer.
It is easy to see that the neural network kernel for the ﬁrst k − 1 layer are unchanged, so we compute them in the usual way as in (A.2).
For the k th layer, it is straightforward to verify that:
[Equation]
The intuition is that the randomness in W (thus f ( θ , . ) ) and φ ( ., . ) are independent, i.e. the former is caused by network parameter initializations and the later is induced by the random features.
The covariance functions for the subsequent layers can be derived by induction, e.g. for the ( k + 1) th layer we have:
[Equation]
In summary, composing the FNN, at any given layer, with the temporal kernel using its random feature representation does not change the inﬁnite-width Gaussian process behavior.
The statement is true for all the deep learning models who also have the Gaussian process behavior, which includes most of the standard neural architectures including RNN, CNN and attention mechanism (Yang, 2019).
The derivations for the NTK, however, is more involved since the gradient on all the layers are affected.
We summarize the result for the L-layer FFN in the following proposition and provide the derivations afterwards.
Proposition A.1.
Suppose f ( k ) (cid:0) θ , ( x , t ) (cid:1) =
vec ( f ( k ) ( θ , x ) ◦ φ ( x , t )) in the standard L-layer FFN.
Let Σ ( h ) T = Σ ( h ) for h = 1 , . .
, k , Σ ( k ) T = Σ ( k ) T ⊗ K T and Σ ( h ) T =
E f ∼N (0 , Σ ( h ) T ) [ σ ( f ) σ ( f )] +for h
= k + 1 , . . .
, L .
If the activation functions σ have polynomially bounded weak derivatives, as the network widths d 1 , . . .
, d L → ∞ , the neural tangent kernel Θ ( L ) converges almost surely to Θ ( L ) T whose partial application on parameters { W ( h ) , b ( h ) } in the
h th -layer is given recursively by:
[Equation]
Proof.
The strategies for deriving the NTK and show the convergence has been discussed in Jacot et al.
; Yang (2019); Arora et al. (2019a).
The key purpose for us presenting the derivations here is to show how the convergence results for the neural-temporal Gaussian (Section 4.2) affects the NTK.
To avoid the cumbersome notations induced by the peripheral intercept terms, here we omit the intercept terms b in the FFN without loss of generality.
We let g ( h ) = 1 √ d h σ (cid:0) f ( h ) ( x , t ) (cid:1) , so the
FFN can be equivalently deﬁned via the recursion: f ( h ) =
W ( h ) g ( h − 1) ( x , t ) .
For the ﬁnal output f (cid:0) θ , ( x , t ) (cid:1) :
= W ( L ) f
( L ) ( x , t ) , the partial derivative to W ( h ) can be given by:
[Equation]
with z ( h ) deﬁned by;
[Equation]
[Equation]
where
[Equation]
Using the above deﬁnitions
[Equation]
we have:
[Equation]
(cid:124)
[Equation]
= (cid:10)
[Equation]
1) ( x [Equation] t ) , g
[Equation] h
[Equation] 1) ( x
[Equation]
[Equation]
) (cid:11) ·
[Equation]
h ) (
[Equation]
t )
[Equation]
h ) ( [Equation] , t
[Equation] ) (cid:11)
We have established in Section 4.2 that
[Equation]
where
[Equation]
By the deﬁnition of z ( h )
[Equation]
we get
[Equation]
The approximation in the third line is made because the W ( h +1) in the right half is replaced by its i [Equation] i.d copy under Gaussian initialization.
This does not change the limit when d h → ∞ when the actionvation functions have polynomially bounded weak derivatives Yang (2019) such the ReLU activation.
Carrying out (A.23) recursively [Equation] we see that
[Equation]
Finally [Equation] we have:
[Equation]
Notice that we use a more compact recursive formulation to state the results in Proposition 1.
It is easy to verify that after expansion, we reach the desired results.
Compared with the original NTK before composing with the temporal kernel (given by (A.4)), the results in Proposition A.1 shares a similar recursion structure.
As a consequence, the previous results for NTK can be directly adopted to our setting.
We list two examples here.
• Following Jacot et al.
(2018), given a training dataset { x i , t i , y i
( t i ) } ni =1 , let f T (cid:0)
θ ( s ) (cid:1) =
(cid:0) f
( θ ( s ) , x 1 , t 1 ) , . . .
, f ( θ ( s ) , x n , t n ) (cid:1) be the network outputs at the s th training step and y T =
(cid:0) y 1 ( t 1 ) , . . .
, y n ( t n ) (cid:1) .
The analysis on the optimization trajectory under inﬁnitesimal learning rate can be conducted via:
[Equation]
where Θ T ( s ) converges almost surely to the NTK Θ ( L ) T in Proposition A.1.
• Following Allen-Zhu et al.
(2019) and Arora et al. (2019b), the generalization performance of the composed time-aware neural network can be explicitly characterized according to the properties of Θ ( L ) T .
A.3. P ROOF FOR T HEOREM
Proof.
We ﬁrst present a technical lemma that is crucial for establishing the duality result under the distributional constraint d f ( S θ T (cid:107) S ) ≤ δ .
Recall that the hidden dimension for the k th layer is d k .
Lemma A.2 (Ben-Tal et al.
Let f be any closed convex function with domain [0 , + ∞ ) , and this conjugate is given by f ∗ ( s )
=
sup t ≥ 0 { ts − f ( t ) } .
Then for any distribution S and any function g : R d k +
→
R , we have
[Equation]
We work with a scaled version of the f-divergence under f ( t ) = 1 k ( t k − 1) (because its dual function has a cleaner form), where the constraint set is now equivalent to { S θ T : d f ( S θ T (cid:107) S ) ≤ δ/k } .
It is easy to check that f ∗ ( s ) = 1 k (cid:48)
[ s ] k (cid:48) + + 1 k with 1 k (cid:48)
+ 1 k = 1 .
Similar to the proof for Proposition 1, we let z :=
[ x , t ] ∈ R d +1 and Z
=
X
× T to simplify the notations.
To explicitly annotate the dependency of the random Fourier features on Ω , which is the random variable corresponding to ω , we deﬁne ˜
φ ( z , Ω ) such that ˜ φ ( z , Ω ) = (cid:2) cos( z (cid:124) Ω 1 ) + cos
( z (cid:124) Ω 2 ) , sin( z (cid:124) Ω 1 ) + sin( z (cid:124) Ω 2 ) (cid:3) , where Ω =
[ Ω 1 , Ω 2 ] .
Then the approximation error, when replacing the sampled Fourier features φ by the original random variable ˜
φ
( z , Ω ) , is given by:
[Equation]
We ﬁrst show that sub-Gaussianity of ∆ n ( Ω ) .
Let { x (cid:48) i } ni
=1 be an i.i.d copy of the observations except for one element j such that x j (cid:54) =
x (cid:48)
j .
Without loss of generality, we assume the last element is different,
i
[Equation]
e. x n (cid:54) =
x (cid:48) n .
Let ∆ (cid:48) n ( Ω ) be computed by replacing x and z with the above x (cid:48) and its corresponding z (cid:48) .
Note that
[Equation]
where the last inequality comes from the fact that the random Fourier features ˜ φ are bounded byand the inﬁnity norm of Σ ( k ) is bounded by M .
The above bounded difference property suggests that ∆ n ( Ω ) is a 4max { 1 ,M } n -sub-Gaussian random variable.
To bound ∆ n ( Ω ) , we use:
[Equation]
[Equation]
Therefore
[Equation]
to bound sup S
[Equation]
we simply need to bound (cid:2) | ∆ n ( Ω ) | 2 (cid:3) .
Using the classical results for sub-Gaussian random variables (Boucheron et al., 2013), for λ ≤ n/ [Equation] we have
[Equation]
Then we take the integral over ω
[Equation]
Jensen’s inequality )
[Equation]
Finally, let the true approximation error be ˆ∆ n ( ω )
= ˆΣ ( k ) ( S θ T ) − Σ
( k ) ( S θ T ) .
Notice that
[Equation]
From (A.28) and (A.29), we are able to bound sup S θ T : d f ( S θ T || S )
∆ n ( Ω ) .
For the second term, recall from Proposition 1 that we have shown the stochastic uniform convergence bound for (cid:12)(cid:12)
˜
φ ( z i , Ω ) (cid:124)
˜
φ ( z j , Ω ) − φ ( z i ) (cid:124) φ ( z j ) (cid:12)(cid:12) under any distributions S θ T . The desired bound for p (cid:16) sup S [Equation] S
[Equation] S [Equation] ˆ∆ n ( ω ) (cid:12)(cid:12) ≥ (cid:15) (cid:17) is obtained after combining all the above results.
A.3. R EPARAMETRIZATION WITH I NVERTIBLE NEURAL NETWORK
In this part, we discuss the idea of constructing and sampling from an arbitrarily complex distribution from a known auxiliary distribution by a sequence of invertible transformations.
Given an auxiliary random variable z following some know distribution q ( z ) , suppose another random variable x is constructed via a one-to-one mapping from z : x = f ( z ) , then the density function of x is given by:
[Equation]
We can parameterize the one-to-one function f ( . ) with free parameters θ and optimize them over the observed evidence such as by maximizing the log-likelihood.
By stacking a sequence of Q one-to-one mappings, i.e. x = f Q ◦ f Q − 1 ◦ . . .
f 1 ( z ) , we can construct complicated density functions.
It is easy to show by chaining that p ( x ) is given by:
[Equation]
Samples from the auxiliary distribution can be transformed to the unknown target distribution in the same manner, and the transformed samples are essentially parameterized by the transformation mappings.
Unfortunately, most standard neural architectures are non-invertible, so we settle on a speciﬁc family of neural networks - the invertible neural network (INN) Ardizzone et al.
A major component of INN is the afﬁne coupling layer .
With z sampled from the auxiliary distribution, we ﬁrst divide z into two halves [ z 1 , z 2 ] and then let:
[Equation]
where s 1 ( γ , · ) (cid:1) , s 1 ( γ , · ) (cid:1) , t 1 ( γ , · ) (cid:1) , t 1 ( γ , · ) (cid:1) can be any function parameterized by different parts of γ .
Here, (cid:12) denotes the element-wise product.
Then the outcome is simply given by:
[Equation]
To see that g ( γ , · ) is invertible so the inverse transform mappings are tractable, it is straightforward to show that g − 1 ( γ , [ v 1 , v 2 ]) is given by:
[Equation]
By stacking multiple afﬁne coupling layers, scalar multiplication and summation actions (which are all invertible), we are able to construct an INN with enough complexity to characterize any non-degenerating distribution.
B S UPPLEMENTARY MATERIAL FOR S ECTION
We provide the detailed dataset description, experiment setup, model conﬁguration, parameter tuning, training procedure, validation, testing, sensitivity analysis and model analysis.
The reported results are averaged over ﬁve iterations.
B. D ATASETS
• Jena weather dataset 1 .
The dataset contains 14 different features such as air temperature, atmospheric pressure, humidity, and other metrics that reﬂect certain aspect of the weather.
The data were collected between between 2009 and 2016 for every 10 minutes, so there are 6 observations in each hour.
A standard task on this dataset is to use 5 days of observations to predict the temperaturehours in the future, which we refer to as the Case 1 .
We use a sliding window to obtain the training, validation and testing samples and make sure they have no overlaps (right panel of Figure A.2).
• Wikipedia trafﬁc.
The Wiki web page trafﬁc records the daily number of visits forWikipedia pages from 2015-07-01 to 2016-12-31.
The features are decoded from the webpage url, where we are able to obtain the project name , e.g. zh.wikipedia.org, the access , e.g. all-access, and the agent , e.g. spider.
We use one-hot encoding to represent the features, and end up with a 14-dimension feature vector for each webpage.
The feature vectors do not change with time.
We use the feature vectors and trafﬁc data from the past 200 days to predict the trafﬁc of the next 14 days, which is also a standard task for this dataset.
The missing data are treated as zero.
• Alibaba online shopping data.
The dataset contains 4,136 online shopping sequences with a total of 1,552 items.
Each shopping sequence has a varying number of time-stamped user-item interactions, from 11 to 157.
We consider the standard next-item recommendation task, where we make a recommendation based on the past interactions.
No user or item features are available.
• Walmart.com e-commerce data.
The session-based online shopping data contains ∼ 12,000 shopping sessions made by 1,000 frequent users, with a total of 2,034 items.
The lengths of the sessions vary from 14 to 87.
In order to be consistent with the Alibaba online shopping data, we do not use the provided item and user features.
We also consider the next-item recommendation task.
Preprocessing, train-validation-test split and metric To ensure fair comparisons across the various models originated from different setting, we minimize the data-speciﬁc preprocessing steps, especially for the time series dataset.
• Jena weather dataset .
We do the train-validation-test split by 60%-20%-20% on the time axis (right panel of Figure A.2).
We ﬁrst standardize the features on the training data, and then use the mean and standard deviation computed on the training data to standardize the validation and testing data, so there is no information leak.
For Case 1 , we use the observations made at each hour (one every six observations) in the most recent 120 hours (5 days) to predict the temperature 12 hours in the future.
For Case 2 , we randomly sample 120 observations from the most recent 120 hours (with a total of 720 observations), to predict the temperature 12 hours in the future.
For Case 3 , we randomly sample 120 observations from the most recent 120 hours (with a total of 720 observations), to predict the temperature randomly sampled from 4 to 12 hours (with a total of 48 observations) in the future.
• Wikipedia trafﬁc.
We use the walk-forward split (illustrated in the left panel of Figure A.2) to test the performance of the proposed approaches under different train-validation-test split schema.
The walk-forward split is helpful when the length of training time series is relatively long compared with the full time series.
For instance, on the Wiki trafﬁc data, the full sequence length is 500, and the training sequence length is 200, so it is impossible to conduct the side-by-side split.
The features are all one-hot encoding, so we do not carry out any preprocessing.
The web trafﬁcs are standardized in the same fashion as the Jena weather data.
For Case 1 , we use the most recent 200 observations to predict the full web trafﬁcs fordays in the future.
For Case 2 , we randomly sample 200 observations from the past 500 days (with a total of 500 observations), to predict the full web trafﬁcs for 14 days in the future.
For Case 3 , we randomly sample 200 observations from the past 500 days (with a total of 500 observations), to predict 6 web trafﬁcs sampled from 0 to 14 days in the future.
• Alibaba data.
We conduct the standard preprocessing steps used in the recommender system literature.
We ﬁrst ﬁlter out items that have less than 5 total occurrences, and then ﬁlter out shopping sequences that has less then 10 interactions.
Using the standard trainvalidation-test split in sequential recommendation, for each shopping sequence, we use all but the last two records as training data, the second-to-last record as validation data, and the last record as testing data.
All the training/validation/testing samples obtained from the real shopping sequences are treated as the positive record.
For each positive record (cid:8) x ( t − k ) , . . .
, x ( t ) , x ( t + 1) (cid:9) , we
[Table] treat each [Table]
[Table] + 1) [Table]
) as a negative sample, which is again a standard implementation in the recommender systems where no negative labels are available.
• Walmart.com data.
We use the same preprocessing steps and train-validation-test split as the Alibaba data.
As for the metrics , we use the standard Mean absolute error (MAE) for the time-series prediction tasks.
For the item recommendation tasks, we use the information retrieval metrics accuracy and discounted cumulative gain (DCG) .
Recall that for each shopping sequence, there is one positive sample and 100 negative samples.
We rank the candidates (cid:8) x
( t + 1) , x (cid:48) 1 ( t + 1) , . . .
, x (cid:48) 100 ( t + 1) (cid:9) according to the model’s output score on each item.
The accuracy checks whether the candidate with the highest score is the positive x ( t + 1) , and the DCG is given by 1 / log (cid:0) rank (cid:0) x ( t + 1) (cid:1)(cid:1) where rank (cid:0) x
( t + 1) (cid:1) is the position at which the positive x ( t + 1) is ranked.
B. M ODEL CONFIGURATION AND IMPLEMENTATION DETAIL
We observe from the empirical results that using kernel addition ( Σ ( h ) T ( x , t ; x (cid:48) , t (cid:48) )
=
Σ ( h ) ( x , x (cid:48) )
+
K T ( x , t ; x (cid:48) , t (cid:48) ) ) and kernel multiplication give similar results in our experiments, but kernel addition is much faster in both training and inference.
Therefore, we choose to use kernel addition as a trick to expedite the computation. Note that kernel addition corresponds to simply adding up the hidden representations from neural network, and the random features from the temporal kernel. We ﬁrst show the conﬁguration and implementation for the models we use in time-series prediction.
All the models take the same inputs for each experiment with the hyperparamters tuned on the validation data.
Note that the VAR model do not have randomness in model initialization and training, so their outputs do not have standard deviations.
For the NN+time , NN+trigo and T-NN models, the temporal structures (time feature) are added to the same part of the neural architectures (illustrated in Figure A.1), and are made to have the same dimension 32 (except for NN+time ).
We will conduct sensitivity analysis later on the dimension.
For the proposed T-NN models, we treat the number of INN (afﬁne coupling) blocks used in building the spectral distribution as tuning parameters.
All the remaining model conﬁgurations are the same across all model variants.
We do not experiment on using regularizations, dropouts or residual connections.
In terms of reparameterization, we draw samples from the auxiliary distribution once at the beginning, so we do not have to resample during the training process.
• VAR .
In the vector autoregression model, each variable is modeled as a linear combination of past values of itself and the past values of other variables in the system.
The order, i.e. the number of past values used, is treated as the hyperparameter, which we select according according to the AIC criteria on the validation data.
For experiments on the Jena weather dataset, we choose the order from { 20 , 40 , 60 , 80 , 100 , 120 } , since the maximum length of the history used for prediction is 120 for all three cases.
Similarly, for experiments on the Wiki trafﬁc dataset, we choose the order from { 40 , 60 , 80 , . . .
• RNN models .
We use the one-layer RNN model the standard RNN cells.
The hidden dimension for the RNN models is selected to be 32 after tuning on the orignal model.
To make time-series prediction, the output of the ﬁnal state is then passed to a two-layer fullyconnected multi-layer perceptron (MLP) with ReLU as the activation function.
We adjust the hidden dimensions of the MLP for each model variant such that they have approximately the same number of parameters.
• CausalCNN models .
We adopt the CausalCNN (Wavenet) architecture from Oord et al.
The original CausalCNN treats the number of ﬁlters, ﬁlter width, dilation rates and number of convolutional layers are hyperparameters.
Here, to avoid the extensive parameter tuning for each model variant, we tune the above parameter of the plain CausalCNN model and adpot them to the other model variants.
Speciﬁcally, we ﬁnd that using 16 ﬁlters where each ﬁlter has a width of 2, with 5 convolutional layers and the dilation rates given by { 2 l } 5 l =1 , gives the best result for all three cases.
Similar to the RNN models, we then pass the output to a two-layer MLP to obtain the prediction.
• Attention models .
We use a single attention block in the self-attention architecture Vaswani et al.
Unlike the ordinary attention in sequence-to-sequence learning, here we need to employ an extra causal mask (shown in Figure 1) to make sure that the model is not using the future information to predict the past.
Other than that, we adopot the same key-query-value attention setting, with their dimension as the tuning parameter.
We ﬁnd that dimension=16 gives the best result in all cases for the original model and we adopt this setting to the rest model variants.
Also, we pass the output to a two-layer MLP to obtain the prediction.
We now discuss the conﬁgurations and implementations for the recommendation models.
We ﬁnd out that the two-tower architecture illustrated in Figure A.3 is widely used for sequential recommendation (Hidasi et al., 2015; Li et al., 2017a).
Each item is ﬁrst passed through an embedding layer, and the history is processed by some sequence processing model , while the target is transformed by some feed-forward neural network (FFN) such as the MLP.
The outputs from the two towers are then combined together, often by concatenating, and then pass to the top-layer FFN to obtain a prediction score.
Hence, we adopt this complex neural architecture to examine our approaches, with the sequence processing model in Figure A.3 replaced by their time-aware counterparts given in
Figure A.1.
To be consistent across the Alibaba and Walmart.com dataset, we use the same model conﬁgurations.
Speciﬁcally, the dimension of the embedding layer is chosen to be 100, the FNN on the sequence learning layer is a single-layer MLP with ReLU as activation, and the prediction layer FNN a twolayer MLP with ReLU as activation.
We keep the modules to be as simple as possible to avoid overcomplications in model selection.
For the T-NN models, we treat the dimension of the
random Fourier features φ , and the number of INN (afﬁne coupling) blocks used in building the spectral distribution as tuning parameters.
We also do not experiment on using regularizations, dropouts or residual connections.
• GRU-Rec .
We use a single layer RNN with GRU cells as the sequence processing model.
We treat the hidden dimension of GRU as a tuning parameter, and according to the validation performance evaluated by the accuracy , dimension=32 gives the best outcome for the plain GRUforRec, and we adopt it for all the RNN variants.
• CNN-Rec .
Similar to the experiments for time-series prediction, we treat the number of convolutional layers, dilation rates, number of ﬁlters and ﬁlter width as tuning parameters.
We select the best settings for the plain CNNforRec and adopt them to all the model variants.
• ATTN-Rec .
We also use the single-head single-block self-attention model as the sequence processing model.
We treat the dimension of the key, query, value matrices (which are of the same dimension) as the turning parameter.
It turns out that dimension=20 gives the best validation performance for ATTNforRec, which we adopt to all the model variants.
Unlike the time-series data where the all the samples have a equal sequence length, the shopping sequences have various lengths.
Therefore, we set the maximum length to be 100 and use the masking layer to take account of the missing entries.
Finally, we mention the implementation to handle the different scales and formats of the timestamps.
The scale of the timespan between events can be very different across datasets, and the absolute value of the timestamps are often less informative, as they could be the linux epoch time or the calendar date.
Hence, for each sequence given by (cid:8) ( x 1 , t 1 ) , . . .
, ( x k , t k ) , ( x k +1 , t k +1 ) (cid:9) where the target is to predict ( x k +1 , t k +1 ) , we convert it to the representation under timespans: (cid:8) (( x 1 , t k +1 − t 1 )) , . . .
, ( x k , t k +1 − t k ) , ( x k +1 , 0) (cid:9) .
We then transform the scale of the timespans to match with the problem setting, for instance, in the online shopping data the timespan is measured by the minute, and in the weather prediction it is measured by the hour.
B. C OMPUTATION
The VAR is implemented using the Python module statsmodels 5 .
The deep learning models are implemented using Tensorﬂow 2.0 on a single Nvidia V100 GPU.
We use the Adam optimizer and adopt the early-stopping training schema where the trainning is stopped if the validation metric stops improving for 5 epochs.
The loss function is mean absolute error for the time series prediction, and binary cross-entropy loss with the softmax function for the recommendation tasks.
The ﬁnal metrics are computed on the hold-out test data using model checkpoints saved during training that has the best validation performance.
We brieﬂy discuss the computation efﬁciency of our approach.
From Figure A.4 we see that the extra computation time for the proposed temporal kernel approach is almost negligible compared with concatenating time to the feature vector.
The advantage is partly due to the fact that we draw samples from the auxiliary distribution only once during training.
Note that this does not interfere with our theoretical results, and greatly speeds up the computation.
B. V ISUAL RESULTS
We visualize the predictions of our approaches on the Jena weather data in Figure A.5.
We see that
T-RNN , T-CNN and T-ATTN all capture the temporal signal well.
In general, T-RNN gives slightly better predictions on the Jena weather dataset.
B. E XTRA ABLATION STUDY
We show the effectiveness of using INN to characterize the temporal kernel, compared with using a known distribution family.
We focus on the Gaussian distribution for illustration, where both the mean and (diagonal of) the covariance are treated as free parameters.
In short, we now parameterize the spectral distribution as Gaussian instead of using INN.
We denote this approach by NN-RF , to differentiate with the temporal kernel approach T-NN .
First, we compare the results between T-NN and NN-RF on the time-series prediction tasks (shown in
Figure A.6).
We see that T-NN uniformly outperforms NN-RF .
The results for the recommendation task is similar, as we show in Table A.3, where T-NN still achieves the best performance.
The numerical results are consistent with Theorem 1 where a more complex distribution family can lead to better outcome.
It also justiﬁes our usage of the INN to parameterize the spectral distribution.
B. S ENSITIVITY ANALYSIS
We conduct sensitivity analysis to reveal the stability of the proposed approaches with respect to the model selections and our proposed learning schema.
Speciﬁcally, we focus on the dimension of the random features φ and the number of INN (afﬁne coupling) blocks when parametrizing the spectral distribution.
The results for the time-series prediction tasks are given in Figure A.7, A. [Table]
[Table]
In general, the pattern is clear with respect to the dimension of φ , that the larger the dimension (within the range we consider), the better the performance.
The result is also reasonable, since a larger dimension for φ may express more temporal information and better characterize the temporal kernel On the other hand, the pattern for the number of INN (afﬁne coupling layers) blocks is less uniform, where in some cases too few INN blocks suffer from under-ﬁtting, while in some other cases too may INN blocks lead to over-ﬁtting.
Therefore, we would recommend using the number of
INN blocks as a hyperparameter, and keep the dimension of φ reasonably large.