Kernel Multimodal Continuous Attention
Anonymous Author(s)
AffiliationAddress email
Abstract
Attention mechanisms average a data representation with respect to probabilityweights.
Recently, Martins et al.
[2020, 2021] proposed continuous attentionmechanisms, focusing on unimodal exponential and deformed exponential familyattention densities: the latter can have sparse support.
Farinhas et al.
[2021]extended to multimodality via Gaussian mixture attention densities.
In this paper,we propose using kernel exponential families [Canu and Smola, 2006] and ournew sparse counterpart, kernel deformed exponential families.
Theoretically, weshow new existence results for both families, and approximation capabilities forthe deformed case.
Lacking closed form expressions for the context vector, weuse numerical integration: we prove exponential convergence for both families.
Experiments show that kernel continuous attention often outperforms unimodalcontinuous attention, and the sparse variant tends to highlight peaks of time series.
Introduction
Attention mechanisms are weighted averages of data representations [Bahdanau et al., 2015] usedto make predictions.
Discrete attention 1) cannot easily handle irregularly spaced observations, and2) attention maps may be scattered, lacking focus.
Martins et al.
[2020, 2021] proposed continuousattention, showing that attention densities maximize the regularized expectation of a function of thedata location (i.e. time).
Special cases lead to exponential and deformed exponential families: thelatter has sparse support.
They form a continuous data representation and take expectations withrespect to attention densities.
In Martins et al.
[2022] they apply this to a transformer architecture.
Martins et al.
[2020, 2021, 2022] used unimodal attention densities, giving importance to one dataregion.
Farinhas et al.
[2021] extended this to multimodal Gaussian mixture attention densities.
However 1) Gaussian mixtures lie in neither the exponential nor deformed exponential families, andare difficult to study in the context of Martins et al.
[2020, 2021]; and 2) they have dense support.
Sparse support can say that certain regions of data do not matter: a region of time has no effect onclass probabilities, or a region of an image is not some object.
We would like to use multimodalexponential and deformed exponential family attention densities, and understand how Farinhas et al.[2021] relates to the framework of Martins et al.
This paper makes three contributions: 1) we introduce kernel deformed exponential families, asparse multimodal density class, and apply it along with the multimodal kernel exponential families[Canu and Smola, 2006] as attention densities.
The latter have been used for density estimation,but not weighting data importance; 2) we theoretically analyze kernel exponential and deformedexponential family i) normalization, ii) approximation and iii) context vector numerical integrationproperties; 3) we apply them to real world datasets, showing that multimodal continuous attentionoutperforms unimodal, and that kernel deformed exponential family densities often highlight thepeaks of time series.
Approximation properties for the kernel deformed case are challenging: similarkernel exponential family results [Sriperumbudur et al., 2017] relied on exponential and logarithm properties to bound the difference of the log-partition functional at two functions: these do not holdfor deformed analogues.
We provide similar bounds by using a functional mean value inequality.
We first review unimodal continuous attention [Martins et al., 2020, 2021].
We motivate multimodalcontinuous attention via time warping.
We next describe kernel exponential families and give a novelnormalization condition relating kernel growth to the base density’s tail decay.
We then proposekernel deformed exponential families, new densities which can have support over disjoint regions.
We describe normalization and approximation capabilities.
Next we describe using these densities forcontinuous attention, including numerical integration convergence analysis.
We show experimentscomparing unimodal and multimodal attention, and conclude with limitations and future work.
Related Work
Attention Mechanisms closely related are Martins et al.
[2020, 2021, 2022], Farinhas et al.
Martins et al.
[2020, 2021] frame continuous attention as an expectation of a value function withrespect to a density, where the density solves an optimization problem.
They only used unimodal(deformed) exponential family densities: we extend this to the multimodal setting by leveragingkernel exponential families and proposing a deformed counterpart.
Farinhas et al.
[2021] proposed amultimodal continuous attention mechanism via a Gaussian mixture.
We show in Appendix A thatthis solves a slightly different optimization problem from Martins et al.
A limitationof Gaussian mixtures is lack of flexible tail decay.
Finally, Martins et al.
[2022] apply continuousattention within a transformer architecture to model long context.
This is a new application ofcontinuous attention rather than an extension of specific continuous attention mechanisms.
Also relevant are Tsai et al.
[2019], Shukla and Marlin [2021, 2022].
Shukla and Marlin [2021]provide an attention mechanism for irregularly sampled time series by use of a continuous-timekernel regression framework, but do not take an expectation of a data representation over time withrespect to a continuous pdf.
Instead they evaluate the kernel regression model at fixed time points.
This describes importance of data at a set of points rather than over continuous regions.
Shukla and
Marlin [2022] extend this to incorporate uncertainty quantification.
Other papers connect attentionand kernels, but focus on discrete attention
[Tsai et al., 2019, Choromanski et al., 2020].
Also relevantare temporal transformer papers, including Xu et al.
[2019], Li et al.
[2019, 2020], Song et al.
However, none have continuous attention densities.
Kernel Exponential Families Canu and Smola [2006] proposed kernel exponential families: Sriperumbudur et al.
[2017] analyzed theory for density estimation.
Wenliang et al.
[2019] parametrizedthe kernel with a deep neural network.
Other density estimation papers include Arbel and Gretton[2018], Dai et al.
[2019], Sutherland et al.
We apply kernel exponential families as attentiondensities to weight a value function which represents the data, rather than for density estimation.
Further, Wenliang et al.
[2019] showed a condition for an unnormalized kernel exponential familydensity to have a finite normalizer.
However, they used exponential power base densities.
We insteadrelate kernel growth rates to the base density tail decay, allowing non-symmetric base densities.
To summarize our theoretical contributions: 1) showing that multimodal continuous attention isrequired to represent time warping 2) introducing kernel deformed exponential families with approximation and normalization analysis 3) improved kernel exponential family normalization results 4)convergence analysis of numerical integration for kernel-based attention 5) characterizing Farinhaset al.
[2021] in terms of the framework of Martins et al.
Continuous Attention Mechanisms
An attention mechanism has: 1) a value function approximating a data representation. 2) an attentiondensity chosen to be ‘similar’ to another data representation, encoding it into a density 3) a context c[Martins et al., 2020] taking an expectation of the value function with respect to the attention density:
c = E T ∼ p
[ V ( T )] .
The value function V : S → R D approximates a data representation, T ∼ p ( t ) is the randomvariable or vector for locations (temporal, spatial, etc) in domain S , and p ( t ) is the attention density(potentially with respect to a discrete measure).
For discrete attention, one could have V ( t ) be a timeseries where t ∈ S a finite set of time points.
One then weights the time series with a probability vector to obtain the context vector.
An example of this for irregularly sampled time series is Shuklaand Marlin [2021].
In the continuous setting V ( t ) could be a curve or realization of a continuous-timestochastic process, and S could be [0 , τ ] where τ is a study end time.
One then weights it with anabsolutely continuous density p ( t ) wrt Lebesgue measure.
If S is a set of spatial locations and onehas image data, then the value function could be a learned representation of an image.
To choose p , one takes a data representation f and finds p ‘similar’ to f , but regularizing p .
Martinset al.
[2020, 2021] did this, formalizing attention mechanisms.
Given a probability space ( S, A , Q ) ,let M 1+ ( S ) be the set of probability densities with respect to Q .
Assume Q is dominated by a σ -finitemeasure ν (i.e. Lebesgue) and that it has density q
=
dQdν with respect to ν .
Let S ⊆ R D , F be afunction class, and Ω : M 1+ ( S )
→ R be a lower semi-continuous, proper, strictly convex functional.
Given f ∈ F , an attention density [Martins et al., 2020] ˆ p : F → R ≥ 0 solves
[Equation]
This maximizes regularized L 2 similarity between p and a data representation f .
If Ω( p ) = 99 (cid:82)
S p ( t ) log p ( t ) dQ ( t ) is the negative differential entropy, the attention density is Boltzmann Gibbsˆ p [ f ]( t ) = exp( f
[Equation] A ( f [Equation] , (3) where A ( f ) ensures (cid:82) S ˆ p [ f ]( t ) dQ = 1 (see Martins et al.
[2020] for proof).
If f ( t ) =
θ T ϕ ( t ) forparameters and statistics θ ∈ R M , ϕ ( t ) ∈ R M respectively, Eqn. 3 becomes an exponential familydensity.
For f in a reproducing kernel Hilbert space (RKHS) H , it becomes a kernel exponentialfamily density [Canu and Smola, 2006], which we propose as an alternative attention density.
One desirable class would be heavy or thin tailed exponential family-like densities.
In exponentialfamilies, the support, or non-zero region of the density, is controlled by the measure Q .
Letting Ω( p )be the α -Tsallis negative entropy Ω α ( p ) [Tsallis, 1988],
[Equation]
then ˆ p [ f ] for f ( t ) =
θ T ϕ ( t ) lies in the deformed exponential family [Tsallis, 1988, Naudts, 2004]
[Equation]
where A α ( f ) again ensures normalization and the density uses the β -exponential
[Equation]
For β < 1 , Eqn. 5 and thus deformed exponential family densities for 1 < α ≤ 2 can return 0 values.
Values α > 1 (and thus β < 1 ) give thinner tails than the exponential family, while α < 1 gives fattertails.
Setting β
= 0 is called sparsemax [Martins and Astudillo, 2016].
In this paper, we assume1 < α ≤ 2 , which is the sparse case studied in Martins et al.
We again propose to replacef ( t )
=
θ T ϕ ( t ) with f ∈ H , which leads to the novel kernel deformed exponential families .
Computing Eqn.
1’s context vector requires parametrizing V ( t ) .
Martins et al.
[2020] parametrize
V : S → R D with B ∈ R D × N as V ( t ; B ) =
B Ψ( t ) and estimate B ∈ R D × N via regularizedmultivariate linear regression.
Here Ψ = { ψ n } Nn =1 is a set of basis functions.
Let L be the number ofobservation locations (times in a temporal setting), D be the observation dimension, and N be thenumber of basis functions.
This involves regressing the observation matrix H ∈ R D
× L on a matrix
F ∈ R N × L of basis functions { ψ n } Nn =1 evaluated at observation locations { t l } Ll =
[Equation]
(6) where ∥ · ∥ F is the Frobenius norm.
Time Warping
We now draw a connection to time warping to show an advantage of our method.
One desirablesummary statistic for classification is an expectation of temporal features with respect to a global density.
However in many processes features may not be aligned in time, and we only observeunaligned curves.
For instance, electrocardiogram (ECG) heartbeat curves have a P-wave, a QRScomplex and a T-wave.
These have similar patterns between heartbeats, but may have differentdurations and peak locations.
Here we show that the expectation of a temporally aligned curve withrespect to a global density is equivalent to the expectation of the unaligned curve with respect to anindividualized density.
However even if the global density is unimodal, the individualized densitymay not be.
We first define the function that aligns a set of features to common reference times.
Definition 4.1.
(Time Warping Function)
Given references times { t 0 j } Kj =1 and individualized times{ t ik } Kk =1 , both in [0 , τ ] , a time warping function
h i : S → R for S
⊆ R ≥ 0 is a strictly increasing,differentiable, invertible function where
[Equation]
Let { X i } ni =1 , X i : S → R be observed curves, each with K features occurring at individualizedtimes { t ik } Kk =1 ⊂ [0 , τ ] increasing in k .
A set of time warping functions { h i } ni =1 map referencetimes to individualized feature times.
One can then compute aligned X ∗ i ( t )
=
X i ( h i ( t )) .
Each
X ∗ i has relevant features at the same times { t 0 k } Kk =1 .
Classically, this requires handcrafting andlocating important features and estimating a warping function.
We could then compute an expectationof the time warped curve with respect to a global fixed density p ( t ) to obtain a summary statistic 140 E
T ∼ p X ∗
i ( T ) of the aligned curve.
The following states that multimodal continuous attention canrepresent such an expectation with an attention density p i , avoiding computing X ∗ i ( t ) .
Lemma 4.2.
(Continuous Attention can Represent Time Warping)
Let h be a time warping function,g = h − 1 and X i : R → R with support on [0 , τ ] .
Assume that Q is dominated by Lebesgue measure νand let q 0 = dQdν .
Then for any fixed density p wrt Q , if g, X i , q 0 , p are continuous almost everywherewe have
[Equation]
where p i ( t ) = p ( g i ( t ))
q
( g i ( t ))
q 0 ( t ) g ′ i ( t ) and p i ( t ) is a valid probability density function.
See Appendix B.1 for proof.
Even if p ( t ) is unimodal, p i ( t ) may not be: see Appendix B.2 for anexample.
Thus we require multimodal continuous attention to represent such statistics.
Kernel Exponential and Deformed Exponential Families
We use kernel exponential families and a new deformed counterpart to obtain flexible attentiondensities solving Eqn.
2 with the same regularizers.
We first review kernel exponential families.
We then give a novel theoretical result describing when an unnormalized kernel exponential familydensity can be normalized.
This says that the normalizing constant exists when the base density hasfast enough tail decay relative to kernel growth.
Next we introduce kernel deformed exponentialfamilies, extending kernel exponential families to have either sparse support, our focus, or fatter tails.
These can attend to non-overlapping time intervals.
We show similar normalization results based onkernel choice and base density.
The normalizing constant exists when the unnormalized density hascompact support and the kernel grows sufficiently slowly.
Following this we show approximationtheory.
We conclude by showing how to compute attention densities in practice.
Kernel exponential families [Canu and Smola, 2006] extend exponential families, replacing f ( t )
=θ T ϕ ( t ) with f ∈ H a reproducing kernel Hilbert space H
[Aronszajn, 1950].
Densities can be written
[Equation]
The second equality follows from the reproducing property.
A challenge is to choose H , Q so thata normalizing constant exists, i.e., (cid:82) S exp( f ( t )) dQ < ∞ .
These densities can approximate anycontinuous density over a compact domain arbitrarily well in KL divergence, Hellinger, and L pdistance [Sriperumbudur et al., 2017].
However relevant integrals including the normalizing constantrequire numerical integration.
To avoid infinite dimensionality one generally assumes a representation of the form f = 168 (cid:80)
Ii =1 γ i k ( · , t i ) , where for density estimation [Sriperumbudur et al., 2017] the t i are the observation locations and this is the solution to a regularized empirical risk minimization problem.
For a given pair H , k , how can we choose Q to ensure that the normalization constant exists?
Wefirst give a simple example of H , f and Q where it does not .
Example 1 .
Let Q be the law of a N (0 , 1) distribution and S = R .
Let H = span { t 3 , t 4 } withk ( t, s )
=
t 3 s 3 + t 4 s 4 and f ( t ) =
t 3 + t 4 =
k ( t, 1) .
Then the following integral diverges.
[Equation]
5.1 Theory for Kernel Exponential Families
We provide sufficient conditions for Q and H so that A ( f ) the log-partition functional exists.
Werelate H ’s kernel growth rate to the tail decay of the random variable or vector T Q with law Q .
Proposition 5.1.
Let ˜ p
( t ) =
exp( f ( t )) where f ∈ H an RKHS with kernel k .
Assume k
( t, t ) ≤
L k ∥ t ∥ ξ 2 + C k for constants L k , C k , ξ > 0 .
Let Q be the law of a random vector T Q , so that
Q
[Equation]
A ) =
P ( T Q ∈ A ) .
Assume ∀ u s.t.
∥
u ∥ 2 = 1 , z >
[Equation]
for some constants η > ξ 2 , C Q , v > 0 .
Then
[Equation]
See Appendix C.1 for proof.
Based on k ( t, t ) ’s growth, we can vary what tail decay rate for T Qensures we can normalize ˜ p ( t ) .
Wenliang et al.
[2019] also proved normalization conditions, butfocused on exponential power density for a specific growth rate of k ( t, t ) rather than relating taildecay to growth rate.
By focusing on tail decay, our result can be applied to non-symmetric basedensities.
Specific kernel bound growth rate terms ξ lead to allowing different tail decay rates.
Corollary 5.2.
For ξ = 4 , T Q can be any sub-Gaussian random vector.
For ξ = 2 it can be anysub-exponential.
For ξ = 0 it can have any density.
See Appendix C.2 for proof.
5.2 Kernel Deformed Exponential Families
We now propose kernel deformed exponential families: flexible sparse non-parametric densitieswhich can be multimodal.
These take deformed exponential families and extend them to use kernelsin the deformed exponential term.
This mirrors kernel exponential families.
We write
[Equation]
where f ∈ H with kernel k .
Fig.
1b shows that they can have support over disjoint intervals.
5.2.1 Normalization Theory
We construct a valid kernel deformed exponential family density from Q and f ∈ H .
We first discussthe deformed log-normalizer.
In exponential family densities, the log-normalizer is the log of thenormalizer.
For deformed exponentials, the following holds.
Lemma 5.3.
Let Z > 0 be a constant.
Then for 1 < α ≤ 2 ,
[Equation]
where
[Equation]
See Appendix D.1 for proof.
We describe a normalization sufficient condition analagous to Proposition 5.1 for the sparse deformed kernel exponential family.
With Lemma 5.3, we can take anunnormalized exp 2 − α
( ˜ f ( t )) and derive a valid normalized kernel deformed exponential family density.
We only require that an affine function of the terms in the deformed-exponential are negative forlarge magnitude t .
Proposition 5.4.
For 1 < α ≤ 2 assume ˜ p
( t ) =
exp 2 − α ( ˜ f ( t )) with ˜ f ∈ H , H is a RKHS withkernel k .
If ∃ C t > 0 s.t. for ∥ t ∥ 2 > C t , ( α − 1) ˜ f
( t ) + 1 ≤ 0 and k ( t, t ) ≤
L k ∥ t ∥ ξ 2 + C k for someξ > 0 , then (cid:82) S exp 2 − α ( ˜ f ( t )) dQ < ∞ .
See Appendix D.2 for proof.
We now construct a valid kernel deformed exponential family densityusing the finite integral.
Corollary 5.5.
Under the conditions of proposition 5.4, assume exp 2 − α ( ˜ f ( t )) > 0 on a set A ⊆ Ssuch
that Q ( A ) > 0 , then ∃ constants Z > 0 ,
A α ( f ) ∈ R such that for f ( t ) =
1 Z
[Equation]
1 ˜ f
( t ) , thefollowing holds
[Equation]
See Appendix D.3 for proof.
We thus estimate ˜ f
( t ) =
Z α − 1 f ( t ) and normalize to obtain a densityof the desired form.
5.2.2 Approximation Theory
Kernel deformed exponential families can approximate continuous densities satisfying a tail conditionon compact domains arbitrarily well in L p norm, Hellinger distance, and Bregman divergence.
Theorem 5.6.
Let q 0 ∈ C ( S ) , such that q 0 ( t ) > 0 for all t ∈ S , where S ⊆ R d is locally compact
Hausdorff and q 0 ( t ) is the density of Q with respect to a dominating measure ν .
Suppose there existsl > 0 such that for any ϵ > 0 , ∃ R > 0 satisfying | p ( t ) − l | ≤ ϵ for any t with ∥ t ∥ 2 > R .
Define
[Equation]
Suppose k ( t, · ) ∈ C 0 ( S ) ∀ t ∈ S and the kernel integration condition (Eqn. 11) holds.
Then kerneldeformed exponential families are dense in P c wrt L r norm, Hellinger distance and Bregmandivergence for the α -Tsallis negative entropy functional.
The proof (D.4) idea is that under a kernel integrability condition, deformed exponential families parametrized by f ∈ H are dense in those parametrized by f ∈ C
( S )
(we denote thoseparametrized by f ∈ C
( S ) as P 0 ).
We can approximate C ( S ) densities satisfying the tail conditionwith P 0 densities, and thus with deformed exponential family densities.
This extends Sriperumbudur et al.
[ approximation to the deformed case: standard log and exponential rules cannot be applied.
It requires bounding Frechet derivatives and applying the functional mean value theorem.
5.3 Using Kernels for Continuous Attention
Here we describe how to compute continuous attention mechanisms with attention densitiesparametrized by functions in an RKHS H in practice.
Algorithm 1 shows the kernel deformedexponential family case: the kernel exponential family case involves a similar algorithm.
Givena base density, kernel, and inducing point locations, we start by computing kernel weights ˜ γ
i for˜ f
( t ) =
Z α − 1 f ( t ) =
(cid:80) Ii =1 ˜ γ
i k ( t, t i ) and estimating the matrix B for basis weights for the valuefunction V ( t ) =
B Ψ( t ) .
Unlike density estimation, this form for f is simply a practical way to obtainf in an RKHS, rather than a solution to an empirical risk minimization problem.
We then compute thenormalizing constant Z = (cid:82) S exp 2 − α ( ˜ f ( t )) dQ via numerical integration and use it to normalize ˜ f
( t )to obtain the attention density p ( t ) .
Finally we compute the context c =
E T ∼ p
[ V ( T )]
=
B E p [Ψ( t )]by taking the expectation of Ψ( T ) with respect to a deformed kernel exponential family density p .
Unlike Martins et al.
[2020, 2021], we lack closed form expressions and use numerical integration.
In the backwards pass we use automatic differentiation.
Note that in some cases we have numericalunderflow when computing the normalizing constant.
We discuss solutions for this in Appendix E.1.
Algorithm
Choose base density q 0 ( t ) and kernel k .
Inducing point locations { t i } I i =
Parameters { ˜ γ i } Ii =1 the weights for ˜ f ( t ) =
( Z ) α − 1 f ( t ) =
(cid:80) Ii =1 ˜ γ
i k ( t, t i ) , matrix B for basis weights for value function V ( t ) =
B Ψ( t ) .
I is number of inducing points.
Forward Pass
Compute Z = (cid:82) exp 2 − α
( ˜ f ( t ))
dQ ( t ) to obtain p ( t ) =
1 Z exp 2 − α ( ˜ f ( t )) via numerical integration
Compute E T ∼ p
[Ψ( T )] via numerical integration
Compute c = E T ∼ p
[ V ( T )]
=
B E p [Ψ( T )]
Backwards Pass use automatic differentiation
5.3.1 Numerical Integration Convergence
The trapezoidal rule’s standard one-dimensional convergence rate is O ( 1 N 2 ) for an integral over a fixedinterval, where N is the number of grid points.
We would like better convergence guarantees.
Wecan achieve exponential convergence for the numerical integrals of kernel exponential and deformedexponential family attention.
We focus on numerical integration over the real line, leaving truncationanalysis and higher dimensions to future work.
We let h > 0 be the grid size.
For functions holomorphic in a strip with rapid decay, the trapezoidal rule has exponential convergence.
For kernel exponential family attention, this gives us O (exp( − C/h )) or exponential convergence forsome C > 0 with appropriate choice of q 0 , V , and k .
Technical details are in E.2, and are based onextending real-valued analytic functions to complex functions analytic/holomorphic on a strip.
Kernel deformed exponential families, however, are not even differentiable, but we can constructa sequence of differentiable approximations by replacing the positive part/ReLU function in thedeformed exponential with softplus for increasing values of the softplus parameter.
Each differentiableapproximation has exponential convergence, and by taking limits as the softplus parameter tends toinfinity we can show that the numerical integral for kernel deformed exponential family attentionitself has exponential convergence.
Technical details are in E.3.
We also show empirical convergence analysis in Appendix E.4 and figure 2 in that appendix.
Bothkernel exponential and deformed exponential families see rapid convergence for 1d attention, providing excellent integral approximations with only 5-10 grid points.
Further, the numerical integralusing softplus is a very close approximation to that using ReLU for softplus parameters 5 and 10 .
Experiments
We investigate how often multimodal continuous attention outperforms unimodal, given the samearchitecture.
We also investigate whether these methods learn rich multimodal densties.
We denotekernel exponential family attention as kernel softmax and the deformed case as kernel sparsemax.
Our architectures have: 1) an encoder maps a discrete time series representation to attention densityparameters.
The value function V ( t ; B ) expresses an embedding of a time series as a linear
[Table] [Table]
LSTM FCN Karim et al.
[2017] is an LSTM+fully convolutional network.
TST is a transformer from
Zerveas et al.
[2021] transformation of basis functions.
The context is c = E p [ V ( T )] , which is used in 4) a classifier.
Fig.3 in the Appendices visualizes this.
For kernel softmax/sparsemax the encoder outputs are the weightsfor the kernel evaluations.
For the Gaussian mixture case, the encoder’s outputs are the mixtureweights and components means and variances.
Here we describe one synthetic and two real dataexperiments.
In Appendix F we describe an additional synthetic irregularly sampled accelerometertime series experiment (uWave) and in Appendix G we describe a sentiment classification experiment.
All experiments were conducted on either a single Titan X GPU or a single 1080.
6.1 Synthetic Experiment: Time Warping
We simulate time warping and do prediction for unimodal vs multimodal attention densities.
Details ofthe original time aligned stochastic process and inverse warping function are in Appendix B.3.
Givenglobal densities for classes p 1 , p 2 and an aligned X ∗ , the class is arg max( ⟨ p 1 , X ∗ ⟩ , ⟨ p 2 , X ∗ ⟩ ) .
Wegenerate 10 , 000 trajectories of length 95 observed at evenly spaced time points in the interval [0 , 1] .
We use two attention mechanisms (heads), one for each class.
Letting V be the value function fit toobserved data, the classifier is softmax ([ ⟨ p i 1 , V ⟩ L 2 , ⟨ p i 2 V ⟩ L 2 ])
where p i 1 and p i 2 are the attentiondensities for the first and second class.
Table 1 shows prediction results from 100 runs along with1 .
96 standard deviation intervals.
Kernel methods outperform by 7 − 10% .
6.2 ECG heartbeat classification
We use a kaggle version 2 of the MIT Arrhythmia Database [Goldberger et al., 2000].
Not all versionsare comparable:
Kachuee et al.
[2018] report results under well-balanced classes (unclear how theyobtain these), while Mousavi and Afghah [2019] augment the dataset with SMOTE.
We do no dataaugmentation, so compare to two popular time series classification baselines in TSAI: a hybrid fullyconvolutional and LSTM networkKarim et al.
[2017], and a transformer Zerveas et al.
The task is to detect abnormal heart beats from ECG.
The five classes are {Normal, Supraventricularpremature, Premature ventricular contraction, Fusion of ventricular and normal, Unclassifiable}.
There are 87,553 training samples and 21,891 test samples.
Each sample is a univariate time seriesof length 187: we pass this through two convolutional layers in order to obtain a multivariate v1.
[Table]
representation of each time step.
We then use an LSTM.
The hidden layer is used to construct discreteattention using Bahdanau et al.
Following Martins et al.
[2021] for unimodal continuoussoftmax and sparsemax, we first output discrete attention weights p
= ( p 1 , · · · , p L ) , p ∈ ∆ L in theprobability simplex
and then compute µ = E p [ T/L ] and σ 2 =
E p
[( T/L ) 2 ] − µ 2 where T ∼ p .
Thevalue function uses Gaussian RBFs.
The final context vector is passed through three feedforwardlayers.
Additional details, including hyperparameters and plots of attention densities, are in Appendix
H. Kernel softmax attention is not particularly interpretable.
For kernel sparsemax, the attentiondensities tend to highlight peaks in the signal.
Particularly they assign high weight to the R wave, thepeak of the QRS complex, of the heartbeat
6.3 FordA Dataset
This is a binary classification dataset for whether a sympton exists in an automotive subsystem.
Eachtime series has 500 sensor observations, and there are 3601 training samples and 1320 test samples.
Table 3 shows results.
Kernel sparsemax outperforms baselines, while kernel softmax also doeswell.
We use the same methods and architecture as the previous section, although hyperparametersare slightly different.
Several methods, including continuous sparsemax and the transformer, failcompletely.
To sanity check we fit SVM with a Gaussian kernel, logistic regression, and a decisiontree: accuracies are under 55% for these, and we conclude that this data poses difficulty for someclassifiers.
Appendix I provides some additional details, along with attention density plots.
Thekernel sparsemax plots often select peaks of a signal while learning rich sparsity patterns.
Discussion
In this paper we extend continuous attention mechanisms to use kernel exponential and deformedexponential family densities.
The latter is a new flexible class of non-parametric densities withsparse support.
We show novel existence properties for kernel exponential and deformed exponentialfamilies, prove approximation properties for the latter, and show exponential convergence of numericalintegration for both attention mechanisms.
We then apply these to the continuous attention frameworkdescribed in Martins et al.
We show results on several datasets.
Kernel attentionmechanisms tend to outperform unimodal attention, often by a large margin.
We also see that in manycases they exhibit multimodality.
Kernel sparsemax in particular learns rich sparsity patterns whilehighlighting peaks of a signal.
This was evident in the ECG example, where it tends to give very highweight to R waves in a signal.
While this paper is more focused on general methods than a specificpotentially dangerous application, potential application areas include wearable sensors and NLP.
7.1 Limitations
A limitation of this work was the use of numerical integration, which scales poorly with locationdimensionality.
While we achieve exponential convergence in the 1d case, it is not clear how this willscale to higher dimensional settings.
This still allows for multiple observation dimensions at a given1d location, i.e. multivariate time series.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We included
the code for two experiments and are working on preparing the rest.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes] We did it for two of our smaller scale experiments where
the datasets were small and sequence lengths were short: the synthetic time warping
experiment and the uWave experiment in Appendix F.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] We mention the type of resources
used.
These are relatively small scale experiments so we don’t need to discuss total
compute as environmental impacts are negligible.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
Overview of Appendices
Appendices A to E are mostly theoretical.
In Appendix A we describe how the Gaussian mixtureattention from Farinhas et al.
[2021] relates to the theoretical framework of Martins et al.
In Appendix B we discuss time warping, showing that sufficiently flexible continuousattention can represent time warping, that in at least in one case a multimodal density is required, andgive some details describing our experiments.
In Appendix C we discuss normalization for kernelexponential families, while in Appendix D we do the same along with approximation results for kerneldeformed exponential families.
In Appendix E we discuss rate of convergence for approximatingthe attention mechanism with numerical integration, and show some synthetic experiments showingrapid convergence empirically.
The remaining appendices are empirical.
In Appendix F and Appendix G, we include two additionalexperiments.
The first analyzes an accelerometer classification dataset, where we add syntheticirregular sampling.
The second analyzes the IMDB dataset, a sentiment classification dataset.
In
Appendix H we provide additional details for the MIT-BIH experiment.
Finally, in Appendix Iwe provide additional details for the FordA experiment.
The latter two appendices in particularfocus on visual comparisons of attention densities.
We find that kernel deformed exponentialfamilies/sparsemax tend to learn more interesting looking attention densities than all of the othermethods, and for ECG kernel sparsemax has the interpretation of selecting regions where the electricalstimuli are largest.
The licenses are: for MIT BIH open data commons attribution.
For UCR time series (uWave and
Ford) the exact license is difficult to find but the citation is Dau et al.
For IMDB it is againdifficult to find the exact license, but the original paper is Maas et al.
[2011] and the website 3 asksthat it be cited.
A Gaussian Mixture Model
In this section we descibe how the Gaussian mixture attention of Farinhas et al.
[2021] relates to theoptimization definition of attention densities in Martins et al.
In fact their attentiondensities solve a related but different optimization problem.
Martins et al.
[2020, 2021] showthat exponential family attention densities maximize a regularized linear predictor of the expectedsufficient statistics of locations.
In contrast, Farinhas et al.
[2021] find a joint density over locationsand latent states, and maximize a regularized linear predictor of the expected joint sufficient statistics.
They then take the marginal location densities to be the attention densities.
Let Ω( p ) be Shannon entropy and consider two optimization problems:
[Equation]
The first is Eqn. 2 with f = θ T ϕ ( t ) and rewritten to emphasize expected sufficient statistics.
If onesolves the second with variables Z , we recover an Exponential family joint density
[Equation]
This encourages the joint density of T, Z to be similar to a complete data representation θ T ϕ ( t, z ) ofboth location variables T and latent variables Z , instead of encouraging the density of T to be similarto an observed data representation θ T ϕ ( t ) .
The latter optimization is equivalent to
[Equation]
The constraint terms c m are determined by θ .
Thus, this maximizes the joint entropy of Z and T ,subject to constraints on the expected joint sufficient statistics.
To recover EM learned Gaussian mixture densities, one must select ϕ m so that the marginal distribution of T will be a mixture of Gaussians, and relate c m to the EM algorithm used to learn themixture model parameters.
For the first, assume that Z is a multinomial random variable with a singletrial taking | Z | possible values and let ϕ ( t, z )
=
( I ( z = 1) , · · · , I ( z = | Z | − 1) , I ( z = 1) t, I ( z =1) t 2 , · · · , I ( z = | Z | ) t, I ( z = | Z | ) t 2 ) .
These are multinomial sufficient statistics with a single trial,followed by the sufficient statistics of | Z | Gaussians multiplied by indicators for each z .
Then p ( T | Z )will be Gaussian, p ( Z ) will be multinomial, and p ( T ) will be a Gaussian mixture.
For contraints,
Farinhas et al.
[2021] have
[Equation]
at each EM iteration: p old ( Z | T = t l ) is the previous iteration’s latent state density conditional on theobservation location, w l are discrete attention weights, and t l is a discrete attention location.
That EMhas this constraint was shown in Wang et al.
This matches expected joint sufficient statisticsto those implied by discrete attention over locations, taking into account the dependence betweenz and t l given by old model parameters.
An alternative is simply to let θ be the output of a neuralnetwork.
While the constraints lack the intuition of Eqn. 9, it avoids the need to run EM.
We focuson this case and use it for our baselines.
B Time Warping
[Figure] B.1 Proof of 4.2
Here we prove that continuous attention can represent time warping.
We change measure to Lebesgue,use the equivalence between Lebesgue and Riemann for almost everywhere continuous functions,use integration by substitution, and then change back.
[Equation]
Further, since h is strictly increasing, g is as well so that g ′ i ( t ) > 0 ∀ t ∈ [0 , τ ] .
This gives us that p i ( t )is non-negative, and we have to show that it integrates to 1 .
[Equation]
B.2 Multimodal Densities are Required
Consider the case where p global ( t ) is an exponential density function and g ( t ) the inverse warpingfunction has the following form [Equation] , else
[Equation]
t else for C 1 = τ (cid:82)
[Equation]
0 exp( W ( u )) du and some twice diferentiable function W .
The role of C 1 is to ensure thatg ( τ )
=
τ .
Then
[Equation]
so that
[Equation]
Now note that depending on the sign and magnitude of W ′′ ( t ) , p ′′
i ( t ) may be either positive ornegative and thus p
i ( t ) may lack a unique optimum and may be multimodal.
B.3 Time Warping Experimental Details
This describes the aligned vs the observed stochastic process.
The original time aligned stochastic process is given by
[Equation]
where Z k ∼ U ( − 4 , 4) , k = 1 , 2 , 3 , 4 .
We instead observe realizations
X i ( t ) =
X ∗ i ( g ( t )) at fixedtime points with the following inverse warp function
[Equation]
t else
The global densities are p 1 ∼ U (0 , 0 . 5) and p 2 ∼ U (0 . 5 , 1) .
C Proof Related to Proposition 5.1C.1 Proof of Proposition 5.1
Here we prove that if the kernel evaluated twice at the same point grows sufficiently slowly withrespect to base density tail decay, then we can normalize.
Proof.
This proof has several parts.
We first bound the RKHS function f and use the generaltail bound we assumed to give a tail bound for the one dimensional marginals T Qd of T Q .
Usingthe RKHS function bound, we then bound the integral of the unnormalized density in terms ofexpectations with respect to these finite dimensional marginals.
We then express these expectationsover finite dimensional marginals as infinite series of integrals.
For each integral within the infiniteseries, we use the finite dimensional marginal tail bound to bound it, and then use the ratio test toshow that the infinite series converges.
This gives us that the original unnormalized density has afinite integral.
We first note, following Wenliang et al.
[2019], that the bound on the kernel in the assumption allowsus to bound f in terms of two constants and the absolute value of the point at which it is evaluated.
[Equation]
We can write T Q =
( T Q 1 , · · · , T QD ) .
Let e d be a standard Euclidean basis vector.
Then by theassumption and setting u = e d we have
[Equation]
Letting Q d be the marginal law,
[Equation]
which will be finite if each E exp( C 2 | T Qd | ξ/ 2 ) < ∞ .
Now letting S d be the relevant dimension of S ,
E
[Equation]
where the inequality follows since S d ⊆ R , exp is a non-negative function and probability measuresare monotonic.
We will show that the second sum converges.
Similar techniques can be shown for first
[Equation]
assumption
Then
[Equation]
C
[Equation]
C
[Equation]
−
[Equation]
η )
Let a j = exp( C 2 | j | ξ/ 2 − vi η ) .
We will use the ratio test to show that the RHS converges.
We have
[Equation]
We want this ratio to be < 1 for large j .
We thus need to select η so that for sufficiently large j , wehave
[Equation]
Assume that η > ξ 2 .
Then
[Equation]
Since the RHS is unbounded for η > ξ 2 , we have that Eqn.
10 holds for sufficiently large j .
By the ratio
[Equation]
is finite.
Thus putting everything together we have
[Equation]
and ˜ p ( t ) can be normalized.
C.2 Proof of Corollary 5.
Here we prove two special cases of kernel growth rate and tail decay.
Proof.
Let ξ = 4 .
Then η > 2 and
[Equation]
<
[Equation]
The second case is similar.
For the uniformly bounded kernel,(cid:90)
[Equation]
[Equation]
D Proofs Related to Kernel Deformed Exponential Family
D.1 Proof of Lemma 5.
Here we prove a key equality for calculating normalization of deformed exponential families.
Proof.
The high level idea is to express a term inside the deformed exponential family that becomes1 /Z once outside.
Recalling the definition of β -exponential in Eqn. 5,
[Equation]
D.2 Proof of Proposition 5.4
Here we prove that if ˜ f ( t ) has compact support and the kernel grows sufficiently slowly, then itsdeformed exponential has a finite integral.
Proof.
[Equation]
where the 2nd to last line is due to the inequality | f ( t ) | ≤ C
+ C
∥ t ∥ ξ/ 2 from the proof in Appendix
C.1.
D.3 Proof of Corollary 5.
Here we prove that we can normalize deformed exponential families under the previous conditions.
Proof.
From proposition 5.4 and the assumption,
[Equation]
for some Z > 0 .
Then
[Equation]
where the second line follows from lemma 5.3.
Set
A α ( f ) = log α
( Z )
and we are done.
D.4 Approximation Theory
This section proves Theorem 5.6.
We start with a Proposition, which says that under a kernelintegration condition, deformed exponential families parametrized by functions in H can approximatesimilar densities parametrized by functions in C 0 ( S ) arbitrarily well.
Proposition D.1.
Define
[Equation]
where S ⊆ R d . Suppose k ( x, · ) ∈ C 0 ( S ) , ∀ x ∈ S and
[Equation]
[Equation]
here M b ( S ) is the space of bounded measures over S .
Then the set of deformed exponential familiesis dense in P 0 wrt L r ( Q ) norm and Hellinger distance.
Proof.
The kernel integration condition tells us that H is dense in C 0 ( S ) with respect to L ∞ norm.
This was shown in Sriperumbudur et al.
For the L r norm, we apply ∥ p f − p g
∥ L r ≤2 M exp ∥
f − g ∥ ∞ from Lemma D.5 with f ∈ C 0 ( S ) , g ∈ H , and f 0 = f .
L 1 convergence implies
Hellinger convergence.
We can then use this for our main proof of Theorem 5.6.
Note that our Bregman divergence resultis analagous to Sriperumbudur et al.
[2017]’s KL divergence result.
KL divergence is Bregmandivergence with the Shannon entropy functional: we show the same for Tsallis entropy, which ismaximized given expected sufficient statistics by deformed exponential families [Naudts, 2004].
The
Bregman divergence describes how close a density’s uncertainty is to its first order approximationevaluated at another density.
Proof.
For any p ∈ P c , define p δ =
p
+ δ 1+ δ .
Then∥ p − p δ
∥
r = δ 1 + δ ∥
p ∥ r →for 1 ≤ r ≤ ∞ .
Thus for any ϵ > 0 , ∃ δ ϵ > 0 such that for any 0 < θ < δ ϵ , we have ∥ p − p θ
∥ r ≤ ϵ ,where p θ ( t ) > 0 for all t ∈ S .
Define f = (cid:16) 1+ θ l + θ (cid:17) 1 − α log 2 − α p θ 1+ θ l + θ .
Since p ∈ C ( S ) , so is f .
Fix any η > 0 and note that
[Equation]
Thus
[Equation]
Since p − l ∈ C 0 ( S ) the set on the second line is bounded.
Thus A is bounded so that f ∈ C 0 ( S ) .
Further, by Lemma 5.
[Equation]
giving us p θ ∈ P 0 .
By Proposition D.1 there is some p g in the deformed kernel exponential familyso that ∥ p θ − p g
∥ L r ( S ) ≤ ϵ .
Thus ∥ p − p g ∥ r ≤ 2 ϵ for any 1 ≤ r ≤ ∞ .
To show convergence in
Helinger distance, note
[Equation]
so that L 1 ( S ) convergence, which we showed, implies Hellinger convergence.
Let us consider the
Bregman divergence.
Note the generalized triangle inequality 4 for Bregman divergence
[Equation]
[Equation]
Term
I
[Equation]
Note that the Bregman divergence is non-negative and thus we only need to worry about an upperbound.
The first term on the rhs clearly vanishes as θ → 0 since p θ → p .
For the second term, wealready showed that ∥ p − p θ ∥ ∞ → 0 .
Term II
Fix θ .
Then term II converges to 0 by Lemma D.5.
Term III
For term III ,
[Equation]
Clearly the first term on the rhs converges by L r convergence.
The L 1 term for the gradient is givenby
[Equation]
so that the inner product terms are bounded as
[Equation]
Lemma D.2.
(Functional Taylor’s Theorem) Let F : X → R where X is a Banach space.
Letf, g ∈ X and let F be k times Gateaux differentiable.
Then we can write
[Equation]
for some c ∈ [0 , 1] .
Proof.
This is simply a consequence of expressing a functional as a function of an ϵ ∈ [0 , 1] , whichrestricts the functional to take input functions between two functions.
We then apply Taylor’stheorem to the function and apply the chain rule for Gateaux derivatives to obtain the resulting Taylorremainder theorem for functionals.
Let G ( η ) =
F ( f + η ( g − f )) .
By Taylor’s theorem we have
[Equation]
and applying the chain rule gives us
[Equation]
Lemma D.3.
(Functional Mean Value Theorem)
Let F : X → R be a Gateaux differentiablefunctional where f, g ∈ X some Banach space with norm ∥ · ∥ .
Then
[Equation]
where h = g + c ( f − g ) for some c ∈ [0 , 1] , F ′ ( h ) is the Gateaux derivative of F , and ∥ · ∥ op is theoperator norm ∥
A ∥ op = inf { c > 0 :
∥
Ax ∥ ≤ c ∥ x ∥∀ x ∈ X } .
Proof.
Consider G ( η ) =
F ( g + η
( f − g )) .
Apply the ordinary mean value theorem to obtain
[Equation]
and thus
[Equation]
Claim 1 .
Consider P ∞ =
{ p f = exp 2 − α ( f − A α ( f )) : f ∈ L ∞ ( S ) } .
Then for p f ∈ P ∞ ,
A α ( f ) ≤ ∥ f ∥ ∞ .
Proof.
[Equation]
where for the second line recall that we assumed that throughout the paper 1 < α ≤ 2 .
Lemma D.4.
Consider P ∞ =
{ p f = exp 2 − α ( f − A α ( f )) : f ∈ L ∞ ( S ) } .
Then the Frechetderivative of A α : L ∞ → R exists.
It is given by the map
[Equation]
Proof.
This proof has several parts.
We first derive the Gateaux differential of p f in a directionψ ∈ L ∞ and as it depends on the Gateaux differential of A α ( f ) in that direction, we can rearrangeterms to recover the latter.
We then show that it exists for any f, ψ ∈ L ∞ .
Next we show that thesecond Gateaux differential of A α ( f ) exists, and use that along with a functional Taylor expansion toprove that the first Gateaux derivative is in fact a Frechet derivative.
In Martins et al.
[2020] they show how to compute the gradient of A α ( θ ) for the finite dimensionalcase: we extend this to the Gateaux differential.
We start by computing the Gateaux differential of p f .
[Equation]
evaluating at η = 0 gives us
[Equation]
Note that by claim 1 we have
[Equation]
We can thus apply the dominated convergence theorem to pull a derivative with respect to η under anintegral.
We can then recover the Gateaux diferential of A α via
[Equation]
where the last line follows as ψ ∈ L ∞ .
Thus the Gateaux derivative exists in L ∞ directions.
Thederivative at f maps ψ : → E ˜ p 2 − α f ( ψ ( T )) i.e. A ′ α
( f )( ψ ) =
E ˜ p
[Equation]
( ψ ( T )) .
We need to show that thisis a Frechet derivative.
To do so, we will take take second derivatives of p f + ηψ ( t ) with respect toη in order to obtain second derivatives of A α ( f + ηψ ) with respect to η .
We will then constructa functional second order Taylor expansion.
By showing that the second order terms convergesufficiently quickly, we will prove that the map ψ : → E ˜ p 2 − α f ( ψ ( T )) is a Frechet derivative.
[Equation]
We need to show that we can again pull the second derivative under the integral.
We already showedthat we can pull the derivative under once (for the first derivative) and we now need to do it again.
We need to show an integrable function that dominates p f + ηψ ( t ) 2 − α
[Equation]
ψ
[Equation] [Equation]
which is in L 1 ( Q ) .
Now applying the dominated convergence theorem
[Equation]
and rearranging gives
[Equation]
since f, ψ ∈ L ∞ .
For the functional Taylor expansion, we have from Lemma D.
[Equation]
)for some ϵ ∈ [0 , 1] .
We thus need to show that for ϵ ∈ [0 , 1] ,(
[Equation]
− α )
[Equation]
[Table] [Table]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
→ 0 as ψ →and plugging this in we obtain the desired result.
Thus the Frechet derivative of A α ( f ) exists.
Lemma D.5.
Define P ∞ =
{ p f = exp 2 − α ( f − A α ( f )) : f ∈ L ∞ ( S ) } where L ∞ ( S ) is the spaceof almost everywhere bounded measurable functions with domain S .
Fix f 0 ∈ L ∞ .
Then for anyfixed ϵ > 0 and p g , p f ∈ P ∞ such that f, g ∈ B ∞
ϵ
( f 0 ) the L ∞ closed ball around f 0 , there existsconstant M exp > 0 depending only on f 0 such that
[Equation]
Further
[Equation]
Proof.
This Lemma mirrors Lemma A.1 in Sriperumbudur et al.
[2017], but the proof is very differentas they rely on the property that exp( x + y )
=
exp( x ) exp
( y ) , which does not hold for β -exponentials.
We thus had to strengthen the assumption to include that f and g lie in a closed ball, and then use thefunctional mean value theorem Lemma D.3 as the main technique to achieve our result.
Consider that by functional mean value inequality,
[Equation]
(13) where h = cf + (1 − c ) g for some c ∈ [0 , 1] .
We need to bound exp β
( h − A α ( h )) and ∥
A α ( f ) −
A α ( g ) ∥ ∞ .
We can show a bound on ∥ h ∥ ∞
[Equation]
so that h is bounded.
Now we previously showed in claim 1 that |
A α ( h ) | ≤ ∥ h ∥ ∞ ≤ ϵ + ∥ f 0 ∥ ∞ .
Since h, A α ( h ) are both bounded exp β ( h − A α ( h )) 2 − α is also.
Now note that by Lemma D.3,
[Equation]
We need to show that ∥
A ′ α ( h ) ∥ op is bounded for f, g ∈ B ϵ ( f 0 ) .
Note that in Lemma D.4 we showedthat
[Equation]
Thus ∥
A
[Equation]
α ∥ op = sup {| A
[Equation]
α
[Equation] [Equation]
m ) | : ∥ m
∥ ∞ = 1 } ≤ 1 .
Let M exp be the bound on exp β ( h −
A α ( h )) .
Then putting everything together we have the desired result
[Equation]
Now
[Equation]
For the inner prodct term, first note that following Martins et al.
[2020] the gradient is given by
[Equation]
Thus
[Equation]
where the second line follows from claim 1.
Further note that by Taylor’s theorem,
[Equation]
for some z between x and y .
Then letting y = p f ( t ) and x = p g ( t ) , we have for some z = h ( t ) lyingbetween p f ( t ) and p g ( t )
that
[Equation]
Since f ∈ L ∞ then applying Claim 1 we have that each p f , p g ∈ L ∞ and thus h is.
Then
[Equation]
so that
[Equation]
Putting it all together we obtain
[Equation]
E Numerical Integration Convergence and Stability Analysis
E.1 Stable Numerical Integration
One issue is numerical underflow when computing the normalizing constant.
In the kernel exponentialfamily case, if | f ( t ) | is very large and f ( t ) < 0 for all evaluated t , then on a computer exp( f ( t )) will round to 0 for all the evaluated t .
Thus Z the normalizing constant will be estimated as 0 innumerical integration and the estimate of exp( f ( t )) /Z
cannot be computed.
A similar issue existsfor the deformed case.
For kernel exponential family, we can use the standard technique used in discrete softmax implementations.
Note
[Equation]
Then instead of normalizing exp( f ( t )) with exp( − A ( f ))
we normalize exp( f ( t ) − C ) with exp( C −
A ( f )) , letting C = sup t f ( t ) .
Taking C = sup t f ( t ) will prevent underflow as if f ( t ) < 0 ∀ t ,f ( t )
− sup f ( t ) ≈
for some t (equality if there is a maximum, which there always is when using afinite set of t for numerical integration).
For the deformed case, recall that by Lemma 5.3, exp 2 − α
[Equation]
f
[Equation]
t [Equation] Z
[Equation]
Z [Equation] f ( t ))for Z > 0 . Then letting log α C = sup t f ( t ) and noting that for x, y > 0 , log α x − log α y =log
[Equation]
⊘
[Equation]
=
[Equation]
− y
[Equation]
+ 1] 1 1 −
[Equation]
+ [Suyari et al., 2004],
[Equation]
(cid:32)(cid:18)
Z ⊘
[Equation]
C (cid:19)
[Equation]
( f ( t ) − log α C ) (cid:33)
Now consider ˜ f
( t ) =
(cid:16) 1 Z ⊘ α C (cid:17) α − 1 f ( t ) .
Thensup
[Equation]
˜ f ( t )
[Equation]
(cid:18)
Z ⊘ α C (cid:19)
[Equation]
log α C
We can thus estimate ˜ f
( t ) , subtract sup t ˜ f ( t ) (max in practice), take the deformed exponential, andnormalize, similar to the kernel exponential family case.
E.2 Kernel Exponential Family Attention
Here we show conditions for which numerical integration of
[Equation]
using the trapezoidal rule is exponentially convergent.
We start by restating a theorem.
This says thatthe trapezoidal rule for numerical integration of holomorphic functions of sufficiently fast decay hasexponential convergence.
A version of this theorem comes from Trefethen and Weideman [2014], butthere are slight issues with the notation and conditions.
A slightly revised statement is in the coursenotes of Lee, which we follow here.
Theorem E.1.
Let w : C → C be analytic in the strip S b = { z ∈ C : | Im ( z ) | < a } for some a > 0 .
Suppose further that w ( z ) → 0 as | z | → ∞ in the strip, and for some M > 0 ,
[Equation]
(17) for all b ∈ ( − a, a ) .
Then leting I = (cid:82) ∞
−∞ w ( x ) dx
and I h
= h (cid:80) ∞
k =
−∞ w ( kh )
[Equation]
[Equation]
Next we give conditions on q 0 ( t ) , V ( t ) , and k so that numerical integration is exponentially convergent.
Specifically that q 0 and V have complex extensions to a strip and that k has exponentiallydecaying Fourier transform.
Corollary E.2.
Assume that q 0 , V have complex extensions to a strip S b = { z ∈ C : | Im ( z ) | < b }and V is bounded by M V .
Also assume that for any t i ∈ R the absolute value of the Fourier transform| ˆ k ( ξ, t i ) | = | (cid:82) ∞
−∞ k ( t, t i ) exp( − 2 πiξt ) dt | ≤ | A ( t i ) exp( − a | ξ | ) | , where A : R → C and a > 0 is afixed constant.
Further assume that f ( t ) = (cid:80) Ii =
1 γ i k ( t, , t i ) , k ( t, t i ) ∈ C 0 , and(cid:90)
[Equation]
(18) for some M > 0 .
Then if w ( z ) = q 0 ( t ) exp( f ( t )) V ( t ) converges uniformly to 0 as | z | → ∞ , thetrapezoidal rule for w ( t ) =
q 0 ( t ) exp( f ( t ))
V ( t ) is exponentially convergent.
Proof.
Note that
[Equation]
and thus| ˆ f ( ξ ) | ≤
I (cid:88) i =
[Equation]
) |
Thus since the fourier transform of f has (at least) exponential decay, by Theorem 3.1 in Stein and
Shakarchi [2010] the extension of f ( t ) to f ( z ) is holomorphic in the strip.
Since compositions andproducts of holomorphic functions are holomorphic, q 0 ( z ) exp( f ( z ))
V ( z ) is holomorphic in the strip.
Further, since for each t i , k ( t, t i ) ∈ C 0 , f is and thus it is bounded and thus its complex extension isbounded, thus the complex exp( f ) is bounded, say by exp( M f ) .
Since V is also bounded, we have(cid:90)
[Equation]
and thus w satisfies the conditions of the previous theorem.
Finally, we show example special cases of q 0 , V, k satisfying those conditions.
Corollary E.3.
If k is a Gaussian kernel, V is a linear combination of Gaussian RBFs, and q 0 ( t ) =1 √ 2 π exp( − t 2 / 2) , then the trapezoidal rule for kernel exponential family attention is exponentiallyconvergent.
Proof.
For the Gaussian kernel, we have k ( t, t i ) =
exp( − ζ 2 ( t − t i ) 2 ) .
Now
[Equation]
and thus|
[Equation]
which satisfies the Fourier decay condition for the kernel.
A similar technique can be applied to V ,and clearly both V and f bounded since they are continuous and vanish at infinity.
Thus exp( f ) isalso bounded.
For the decay of q , note
[Equation]
satisfying the integration condition Eqn. 17.
It remains to show that w ( z ) → 0 .
Note that since V, fare bounded and q ( z ) → 0 (which was shown in proving the integration condition above) in the stripas | z | → ∞ , w ( z ) → 0 as | z | → ∞ .
E.3 Convergence of Numerical Integration for Kernel Deformed Exponential FamilyAttentionE.3.1 Smooth Approximation to Kernel Deformed Exponential Family Attention
Definition E.4.
Let
[Equation] [Equation]
(19) for some ρ > 0 .
Claim 2 .
exp ρ,β ( z ) is holomorphic on the strip { z ∈ C :
Im ( z ) ∈ [ − π/ 2 , π/ 2] } for the principalbranch of log .
Proof.
Note that this proof is adapted from [https://math.stackexchange.com/users/8508/robertisrael].
We show it for exp 0 ,
but the idea can easily be extended to more general ρ and β .
Note thatthe principal branch of log z is analytic outside of B = { z = x + iy : −∞ < x ≤ 0 , y = 0 } .
Thusexp 0 , 0 is analytic as long as 1 + exp(1 + z ) is not in B .
Now exp(1+ z ) = exp( x +1) exp( iy ) , so exp(1+ z ) will be in { z = x + iy : x ∈ ( −∞ , − 1] , y = 0 }when exp( x + 1) ≥ 1 , exp( iy )
= − 1 , i.e. x ≥ − 1 ,
y =
π
+ 2 πn, n ∈ N .
The y condition will notbe satisfied on the strip above and thus exp 0 ,
( z ) is holomorphic on that strip.
Corollary E.5.
If k is a Gaussian kernel, V is a linear combination of Gaussian RBFs, andq 0 ( t )
= 1 √ 2 π exp( − t 2 / 2) , then the trapezoidal rule for softplus approximation to kernel deformedexponential family attention is exponentially convergent.
Proof.
Apply Theorem E.1.
We already showed in the kernel exponential family section that q 0 and
V are holomorphic in the strip, and we showed that exp ρ,β is in the previous claim.
We also showedthat V is bounded and that q 0 satisfies the integration condition.
It remains to show that exp ρ,β ( f ( t ))is bounded (and thus its complex extension is).
We show it for exp ρ, 0 ( f ( t )) but again the idea can beextended to general β .
Note that
[Equation]
since f is bounded.
Further, fixing t , log(1 + exp( ρ (1 + t ))) /ρ is monotonically decreasing as afunction of ρ and thus for all ρ ≥ 1 ,
[Equation]
Thus w satisfies both the integration condition and the convergence condition.
E.3.2 Using the Smooth Approximation to Bound the Numerical Integral for KernelSparsemax Attention
Let I h be the numerical integral proportional to kernel deformed exponential family attention, I h,s beits softplus approximation, I be the true kernel deformed exponential family integral, and I s be itssoftplus approximation.
Then
[Equation]
We already bounded | I h,s −
I s | in the previous subsection, so we will bound the other two terms onthe right hand side.
E.3.3 Bounding I s − I
We first bound the difference between the softplus integral and the integral using ReLU.
By Hoelder’sinequality,
[Equation]
E.3.4 Bounding I h − I h
[Equation]
s
[Equation]
where the last line again uses Hoelder’s inequality.
E.3.5 Putting it All Together
We now have, for any ρ > 0 ,
[Equation]
and since this holds for all ρ > 0 we can take ρ → ∞ and we have
[Equation]
E.4 Synthetic Experiments: Convergence
We now analyze convergence of the trapezoidal rule for multimodal continuous attention usingnumerical integration empirically using synthetic experiments.
We define
[Equation]
( t )
[Equation]
I (cid:88) i
[Equation]
( t
[Equation]
t i ) and
[Equation]
( t )
[Equation]
(cid:88)
[Equation]
with γ i , B i ∼ U ( − 1 , 1) .
We set I = 10 and t i to be evenly spaced in the interval [0 , 1] .
We thencompute
[Equation]
which is proportional to attention for kernel exponential families.
We also compute
[Equation]
proportional to the sparsemax case of kernel deformed exponential family attention.
Finally, we alsouse the softplus approximation
[Equation]
We simulate 100 times.
Figure 2a compares the convergence as the number of grid points in thetrapezoidal rule increases.
2b shows a single case of the integral value using deformed exponentialvs the softplus approximation for various values of ρ and numbers of grid points.
We see that forρ = 5 , 10 the integral using softplus approximation is essentially indistinguishable from that usingthe positive part/ReLU.
F uWave Experiment
[Figure]
We investigate: 1) does a large number of unimodal attention heads, as suggested in Martins et al.[2020, 2021], perform well when multimodality is needed? 2) can this method work well forirregularly sampled time series?
3) Can we learn interesting multimodal attention densities.
We analyze uWave [Liu et al., 2009]: accelerometer time series with eight gesture classes.
We follow
Li and Marlin [2016]’s split into 3,582 training observations and 896 test observations: sequenceshave length 945 .
We do synthetic irregular sampling and uniformly sample 10% of the observations.
Because of this our results are comparable to other uWave irregular sampling papers Li and Marlin[2016], Shukla and Marlin [2019], but not to results using the full time series.
Table 4 shows the results.
Our highest accuracy is 94 .
26% , the multi-head unimodal case’s best is74 .
69% , and the mixture’s best is 81 .
Since this dataset is small, we report ± 1 .
96 standarddeviations from 10 runs.
We outperform the results of Li and Marlin [2016], who report a highestaccuracy of 91 .
41% , and perform similarly to Shukla and Marlin [2019] (their figure suggestsapproximately 94% accuracy).
Fig.
1 shows attention densities for one of the attention heads for the
[Table]
Note that due to the irregular sampling, this is only comparable to Li and Marlin [2016], Shukla and
Marlin [2019] attention substantially outperforms and first three classes.
This takes one attention density for each time series in the test set of each classand plots it.
Within the same class, all attention densities for the head (one for each time series) areplotted.
The plot shows two things: firstly, attention densities have support over non-overlappingtime intervals.
This cannot be done with Gaussian mixtures, and the intervals would be the same foreach density in the exponential family case.
Secondly, there is high similarity of attention densitieswithin each class, but low similarity between classes.
Appendix F describes additional details
[Table] [Table]
We experiment with N = 64 , 128 basis functions, and use a learning rate of 1 e − 4 .
We use H =
100attention mechanisms, or heads.
Unlike Vaswani et al.
[2017], our use of multiple heads is slightlydifferent as we use the same value function for each head, and only vary the attention densities.
Additional architectural details are given below.
Table 5 summarizes the hypermarameters andtraining details.
F.1 Value Function
The value function uses regularized linear regression on the original time series observed at randomobservation times (which are not dependent on the data) to obtain an approximation V ( t ; B ) =
B Ψ( t ) ≈ X ( t ) .
The H in Eqn.
6 is the original time series.
F.1.1 Encoder
In the encoder, we use the value function to interpolate the irregularly sampled time series at theoriginal points.
This is then passed through a convolutional layer with 4 filters and filter size 5followed by a max pooling layer with pool size 2 .
This is followed by one hidden layer with 256units and an output v of size 256 .
The attention densities for each head h = 1 , · · · , H are then
[Equation]
for vectors w h, 1 , w h, 2 and matrices W h and heads h = 1 ,
· · · , H
F.1.2 Attention Mechanism
After forming densities and normalizing, we have densities p 1 ( t ) , · · · , p H ( t ) , which we use tocompute context scalars
[Equation]
T )]
We compute these expectations using numerical integration to compute basis function expectations 898 E p h [ ψ n ( T )] and a parametrized value function V ( t ) =
Bψ ( t ) as described in section 3 .
F.1.3 Classifier
The classifier takes as input the concatenated context scalars as a vector.
A linear layer is thenfollowed by a softmax activation to output class probabilities.
G IMBD Experiments
We extend Martins et al.
[2020]’s code 5 for the IMDB sentiment classification dataset [Maas et al.
This starts with a document representation v computed from a convolutional neural networkand uses an LSTM attention model.
We use a Gaussian base density and kernel, and divide theinterval [0 , 1] into I = 10 inducing points where we evaluate the kernel in f ( t ) =
(cid:80) Ii =1 γ i k ( t, t i ) .
Table 6 shows results.
On average, kernel exponential and deformed exponential family slightlyoutperforms the continuous softmax and sparsemax.
The continuous softmax/sparsemax results arefrom running their code.
H MIT BIH: Additional Details
We plot attention densities for randomly selected examples under the different density classes, andfind that only kernel deformed exponential families/sparsemax learn interpretable attention densities,which focus on regions where the electrical signals from the heart are strong.
Figures 5 and 6 showattention densities vs original signals for continuous softmax and sparsemax, respectively.
Both areonly able to learn simple unimodal densities.
Figure 7 shows the same for Gaussian mixture attention.
[Table] [Table]
These do not look very multimodal, despite having one component per time point.
This is likely dueto lack of separation between components.
However, we do see that the shapes look more flexiblethan in single Gaussian or truncated parabola cases.
Figure 8 shows attention densities vs originalsignals for kernel softmax attention.
While not particularly interpretable, it learns densities similarto exponential densities without them being specified, a benefit of being a non-parametric density.
Figure 9 shows the same for kernel sparsemax.
These show interesting highlighting of waves, whichdescribe electrical signals passing through the heart conduction system.
There is a particular focus onthe R wave, the largest peak in a heartbeat, representing electrical stimulus in the main ventricularmass.
Note that given the relatively complex model structure and reasonably high capacity relative to theoriginal time series length (512 LSTM hidden units per time step for a univariate time series of length187), models without an interpretable attention density may still perform well by taking advantageof capacity elsewhere in the model.
However, a model that selects ECG waves may be useful inconvincing specialists of its value.
H.1 General Architecture
Our general architecture is shown in 4.
In the first part, two convolutional layers of filter size 5 and 24filters with padding map the original univariate time series from R 187 to a multivariate representation 933 R 187 × 24 .
This is then passed to an LSTM.
Context vectors are computed using either the originalhidden states (discrete attention) or a continous-time representation (continuous attention).
Thecontext vector is then fed into a feedforward network for final classification.
The entire architectureis trained end-to-end.
Hyperparameters are described in Table 7.
H.2 Value Function
The value function uses regularized linear regression on the hidden states of an LSTM to obtain anapproximation V ( t ; B ) =
B Ψ( t ) ≈ h t .
The H in Eqn.
6 is the set of all hidden states.
H.3 Encoder
The encoder takes the hidden layer of the LSTM as input.
For the hidden units h t ∈ R p for a giventime step, it computes
[Equation]
Note that this is written slightly differently from the form in the main paper.
H.4 Attention Mechanism
The attention mechanism takes the parameters from the encoder and forms an attention density.
Itthen computes
[Equation]
for input to the classifier.
H.5 Classifier
The classifier has three feedforward layers, where the first two have ReLU activation functions.
I FordA: Additional Details
We again plot attention densities for randomly selected examples under each density class.
We findthat only the kernel methods appear truly multimodal visually (Gaussian mixture does not), and thekernel sparsemax case actually highlights specific peaks and troughs of the original signal, suggesting a focus which
[Figure]
the ability to attend to higher frequencies.
Figure 13 shows rescaled kernel softmax attention densitiesalong with the original series.
These are generally smooth and show rich multimodality.
Figure 14shows the same for kernel sparsemax.
These are very interesting, and learn rich sparsity patterns andoften highlight the peaks and troughs of the series, while taking 0 values over many regions wherethe series becomes negative.
This dataset uses the same architecture as the MIT BIH dataset.
However, several hyperaparametersare different, as mentioned in Table 8.
In particular, we use more value basis functions (256) as thesequence is longer, and a smaller number of LSTM hidden units (128).
On this dataset, claiminginterpretability is more difficult because it is engine noise while the class meanings are unknown:only the binary labels are present.
However it is still interesting that kernel sparsemax is able to selectindividual waves and exhibit rich sparsity patterns.