Kernel Multimodal Continuous Attention
Abstract
Attention mechanisms average a data representation with respect to probability weights.
Recently, [23–25] proposed continuous attention, focusing on unimodal exponential and deformed exponential family attention densities: the latter can have sparse support.
[8] extended to multimodality via Gaussian mixture attention densities.
In this paper, we propose using kernel exponential families [4] and our new sparse counterpart, kernel deformed exponential families.
Theoretically, we show new existence results for both families, and approximation capabilities for the deformed case.
Lacking closed form expressions for the context vector, we use numerical integration: we prove exponential convergence for both families.
Experiments show that kernel continuous attention often outperforms unimodal continuous attention, and the sparse variant tends to highlight time series peaks
Introduction
Attention mechanisms [3] are weighted averages of data representations used to make predictions.
Discrete attention 1) cannot easily handle irregularly spaced observations, and 2) attention maps may be scattered, lacking focus.
[23, 24] proposed continuous attention, showing that attention densities maximize the regularized expectation of a function of the data location (i.e. time).
Special cases lead to exponential and deformed exponential families: the latter has sparse support.
They form a continuous data representation and take expectations with respect to attention densities.
In [25] they apply this to a transformer architecture.
[23–25] used unimodal attention densities, giving importance to one data region.
[8] extended this to multimodal Gaussian mixture attention densities.
However 1) Gaussian mixtures lie in neither the exponential nor deformed exponential families, and are difficult to study in the context of [23, 24]; and 2) they have dense support.
Sparse support can say that certain regions of data do not matter: a region of time has no effect on class probabilities, or a region of an image is not some object.
We would like to use multimodal exponential and deformed exponential family attention densities, and understand how [8] relates to the framework of [23, 24].
This paper makes three contributions: 1) we introduce kernel deformed exponential families, a sparse multimodal density class, and apply it along with the multimodal kernel exponential families [4] as attention densities.
The latter have been used for density estimation, but not weighting data importance; 2) we theoretically analyze kernel exponential and deformed exponential family i) normalization, ii) approximation and iii) context vector numerical integration properties; 3) we apply them to real world datasets, showing that multimodal continuous attention outperforms unimodal, and that kernel deformed exponential family densities often highlight the peaks of time series.
Approximation properties for the kernel deformed case are challenging: similar kernel exponential family results [33] relied on exponential and logarithm properties to bound the difference of the log-partition functional at two functions: these do not hold for deformed analogues.
We provide similar bounds by using a mean value inequality and bounding a functional derivative.
We first review unimodal continuous attention [23, 24].
We motivate multimodal continuous attention via time warping.
We next describe kernel exponential families and give a novel normalization condition relating kernel growth to the base density’s tail decay.
We then propose kernel deformed exponential families, new densities with support over potentially disjoint regions.
We describe normalization and approximation capabilities.
Next we describe using these densities for continuous attention, including numerical integration convergence analysis.
We show experiments comparing unimodal and multimodal attention, and conclude with limitations and future work.
Related Work
Attention Mechanisms closely related are [23–25, 8].
[23, 24] frame continuous attention as an expectation of a value function with respect to a density, where the density solves an optimization problem.
They only used unimodal (deformed) exponential family densities: we extend this to the multimodal setting by leveraging kernel exponential families and proposing a deformed counterpart.
[8] proposed a multimodal continuous attention mechanism via a Gaussian mixture.
We show in
Appendix A that this solves a slightly different optimization problem from [23, 24].
A limitation of Gaussian mixtures is lack of flexible tail decay.
Finally, [25] apply continuous attention within a transformer architecture to model long context.
This is a new application of continuous attention rather than an extension of specific continuous attention mechanisms.
Also relevant are [40, 30, 31].
[30] provide an attention mechanism for irregularly sampled time series by use of a continuous-time kernel regression framework, but do not take an expectation of a data representation over time with respect to a continuous pdf.
Instead they evaluate the kernel regression model at fixed time points.
This describes importance of data at a set of points rather than over continuous regions.
[31] extend this to incorporate uncertainty quantification.
Other papers connect attention and kernels, but focus on discrete attention [40, 5].
Also relevant are temporal transformer papers, including [45, 15, 17, 32].
However, none have continuous attention densities.
Kernel Exponential Families [4] proposed kernel exponential families: [33] analyzed theory for density estimation.
[44] parametrized the kernel with a deep neural network.
Other density estimation papers include [1, 6, 36].
We apply kernel exponential families as attention densities to weight a value function which represents the data, rather than for density estimation.
Further, [44] showed a condition for an unnormalized kernel exponential family density to have a finite normalizer.
However, they used exponential power base densities.
We instead relate kernel growth rates to the base density tail decay, allowing non-symmetric base densities.
To summarize our theoretical contributions: 1) showing that multimodal continuous attention is required to represent time warping 2) introducing kernel deformed exponential families with approximation and normalization analysis 3) improved kernel exponential family normalization results 4) stability and convergence analysis of numerical integration for kernel-based continuous attention 5) characterizing [8] in terms of the framework of [23, 24].
Continuous Attention Mechanisms
An attention mechanism has: 1) a value function: a raw or learned data representation 2) an attention density chosen to be ‘similar’ to another data representation, encoding it into a density 3) a context c [23] taking an expectation of the value function with respect to the attention density:
[Equation]
The value function V : S → R D is a data representation, T ∼ p ( t ) is the random variable or vector for locations (temporal, spatial, etc) in domain S , and p ( t ) is the attention density (potentially with respect to a discrete measure).
For discrete attention, one could have V ( t ) be a time series where t ∈ S a finite set of time points.
One then weights the time series with a probability vector to obtain the context vector.
An example of this for irregularly sampled time series is [30].
For each attention mechanism (eqn. 3 in their paper), S is the set of observed time points, the value function maps the observed time points to their respective observation values, and the attention probability mass function is the output of normalized kernel evaluations between observation time points and a reference time point.
In the continuous setting V ( t ) could be a curve or realization of a continuous-time stochastic process, and S could be [0 , τ ] where τ is a study end time.
One then weights it with a continuous probability density.
If S is a set of spatial locations and one has image data, then the value function could be a raw image or learned representation of an image.
Finally, for self-attention [18],[42], V is a linear transformation of a sequence, and the expectation is conditional on a transformation of a specific token (query).
To choose p , one takes a data representation f and finds p ‘similar’ to f , but regularizing p .
[23, 24] did this, formalizing attention mechanisms.
Given a measure space ( S, A , Q ) , let M 1+ ( S ) be the set of probability densities with respect to Q .
Assume Q is dominated by a σ -finite measure ν (i.e.
Lebesgue) and that it has Radon Nikodym derivative q
= dQdν with respect to ν .
Let S ⊆ R D , F be a function class, and Ω : M 1+ ( S )
→ R be a lower semi-continuous, proper, strictly convex functional.
Given f ∈ F , an attention density [23] ˆ p : F → R ≥ 0 solves
[Equation]
This maximizes regularized L 2 similarity between p and a data representation f .
If Ω( p ) =
(cid:82)
S p ( t ) log p ( t ) dQ ( t ) is the negative differential entropy, the attention density is Boltzmann Gibbs
[Equation]
where A ( f ) ensures (cid:82) S ˆ p [ f ]( t ) dQ = 1 (see [23] for proof).
If f ( t ) =
θ T ϕ ( t ) for parameters and statistics θ ∈ R M , ϕ ( t )
∈ R M respectively, Eqn. 3 becomes an exponential family density.
For f in a reproducing kernel Hilbert space (RKHS) H , it becomes a kernel exponential family density [4], which we propose as an alternative attention density.
One desirable class would be heavy or thin tailed exponential family like densities.
In exponential families, the support, or non-zero region of the density, is controlled by the measure Q .
Letting Ω( p ) be the α -Tsallis negative entropy Ω α ( p ) [41],
[Equation]
then ˆ p [ f ] for f ( t ) =
θ T ϕ ( t ) lies in the deformed exponential family [41, 27]
[Equation]
where A α ( f ) again ensures normalization and the density uses the β -exponential
[Equation]
For β < 1 , Eqn. 5 and thus deformed exponential family densities for 1 < α ≤ 2 can return 0 values.
Values α > 1 (and thus β < 1 ) give thinner tails than the exponential family, while α < 1 gives fatter tails.
Setting β
= 0 is called sparsemax [22].
In this paper, we assume 1 < α ≤ 2 , which is the sparse case studied in [23].
We again propose to replace f ( t )
= θ T ϕ ( t ) with f ∈ H , which leads to the novel kernel deformed exponential families .
Computing Eqn.
1’s context vector requires parametrizing V ( t ) .
[23] parametrize V : S → R D with B ∈ R D × N as V ( t ; B ) =
B Ψ( t ) and estimate B ∈ R D × N via regularized multivariate linear regression.
Here Ψ = { ψ n } Nn =1 is a set of basis functions.
Let L be the number of observation locations (times in a temporal setting), D be the observation dimension, and N be the number of basis functions.
Using Frobenius norm ∥ · ∥ F , this involves regressing the observation matrix H ∈ R D
× L on a matrix F ∈ R N × L of basis functions { ψ n } Nn =1 evaluated at observation locations { t
[Equation] [Equation]
Time Warping
We now draw a connection to time warping [28] to show an advantage of our method.
One summary statistic for classification is a global weighted average pooling: an expectation of temporal features.
However in many processes features may not be aligned in time, and we only observe unaligned curves due to a discrepancy between individual system times and clock time: this is known as phase variation [21].
For instance, electrocardiogram (ECG) heartbeat curves have a P-wave, a QRS complex and a T-wave.
These have similar patterns between heartbeats, but may have different durations and peak locations.
Here we show that the expectation of a temporally aligned curve with respect to a global density is equivalent to the expectation of the unaligned curve with respect to an individualized density.
However even if the global density is unimodal, the individualized density may not be.
We first define the function that aligns a set of features to common reference times.
Definition 4.1.
(Time Warping Function)
Given references times { t 0 k } Kk =1 and individualized times { t ik } Kk =1 , both in [0 , τ ] , a time warping function h
i : S → R for S
⊆
R ≥ 0 is a strictly increasing, differentiable, invertible function where
[Equation]
Let { X i } ni =1 , X i : S → R be observed curves, each with K features occurring at individualized times { t ik } Kk =1 ⊂ [0 , τ ] increasing in k .
A set of time warping functions { h i } ni =1 map reference times to individualized feature times.
One can then compute aligned X ∗ i ( t )
=
X i ( h i ( t )) .
Each
X ∗ i has relevant features at the same times { t 0 k } Kk =1 .
Classically, this requires handcrafting and locating important features and estimating a warping function.
We could then compute an expectation of the time warped curve with respect to a global fixed density p ( t ) to obtain a summary statistic E
T ∼ p X ∗
i ( T ) of the aligned curve.
The following states that multimodal continuous attention can represent such an expectation with an attention density p i , avoiding computing X ∗ i ( t ) .
Lemma 4.2.
(Continuous Attention can Represent Time Warping)
Let h be a time warping function, g = h − 1 and X i : R → R with support on [0 , τ ] .
Assume that Q is dominated by Lebesgue measure ν and let q 0 = dQdν .
Then for any fixed density p wrt Q , if g, X i , q 0 , p are continuous almost everywhere we have
[Equation]
where p i ( t ) = p ( g i ( t ))
q
( g i ( t ))
q 0 ( t ) g ′ i ( t ) and p i ( t ) is a valid probability density function.
See Appendix B.1 for proof.
Even if p ( t ) is unimodal, p i ( t ) may not be: see Appendix B.2 for an example.
Thus we require multimodal continuous attention to represent such statistics.
Kernel Exponential and Deformed Exponential Families
We use kernel exponential families and a new deformed counterpart to obtain flexible attention densities solving Eqn.
2 with the same regularizers.
We first review kernel exponential families.
We then give a novel theoretical result describing when an unnormalized kernel exponential family density can be normalized.
This says that the normalizing constant exists when the base density has fast enough tail decay relative to kernel growth.
This result allows us to verify that a choice of base measure and kernel lead to a valid attention density and thus attention mechanism.
Next we introduce kernel deformed exponential families, extending kernel exponential families to have either sparse support, our focus, or fatter tails.
These can attend to non-overlapping time intervals.
We show similar normalization results based on kernel choice and base density.
The normalizing constant exists when the unnormalized density has compact support and the kernel grows sufficiently slowly.
Following this we show approximation theory.
We conclude by showing how to compute attention densities in practice.
Kernel exponential families [4] extend exponential families, replacing f ( t )
= θ T ϕ ( t ) with f ∈ H a reproducing kernel Hilbert space H [2].
Densities can be written
[Equation]
Eqn. 8 follows from the reproducing property.
A challenge is to choose H , Q so that a normalizing constant exists, i.e., (cid:82) S exp( f ( t )) dQ < ∞ .
These densities can approximate any continuous density over a compact domain arbitrarily well in KL divergence, Hellinger, and L p distance [33].
However relevant integrals including the normalizing constant require numerical integration.
To avoid infinite dimensionality one generally assumes a representation of the form f = (cid:80)
Ii =1 γ i k ( · , t i ) , where for density estimation [33] the t i are the observation locations and this is the solution to a regularized empirical risk minimization problem.
This requires using one parameter per observation value.
This model complexity may not be necessary, and often one chooses a set of inducing points [38] { t i } Ii =1 where I is less than the number of observation locations.
For a given pair H , k , how can we choose Q to ensure that the normalization constant exists?
We first give a simple example of H , f and Q where it does not .
Example 1 .
Let Q be the law of a N (0 , 1) distribution and S = R .
Let H = span { t 3 , t 4 } with k ( t, s )
=
t 3 s 3 + t 4 s 4 and f ( t ) =
t 3 + t 4 =
k ( t, 1) .
Then the following integral diverges.
[Equation]
5.1 Theory for Kernel Exponential Families
We provide sufficient conditions for Q and H so that A ( f ) the log-partition functional exists.
We relate H ’s kernel growth rate to the tail decay of the random variable or vector T Q with law Q .
Proposition 5.1.
Let ˜ p
( t ) =
exp( f ( t )) where f ∈ H an RKHS with kernel k .
Assume k
( t, t ) ≤
L k ∥ t ∥ ξ 2 + C k for constants L k , C k , ξ > 0 .
Let Q be the law of a random vector T Q , so that
Q ( A ) =
P ( T Q ∈ A ) .
Assume ∀ u s.t.
[Equation]
u
[Equation]
2 = 1 , z
[Equation] [Equation]
for some constants η > ξ 2 , C Q , v > 0 .
Then
[Equation]
See Appendix C.1 for proof.
Based on k ( t, t ) ’s growth, we can vary what tail decay rate for T Q ensures we can normalize ˜ p ( t ) .
[44] also proved normalization conditions, but focused on exponential power density for a specific growth rate of k ( t, t ) rather than relating tail decay to growth rate.
By focusing on tail decay, our result can be applied to non-symmetric base densities.
Specific kernel bound growth rate terms ξ lead to allowing different tail decay rates.
Corollary 5.2.
For ξ = 4 , T Q can be any sub-Gaussian random vector.
For ξ = 2 it can be any sub-exponential.
For ξ = 0 it can be any probability density.
See Appendix C.2 for proof.
5.2 Kernel Deformed Exponential Families
We now propose kernel deformed exponential families, which are flexible sparse non-parametric densities: these can be multimodal.
They take deformed exponential families and extend them to use kernels in the deformed exponential term.
This mirrors kernel exponential families.
We write
[Equation]
where f ∈ H with kernel k .
Fig.
1 shows that they can have support over disjoint intervals.
5.2.1 Normalization Theory
We construct a valid kernel deformed exponential family density from Q and f ∈ H .
We first discuss the deformed log-normalizer.
In exponential family densities, the log-normalizer is the log of the normalizer.
For deformed exponentials, the following holds.
Lemma 5.3.
Let Z > 0 be a constant.
Then for 1 < α ≤ 2 ,
[Equation]
where
[Equation]
See Appendix D.1 for proof.
We now describe a normalization sufficient condition analagous to
Proposition 5.1 for deformed kernel exponential families.
With Lemma 5.3 and an unnormalized exp 2 − α ( ˜ f ( t ))
we derive a valid normalized kernel deformed exponential family density.
We only require that an affine function of the terms in the deformed-exponential is negative for large ∥ t ∥ 2 .
Proposition 5.4.
For 1 < α ≤ 2 assume ˜ p
( t ) =
exp 2 − α ( ˜ f ( t )) with ˜ f ∈ H , H is a RKHS with kernel k .
If ∃ C t > 0 s.t. for ∥ t ∥ 2 > C t , ( α − 1) ˜ f
( t ) + 1 ≤ 0 and k ( t, t ) ≤
L k ∥ t ∥ ξ 2 + C k for some ξ > 0 , then
[Equation]
See Appendix D.2 for proof.
We now construct a valid kernel deformed exponential family density using the finite integral.
Corollary 5.5.
Under proposition 5.4’s conditions, assume exp 2 − α
( ˜ f ( t )) > 0 on a set A ⊆ S where
Q ( A ) > 0 , then ∃ constants Z > 0 ,
A α ( f ) ∈ R such that for f ( t ) =
[Equation]
, the following holds
[Equation]
See Appendix D.3 for proof.
We thus estimate ˜ f
( t ) =
Z α − 1 f ( t ) and normalize to obtain a density of the desired form.
5.2.2 Approximation Theory
Kernel deformed exponential families can approximate continuous densities satisfying a tail condition on compact domains arbitrarily well in L p norm, Hellinger distance, and Bregman divergence.
Theorem 5.6.
Let q 0 ∈ C ( S ) , such that q 0 ( t ) > 0 for all t ∈ S , where S ⊆ R d is locally compact
Hausdorff and q 0 is the Radon Nikodym derivative of measure Q with respect to a dominating measure ν .
Suppose there exists l > 0 such that for any ϵ > 0 , ∃ R > 0 satisfying | p ( t ) − l | ≤ ϵ for any t with ∥ t ∥ 2 > R .
Define
[Equation]
Suppose k ( t, · ) ∈ C 0 ( S ) ∀ t ∈ S and the kernel integration condition (Eqn. 12) holds.
Then kernel deformed exponential families are dense in P c wrt L r norm, Hellinger distance and Bregman divergence for the α -Tsallis negative entropy functional.
The proof (Appendix D.4) idea is that under a kernel integrability condition, deformed exponential families parametrized by f ∈ H are dense in those parametrized by f ∈ C 0 ( S ) 2 (we denote those parametrized by f ∈ C 0 ( S ) as P 0 ).
We can approximate C ( S ) densities satisfying the tail condition with P 0 densities, and thus with deformed exponential family densities.
This extends [33]’s approximation to the deformed case: standard log and exponential rules cannot be applied.
It requires bounding functional derivatives and applying a mean value inequality.
5.3 Using Kernels for Continuous Attention
Here we describe how to compute continuous attention mechanisms with attention densities parametrized by functions in an RKHS H in practice.
Algorithm 1 shows the kernel deformed exponential family case: the kernel exponential family case involves a similar algorithm.
Given a base measure, kernel, and inducing point locations, we start by computing kernel weights ˜ γ
i for ˜ f ( t ) =
Z α − 1 f ( t ) =
(cid:80) Ii =1 ˜ γ
i k ( t, t i ) and estimating the matrix B for basis weights for the value function V ( t ) =
B Ψ( t ) .
Unlike density estimation, this form for f is simply a practical way to obtain f in an RKHS, rather than a solution to an empirical risk minimization problem.
We then compute the normalizing constant Z = (cid:82) S exp 2 − α
( ˜ f ( t )) dQ via numerical integration and use it to normalize ˜ f ( t ) to obtain the attention density p ( t ) .
Finally we compute the context c =
E T ∼ p
[ V ( T )]
=
B E p [Ψ( t )] by taking the expectation of Ψ( T ) with respect to a deformed kernel exponential family density p .
Unlike [23, 24], we lack closed form expressions and use numerical integration.
In the backwards pass we use automatic differentiation.
Note that in some cases we have numerical underflow when computing the normalizing constant.
We also found that using FP16 precision, the default on some newer GPUs for Pytorch up to 1.11, leads to worse performance, sometimes dramatically so, and thus recommend using FP32.
We discuss solutions for the underflow issue in Appendix E.1.
Algorithm
Choose q 0 ( t ) and kernel k .
Inducing point locations { t i } Ii =
Parameters { ˜ γ i } Ii =1 the weights for ˜ f ( t ) =
( Z ) α − 1 f ( t ) =
(cid:80) Ii =1 ˜ γ
i k ( t, t i ) , matrix B for basis weights for value function V ( t ) =
B Ψ( t ) .
I is number of inducing points.
Forward Pass
Compute Z = (cid:82) exp 2 − α
( ˜ f ( t ))
dQ ( t ) to obtain p ( t ) =
1 Z exp 2 − α ( ˜ f ( t )) via numerical integration
Compute E T ∼ p
[Ψ( T )] via numerical integration
Compute c = E T ∼ p
[ V ( T )]
=
B E p [Ψ( T )]
Backwards Pass use automatic differentiation
5.3.1 Numerical Integration Convergence
The trapezoidal rule’s standard one-dimensional convergence rate is O ( 1 N 2 ) for an integral over a fixed interval, where N is the number of grid points.
We would like better convergence guarantees.
We can achieve exponential convergence for the numerical integrals of kernel exponential and deformed exponential family attention.
We focus on numerical integration over the real line, leaving truncation analysis and higher dimensions to future work.
We let h > 0 be the grid size.
For functions holomorphic in a strip with rapid decay, the trapezoidal rule has exponential convergence.
For kernel exponential family attention, this gives us O (exp( − C/h )) or exponential convergence for some C > 0 with appropriate choice of q 0 , V , and k .
Technical details are in E.2, and are based on extending real-valued analytic functions to complex functions analytic/holomorphic on a strip.
Kernel deformed exponential families, however, are not even differentiable, but we can construct a sequence of differentiable approximations by replacing the positive part/ReLU function in the deformed exponential with softplus for increasing values of the softplus parameter.
Each differentiable approximation has exponential convergence, and by taking limits as the softplus parameter tends to infinity we can show that the numerical integral for kernel deformed exponential family attention itself has exponential convergence.
Technical details are in E.3.
We also show empirical convergence analysis in Appendix E.4 and figure 2 in that appendix.
Both kernel exponential and deformed exponential families see rapid convergence for 1d attention, pro-
[Table]
Experiments
We investigate how often multimodal continuous attention outperforms unimodal, given the same architecture.
We also investigate whether these methods learn rich multimodal densties.
We denote kernel exponential family attention as kernel softmax and the deformed case as kernel sparsemax.
Our architectures have: 1) an encoder maps a discrete time series representation to attention density parameters.
The value function V ( t ; B ) expresses an embedding of a time series as a linear transformation of basis functions.
The context is c = E p [ V ( T )] , which is used in 4) a classifier.
Fig.
3 in the Appendices visualizes this.
For kernel softmax/sparsemax the encoder outputs are the weights for the kernel evaluations.
For the Gaussian mixture case, the encoder’s outputs are the mixture weights and components means and variances.
Here we describe one synthetic and three real data experiments.
In Appendix E we provide an additional ECG classification experiment.
We provide confidence intervals for our three smaller experiments (at most 10 , 000 samples).
Our UWave and ECG experiments were done on a Titan X GPU, IMDB on a 1080, and FordA on an A40.
We found that the A40 provides very different results out of the box for both accuracy/F1 and attention densities.
As an example, Figure 8 was done on a Titan X with an older version of Pytorch, while
Figure 7 was done with an A40 with Pytorch 1.12.
Code is in our repository 3 , where we discuss the flags used to control precision on recent GPUs and Pytorch versions.
We discuss computational/ memory complexity in Appendix F, and give a summary in Table 5.
We provide wall clock times in
Table 6.
6.1 Synthetic Experiment: Time Warping
We simulate time warping and do prediction for unimodal vs multimodal attention densities.
Details of the original time aligned stochastic process and inverse warping function are in Appendix B.3.
Given global densities for classes p 1 , p 2 and aligned X ∗ , the class is arg max( ⟨ p 1 , X ∗ ⟩ , ⟨ p 2 , X ∗ ⟩ ) .
We generate 10 , 000 trajectories of length 95 observed at evenly spaced time points in the interval [0 , 1] .
We use two attention mechanisms (heads), one for each class.
Letting V be the value function fit to observed data, the classifier is softmax ([ ⟨ p i 1 , V ⟩ L 2 , ⟨ p i 2 V ⟩ L 2 ])
where p i 1 and p i 2 are the attention densities for the first and second class.
Table 1 shows prediction results from 100 runs along with 1 .
96 standard deviation intervals.
Kernel methods outperform by 7 − 10% .
6.2 FordA Dataset
This is a binary classification dataset for whether a sympton exists in an automotive subsystem.
Each time series has 500 sensor observations, and there are 3601 training samples and 1320 test samples.
Table 2 shows results.
Kernel sparsemax outperforms most baselines 4 , while kernel softmax also does well.
We use the same methods and architecture as the previous section, although hyperparameters are slightly different.
Several methods, including discrete softmax, continuous sparsemax and the transformer, have very poor performance.
To sanity check we fit SVM with a Gaussian kernel, logistic regression, and a decision tree: accuracies are under 55% for these, and we conclude that
[Table] [Table]
this data poses difficulty for some classifiers.
Appendix G provides some additional details, along with attention density plots.
The kernel softmax plots often highlight zero crossings.
The kernel sparsemax plots often select peaks of a signal while learning rich sparsity patterns.
6.3 uWave Experiment: Gesture Classification
We investigate: 1) Does a large number of unimodal attention heads, as suggested in [23, 24], perform well when multimodality is needed? 2) Can this method work well for irregularly sampled time series? 3) Can we learn interesting multimodal attention densities?
We analyze uWave [19]: accelerometer time series with eight gesture classes.
We follow [16]’s split into 3,582 training observations and 896 test observations: sequences have length 945 .
We do synthetic irregular sampling and uniformly sample 10% of the observations.
Because of this our results are comparable to other uWave irregular sampling papers [16, 29], but not to results using the full time series.
Table 3 shows the results.
Our highest accuracy is 94 .
26% , the multi-head unimodal case’s best is 74 .
69% , and the mixture’s best is 81 .
We outperform the results of [16], who report a highest accuracy of 91 .
41% , and perform similarly to [29] (their figure suggests approximately 94% accuracy).
Fig.
10 shows attention densities for one of the attention heads for the first three classes.
This takes one attention density for each time series of each class and plots it.
Within the same class, all attention densities for the head (one for each time series) are plotted.
The plot shows two things: firstly, attention densities have support over non-overlapping time intervals.
This cannot be done with
Gaussian mixtures, and the intervals would be the same for each density in the exponential family case.
Secondly, there is high similarity of attention densities within each class, but low similarity between classes.
Appendix H describes additional details.
[Table]
6.4 IMBD Sentiment Classification
We extend [23]’s code 5 for IMDB sentiment classification [20].
This uses a document representation v from a convolutional network and an LSTM attention model.
We use a Gaussian base density and kernel, and divide the interval [0 , 1] into I =
10 inducing points where we evaluate the kernel in f ( t ) =
(cid:80) I i =
1 γ i k ( t, t i ) .
Table 4 shows results.
On average, kernel exponential and deformed exponential family slightly outperforms the continuous softmax and sparsemax.
The continuous softmax/sparsemax results are from running their code.
Discussion
In this paper we extend continuous attention mechanisms to use kernel exponential and deformed exponential family densities.
The latter is a new flexible class of non-parametric densities with sparse support.
We show novel existence properties for kernel exponential and deformed exponential families, prove approximation properties for the latter, and show exponential convergence of numerical integration for both attention mechanisms.
We then apply these to the continuous attention framework described in [23, 24].
We show results on several datasets.
Kernel attention mechanisms tend to outperform unimodal attention, sometimes by a large margin.
We also see that in many cases they exhibit multimodality.
Kernel sparsemax in particular learns rich sparsity patterns while highlighting peaks of a signal.
This was evident in Figure 1.
The engine noise example mimicked certain parts of the signal, while in the ECG example, it tends to give very high weight to R waves in a signal.
While this paper is more focused on general methods than a specific potentially dangerous application, potential application areas include wearable sensors and NLP, and general negative societal impacts in those application areas could apply to this work.
7.1 Limitations
A limitation was the use of numerical integration, which scales poorly with location dimensionality.
While we achieve 1D exponential convergence, we still must investigate whether we can extend this to higher dimensions.
This still allows for multiple observation dimensions at a given 1d location, i.e. multivariate time series or language tasks.
Funding and Other Acknowledgments
Alexander Moreno is supported by Luminous Computing, and was previously supported by NIH 1-P41-EB028242-01A1, U01CA229437, and a Google CMD-IT Flip-Alliance fellowship.
Zhenke
Wu is partly supported by NIH U01CA229437 and an investigator grant from Precision Health
Initiative at University of Michigan, Ann Arbor.
Supriya Nagesh is supported by NSF CNS1823201.
Walter Dempsey is supported by P50 DA054039.
We thank Marcos Treviso and André Martins for helpful discussions about their prior work and help with understanding their code.
Table
of
Contents A Gaussian Mixture Model
B Time Warping
B.1 Proof of 4.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.2 Multimodal Densities are Required . . . . . . . . . . . . . . . . . . . . . . . .
B.3 Time Warping Experimental Details . . .
C Proof Related to Proposition 5.
C.1 Proof of Proposition 5.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.2 Proof of Corollary 5.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D Proofs Related to Kernel Deformed
Exponential
Family
D.1 Proof of Lemma 5.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.2 Proof of Proposition 5.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.3 Proof of Corollary 5.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.4 Approximation Theory . .
E Numerical Integration Stability and Convergence Analysis
E.
1 Stable
Numerical Integration
E
Kernel Exponential Family Attention . . . . . . . . . . . . . . . . . . . . . . .
E
.3 Convergence
of Numerical Integration
for
[Table] [Table]
.4 Synthetic Experiments: Convergence
[Table]
Computational/Memory Complexity
G FordA
[Table]
Additional
[Table]
H Additional uWave
[Table]
H.1 Value Function
I ECG Heartbeat
Classification: MIT BIH
[Table]
Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
[Table]
[Table] [Table]
Attention Mechanism
[Table]
Classifier
Overview of Appendices
Appendices A to E are mostly theoretical.
In Appendix A we describe how the Gaussian mixture attention from [8] relates to the theoretical framework of [23, 24].
In Appendix B we discuss time warping, showing that sufficiently flexible continuous attention can represent time warping, that in at least in one case a multimodal density is required, and give some details describing our experiments.
In Appendix C we discuss normalization for kernel exponential families, while in Appendix D we do the same along with approximation results for kernel deformed exponential families.
In Appendix E we discuss rate of convergence for approximating the attention mechanism with numerical integration, and show some synthetic experiments showing rapid convergence empirically.
The remaining appendices are empirical.
In Appendix H, we include an additional experiment analyzing an accelerometer classification dataset, where we add synthetic irregular sampling.
In
Appendix I we provide additional details for the MIT-BIH experiment.
Finally, in Appendix G we provide additional details for the FordA experiment.
The latter two appendices in particular focus on visual comparisons of attention densities.
We find that kernel deformed exponential families/sparsemax tend to learn more interesting looking attention densities than all of the other methods, and for ECG kernel sparsemax has the interpretation of selecting regions where the electrical stimuli are largest.
The licenses are: for MIT BIH open data commons attribution.
For UCR time series (uWave and
Ford) the exact license is difficult to find but the citation is [7].
For IMDB it is again difficult to find the exact license, but the original paper is [20] and the website 6 asks that it be cited.
A Gaussian Mixture Model
In this section we descibe how the Gaussian mixture attention of [8] relates to the optimization definition of attention densities in [23, 24].
In fact their attention densities solve a related but different optimization problem.
[23, 24] show that exponential family attention densities maximize a regularized linear predictor of the expected sufficient statistics of locations.
In contrast, [8] find a joint density over locations and latent states, and maximize a regularized linear predictor of the expected joint sufficient statistics.
They then take the marginal location densities to be the attention densities.
Let Ω( p ) be Shannon entropy and consider two optimization problems:
[Equation]
The first is Eqn. 2 with f = θ T ϕ ( t ) and rewritten to emphasize expected sufficient statistics.
If one solves the second with variables Z , we recover an Exponential family joint density
[Equation]
This encourages the joint density of T, Z to be similar to a complete data representation θ T ϕ ( t, z ) of both location variables T and latent variables Z , instead of encouraging the density of T to be similar to an observed data representation θ T ϕ ( t ) .
Let S Z be the domain for latent variables Z .
The latter optimization is equivalent to
[Equation]
The constraint terms c m are determined by θ .
Thus, this maximizes the joint entropy of Z and T , subject to constraints on the expected joint sufficient statistics.
To recover EM learned Gaussian mixture densities, one must select ϕ m so that the marginal distribution of T will be a mixture of Gaussians, and relate c m to the EM algorithm used to learn the mixture model parameters.
For the first, assume that Z is a multinomial random variable with a single trial taking | Z | possible values and let ϕ ( t, z )
=
( I ( z = 1) , . . .
, I ( z = | Z | − 1) , I ( z = 1) t, I ( z = 1) t 2 , . .
, I ( z = | Z | ) t, I ( z = | Z | ) t 2 ) .
These are multinomial sufficient statistics with a single trial, followed by the sufficient statistics of | Z | Gaussians multiplied by indicators for each z .
Then p ( T | Z ) will be Gaussian, p ( Z ) will be multinomial, and p ( T ) will be a Gaussian mixture.
For contraints, [8] have
[Equation]
at each EM iteration: p old ( Z | T = t l ) is the previous iteration’s latent state density conditional on the observation location, w l are discrete attention weights, and t l is a discrete attention location.
That
EM has this constraint was shown in [43].
This matches expected joint sufficient statistics to those implied by discrete attention over locations, taking into account the dependence between Z and T given by old model parameters.
An alternative is simply to let θ be the output of a neural network.
While the constraints lack the intuition of Eqn. 10, it avoids the need to run EM.
We focus on this case and use it for our baselines.
B Time Warping B.1 Proof of 4.
Here we prove that continuous attention can represent time warping.
We change measure to Lebesgue, use the equivalence between Lebesgue and Riemann for almost everywhere continuous functions, use integration by substitution, and then change back.
[Equation]
Further, since h is strictly increasing, g is as well so that g ′ i ( t ) > 0 ∀ t ∈ [0 , τ ] .
This gives us that p i ( t ) is non-negative, and we have to show that it integrates to 1 .
[Equation]
B.2 Multimodal Densities are Required
Consider the case where p global ( t ) is an exponential density function and g ( t ) the inverse warping function has the following form
[Equation]
else
[Equation]
t
[Equation]
for C 1 = τ
[Equation]
0 exp( W ( u )) du and some twice diferentiable function W .
The role of C 1 is to ensure that g ( τ ) = τ .
Then
[Equation]
so that
[Equation]
Now note that depending on the sign and magnitude of W ′′ ( t ) , p ′′
i ( t ) may be either positive or negative and thus p
i ( t ) may lack a unique optimum and may be multimodal.
B.3 Time Warping Experimental Details
This describes the aligned vs the observed stochastic process.
The original time aligned stochastic process is given by
[Equation]
where Z k ∼ U ( − 4 , 4) , k = 1 , 2 , 3 , 4 .
We instead observe realizations
X i ( t ) =
X ∗ i ( g ( t )) at fixed time points with the following inverse warp function
[Equation]
The global densities are p 1 ∼ U (0 , 0 . 5) and p 2 ∼ U (0 . 5 , 1) .
C Proof Related to Proposition 5. C.1 Proof of Proposition 5.
Here we prove that if the kernel evaluated twice at the same point grows sufficiently slowly with respect to base density tail decay, then we can normalize.
Proof.
This proof has several parts.
We first bound the RKHS function f and use the general tail bound we assumed to give a tail bound for the one dimensional marginals T Qd of T Q .
Using the RKHS function bound, we then bound the integral of the unnormalized density in terms of expectations with respect to these finite dimensional marginals.
We then express these expectations over finite dimensional marginals as infinite series of integrals.
For each integral within the infinite series, we use the finite dimensional marginal tail bound to bound it, and then use the ratio test to show that the infinite series converges.
This gives us that the original unnormalized density has a finite integral.
We first note, following [44], that the bound on the kernel in the assumption allows us to bound f in terms of two constants and the absolute value of the point at which it is evaluated.
[Equation]
We can write T Q =
( T Q 1 , . . .
, T QD ) .
Let e d be a standard Euclidean basis vector.
Then by the assumption and setting u = e d we have
[Equation]
Letting Q d be the marginal law,
[Equation]
which will be finite if each E exp( C 2 | T Qd | ξ/ 2 ) < ∞ .
Now letting S d be the relevant dimension of S ,
[Equation]
where the inequality follows since S d ⊆ R , exp is a non-negative function and probability measures are monotonic.
We will show that the second sum converges.
Similar techniques can be shown for the first sum.
Note that for j ≥
[Equation]
assumption
Then
[Equation]
Let a j = exp( C 2 | j | ξ/ 2 − vi η ) .
We will use the ratio test to show that the RHS converges.
We have
[Equation]
We want this ratio to be < 1 for large j .
We thus need to select η so that for sufficiently large j , we have
[Equation]
Assume that η > ξ 2 .
Then
[Equation]
Since the RHS is unbounded for η > ξ 2 , we have that Eqn.
11 holds for sufficiently large j .
By the ratio test E
[Equation]
is finite.
Thus putting everything together we have
[Equation]
and ˜ p ( t ) can be normalized.
C.2 Proof of Corollary 5.
Here we prove two special cases of kernel growth rate and tail decay.
Proof.
Let ξ = 4 .
Then η > 2 and
[Equation]
The second case is similar.
For the uniformly bounded kernel,
[Equation]
The first line follows from Cauchy Schwarz and ξ =
D Proofs Related to Kernel Deformed Exponential Family D.1 Proof of Lemma 5.
Here we prove a key equality for calculating normalization of deformed exponential families.
Proof.
The high level idea is to express a term inside the deformed exponential family that becomes 1 /Z once outside.
Recalling the definition of β -exponential in Eqn. 5,
[Equation]
Note that one could potentially also prove this starting with Proposition 1 and B.3 from [23].
However after some investigation we found our approach to be easier.
D.2 Proof of Proposition 5.
Here we prove that if ˜ f ( t ) has compact support and the kernel grows sufficiently slowly, then its deformed exponential has a finite integral.
Proof.
[Equation]
where the 2nd to last line is due to the inequality | f ( t ) | ≤ C
+ C
∥ t ∥ ξ/ 2 from the proof in Appendix
C.1.
D.3 Proof of Corollary 5.
Here we prove that we can normalize deformed exponential families under the previous conditions.
Proof.
From proposition 5.4 and the assumption,
[Equation]
for some Z > 0 .
Then (cid:90)
[Equation]
where the second line follows from lemma 5.3.
Set
A α ( f ) = log α
( Z )
and we are done.
D.4 Approximation Theory
This section proves Theorem 5.6.
We start with a Proposition, which says that under a kernel integration condition, deformed exponential families parametrized by functions in H can approximate similar densities parametrized by functions in C 0 ( S ) arbitrarily well.
Proposition D.1.
Define
[Equation]
where S ⊆ R d . Suppose k ( x, · ) ∈ C 0 ( S ) , ∀ x ∈ S and
[Equation]
here M b ( S ) is the space of bounded measures over S .
Then the set of deformed exponential families is dense in P 0 wrt L r ( Q ) norm and Hellinger distance.
Proof.
The kernel integration condition tells us that H is dense in C 0 ( S ) with respect to L ∞ norm.
This was shown in [34].
For the L r norm, we apply ∥ p f − p g
∥ L r ≤ 2 M exp ∥
f − g ∥ ∞ from Lemma
D.5 with f ∈ C 0 ( S ) , g ∈ H , and f 0 = f .
L 1 convergence implies Hellinger convergence.
We can then use this for our main proof of Theorem 5.6, which we restate here for reference.
Note that our Bregman divergence result is analagous to [33]’s KL divergence result.
KL divergence is
Bregman divergence with the Shannon entropy functional: we show the same for Tsallis entropy, which is maximized given expected sufficient statistics by deformed exponential families [27].
The
Bregman divergence describes how close a density’s uncertainty is to its first order approximation evaluated at another density.
Theorem D.2.
Let q 0 ∈ C ( S ) , such that q 0 ( t ) > 0 for all t ∈ S , where S ⊆ R d is locally compact
Hausdorff and q 0 is the Radon Nikodym derivative of measure Q with respect to a dominating measure ν .
Suppose there exists l > 0 such that for any ϵ > 0 , ∃ R > 0 satisfying | p ( t ) − l | ≤ ϵ for any t with ∥ t ∥ 2 > R .
Define
[Equation]
Suppose k ( t, · ) ∈ C 0 ( S ) ∀ t ∈ S and the kernel integration condition (Eqn. 12) holds.
Then kernel deformed exponential families are dense in P c wrt L r norm, Hellinger distance and Bregman divergence for the α -Tsallis negative entropy functional.
Proof.
For any p ∈ P c , define p δ =
p
+ δ 1+ δ .
Then
[Equation]
→for 1 ≤ r ≤ ∞ .
Thus for any ϵ > 0 , ∃ δ ϵ > 0 such that for any 0 < θ < δ ϵ , we have ∥ p − p θ
∥ r ≤ ϵ , where p θ ( t ) > 0 for all t ∈ S .
Define f = (cid:16) 1+ θ l + θ (cid:17) 1 − α log 2 − α p θ 1+ θ l + θ .
Since p ∈ C ( S ) , so is f .
Fix any η > 0 and note that f ( t )
[Equation]
(cid:32) (cid:18)
[Equation]
(cid:19)
[Equation]
η (cid:33) p θ
[Equation] [Equation]
θ ) ≥ ( l + θ )
[Equation]
−
[Equation]
(cid:32)(cid:18)
[Equation]
θ l
[Equation]
θ (cid:19) α −
[Equation]
η (cid:33) p + θ 1 + θ (1 + θ ) ≥ ( l + θ ) exp 2 − α (cid:32)(cid:18)
+ θ l + θ (cid:19)
α −η (cid:33) p − l ≥ ( l + θ ) (cid:32) exp 2 − α (cid:32)(cid:18)
+ θ l + θ (cid:19)
α −η (cid:33) −(cid:33)
Thus
A = { t : f ( t ) ≥ η } = (cid:40) p − l ≥ ( l + θ ) (cid:32)
[Equation]
(cid:32)(cid:18)
[Equation]
(cid:19)
[Equation]
(cid:33)(cid:41)
Since p − l ∈ C 0 ( S ) the set on the second line is bounded.
Thus A is bounded so that f ∈ C 0 ( S ) .
Further, by Lemma 5.
[Equation]
giving us p θ ∈ P 0 .
By Proposition D.1 there is some p g in the deformed kernel exponential family so that ∥ p θ − p g
∥ L r ( S ) ≤ ϵ .
Thus ∥ p − p g ∥ r ≤ 2 ϵ for any 1 ≤ r ≤ ∞ .
To show convergence in
Helinger distance, note
[Equation]
so that L 1 ( S ) convergence, which we showed, implies Hellinger convergence.
Let us consider the
Bregman divergence.
Note the generalized triangle inequality 7 for Bregman divergence
[Equation]
Term
I
[Equation]
Note that the Bregman divergence is non-negative and thus we only need to worry about an upper bound.
The first term on the rhs clearly vanishes as θ → 0 since p θ → p and we can pull the limit under the integral since p, p θ are bounded.
For the second term, we already showed that ∥ p − p θ ∥ ∞ → 0 as θ → 0 .
Term II
Fix θ .
Then term II converges to 0 by Lemma D.5.
Term III
For term III ,
[Equation]
Clearly the first term on the rhs converges by L r convergence.
The L 1 term for the gradient is given by
[Equation]
so that the inner product terms are bounded as
[Equation]
Fixing θ and letting g → θ the RHS goes to 0 .
D.4.1 Some Supporting Lemmas and Claims for Approximation Theory
Lemma D.3.
(Functional Mean Value Theorem)
Let F : X → R be a Gateaux differentiable functional where f, g ∈ X some Banach space with norm ∥ · ∥ .
Then
[Equation]
where h = g + c ( f − g ) for some c ∈ [0 , 1] , F ′ ( h ) is the Gateaux derivative of F , and ∥ · ∥ op is the operator norm
∥
A ∥ op = inf { c > 0 :
∥
Ax ∥ ≤ c ∥ x ∥∀ x ∈ X } .
Proof.
Consider G ( η ) =
F ( g + η
( f − g )) .
Apply the ordinary mean value theorem to obtain
[Equation]
and thus
[Equation]
Claim 1 .
Consider P ∞ =
{ p f = exp 2 − α ( f − A α ( f )) : f ∈ L ∞ ( S ) } .
Then for p f ∈ P ∞ ,
A α ( f ) ≤ ∥ f ∥ ∞ .
Proof.
[Equation]
where for the second line recall that we assumed that throughout the paper 1 < α ≤ 2 .
Lemma D.4.
Consider P ∞ =
{ p f = exp 2 − α ( f − A α ( f )) : f ∈ L ∞ ( S ) } .
Then the Gateaux derivative of A α : L ∞ → R is given by the map
[Equation]
Proof.
This proof has several parts.
We first derive the Gateaux differential of p f in a direction ψ ∈ L ∞ and as it depends on the Gateaux differential of A α ( f ) in that direction, we can rearrange terms to recover the latter.
We then show that it exists for any f, ψ ∈ L ∞ .
Next we show that the second Gateaux differential of A α ( f ) exists, and use that along with a functional Taylor expansion to prove that the first Gateaux derivative is in fact a Frechet derivative.
In [23] they show how to compute the gradient of A α ( θ ) for the finite dimensional case: we extend this to the Gateaux differential.
We start by computing the Gateaux differential of p f .
[Equation]
evaluating at η = 0 gives us
[Equation]
Note that since p f + ηψ ( t ) is a probability density function, it is integrable and thus we can apply the dominated convergence theorem to pull a derivative with respect to η under an integral.
Noting that the integral of a density function is 1 and thus its derivative is 0 , we can then recover the Gateaux diferential of A α via
[Equation]
where the last line follows as ψ ∈ L ∞ .
Thus the Gateaux derivative exists in L ∞ directions.
The derivative at f maps ψ : → E
[Equation]
Lemma D.5.
Define P ∞ =
{ p f = exp 2 − α ( f − A α ( f )) : f ∈ L ∞ ( S ) } where L ∞ ( S ) is the space of almost everywhere bounded measurable functions with domain S .
Fix f 0 ∈ L ∞ .
Then for any fixed ϵ > 0 and p g , p f ∈ P ∞ such that f, g ∈ B ∞
ϵ
( f 0 ) the L ∞ closed ball around f 0 , there exists constant M exp > 0 depending only on f 0 such that
[Equation]
Further
[Equation]
Proof.
This Lemma mirrors Lemma A.1 in [33], but the proof is very different as they rely on the property that exp( x + y ) =
exp( x ) exp
( y ) , which does not hold for β -exponentials.
We thus had to strengthen the assumption to include that f and g lie in a closed ball, and then use the functional mean value theorem Lemma D.3 as the main technique to achieve our result.
Consider that by the mean value inequality,
[Equation]
where h = cf + (1 − c ) g for some c ∈ [0 , 1] .
This implies
[Equation]
We need to bound exp β
( h − A α ( h )) and ∥
A α ( f ) − A α ( g ) ∥ ∞ .
We can show a bound on ∥ h ∥ ∞
[Equation]
so that h is bounded.
Now we previously showed in claim 1 that |
A α ( h ) | ≤ ∥ h ∥ ∞ ≤ ϵ + ∥ f 0 ∥ ∞ .
Since h, A α ( h ) are both bounded exp β ( h − A α ( h )) 2 − α is also.
Now note that by Lemma D.3,
[Equation]
We need to show that ∥
A ′ α ( h ) ∥ op is bounded for f, g ∈ B ϵ ( f 0 ) .
Note that in Lemma D.4 we showed that
[Equation]
Thus ∥ A ′ α ∥ op = sup {| A ′ α ( h )( m ) | : ∥ m
∥ ∞
= 1 } ≤ 1 .
Let M exp be the bound on exp β ( h −
A α ( h )) .
Then putting everything together we have the desired result
[Equation]
Now
[Equation] [Equation]
For the inner prodct term, first note that following [23] the gradient is given by
[Equation]
Thus
[Equation]
where the second line follows from claim 1.
Further note that by Taylor’s theorem,
[Equation]
for some z between x and y .
Then letting y = p f ( t ) and x = p g ( t ) , we have for some z = h ( t ) lying between p f ( t ) and p g ( t ) that
[Equation]
[Equation]
Since f ∈ L ∞ then applying Claim 1 we have that each p f , p g ∈ L ∞ and thus h is.
Then
[Equation] [Equation]
so that
[Equation]
Putting it all together we obtain
[Equation]
E Numerical Integration Stability and Convergence Analysis E.1 Stable Numerical Integration
One issue is numerical underflow when computing the normalizing constant.
In the kernel exponential family case, if | f ( t ) | is very large and f ( t ) < 0 for all evaluated t , then on a computer exp( f ( t )) will round to 0 for all the evaluated t .
Thus Z the normalizing constant will be estimated as 0 in numerical integration and the estimate of exp( f ( t )) /Z
cannot be computed.
A similar issue exists for the deformed case.
For kernel exponential family, we can use the standard technique used in discrete softmax implementations (see [10] chapter 4).
Note
[Equation]
Then instead of normalizing exp( f ( t )) with exp( − A ( f ))
we normalize exp( f ( t ) − C ) with exp( C −
A ( f )) , letting C = sup t f ( t ) .
Taking C = sup t f ( t ) will prevent underflow as if f ( t ) < 0 ∀ t , f ( t )
− sup f ( t ) ≈
for some t (equality if there is a maximum, which there always is when using a finite set of t for numerical integration).
For the deformed case, recall that by Lemma 5.3, exp 2 − α ( f ( t ) − log α Z )
= 1 Z exp 2 − α
( Z α − 1 f ( t )) for Z > 0 .
Then letting log α C = sup t f ( t ) and noting that for x, y > 0 , log α x − log α y = log α ( x ⊘ α y )
=
[ x 1 − α − y 1 − α + 1] 1 1 − α + [37], exp
[Equation]
f
[Equation]
) = exp
[Equation]
( f (
[Equation]
− log
[Equation]
C + log
[Equation]
C − log α Z ) =
[Equation]
Now consider ˜ f
( t ) =
(cid:16)
[Equation]
α − 1 f ( t ) .
Then
[Equation]
We can thus estimate ˜ f
( t ) , subtract sup t ˜ f ( t ) (max in practice), take the deformed exponential, and normalize.
The computational operations are nearly identical to the kernel exponential family case, although the reasons for their validity are very different.
E.2 Kernel Exponential Family Attention
Here we show conditions for which numerical integration of
[Equation]
using the trapezoidal rule is exponentially convergent.
We start by restating a theorem.
This says that the trapezoidal rule for numerical integration of holomorphic functions of sufficiently fast decay has exponential convergence.
A version of this theorem comes from [39], but there are slight issues with the notation and conditions.
A slightly revised statement is in the course notes of [14], which we follow here.
Theorem E.1.
Let w : C → C be analytic in the strip S b = { z ∈ C : | Im ( z ) | < a } for some a > 0 .
Suppose further that w ( z ) → 0 as | z | → ∞ in the strip, and for some M > 0 ,
[Equation]
[Equation]
for all b ∈ ( − a, a ) .
Then leting I = (cid:82) ∞
−∞ w ( x ) dx
and I h
= h (cid:80) ∞
k =
−∞ w ( kh ) ,
[Equation]
Next we give conditions on q 0 ( t ) , V ( t ) , and k so that numerical integration is exponentially convergent.
Specifically that q 0 and V have holomorphic complex extensions to a strip and that k has exponentially decaying Fourier transform.
Corollary E.2.
Assume that q 0 , V have holomorphic complex extensions to a strip S b = { z ∈ C : | Im ( z ) | < b } , q 0 ( z ) → 0 as | z | → ∞ and V is bounded by M V .
Also assume that for any t i ∈ R the absolute value of the Fourier transform | ˆ
[Equation]
| A
[Equation]
exp( − a | ξ | ) | , where A : R → C and a > 0 is a fixed constant.
Further assume that f
[Equation]
i
[Equation] [Equation]
, and
[Equation]
for some M > 0 .
Then if w ( z ) = q 0 ( z ) exp
( f ( z ))
V ( z ) → 0 as | z | → ∞ , the trapezoidal rule for w ( t ) =
q 0 ( t ) exp( f ( t ))
V ( t ) is exponentially convergent.
Proof.
Note that
[Equation]
and thus
[Equation]
Thus since the fourier transform of f has (at least) exponential decay, by Theorem 3.1 in [35] the extension of f ( t ) to f ( z ) is holomorphic in the strip.
Since compositions and products of holomorphic functions are holomorphic, q 0 ( z ) exp( f ( z ))
V ( z ) is holomorphic in the strip.
Further, since for each t i , k ( t, t i ) ∈ C 0 , f is and thus it is bounded and thus its complex extension is bounded, say by M F , thus the complex exp( f ) is bounded by exp( M f ) .
Since V is also bounded, we have
[Equation]
Finally,
[Equation]
and thus w satisfies the conditions of the previous theorem.
Finally, we show example special cases of q 0 , V, k satisfying those conditions.
Corollary E.3.
If k is a Gaussian kernel, V is a linear combination of Gaussian RBFs, and q 0 ( t ) =√ 2 π exp( − t 2 / 2) , then the trapezoidal rule for kernel exponential family attention is exponentially convergent.
Proof.
For the Gaussian kernel, we have k ( t, t i ) =
exp( − ζ 2 ( t − t i ) 2 ) .
Now
[Equation]
and thus since exp( − iξt i ) lies on the unit circle,
[Equation]
This satisfies the Fourier decay condition for the kernel so that functions f ∈ H have holomorphic extensions to the strip.
A similar technique can be applied to V and q 0 to show that they have holomorphic extensions to the strip, and clearly both V and f are bounded since they are continuous and vanish at infinity.
Thus exp( f ) is also bounded.
For the decay of q 0 , note
[Equation]
satisfying the integration condition Eqn. 18.
It remains to show that w ( z ) → 0 .
First note that
[Equation]
Since z ∈ S b , | z | → ∞ requires | x | → ∞ , and 1 √ 2 π exp( a 2 / 2) exp( − x 2 / 2) → 0 as | x | → ∞ .
Thus q 0 ( z ) → 0 as | z | → ∞ in the strip.
Note that since V, f are bounded and q 0 ( z ) → 0 in the strip as | z | → ∞ , w ( z ) → 0 as | z | → ∞ in the strip.
E.3 Convergence of Numerical Integration for Kernel Deformed Exponential Family Attention E.3.1 Smooth Approximation to Kernel Deformed Exponential Family Attention
Definition E.4.
Let
[Equation]
for some ρ > 0 .
Claim 2 .
exp ρ,β ( z ) is holomorphic on the strip { z ∈ C :
Im ( z ) ∈ [ − π/ 2 , π/ 2] } for the principal branch of log .
Proof.
This proof is adapted from [11].
We show it for exp 0 ,
but the idea can easily be extended to more general ρ and β .
Note that the principal branch of log z is analytic outside of B = { z = x + iy : −∞ < x ≤ 0 , y = 0 } .
Thus exp 0 , 0 is analytic as long as 1 + exp(1 + z ) is not in B .
Now exp(1+ z ) = exp( x +1) exp( iy ) , so exp(1+ z ) will be in { z = x + iy : x ∈ ( −∞ , − 1] , y = 0 } when exp( x + 1) ≥ 1 , exp( iy )
= − 1 , i.e. x ≥ − 1 ,
y =
π
+ 2 πn, n ∈ N .
The y condition will not be satisfied on the strip above and thus exp 0 ,
( z ) is holomorphic on that strip.
Corollary E.5.
If k is a Gaussian kernel, V is a linear combination of Gaussian RBFs, and q 0 ( t ) = 1 √ 2 π exp( − t 2 / 2) , then the trapezoidal rule for softplus approximation to kernel deformed exponential family attention is exponentially convergent.
Proof.
Apply Theorem E.1.
We already showed in the kernel exponential family section that q 0 and
V are holomorphic in the strip, and we showed that exp ρ,β is in the previous claim.
We also showed that V is bounded and that q 0 satisfies the integration condition and q 0 ( z ) → 0 as | z | → ∞ in the strip.
It remains to show that exp ρ,β ( f ( t )) is bounded (and thus its complex extension is).
We show it for exp ρ, 0 ( f ( t )) but again the idea can be extended to general β .
Note that since log(1 + x ) ≤ x for x > − 1 ,
[Equation]
since f is bounded.
Further, fixing t , log(1 + exp( ρ (1 + t ))) /ρ is monotonically decreasing as a function of ρ and thus for all ρ ≥ 1 ,
[Equation]
Thus w satisfies both the integration condition and the convergence condition.
E.3.2 Using the Smooth Approximation to Bound the Numerical Integral for Kernel Sparsemax Attention
Let I h be the numerical integral proportional to kernel deformed exponential family attention, I h,s be its softplus approximation, I be the true kernel deformed exponential family integral, and I s be its softplus approximation.
Then
[Equation]
We already bounded | I h,s −
I s | in the previous subsection, so we will bound the other two terms on the right hand side.
E.3.3 Bounding I s − I
We first bound the difference between the softplus integral and the integral using the positive part/ReLU.
By Hoelder’s inequality,
[Equation]
E.
Bounding I
h − I h,s |
I h,s −
I h | = | h ∞ (cid:88) k =
−∞ q 0 ( kh ) V ( kh )
[ln(1 + exp( ρ (1 + kh ))) /ρ
− max(0 , 1 + kh )] | = | h ⟨ q 0 ( h · ) V ( h · ) , ln(1 + exp( ρ (1 + h · )))
/ρ
− max(0 , 1 + h · ) ⟩ l 2 | ≤ h (cid:32) ∞ (cid:88) k =
−∞ q 0 ( hk ) | V ( hk ) | (cid:33) ln(2) /ρ where the last line again uses Hoelder’s inequality.
E.3.5 Putting it All Together
We now have, for any ρ > 0 ,
[Equation]
and since this holds for all ρ > 0 we can take ρ → ∞ and we have
[Equation]
E.4 Synthetic Experiments: Convergence
We now analyze convergence of the trapezoidal rule for multimodal continuous attention using numerical integration empirically using synthetic experiments.
We define
[Equation]
and
[Equation]
with γ i , B i ∼ U ( − 1 , 1) .
We set I = 10 and t i to be evenly spaced in the interval [0 , 1] .
We then compute
[Equation]
which is proportional to attention for kernel exponential families.
We also compute
[Equation]
proportional to the sparsemax case of kernel deformed exponential family attention.
Finally, we also use the softplus approximation
[Equation]
Note that we do not use the ˜ f and ˜ γ notation for the kernel deformed case as the main point is to examine the numerical integration convergence rather than to formally handle normalization.
We simulate 100 times.
Figure 2a compares the convergence as the number of grid points in the trapezoidal rule increases.
2b shows a single case of the integral value using deformed exponential vs the softplus approximation for various values of ρ and numbers of grid points.
We see that for ρ = 5 , 10 the integral using softplus approximation is essentially indistinguishable from that using the positive part/ReLU.
F Computational/Memory Complexity
Our attention mechanisms require computation in three steps:
Estimating the value function parameters with regularized multivariate linear regression.
This was already required by [23, 24].
This involves computing HF T ( FF T + λI N ) − 1 for
F ∈ R N × L , H ∈ R D × L .
We thus need to compute
• The Gram matrix FF T ∈ R N × N .
This is O ( N 2 L ) .
• The inverse ( FF T + λI N ) − 1 ∈ R N × N .
This is O ( N 3 ) .
• The matrix multiplication HF T ∈ R D × N .
This is O ( DNL ) .
• The matrix multiplication giving HF T ( FF T + λI N ) − 1 ∈ R D × N .
This is O ( DNL ) .
This gives a final cost of O ( N 3 + N 2 L + DNL ) .
Note that if we do this for raw data, we only need to do this once, while if we do this for activations, we need to do it at every iteration.
Computing the normalizing constant of the attention density.
• This requires computing exp( ˜ f ( t ))
q 0 ( t ) for each numerical integration grid point 1 , . . .
, G . • Each term has cost O ( I ) , where I is the number of inducing points.
The final cost is O ( GI ) .
Computing the numerical integral/context vector.
• We need to compute exp( ˜ f ( t ))
q 0 ( t ) ψ n ( t ) for each basis function n = 1 , . . .
, N and each numerical integration grid point 1 , . .
, G . •
Each such computation has cost O ( DI ) , since we have D dimensions and I inducing points.
The total cost is thus O ( NGDI ) , linear in the number of basis functions, grid evaluation points, dimensions, and inducing points.
Applying the final linear transformation.
This gives total computation O ( N 3 + N 2 L + DNL + NGDI ) .
The first three terms were already required in [23, 24], while the last is due to numerical integration.
For memory, we require
[Table] [Table]
All the matrices of the value function.
This gives O ( N 2 + DN ) .
All of the evaluations for numerical integration before integration happens.
This requires O ( NGD ) .
Note that in principle one could do the numerical integrations for each basis function and dimension sequentially, without storing all of them at once.
In this case the memory requirements would be O ( G ) .
We are uncertain whether or not this requires custom code, but regardless neither memory requirement is excessive for most applications.
The total memory cost is thus O ( N 2 + NGD ) , where the first term was already present in [23, 25] and the second is due to numerical integration.
These results are summarized in Table 5.
G FordA: Additional Details
We plot attention densities for randomly selected examples under each density class.
We find that only the kernel methods appear truly multimodal visually (Gaussian mixture does not), and the kernel sparsemax case actually highlights specific peaks and troughs of the original signal, suggesting the ability to attend to higher frequencies.
Figure 7 shows rescaled kernel softmax attention densities along with the original series.
These are generally smooth and show rich multimodality, and often highlight 0 crossings where the signal decreases.
Figure 9 shows the same for kernel sparsemax.
These are very interesting, and learn rich sparsity patterns and often highlight the peaks and troughs of the series, while taking 0 values over many regions where the series becomes negative.
This dataset uses the architecture in Fig.
However, several hyperaparameters are different, as mentioned in Table 7.
In particular, we use more value basis functions (256) as the sequence is longer, and a smaller number of LSTM hidden units (128).
On this dataset, claiming interpretability is more difficult because it is engine noise while the class meanings are unknown: only the binary labels are present.
However it is still interesting that kernel sparsemax is able to select individual waves and exhibit rich sparsity patterns.
[Table]
[Figure]
Batch
[Figure] [Figure]
Basis Functions 64,
[Figure]
Points
[Figure]
Optimizer
Adam
[Figure]
1e
[Figure]
Epochs
H Additional uWave Details
We experiment with N = 64 , 128 basis functions, and use a learning rate of 1 e − 4 .
We use H =attention mechanisms, or heads.
Unlike [42], our use of multiple heads is slightly different as we use the same value function for each head, and only vary the attention densities.
Additional architectural details are given below.
Table 8 summarizes the hypermarameters and training details.
H.1 Value Function
The value function uses regularized linear regression on the original time series observed at random observation times (which are not dependent on the data) to obtain an approximation V ( t ; B ) =
B Ψ( t ) ≈ X ( t ) .
The H in Eqn.
6 is the original time series.
H.1.1 Encoder
In the encoder, we use the value function to interpolate the irregularly sampled time series at the original points.
This is then passed through a convolutional layer with 4 filters and filter sizefollowed by a max pooling layer with pool size 2 .
This is followed by one hidden layer withunits and an output v of size 256 .
The attention densities for each head h
= 1 , . .
, H are then
[Equation]
for vectors w h, 1 , w h, 2 and matrices W h and heads h
= 1 , . .
, H
H.1.2 Attention Mechanism
After forming densities and normalizing, we have densities p 1 ( t ) , . .
p H ( t ) , which we use to compute context scalars
[Equation]
We compute these expectations using numerical integration to compute basis function expectations E p h
[ ψ n ( T )] and a parametrized value function V ( t ) =
Bψ ( t ) as described in section 3 .
[Table]
The classifier takes as input the concatenated context scalars as a vector.
A linear layer is then followed by a softmax activation to output class probabilities.
I ECG Heartbeat Classification: MIT BIH
We use a kaggle version 8 of the MIT Arrhythmia Database [9].
Not all versions are comparable: [12] report results under well-balanced classes (unclear how they obtain these), while [26] augment the dataset with SMOTE.
We do no data augmentation, so compare to two time series classification baselines in TSAI: a hybrid fully convolutional and LSTM network[13], and a transformer [46].
The task is to detect abnormal heart beats from ECG.
The five classes are {Normal, Supraventricular premature, Premature ventricular contraction, Fusion of ventricular and normal, Unclassifiable}.
There data has a 87,553/21,891 train/test split.
Each sample is a univariate time series of length 187: we pass this through two convolutional layers in order to obtain a multivariate representation of each time step.
We then use an LSTM.
The hidden layer is used to construct discrete attention using [3].
Following [24] for unimodal continuous softmax and sparsemax, we first output discrete attention weights p
=
( p 1 , . . .
, p L ) , p ∈ ∆ L in the probability simplex and then compute µ = E p [ T/L ] and σ 2 = E p
[( T/L ) 2 ] − µ 2 where T ∼ p .
The value function uses Gaussian RBFs.
The final context vector is passed through three feedforward layers.
Kernel softmax attention is not particularly interpretable.
For kernel sparsemax, the attention densities tend to highlight peaks in the signal.
Particularly they assign high weight to the R wave, the peak of the QRS complex, of the heartbeat.
We use a kaggle version 9 of the MIT Arrhythmia Database [9].
Not all versions are comparable: [12] report results under well-balanced classes (unclear how they obtain these), while [26] augment the dataset with SMOTE.
We do no data augmentation, so compare to two time series classification baselines in TSAI: a hybrid fully convolutional and LSTM network[13], and a transformer [46].
The task is to detect abnormal heart beats from ECG.
The five classes are {Normal, Supraventricular premature, Premature ventricular contraction, Fusion of ventricular and normal, Unclassifiable}.
There data has a 87,553/21,891 train/test split.
Each sample is a univariate time series of length 187: we pass this through two convolutional layers in order to obtain a multivariate representation of each time step.
We then use an LSTM.
The hidden layer is used to construct discrete attention using [3].
Following [24] for unimodal continuous softmax and sparsemax, we first output discrete attention weights p
=
( p 1 , . . .
, p L ) , p ∈ ∆ L in the probability simplex and then compute µ = E p [ T/L ] and σ 2 = E p
[( T/L ) 2 ] − µ 2 where T ∼ p .
The value function uses Gaussian RBFs.
The final context vector is passed through three feedforward layers.
Additional details, including hyperparameters and plots of attention densities, are in Appendix I. Kernel softmax attention is not particularly
[Table]
interpretable.
For kernel sparsemax, the attention densities tend to highlight peaks in the signal.
Particularly they assign high weight to the R wave, the peak of the QRS complex, of the heartbeat.
We plot attention densities for randomly selected examples under the different density classes, and find that only kernel deformed exponential families/sparsemax learn interpretable attention densities, which focus on regions where the electrical signals from the heart are strong.
Figures 12 andshow attention densities vs original signals for continuous softmax and sparsemax, respectively.
Both are only able to learn simple unimodal densities.
Figure 14 shows the same for Gaussian mixture attention.
These do not look very multimodal, despite having one component per time point.
This is likely due to lack of separation between components.
However, we do see that the shapes look more flexible than in single Gaussian or truncated parabola cases.
Figure 15 shows attention densities vs original signals for kernel softmax attention.
While not particularly interpretable, it learns densities similar to exponential densities without them being specified, a benefit of being a non-parametric density.
Figure 16 shows the same for kernel sparsemax.
These show interesting highlighting of waves, which describe electrical signals passing through the heart conduction system.
There is a particular focus on the R wave, the largest peak in a heartbeat, representing electrical stimulus in the main ventricular mass.
Note that given the relatively complex model structure and reasonably high capacity relative to the original time series length (512 LSTM hidden units per time step for a univariate time series of length 187), models without an interpretable attention density may still perform well by taking advantage of capacity elsewhere in the model.
However, a model that selects ECG waves may be useful in convincing specialists of its value.
I.1 General Architecture
Our general architecture is shown in 11.
In the first part, two convolutional layers of filter size 5 andfilters with padding map the original univariate time series from R 187 to a multivariate representation R 187 × 24 .
This is then passed to an LSTM.
Context vectors are computed using either the original hidden states (discrete attention) or a continous-time representation (continuous attention).
The context vector is then fed into a feedforward network for final classification.
The entire architecture is trained end-to-end.
Hyperparameters are described in Table 10.
I.2 Value Function
The value function uses regularized linear regression on the hidden states of an LSTM to obtain an approximation V ( t ; B ) =
B Ψ( t ) ≈ h t .
The H in Eqn.
6 is the set of all hidden states.
I.3 Encoder
The encoder takes the hidden layer of the LSTM as input.
For the hidden units h t ∈ R p for a given time step, it computes
[Equation]
Note that this is written slightly differently from the form in the main paper.
I.4 Attention Mechanism
The attention mechanism takes the parameters from the encoder and forms an attention density.
It then computes
[Equation]
)] for input to the classifier.
I.5 Classifier
The classifier has three feedforward layers, where the first two have ReLU activation functions.