T RANSFORMERS ARE D EEP I NFINITE -D IMENSIONAL N ON -M ERCER B INARY K ERNEL M ACHINES
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite their ubiquity in core AI ﬁelds like natural language processing, the mechanics of deep attention-based neural networks like the “Transformer” model are not fully understood.
In this article, we present a new perspective towards understanding how Transformers work.
In particular, we show that the “dot-product attention” that is the core of the Transformer’s operation can be characterized as a kernel learning method on a pair of Banach spaces.
In particular, the Transformer’s kernel is characterized as having an inﬁnite feature dimension.
Along the way we generalize the standard kernel learning problem to what we term a “binary” kernel learning problem, where data come from two input domains and a response is deﬁned for every cross-domain pair.
We prove a new representer theorem for these binary kernel machines with non-Mercer (indeﬁnite, asymmetric) kernels (implying that the functions learned are elements of reproducing kernel Banach spaces rather than Hilbert spaces), and also prove a new universal approximation theorem showing that the Transformer calculation can learn any binary non-Mercer reproducing kernel Banach space pair.
We experiment with new kernels in Transformers, and obtain results that suggest the inﬁnite dimensionality of the standard
Transformer kernel is partially responsible for its performance.
This paper’s results provide a new theoretical understanding of a very important but poorly understood model in modern machine learning.
I NTRODUCTION
Since its proposal by Bahdanau et al.
(2015), so-called neural attention has become the backbone of many state-of-the-art deep learning models.
This is true in particular in natural language processing (NLP), where the Transformer model of Vaswani et al.
(2017) has become ubiquitous.
This ubiquity is such that much of the last few years’ NLP breakthroughs have been due to developing new training regimes for Transformers (Radford et al., 2018;
Devlin et al., 2019; Yang et al., 2019; Liu et al., 2019;
Wang et al., 2019a; Joshi et al., 2020; Lan et al., 2020; Brown et al., 2020, etc.).
Like most modern deep neural networks, theoretical understanding of the Transformer has lagged behind the rate of Transformer-based performance improvements on AI tasks like NLP.
Recently, several authors have noted Transformer operations’ relationship to other, better-understood topics in deep learning theory, like the similarities between attention and convolution (Ramachandran et al., 2019; Cordonnier et al., 2020) and the design of the residual blocks in multi-layer Transformers (e.g., Lu et al.
(2019); see also the reordering of the main learned (fully-connected or attentional) operation, elementwise nonlinearity, and normalization in the original Transformer authors’ ofﬁcial reference codebase (Vaswani et al., 2018) and in some more recent studies of deeper Transformers (Wang et al., 2019b) to the “pre-norm” ordering of normalize, learned operation, nonlinearity, add residual ordering of modern (“v2”)
Resnets (He et al., 2016)).
In this paper, we propose a new lens to understand the central component of the Transformer, its “dot-product attention” operation.
In particular, we show dot-product attention can be characterized as a particular class of kernel method (Schölkopf & Smola, 2002).
More speciﬁcally, it is a so-called indeﬁnite and asymmetric kernel method, which refer to two separate generalizations of the classic class of kernels that does not require the classic assumptions of symmetry and positive (semi-) deﬁniteness
(Ong et al., 2004; Balcan et al., 2008; Zhang et al., 2009; Wu et al., 2010; Loosli et al.,
2016; Oglic & Gärtner, 2018; 2019, etc.).
We in fact show in Theorem 2 below that dot-product attention can learn any asymmetric indeﬁnite kernel.
This insight has several interesting implications.
Most immediately, it provides some theoretical justiﬁcation for one of the more mysterious components of the Transformer model.
It also potentially opens the door for the application of decades of classic kernel method theory towards understanding one of today’s most important neural network models, perhaps similarly to how tools from digital signal processing are widely used to study convolutional neural networks.
We make a ﬁrst step on this last point in this paper, proposing a generalization of prior kernel methods we call “binary” kernel machines, that learns how to predict distinct values for pairs of elements across two input sets, similar to an attention model.
The remainder of this paper is organized as follows.
Section 2 reviews the mathematical background of both Transformers and classic kernel methods.
Section 3 presents the deﬁnition of kernel machines on reproducing kernel Banach spaces (RKBS’s) that we use to characterize Transformers.
In particular we note that the Transformer can be described as having an inﬁnite-dimensional feature space.
Section 4 begins our theoretical results, explicitly describing the Transformer in terms of reproducing kernels, including explicit formulations of the Transformer’s kernel feature maps and its relation to prior kernels.
Section 5 discusses Transformers as kernel learners, including a new representer theorem and a characterization of stochastic-gradient-descent-trained attention networks as approximate kernel learners.
In Section 6, we present empirical evidence that the inﬁnite-dimensional character of the
Transformer kernel may be somewhat responsible for the model’s effectiveness.
Section 7 concludes and summarizes our work.
B ACKGROUND AND R ELATED W ORK 2. T RANSFORMER N EURAL N ETWORK M ODELS
The Transformer model (Vaswani et al., 2017) has become ubiquitous in many core AI applications like natural language processing.
Here, we review its core components.
Say we have two ordered sets of vectors, a set of “source” elements { s 1 , s 2 , . . .
, s S } , s j ∈ R d s and a set of “target” elements { t 1 , t 2 , . .
. , t T } , t i ∈ R d t .
In its most general form, the neural-network “attention” operation that forms the backbone of the Transformer model is to compute, for each t i , a t i -speciﬁc embedding of the source sequence { s j } Sj =1 .
The particular function used in the Transformer is the so-called “scaled dot-product” attention, which takes the form
[Equation]
where W V , W K ∈ R d s × d , and
W Q ∈ R
d t × d are learnable weight matrices, usually called the “value,” “key,” and “query” weight matrices, respectively.
Usually multiple so-called “attention heads” with independent parameter matrices implement several parallel computations of (1), with the Cartesian product (vector concatenation) of several d -dimensional head outputs forming the ﬁnal output t 0 i .
Usually the unnormalized a ij ’s are called attention scores or attention logits, and the normalized α ij ’s are called attention weights.
In this paper, we restrict our focus to the dot-product formulation of attention shown in (1).
Several other alternative forms of attention that perform roughly the same function (i.e., mapping from R d s × R d t to R ) have been proposed (Bahdanau et al., 2015;
Luong et al., 2015; Veliˇckovi´c et al., 2018; Battaglia et al., 2018, etc.)
but the dot-product formulation of the Transformer is by far the most popular.
2. K ERNEL M ETHODS AND G ENERALIZATIONS OF K ERNELS
Kernel methods (Schölkopf & Smola, 2002; Steinwart & Christmann, 2008, etc.) are a classic and powerful class of machine learning methods.
The key component of kernel methods are the namesake kernel functions , which allow the efﬁcient mapping of input data from a low-dimensional data domain, where linear solutions to problems like classiﬁcation or regression may not be possible, to a high- or inﬁnite-dimensional embedding domain, where linear solutions can be found.
Given two nonempty sets X and Y , a kernel function κ is a continuous function κ : X × Y → R .
In the next few sections, we will review the classic symmetric and positive (semi-) deﬁnite, or Mercer, kernels, then discuss more general forms.
2.2. S YMMETRIC AND P OSITIVE S EMIDEFINITE (M ERCER ) KERNELS
If X = Y , and for all x i , x j ∈ X
=
Y , a particular kernel κ has the properties
[Equation]
where K in (2b) is the Gram matrix, deﬁned as K ij = κ ( x i , x j ) , then κ is said to be a Mercer kernel.
For Mercer kernels, it is well-known that, among other facts, (i) we can deﬁne a Hilbert space of functions on X , denoted H κ (called the reproducing kernel Hilbert space, or RKHS, associated with the reproducing kernel κ ), (ii)
H κ has for each x a (continuous) unique element δ x called a point evaluation functional, with the property f ( x ) =
δ x ( f ) ∀ f ∈ H κ , (iii) κ has the so-called reproducing property , h f, κ ( x, · ) i H κ = f ( x ) ∀ f ∈ H κ , where h· , ·i H κ is the inner product on H κ , and (iv) we can deﬁne a “feature map” Φ : X → F H , where F H is another Hilbert space sometimes called the feature space, and κ ( x, y ) =
h Φ( x ) , Φ( y ) i F H (where h· , ·i F H is the inner product associated with
F H ).
This last point gives rise to the kernel trick for RKHS’s.
From a machine learning and optimization perspective, kernels that are symmetric and positive (semi-) deﬁnite (PSD) are desirable because those properties guarantee that empirical-risk-minimization kernel learning problems like support vector machines (SVMs), Gaussian processes, etc. are convex.
Convexity gives appealing guarantees for the tractability of a learning problem and optimality of solutions.
2.2. L EARNING WITH NON -M ERCER KERNELS
Learning methods with non-Mercer kernels, or kernels that relax the assumptions (2), have been studied for some time.
One line of work (Lin & Lin, 2003; Ong et al., 2004; Chen & Ye, 2008; Luss & D’aspremont, 2008; Alabdulmohsin et al., 2015; Loosli et al., 2016; Oglic & Gärtner, 2018; 2019, etc.) has focused on learning with symmetric but indeﬁnite kernels, i.e., kernels that do not satisfy (2b).
Indeﬁnite kernels have been identiﬁed as reproducing kernels for so-called reproducing kernel
Kre ˘ ın spaces (RKKS’s)
since Schwartz (1964) and Alpay (1991).
Replacing a Mercer kernel in a learning problem like an SVM with an indeﬁnite kernel makes the optimization problem nonconvex in general (as the kernel Gram matrix K is no longer always PSD).
Some early work in learning with indeﬁnite kernels tried to ameliorate this problem by modifying the spectrum of the Gram matrix such that it again becomes PSD (e.g., Graepel et al., 1998;
Roth et al., 2003; Wu et al., 2005).
More recently, Loosli et al.
; Oglic & Gärtner (2018), among others, have proposed optimization procedures to learn in the RKKS directly.
They report better performance on some learning problems when using indeﬁnite kernels than either popular Mercer kernels or spectrally-modiﬁed indeﬁnite kernels, suggesting that sacriﬁcing convexity can empirically give a performance boost.
This conclusion is of course reminiscent of the concurrent experience of deep neural networks, which are hard to optimize due to their high degree of non-convexity, yet give superior performance to many other methods.
Another line of work has explored the application of kernel methods to learning in more general
Banach spaces, i.e., reproducing kernel Banach spaces (RKBS’s)
(Zhang et al., 2009).
Various constructions to serve as the reproducing kernel for a Banach space (replacing the inner product of an
RKHS) have been proposed, including semi-inner products (Zhang et al., 2009), positive-deﬁnite bilinear forms via a Fourier transform construction (Fasshauer et al., 2015), and others (Song et al., 2013; Georgiev et al., 2014, etc.).
In this work, we consider RKBS’s whose kernels may be neither symmetric nor PSD.
A deﬁnition of these spaces is presented next.
G ENERAL REPRODUCING KERNEL B ANACH SPACES
Recently, Georgiev et al.
(2014), Lin et al.
(2019), and
Xu & Ye (2019) proposed similar deﬁnitions and constructions of RKBS’s and their reproducing kernels meant to encompass prior deﬁnitions.
In this paper, we adopt a fusion of the deﬁnitions and attempt to keep the notation as simple as possible to be sufﬁcient for our purposes.
Deﬁnition 1 ( Reproducing kernel Banach space (Xu & Ye, 2019, Deﬁnition 2.1; Lin et al., 2019,
Deﬁnitions 1.1 & 1.2; Georgiev et al., 2014)) .
Let X and Y be nonempty sets, κ a measurable function called a kernel, κ : X × Y
→
R , and B X and B Y Banach spaces of real measurable functions on X and Y , respectively.
Let h· , ·i B X ×B Y : B X × B Y → R be a nondegenerate bilinear mapping such that
[Equation]
Then, B X and B Y are a pair of reproducing kernel Banach spaces (RKBS’s) on X and Y , respectively, and κ is their reproducing kernel.
Line (3a) (resp.
(3c)) says that, if we take κ , a function of two variables x ∈ X and y ∈ Y , and ﬁx x (resp.
y ), then we get a function of one variable.
This function of one variable must be an element of
B Y (resp.
B X ).
Lines (3b) and (3d) are the reproducing properties of κ .
For our purposes, it will be useful to extend this deﬁnition to include a “feature map” characterization similar to the one used in some explanations of RKHS’s
(Schölkopf & Smola, 2002, Chapter 2).
Deﬁnition 2 ( Feature maps for RKBS’s
(Lin et al., 2019, Theorem 2.1; Georgiev et al., 2014)) .
For a pair of RKBS’s as deﬁned in Deﬁnition 1, suppose that there exist mappings Φ X : X → F X , Φ Y :
Y → F Y , where F X and F Y are Banach spaces we will call the feature spaces, and a nondegenerate bilinear mapping h· , ·i F X ×F Y : F X × F Y → R such that
[Equation]
In this case, the spaces B X and B Y can be deﬁned as (Xu & Ye, 2019;
Lin et al., 2019)
[Equation]
Remark 1 .
We brieﬂy discuss how to understand the spaces given by (5).
Consider (5a) for example.
It is a space of real-valued functions of one variable x , where the function is also parameterized by a v .
Picking a v ∈ F Y in (5a) deﬁnes a manifold of functions in B X .
This manifold of functions with ﬁxed v varies with the function Φ X .
Evaluating a function f v in this manifold at a point x is deﬁned by taking the bilinear product of Φ X ( x ) and the chosen v .
This also means that we can combine (4) and (5) to say
[Equation]
all
[Equation]
Remark 2 .
If Φ X ( x ) and Φ Y ( y ) can be represented as countable sets of real-valued measurable functions, { φ X ( x ) ‘ } ‘ ∈ N and { φ Y ( y ) ‘ } ‘ ∈ N for ( φ X ) ‘ : X → R and ( φ Y ) ‘ : Y → R (i.e.,
F X , F Y ⊂ Q ‘ ∈ N R ); and h u , v
i F X ×F Y
=
P ‘ ∈ N u
‘ v ‘ for u ∈ F X , v ∈ F Y ; then the “feature map” construction, whose notation we borrow from Lin et al.
(2019), corresponds to the “generalized
Mercer kernels” of Xu & Ye (2019).
D OT -P RODUCT A TTENTION AS AN RKBS K ERNEL
We now formally state the formulation for dot-product attention as an RKBS learner.
Much like with
RKHS’s, for a given kernel and its associated RKBS pair, the feature maps (and also the bilinear mapping) are not unique.
In the following, we present a feature map based on classic characterizations of other kernels such as RBF kernels (e.g., Steinwart et al.
Proposition 1.
The (scaled) dot-product attention calculation of (1) is a reproducing kernel for an RKBS in the sense of Deﬁnitions 1 and 2, with the input sets X and Y being the vector spaces from which the target elements { t i } Ti =1 , t i ∈ R d t and source elements { s j } Sj =1 , s
j ∈ R d s are drawn, respectively; the feature maps
[Equation]
where q ‘ is the ‘ th element of q = W Q t , k ‘ is the ‘ th element of k = W K s , W Q ∈ R
d × d t
W K ∈ R d × d s , with d ≤ d s , d t and rank( W Q ) =
rank( W K )
=
d ; the bilinear mapping h Φ X ( t ) , Φ Y ( s ) i F X ×F Y =
Φ X ( t ) · Φ Y ( s ) ; and the Banach spaces
[Equation]
with the “exponentiated query-key kernel,” κ
[Equation]
the associated reproducing kernel.
The proof of Proposition 1 is straightforward and involves verifying (9) by multiplying the two inﬁnite series in (7), then using the multinomial theorem and the Taylor expansion of the exponential.
In the above and when referring to Transformer-type models in particular rather than RKBS’s in general, we use t , s , q , and k for x, y, u , and v , respectively, to draw the connection between the elements of the RKBS’s and the widely-used terms “target,” “source,” “query,” and “key.”
The rank requirements on W Q and W K mean that span( { Φ X ( t ) , t ∈ X} )
=
F X and span( { Φ Y ( s ) , s ∈ Y} )
=
F Y .
This in turn means that the bilinear mapping is nondegenerate.
Remark 3 .
Now that we have an example of a pair of RKBS’s, we can make more concrete some of the discussion from Remark 1.
Examining (8a), for example, we see that when we select a k ∈ F Y , we deﬁne a manifold of functions in B X where k is ﬁxed, but W Q can vary.
Similarly, selecting a q ∈ F X deﬁnes a manifold in B Y .
Selecting an element from both F X and F Y locks us into one element each from B X and B Y , which leads to the equality in (6).
Remark 4 .
Examining (8)-(9), we can see that the element drawn from F Y that parameterizes the element of B X , as shown in (8a), is a function of Φ Y (and vice-versa for (8b)).
This reveals the exact mechanism in which the Transformer-type attention computation is a generalization of the RKBS’s considered by Fasshauer et al.
(2015), Lin et al.
(2019), Xu & Ye (2019), etc., for applications like
SVMs, where one of these function spaces is considered ﬁxed.
Remark 5 .
Since the feature maps deﬁne the Banach spaces (5), the fact that the parameters W Q and W K are learned implies that Transformers learn parametric representations of the RKBS’s themselves.
This is in contrast to classic kernel methods, where the kernel (and thus the reproducing space) is usually ﬁxed.
In fact, in Theorem 2 below, we show that (a variant of) the Transformer architecture can approximate any RKBS mapping.
Remark 6 .
The symmetric version of the exponentiated dot product kernel is known to be a reproducing kernel for the so-called Bargmann space (Bargmann, 1961) which arises in quantum mechanics.
Remark 7 .
Notable in Proposition 1 is that we deﬁne the kernel of dot-product attention as including the exponential of the softmax operation.
The output of this operation is therefore not the attention scores a ij but rather the unnormalized attention weights, ¯ α ij = α ij P j α ij .
Considering the exponential as a part of the kernel operation reveals that the feature spaces for the Transformer are in fact inﬁnite-dimensional in the same sense that the RBF kernel is said to have an inﬁnite-dimensional feature space.
In Section 6, we ﬁnd empirical evidence that this inﬁnite dimensionality may be partially responsible for the Transformer’s effectiveness.
T RANSFORMERS AS KERNEL LEARNERS 5. T HE BINARY RKBS LEARNING PROBLEM AND ITS REPRESENTER THEOREM
Most kernel learning problems take the form of empirical risk minimization problems.
For example, if we had a learning problem for a ﬁnite dataset ( x 1 , z 1 ) , . . .
, ( x n , z n ) , x i ∈ X , z i ∈ R and wanted to learn a function f : X → R in an RKHS H κ , the learning problem might be written as
[Equation]
where L : X × R × R → R is a convex loss function, R : [0 , ∞ )
→ R is a strictly increasing regularization function, and λ is a scaling constant.
Recent references that consider learning in
RKBS’s
(Georgiev et al., 2014; Fasshauer et al., 2015; Lin et al., 2019; Xu & Ye, 2019) consider similar problems to (10), but with the RKHS H replaced with an RKBS.
The kernel learning problem for attention, however, is different from (10) in that, as we discussed in the previous section, we need to predict a response z ij (i.e., the attention logit) for every pair ( t i , s j ) .
This motivates a generalization of the classic class of kernel learning problems that operates on pairs of input spaces.
We discuss this generalization now.
Deﬁnition 3 (Binary kernel learning problem - regularized empirical risk minimization) .
Let X and Y be nonempty sets, and B X and B Y RKBS’s on X and Y , respectively.
Let h· , ·i B X ×B Y :
B X × B Y → R be a bilinear mapping on the two RKBS’s.
Let Φ X : X → F X and Φ Y :
Y → F Y be ﬁxed feature mappings with the property that h Φ X ( x i ) , Φ Y ( y i )
i F X ×F Y
[Equation] [Equation]
Say { x 1 , . . .
, x n x } , x i ∈ X , { y 1 , . . .
, y n y } , y j ∈ Y , and { z ij } i =1 ,...,n x ; j =1 ,...,n y , z ij ∈ R is a ﬁnite dataset where a response z ij is deﬁned for every ( i, j ) pair of an x i and a y j .
Let L :
X × Y × R × R → R be a loss function that is convex for ﬁxed ( x i , y j , z i,j ) , and R X : [0 , ∞ )
→ R and R Y : [0 , ∞ ) → R be convex, strictly increasing regularization functions.
A binary empirical risk minimization kernel learning problem for learning on a pair of RKBS’s takes the form
[Equation]
where λ X and λ Y are again scaling constants.
Remark 8 .
The idea of a binary kernel problem that operates over pairs of two sets is not wholly new: there is prior work both in the collaborative ﬁltering (Abernethy et al., 2009) and tensor kernel method (Tao et al., 2005; Kotsia & Patras, 2011;
He et al., 2017) literatures.
Our problem and results are new in the generalization to Banach rather than Hilbert spaces: as prior work in the RKBS literature (Micchelli et al., 2004; Zhang & Zhang, 2012; Xu & Ye, 2019, etc.) notes, RKBS learning problems are distinct from RKHS ones in their additional nonlinearity and/or nonconvexity.
An extension of binary learning problems to Banach spaces is thus motivated by the Transformer setting, where a kernel method is in a context of a nonlinear and nonconvex deep neural network, rather than as a shallow learner like an SVM or matrix completion.
For more discussion, see Appendix A.
Virtually all classic kernel learning methods ﬁnd solutions whose forms are speciﬁed by so-called representer theorems .
Representer theorems state that the solution to a regularized empirical risk minimization problem over a reproducing kernel space can be expressed as a linear combination of evaluations of the reproducing kernel against the dataset.
Classic solutions to kernel learning problems thus reduce to ﬁnding the coefﬁcients of this linear combination.
Representer theorems exist in the literature for RKHS’s (Kimeldorf & Wahba, 1971; Schölkopf et al., 2001; Argyriou et al., 2009), RKKS’s
(Ong et al., 2004; Oglic & Gärtner, 2018), and RKBS’s
(Zhang et al., 2009; Zhang &
Zhang, 2012; Song et al., 2013; Fasshauer et al., 2015; Xu & Ye, 2019; Lin et al., 2019).
Fasshauer et al.
(2015, Theorem 3.2), Xu & Ye (2019, Theorem 2.23), and Lin et al.
(2019, Theorem 4.7) provide representer theorems for RKBS learning problems.
However, their theorems only deal with learning problems where datapoints come from only one of the sets on which the reproducing kernel is deﬁned (i.e., only X but not Y ), which means the solution sought is an element of only one of the Banach spaces (e.g., f : X → R , f ∈ B X ).
Here, we state and prove a theorem for the more-relevant-to-Transformers binary case presented in Deﬁnition 3.
Theorem 1.
Suppose we have a kernel learning problem of the form in (11) .
Let κ : X × Y → R be the reproducing kernel of the pair of RKBS’s B X and B Y satisfying Deﬁnitions 1 and 2.
Then, given some conditions on B X and B Y (see Appendix B), the regularized empirical risk minimization problem (11) has a unique solution pair ( f ∗ , g ∗ ) , with the property that
[Equation]
where ι ( f ) (resp.
ι ( g ) ) denotes the Gâteaux derivative of the norm of f (resp.
g )
with the convention that ι (0) (cid:44) 0 , and where ξ i , ζ j ∈ R .
Proof.
See Appendix B.
5. A NEW APPROXIMATE KERNEL LEARNING PROBLEM AND UNIVERSAL APPROXIMATION THEOREM
The downside of ﬁnding solutions to kernel learning problems like (10) or (11) of the form (12) as suggested by representer theorems is that they scale poorly to large datasets.
It is well-known that for an RKHS learning problem, ﬁnding the scalar coefﬁcients by which to multiply the kernel evaluations takes time cubic in the size of the dataset, and querying the model takes linear time.
The most popular class of approximation techniques are based on the so-called Nyström method, which constructs a low-rank approximation of the kernel Gram matrix and solves the problem generated by this approximation (Williams & Seeger, 2001).
A recent line of work (Gisbrecht & Schleif, 2015;
Schleif & Tino, 2017; Oglic & Gärtner, 2019) has extended the Nyström method to RKKS learning.
In this section, we characterize the Transformer learning problem as a new class of approximate kernel methods – a “distillation” approach, one might call it.
We formally state this idea now.
Proposition 2 (Parametric approximate solutions of binary kernel learning problems) .
Consider the setup of a binary kernel learning problem from Deﬁnition 3.
We want to ﬁnd approximations to the solution pair ( f ∗ , g ∗ ) .
In particular, we will say we want an approximation
ˆ κ : X ×Y → R such that
[Equation]
for all x ∈ X and y ∈ Y .
Comparing (13) to (6) suggests a solution: to learn a function ˆ κ that approximates κ .
In particular, (6) suggests learning explicit approximations of the feature maps, i.e.,
[Equation]
In fact, it turns out that the Transformer query-key mapping (1) does exactly this.
That is, while the Transformer kernel calculation outlined in Propostion 1 is ﬁnite-dimensional, it can in fact approximate the potentially inﬁnite-dimensional optimal solution ( f ∗ , g ∗ ) characterized in Theorem 1.
This fact is proved next.
Theorem 2.
Let X ⊂ R d t and Y ⊂ R d s be compact; t ∈ X , s ∈ Y ; and let q ‘ :
X → R and k ‘ : Y → R for ‘ = 1 , . .
, d be two-layer neural networks with m hidden units.
Then, for any continuous function F : X × Y → R and (cid:15) > 0 , there are integers m, d > 0 such that
[Equation]
Proof.
See Appendix C.
We now outline how Theorem 2 relates to Transformers.
If we concatenate the outputs of the two-layer neural networks { q ‘ } d‘ =1 and { k ‘ } d‘ =1 into d -dimensional vectors q : R d t → R d and k : R d s → R d , then the dot product q ( t ) T k ( s ) denoted by the sum in (14) can approximate any real-valued continuous function on X × Y .
Minus the usual caveats in applications of universal approximation theorems (i.e., in practice the output elements share hidden units rather than having independent ones), this dot product is exactly the computation of the attention logits a ij , i.e.,
F ( t , s ) ≈ log κ ( t , s ) for the F in (14) and the κ in (9) up to a scaling constant √ d .
Since the exponential mapping between the attention logits and the exponentiated query-key kernel used in Transformers is a one-to-one mapping, if we take F ( t , s )
= log (cid:10) f ∗ Φ Y ( s ) , g ∗ Φ X ( t ) (cid:11) B
X ×B Y , then we can use a Transformer’s dot-product attention to approximate the optimal solution to any
RKBS solution arbitrarily well.
The core idea of an attention-based deep neural network is then to learn parametric representations of q ‘ and k ‘ via stochastic gradient descent.
Unlike traditional representer-theorem-based learned functions, training time of attention-based kernel machines like deep Transformers (generally, but with no guarantees) scale sub-cubically with dataset size, and evaluation time stays constant regardless of dataset size.
I S THE EXPONENTIATED DOT PRODUCT ESSENTIAL TO T RANSFORMERS ?
[Table]
We study modiﬁcations of the Transformer with several kernels used in classic kernel machines.
We train on two standard machine translation datasets and two standard sentiment classiﬁcation tasks.
For machine translation, IWSLT14 DE-EN is a relatively small dataset, while WMT14 EN-FR is a considerably larger one.
For sentiment classiﬁcation, we consider SST-2 and SST-5.
We retain the standard asymmetric query and key feature mappings, i.e., q = W Q t and k = W K s , and only modify the kernel κ : R d × R d → R ≥ 0 .
In the below, τ > 0 and γ ∈ R are per-head learned scalars.
Our kernels of interest are:
the (scaled) exponentiated dot product (EDP), κ ( q , t ) =
exp( q T k / √ d ) , i.e., the standard Transformer kernel; 2.
the radial basis function (RBF) kernel, κ ( q , t ) = exp( k − τ / √ d ( q − k ) k 22 ) , where k · k 2 is the standard 2-norm.
It is well-known that the RBF kernel is a normalized version of the exponentiated dot-product, with the normalization making it translation-invariant; 3.
the vanilla L2 distance, κ ( q , t ) =
τ / √ d k q − k k 2 ; 4.
an exponentiated version of the intersection kernel, κ ( q , t ) = exp( P d‘ =1 min( q ‘ , k ‘ )) .
The symmetric version of the intersection kernel was popular in kernel machines for computer
vision applications (Barla et al., 2003; Grauman & Darrell,
Maji et al., 2008, etc
, and is usually characterized as having an associated RKHS that is a subspace of the function space
L 2 (i
e., it is inﬁnite-dimensional in the sense of having a feature space of continuous functions, as opposed to the inﬁnite-dimensional inﬁnite series of the EDP and RBF kernels); 5.
a quadratic polynomial kernel, κ ( q , k ) =
( 1 / √ d q T k +
γ ) 2 .
Full implementation details are provided in Appendix D.
Results for machine translation are presented in Table 1.
Several results stand out.
First, the exponentiated dot product, RBF, and exponentiated intersection kernels, which are said to have inﬁnite-dimensional feature spaces, indeed do perform better than kernels with lower-dimensional feature maps such as the quadratic kernel.
In fact, the RBF and EDP kernels perform about the same, suggesting that a deep Transformer may not need the translation-invariance that makes the RBF
[Table]
kernel preferred to the EDP in classic kernel machines.
Intriguingly, the (unorthodox) exponentiated intersection kernel performs about the same as the two than the EDP and RBF kernels on IWSLT
DE-EN, but slightly worse on WMT14 EN-FR.
As mentioned, the EDP and RBF kernels have feature spaces of inﬁnite series, while the intersection kernel corresponds to a feature space of continuous functions.
On both datasets, the quadratic kernel performs slightly worse than the best inﬁnite-dimensional kernel, while the L2 distance performs signiﬁcantly worse.
Results for sentiment classiﬁcation appear in Table 2.
Unlike the machine translation experiments, the inﬁnite-dimensional kernels do not appear strictly superior to the ﬁnite-dimensional ones on this task.
In fact, the apparent loser here is the exponentiated intersection kernel, while the L2 distance, which performed the worst on machine translation, is within a standard deviation of the top-performing kernel.
Notably, however, the variance of test accuracies on sentiment classiﬁcation means that it is impossible to select a statistically signiﬁcant “best” on this task.
It is possible that the small inter-kernel variation relates to the relative simplicity of this problem (and relative smallness of the dataset) vs. machine translation: perhaps an inﬁnite-dimensional feature space is not needed to obtain
Transformer-level performance on this learning problem.
It is worth noting that the exponentiated dot product kernel (again, the standard Transformer kernel) is a consistent high performer.
This may be experimental evidence for the practical usefulness of the universal approximation property they enjoy (c.f. Theorem 2).
The relatively small yet statistically signiﬁcant performance differences between kernels is reminiscent of the same phenomenon with activation functions (ReLU, ELU, etc.) for neural nets.
Moreover, the wide inter-kernel differences in performance for machine translation, compared against the much smaller performance differences on the SST sentiment analysis tasks, demonstrates an opportunity for future study on this apparent task- and dataset-dependency.
As a whole, these results suggest that kernel choice may be an additional design parameter for Transformer networks.
C ONCLUSION
In this paper, we drew connections between classic kernel methods and the state-of-the-art Transformer networks.
Beyond the theoretical interest in developing new RKBS representer theorems and other kernel theory, we gained new insight into what may make Transformers work.
Our experimental results suggest that the inﬁnite dimensionality of the Transformer kernel makes it a good choice in application, similar to how the RBF kernel is the standard choice for e.g. SVMs.
Our work also reveals new avenues for Transformer research.
For example, our experimental results suggest that choice of Transformer kernel acts as a similar design choice as activation functions in neural net design.
Among the new open research questions are (1) whether the exponentiated dot-product should be always preferred, or if different kernels are better for different tasks (c.f. how GELUs have recently become very popular as replacements for ReLUs in Transformers), (2) any relation between vector-valued kernels used for structured prediction (Álvarez et al., 2012) and, e.g., multiple attention heads, and (3) the extension of Transformer-type deep kernel learners to non-Euclidean data (using, e.g., graph kernels or kernels on manifolds).
A D EEP NEURAL NETWORKS LEAD TO B ANACH SPACE ANALYSIS
Examining the kernel learning problem (11), it may not immediately clear why the reproducing spaces on X and Y need be Banach spaces rather than Hilbert spaces.
Suppose for example that we have two RKHS’s H X and H Y on X and Y , respectively.
Then, we can take their tensor product
H X ⊗ H Y as an RKHS on X × Y , with associated reproducing kernel κ X×Y ( x 1 ⊗ y 1 , x 2 ⊗ y 2 ) =
κ X ( x 1 , x 2 ) κ Y ( y 1 , y 2 ) , where κ X and κ Y are the reproducing kernels of H X and H Y , respectively, and x 1 ⊗ y 1 , x 2 ⊗ y 2 ∈ X ⊗ Y .
The solutions to a regularized kernel problem like (11) would then be drawn from H X and H Y .
This setup is similar to those studied in, e.g., Abernethy et al.
He et al.
In a shallow kernel learner like an SVM, the function in the RKHS can be characterized via its norm.
Representer theorems allow for the norm of the function in the Hilbert space to be calculated from the scalar coefﬁcients that make up the solution.
On the other hand, for a Transformer layer in a multilayer neural network, regularization is usually not done via norm penalty as shown in (11).
In most applications, regularization is done via dropout on the attention weights a ij as well as via the implicit regularization obtained from subsampling the dataset during iterations of stochastic gradient descent.
While dropout has been characterized as a form of weight decay (i.e., a variant of p-norm penalization) for linear models (Baldi & Sadowski, 2013; Wager et al., 2013, etc.), recent work has shown that dropout induces a more complex regularization effect in deep networks (Helmbold &
Long, 2017; Arora et al., 2020, etc.).
Thus, it is difﬁcult to characterize the norm of the vector spaces we are traversing when solving the general problem (11) in the context of a deep network.
This can lead to ambiguity as to whether the norm being regularized as we traverse the solution space is a
Hilbert space norm.
If f and g are inﬁnite series or L p functions, for example, their resident space is only a Hilbert space if the associated norm is the ‘ 2 or L 2 norm.
This motivates the generalization of kernel learning theorems to the general Banach space setting when in the context of deep neural networks.
B P ROOF OF T HEOREMB. P RELIMINIARIES
To prove this theorem, we ﬁrst need some results and deﬁnitions regarding various properties of
Banach spaces (Megginson, 1998).
These preliminaries draw from Xu & Ye (2019) and Lin et al.
Two metric spaces ( M , d M ) and ( N , d N ) are said to be isometrically isomorphic if there exists a bijective mapping T : M → N , called an isometric isomorphism, such that for all m ∈ M , d N ( T ( m )) =
d M ( m ) (Megginson, 1998, Deﬁnition 1.4.13).
The dual space of a vector space V over a ﬁeld F , which we will denote V ∗ , is the space of all continuous linear functionals on V , i.e.,
[Equation]
and continuous } .
[Equation]
A normed vector space V is reﬂexive if it is isometrically isomorphic to V ∗∗ , the dual space of its dual space (a.k.a.
its double dual).
For a normed vector space V , the dual bilinear product , which we will denote h· , ·i V (i.e., with only one subscript, as opposed to e.g., h· , ·i B X ×B Y ), is deﬁned on V and V ∗ as h f, g i V (cid:44) g
( f ) for f ∈ V , g ∈ V ∗ .
Given a normed vector space V and its dual space V ∗ , let U ⊆ V and W ⊆ V ∗ .
The annihilator of
U in V ∗ and the annihilator of W in V , denoted U ⊥ and ⊥ W respectively, are (Megginson, 1998,
Deﬁnition 1.10.14)
[Equation]
A normed vector space V is called strictly convex if k tv 1 + (1 − t ) v 2 k V < 1 whenever k v 1 k V =
k v 2 k V = 1 , 0 < t < 1 , where v 1 , v 2 ∈ V and k · k V denotes the norm of V (Megginson, 1998,
Deﬁnition 5.1.1; citing Clarkson, 1936 and Akhiezer & Krein, 1962).
A nonempty subset A of a metric space ( M , d M ) is called a Chebyshev set if, for every element m ∈ M , there is exactly one
element c ∈
A such that d M ( m, c ) =
d M ( m, A ) (Megginson, 1998,
Deﬁnition 5.1.17) (where recall the distance between a point m and a set A in a metric space is equal to inf
c ∈A d M ( m, c ) ).
If a normed vector space V is reﬂexive and strictly convex, then every nonempty closed convex subset of V is a Chebyshev set (Megginson, 1998, Corollary 5.1.19; citing
Day, 1941).
For a normed vector space V and v, w ∈ V , the Gâteaux derivative of the norm (Megginson, 1998,
Deﬁnition 5.4.15) at v in the direction of w is deﬁned as
[Equation]
If the Gâteaux derivative of the norm at v in the direction of w exists for all w ∈ V , then k · k V is said to be Gâteaux differentiable at v .
A normed vector space V is called Gâteaux differentiable or smooth if its norm is Gâteaux differentiable at all v ∈ V (Megginson, 1998, Corollary 5.4.18).
The smoothness of a normed vector space V implies that, if we deﬁne a “norm operator” ρ on V , ρ ( v ) (cid:44) k v k V , then for each v ∈ V \ { 0 } , there exists a continuous linear functional d G ρ ( v ) on V such that (Xu & Ye, 2019, p. 24)
[Equation]
for all w ∈ V .
Since the Gâteaux derivative of the norm is undeﬁned at 0, following Xu & Ye (2019, Equation 2.16);
Lin et al.
(2019, p. 20); etc., we deﬁne a regularized Gâteaux derivative of the norm operator on V , ι ( v ) (cid:44) (cid:26)
[Equation]
G
[Equation]
v ) when v 6 = when v =(B.2) for v ∈ V .
Given two vector spaces V and W deﬁned over a ﬁeld F , the direct sum , denoted V ⊕ W , is the vector space with elements ( v, w )
∈ V ⊕ W for v ∈ V , w ∈ W with the additional structure
[Equation]
c
[Equation]
If V and W are normed vector spaces with norms k · k V and k · k W , respectively, then we will say
V ⊕ W has the norm
[Equation]
Megginson (1998, Deﬁnition 1.8.1) calls (B.3) a “1-norm” direct sum norm, but notes that other norm-equivalent direct sum norms such as a 2-norm and inﬁnity-norm are possible.
Some other useful facts about direct sums are:
• if V and W are both strictly convex, then V ⊕ W is strictly convex (Megginson, 1998, Theorem 5.1.23); • if V and W are both reﬂexive, then V ⊕W is reﬂexive (Megginson, 1998, Corollary 1.11.20); • and if V and W are both smooth
, then V ⊕ W is smooth (Megginson, 1998, Theorem 5.4.22).
An element v of a normed vector space V is said to be orthogonal (or Birkhoff-James orthogonal) to another element w ∈ V if (Birkhoff, 1935; James, 1947)
[Equation]
If W ⊆ V , then we say v ∈ V is orthogonal to W if v is orthogonal to all w ∈ W .
Now we can state a lemma regarding orthogonality in RKBS’s.
Lemma 1 (Xu & Ye, 2019, Lemma 2.21) .
If the RKBS B is smooth, then f ∈ B is orthogonal to g ∈ B if and only if h g, ι ( f ) i B = 0 , where h· , ·i B means the dual bilinear product as given in (B.1) and ι is the regularized Gâteaux derivative from (B.2) .
Also, an f ∈ B \ { 0 } is orthogonal to a subspace N ⊆ B if and only if h h, ι ( f ) i B = 0 for all h ∈ N .
B. M INIMUM - NORM INTERPOLATION ( OPTIMAL RECOVERY )
Following Fasshauer et al.
; Xu & Ye (2019); Lin et al.
(2019), we ﬁrst prove a representer theorem for a simpler problem – that of perfect interpolation while minimizing the norm of the solution – before proceeding to the representer theorem for the empirical risk minimization problem (11) in the next section.
Deﬁnition 4 (Minimum-norm interpolation in a pair of RKBS’s) .
Let X and Y be nonempty sets, and
B X and B Y RKBS’s on X and Y , respectively.
Let h· , ·i B X ×B Y :
B X ×B Y → R be a bilinear mapping on the two RKBS’s,
Φ X : X → F X and Φ Y :
Y → F Y .
Let κ ( x i , y i )
=
h
Φ X ( x ) , Φ Y ( y )
i F =
h f Φ Y ( y i ) , g Φ X ( x i ) i
B
X ×B
Y be a reproducing kernel of X and Y satisfying Deﬁnitions 1 and 2.
Say { x 1 , . . .
, x n x } , x i ∈ X , { y 1 , . . .
, y n y } , y i ∈ Y , and { z ij } i
=1 ,...,n x ; j =1 ,...,n y , z ij ∈ R is a ﬁnite dataset where a response z ij is deﬁned for every ( i, j ) pair of an x i and a y j .
The minimum-norm interpolation problem is
[Equation]
such that ( f, g ) ∈ N X , Y , Z (B.4) where
[Equation]
∀ i, j } .
(B.5)
To discuss the solution of (B.4), we ﬁrst need to establish the condition for the existence of a solution.
The following is a generalization of a result from Section 2.6 of Xu & Ye (2019).
Lemma 2.
If the set { κ ( x i , · ) } n x i =1 is linearly independent in B Y , the set { κ ( · , y j ) } n y j =1 is linearly independent in B X , and the bilinear mapping h· , ·i B X ×B Y :
B X × B Y → R is nondegenerate, then
N X , Y , Z (B.5) is nonempty.
Proof.
From the deﬁnition of κ (3) and the bilinearity of h· , ·i B X ×B Y , we can write that
D f, n
[Equation]
i =
c i κ ( x i , · ) E B X
[Equation]
Y = n x X i
=
c i h f, κ ( x i , · ) i B X ×B Y = n x X i
=
c
i f ( x i ) for all f ∈ B X for c i ∈ R , and that
D n y X
j
=
c
[Equation]
κ ( · , y
[Equation]
) , g E
[Equation]
n
[Equation]
X
[Equation]
c
[Equation] [Equation]
, y
[Equation]
) , g
[Equation]
B X ×B Y = n y X
j = c j g ( y j ) for all g ∈ B Y for c j ∈ R .
This means that n x X i
=
c i κ ( x i , · ) =
if and only if n x X i
=
c i f ( x i ) = 0 for all f ∈ B X and n y X j = c
j κ ( · , y j ) =
if and only if n y X
j = c j g ( y j ) = 0 for all g ∈ B Y .
This shows that linear independence of { κ ( x i , · ) } n x i =1 and { κ ( · , y j ) } n y j =1 imply linear independence of { f ( x i ) } n x i =1 and { g ( y j ) } n y j =1 , respectively.
Then, considering the nondegeneracy of the bilinear mapping h· , ·i B X ×B Y , we can say that
[Equation]
if and only if
[Equation]
From this, we can see that linear independence of { κ ( x i , · ) } n x i =1 and { κ ( · , y j ) } n y j =1 , and the nondegeneracy of h· , ·i B X ×B Y ensure the existence of at least one ( f ∗ , g ∗ ) pair in N X , Y , Z .
Now we can prove a lemma characterizing the solution to (B.4).
Lemma 3.
Consider the minimum-norm interpolation problem from Deﬁnition 4.
Assume that B X and B Y are smooth, strictly convex, and reﬂexive, 2 and that { κ ( x i , · ) } n x i =1 and { κ ( · , y j ) } n y j =1 are linearly independent.
Then, (B.4) has a unique solution pair ( f ∗ , g ∗ ) , with the property that
[Equation]
where ι ( · ) is the regularized Gâteaux derivative as deﬁned in (B.2) , and where ξ i , ζ j ∈ R .
Proof.
The existence of a solution pair is given by the linear independence of { κ ( x i , · ) } n x i =1 and { κ ( · , y j ) } n y j =1 and Lemma 2.
Uniqueness of the solution will be shown by showing that N X , Y , Z is closed and convex and a subset of a strictly convex and reﬂexive set, which ensures it is a Chebyshev set.
Since B X and B Y are strictly convex and reﬂexive, their direct sum B X ⊕ B Y is strictly convex and reﬂexive, as we noted in section B.1.
Now we analyze N X , Y , Z .
We ﬁrst show convexity.
Pick any ( f, g ) , ( f 0 , g 0 ) ∈ N X , Y , Z and t ∈ (0 , 1) .
Then note that for any ( x i , y j , z i,j ) ,
[Equation]
thus showing that N X , Y , Z is convex.
Closedness may be shown by the strict convexity of its superset
B X ⊕ B Y and the continuity of h· , ·i B 1 ×B 2 .
Thus, the closed and convex N X , Y , Z ⊆
B X ⊕ B Y is a
Chebyshev set, implying a unique ( f ∗ , g ∗ )
∈ N X , Y , Z with
[Equation]
Now we characterize this solution ( f ∗ , g ∗ ) .
Similar to proofs of the classic RKHS representer theorem (Schölkopf et al., 2001) and those of earlier RKBS representer theorems (Xu & Ye, 2019;
Lin et al., 2019, etc.), we approach this via orthogonal decomposition.
Consider the following set of function pairs ( f, g ) that map all data pairs ( x i , y j ) to 0 :
[Equation]
We can see that N X , Y , 0 is closed under addition and scalar multiplication, making it a subspace of
B X ⊕ B Y .
Taking our optimal ( f ∗ , g ∗ ) , we can see that
[Equation]
thus showing that ( f ∗ , g ∗ ) is orthogonal to the subspace N X , Y , 0 .
Consider the left and right preimages of N X , Y , 0 under h· , ·i B X ×B Y :
[Equation]
Since h· , g i − 1 B X ×B Y [ N X , Y , 0 ] ⊆
B X and h f, ·i − 1 B X ×B Y [ N X , Y , 0 ] ⊆
B Y , we can consider them as normed vector spaces with norms k · k B X and k · k B Y , respectively.
From (B.7) and the deﬁnition of the direct sum norm (B.3),
[Equation]
for
[Equation]
f
[Equation] [Equation]
for
[Equation]
] , for arbitrary
[Equation]
We can then use (B.8) and Lemma 1 to say h f, ι ( f
[Equation]
) i B X = 0 for all f ∈ h· , g i − 1 B X ×B Y [ N X , Y , 0 ] , for arbitrary g ∈ B Y
[Equation]
g,
[Equation]
g
[Equation] [Equation]
= 0 for all g ∈ h f, ·i − 1 B X ×B Y [ N X , Y , 0 ] , for arbitrary f ∈ B X which means
[Equation]
From (B.9a) and (3a)-(3b), f ∗ ∈ [
[Equation]
Y
[Equation]
g
[Equation]
[Equation] [Equation]
=
[Equation]
h
[Equation] [Equation]
=
[Equation]
= ⊥ span { κ ( x i , · ); i = 1 , . . . , n x } .
(B.10)
And from (B.9b) and (3c)-(3d), g ∗ ∈ [ f ∈B X
[Equation]
=
[Equation]
span { κ ( ·
[Equation]
y = 1 ,
[Equation]
n
[Equation]
(B.11)
Combining (B.9a) and (B.10), we get ι ( f ∗ ) ∈ (cid:0) ⊥ span { κ ( x i , · ); i = 1 , . . . , n x } (cid:1) ⊥
= span { κ ( x i , · ); i = 1 , . . . , n x } (B.12) and by combining (B.9b) and (B.11), we get
[Equation]
where in both (B.12) and (B.13) we use Proposition 2.6.6 of Megginson (1998) regarding properties of annihilators.
From (B.12) and (B.13) we can write that there exist two sets of parameters ξ 1 , . . . , ξ n x ∈ R and ζ 1 , . . .
, ζ n y ∈ R such that
[Equation]
thus proving the claim.
B. M AIN P ROOF
Before beginning the proof, we state the following lemma regarding the existence of solutions of convex optimization problems on Banach spaces:
Lemma 4 (Ekeland & Témam, 1999, Chapter II, Proposition 1.2) .
Let B be a reﬂexive Banach space and S a closed, convex, and bounded (with respect to k · k B ) subset of B .
Let f : S
→ R ∪ { + ∞} be a convex function with a closed epigraph (i.e., it satisﬁes the condition that ∀ c ∈ R ∪ { + ∞} , the set { v ∈ S : f ( v ) ≤ c } is closed).
Then, the optimization problem inf v ∈S f ( v ) has at least one solution.
Xu & Ye (2019) and Lin et al.
(2019) also reference Ekeland & Turnbull (1983) as a source for
Lemma 4.
We now restate Theorem 1 with the conditions on B X and B Y ﬁlled in.
Theorem 1, Revisited.
Suppose we have a kernel learning problem of the form in (11) .
Let κ :
X × Y → R , κ
[Equation]
i , y i ) =
h
Φ X ( x i ) , Φ
[Equation]
, g
[Equation]
X
[Equation] [Equation]
B X ×B Y be a reproducing kernel satisfying Deﬁnitions 1 and 2.
Assume that { κ ( x i , · ) } n x i =1 is linearly independent in B Y and that { κ ( · , y j ) } n y j =1 is linearly independent in B X .
Assume also that B X and B Y are reﬂexive, strictly convex, and smooth.
Then, the regularized empirical risk minimization problem (11) has a unique solution pair ( f ∗ , g ∗ ) , with the property that
[Equation]
where ξ i , ζ j ∈ R .
Proof.
As before, we begin by proving existence and uniqueness of the solution pair ( f ∗ , g ∗ ) .
We ﬁrst prove uniqueness using some basic facts about convexity.
Assume that there exist two distinct minimizers ( f ∗ 1 , g ∗ 1 ) , ( f ∗ 2 , g ∗ 2 )
∈ B X ⊕ B Y .
Deﬁne ( f ∗ 3 , g ∗ 3 ) = 1 / 2 [( f ∗ 1 , g ∗ 1 ) +
( f ∗ 2 , g ∗ 2 )] .
Then, since
B X and B Y are strictly convex, we have
[Equation]
and since R X and R Y are convex and strictly increasing,
[Equation]
and
[Equation]
Consider the regularized empirical risk minimization cost function (11)
[Equation]
where we use the shorthand
[Equation]
We have that R X ( k · k B X ) and R Y ( k · k B Y ) are both convex via identities about composition of convex functions.
The function L ( f, g ) is also convex since all the functions in the summand are convex in f and g .
Then, since we have assumed that T ( f ∗ 1 , g ∗ 1 ) =
T ( f ∗ 2 , g ∗ 2 ) , by plugging in some of the above inequalities we can write
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
B X
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
contradicting that ( f ∗ 1 , g ∗ 1 ) is a minimizer, and thus showing uniqueness of the solution.
We now prove existence via Lemma 4.
We already know that T ( · ) is convex.
From the bilinearity of h· , ·i B X ×B Y and the convexity of L , L is continuous in f and g .
Since the regularization functions R X and R Y are convex and strictly increasing, is follows that the functions R X ( k f k B X ) and R Y ( k g k B Y ) are continuous in f and g , respectively.
Thus, T ( f, g ) is continuous.
Consider the set
[Equation]
The set E is nonempty (it contains at least (0 , 0) ), and we can see that
[Equation]
showing that E is bounded.
So, by Lemma 4, we are guaranteed the existence of an optimal solution ( f ∗ , g ∗ ) .
Pick any ( f, g ) ∈
B X × B Y and consider the set
[Equation]
i.e., the set of pairs of points ( x i , y j ) along with the value that the function pair ( f, g ) maps to via the bilinear form at the pair of points ( x i , y j ) .
From Lemma 3, there exists an element ( f 0 , g 0 )
∈
B X × B Y such that ( f 0 , g 0 ) interpolates D f,g perfectly, i.e.,
[Equation]
[Equation]
n y whose Gâteaux derivatives of norms satisfy
[Equation]
Further, this element ( f 0 , g 0 ) obtains the minimum-norm interpolation of D f,g , i.e.,
[Equation]
This last fact implies
[Equation]
Therefore, the unique optimal solution ( f ∗ , g ∗ ) also satisﬁes
[Equation]
which implies that suitable parameters ξ 1 , . .
ξ n x ∈ R and ζ 1 , . .
, ζ n y ∈ R exist such that
[Equation]
proving the claim.
C P ROOF OF T HEOREM
First, we state the following well-known lemma.
Lemma 5.
For any two compact Hausdorff spaces X and Y , continuous function κ : X × Y → R , and (cid:15) > 0 , there exists an integer d > 0 and continuous functions φ ‘ : X → R , ψ ‘ : Y → R , ‘ = 1 , . . .
, d such that
[Equation]
Proof.
The product space of two compact spaces X and Y , X × Y , is of course compact by
Tychonoff’s theorem.
Consider the algebra
[Equation]
It is easy to show (i) that A is an algebra, (ii) that A is a subalgebra of the real-valued continuous functions on X × Y , and (iii) that A separates points.
Then, combining the aforementioned facts, by the Stone-Weierstrass theorem A is dense in the set of real-valued continuous functions on X ×Y .
Remark 9 .
In addition to helping us prove Theorem 2 below, Lemma 5 also serves as somewhat of an analog to Mercer’s theorem for the more general case of asymmetric, non-PSD kernels.
It is however weaker than Mercer’s theorem in that the non-PSD nature of κ means that the functions in the sum cannot be considered as eigenfunctions (with φ ‘ = ψ ‘ ) with associated nonnegative eigenvalues.
Now we proceed to the proof of Theorem 2.
Proof.
To keep the equations from becoming too cluttered, below we use q ( t ) , k ( s ) , φ ( t ) , ψ ( s ) ∈ R d as the vector concatenation of the scalar functions { q ‘ ( t ) } , { k ‘ ( s ) } , { φ ‘ ( t ) } , and { ψ ‘ ( s ) } , ‘ = 1 , . . .
, d , respectively.
All sup norms are with respect to X × Y .
Our proof proceeds similarly to the proof of Theorem 5.1 of Okuno et al.
We generalize their theorem and proof to non-Mercer kernels and simplify some intermediate steps.
First, by applying
Lemma 5, we can write that for any (cid:15) 1 , there is a d such that
[Equation]
Now we consider the approximation of φ ‘ and ψ ‘ by q ‘ and k ‘ , respectively.
By the universal approximation theorem of multilayer neural networks (Cybenko, 1989; Hornik et al., 1989; Funahashi, 1989; Attali & Pagès, 1997, etc.), we know that for any functions φ ‘ : X → R , ψ ‘ : Y → R and scalar (cid:15) 2 > 0 , there is an integer m >
such that if q ‘ : X → R and k ‘ : Y → R are two-layer neural networks with m hidden units, then
[Equation]
for all ‘ .
Now, beginning from (14), we can write
[Equation] [Equation]
where the ﬁrst inequality uses the triangle inequality and the second uses the Cauchy-Schwarz inequality.
Finally, we can combine (C.1)-(C.4) to write
[Equation]
Picking (cid:15) 1 and (cid:15) 2 appropriately, e.g. (cid:15)
= (cid:15) / 2 and (cid:15) 2 ≤
[Equation]
k
[Equation]
sup
[Equation] [Equation]
(cid:15)
[Equation]
, completes the proof.
D E XPERIMENT I MPLEMENTATION D ETAILS
Datasets The 2014 International Workshop on Spoken Language Translation (IWSLT14) machine translation dataset is a dataset of transcriptions of TED talks (and translations of those transcriptions).
We use the popular German to English subset of the dataset.
We use the 2014 “dev” set as our test set, and a train/validation split suggested in demo code for fairseq (Ott et al., 2019), where every 23rd line in the IWSLT14 training data is held out as a validation set.
The 2014 ACL Workshop on Statistical Machine Translation (WMT14) dataset is a collection of
European Union Parliamentary proceedings, news stories, and web text with multiple translations.
We use newstest2013 as our validation set and newstest2014 as our test set.
The Stanford Sentiment Treebank (SST) (Socher et al., 2013) is a sentiment analysis dataset with sentences taken from movie reviews.
We use two standard subtasks: binary classiﬁcation (SST-2) and ﬁne-grained classiﬁcation (SST-5).
SST-2 is a subset of SST-5 with neutral-labeled sentences removed.
We use the standard training/validation/testing splits, which gives splits of 6920/872/ on SST-2 and 8544/1101/2210 on SST-5.
Data Preprocessing On both translation datasets, we use sentencepiece 3 to tokenize and train a byte-pair encoding (Sennrich et al., 2016) on the training set.
We use a shared BPE vocabulary across the target and source languages.
Our resulting BPE vocabulary size is 8000 for IWSLT
DE-EN and 32000 for WMT14 EN-FR.
For SST, we train a sentencepiece BPE for each subtask separately, obtaining BPE vocabularies of size 7465 for SST-2 and 7609 for SST-5.
Models Our models are written in Pytorch (Paszke et al., 2019).
We make use of the Fairseq (Ott et al., 2019) library for training and evaluation.
In machine translation, we use 6 Transformer layers in both the encoder and decoder.
Both Transformer sublayers (attention and the two fully-connected layers) have a residual connection with the “pre-norm” (Wang et al., 2019b) ordering of Layer normalization -> Attention or FC -> ReLU -> Add residual.
We use an embedding dimension of 512 for the learned token embeddings.
For IWSLT14, the attention sublayers use 4 heads with a per-head dimension d of 128 and the fully-connected sublayers have a hidden dimension of 1024.
For WMT14, following Vaswani et al.
(2017)’s “base” model, the attention layers have 8 heads with a per-head dimension d of 64 and the fully-connected sublayers have a hidden dimension of 2048.
For SST, we use a very small, encoder-only, Transformer variant, with only two Transformer layers.
The token embedding dimension is 64, each Transformer self-attention sublayer has 4 heads with per-head dimension d of 16, and the fully-connected sublayers have a hidden dimension of 128.
To produce a sentence classiﬁcation, the output of the second Transformer layer is average-pooled over the non-padding tokens, then passed to a classiﬁcation head.
This classiﬁcation head is a two-layer neural network with hidden dimension 64 and output dimension equal to the number of classes; this output vector becomes the class logits.
Training We train with the Adam optimizer (Kingma & Ba, 2015).
Following Vaswani et al.
(2017), for machine translation we set the Adam parameters β 1 = 0 .
9 , β 2 = 0 .
On IWSLT14 DE-EN, we schedule the learning rate to begin at 0.001 and then multiply by a factor of 0.1 when the validation BLEU does not increase for 3 epochs.
FOR WMT14 EN-FR, we decay proportionally to the inverse square root of the update step using Fairseq ’s implementation.
For both datasets, we also use a linear warmup on the learning rate from 1e-7 to 0.001 over the ﬁrstupdate steps.
On IWSLT14 DE-EN, we end training when the BLEU score does not improve for 7 epochs on the validation set.
On WMT14 EN-FR, we end training after 100k gradient updates (inclusive of the warmup stage), which gives us a ﬁnal learning rate of 0.0002.
We train on the cross-entropy loss and employ label smoothing of 0.1.
We use minibatches with a maximum of about 10k source tokens on
IWSLT14 DE-EN and 25k on WMT14 EN-FR Also on WMT14, we ignore sentences with more than 1024 tokens.
For both SST subtasks, we also use a linear warmup from 1e-7 over 4000 warmup steps, but use an initial post-warmup learning rate of 0.0001.
Similar to IWSLT14, we decay the learning rate by multiplying by 0.1 when the validation accuracy does not increase for 3 epochs, and end training when the validation accuracy does not improve for 8 epochs.
Evaluation for machine translation Following Vaswani et al.
, we use beam-search decoding, with a beam length of 4 and a length penalty of 0.6, to generate sentences for evaluation.
We use sacrebleu (Post, 2018) to generate BLEU scores.
We report whole-word case-sensitive BLEU.