L ONG -T AILED L EARNING R EQUIRES F EATURE L EARNING
A BSTRACT
We propose a simple data model inspired from natural data such as text or images, and use it to study the importance of learning features in order to achieve good generalization.
Our data model follows a long-tailed distribution in the sense that some rare subcategories have few representatives in the training set.
In this context we provide evidence that a learner succeeds if and only if it identiﬁes the correct features, and moreover derive non-asymptotic generalization error bounds that precisely quantify the penalty that one must pay for not learning features.
I NTRODUCTION
Part of the motivation for deploying a neural network arises from the belief that algorithms that learn features/representations generalize better than algorithms that do not.
We try to give some mathematical ballast to this notion by studying a data model where, at an intuitive level, a learner succeeds if and only if it manages to learn the correct features.
The data model itself attempts to capture two key structures observed in natural data such as text or images.
First, it is endowed with a latent structure at the patch or word level that is directly tied to a classiﬁcation task.
Second, the data distribution has a long-tail, in the sense that rare and uncommon instances collectively form a signiﬁcant fraction of the data.
We derive non-asymptotic generalization error bounds that quantify, within our framework, the penalty that one must pay for not learning features.
We ﬁrst prove a two part result that quantiﬁes precisely the necessity of learning features within the context of our data model.
The ﬁrst part shows that a trivial nearest neighbor classiﬁer performs perfectly when given knowledge of the correct features.
The second part shows it is impossible to a priori craft a feature map that generalizes well when using a nearest neighbor classiﬁcation rule.
In other words, success or failure depends only on the ability to identify the correct features and not on the underlying classiﬁcation rule.
Since this cannot be done a priori , the features must be learned.
Our theoretical results therefore support the idea that algorithms cannot generalize on long-tailed data if they do not learn features.
Nevertheless, an algorithm that does learn features can generalize well.
Speciﬁcally, the most direct neural network architecture for our data model generalizes almost perfectly when using either a linear classiﬁer or a nearest neighbor classiﬁer on the top of the learned features.
Crucially, designing the architecture requires knowing only the meta structure of the problem, but no a priori knowledge of the correct features.
This illustrates the built-in advantage of neural networks; their ability to learn features signiﬁcantly eases the design burden placed on the practitioner.
Subcategories in commonly used visual recognition datasets tend to follow a long-tailed distribution (Salakhutdinov et al., 2011; Zhu et al., 2014; Feldman & Zhang, 2020).
Some common subcategories have a wealth of representatives in the training set, whereas many rare subcategories only have a few representatives.
At an intuitive level, learning features seems especially important on a long-tailed dataset since features learned from the common subcategories help to properly classify test points from a rare subcategory.
Our theoretical results help support this intuition.
We note that when considering complex visual recognition tasks, datasets are almost unavoidably long-tailed (Liu et al., 2019) — even if the dataset contains millions of images, it is to be expected that many subcategories will have few samples.
In this setting, the classical approach of deriving asymptotic performance guarantees based on a large-sample limit is not a fruitful avenue.
General- ization must be approached from a different point of view (c.f. Feldman (2020) for very interesting work in this direction).
In particular, the analysis must be non-asymptotic.
One of our main contribution is to derive, within the context of our data model, generalization error bounds that are non-asymptotic and relatively tight — by this we mean that our results hold for small numbers of data samples and track reasonably well with empirically evaluated generalization error.
In Section 2 we introduce our data model and in Section 3 we discuss our theoretical results.
For the simplicity of exposition, both sections focus on the case where each rare subcategory has a single representative in the training set.
Section 4 is concerned with the general case in which each rare subcategory has few representatives.
Section 5 provides an overview of our proof techniques.
Finally, in Section 6, we investigate empirically a few questions that we couldn’t resolve analytically.
In particular, our error bounds are restricted to the case in which a nearest neighbor classiﬁcation rule is applied on the top of the features — we provide empirical evidence in this last section that replacing the nearest neighbor classiﬁer by a linear classiﬁer leads to very minimal improvement.
This further support the notion that, on our data model, it is the ability to learn features that drives success, not the speciﬁc classiﬁcation rule used on the top of the features.
Related work.
By now, a rich literature has developed that studies the generalization abilities of neural networks.
A major theme in this line of work is the use of the PAC learning framework to derive generalization bounds for neural networks (e.g. Bartlett et al.
; Neyshabur et al.
Golowich et al.
Arora et al.
; Neyshabur et al.
(2018)), usually by proving a bound on the difference between the ﬁnite-sample empirical loss and true loss.
While powerful in their generality, such approaches are usually task independent and asymptotic; that is, they are mostly agnostic to any idiosyncrasies in the data generating process and need a statistically meaningful number of samples in the training set.
As such, the PAC learning framework is not well-tailored to our speciﬁc aim of studying generalization on long-tailed data distributions; indeed, in such setting, a rare subcategory might have only a handful of representatives in the training set.
After breakthrough results (e.g. Jacot et al.
Du et al.
Allen-Zhu et al.
; Ji & Telgarsky (2019)) showed that vastly over-parametrized neural networks become kernel methods (the so-called Neural Tangent Kernel or NTK) in an appropriate limit, much effort has gone toward analyzing the extent to which neural networks outperform kernel methods (Yehudai & Shamir, 2019;
Wei et al., 2019; Reﬁnetti et al., 2021; Ghorbani et al., 2019; 2020; Karp et al., 2021; Allen-Zhu &
Li, 2019; 2020; Li et al., 2020; Malach et al., 2021).
Our interest lies not in proving such a gap for its own sake, but rather in using the comparison to gain some understanding on the importance of learning features in computer vision and NLP contexts.
Analyses that shed theoretical light onto learning with long-tailed distributions (Feldman, 2020;
Brown et al., 2021) or onto speciﬁc learning mechanisms (Karp et al., 2021) are perhaps closest to our own.
The former analyses (Feldman, 2020; Brown et al., 2021) investigate the necessity of memorizing rare training examples in order to obtain near-optimal generalization error when the data distribution is long-tailed.
Our analysis differs to the extent that we focus on the necessity of learning features and sharing representations in order to properly classify rare instances.
Like us, the latter analysis (Karp et al., 2021) also considers a computer vision inspired task and uses it to compare a neural network to a kernel method, with the ultimate aim of studying the learning mechanism involved.
Their object of study (ﬁnding a sparse signal in the presence of noise), however, markedly differs from our own (learning with long-tailed distributions).
T HE D ATA M ODEL
We begin with a simple example to explain our data model and to illustrate, at an intuitive level, the importance of learning features when faced with a long-tailed data distribution.
For the sake of exposition we adopt NLP terminology such as ‘words’ and ‘sentences,’ but the image-based terminology of ‘patches’ and ‘images’ would do as well.
The starting point is a very standard mechanism for generating observed data from some underlying collection of latent variables.
Consider the data model depicted in Figure 1.
We have a vocabulary of n w = 12 words and a set of n c = 3 concepts:
V = { potato, cheese, carrots, chicken , . . . } and
C = { vegetable, dairy, meat } . [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
{+/LYZflpoYMhrByjThavVZ0uqKUPXss0suBSTPtnkpqA1TRtBpbiZGHQUyNw9ipOkkDnQ3cqKugbO1XVjdZ401vV04vfbyZY01wDyRtlK3nXiLZ2G9N95J07iPqeRcM2UEFSmtnth0czD2NrotqsL2CjJxmHT3OG52vCsKvqfK91w9Vdet7kZCzcnahvMjqcfp+GX4/Hp237Y9shL8oq8ISF5T07JZ3JGZoQRS36Qn+TX4I+
+}
+17ByvpYKf3PCNry3vxF1eZGF0=</latexit>
The 12 words are partitioned into the 3 concepts as shown on the left of Figure 1.
We also havesequences of concepts of length L = 5 .
They are denoted by c 1 , c 2 , c 3 and c (cid:48) 1 , c (cid:48) 2 , c (cid:48) 3 .
[-/LYZflpoYMhrByjThavVZ0uqKUPXss0suBSTPtnkpqA1TRtBpbiZGHQUyNw9ipOkkDnQ3cqKugbO1XVjdZ401vV04vfbyZY01wDyRtlK3nXiLZ2G9N95J07iPqeRcM2UEFSmtnth0czD2NrotqsL2CjJxmHT3OG52vCsKvqfK91w9Vdet7kZCzcnahvMjqcfp+GX4/Hp237Y9shL8oq8ISF5T07JZ3JGZoQRS36Qn+TX4I+
-]
Sequences of concepts are latent variables that generate sequences of words.
For example [ dairy , dairy , veggie , meat , veggie ] generates −→ [ cheese , butter , lettuce , chicken , leek ]
The sequence of words on the right was obtained by sampling each word uniformly at random from the corresponding concept.
For example, the ﬁrst word was randomly chosen out of the dairy concept ( butter, cheese, cream, yogurt ), and the last word was randomly chosen out of the vegetable concept ( potato, carrot, leek, lettuce .)
Sequences of words will be referred to as sentences.
The non-standard aspect of our model comes from how we use the ‘latent-variable → observeddatum’ process to form a training distribution.
The training set in Figure 1 is made of 15 sentences split into R = 3 categories.
The latent variables c (cid:48) 1 , c (cid:48) 2 , c (cid:48) 3 each generate a single sentence, whereas the latent variables c 1 , c 2 , c 3 each generate 4 sentences.
We will refer to c 1 , c 2 , c 3 as the familiar sequences of concepts since they generate most of the sentences encountered in the training set.
On the other hand c (cid:48) 1 , c (cid:48) 2 , c (cid:48) 3 will be called unfamiliar .
Similarly, a sentence generated by a familiar (resp.
unfamiliar) sequence of concepts will be called a familiar (resp.
unfamiliar) sentence.
The former represents a datum sampled from the head of a distribution while the latter represents a datum sampled from its tail.
We denote by x r,s the s th sentence of the r th category, indexed so that the ﬁrst sentence of each category is an unfamiliar sentence and the remaining ones are familiar.
Suppose now that we have trained a learning algorithm on the training set described above and that at inference time we are presented with a previously unseen sentence generated by the unfamiliar sequence of concept c (cid:48)
= [ dairy , dairy , veggie , meat , veggie ] .
To ﬁx ideas, let’s say that sentence is:
[Equation]
This sentence is hard to classify since there is a single sentence in the training set that has been generated by the same sequence of concepts, namely
[Equation]
Moreover these two sentences do not overlap at all (i.e. the i th word of x test is different from the i th word of x 1 , 1 for all i .)
To properly classify x test , the algorithm must have learned the equivalences butter ↔ cheese , yogurt ↔ butter , carrot ↔ lettuce , and so forth.
In other words, the algorithm must have learned the underlying concepts.
Nevertheless, a neural network with a well-chosen architecture can easily succeed at such a classiﬁcation task.
Consider, for example, the network depicted on Figure 2.
Each word of the input sentence, after being encoded into a one-hot-vector, goes through a multi-layer perceptron (MLP 1 on the ﬁgure) shared across words.
The output is then normalized using LayerNorm (Ba et al., 2016) to produce a representation of the word.
The word representations are then concatenated into a single vector that goes through a second multi-layer perceptron (MLP 2 on the ﬁgure).
norm
MLPnorm MLPnorm
MLPnorm MLPnorm
MLP[ cheese , butter , lettuce , chicken , leek ] <latexit sha1_base64="K6RTHw7d8GLJDGOry1WU2zlGj5g=">AAACwXicdVFdS9xAFJ3Ej9rU1tU+9mVwUUoJS1IK1TdBH3y00FUhCctk9u7udCczYeZGXEN+
[Table]
[Table]
+qG/9KZ2ssairFwbOPfdczuVMXkphMYruPX9tfWPz3db74MP2x087vd29S6srw2HItdTmOmcWpFAwRIESrksDrMglXOXz03Z+dQPGCq1+46KErGBTJSaCM3TUqHeX5jAVqi4YGnH7rUlkFtDkMEW4xZrPACw0IaUdkVeIYBzR9RIQK94qDunjiuBzUE8YCTBvXJfRgKagxv+tRr1+NIiWRVdB3IE+6epitOtF6VjzqgCFXDJrkzgqMauZQcElNEFaWSgZn7MpJA4qVoDN6mVIDT1wzJhOtHFPIV2yTzdqVli7KHKndBfO7MtZS742SyqcHGW1UGWFoPiD0aSSFDVtE6djYYCjXDjAuBHuVspnzDDuolxxwVkRdmbh40HPNO0EtZY2tMjaqN3vwRuSJnAZxy8TXQXD74PjQfzrR/8k7MLeIl/IPvlKYvKTnJBzckGGhJO/3qa34/X8M/+PX/rmQep73c5n8qz8+h8dztxe</latexit>
[Figure]
This network, if properly trained, will learn to give similar representations to words that belong to the same concept.
Therefore, if it correctly classiﬁes the train point x 1 , 1 given by (2), it will necessarily correctly classify the test point x test given by (1).
So the neural network is able to classify the previously unseen sentence x test despite the fact that the training set contains a single example with the same underlying sequence of concepts.
This comes from the fact that the neural network learns features and representations from the familiar part of the training set (generated by the head of the distribution), and uses these, at test time, to correctly classify the unfamiliar sentences (generated by the tail of the distribution).
In other words, because it learns features, the neural network has no difﬁculty handling the long-tailed nature of the distribution.
To summarize, the variables L, n w , n c , R and n spl parametrize instances of our data model.
They denote, respectively, the length of the sentences, the number of words in the vocabulary, the number of concepts, the number of categories, and the number of samples per category.
So in the example presented in Figure 1 we have L = 5 , n w = 12 , n c = 3 , R = 3 and n spl = 5 (four familiar sentences and one unfamiliar sentence per category).
The vocabulary V and set of concepts C are discrete sets with |V| = n w and |C| = n c , rendered as V = { 1 , . .
. , n w } and C = { 1 , . .
. , n c } for concreteness.
A partition of the vocabulary into concepts, like the one depicted at the top of Figure 1, is encoded by a function ϕ : V → C that assigns words to concepts.
We require that each concept contains the same number of words, so that ϕ satisﬁes
[Equation]
all [Equation] and we refer to such a function ϕ : V → C satisfying (3) as equipartition of the vocabulary.
The set Φ = { All functions ϕ from V to C that satisfy (3) } denotes the collection of all such equipartitions, while the data space and latent space are denoted
X = V L and
Z = C L , respectively.
Elements of X are sentences of L words and they take the form x = [ x 1 , x 2 , . . .
, x L ] , while elements of Z take the form c =
[ c 1 , c 2 , . .
c L ] and correspond to sequences of concepts.
In the context of this work, a feature map refers to any function ψ : X (cid:55)→ F from data space to feature space.
The feature space F can be any Hilbert space (possibly inﬁnite dimensional) and we denote by (cid:104)· , ·(cid:105) F the associated inner product.
Our analysis applies to the case in which a nearest neighbor classiﬁcation rule is applied on the top of the extracted features.
Such rule works as follow: given a test point x , the inner products (cid:104) ψ ( x ) , ψ ( y ) (cid:105) F are evaluated for all y in the training set; the test point x is then given the label of the training point y that led to the highest inner product
S TATEMENT AND D ISCUSSION OF M AIN R ESULTS
Our main result states that, in the context of our data model, features must be tailored (i.e. learned) to each speciﬁc task.
Speciﬁcally, it is not possible to ﬁnd a universal feature map ψ : X → F that performs well on a collection of tasks like the one depicted on Figure 1.
In the context of this work, a task refers to a tuple
[Equation]
that prescribes a partition of the vocabulary into concepts, R familiar sequences of concepts, and R unfamiliar sequences of concepts.
Given such a task T we generate a training set S as described in the previous section.
This training set contains R × n spl sentences split over R categories, and each category contains a single unfamiliar sentence.
Randomly generating the training set S from the task T corresponds to sampling
[Equation] from a distribution D train T deﬁned on the space X R × n spl and parametrized by the variables in (4) (the appendix provides an explicit formula for this distribution).
We measure performance of an algorithm by its ability to generalize on previously unseen unfamiliar sentences.
Generating an unfamiliar sentence amounts to drawing a sample
[Equation]
from a distribution D test T on the space X parametrized by the variables ϕ, c (cid:48) 1 , . . .
, c (cid:48)
R in (4) that determine unfamiliar sequences of concepts.
Finally, associated with every task T we have a labelling function f T : X → { 1 , . . .
, R } that assigns the label r to sentences generated by either c r or c (cid:48)
r (this function is ill-deﬁned if two sequences of concepts from different categories are identical, but this issue is easily resolved by formal statements in the appendix).
Summarizing our notations, for every task T ∈ Φ × Z 2 R we have a distribution D train T on the space
X R × n spl , a distribution D test T on the space X , and a labelling function f T .
Given a feature space F , a feature map ψ : X → F , and a task T ∈
Φ × Z 2 R , the expected generalization error of the nearest neighbor classiﬁcation rule on unfamiliar sentences is given by:
[Equation]
For simplicity, if the test point has multiple nearest neighbors with inconsistent labels in the training set (and so the arg max returns multiple training points y ), we will count the classiﬁcation as a failure for the nearest neighbor classiﬁcation rule.
We therefore replace (5) by the more formal (but more cumbersome) formula err [Equation] S [Equation]
F such that f T
[Equation]
y
[Equation]
to make this explicit.
Our main theoretical results concern performance of a learner not on a single task T but on a collection of tasks T =
{T 1 , T 2 , . .
, T N tasks } , and so we deﬁne
[Equation]
as the expected generalization error on such a collection T of tasks.
As a task refers to an element of the discrete set Φ × Z 2 R , any subset T ⊂ Φ × Z 2 R deﬁnes a collection of tasks.
Our main result concerns the case where the collection of tasks T =
Φ × Z 2 R consists in all possible tasks that one might encounter.
For concreteness, we choose speciﬁc values for the model parameters and state the following special case of our main theorem (Theorem 3 at the end of this section) —
Theorem 1.
Let L = 9 , n w = 150 , n c = 5 , R = 1000 and n spl ≥ 2 .
Let T =
Φ × Z 2 R .
Then
[Equation]
for all feature spaces F , and all feature maps ψ : X (cid:55)→ F .
In other words, for the model parameters speciﬁed above, it is not possible to design a ‘task-agnostic’ feature map ψ that works well if we are uniformly uncertain about which speciﬁc task we will face.
Indeed, the best possible feature map will fail at least 98 .
4% of the time at classifying unfamiliar sentences (with a nearest-neighbor classiﬁcation rule), where the probability is with respect to the random choices of the task, of the training set, and of the unfamiliar test sentence.
Interpretation: Our desire to understand learning demands that we consider a collection of tasks rather than a single one, for if we consider only a single task then the problem, in our setting, becomes trivial.
Indeed, assume T =
{T 1 } with T 1 = ( ϕ ; c 1 , . . .
, c R ; c (cid:48) 1 , . . .
, c (cid:48) R ) consists only of a single task.
With knowledge of this task we can easily construct a feature map ψ : X → R Ln c that performs perfectly.
Indeed, the map
[Equation]
that simply ‘replaces’ each word x (cid:96) of the input sentence by the one-hot-encoding e ϕ ( x (cid:96) ) of its corresponding concept will do.
1 A bit of thinking reveals that the nearest neighbor classiﬁcation rule associated with feature map (8) perfectly solves the task T 1 .
This is due to the fact that sentences generated by the same sequence of concepts are mapped by ψ to the exact same location in feature space.
As a consequence, the nearest neighbor classiﬁcation rule will match the unfamiliar test sentence x to the unique training sentence y that occupies the same location in feature space, and this training sentence has the correct label by construction (assuming that sequences of concepts from different categories are distinct).
To put it formally:
Theorem 2.
Given a task T ∈
Φ × Z 2 R satisfying c (cid:48) r (cid:54) =
c (cid:48) s and c (cid:48)
r (cid:54) =
c s for all r (cid:54) =
s , there exists a feature space F and a feature map ψ : X (cid:55)→ F such that err ( F , ψ, T )
= 0 .
Consider now the case where
T =
{T 1 , T 2 } consists of two tasks.
According to Theorem 2 there exists a ψ that perfectly solves T 1 , but this ψ might perform poorly on T 2 , and vice versa.
So, it might not be possible to design good features if we do not know a priori which of these tasks we will face.
Theorem 1 states that, in the extreme case where T contains all possible tasks, this is indeed the case — the best possible ‘task-agnostic’ features ψ will perform catastrophically on average.
In other words, features must be task-dependent in order to succeed.
To draw a very approximate analogy, imagine once again that T
= {T 1 } and that T 1 represents, say, a hand-written digit classiﬁcation task.
A practitioner, after years of experience, could hand-craft a very good feature map ψ that performs almost perfectly for this task.
If we then imagine the case T = {T 1 , T 2 } where T 1 represents a hand-written digit classiﬁcation task and T 2 represents, say, an animal classiﬁcation task, then it becomes more difﬁcult for a practitioner to handcraft a feature map ψ that works well for both tasks.
In this analogy, the size of the set T encodes the amount of knowledge the practitioner has about the speciﬁc tasks she will face.
The extreme choice
T =
Φ × Z 2 R corresponds to the practitioner knowing nothing beyond the fact that natural images are made of patches .
Theorem 1 quantiﬁes, in this extreme case, the impossibility of hand-crafting a feature map ψ knowing only the range of possible tasks and not the speciﬁc task itself.
In a realistic setting the collection of tasks T is smaller, of course, and the data generative process itself is more coherent than in our simpliﬁed setup.
Nonetheless, we hope our analysis sheds some light on some of the essential limitations of algorithms that do not learn features.
Finally, our empirical results (see Section 6) show that a simple algorithm that learns features does not face this obstacle.
We do not need knowledge of the speciﬁc task T in order to design a good neural network architecture, but only of the family of tasks T =
Φ × Z 2 R that we will face.
Indeed, the architecture in Figure 2 succeeds at classifying unfamiliar test sentences more than 99% of the time.
This probability, which we empirically evaluate, is with respect to the choice of the task, the choice of the training set, and the choice of the unfamiliar test sentence (we use the values of
L, n w , n c and R from Theorem 1, and n spl = 6 , for this experiment).
Continuing with our approximate analogy, this means our hypothetical practitioner needs no domain speciﬁc knowledge beyond the patch structure of natural images when designing a successful architecture.
In sum, successful feature design requires task-speciﬁc knowledge while successful architecture design requires only knowledge of the task family.
Main Theorem: Our main theoretical result extends Theorem 1 to arbitrary values of L , n w , n c , n spl and R .
The resulting formula involves various combinatorial quantities.
We denote by (cid:0) n k (cid:1) the binomial coefﬁcients and by (cid:8) nk (cid:9) the Stirling numbers of the second kind.
Let N = { 0 , 1 , 2 , . . . } and let γ, ˆ γ : N L +1 → N be the functions deﬁned by γ ( k ) :
= (cid:80) L +1 i =
( i − 1)
k i and ˆ γ ( k ) :
= (cid:80) L +1 i =
ik i , respectively.
We then deﬁne, for 0 ≤ (cid:96) ≤ L , the sets
S (cid:96) :
= (cid:8) k ∈ N L +1 : ˆ γ ( k ) =
n w and (cid:96) ≤ γ ( k ) ≤ L (cid:9) .
We let S = S 0 , and we note that the inclusion S (cid:96) ⊂
S always holds.
Given k ∈ N L +1 we denote by
A [Equation]
L [Equation] [Equation]
iA
[Equation] all j and n [Equation] all
[Equation]
the set of k -admissible matrices.
Finally, we let f , g : S
→ R be the functions deﬁned by f
[Equation]
A i,
[Equation]
A
[Equation]
[Equation]
A i,n
[Equation]
[Equation]
and
[Equation]
) i !
[Equation]
 
[Equation]
i
[Equation]
k
[Equation]
i
[Equation]
γ
[Equation]
k )
[Equation]
[Equation]
n L − i
[Equation]


[Equation]
respectively.
With these deﬁnitions at hand, we may now state our main theorem.
Theorem 3 (Main Theorem) .
Let T =
Φ × Z 2 R .
Then
[Equation]
for all feature spaces F , all feature maps ψ : X (cid:55)→ F , and all 0 ≤ (cid:96) ≤ L .
The combinatorial quantities involved appear a bit daunting at a ﬁrst glance, but, within the context of the proof, they all take a quite intuitive meaning.
The heart of the proof involves the analysis of a measure of concentration that we call the permuted moment, and of an associated graph-cut problem.
The combinatorial quantities arise quite naturally in the course of analyzing the graph cut problem.
We provide a quick overview of the proof in Section 5, and refer to the appendix for full details.
For now, it sufﬁces to note that we have a formula (i.e. the right hand side of (9)) that can be exactly evaluated with a few lines code.
This formula provides a relatively tight lower bound for the generalization error.
Theorem 1 is then a direct consequence — plugging L = 9 , n w = 150 , n c = 5 , R = 1000 and (cid:96) =
7 in the right hand side of (9) gives the claimed 98 .
4% lower bound.
M ULTIPLE U NFAMILIAR S ENTENCES PER C ATEGORY
C a t e g
o r y C a t e g o r y C a t
[Equation]
g o r y
[Equation]
x
[Equation]
] x
[Equation]
[Equation]
pork
[Equation]
carrot ]
[Equation]
[ lettuce
[Equation]
chicken
[Equation]
potato
[Equation]
butter ] x
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table]
/wAlxQ1xxUidLm0hgJdHr98PP8344bkyDRIzH33d2Oze6N2/t3e7duXvv/oP9g4efEpZyAlPCKOPnLk6ABhFMRSAonMcccOhSOHMX73L72RJ4ErDoo1jF4IR4FgV+QLBQqouD3d+DQc92YRZEMsSCB1fPM4s6PWm7PrrKLuTEzBAavkVDhKyhLeBKeNJNhQCejZSh0IQyZnyh9uWWKgL5FrXuPblis5SL/GAH2bZ6aniTJh7JM6rDuQB+DY7MA7KAqI5YEkKblFsRj0pEq3kgqjjMARKoQGuk9NJQS0lBiJQUrtpUZ93K4LjBQGd4nUCF32RUqpaSYM6ZWJMabmAX9cixt6FfNKC3m1V1+5qLTr1KOWYCC7amstHzNsSX/1XuEqIxcmW5q2rDQgO3patLbUPkVfPdGwwu9vvjw3GxUFMwtdA39DpVF2Vse4ykIUSCUJwkljmOhSMxFwGhkPXsNIEYkwWegaXECIeQOLK4oBl6qjQe8hlXbyRQoa1HSBwmySqv11NFcp5s23Jlm81Khf/akUEUpwIisgbyU4oEQ/ltR17AgQi6UgImPFBcEZljjokqZgNFzMORBhuVhDZ8cotgjCajROC8Y+qvkafuga+EIisZrlZAKbvMJJ+5mVQ1HSH9OW64zjiotmvP3OWocG74cfCq846Vi3qUTwSXhIUhVq1dtz2zTEdKux5VGPLZ65tZ1hKz3IpZM/pXlLcVpVPeDFMzZm5PVFOYTg7fHJofJv2TgR62PeOx8cR4ZpjGK+PEeG+cGlODdOzO586Xztfut+6P7s/ur7Xr7o6OeWRsrO6fv53GEeU=</latexit>
The two previous sections were concerned with the case in which each unfamiliar sequence of concepts has a single representative in the training set.
In this section we consider the more general case in which each unfamiliar sequence of concepts has n ∗ representatives in the training set.
Figure 3 depict an example with n ∗ = 2 and n spl = 6 , meaning that each category contains 2 unfamiliar sentences and 4 familiar sentences.
The other model parameters, in this example, are L = 5 , n w = 12 , n c = 3 and R = 3 .
Using a simple union bound, inequality (9) easily extends to this situation — the resulting formula is a bit cumbersome so we present it in the appendix (see Theorem 7).
In the concrete case where L = 9 , n w = 150 , n c = 5 , R = 1000 this formula simpliﬁes to
[Equation]
all
[Equation]
and all ψ ,
[Equation]
therefore exhibiting an afﬁne relationship between the error rate and the number n ∗ of unfamiliar sentences per category.
Note that choosing n ∗ = 1 in (10) leads to a 98 .
4% lower bound on the error rate, therefore recovering the result from Theorem 1.
This lower bound then decreases by 1 . 5% with each additional unfamiliar sentence per category in the training set.
We would like to emphasize one more time the importance of non-asymptotic analysis in the long-tailed learning setting.
For example, in inequality (10), the difﬁculty lies in obtaining a value as small as possible for the coefﬁcient in front of n ∗ .
We accomplish this via a careful analysis of the graph cut problem associated with our data model.
P ROOF O UTLINE — P ERMUTED M OMENT AND O PTIMAL F EATURE M AP
The proof involves two main ingredients.
First, the key insight of our analysis is the realization that generalization in our data model is closely tied to the permuted moment of a probability distribution.
To state this central concept, it will prove convenient to think of probability distributions on X as vectors p ∈ R N with N = |X| , together with indices 0 ≤ i ≤ N − 1 given by some arbitrary (but ﬁxed) indexing of the elements of data space.
Then p i denotes the probability of the i th element of
X in this indexing.
We use S N to denote the set of permutations of { 0 , 1 , . . .
, N − 1 } and σ ∈ S N to refer to a particular permutation.
The t th permuted moment of the probability vector p ∈ R N is
[Equation]
Since (11) involves a maximum over all possible permutations, the deﬁnition clearly does not depend on the way the set X was indexed.
In order to maximize the sum, the permutation σ must match the largest values of p i with the largest values of ( i/N ) t , so the maximizing permutation simply orders the entries p i from smallest to largest.
A very peaked distribution that gives large probability to only a handful of elements of X will have large permuted moment.
Because of this, the permuted moment is akin to the negative entropy; it has large values for delta-like distributions and small values for uniform ones.
From deﬁnition (11) it is clear that 0 ≤ H t ( p ) ≤ 1 for all probability vectors p , and it is easily veriﬁed that the permuted moment is convex.
These properties, as well as various useful bounds for the permuted moment, are presented and proven in the appendix.
Second, we identify a speciﬁc feature map, ψ (cid:63) :
X → F (cid:63) , which is optimal for a collection of tasks closely related to the ones considered in our data model.
Leveraging the optimality of ψ (cid:63) on these related tasks allows us to derive an error bound that holds for the tasks of interest.
The feature map ψ (cid:63) is better understood through its associated kernel, which is given by the formula
[Equation]
Up to normalization, K (cid:63) ( x , y ) simply counts the number of equipartitions of the vocabulary for which sentences x and y have the same underlying sequence of concepts.
Intuitively this makes sense, for the best possible kernel must leverage the only information we have at hand.
We know the general structure of the problem (words are partitioned into concepts) but not the partition itself.
So to try and determine if sentences ( x , y ) were generated by the same sequence of concepts, the best we can do is to simply try all possible equipartitions of the vocabulary and count how many of them wind up generating ( x , y ) from the same underlying sequence of concepts.
A high count makes it more likely that ( x , y ) were generated by the same sequence of concepts.
The optimal kernel K (cid:63) does exactly this, and provides a good (actually optimal, see the appendix) measure of similarity between pairs of sentences.
For ﬁxed x ∈ X , the function y
(cid:55)→ K (cid:63) ( x , y ) deﬁnes a probability distribution on data space.
The connection between generalization error, permuted moment, and optimal feature map, come from the fact that sup
[Equation]
(13) and so, up to a small error 1 /R , it is the permuted moments of K (cid:63) that determine the success rate.
We then obtain the lower bound (9) by studying these moments in great detail.
A simple union bound is then used to obtain inequalities such as (10).
E MPIRICAL R ESULTS
We conclude by presenting empirical results that complement our theoretical ﬁndings.
The full details of these experiments (training procedure, hyperparameter choices, number of experiments ran to estimate the success rates, and standard deviations of these success rates), as well as additional experiments, can be found in Appendix E. Codes are available at https://anonymous.4open.
science
r
Long_Tailed_Learning_Requires_Feature_Learning-
17C
Parameter Settings.
We consider ﬁve parameter settings for the data model depicted in Figure 3.
Each setting corresponds to a column in Table 1.
In all ﬁve settings, we set the parameters
L = 9 , n w = 150 , n c = 5 and R = 1000 to the values for which the error bound (10) holds.
We choose values for the parameters n spl and n ∗ so that the i th column of the table corresponds to a setting in which the training set contains 5 familiar and i unfamiliar sentences per category.
Recall that n spl is the total number of samples per category in the training set.
So the ﬁrst column of the table corresponds to a setting in which each category contains 5 familiar sentences and 1 unfamiliar
[Table]
sentence, whereas the last column corresponds to a setting in which each category contains 5 familiar sentences and 5 unfamiliar sentences.
Algorithms.
We evaluate empirically seven different algorithms.
The ﬁrst two rows of the table correspond to experiments in which the neural network in Figure 2 is trained with SGD and constant learning rate.
At test time, we consider two different strategies to classify test sentences.
The ﬁrst row of the table considers the usual situation in which the trained neural network is used to classify test points.
The second row considers the situation in which the trained neural network is only used to extract features (i.e. the concatenated words representation right before MLP2).
The classiﬁcation of test points is then accomplished by running a nearest neighbor classiﬁer on these learned features.
The third (resp. sixth) row of the table shows the results obtained when running a nearest neighbor algorithm (resp.
SVM) on the features ψ (cid:63) of the optimal feature map.
By the kernel trick, these algorithms only require the values of the optimal kernel (cid:104) ψ (cid:63) ( x ) , ψ (cid:63) ( y ) (cid:105) F (cid:63) , computed via (12), and not the features ψ (cid:63) themselves.
The fourth (resp. seventh) row shows results obtained when running a nearest neighbor algorithm (resp.
SVM) on features extracted by the simplest possible feature map, that is
[Equation]
where e x (cid:96) denotes the one-hot-encoding of the (cid:96) th word of the input sentence.
Finally, the last row considers a SVM with Gaussian Kernel (also called RBF kernel).
Results.
The ﬁrst two rows of the table correspond to algorithms that learn features from the data; the remaining rows correspond to algorithms that use a pre-determined (not learned) feature map.
Table 1 reports the success rate of each algorithm on unfamiliar test sentences.
A crystal-clear pattern emerges.
Algorithms that learn features generalize almost perfectly, while algorithms that do not learn features catastrophically fail.
Moreover, the speciﬁc classiﬁcation rule matters little.
For example, replacing MLP2 by a nearest neighbor classiﬁer on the top of features learned by the neural network leads to equally accurate results.
Similarly, replacing the nearest neighbor classiﬁer by a SVM on the top of features extracted by ψ (cid:63) or ψ one − hot leads to almost equally poor results.
The only thing that matters is whether or not the features are learned.
Finally, inequality (10) gives an upper bound of 0 .
015 n ∗ + 1 / 1000 on the success rate of the nearest neighbor classiﬁcation rule applied on the top of any possible feature map (including ψ (cid:63) and ψ one − hot ).
The ﬁfth row of
Table 1 compares this bound against the empirical accuracy obtained with ψ (cid:63) and ψ one − hot , and the comparison shows that our theoretical upper bound is relatively tight.
When n ∗ = 1 our main theorem states that no feature map can succeed more than 1 .
6% of the time on unfamiliar test sentences (ﬁfth row of the table).
At ﬁrst glance this appears to contradict the empirical performance of the feature map extracted by the neural network, which succeeds 99% of the time (second row of the table).
The resolution of this apparent contradiction lies in the order of operations.
The point here is to separate hand crafted or ﬁxed features from learned features via the order of operations.
If we choose the feature map before the random selection of the task then the algorithm performs poorly since it uses unlearned, task-independent features.
By contrast, the neural network learns a feature map from the training set, and since the training set is generated by the task, this process takes place after the random selection of the task.
It therefore uses taskdependent features, and the network performs almost perfectly for the speciﬁc task that generated its training set.
But by our main theorem, it too must fail if the task changes but the features do not.
Acknowledgements
Xavier Bresson is supported by NUS-R-252-000-B97-133 and A*STAR
Grant ID A20H4g2141.
Appendix
In Section A we prove a few elementary properties of the permuted moment (11).
Section B is devoted to the proof of inequality (13), which we restate here for convenience:
[Equation]
where the collection of tasks T =
Φ × Z 2 R consists in all possible tasks that one might encounter.
Inequality (14) plays a central role in our work as it establishes the connection between the generalization error, the permuted moment, and the optimal kernel K (cid:63) deﬁned by (12).
The proof is non-technical and easily accessible.
In Section C we provide the following upper bound on the permuted moment of the optimal kernel:
[Equation]
for all 0 ≤ (cid:96) ≤ L.
The proof is combinatorial in nature, and involves the analysis of a graph-cut problem.
Combining (14) and (15) establishes Theorem 3.
In Section D we consider the case in which each unfamiliar sequence of concepts has n ∗ representatives in the training set.
A simple union bound shows that, in this situation, inequality (14) becomes
[Equation]
Combining (16) and (15) then provides our most general error bound, see Theorem 7.
Inequality (10) in the main body of the paper is just a special case of Theorem 7.
Finally, in Section E, we provide the full details of the experiments.
A P ROPERTIES OF THE P ERMUTED M OMENT
The permuted moment, in Section 5, was deﬁned for probability vectors only.
It will prove convenient to consider the permuted moment of nonnegative vectors as well.
We denote by R + = [0 , + ∞ ) the nonnegative real numbers, and by R N + the vectors with N nonnegative real entries indexed from i =
to i = N − 1 .
The permuted moment of u ∈ R N + is then given by
[Equation]
where S N denote the set of permutations of { 0 , 1 , . .
, N − 1 } .
The concept of an ordering permutation will prove useful in the next lemma.
Deﬁnition 1.
σ ∈ S N is said to be an ordering permutation of u ∈ R N if
[Equation]
The lemma below shows that the permutation maximizing (17) is the one that sorts the entries
u
i from smallest to largest.
Lemma 1.
Let u ∈ R N + and let σ ∗ be an ordering permutation of u .
Then
[Equation] [Equation]
Proof.
The optimization problem (19) can be formulated as ﬁnding a pairing between the u i ’s and the ( i/N ) t ’s that maximizes the sum of the product of the pairs.
An ordering permutation of u corresponds to pairing the smallest entry of u to (0 /N )
t , the second smallest entry to (1 /N )
t , the third smallest entry to (2 /N )
t , and so forth.
This pairing is clearly optimal.
In light of the previous lemma, we see that computing the permuted moment of a vector u can be accomplished as follow: 1) sort the entries of u from smallest to largest; 2) compute the dot product between this sorted vector and the vector
[Equation]
Let us now focus on the case where u is a probability distribution.
If u is very peaked, it must have a large permuted moment since, after sorting, most of the mass concentrates on the high values of (20) located on the right.
On the contrary, if u is very spread, it must have small permuted moment since it ‘wastes’ its mass on small values of (20).
Because of this, the permuted moment is akin to the negative entropy; it has large values for delta-like distributions and small values for uniform ones.
We now show that the permuted moment is subaddiditive and one-homogeneous on R N + (as a consequence it is convex on the set of probability vectors) and we derive some elementary (cid:96) 1 and (cid:96) ∞ bounds.
We denote by (cid:107) u (cid:107) p the (cid:96) p -norm of a vector u .
In particular, if u ∈ R N + , we have (cid:107) u (cid:107)
[Equation]
(cid:88) i =
u i and
[Equation]
With this notation in hand, we can now state our lemma:
Lemma 2.
(i) H t ( u + v ) ≤ H t ( u ) + H t ( v ) for all u , v ∈ R N + .
(ii) H t ( c u ) =
c H t ( u ) for all u ∈ R N + and all c ≥ 0 .
(iii) H t ( u ) ≤ (cid:107) u (cid:107) 1 for all u ∈ R N + .
(iv) H t ( u ) ≤ Nt +1 (cid:107) u (cid:107) ∞ for all u ∈ R N + .
Proof.
Properties (i) and (ii) are obvious.
To prove (iii) and (iv), deﬁne w i
= ( i/N ) t and note that (cid:107)
w (cid:107) ∞
≤and
[Equation]
Then (iii) comes from H t ( u ) ≤ (cid:107) w (cid:107) ∞ (cid:107) u (cid:107) 1 whereas (iv) comes from H t ( u ) ≤ (cid:107) w (cid:107) 1 (cid:107) u (cid:107) ∞ .
We conclude this section with a slightly more sophisticated bound that holds for probability vectors — this bound will play a central role in Section C.
Lemma 3.
Suppose p ∈ R N + , and suppose (cid:80) Ni =1 p
i
= 1 .
Then
H t ( p ) ≤ (cid:32) 1 −
N − 1 (cid:88) i =
[Equation]
min { p i , λ
[Equation]
(cid:33)
+
[Equation]
t +for all λ ≥ 0 .
Proof.
Fix a λ ≥ 0 and deﬁne the vectors u and v as follow:
[Equation]
Note that this two vectors are non-negative and sum to p .
We can therefore use Lemma 2 to obtain
[Equation]
(cid:107)
B P ERMUTED M OMENT OF K (cid:63) AND G ENERALIZATION E RROR
This section is devoted to the proof of inequality (14).
We start by recalling a few deﬁnitions.
The vocabulary, set of concepts, data space, and latent space are
V = { 1 , . . . , n w } ,
C = { 1 , . . . , n c } ,
X = V L and
Z = C L respectively.
Elements of X are sentences of L words and they take the form x = [ x 1 , x 2 , . . .
, x L ] , while elements of Z take the form c =
[ c 1 , c 2 , . .
c L ] and correspond to sequences of concepts.
We also recall that the collection of all equipartitions of the vocabulary is denoted by
[Equation]
All functions ϕ from V to C that satisfy | ϕ − 1 ( { c } ) | =
s c for all c (cid:9) where s c := n w /n c
denote the size of the concepts.
Given ϕ ∈ Φ , we denote by ˚ ϕ : X → C the function
[Equation]
that operates on sentences element-wise.
The informal statement “the sentence x is randomly generated by the sequence of concepts c ” means that x is sampled uniformly at random from the set ˚
[Equation]
We will often do the abuse of notation of writing ˚ ϕ − 1 ( c ) instead of ˚ ϕ − 1 ( { c } ) .
We now formally deﬁne the sampling process associated with our main data model.
Sampling Process DM:
[Equation]
ii)
For r = 1 , . . .
, R : • Sample ( x r, 1 , . . .
, x r,n unf ) uniformly at random in ˚ ϕ − 1 ( c (cid:48) r ) × . . .
× ˚ ϕ − 1 ( c (cid:48) r ) .
• Sample ( x r,n unf +1 , . . .
, x r,n spl ) uniformly at random in ˚ ϕ − 1 ( c r ) × . . .
× ˚ ϕ − 1 ( c r ) .
(iii) Sample x test uniformly at random in ˚ ϕ − 1 ( c (cid:48) 1 ) .
Step (i) of the above sampling process consists in selecting at random a task T among all possible tasks.
Step (ii) consists in generating a training set 2 S ∈ X R × n spl exactly as depicted on Figure 1: each unfamiliar sequence of concept c (cid:48) r generates n unf sentences, whereas each familiar sequence of concept c r generates n fam sentences (recall that the number of samples per category is n spl = n unf + n fam ).
Finally step (iii) consists in randomly generating an unfamiliar test sentence x test ∈ X .
Without loss of generality we assume that this test sentence is generated by the unfamiliar sequence of concept c (cid:48) 1 .
We denote by (cid:37) DM the p.d.f.
of the sampling process DM.
This function is deﬁned on the sample space
[Equation]
A sample from Ω DM takes the form
[Equation]
The test sentence
[Equation]
and we have the following formula for (cid:37) DM
[Equation]
where 1 ˚ ϕ − 1 ( c r ) and 1 ˚ ϕ − 1 ( c (cid:48) r ) denote the indicator functions of the set ˚ ϕ − 1 ( c r ) and ˚ ϕ − 1 ( c r ) respec- tively.
Let us compute a few marginals of (cid:37)
DM in order to verify that it is indeed the p.d.f.
of the sampling process DM.
Writing ω
= ( T , S, x test ) , summing over the variables S and x test , and using the fact that (cid:80) x ∈X 1 ˚ ϕ
[Equation]
x
[Equation]
= |
[Equation]
c
[Equation]
| , we obtain
[Equation]
This shows that each task T is equiprobable.
Summing over the variable S gives
[Equation]
This shows that, given a task T , the test sentence x test is obtained by sampling uniformly at random from ˚ ϕ − 1 ( c (cid:48) 1 ) .
A similar calculation shows that, given a task T , the train sentence x r,s is obtained by sampling uniformly at random from ˚ ϕ − 1 ( c r ) if s ≥ 1 , and from ˚ ϕ − 1 ( c (cid:48) r ) if s = 1 .
Given a feature space F and a feature map ψ : X → F , we deﬁne the events E F ,ψ ⊂ Ω DM as follow:
[Equation]
There exists 1 ≤ s ∗ ≤ n spl such that
[Equation]
Note that this event consists in all the outcomes
ω =
( T , S, x test ) for which the feature map ψ associates the test sentence x test to a train point x r,s from the ﬁrst category.
Since by construction, x test belongs to the ﬁrst category, E F ,ψ consists in all the outcomes for which the nearest neighbor classiﬁcation rule is ‘successful’.
As a consequence, when T =
Φ × Z 2 R , the generalization error can be expressed as
[Equation]
where P DM denote the probability measure on Ω DM induced by (cid:37) DM .
Equation (24) should be viewed as our ‘fully formal’ deﬁnition of the quantity err ( F , ψ, T ) , as opposed to the more informal deﬁnition given earlier by equations (6) and (7).
The goal of this section is to prove inequality (14), which, in light of (24) is equivalent to
[Equation]
which in turn equivalent to
[Equation]
where the event E K is deﬁned by
E K =
(cid:110) ω ∈ Ω DM : There exists 1 ≤ s ∗ ≤ n spl such that
K ( x test , x 1 ,s ∗ ) > K ( x test , x r,s ) for all 2 ≤ r ≤ R and all 1 ≤ s ≤ n spl (cid:111) (27) and where the supremum is taken over all kernels K : X × X → R which are symmetric positive semideﬁnite.
We will actually prove a slightly stronger result, namely
[Equation] [Equation]
where the supremum is taken over all functions K : X × X → R that satisfy K ( x , y )
=
K ( y , x ) for all ( x , y ) ∈
X × X .
The rest of the section is devoted to proving (28).
In Subsection B.1 we start by considering a simpler data model — for this simpler data model we are able to show that the function ψ (cid:63) implicitly deﬁned by (12) is the best possible feature map (we actually only work with the associated kernel K (cid:63) , and never need ψ (cid:63) itself).
We also show that the success rate is exactly equal to the permuted moment of K (cid:63) — see Theorem 4, which is is the central result of this section.
In the remaining subsections, namely Subsection B.2 and Subsection
B.3, we leverage the bound obtained for the simpler data model in order to obtain bound (28) for the main data model.
These two subsections are mostly notational.
The core of the analysis takes place in Subsection B.1.
B. A S IMPLER D ATA M ODEL
We start by presenting the sampling process associated with our simpler datamodel.
Sampling Process SDM:
(i)
Sample ϕ uniformly at random in Φ .
Sample c 1 , c 2 , . .
c t +1 uniformly at random in Z .
(ii)
For 1 ≤ r ≤ t + 1 : Sample x r uniformly at random in ˚ ϕ − 1 ( c r ) .
(iii) Sample x test uniformly at random in ˚ ϕ − 1 ( c 1 ) .
The function
[Equation]
on Ω SDM :
=
Φ × C t
+1 ×
X t +2 is the p.d.f.
of the above sampling process.
We use P SDM to denote the probability measure on Ω SDM induced by this function.
The identity in the next theorem is the central result of this section.
Theorem 4.
Let K denote the set of all symmetric functions from X × X to R .
Then sup
[Equation]
In (30), K (cid:63) x stands for the function K ( x , · ) .
Theorem 4 establishes an intimate connection between the permuted moment and the ability of any ﬁxed feature map (or equivalently, any ﬁxed kernel) to generalize well in our framework.
The sampling process considered in this theorem involves two points, x test and x 1 , generated by the same sequence of concepts c 1 , and t ‘distractor’ points x 2 , . . .
, x t +1 generated by different sequences of concepts.
Success for the kernel K means correctly recognizing that x test is more ‘similar’ to x 1 than any of the distractors, and the success rate in (30) precisely quantiﬁes its ability to do so as a function of the number t of distractors.
The theorem shows that the probability of success for the best possible kernel at this task is exactly equal to the averaged t th -permuted moment of K (cid:63) x , so it elegantly quantiﬁes the generalization ability of the best possible ﬁxed feature map in term of the permuted moment.
We also provide an explicit construction for a kernel K ( x , y ) that achieves the supremum in (30) — First, choose a kernel ε ( x , y ) that satisﬁes
(i) ε ( x , y ) (cid:54) =
ε ( x , z ) for all x , y , z ∈ X with y (cid:54) =
z .
(ii)
≤ ε ( x , y ) ≤ 1 for all x , y ∈ X .
and then deﬁne the following perturbation
[Equation]
(31) of K (cid:63) .
Any such kernel is a maximizer of the optimization problem in (30), so we may think of perturbations of K (cid:63) as bona-ﬁde optimal.
The rest of this subsection is devoted to the proof of Theorem 4, and we also show, in the course of the proof, that (31) is a maximizer of the optimization problem in (30).
We use K to denote the set of all symmetric functions from X ×X to R .
We will refers to such functions as ‘kernel’ despite the fact that these functions are not necessarily positive semi-deﬁnite.
Proving Theorem 4 requires that we study the following optimization problem:
[Equation]
) for
[Equation]
2 ≤ r
[Equation]
over all kernels K ∈ K .
[Equation]
We recall the deﬁnition of the optimal kernel,
[Equation]
where s c = n w /s c denotes the size of a concept.
We start with the following simple lemma:
Lemma 4.
The function K (cid:63) x ( · ) =
K (cid:63) ( x , · ) is a probability distribution on X .
Proof.
First note that K (cid:63) can be written as
[Equation]
Since ϕ maps exactly s c words to each concept c ∈ { 1 , . . .
, n c } , we have that
[Equation] [Equation]
Therefore (cid:88)
[Equation]
We now show that the marginal of the p.d.f.
(cid:37) SDM is related to K (cid:63) .
Lemma 5.
For all x 1 , . . .
, x t +1 and x test in X we have (cid:88)
[Equation]
Proof.
Identity (36) can be expressed as (cid:12)(cid:12)
˚ ϕ − 1 ( c ) (cid:12)(cid:12) =
s Lc for all c ∈ C .
As a consequence, deﬁnition (29) of (cid:37) SDM ( ω ) simpliﬁes to
[Equation]
where the constant α is given by
[Equation]
In the above we have used the fact that |C| = n Lc and |X| =
n Lw .
We then note that the identity 1 ˚ ϕ
[Equation]
x ) = 1 { ˚ ϕ ( x )=
c } implies
[Equation]
for all x
[Equation]
y ∈ X .
Summing (37) over the variables c 1 , . . .
, c t +
we obtain
[Equation]
where we have used (38) and (39) to obtain the last equality.
Summing the above over the variable ϕ gives K (cid:63) ( x 1 , x test ) / |X| t
+1 .
The next lemma provides a purely algebraic (as opposed to probabilistic) formulation for the functional E ( K ) deﬁned in (32).
Lemma 6.
The functional E : K → R can be expressed as
[Equation]
Proof.
Let g : X t +2 × K
→ { 0 , 1 } be the indicator function deﬁned by
[Equation]
if
[Equation]
test
[Equation] [Equation]
K
[Equation]
test
[Equation]
x
[Equation]
) for all 2 ≤ r ≤ t +0 otherwise
Let ω denote the sample ( ϕ ; c 1 , . .
c t +1 ; x 1 , . . .
, x t +1 ; x test ) .
Since g only depends on the last t + variables of ω , we have
[Equation] [Equation]
where we have used Lemma 5 to go from (43) to (44).
Writing the indicator function g as a product of indicator functions,
[Equation]
we obtain the following expression for the term appearing between parentheses in (45):
[Equation]
Changing the name of variables x test , x 1 to x , y gives (40).
We now use expression (40) for E ( K ) and reformulate optimization problem (32)-(33) into an equivalent optimization problem over symmetric matrices.
Putting an arbitrary ordering on the set X (starting with i = 0 ) and denoting by K (cid:63)ij the value of the kernel K (cid:63) on the pair that consists of the i th and j th element of X , we see that optimization problem (32)-(33) can be written as
Maximize
[Equation]
over all symmetric matrices K ∈ R N × N
[Equation]
[Equation]
In the above we have used the letter N to denote the cardinality of X , that is N = n Lw , and we have used the notation [ N ] = { 0 , 1 , . . .
, N − 1 } .
Before solving the matrix optimization problem (46)-(47), we start with a simpler vector optimization problem.
Let p (cid:63) be a probability vector, that is p (cid:63) ∈
R N + with (cid:80) Ni =1 p
i = 1 , and consider the optimization problem:
[Equation]
over all vector v ∈ R N .
[Equation]
Recall from Deﬁnition 1 that an ordering permutation of a vector v is a permutation that sorts its entries from smallest to largest.
We will say that two vectors v , w ∈ R N have the same ordering if there exist σ ∈ S N which is ordering for both v and w .
The following lemma is key — it shows that the optimization problem (48)–(49) has a simple solution.
Lemma 7.
The following identity
[Equation]
holds.
Moreover, the supremum is achieved by any vector v ∈ R N that has mutually distinct entriesand that has the same ordering than p (cid:63) .
Proof.
Let Distinct ( R N ) denote the vectors of R N that have mutually distinct entries.
We will ﬁrst show that
[Equation]
To do this we show that for any v ∈ R N , there exists w ∈ Distinct ( R N ) such that
[Equation]
There are many ways to construct such a w .
One way is to simply set w j = σ − 1 ( j ) for some permutation σ that orders v .
Indeed, note that σ − 1 ( j ) provides the position of v j in the sequence of inequality (18).
Therefore if v j (cid:48) < v j we must have that σ − 1 ( j (cid:48) )
<
σ − 1 ( j ) .
This implies
[Equation]
for all j ∈ [ N ] which in turn implies (51).
Because of (50) we can now restrict our attention to v ∈ Distinct ( R N ) .
Note that if v ∈
Distinct ( R N ) , then it has a unique ordering permutation σ ,
[Equation]
and, recalling that σ − 1 ( j ) provide the position of v j in the above ordering, we clearly have that
[Equation]
Therefore, if v ∈ Distinct ( R N ) and if σ denotes its unique ordering permutation, e ( v ) can be expressed as
[Equation]
Looking at deﬁnition (11) of the permuted moment, it is then clear that e ( v ) ≤ H t ( p (cid:63) ) for all v ∈ Distinct ( R N ) .
We then note that if v ∈ Distinct ( R N ) has the same ordering than p (cid:63) , then its unique ordering permutation σ must also be an ordering permutation of p (cid:63) .
Then (52) combined with Lemma 1 implies that e ( v ) =
H t ( p (cid:63) ) .
This concludes the proof.
Relaxing the symmetric constraint in the optimization problem (46)-(47) gives the following unconstrained problem over all N -by- N matrices:
[Equation]
over all matrices K ∈ R N × N
[Equation]
Let us denote by K (cid:63)i, : the i th row of the matrix K (cid:63) and remark that K (cid:63)i, : is a probability vector (because K (cid:63) ( x , · ) is a probability distribution on X , see Lemma 4).
We then note that the above unconstrained problem decouples into N separate optimization problems of the type (48)-(49) in which the probability vector p (cid:63) must be replaced by the probability vector K (cid:63)i, : .
Using Lemma 7 we therefore have that any K ∈ R N × N that satisﬁes, for each 0 ≤ i ≤ N − 1 ,
(a) The entries of K i, : are mutually distinct, (b) K i, : and K (cid:63)i, : have the same ordering,
must be a solution of (53)-(54).
Lemma 7 also gives:
[Equation]
Now let ε ∈ R N × N be a symmetric matrix that satisﬁes:
(i) ε ij (cid:54) = ε ij (cid:48) for all i, j, j (cid:48) ∈ [ N ] with j (cid:54) =
j (cid:48) , (ii) 0 ≤ ε ij ≤ 1 for all i, j ∈ [ N ] ,
and deﬁne the following perturbation of the matrix K (cid:63) :
[Equation]
Recalling deﬁnition (35) of the kernel K (cid:63) , it is clear that for each i, j ∈ [ N ] , we have
[Equation]
integer
[Equation]
As a consequence perturbing K (cid:63) by adding to its entries quantities smaller than 1 / ( s Lc | Φ | ) can not change the ordering of its rows.
Therefore the kernel K deﬁned by (55) satisﬁes (b).
It also satisﬁes (a).
Indeed, if K (cid:63)ij = K (cid:63)ij (cid:48) and j (cid:54) =
j (cid:48) , then we clearly have that K ij (cid:54) =
K ij (cid:48) due to (i).
On the other hand if K (cid:63)ij (cid:54) =
K (cid:63)ij (cid:48) , then K ij (cid:54) =
K ij (cid:48) due to (ii) and (56).
We have therefore constructed a symmetric matrix that is a solution of the optimization problem (53)-(54).
As a consequence we have
[Equation]
where K should now be interpreted as the set of N -by- N symmetric matrices.
The above equality proves Theorem 4, and we have also shown that the perturbed kernel (55) achieves the supremum.
B. C ONNECTION B ETWEEN THE T WO S AMPLING P ROCESSES
In this subsection we show that the p.d.f.
of Sampling Process SDM can be obtained by marginalizing the p.d.f.
of Sampling Process DM over a subset of the variables.
We also compute another marginal of (cid:37) DM that will prove useful in the next subsection.
Recall that
[Equation]
on Ω DM :
=
Φ × C 2 R × X R × n spl +1 is the p.d.f.
of the sampling process for our main data model.
Samples from Ω DM take the form
[Equation]
We separate these variables into two groups, ω = ( ω a , ω b ) , where
[Equation]
The variable ω a belongs to Ω a =
Φ ×C 2 R ×X 2 R +1 , and the variable ω b belongs to Ω b = X R ( n spl − 2) .
Note that the variables in ω a contains, among other, 2 R sequences of concepts and 2 R training points (the ﬁrst and second training points of each category).
Each of these 2 R training points is generated by one of the 2 R sequences of concepts.
So the variables involved in ω a are generated by a process similar to the one involved in the simpler data model.
The following lemma shows that p.d.f.
of ω a , after marginalizing ω b , is indeed (cid:37) SDM .
Lemma 8.
For all ω a ∈ Ω a we have
[Equation]
Recalling the deﬁnition (29) of (cid:37) SDM
[Equation]
and letting t + 1 = 2 R
[Equation]
we see that the above lemma states that
[Equation]
and Ω a = Ω SDM .
Proof of Lemma 8.
We start by reorganizing the terms involved in the product deﬁning (cid:37) DM so that the variables in ω a and ω b are clearly separated:
[Equation]
To demonstrate the process, let us start by summing the above formula over the ﬁrst variable of ω b , namely x 1 , 3 .
Since this variable only occurs in the last term of the above product, we have:
[Equation]
Since (cid:80) x ∈X 1 ˚ ϕ − 1 ( c 1 ) ( x ) =
|
˚ ϕ − 1 ( c 1 ) | , the last term of the above product is equal to 1 and can therefore be omitted.
Repeating this process for all the x r,s that constitute ω b leads to the desired result.
In the next subsection we will need the marginal of (cid:37) DM with respect to another set of variables.
To this aim we write ω =
( ω c , ω d ) where
[Equation]
Note that all the unfamiliar training points are contained in ω d .
The test point and the familiar training points are in ω c .
We also let Ω c =
Φ × X R ( n spl − 1)+1 and Ω d = C 2 R × X R .
Lemma 9.
For all ω c ∈ Ω c we have
[Equation]
Proof.
We reorganizing the terms involved in the product deﬁning (cid:37) DM so that the variables in ω c and ω d are separated:
[Equation]
Summing the above formula over the last variable involved in ω d , namely x R, 1 , gives
[Equation]
The last term in the above product is equal to 1 and can therefore be omitted.
Iterating this process gives (cid:88)
[Equation]
We then use the fact that (cid:80) c (cid:48) 1 ∈C 1 ˚ ϕ − 1 ( c (cid:48) 1 ) ( x test ) = 1 , see (38), together with (cid:12)(cid:12)
˚ ϕ − 1 ( c (cid:48) 1 ) (cid:12)(cid:12) =
s
Lc , see (36), to obtain (cid:88)
[Equation]
) ( x r
[Equation]
s )
[Equation]
ϕ
[Equation]
| (cid:33)
We then sum over c (cid:48) 2 , . . .
, c (cid:48) R .
Since these variables are not involved in the above formula we get
[Equation]
where we have used |C| s Lc =
n Lc s Lc = |X| to obtain the last equality.
Summing over c 1 gives (cid:88) c 1 ∈C
[Equation]
=
[Equation]
To obtain the last equality we have used (39) but for a product of n spl − 1 indicator functions instead of just two.
Iterating this process we obtain (cid:88)
[Equation]
˚ ϕ
[Equation]
˚ ϕ
[Equation]
x
[Equation]
=
[Equation] [Equation]
˚ ϕ ( x r
[Equation]
| ˚ ϕ − 1 ( c r ) | n spl −
Using one more time that (cid:12)(cid:12)
˚ ϕ − 1 ( c r ) (cid:12)(cid:12) =
s Lc and |C| s Lc = |X| gives to the desired result.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
/+Xi0rMjpJKqKJEULx9KCslRU2br6BTYYCjXDnBuBFuV8rnzDDunLz7SjMbtZa2sSa8b8RDcflpFAaj8OJgcPq2M2mL7JE35D0JySE5JWfknIwJJ3886n3w9v3X/rH/2f/alvpe17NL7oT/7S/dusoL</latexit> <latexit sha1_base64="ng4QEm9z7lXRCJUvu2NbhpgDzps=">AAACk3icbZHdatswFMdlt9u67Ctt2VVvxLKFrYxgj0ILpVC2XfRm0MLSFmwvyMpxIiJLRjouCcZvsZfbG+wZdjU5NqMfOyD463c+dPgrLaSwGAS/PH9j89HjJ1tPe8+ev3j5qr+9c2l1aTiMuZbaXKfMghQKxihQwnVhgOWphKt08aXJX92AsUKr77gqIMnZTIlMcIYOTfo/4xRmQlU5QyOW+3Ukk14Vpxld1j9ihCVWCBZrSocndEhpNGzhSs9Kg/VHx1uQlohgHOjunBmjm4Ih7cBc8AWo26QtcSMS2otBTf8tMekPglGwDvpQhJ0YkC7OJ/3f8VTzMgeFXDJrozAoMKmYQcEl1L24tFAwvmAziJxULAebVGv3avrOkSnNtHFHIV3T2x0Vy61d5amrdAvO7f1cA/+Xi0rMjpJKqKJEULx9KCslRU2br6BTYYCjXDnBuBFuV8rnzDDunLz7SjMbtZa2sSa8b8RDcflpFAaj8OJgcPq2M2mL7JE35D0JySE5JWfknIwJJ3886n3w9v3X/rH/2f/alvpe17NL7oT/7S/dusoL</latexit>
[Figure]
C ONCLUSION OF P ROOF
We now establish the desired upper bound (28), which we restate below for convenience:
[Equation]
where
E
[Equation]
: There exists 1 ≤ s ∗ ≤ n spl such that
[Equation]
We recall that the test point x test is generated by the unfamiliar sequence of concepts c (cid:48) 1 and that it belongs to category 1, see Figure 4.
The event E K describes all the outcomes in which the training point most similar to x test (where similarity is measured with respect to the kernel K ) belongs to the ﬁrst category.
There are two very distinct cases within the event E K : the training point most similar to x test can be x 1 , 1 — this corresponds to a ‘meaningful success’ in which the learner recognizes that x 1 , 1 is generated by the same sequence of concepts than x test , see Figure 4.
Or the training point most similar to x test can be one of the points x 1 , 2 , . . .
, x 1 ,n spl — this corresponds to a ‘lucky success’ because x 1 , 2 , . . .
, x 1 ,n spl are not related to x test (they are generated by a different sequence of concept, see Figure 4).
To make this discussion formal, we ﬁx a kernel K ∈ K , and we partition the event E K as follow
[Equation]
where
[Equation]
≤
[Equation]
spl
[Equation]
The next two lemmas provide upper bounds for the probability of the events E meaningful and E luck .
Lemma 10.
[Equation]
Proof.
Deﬁne the event
[Equation]
This events involves only the ﬁrst two training points of each category.
On the example depicted on Figure 4, that would be the points x 1 , 1 and x 1 , 2 , the points x 2 , 1 and x 2 , 2 , and ﬁnally the points x 3 , 1 and x 3 , 2 .
The event A consists in all the outcomes in which, among these 2 R training points, x 1 , 1 is most similar to x test .
We then make two key remarks.
First, these 2 R points are generated by 2 R distinct sequences on concepts — so if we restrict our attention to these 2 R points, we are in a situation very similar to the simpler data model (cid:37) SDM (i.e. we ﬁrst generate 2 R sequences of concepts, then from each sequence of concepts we generate a single training point, and ﬁnally we generate a test point from the ﬁrst sequence of concepts.)
We will make this intuition precise by appealing to the fact that (cid:37) SDM is the marginal of (cid:37) DM , and this will allow us to obtain a bound for P DM [ A ] in term of the permuted moment of K (cid:63) .
The second remark is that E meaningful is clearly contained in A , and therefore we have
[Equation]
so an upper bound for P DM
[ A ] is also an upper bound for P DM [ E meaningful ] .
Let us rename some of the variables.
We deﬁne d 1 , . . .
, d 2 R , and y 1 , . . .
, y 2 R as follow: d 2 r
[Equation]
1 = c (cid:48)
r and d 2 r = c r for r = 1 , . . .
, R y 2 r − 1 = x r,and y 2 r = x r,for r = 1 , . . .
, R
On the example depicted on Figure 4, that would be:
[Equation] [Equation]
In other words, the y r ’s are the ﬁrst two training points of each category and the d r ’s are their corresponding sequence of concepts.
With these notations it is clear that the training points y r are generated by distinct sequences of concepts, and that the test point x test is generated by the same sequence of concepts than y 1 .
Moreover the event A can now be conveniently written as
[Equation]
Let h : X 2 R +
→ R be the indicator function deﬁned by h ( y 1 , . . .
, y 2 R , x test ) =
(cid:26)if K ( x test , y 1 ) > K ( x test , x r ) for all 2 ≤ r ≤ 2 R 0 otherwise
We now recall the splitting ω
= ( ω a , ω b ) described in (58)-(59) and note that ω a can be written as
[Equation]
x test )
Since h only depends on the variables involved in ω a , and since, according to Lemma 8, (cid:80)
[Equation]
, ω b ) = (cid:37) SDM ( ω a ) , we obtain
P
[Equation] [Equation]
where we have used Theorem 4 in order to get the last inequality (with the understanding that t + 1 = 2 R .)
Combining the above bound with (66) concludes the proof.
We now estimate the probability of the event E luck .
Lemma 11.
P DM [ E luck ] ≤
[Equation]
R.
Proof.
For 1 ≤ r ≤ R , we deﬁne the events
[Equation]
Note that the events B r involve only the training points with an index s ≥ 2 : these are the familiar training points.
On the example depicted on Figure 4, these are the training points generated by c 1 , c 2 and c 3 .
Let us pursue with this example.
The event B 1 consists in all the outcomes in which one of the points generated by c 1 is more similar to x test than any of the points generated by c 2 and c 3 . The event B 2 consists in all the outcomes in which one of the points generated by c 2 is more similar to x test than any of the points generated by c 1 and c 3 .
And ﬁnally the event B 3 consists in all the outcomes in which one of the points generated by c 3 is more similar to x test than any of the points generated by c 1 and c 2 .
Importantly, the test point x test is generated by the unfamiliar sequence of concepts c (cid:48) 1 , and this sequence of concept is unrelated to the sequence c 1 , c 2 and c 3 .
So one would expect that, from simple symmetry, that
[Equation]
We will prove (67) rigorously, for general R , using Lemma 9 from the previous subsection.
But before to do so, let us show that (67) implies the desired upperbound on the probability of E luck .
First, note that E luck ⊂ B 1 and therefore
[Equation]
Then, note that B 1 , B 2 and B 3 are mutually disjoints, and therefore, continuing with the same example,
[Equation]
which, combined with (67) and (68), gives P DM [ E luck ] ≤ 1 / 3 as desired.
We now provide a formal proof of (67).
As in the proof of the previous lemma, it is convenient to rename some of the variables.
Let denote by fam r the variable that consists in the familiar training points from category r :
[Equation]
With this notation we have fam r,s = x r,s +1 .
We now recall the splitting ω
= ( ω c , ω d ) described in (61)-(62), and note that ω c can be written as
[Equation]
Using Lemma 9 we have
[Equation] [Equation]
where α is the constant appearing in front of the product in Lemma 9 an h : Φ × X n spl − 1 → { 0 , 1 } is the indicator function implicitly deﬁned in equality (71)-(72).
With the slight abuse of notation of viewing fam r as a set instead of a tuple, let us rewrite the event B r as
[Equation]
We also deﬁne the corresponding indicator function
[Equation]
We now compute P DM ( B 1 ) (the formula for the other P DM ( B r ) are obtained in a similar manner.)
Recall from (69) that the variables involved in fam r only appear in ω c .
Therefore
[Equation]
where we have used (72) to obtain the last equality.
Let us now compare P DM ( B 1 ) and P DM ( B 2 ) .
Letting Z :=
X n spl − 1 , and recalling that ω c = ( ϕ ; fam 1 , . .
. , fam R ; x test ) , we obtain:
P DM [ B 1 ] = (cid:88)
[Equation]
From the above it is clear that P DM [ B 1 ]
= P DM [ B 2 ] (as can be seen by exchanging the name of the variables fam 1 and fam 2 ).
Similar reasoning shows that the events B r are all equiprobable, which concludes the proof.
Combining Lemma 10 and 11, together with the fact that E K =
E correct ∪ E luck , concludes the proof of (63).
Inequality (63) implies inequality (26), which itself is equivalent to inequality (14).
C U PPER BOUND FOR THE P ERMUTED M OMENT OF K (cid:63)
This section is devoted to the proof of inequality (15), which we state below as a theorem for convenience.
Theorem 5.
For all 0 ≤ (cid:96) ≤ L , we have the upper bound
[Equation]
The rather intricate formula for the function f
and g can be found in the main body of the paper, but we will recall them as we go through the proof.
We also recall that the optimal kernel is given by the formula:
[Equation]
The key insight to derive the upper bound (76) is to note that each pair of sentences ( x , y ) induces a graph on the vocabulary { 1 , 2 , . .
. , n w } , and that the quantity
[Equation]
[Table]
can be interpreted as the number of equipartitions of the vocabulary that do not sever any of the edges of the graph.
This graph-cut interpretation of the optimal kernel is presented in detail in Subsection
C.1.
In Subsection C.2 we derive a formula for K (cid:63) which is more tractable than (77).
To do this we partition X × X into subsets on which K (cid:63) is constant, then provide a formula for the value of
K (cid:63) on each of these subsets (c.f. Lemma 16).
With this formula at hand, we then appeal to Lemma 3 to derive a ﬁrst bound for the permuted moment of K (cid:63) (c.f. Lemma 17).
This ﬁrst bound is not fully explicit because it involves the size of the subsets on which K (cid:63) is constant.
In section C.3 we appeal to Cayley’s formula, a classical result from graph theory, to estimate the size of these subsets (c.f. Lemma 18) and therefore conclude the proof of Theorem 5.
We now introduce the combinatorial notations that will be used in this section, and we recall a few basics combinatorial facts.
We denote by N = { 0 , 1 , 2 , . . .
the nonnegative integers.
We use the standard notations (cid:18)
[Equation]
and
[Equation]
for the binomial and multinomial coefﬁcients, with the understanding that 0!
= 1 .
We recall that multinomial coefﬁcients can be interpreted as the number of ways of placing n distinct objects into m distinct bins, with the constraint that k 1 objects must go in the ﬁrst bin, k 2 objects must go in the second bin, and so forth.
The Stirling numbers of the second kind (cid:8) nk (cid:9) are close relatives of the binomial coefﬁcients.
(cid:8) nk (cid:9) stands for the number of ways to partition a set of n objects into k nonempty subsets.
To give a simple example, (cid:8) 42 (cid:9) = 7 because there are 7 ways to partition the set { 1 , 2 , 3 , 4 } into two nonempty subsets, namely:
[Equation]
Stirling numbers are easily computed via the following variant of Pascal’s recurrence formula 4 :
[Equation]
The above formula is easily derived from the deﬁnition of the Stirling numbers as providing the number of ways to partition a set of n objects into k nonempty subsets (see for example chapterof ? ).
Table 2 shows the ﬁrst few Stirling numbers.
We recall that an undirected graph is an ordered pair G =
( V, E ) , where V is the vertex set and
E ⊂ {{ v, v (cid:48) } : v, v (cid:48) ∈ V and v (cid:54) = v (cid:48) } is the edge set.
Edges are unordered pairs of distinct vertices (so loops are not allowed.)
A tree is a connected graph with no cycles.
A tree on n vertices has exactly n − 1 edges.
Cayley’s formula states that there are n n − 2 ways to put n − 1 edges on n labeled vertices in order to make a tree.
We formally state this classical result below:
Lemma 12 (Cayley’s formula) .
There are n n − 2 trees on n labeled vertices.
C. G RAPH -C UT F ORMULATION OF THE O PTIMAL K ERNEL
In this section we consider undirected graphs on the vertex set
[Equation]
Since the data space X consists of sentences of length L , graphs that have at most L edges will be of particular importance.
We therefore deﬁne:
[Equation]
graphs
[Equation]
have
[Equation]
most L edges
[Equation]
In other words, G consists in all the graphs G =
( V , E ) whose edge set E has cardinality less or equal to L .
Since these graphs all have the same vertex set, we will often identify them with their edge set.
We now introduce a mapping between pairs of sentences containing L words, and graphs containing at most L edges.
Deﬁnition 2.
The function ζ : X × X
→ G is deﬁned by
[Equation]
The right hand side of (78) is a set of at most L edges.
Since graphs in G are identiﬁed with their edge set, ζ indeed deﬁne a mapping from X × X to G .
Let us give a few examples illustrating how the map ζ works.
Suppose we have a vocabulary of n w = 10 words and sentences of length L = 6 .
Consider the pair of sentences (
x , y ) ∈
X × X where
[Equation]
Then ζ ( x , y ) is the set of 3 edges
[Equation]
which indeed deﬁne a graph on V .
Note that position (cid:96) =
2 and (cid:96) =
4 of ( x , y ) ‘code’ for the same edge { 2 , 5 } , position 5 codes for the edge { 9 , 2 } , and position 6 codes for the edge { 7 , 1 } .
On the other hand, position 1 and 3 do not code for any edge: indeed, since x 1 = y 1 and x 3 = y 3 , these two positions do not contribute any edges to the edge set deﬁned by (78).
We will say that positions 1 and 3 are silent .
We make this terminology formal in the deﬁnition below:
Deﬁnition 3.
Let ( x , y ) ∈
X × X .
If x (cid:96) =
y (cid:96) for some 1 ≤ (cid:96) ≤ L , we say that position (cid:96) of the pair ( x , y ) is silent.
If x (cid:96) (cid:54) = y (cid:96) for some 1 ≤ (cid:96) ≤ L , we say that position (cid:96) of the pair ( x , y ) codes for the edge { x (cid:96) , y (cid:96) } .
Note that if ( x , y ) has some silent positions, or if multiple positions codes for the same edge, then the graph ζ ( x , y ) will have strictly less than L edges.
On the other hand, if none of these take place, then ζ ( x , y ) will have exactly L edges.
For example the pair of sentences
[Equation]
does not have silent positions, and all positions code for different edges.
The corresponding graph
[Equation]
has the maximal possible number of edges, namely L = 6 edges.
From the above discussion, it is clear that any graph with L or less edges can be expressed as ζ ( x , y ) for some pair of sentences ( x , y ) ∈
X × X .
Therefore ζ : X × X → G is surjective.
On the other hand, different pair of sentences can be mapped to the same graph.
Therefore ζ is not injective.
We now introduce the following function.
Deﬁnition 4 (Number of cut-free equipartitions of a graph) .
The function I : G → N is deﬁned by :
[Equation]
[Figure]
Recall that Φ is the set of maps ϕ : V → { 1 , . . . , n c } that satisfy | ϕ − 1 ( c ) | =
s c for all 1 ≤ c ≤ n c .
Given a graph G , the quantity I ( G ) can therefore be interpreted as the number of ways to partition the vertices into n c labelled subsets of equal size so that no edges are severed (i.e. two connected vertices must be in the same subset.)
In other words, I ( G ) is the number of “cut-free” equipartition of the graph G , see Figure 5 for an illustration.
Note that if the graph G is connected, then I ( G ) =since any equipartition of the graph will severe some edges.
On the other hand, if the graph G has no edges, then I ( G ) = | Φ | since there are no edges to be cut (and therefore any equipartition is acceptable.)
The optimal kernel K (cid:63) can be expressed as a composition of the function ζ
and I .
Indeed:
[Equation] [Equation]
[Equation] [Equation]
where we have simply used that ζ ( x , y ) :
= (cid:83) 1 ≤ (cid:96) ≤
L x
[Equation]
(cid:54) =
y
[Equation]
{{ x (cid:96) , y (cid:96) } (cid:9) to go from (82) to (83).
We will refer to (84) as the graph-cut formulation of the optimal kernel.
We have discussed earlier that the function ζ : X × X → G is surjective but not injective.
We conclude this subsection with a lemma that provides an exact count of how many distinct ( x , y ) are mapped by ζ to a same graph G .
Lemma 13.
Suppose G ∈ G has m edges.
Then
[Equation]
Proof.
Fix once and for all a graph G =
( V , E ) with edge set E = { e 1 , . . .
, e m } where m ≤ L .
Given 0 ≤ α ≤ L , deﬁne the set
O
[Equation]
, y )
[Equation]
: ζ ( x , y )
=
G and ( x , y ) has exactly α non-silent positions } .
We start by noting that the set O α is empty for all α < m : indeed, since G has m edges, at least m positions of a pair ( x , y ) must be coding for edges (i.e., must be non-silent) in order to have ζ ( x , y )
=
G .
We therefore have the following partition:
[Equation]
and
[Equation]
To conclude the proof, we need to show that
[Equation]
Consider the following process to generate an ordered pair ( x , y ) that belong to O α : we start by deciding which positions of ( x , y ) are going to be silent, and which positions are going to code for which edge of the graph G .
This is equivalent to choosing a map ρ : { 1 , . . .
, L } → { e 1 , . . .
, e m , s } where { 1 , . . .
, L } denotes the L positions, e 1 , . . .
, e m denote the m edges of the graph G , and s is the silent symbol.
Choosing a map ρ correspond to assigning a “role” to each position: ρ ( (cid:96) )
=
e i means that position (cid:96) is given the role to code for edge e i , and ρ ( (cid:96) )
= s means that position (cid:96) is given the role of being silent.
The map ρ must satisfy:
[Equation]
and
[Equation]
because L − α position must be silent and each edge must be represented by at least one position.
The number of maps ρ : { 1 , . . .
, L } → { e 1 , . . .
, e m , s } that satisﬁes (86) is equal to (cid:18)
[Equation]
m !
Indeed, we need to choose the L − α positions that will be mapped to the silent symbol s : there are (cid:0)
LL − α (cid:1) ways of accomplishing this.
We then partition the α remaining positions into m non-empty subsets: there are (cid:8) αm (cid:9) ways of accomplishing this.
We ﬁnally map each of these non-empty subset to a different edge: there are m !
ways of accomplishing this.
We have shown that there are (cid:0) Lα (cid:1)(cid:8) αm (cid:9) m !
ways to assign roles to the positions.
Let say that position (cid:96) is assigned the role of coding for edge { v, v (cid:48) } .
Then we have two choices to generate entries x (cid:96) and y (cid:96) : either x (cid:96) = v and y (cid:96) = v (cid:48) , or x (cid:96) = v (cid:48) and y (cid:96) =
v. Since α positions are coding for edges, this lead to the factor 2 α in equation (85).
Finally, if the position (cid:96) is silent, then we have n w choices to generate entries x (cid:96) and y (cid:96) (because we need to choose v ∈ V such that x (cid:96) =
y (cid:96) = v ) , hence the factor n L − α w appearing in (85).
C. L EVEL S ETS OF THE O PTIMAL K ERNEL
We recall that a connected component (or simply a component) of a graph is a connected subgraph that is not part of any larger connected subgraph.
Since graphs in G have at most L edges, their components contain at most L + 1 vertices.
This comes from the fact that the largest component that can be made with L edges contains L + 1 vertices.
It is therefore natural, given a vector k =
[ k 1 , . .
k L +1 ] ∈ N L +1 , to deﬁne
G k :=
{G ∈ G : G has exactly k 1 components of size 1 , exactly k 2 components of size 2 , ..., exactly k L +1 components of size L + 1 } (87) where the size of a component refers to the number of vertices it contains.
We recall that N = { 0 , 1 , 2 , . . .
} therefore some of the entries of the vector k can be equal to zero.
Note that components of size 1 are simply isolated vertices.
The following lemma identify which k ∈ N L +1 lead to nonempty G k .
Lemma 14.
The set G k is not empty if and only if k satisﬁes
L +1 (cid:88) i =
ik i = n w and
L +1 (cid:88)
[Equation] [Equation]
Proof.
Suppose G k is not empty.
Then there exists a graph G ∈ G that has exactly k i components of size i , for 1 ≤ i ≤ L + 1 .
A component of size i contains i vertices (by deﬁnition) and at least i − 1 edges (otherwise it would not be connected.)
Since G ∈ G it must have n w vertices and at most L edges.
Therefore (88) must hold.
Suppose that k ∈ N L +1 satisﬁes (88).
Then we can easily construct a graph G on V that has a number of edges less or equal to L , and that has exactly k i components of size i , for 1 ≤ i ≤ L +1 .
To do this we ﬁrst partition the vertices into subsets so that there are k i subsets of size i , for 1 ≤ i ≤ L +1 .
We then put i − 1 edges on each subset of size i so that they form connected components.
The resulting graph has k i components of size i , for 1 ≤ i ≤ L + 1 , and (cid:80) L +1 i =1 ( i − 1) k
i edges.
The previous lemma allows us to partition G into non-empty subsets as follow:
[Equation]
for all k
[Equation]
S , and
[Equation] [Equation]
where S := (cid:40) k ∈ N
[Equation]
+1 :
[Equation]
i =
ik i = n w and
L +
[Equation] [Equation]
Recall that I ( G ) count the number of equipartitions that do not severe edges of G .
The next lemma shows that two graphs that belongs to the same subset G k have the same number of cut-free equipartitions, and it provides a formula for this number in term of the index k of the subset.
Lemma 15.
Suppose k ∈ S and deﬁne the set of admissible assignment matrices
A
[Equation]
k
[Equation]
for all i and
L
[Equation]
1 (cid:88)
[Equation]
for all j
[Equation]
Then for all G ∈ G k , we have that
[Equation]
Let us remark that, since 0!
= 1 , the multinomial coefﬁcient (cid:0)
[Equation]
[Equation]
(cid:1) appearing in (92) is equal to 1 when k i is equal to 0 .
Poof of Lemma 15.
Let k ∈ S and ﬁx once and for all a graph G ∈ G k .
Deﬁne the set Ψ = {
ϕ ∈ Φ : ϕ ( v ) =
ϕ ( v (cid:48) ) for all edge { v, v (cid:48) } of the graph G} so that I ( G ) = | Ψ | .
Note that a map ϕ that belongs to Ψ must map all vertices that are in a connected component to the same concept (otherwise some edges of G would be severed.)
So a map ϕ ∈ Ψ can be viewed as assigning connected components to concepts.
Given a matrix A ∈ N ( L +1) × n c we deﬁne the set: Ψ
[Equation]
= { ϕ ∈ Ψ : ϕ assigns
A ij components of size i to concept j , for all 1 ≤ i ≤ L +1 and 1 ≤ j ≤ n c } .
We then note that the set Ψ A is empty unless the matrix A satisﬁes:
[Equation]
The ﬁrst constraint states that the total number of connected components of size i is equal to k i (because G ∈ G k ).
The second constraint states that concept j must receive a total of s c vertices (because ϕ ∈ Φ .)
The matrices that satisfy these two constraints constitute the set
A k deﬁned in (91).
We therefore have the following partition of the set Ψ :
[Equation]
To conclude the proof, we need to show that
[Equation]
To see this, consider the k i components of size i .
The number of ways to assign them to the n c concepts so that concept j receives A ij of them is equal to the multinomial coefﬁcient (cid:0)
k
i A
i, 1 ,A i, 2 ,...,A i,nc (cid:1) .
Repeating this reasonning for the components of each size gives (93).
We now leverage the previous lemma to obtain a formula for K (cid:63) .
For k ∈ S we deﬁne Ω k :=
ζ − 1 ( G k ) .
Since ζ : X × X → G is surjective, partition (89) of G induces the following partition of X × X :
[Equation]
Using the graph-cut formulation of the optimal kernel together with Lemma 15 we therefore have
K (cid:63) ( x , y
[Equation]
=
[Equation]
The above formula is key to our analysis.
We restate it in the lemma below, but in a slightly different format that will better suit the rest of the analysis.
Let f : S
→ R be the function deﬁned by
[Equation]
We then have:
Lemma 16 (Level set decomposition of K (cid:63) ) .
The kernel K (cid:63) is constant on each subsets Ω k of the partition (94).
Moreover we have
[Equation]
for all ( x , y ) ∈
Ω k and for all k ∈ S .
Proof.
The quantity | Φ | appearing in (95) can be interpreted as the number of ways to assign the n w words to the n c concepts so that each concept receives s c words.
Therefore
[Equation]
Combined with the fact that |X| =
n Lw , this leads to the desired formula for K (cid:63) .
The above lemma provides us with the level sets of the optimal kernel.
Together with Lemma 3, this allows us to derive the following upper bound for the permuted moment of K (cid:63) .
Lemma 17.
Let Ω = X
× X .
The inequality
[Equation]
holds for all S (cid:48)
⊂ S .
Proof.
Let S (cid:48)
⊂ S and deﬁne:
[Equation]
where we have used the fact that K (cid:63) is equal to f ( k ) / |X| on Ω k .
We then appeal to Lemma 3 to obtain:
[Equation]
=
λ |X| t
[Equation]
[Equation] [Equation]
|X|
[Equation] [Equation]
( x
[Equation]
y )
[Equation]
λ } (99)
[Equation]
|X| t +
[Equation] [Equation]
|X| (cid:88)
[Equation]
x
[Equation]
y )
[Equation]
k
[Equation]
x
[Equation]
y )
[Equation]
λ } (100)
[Equation] [Equation]
[Equation]
where we have use the fact that X × X
= (cid:83) k ∈S Ω k to go from (98) to (99), and the fact that
K (cid:63) ( x , y ) ≤ λ on (cid:83) k ∈S (cid:48)
Ω k to go from (100) to (101).
To conclude the proof, we simply note that λ |X| =
max k ∈S (cid:48)
f ( k ) according to our deﬁnition of λ.
The bound provided by the above lemma is not fully explicit because it involves the size of level sets Ω k .
In the next section, we appeal to Cayley’s formula to obtain a lower bound for | Ω k | .
C. F OREST L OWER B OUND FOR THE S IZE OF THE L EVEL S ETS
We recall that a forest is a graph whose connected components are trees (equivalently, a forest is a graph with no cycles.)
Let us deﬁne:
F := {G ∈ G : G is a forest } .
In other words, F is the set of forests on V = { 1 , 2 , . . . , n w } that have at most L edges.
We obviously have the following lower bound on the size of the level sets:
[Equation]
In this subsection, we use Cayley’s formula to derive an explicit formula for (cid:12) (cid:12) ζ
[Equation]
G k ∩ F ) (cid:12) (cid:12) .
We start with the following lemma:
Lemma 18.
Let k ∈ S , then
[Equation]
Proof.
First we note that (104) can be written as
[Equation]
We now explain the above formula.
The set G k ∩ F consists in all the forests that have exactly ktrees of size 1 , k 2 trees of size 2 , ..., k L +1 trees of size L + 1 .
In order to construct a forest with this speciﬁc structure, we start by assigning the n w vertices to L + 1 bins, with bin 1 receiving kvertices, bin 2 receiving 2 k 2 vertices, ..., bin L + 1 receiving ( L + 1)
k L +1 vertices.
The number of ways of accomplishing this is
[Equation]
Let us now consider the vertices in bin i for some
i ≥ 2 .
We claim that there are
[Equation]
ways of putting edges on these ik i vertices in order to make k i trees of size i .
Indeed, there arek i !
(cid:0)
ik i i,i,...,i (cid:1) ways of partitioning the vertices into k
i disjoint subsets of size i , and then, according to
Cayley’s formula, there are i i − 2 ways of putting edges on each of these subsets so that they form a tree.
To conclude, we remark that there is obviously only one way to to make k 1 trees of size 1 out of the k 1 vertices in the ﬁrst bin.
Recall that a tree on n vertices always has n − 1 edges.
So a graph that belongs to G k ∩ F has
[Equation]
edges since it is made of k 1 trees of size 1 , k 2 trees of size 2 , ..., k L +1 trees of size ( L +1) .
The fact that all graphs in G k ∩ F have the same number of edges allows us to to obtain an explicit formula for | ζ − 1 ( G k ∩ F ) | by combining combine Lemma 13 and 18, namely
[Equation]
This lead us to deﬁne the function g : S → R by
[Equation]

 where γ ( k ) =
L +1 (cid:88) i
=
( i − 1) k i .
Recalling (103) we therefore have that
[Equation]
all k ∈ S
[Equation]
Combining the above inequality with Lemma 17 we obtain:
Theorem 6.
The inequality
[Equation]
holds for all S (cid:48)
⊂ S .
The above theorem is more general than Theorem 5 — indeed, in Theorem 5, the choice of the subset S (cid:48) is restricted to the L + 1 candidates:
[Equation]
i
[Equation]
ik
[Equation]
and (cid:96)
[Equation]
L +
[Equation]
(cid:88) i =
[Equation]
( i −
[Equation]
k
[Equation]
L (cid:41) where (cid:96) = 0 , 1 , . . . , L.
When L = 9 , n w = 150 , n c = 5 and t = 1999 (these are the parameters used in Theorem 1), choosing S (cid:48)
=
S 7 leads to a relatively tight upper bound.
When L = 15 , n w = 30 , n c = 5 and t = 5999 (these are the parameters corresponding the the second experiment of the experimental section), choosing S (cid:48)
=
S 11 gives a good upper bound.
D M ULTIPLE U NFAMILIAR S ENTENCES PER C ATEGORY
In the data model depicted in Figure 1, each unfamiliar sequence of concept has a single representative in the training set.
In this section we consider the more general case in which each unfamiliar sequence of concepts has n ∗ representatives in the training set, where
[Equation]
An example with n ∗ = 2 is depicted in Figure 6.
The variables L, n w , n c , R, n spl and n ∗ parametrize instances of this more general data model, and the associated sampling process is:
Sampling Process DM2:
(i) Sample T =
( ϕ ; c 1 , . . .
, c R ; c (cid:48) 1 , . . .
, c (cid:48) R ) uniformly at random in T =
Φ × Z 2 R .
(ii) For r = 1 , . . .
, R : • Sample ( x r, 1 , . . .
, x r,n ∗ ) uniformly at random in ˚ ϕ − 1 ( c (cid:48) r ) × . . .
× ˚ ϕ − 1 ( c (cid:48) r ) .
• Sample ( x r,n ∗ +1 , . . .
, x r,n spl ) uniformly at random in ˚ ϕ − 1 ( c r ) × . . .
× ˚ ϕ − 1 ( c r ) .
(iii) Sample x test uniformly at random in ˚ ϕ − 1 ( c (cid:48) 1 ) .
Our analysis easily adapts to this more general case and gives:
Theorem 7.
Let T =
Φ × Z 2 R .
Then
1 − err ( F , ψ, T ) ≤ n ∗ (cid:34)(cid:32) 1 − (cid:88) k ∈S (cid:96) f
( k ) g ( k )
[Equation]
(cid:55)→ F ,
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
/+Xi0rMjpJKqKJEULx9KCslRU2br6BTYYCjXDnBuBFuV8rnzDDunLz7SjMbtZa2sSa8b8RDcflpFAaj8OJgcPq2M2mL7JE35D0JySE5JWfknIwJJ3886n3w9v3X/rH/2f/alvpe17NL7oT/7S/dusoL</latexit>
Theorem 3 in the main body of the paper can be viewed as a special case of the above theorem — indeed, setting n ∗ = 1 in inequality (105) we exactly recover (9).
In order to prove Theorem 7, we simply need to revisit subsection B.3.
We denote by Ω DM2 and P
DM2 the sample space and probability measure associated with the sampling process DM2.
As in subsection B.3, given a kernel K ∈ K , we deﬁne the event
E K =
(cid:110) ω ∈ Ω DM2 : There exists 1 ≤ s ∗ ≤ n spl such that
[Equation]
Such event describes all outcomes corresponding to successful classiﬁcation of the test point x test .
For simplicity let us assume that n ∗ = 2 (therefore matching the scenario depicted in Figure 6).
We further partition the event E K according to which training point from the ﬁrst category is most similar to the test point:
[Equation]
The event E (1) meaningful consists in all the outcomes where, among the points from ﬁrst category, x 1 , 1 is the most similar to x test , E (2) meaningful consists in all the outcomes where, among the points from ﬁrst category, x 1 , 2 is the most similar to x test , and E luck consists in all the remaining cases.
Formally we have:
[Equation]
ω
∈
Ω DM2 :
there exists 3 ≤ s ∗ ≤ n spl such that
[Equation]
Exactly as in subsection B.3, we then prove that
[Equation]
The proof of inequality (107) is essentially identical to the proof of Lemma 10.
We deﬁne the event
[Equation]
and the event
[Equation]
The x ’s involved in the deﬁnition of the event
A (1) are highlighted in yellow in Figure 6.
Crucially they are all generated by different sequences of concepts, except for x 1 , 1 and x test .
We can therefore appeal to Theorem 4 to obtain
[Equation]
since there is a total of t = 2 R − 1 ‘distractors’ (the ‘distractors’ in Figure 6 are x 1 , 3 , x 2 , 1 , x 2 , 3 , x 3 , 1 and x 3 , 3 ).
We then use the fact E (1) meaningful ⊂
A (1) to obtain (107) with i = 1 .
The case i = 2 is exactly similar.
We now prove (108).
The proof is similar to the proof of Lemma 11.
For 1 ≤ r ≤ R , we deﬁne the events
[Equation]
By symmetry, these events are equiprobable.
They also are mutually disjoints, and therefore P
DM2 [ B r ] ≤ 1 /R .
Inequality (108) then comes from the fact that E luck ⊂ B 1 .
Combining (106), (107), (108) then gives
[Equation]
and, going back to the general case where n ∗ denotes the number of representatives that each sequence of unfamiliar concepts has in the training set,
[Equation]
which in turn implies (16).
Combining inequalities (15) and (16) then concludes the proof of Theorem 7.
E D ETAILS OF THE E XPERIMENTS
In this section we provide the details of the experiments described in Section 6, as well as additional experiments.
Table 4 provides the results of experiments in which the parameters L , n w , n c and R are set to
L = 9 , n w = 150 , n c = 5 ,
R = 1000 .
[Table] [Table]
The parameters n spl and n ∗ are chosen so that the training set contains 5 familiar sentences per category
[Table]
and between 1 and 5 unfamiliar sentences per category.
Table 3 is identical to Table 1 in
Section 6, with the exception that it contains additional information (i
[Table]
e. the standard deviations of the obtained accuracies).
The abbreviation NN appearing in Table 3 stands for ‘Nearest Neighbor’.
Table 4 provides the results of additional experiments in which the parameters L , n w , n c and R are set to
L = 9 , n w = 50 , n c = 5 ,
R = 1000 .
The parameters n spl and n ∗ are chosen, as in the previous set experiments, so that the training set contains 5 familiar sentences per category, and between 1 and 5 unfamiliar sentences per category.
The tasks considered in this set of experiments are easier due to the fact that the vocabulary is smaller ( n w = 50 instead of n w = 150 ).
E. N EURAL N ETWORK E XPERIMENTS
We consider the neural network in Figure 2.
The number of neurons in the input, hidden and output layers of the MLPs constituting the neural network are set to:
[Equation]
For each of the 10 possible parameter settings in Table 3 and Table 4, we do 104 experiments.
For each experiment we generate:
• A training set containing R × n spl sentences.
• A test set containing 10 , 000 unfamiliar sentences (10 sentences per category).
We then train the neural network with stochastic gradient descent until the training loss reaches 10 − 4 (we use a cross entropy loss).
The learning rate is set to 0 .
01 (constant learning rate), and the batch size to 100 .
At test time, we either use the neural network to classify the test points (ﬁrst row of the tables) or we use a nearest neighbor classiﬁcation rule on the top of the features extracted by the neural network (second row of the tables).
The mean and standard deviation of the 104 test accuracies, for each of the 10 settings, and for each of the two evaluation strategies, is reported in the ﬁrst two rows of Table 3 and Table 4.
E. N EAREST -N EIGHBOR E XPERIMENTS
In these experiments we use a nearest neighbor classiﬁcation rule on the top of features extracted by ψ (cid:63) (third row of Table 3 and 4) or ψ one − hot (fourth row).
For each of the 10 possible parameter settings in Table 3 and Table 4, we do 50 experiments.
For each experiment we generate:
• A training set containing R × n spl sentences.
• A test set containing 1 , 000 unfamiliar sentences (one sentences per category).
In order to perform the nearest neighbor classiﬁcation rule on the features extracted by ψ (cid:63) , one needs to evaluate the kernel K (cid:63) ( x , y )
=
(cid:104) ψ (cid:63) ( x ) , ψ (cid:63) ( y ) (cid:105) F (cid:63) for each pair of sentences.
Computing
K (cid:63) ( x , y ) requires an expensive combinatorial calculation which is the reason why we perform fewer experiments and use a smaller test set than in E.1.
In order to break ties, the values of K (cid:63) ( x , y ) are perturbed according to (31).
With the parameter setting L = 9 , n w = 50 , n c = 5 and R = 1000 , our theoretical lower bound for the generalization error is err
[Equation]
for all F and all ψ , (112) which is obtained by choosing (cid:96) = 6 in inequality (105).
This lead to an upper bound of 0 .
073 n ∗ + 1 /R on the success rate.
This upper bound is evaluated for n ∗ ranging from 1 to 5 in the ﬁfth row of Table 4.
E. SVM ON F EATURES E XTRACTED BY ψ one − hot AND SVM WITH G AUSSIAN K ERNEL
For each of the 10 possible parameter settings in Table 3 and Table 4, we do 100 experiments.
For each experiment we generate:
• A training set containing R × n spl sentences.
• A test set containing 10 , 000 unfamiliar sentences (10 sentences per category).
We use the feature map ψ one − hot (which simply concatenates the one-hot-encodings of the words composing a sentence) to extract features from each sentence.
These features are further normalized according to the formula
[Equation]
where p =
[Equation]
113) so that they are centered around 0 and are O (1) .
We then use the SVC function of Scikit-learn
Pedregosa et al.
(2011), which itself relies on the LIBSVM library Chang & Lin (2011), in order to run a soft multiclass SVM algorithm on these features.
We tried various values for the parameter controlling the (cid:96) 2 regularization in the soft-SVM formulation, and found that the algorithm, on this task, is not sensitive to this choice — so we chose C = 1 .
The results are reported in the seventh row of both tables.
We also tried a soft SVM with Gaussian Kernel
[Equation]
applied on the top of features extracted by ψ one − hot and normalized according to (113).
We use the
SVC function of Scikit-learn with (cid:96) 2 regularization parameter set to C = 1 .
For the experiments in
Table 3 ( n w = 150 ), the parameter γ involved in the deﬁnition of the kernel was set to γ = 0 .when
n ∗ ∈ { 1 , 2 } and to γ = 0 .
1 when n ∗ ∈ { 3 , 4 , 5 } .
For the experiments in Table 4 ( n w = 50 ), it was set to γ = 0 .
75 when n ∗ = 1 , to γ = 0 .
1 when n ∗ = 2 , and ﬁnally to γ = 0 .
005 when n ∗ ∈ { 3 , 4 , 5 } .
[Table]
E. SVM ON F EATURES E XTRACTED BY ψ (cid:63)
Applying a SVM on the feature extracted by ψ (cid:63) is equivalent to running a kernelized SVM with kernel K (cid:63) .
A naive implementation of such algorithm leads to very poor results on our data model.
For such algorithm to not completely fail
[Table]
it is important to carefully “rescale” K (cid:63) so that the eigenvalues of the corresponding Gram matrix are well behaved.
Recall that
[Equation]
and let ξ : R → R be a strictly increasing function.
Since the nearest neighbor classiﬁcation rule works by comparing the values of K (cid:63) on various pairs of points
[Table]
it is clear that using the kernel K (cid:63)(cid:63) ( x , y ) =
ξ ( K (cid:63) ( x , y )) is equivalent to using the kernel K (cid:63) ( x , y ) .
In particular, choosing ξ ( x ) :
=
log(1 + ( n Lw /α ) x ) gives the following family of optimal kernels:
[Equation]
To be clear, all these kernels are exactly equivalent to the the kernel K (cid:63) when using a nearest neighbor classiﬁcation rule.
However, they lead to different algorithms when used for kernelized SVM.
We have experimented with various choice of the function ξ and found out that this logarithmic scaling works well for kernelized SVM.
For each of the 10 possible parameter settings in Table 3 and Table 4, we do 10 experiments.
For each experiment we generate:
• A training set containing R × n spl sentences.
• A test set containing 1 , 000 unfamiliar sentences (one sentences per category).
Let us denote by x train i , 1 ≤ i ≤ R × n spl , the data points in one of these training set, and by x test i , 1 ≤ i ≤ 1000 , the data points in the corresponding test set.
In order to run the kernelized SVM algorithm we need to form the Gram matrices
[Equation]
Constructing each of these Gram matrices takes a few days on CPU.
We then use the SVC function of Scikit-learn to run a soft multiclass kernelized-SVM algorithm.
We tried various values for the parameter controlling the (cid:96) 2 and found that the algorithm is not sensitive to this choice — so we chose C = 1 .
The algorithm, on the other hand, is quite sensitive to the choice of the hyperparamater α deﬁning the kernel K (cid:63)(cid:63)α .
We experimented with various choices of α and found that choosing the smallest α that makes the Gram matrix G train positive deﬁnite works well (note that the Gram matrix should be positive semideﬁnite for the kernelized SVM method to make sense).
In Table 5 we show an example, on a speciﬁc pair of train and test set 5 , of how the eigenvalues of G train and the test accuracy depends on α .