Unlabelled Sample Compression Schemes for Intersection-Closed Classes and Extremal Classes
Anonymous Author(s)
AfﬁliationAddress email
Abstract
The sample compressibility of concept classes plays an important role in learningtheory, as a sufﬁcient condition for PAC learnability, and more recently as an avenuefor robust generalisation in adaptive data analysis.
Whether compression schemesof size O ( d ) must necessarily exist for all classes of VC dimension d is unknown,but conjectured to be true by Warmuth.
Recently Chalopin, Chepoi, Moran, and
Warmuth (2018) gave a beautiful unlabelled sample compression scheme of size
VC dimension for all maximum classes: classes that meet the Sauer-Shelah-Perles
Lemma with equality.
They also offered a counterexample to compression schemesbased on a promising approach known as corner peeling.
In this paper we simplifyand extend their proof technique to deal with so-called extremal classes of VCdimension d which contain maximum classes of VC dimension d − 1 .
A criterionis given which would imply that all extremal classes admit unlabelled compressionschemes of size d .
We also prove that all intersection-closed classes with VCdimension d admit unlabelled compression schemes of size at most 11 d .
Introduction
Littlestone and Warmuth (1986) initiated the study of sample compression schemes as an alternativecharacterisation of probably approximately correct (PAC) learnability, to ﬁnite Vapnik-Chervonenkis(VC) dimension.
Informally a sample compression scheme for a concept class comprises a compressorand reconstructor: while the compressor represents samples labelled by the class as a labelled orunlabelled subsample, the reconstructor must be able to recover a concept consistent with the originallabelled sample given only this representation.
Littlestone and Warmuth (1986) demonstrated aremarkably simple proof of PAC learnability for any concept classes with labelled compressionschemes provided compressed representations have bounded size, constant in the original samplesize.
They conjectured that bounded labelled compression schemes should also be necessary for PAClearnability—completing the characterisation—speciﬁcally that all concept classes of VC dimensiond have labelled compression schemes of size d .
This conjecture, for size O ( d ) , remains open nowmore than 35 years later, and stands as one of the oldest open problems in learning theory.
In additionto proving PAC generalisation bounds (von Luxburg, Bousquet, and Schölkopf, 2004; Langford,2005), compression has found important connections to generalisation in the adaptive data analysissetting (Cummings, Ligett, Nissim, Roth, and Wu, 2016).
While a characterisation of learnability by compression would be completed by a relaxation ofthe original Littlestone and Warmuth (1986) conjecture from labelled schemes of size d to sizepoly ( d ) , for all VC- d classes, Warmuth (2003) later conjectured O ( d ) .
In introducing unlabelledcompression schemes—where the compressor’s representations omit labels, Kuzmin and Warmuth(2007) conjectured O ( d ) unlabelled compression schemes. Clearly unlabelled schemes of any sizeimply labelled compression schemes of the same size.
Notable progress towards resolving the sample compression conjecture has involved compressingfamilies of concept classes.
Shortly after the original conjecture Floyd (1989) proved that allmaximum concept classes C (those meeting the Sauer-Shelah-Perles Lemma with equality) canbe label compressed to size VC ( C ) (Floyd and Warmuth, 1995).
Chalopin et al.
(2018) recentlyproved the same bound but with unlabelled compression, while also providing an elegant counterexample to the same result for so-called corner peeling schemes.
This shows that the constructions ofcorner peeling schemes by Kuzmin and Warmuth (2007) and Rubinstein and Rubinstein (2012) areincorrect.
Moran and Yehudayoff (2016) proved that all VC- d classes can be label compressed tosize exponential in d .
Recently Pálvölgyi and Tardos (2020) offered examples of VC classes whichcannot admit unlabelled compression schemes of size d .
Moran and Warmuth (2016) discover labelled compression schemes for extremal classes, whichcontain cubes for any coordinates shattered by the class, and generalise maximum classes.
It isunknown if all extremal classes admit unlabelled compression schemes of size O ( d ) .
In this paper, wegive a criterion on extremal classes to admit unlabelled compression schemes of size VC dimension
A promising approach to the general conjecture—and a strong motivation for compressing specialfamilies of concept classes–is via embedding.
In this sense, there is interest in embedding general VC-classes into larger classes with good compression properties.
Rubinstein, Rubinstein, and Bartlett(2015) proved that VC- d classes cannot in general be embedded in maximum classes of VC dimensionless than 2 d − 1 , leaving O ( d ) -maximum embeddings still open.
Intersection-closed classes—those closed under coordinate-wise multiplication—are another promising focus for compression, as any class can be embedded into an intersection-closed class.
In thispaper, we show that any intersection-closed class can be embedded into an extremal intersectionclosed class with only a linear increase in VC dimension.
These extremal intersection-closed classeshave VC-sized unlabelled compression schemes by Chalopin et al.
As such, we establish forthe ﬁrst time that intersection-closed classes can be unlabelled compressed to size linear in their VCdimension.
We offer a short proof of the result by Chalopin et al.
(2018) that maximum classes have
VC dimension d -size unlabelled compression schemes and extend this to extremal classes whichcontain maximum classes of VC dimension d − 1 .
Finally we give an efﬁcient method to compute the VC dimension of the intersection closure of aclass.
Preliminaries
Given n ∈ N we call a subset of the binary n -cube C ⊆ { 0 , 1 } n , a concept class with elements v ∈ Cconcepts .
These come from restrictions of classiﬁer families, on samples of n instances.
1 We equateeach instance in [ n ] = { 1 , . . . , n } with coordinates , axes or colours of the n -cube.
An importantcombinatorial parameter is the VC dimension, which is used to bound concept class cardinality.
Deﬁnition 2.1 (Vapnik and Chervonenkis, 1971) .
The VC dimension of concept class C ⊆{ 0 , 1 } n is deﬁned as VC ( C ) =
max (cid:8) | I | : I ⊆
[ n ] , proj I ( C )
= { 0 , 1 } | I | (cid:9) , where proj I ( C ) = 73 (cid:8) ( v i ) i ∈ I : v ∈ C (cid:9) is the set of coordinate projections of the concepts of C on coordinates I ⊆
[ n ] .
Lemma 2.2 (Sauer-Shelah-Perles Lemma, Vapnik and Chervonenkis, 1971; Shelah, 1972; Sauer,1972) .
The cardinality of any concept class C ⊆ { 0 , 1 } n is bounded by |C| ≤ (cid:80) VC( C ) i = (cid:0) ni (cid:1) .
Concept classes meeting Sauer’s Lemma with equality are called maximum (Welzl, 1987).
A cubical–complex structure can be put on concept classes C ⊆ { 0 , 1 } n .
To begin, each v ∈ C is a vertex,and pairs of Hamming-1 separated vertices are edges, of the one-inclusion graph of C (Haussler, Littlestone, and Warmuth, 1994).
Each higher-order cube is composed of points varying over a setof coordinates, the cube’s colours ; a cube’s common values on non-colour coordinates make up itsanchor .
(An equivalent view is that d -cubes arise as pre-images of points in the binary m -cube undera projection from the binary m + d -cube to the binary m -cube.)
Any d -maximum class has a specialstructure as a complete collection of d -cubes:
Deﬁnition 2.3 (Rubinstein, Bartlett, and Rubinstein, 2009) .
A set of d -cubes S of cardinality (cid:0) nd (cid:1) iscalled d -complete if for all I ⊆
[ n ] of cardinality d , there exists c I ∈ S with proj
I ( c I ) = { 0 , 1 } d .
Another property of a concept class C is of being shortest-path closed —also known as isometric(Chalopin et al., 2018)—where every pair of points in C has a shortest path contained in C (Kuzminand Warmuth, 2007; Rubinstein and Rubinstein, 2012).
We will be especially interested in extremal classes—also known as ample classes (Chalopin et al.,2018).
A class C is extremal if whenever C shatters some I ⊆
[ n ] , there is a subcube c ⊆ C whichmaps bijectively to the | I | -cube via proj
I ( · ) .
All maximum classes are extremal and extremal classesare shortest-path closed (Moran and Warmuth, 2016), while the complements of extremal classesare themselves extremal (Bandelt, Chepoi, Dress, and Koolen, 2006; Bollobás and Radcliffe, 1995; Lawrence, 1983).
An extremal counter-part to Sauer’s Lemma was independently discovered by Pajor (1985), Bollobás and Radcliffe (1995), Dress (1997), and Anstee, Rónyai, and Sali (2002).
Lemma 2.4 (Sandwich Lemma) .
For any extremal class C , its cardinality is equal to both the numberof shatterings of C and the number of cube types (an equivalence class of cubes which all have thesame set of colours) in C .
Extremal and maximum classes and the VC dimension, have an elegant recursive structure that is beststudied by inductive arguments.
One such tool is the projection operator proj I ( C ) .
The reduction C I ⊆
proj
I ( C ) is the subset of points with non-unique pre-images under the projection, while thetail is the subset of proj
I ( C ) with unique pre-image under the projection (Welzl, 1987; Kuzmin and Warmuth, 2007).
Unlabelled compression schemes are deﬁned by representation mappings .
A party known as thecompressor uses a representation mapping to summarise a C -labelled sample of any size with asmall unlabelled subset of instances.
A second party, the reconstructor , can invert this representationthanks to condition 1, to a concept that is consistent with the original labelled sample, thanks tocondition 2, (Chalopin et al., 2018).
Deﬁnition 2.5 (Kuzmin and Warmuth, 2007) .
Consider C ⊆ { 0 , 1 } n . Mapping r is a representationmapping of size k < n of C if it satisﬁes
r is an injection between C and subsets of [ n ] of size at most k ; and
(non-clashing) proj r ( v ) ∪ r ( v (cid:48) )
( v ) (cid:54) =
proj r ( v ) ∪ r ( v (cid:48) )
( v (cid:48) ) for all v (cid:54) = v (cid:48) ∈ C .
The intersection closure operator
Examples of intersection-closed classes include axis-aligned hyperrectangles in R m , monomials,which are unions of orthogonal subcubes, and any union of orthogonal intersection-closed subclasses (Moran and Warmuth, 2016).
Closed-below classes (Rubinstein et al., 2009) are intersectionclosed and extremal.
All extremal classes of VC dimension 1 are intersection closed, so long as theorigin is located in the class (Moran and Warmuth, 2016).
We will use the notation v (cid:12) w to denote the intersection of the vertices v, w .
This is just theoperation of bitwise Boolean multiplication v (cid:12) w = ( v 1 w 1 , . . .
, v n w n ) .
We can likewise letu 1 (cid:12) . . .
(cid:12) u k
=
( (cid:81) k
i = 1 u i 1 , . . .
, (cid:81) k
i = 1 u i n ) denote the (iterated) intersection of vertices u 1 , . .
u k .
Deﬁnition 3.1.
A class C in the binary n -cube is called intersection closed if v (cid:12) w ∈ C , wheneverv, w ∈ C .
Let C =
(cid:12) ( C ) =
{ u 1 (cid:12) . . .
(cid:12) u k ) : k ≤ n, { u 1 , . . .
, u k } ⊆ C} denote the intersectionclosure of any class C obtained via intersections of all ﬁnite subsets of C ’s concepts.
Lemma 3.2.
For any concept class C ⊂ { 0 , 1 } n , the intersection closure C is the unique smallestintersection-closed class containing C , and it is contained in any intersection-closed class containing
C .
Moreover for any J ⊆ [ n ] , we must have proj J ( C ) =
proj J (cid:0) C (cid:1) .
We now give a useful result on computing the VC dimension of the intersection closure of any class.
Deﬁnition 3.3.
Suppose C is a class in the binary n -cube with VC dimension d .
We will say that
C satisﬁes the k -close cube condition if there is a vertex v ∈ { 0 , 1 } n and a complete union of( n − k − 1) -cubes in the complement of C so that each cube is distance at most one from v .
Theorem 3.4.
Suppose C is a class in the binary n -cube.
If the VC dimension d of the intersectionclosure C of C is at most k then C satisﬁes the k -close cube condition. Conversely if C satisﬁes thek -close cube condition, then there is a choice of origin so that the intersection closure C of C has VCdimension at most k relative to this origin. Proof. Suppose ﬁrst that d ≤ k .
Then any projection p of C to a ( k + 1) -cube is not onto.
Assumep ( C ) contained the vertex w = (1 , 1 , . .
. , 1) and all vertices w i = . , 1) with 0 inthe i th position and all other entries 1 . Then since p ( C ) is intersection closed, the image is the whole( k + 1) -cube contrary to assumption.
The reason is that taking intersections of combinations of thevertices w i gives all the vertices in the ( k + 1) -cube.
So at least one of the vertices w, w 1 , . . .
, w k +1must not be in p ( C ) .
But then there is an ( n − k − 1) -cube with anchor of this form in the complementof C by taking the inverse image under p of this vertex.
By choosing v as the vertex with all entries1 in the n -cube, we have shown the existence of a collection of cubes in the complement of C asclaimed. For clearly each of our ( n − k − 1) -cubes has distance at most one from v . Conversely suppose that C has a collection of cubes in its complement as above. We claim that d ≤ kfor a suitable choice of origin.
For suppose that any projection p of C to the ( k + 1) -cube is selected. By Lemma 3.2, p ( C ) = p ( C ) . But now by assumption, if we choose the origin so that v is the vertexwith all entries 1 , then there is an ( n − k − 1) -cube in the complement of C which has anchor withentries all 1 ’s except for at most one entry 0 and the anchor projects to a vertex of the ( k + 1) -cube. Hence the projection p of C is not onto.
We end with a useful property of the intersection closure operator.
We will abuse notation by sayinga class A is shortest path closed inside a class B , if A ⊂ B and any two vertices of A are connectedby a shortest path inside B .
Theorem 3.5.
Suppose C is shortest-path closed. Then the intersection closure C is also shortest-pathclosed.
Proof.
We ﬁrst observe the easy fact that if λ is a shortest path in C consisting of verticesv 1 , v 2 , . .
. , v k and u ∈ C then u (cid:12) λ := { u (cid:12) v i : i ∈ [ k ] } is a shortest path in C possiblywith repetitions.
For an edge between successive vertices v i , v i +1 is transformed into u (cid:12) v i , u (cid:12) v i +1which is either an edge of the same colour or a single vertex.
Our main claim that if C is shortest-path closed then
X
=
{ v : v = u (cid:12) w, u, w ∈ C} is shortest pathclosed inside C .
Once this claim is established, the proof follows easily by repeating this step.
For wecan apply the claim to the class X to prove that the class { v : v = u (cid:12) w, u, w ∈ X} consisting ofall elements of C together with intersections of up to 4 elements of C is shortest-path closed.
inside
X = C .
Repeating at most log 2 n times, C is obtained and is shortest-path closed.
(Note that sincev = v (cid:12) v , elements of C can be written as intersections of pairs of elements of C ).
The ﬁrst step in proving the main claim is to observe that there is a shortest path in X connecting anyelement u ∈ C to any element u (cid:12) v ∈ X .
To construct this path, start with a shortest path λ betweenu, v in C and then form u (cid:12) λ .
As above this is a shortest path, possibly with repetitions, betweenu, u (cid:12) v in X .
We conclude that given two vertices u, v ∈ C , there is a shortest path in X from u tov passing through u (cid:12) v , by concatenating shortest paths in X from u to u (cid:12) v and u (cid:12) v to v .
To prove the main claim, choose elements v, v (cid:48)
∈ X .
We want to ﬁnd a shortest path in C connectingv, v (cid:48) .
Let v = u (cid:12) w, v (cid:48)
=
u (cid:48) (cid:12) w (cid:48) where u, w, u (cid:48) , w (cid:48) ∈ C .
By the ﬁrst step, there are shortest pathsλ, λ (cid:48) in X connecting u, u (cid:48) and w, w (cid:48) and passing through u (cid:12) u (cid:48) , w (cid:12) w (cid:48) respectively.
Next, observe that v (cid:12) v (cid:48) =
u (cid:12) w (cid:12) u (cid:48) (cid:12) w (cid:48) =
u (cid:12) u (cid:48) (cid:12) w (cid:12) w (cid:48) .
Split λ, λ (cid:48) into λ i , λ (cid:48)
i for i = 1 , 2so that λ 1 , λ (cid:48) 1 join u, w and u (cid:12) u (cid:48) , w (cid:12) w (cid:48) respectively and λ 2 , λ (cid:48)
2 join u (cid:12) u (cid:48) , w (cid:12) w (cid:48) and u (cid:48) , w (cid:48)respectively.
To establish the main claim, it clearly sufﬁces to construct shortest paths in C from v tov (cid:12) v (cid:48) snd from v (cid:12) v (cid:48) to v (cid:48) .
By symmetry, we will focus on the ﬁrst of these.
We will prove a slightly stronger claim.
Namely assume λ 1 consists of ordered vertices
u =
z 1 , z 2 , . .
z k = u
(cid:12) u (cid:48) and λ (cid:48) 1 has ordered vertices w = y 1 , y 2 , . . .
, y r = w (cid:12) w (cid:48) .
We will showthat a shortest path in X connecting v, v (cid:12) v (cid:48) can be chosen to have all vertices of the form
z i (cid:12) y j .
To complete the proof, consider the rectangular grid of intersections
z i (cid:12) y j , for 1 ≤ i ≤ k and1 ≤ j ≤ r with k, r ≥ 1 .
We claim that there is a shortest path between the diagonally oppositeentries v =
z 1 (cid:12) y 1 , v (cid:12) v (cid:48)
=
z k (cid:12)
y r using vertices in this grid.
It sufﬁces to use the boundary of the grid to do this.
So for example, consider the ﬁrst path
z i (cid:12) y 1 for1 ≤ i ≤ k , followed by a second path
z k (cid:12) y j for 1 ≤ j ≤ r .
By the ﬁrst observation in this proof, the ﬁrst path is shortest with possible repetitions, as is the second path.
To complete the argument,we need to show that the concatenation of these two paths is also shortest with possible repetitions.
Notice that the path λ
1 =
z 1 , z 2 , . .
z k runs between ends z 1 , z k where z k is between the originof the binary n -cube and z 1 .
So the same follows for the path with possible repetitions
z i (cid:12) y 1 for1 ≤ i ≤ k .
The second path z k (cid:12) y j for 1 ≤ j ≤ r is of the same form and hence the concatenationof these two paths is also shortest with repetitions and the theorem proof is complete.
Extremal intersection-closed classes
Theorem 4.1.
Suppose that C is shortest-path and intersection closed.
Then C is extremal.
Proof.
Suppose a projection p : { 0 , 1 } n → { 0 , 1 } k shatters C .
We will prove there is a k -subcube cof C which witnesses the shattering.
Consider all the vertices of C which map to the vertex w = (1 , 1 , . .
, 1) in the k -cube.
Choose avertex v in this set which has smallest distance to the origin.
(Since C is shortest-path closed andintersection closed, the origin is in C and distance in C is the same as distance in the n -cube.)
Our claim is that there are vertices v i ∈ C , i ∈
[ k ] , so that the distance from v to each v i is one andp ( v i ) =
, 1) =
w i the vertex in the k -cube with all coordinates 1 except a single 0 atthe i th coordinate.
Once this claim is established, it follows that since C is intersection closed, takingthe intersections of subcollections of the vertices { v i : i ∈ [ k ] } generates the required k -subcube cof C .
Note c witnesses the shattering, since projection commutes with intersection and the verticesw, w i generate c under the operation of taking intersections.
To prove the claim, choose any vertex u ∈ C
so that p ( u ) =
w
i .
Since C is intersection closed,u (cid:48)
= v (cid:12) u ∈ C .
Then p ( u (cid:48) )
=
p ( v (cid:12) u ))
=
p ( v ) (cid:12) p ( u ) = w (cid:12) w i = w i .
Note that u (cid:48) is a vertexbetween v and the origin, i.e. , for any coordinate where u (cid:48) has value 1 then v must also.
Since C isshortest-path closed, a shortest path λ exists in C from v to u (cid:48) .
Moreover λ has vertices in order withdistances decreasing to the origin, since u (cid:48) is between v and the origin.
Let v (cid:48) be the ﬁrst vertex on λ after v .
Since p ( λ ) is a shortest path in the k -cube from w to w i , itfollows that p ( v (cid:48) ) is either w or w i .
If the former, this contradicts our choice of v as a closest vertexin C to the origin which projects to w .
If the latter then we have found our vertex v i with distance 1to v and which projects to w i .
Remark 4.2 .
Chalopin et al.
(2018) discusses the signiﬁcance of properties of extremal intersectionclosed classes, also known as conditional antimatroids .
Remark 4.3 .
There is a partial order on the binary n -cube given by w ≤ v when w is between v andthe origin or equivalently there is a shortest path from v to the origin passing through w .
Choose anarbitrary ordering on [ n ] .
We will denote by λ ( v, w ) the shortest path between v, w with verticesv =
v 1 , v 2 , . . .
v k = w where v i .v
i +1 differ in the ﬁrst coordinate x j in the ordering of indices jamongst all coordinates where v i , w differ.
The main result of this section is the following:
Theorem 4.4.
Any intersection-closed class C embeds in a shortest-path closed and intersectionclosed class C ∗ so that d ∗ ≤ 11 d where d, d ∗ are the VC dimensions of C , C ∗ respectively.
Toconstruct C ∗ , for every pair of vertices v, w ∈ C with w < v , add all the vertices in λ ( v, w ) to C .
Lemma 4.5.
If C is intersection closed then C ∗ as constructed by Theorem 4.4 is intersection closed.
Proof.
By deﬁnition, for any vertices u, u (cid:48)
∈ C ∗ , there are vertices v, w, v (cid:48) , w (cid:48) ∈ C
so that u ∈λ ( v, w ) , u (cid:48) ∈ λ ( v (cid:48) , w (cid:48) ) .
Moreover, w ≤ v, w (cid:48) ≤ v (cid:48) .
Since C is intersection closed, both v (cid:12) v (cid:48) and w (cid:12) w (cid:48) are in C .
We claim that u (cid:12) u (cid:48) is on the pathλ ( v (cid:12) v (cid:48) , w (cid:12) w (cid:48) ) —note that as part of this claim, we require that w (cid:12) w (cid:48) ≤ v (cid:12) v (cid:48) .
We begin with the latter claim.
This is easy— w (cid:12) w (cid:48) has entry 1 at precisely the coordinates whereboth w, w (cid:48) have entry 1 .
But since w ≤ v, w (cid:48) ≤ v (cid:48) , v, v (cid:48) has entry 1 at all the coordinates wherew, w (cid:48) respectively has entry 1 .
So for any coordinate where w (cid:12) w (cid:48) has entry 1 , it follows immediatelythat both v, v (cid:48) have entry 1 and so does v (cid:12) v (cid:48) .
Next, the set of coordinates where v has entry 1 and u has entry 0 forms an initial segment of thecoordinates where v has entries 1 , using the ordering of the coordinates, since u ∈ λ ( v, w ) .
A similaranalysis applies to the set of coordinates of u (cid:48) which are 0 but the entries for v (cid:48) are 1 .
To complete the proof that C ∗ is intersection closed, we show that the set of coordinates where theentries of v (cid:12) v (cid:48) are 1 and the entries of u (cid:12) u (cid:48) are 0 , forms an initial segment of the ordering of thecoordinates where the entries of v (cid:12) v (cid:48) are 1 .
It will then follow
that u (cid:12) u (cid:48) ∈
λ ( v (cid:12) v (cid:48) , w (cid:12) w (cid:48) ) .
Now the set of coordinates where v (cid:12) v (cid:48) has entries 1 are exactly those coordinates where both v, v (cid:48)have entries 1 .
Moreover u (cid:12) u (cid:48) has entry 0 at a coordinate if either u or u (cid:48) has entry 0 .
We completethe proof using an argument by contradiction.
Suppose that there is a coordinate x i such that u (cid:12) u (cid:48)has entry 1 but this occurs before a coordinate x j where u (cid:12) u (cid:48) has entry 0 and moreover v (cid:12) v (cid:48) hasentry 1 .
Then at x j , either u or u (cid:48) has entry 0 .
Without loss of generality assume the former.
Sinceu (cid:12) u (cid:48) has entry 1 at x i it follows that u has entry 1 at x i .
Note also that v has entry 1 at both x i , x j .
But this is a contradiction since the coordinates where v has entry 1 but u has entry 0 should form aninitial segment of the set of coordinates where v has entry 1 .
But we have found j > i where u hasentry 1 at x i and 0 at x j .
This establishes that C ∗ is intersection closed.
Lemma 4.6.
If C is intersection closed then C ∗ as constructed by Theorem 4.4 is shortest-path closed.
Proof.
By Lemma 4.5, it sufﬁces to show that if v, w ∈ C ∗ then there are shortest paths from v, w tov (cid:12) w , since putting these together will give a shortest path from v to w .
Next, it sufﬁces to showthere is a shortest path in C ∗ between any pair of vertices v, u in C ∗ , where u ≤ v , for we can thenapply this to the pairs v, v (cid:12) w and w, v (cid:12) w .
We claim that there is a vertex v 1 ∈ C so that u ≤ v 1 , v ≤ v 1 and there are shortest paths in C ∗ joiningboth v 1 , u and v 1 , v .
Firstly, since v ∈ C ∗ , there are vertices v 1 , v 2 ∈ C so that v is in λ ( v 1 , v 2 ) .
Hence v ≤ v 1 and there is a shortest path in C ∗ from v 1 to v , namely a segment of λ ( v 1 , v 2 ) .
Next, since u ≤ v , then u ≤ v 1 .
Moreover u lies on a path λ
( u 1 , u 2 ) where u 1 , u 2 ∈ C , since u ∈ C ∗ .
Then v 1 (cid:12) u 1 ∈ C and v 1 (cid:12) u 1 ≤ u 1 .
Since u ≤ v 1 , u ≤ u 1 , it follows that u ≤ v 1 (cid:12) u 1 .
Notice that u must lie on λ ( v 1 (cid:12) u 1 , u 2 ) .
To verify this, observe all the coordinates where entriesof v 1 (cid:12) u 1 differ from entries of u are also coordinates where u 1 entries differ from u entries.
Consequently, coordinates where entries of u and u 2 differ, are lower in the coordinate ordering thanthose where entries of v 1 (cid:12) u 1 and u differ.
So this conﬁrms that u ∈ λ ( v 1 (cid:12) u 1 , u 2 ) .
Combining λ ( v 1 , v 1 (cid:12) u 1 ) and λ ( v 1 (cid:12) u 1 , u ) gives a shortest path from v 1 to u in C ∗ .
So thiscompletes the proof of the claim about v 1 .
To recapitulate, we have constructed shortest paths in C from v 1 to both u, v in C ∗ , where v ≤ v 1 , u ≤v 1 , u ≤ v .
To complete the proof that there is a shortest path from v to u in C , we use induction onthe Hamming distance from v 1 to v .
To begin the induction, suppose this distance is 0 .
Then clearlywe are done, since there is a shortest path in C ∗ from v to u .
Assume the distance from v 1 to v is k and the result is true for v 1 ∗ , v ∗ , u ∗ as above, where thedistance from v 1 ∗ to v ∗ is less than k .
Let v (cid:48) be the ﬁrst vertex in the shortest path from v 1 to v .
Ouraim is to show that there is a shortest path in C ∗ from v (cid:48) to u .
By induction, it then follows there is ashortest path in C ∗ from v to u .
The argument is similar to that in Theorem 3.5.
Let v 1 = y 1 , y 2 , . . .
y r
= u be a shortest path in
C ∗ .
We can construct a shortest path v (cid:48)
= v (cid:48) (cid:12) y 1 , v (cid:48) (cid:12) y 2 , . .
, v (cid:48) (cid:12) y r =
u where there is a singlerepetition.
Namely, since one of the edges joining y j , y j +1 must have the same colour as the edgebetween v 1 , v (cid:48) , then v (cid:48) (cid:12) y
j
= v (cid:48) (cid:12) y
j +1 .
All other pairs of vertices are distance one apart and since
C ∗ is intersection closed, this shortest path is in C ∗ .
So this completes the proof.
Proof.
(Theorem 4.4)
It remains to prove that the VC dimension of C ∗ is at most 11 d .
Choose any projection p : { 0 , 1 } n → { 0 , 1 } 11 d .
By Sauer’s Lemma, since the VC dimension of p ( C )is at most d , the cardinality | p ( C ) | is at most F ( d ; 11 d, 1 2 ) × 2 11 d where F ( k ; n, p ) is the cumulativebinomial probability function.
Algorithm 1 constructing C ∗ involves ﬁnding and adjoining shortest paths connecting suitable orderedpairs of points in C .
So |C ∗ | ≤ n × |C| 2 .
Similarly we see that | p ( C ∗ ) | ≤ 11 d × | p ( C ) | 2 , sincethe process of adjoining shortest paths is preserved under projection.
Hence | p ( C ∗ | ≤ 11 d ×
F ( d ; 11 d, 12 ) 2 × 2 22 d × 12 .
Note that as pairs ( v, w ) are ordered by requiring that w ≤ v , we get afactor of 12 .
We claim that this upper bound is strictly less than 2 11 d .
Then p cannot shatter and the
VC dimension of C ∗ is at most 11 d − 1 .
The key idea is to use Chernoff’s inequality to estimate F ( d ; 11 d, 12 ) .
This is:
[Equation]
where D ( 111 || 12 ) = 111 log 211 + (1 − 111 ) log 2(1 − 111 ) .
Substituting this into the previous upper bound and simplifying, we obtain | p ( C ∗ ) | ≤ 11 d × 2 22 dexp(22 d log 11 − 2 d log 2 − 20 d log 20) .
We can further simplify this to | p ( C ∗ ) | ≤ 11 d × 11 22 d × 10 − 20 d × 12 .
To complete the argument, we need to show the right side is strictly less than 2 11 d .
Hence it sufﬁces to show that
[Equation]
Now if d ≥ 5 then 11 1 d < 1 .
7 and d 1 d < 1 . 4 .
So we can compute the left side is at most1 .
7 × 1 .
× 121 × 111020 < 1937 whereas the right side is 2048 .
(Here we leave out the factor 2 − 1 d .)
If d = 4 then 112 14 < 1 . 6 and 4 14 < 1 .
So we can compute the left side is as at most 1 . 6 × 1 . 42 ×121 × 111020 =
1810 whereas the right side is 2048 .
So this completes the case when d ≥ 4 .
For theother cases, it is convenient to work directly, as Chernoff’s inequality is not sharp.
For d = 1 , | p ( C ) | ≤ 12 , as it has VC dimension at most 1 in the 11 -cube.
Hence | p ( C ∗ ) | ≤11 × 12 2 × 12 = 792 which is less than 2048 .
Next suppose d = 2 .
| p ( C ) | ≤ 254 , as it has VC dimension at most 2 in the 22 -cube.
Hence | p ( C ∗ ) |is bounded above by 22 × 254 2 × 12 = 709 , 676 which is less than 2048 2 = 4 , 194 , 304 .
Finally assume d = 3 .
| p ( C ) | ≤ 6018 , as it has VC dimension at most 3 in the 33 -cube.
Hence | p ( C ∗ ) |is bounded above by 33 × 6018
× 12 = 597 , 569 , 346 which is less than 2048 3 = 8 , 589 , 934592 .
Corollary 4.7.
If C is an intersection-closed class, then C has a (corner peeling) unlabelled compression scheme of size at most 11 d .
Proof.
This follows immediately by combining Theorem 4.4 and (Chalopin et al., 2018, Theorem4.9).
Unlabelled compression schemes for extremal classes
The aim is to give a condition on extremal classes which implies that they have unlabelled compressionschemes.
We then give some situations where this criterion is satisﬁed.
We conjecture that it is alwaysvalid.
5.1 High-level strategy
The idea is inspired by Chalopin et al.
Start with a pair ( C , D ) consisting of an extremal classand a proper extremal subclass.
The condition is that for every cube in C which is not maximal, thereis a cube with the same colours in D .
We call this the cubical colour condition of the pair C , D andabbreviate it by ccc .
We will construct a bijection φ between the set S of maximal cubes c of C not in D and C \ D , so thatv = φ ( c ) ∈ c .
Note the cardinalities of these two sets are the same, by the Sandwich Lemma 2.4applied to the two extremal classes C , D .
Then φ − 1 : C \D → S gives a representation map for C \D ,which also separates the vertices of C \ D from the vertices of D .
Therefore, any representation of Dcan be combined with the representation of C \ D to give a representation of C .
Suppose we can ﬁnd a sequence of proper pairs of extremal subclasses ( C , D ) , ( D , E ) , . . .
, ( X , Y ) ,all satisfying ccc , ending with VC ( Y ) ≤ 2 .
Then we can start with a representation map of the VC-2class Y (Chalopin et al., 2018) and extend to a representation of X , then extending the representationall the way to C using this strategy.
5.2 Detailed construction
We construct φ : C \D → S and show it is a bijection.
To begin with, denote the extremal class whichis the complement of D by D ∗ .
Then every vertex of C \ D is in at least one maximal cube of D ∗ .
We say that two cubes c, c ∗ in { 0 , 1 } n are complementary if the dimensions d, d ∗ of c, c ∗ satisfyd + d ∗ = n and c ∩ c ∗ is a single vertex.
Recall that S is the set of maximal cubes c of C not in D .
Denote the set of maximal cubes of D ∗ which intersect C by S ∗ .
Lemma 5.1.
Assume that ( C , D ) is a proper pair of extremal classes satisfying ccc .
If c ∈ S , thenthere is a complementary maximal cube c ∗ of D ∗ .
Conversely, if c ∗ is a maximal cube of S ∗ , then thereis a complementary cube c ∈ S .
A maximal cube c ∗ of S ∗ , cannot have two different complementarycubes in S .
Given Lemma 5.1, we will call v = c
∩ c ∗ the special vertex of c .
We now note properties of v ,assuming Lemma 5.1.
Corollary 5.2.
Assume that ( C , D ) is a proper pair of extremal classes satisfying ccc .
Let v =
c ∩ c ∗where c ∈ S and c
∗ ∈ S ∗ are complementary maximal cubes.
Then
• The set of vertices v = c ∩ c ∗ , for all pairs of complementary maximal cubes c ∈ S andc ∗ ∈ S ∗ , is C \ D .
• The mappings between c, c ∗ , v are bijections between S , S ∗ and C \ D .
Proof.
Lemma 5.1 gives a bijection between S and S ∗ , by the mapping from cubes c to complementarycubes c ∗ .
Moreover by the Sandwich Lemma applied to extremal classes C , D , the cardinalities of
C \ D , S ∗ are the same.
Since the map from maximal cubes in S to special vertices is an injection by
Lemma 5.1, it must be a bijection onto C \ D .
Proof of Lemma 5.1.
Let p : { 0 , 1 } n → { 0 , 1 } d be the projection to the colours of c ∈ S .
Recall cccfor the pair C , D says that for any cube of C which is not maximal, there is a cube of D with thesame set of colours.
So p ( D ) = { 0 , 1 } d \ { w } for some vertex w .
For the only other case is thatp ( D ) = { 0 , 1 } d .
But since D is extremal, this would imply D contained a cube with the same coloursas c .
This contradicts the assumption that c is maximal in C , since for an extremal class, maximalcubes are unique for their sets of colours.
Deﬁne v, c ∗ by v = p − 1 ( w ) ∩ c, c ∗ =
p − 1 ( w ) respectively.
Clearly c ∩ c ∗ = v .
It remains to showthat c ∗ is maximal in D ∗ and c ∗ cannot intersect some other cube of S in a single vertex.
Suppose that c ∗ was properly contained in a larger cube ˜ c in D ∗ .
Then p (˜ c ) would properly contain{ w } .
Moreover ˜ c = p − 1 p (˜ c ) .
But this contradicts our observation above that p ( D ) = { 0 , 1 | d \ { w } .
So this completes the argument that c ∗ is maximal.
Next, assume that c ∗ is complementary to a second maximal cube ˆ c ∈ S .
Consider the projection ˆ pof { 0 , 1 } n to { 0 , 1 } ˆ d given by the ˆ d colours of ˆ c .
As above, ˆ p ( D ) consists of { 0 , 1 } ˆ d with a singlevertex ˆ w removed.
Moreover, ˆ p − 1 ( ˆ w ) is a maximal cube c (cid:48) of D ∗ .
Now as c ∗ ∩ ˆ c = u ∗ is a single vertex, ˆ p ( c ∗ )
= ˆ p ( u ∗ ) = u .
As the cubes c ∗ , ˆ c are complementary,c ∗ =
ˆ p − 1 ( u ) .
But now we see that the colours of the anchor of c ∗ are both the colours of c and of ˆ c .
This gives a contradiction, since two maximal cubes of C cannot have the same set of colours.
Thiscompletes the proof of Lemma 5.1.
Finally we show that the map from a special vertex v to the colours of the corresponding maximalcube c of S gives a representation mapping of C \ D which also separates C \ D from D .
The second property follows immediately.
For recall that if p is the projection given by the d coloursof c , then p ( D ) = { 0 , 1 } d \ { p ( v ) } .
So the colours of c clearly separate v from all the vertices of D .
By this we mean that given any vertex w ∈ D , v, w differ on at least one coordinate from the coloursof c .
Next suppose two vertices v, v (cid:48) have the same values at all the coordinates corresponding to thecolours of their corresponding cubes c, c (cid:48)
∈ S of dimensions d, d (cid:48) respectively.
Project { 0 , 1 } n tothe union of the colours of c, c (cid:48) by a mapping p (cid:48)(cid:48) .
The image of p (cid:48)(cid:48) is a cube { 0 , 1 } d (cid:48)(cid:48) of dimensiond (cid:48)(cid:48) ≤ d + d (cid:48) .
In this cube, C , D map to extremal classes p (cid:48)(cid:48)
( C ) =
C (cid:48)(cid:48) , p (cid:48)(cid:48)
( D ) =
D (cid:48)(cid:48) .
We want to check that the pair of extremal classes ( C (cid:48)(cid:48) , D (cid:48)(cid:48) ) satisfy ccc .
But this follows easily.
Choose any non maximal cube ˜ c ∈ C (cid:48)(cid:48) .
As C is extremal, there is a cube c 0 in C which projectsone-to-one to ˜ c and therefore has the same colours as ˜ c .
Since c 0 is not maximal, there is a cube c 1 in
D with the same colours as c 0 .
Then p (cid:48)(cid:48) ( c 1 ) is the required cube in D (cid:48)(cid:48) with the same colours as ˜ c asrequired.
Next, c, c (cid:48) project one-to-one by p (cid:48)(cid:48) to cubes with colours spanning all of { 0 , 1 } d (cid:48)(cid:48) .
Finally we claimthat p (cid:48)(cid:48) ( v ) (cid:54) =
p (cid:48)(cid:48)
( v (cid:48) ) .
Since p (cid:48)(cid:48) ( v ) , p (cid:48)(cid:48) ( v (cid:48) ) are the special points for p (cid:48)(cid:48)
( c ) , p (cid:48)(cid:48)
( c (cid:48) ) in { 0 , 1 } d (cid:48) (cid:48) , theycannot coincide, by Lemma 5.1.
But then p (cid:48)(cid:48) ( v ) , p (cid:48)(cid:48) ( v (cid:48) ) must have at least one different value of theircoordinates at the colours of c, c (cid:48) and hence the same is true for v, v (cid:48) .
So this gives a representationfor C \ D , as we have veriﬁed the no-clashing condition.
Applications
Theorem 6.1.
Suppose that C is an extremal class of VC dimension d containing a maximum class D of VC dimension d − 1 .
Then C has a d -size unlabelled compression scheme.
Proof.
This follows immediately by the construction in Section 5.
We only have to check that thereis a sequence of pairs of extremal classes starting at C , D , all satisfying ccc .
But this is easy sinceany maximum class of VC dimension i is a union of a complete collection of maximal cubes ofdimension i .
Moreover any maximum class of VC dimension i has reductions which are products ofmaximum classes of VC dimension i − 1 with { 0 , 1 } .
Theorem 6.2.
Suppose that C is an extremal class of VC dimension 2 .
Then C contains a properextremal class D so that for every edge of C , there is an edge of D of the same colour.
Hence C has acompression scheme of size 2 .
Proof.
Rubinstein and Rubinstein (2008) develops the idea of splitting classes by their reduction(associated with a particular projection direction): for extremal classes, splitting the reductioninduces connected components that are themselves extremal.
Choose a reduction C 1 of C whichsplits off a component E with a smallest number of vertices, amongst all choices of reductions andcomplementary components.
Let D =
C \ E .
We claim that D is extremal and contains edges of eachcolour in C .
The reason is that for any other reduction C 2 of C , if C 2 ∩ E (cid:54) = ∅ , then C 2 ∩ C 1 (cid:54) = ∅ .
For if
C 2 ∩ C 1 = ∅ , there is a smaller component which C 2 splits off from E , which is a contradiction.
Butthe condition C 2 ∩ C 1 (cid:54) = ∅ implies that D has edges of each colour of C .
Showing D is extremal isstraightforward.
Remark 6.3 .
Chalopin et al.
(2018) showed that VC- 2 extremal classes have unlabelled 2 -compressionschemes.
Theorem 6.2 gives an alternate proof using our criterion for extremal classes to have suchschemes in Section 5.
Conclusion
Chalopin et al.
(2018) constructed unlabelled compression schemes for maximum classes and extremalintersection-closed classes, while Moran and Warmuth (2016) developed labelled compressionschemes for extremal classes.
Here we have shown that intersection-closed classes can be embeddedinto extremal intersection-closed classes with an increase in VC dimension from d to 11 d and so have11 d -size unlabelled compression schemes.
We simplify and extend the d -size unlabelled compressionschemes for maximum classes due to Chalopin et al.
(2018), to apply to extremal classes of VCdimension d which contain maximum classes of VC dimension d − 1 .
We also give a sufﬁcientcondition for extremal classes to admit such compression schemes.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope? [Yes] (b) Did you describe the limitations of your work?
[Yes] See throughout for precise theoretical claims, and Section 7 for further directions addressing limitations.
(c) Did you discuss any potential negative societal impacts of your work? [No] The work
seeks constructions of sample compression schemes for PAC learnable concept classes. The focus of
the work is on learning theory.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [Yes] See the
assumptions written in each of our results. (b) Did you include complete proofs of all theoretical results? [Yes] All theoretical results are accompanied by complete proofs.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)? [N/A] No experiments
were run for this work. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they
were chosen)? [N/A] (c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)? [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)? [N/A]
If you are using existing assets (e g
, code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [N/A] We are not using
existing assets or curating/releasing new assets. (b) Did you mention the license of the assets? [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable? [N/A] This work involved no crowdsourcing or human subject research.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] ) Did you include the estimated hourly wage paid to participants and the total amount
[Table] spent on participant compensation? [N/A [Table]