Unlabelled Sample Compression Schemes for Intersection-Closed Classes and Extremal Classes
Anonymous Author(s)
AfﬁliationAddress email
Abstract
The sample compressibility of concept classes plays an important role in learningtheory, as a sufﬁcient condition for PAC learnability, and more recently as an avenuefor robust generalisation in adaptive data analysis.
Whether compression schemesof size O ( d ) must necessarily exist for all classes of VC dimension d is unknown,but conjectured to be true by Warmuth.
Recently Chalopin, Chepoi, Moran, and
Warmuth (2018) gave a beautiful unlabelled sample compression scheme of size
VC dimension for all maximum classes: classes that meet the Sauer-Shelah-Perles
Lemma with equality.
They also offered a counterexample to compression schemesbased on a promising approach known as corner peeling.
In this paper we simplifyand extend their proof technique to deal with so-called extremal classes of VCdimension d which contain maximum classes of VC dimension d − 1 .
A criterionis given which would imply that all extremal classes admit unlabelled compressionschemes of size d .
We also prove that all intersection-closed classes with VCdimension d admit unlabelled compression schemes of size at most 11 d .
Introduction
Littlestone and Warmuth (1986) initiated the study of sample compression schemes as an alternativecharacterisation of probably approximately correct (PAC) learnability, to ﬁnite Vapnik-Chervonenkis(VC) dimension.
Informally a sample compression scheme for a concept class comprises a compressorand reconstructor: while the compressor represents samples labelled by the class as a labelled orunlabelled subsample, the reconstructor must be able to recover a concept consistent with the originallabelled sample given only this representation.
Littlestone and Warmuth (1986) demonstrated aremarkably simple proof of PAC learnability for any concept classes with labelled compressionschemes provided compressed representations have bounded size, constant in the original samplesize.
They conjectured that bounded labelled compression schemes should also be necessary for PAC learnability—completing the characterisation—speciﬁcally that all VC- d concept classes (thosehaving VC dimension d ) have labelled compression schemes of size d .
This conjecture, for size O ( d ) , remains open now more than 35 years later, and stands as one of the oldest open problemsin learning theory.
In addition to proving PAC generalisation bounds (von Luxburg, Bousquet, and Schölkopf, 2004; Langford, 2005), compression has found important connections to generalisation inthe adaptive data analysis setting (Cummings, Ligett, Nissim, Roth, and Wu, 2016).
While Littlestone and Warmuth (1986) conjectured the existence of labelled schemes of size d forall VC- d classes, Warmuth (2003) later relaxed this to O ( d ) , while Kuzmin and Warmuth (2007)conjectured size O ( d ) unlabelled compression schemes.
(In unlabelled schemes, the compressor’srepresentations omit labels; such schemes of size k imply labelled schemes of size k .) Indeed VC- dclasses are known that cannot be unlabelled compressed to size d (Pálvölgyi and Tardos, 2020).
Notable progress towards resolving the sample compression conjecture has involved compressingfamilies of concept classes.
Shortly after the original conjecture Floyd (1989) proved that allmaximum concept classes C (those meeting the Sauer-Shelah-Perles Lemma with equality) canbe label compressed to size VC ( C ) (Floyd and Warmuth, 1995).
Chalopin et al.
(2018) recentlyproved the same bound but with unlabelled compression, while also providing an elegant counterexample to the same result for so-called corner peeling schemes.
This shows that the constructions ofcorner peeling schemes by Kuzmin and Warmuth (2007) and Rubinstein and Rubinstein (2012) areincorrect.
Moran and Yehudayoff (2016) proved that all VC- d classes can be label compressed tosize exponential in d .
Recently Pálvölgyi and Tardos (2020) offered examples of VC classes whichcannot admit unlabelled compression schemes of size d .
Moran and Warmuth (2016) discover labelled compression schemes for extremal classes, whichcontain cubes for any coordinates shattered by the class, and generalise maximum classes.
It isunknown if all extremal classes admit unlabelled compression schemes of size O ( d ) .
In this paper, wegive a criterion on extremal classes to admit unlabelled compression schemes of size VC dimension
A promising approach to the general conjecture—and a strong motivation for compressing specialfamilies of concept classes–is via embedding, as a compression scheme restricts to any subclass.
So, there is interest in embedding general VC-classes into larger classes with good compressionproperties.
Rubinstein, Rubinstein, and Bartlett (2015) proved that VC- d classes cannot always beembedded in maximum classes of VC dimension less than 2 d − 1 . Embedding VC- d classes intomaximum classes of VC-dimension O ( d ) could still achieve compression to size O ( d ) in general.
Intersection-closed classes—those closed under coordinate-wise multiplication—are another promising focus for compression, as any class can be embedded into an intersection-closed class.
In thispaper, we show that any intersection-closed class can be embedded into an extremal intersectionclosed class with the VC dimension increasing from d to 11 d .
These extremal intersection-closedclasses have VC-sized unlabelled compression schemes by Chalopin et al.
As such, weestablish for the ﬁrst time that intersection-closed classes can be unlabelled compressed to size linearin their VC dimension.
We offer a short proof of the result by Chalopin et al.
(2018) that maximumclasses have VC dimension
d -size unlabelled compression schemes and extend this to extremalclasses which contain maximum classes of VC dimension d − 1 .
Finally we give an efﬁcient methodto compute the VC dimension of the intersection closure of a class.
Preliminaries
Given n ∈ N we call a subset of the binary n -cube C ⊆ { 0 , 1 } n , a concept class with elements v ∈ Cconcepts .
These come from restrictions of classiﬁer families, on samples of n instances.
1 We equateeach instance in [ n ] = { 1 , . . . , n } with coordinates , axes or colours of the n -cube.
An importantcombinatorial parameter is the VC dimension, which is used to bound concept class cardinality.
Deﬁnition 2.1 (Vapnik and Chervonenkis, 1971) .
The VC dimension of concept class C ⊆{ 0 , 1 } n is deﬁned as VC ( C ) =
max (cid:8) | I | : I ⊆
[ n ] , proj I ( C )
= { 0 , 1 } | I | (cid:9) , where proj I ( C ) = 72 (cid:8) ( v i ) i ∈ I : v ∈ C (cid:9) is the set of coordinate projections of the concepts of C on coordinates I ⊆
[ n ] .
C is said to shatter I when the projection proj I ( C ) = { 0 , 1 } | I | is onto the | I | -cube.
Lemma 2.2 (Sauer-Shelah-Perles Lemma, Vapnik and Chervonenkis, 1971; Shelah, 1972; Sauer,1972) .
The cardinality of any concept class C ⊆ { 0 , 1 } n is bounded by |C| ≤ (cid:80) [Table]
Concept classes meeting equality in Sauer’s Lemma are called maximum (Welzl, 1987).
A cubical–complex structure can be put on concept classes C ⊆ { 0 , 1 } n .
To begin, each v ∈ C is a vertex, andpairs of Hamming-1 separated vertices are edges, of the one-inclusion graph of C (Haussler, Littlestone, and Warmuth, 1994).
Each higher-order cube is all points varying over a set of coordinates, thecube’s colours ; a cube’s common values on non-colour coordinates make up its anchor .
(Equivalentlyd -cubes are pre-images of points under a projection from the binary ( m + d ) -cube to the binarym -cube.)
Any d -maximum class has a special structure as a complete collection of d -cubes:
Deﬁnition 2.3 (Rubinstein, Bartlett, and Rubinstein, 2009) .
A set of d -cubes S of cardinality (cid:0) nd (cid:1) iscalled d -complete if for all I ⊆
[ n ] of cardinality d , there exists c I ∈ S with proj
I ( c I ) = { 0 , 1 } d .
Another property of a concept class C is of being shortest-path closed —also known as isometric(Chalopin et al., 2018)—where every pair of points in C has a shortest path contained in C (Kuzminand Warmuth, 2007; Rubinstein and Rubinstein, 2012).
We will be especially interested in extremalclasses—also known as ample classes (Chalopin et al., 2018).
Deﬁnition 2.4.
A class C ⊆ { 0 , 1 } n is extremal if whenever C shatters some I ⊆
[ n ] , then there is asubcube c ⊆ C which maps bijectively to the | I | -cube via proj I ( · ) .
That is, proj I ( c ) = { 0 , 1 } | I | .
All maximum classes are extremal and extremal classes are shortest-path closed (Moran and Warmuth,2016), while the complements of extremal classes are themselves extremal (Bandelt, Chepoi, Dress,and Koolen, 2006; Bollobás and Radcliffe, 1995; Lawrence, 1983).
As an example, a VC onedimensional maximum (respectively extremal) class is a tree with one (respectively at most one) edgefor each colour.
An extremal counter-part to Sauer’s Lemma was independently discovered by Pajor(1985), Bollobás and Radcliffe (1995), Dress (1997), and Anstee, Rónyai, and Sali (2002).
Lemma 2.5 (Sandwich Lemma) .
For any extremal class C , its cardinality is equal to both the numberof shatterings of C and the number of cube types (an equivalence class of cubes which all have thesame set of colours) in C .
Extremal and maximum classes have an elegant recursive structure that is best studied by inductivearguments.
One such tool is the projection operator proj I ( C ) .
The reduction C I ⊆
proj
I ( C ) isthe subset of points with non-unique pre-images under the projection, while the tail is the subset ofproj
I ( C ) with unique pre-image under the projection (Welzl, 1987; Kuzmin and Warmuth, 2007).
Unlabelled compression schemes are deﬁned by representation mappings .
A party known as thecompressor uses a representation mapping to summarise a C -labelled sample of any size with asmall unlabelled subset of instances.
A second party, the reconstructor , can invert this representationthanks to condition 1, to a concept that is consistent with the original labelled sample, thanks tocondition 2, (Chalopin et al., 2018).
Deﬁnition 2.6 (Kuzmin and Warmuth, 2007) .
Mapping r is a representation mapping of size k < n ,also known as an unlabelled compression scheme of size k , of class C ⊆ { 0 , 1 } n if it satisﬁes:1.
r is an injection between C and subsets of [ n ] of size at most k ; and
(non-clashing) proj r ( v ) ∪ r ( v (cid:48) )
( v ) (cid:54) =
proj r ( v ) ∪ r ( v (cid:48) )
( v (cid:48) ) for all v (cid:54) = v (cid:48) ∈ C .
The intersection closure operator
Examples of intersection-closed classes include axis-aligned hyperrectangles in R m , monomials,which are unions of orthogonal subcubes, and any union of orthogonal intersection-closed subclasses (Moran and Warmuth, 2016).
Closed-below classes (Rubinstein et al., 2009) are intersectionclosed and extremal.
All extremal classes of VC dimension 1 are intersection closed, so long as theorigin is located in the class (Moran and Warmuth, 2016).
We will use the notation v (cid:12) w to denote the intersection of the vertices v, w .
This is just theoperation of bitwise Boolean multiplication v (cid:12) w = ( v 1 w 1 , . . .
, v n w n ) .
We can likewise letu 1 (cid:12) . . .
(cid:12) u k
=
( (cid:81)
ki =1 u i 1 , . . .
, (cid:81)
ki =1 u in ) denote the (iterated) intersection of vertices u 1 , . . .
, u k .
Deﬁnition 3.1.
A class C in the binary n -cube is called intersection closed if v (cid:12) w ∈ C , wheneverv, w ∈ C .
Let C =
(cid:12) ( C ) =
{ u 1 (cid:12) . . .
(cid:12) u k ) : k ≤ n, { u 1 , . . .
, u k } ⊆ C} denote the intersectionclosure of any class C obtained via intersections of all ﬁnite subsets of C ’s concepts.
Lemma 3.2.
For any concept class C ⊂ { 0 , 1 } n , the intersection closure C is the unique smallestintersection-closed class containing C , and it is contained in any intersection-closed class containing
C .
Moreover for any J ⊆ [ n ] , we must have proj J ( C ) =
proj J (cid:0) C (cid:1) .
Next is a useful result on computing the VC dimension of the intersection closure of a class.
Itsproof is relegated to Appendix A.
The key idea is that to verify proj J (cid:0) C (cid:1) shatters, it is necessary andsufﬁcient to show C has | J | + 1 vertices which project to the vertices with at most one coordinate 0 .
Deﬁnition 3.3.
Suppose C is a class in the binary n -cube with VC dimension d .
We will say that
C satisﬁes the k -close cube condition if there is a vertex v ∈ { 0 , 1 } n and a complete union of( n − k − 1) -cubes in the complement of C so that each cube is distance at most one from v . For anyk (cid:48) > k , C then also satisﬁes the k (cid:48) -close cube condition.
Theorem 3.4.
Suppose C is a class in the binary n -cube.
The smallest VC dimension d for theintersection closure of C over all choices of origin, is the same as the smallest k for which C satisﬁesthe k -close cube condition.
We end with a useful property of the intersection closure operator.
We will abuse notation by saying aclass A is shortest path closed inside a class B , if A ⊂ B and any two vertices of A are connected bya shortest path inside B .
The main idea in the following theorem is that if C is shortest-path closed,then X = { v : v = u (cid:12) w, u, w ∈ C} is shortest path closed inside C .
Theorem 3.5.
If C is shortest-path closed then its intersection closure C is also shortest-path closed.
Proof.
We ﬁrst observe the easy fact that if λ is a shortest path in C consisting of verticesv 1 , v 2 , . .
. , v k and u ∈ C then u (cid:12) λ := { u (cid:12) v i : i ∈ [ k ] } is a shortest path in C possiblywith repetitions.
For an edge between successive vertices v i , v i +1 is transformed into u (cid:12) v i , u (cid:12) v i +1which is either an edge of the same colour or a single vertex.
Our main claim that if C is shortest-path closed then
X
=
{ v : v = u (cid:12) w, u, w ∈ C} is shortest pathclosed inside C .
Once this claim is established, the proof follows easily by repeating this step.
For wecan apply the claim to the class X to prove that the class { v : v = u (cid:12) w, u, w ∈ X} consisting ofall elements of C together with intersections of up to 4 elements of C is shortest-path closed.
inside
X = C .
Repeating at most log 2 n times, C is obtained and is shortest-path closed.
(Note that sincev = v (cid:12) v , elements of C can be written as intersections of pairs of elements of C ).
The ﬁrst step in proving the main claim is to observe that there is a shortest path in X connecting anyelement u ∈ C to any element u (cid:12) v ∈ X .
To construct this path, start with a shortest path λ betweenu, v in C and then form u (cid:12) λ .
As above this is a shortest path, possibly with repetitions, betweenu, u (cid:12) v in X .
We conclude that given two vertices u, v ∈ C , there is a shortest path in X from u tov passing through u (cid:12) v , by concatenating shortest paths in X from u to u (cid:12) v and u (cid:12) v to v .
To prove the main claim, choose elements v, v (cid:48)
∈ X .
We want to ﬁnd a shortest path in C connectingv, v (cid:48) .
Let v = u (cid:12) w, v (cid:48)
=
u (cid:48) (cid:12) w (cid:48) where u, w, u (cid:48) , w (cid:48) ∈ C .
By the ﬁrst step, there are shortest pathsλ, λ (cid:48) in X connecting u, u (cid:48) and w, w (cid:48) and passing through u (cid:12) u (cid:48) , w (cid:12) w (cid:48) respectively.
Next, observe that v (cid:12) v (cid:48) =
u (cid:12) w (cid:12) u (cid:48) (cid:12) w (cid:48) =
u (cid:12) u (cid:48) (cid:12) w (cid:12) w (cid:48) .
Split λ, λ (cid:48) into λ i , λ (cid:48)
i for i = 1 , 2so that λ 1 , λ (cid:48) 1 join u, w and u (cid:12) u (cid:48) , w (cid:12) w (cid:48) respectively and λ 2 , λ (cid:48)
2 join u (cid:12) u (cid:48) , w (cid:12) w (cid:48) and u (cid:48) , w (cid:48)respectively.
To establish the main claim, it clearly sufﬁces to construct shortest paths in C from v tov (cid:12) v (cid:48) and from v (cid:12) v (cid:48) to v (cid:48) .
By symmetry, we will focus on the ﬁrst of these.
We will prove a slightly stronger claim.
Namely assume λ 1 consists of ordered vertices
u =
z 1 , z 2 , . .
z k = u
(cid:12) u (cid:48) and λ (cid:48) 1 has ordered vertices w = y 1 , y 2 , . . .
, y r = w (cid:12) w (cid:48) .
We will showthat a shortest path in X connecting v, v (cid:12) v (cid:48) can be chosen to have all vertices of the form
z i (cid:12) y j .
To complete the proof, consider the rectangular grid of intersections
z i (cid:12) y j , for 1 ≤ i ≤ k and1 ≤ j ≤ r with k, r ≥ 1 .
We claim that there is a shortest path between the diagonally oppositeentries v =
z 1 (cid:12) y 1 , v (cid:12) v (cid:48)
=
z k (cid:12)
y r using vertices in this grid.
It sufﬁces to use the boundary of the grid to do this.
So for example, consider the ﬁrst path
z i (cid:12) y 1 for1 ≤ i ≤ k , followed by a second path
z k (cid:12) y j for 1 ≤ j ≤ r .
By the ﬁrst observation in this proof,the ﬁrst path is shortest with possible repetitions, as is the second path.
To complete the argument,we need to show that the concatenation of these two paths is also shortest with possible repetitions.
Notice that the path λ
1 =
z 1 , z 2 , . .
z k runs between ends z 1 , z k where z k is between the originof the binary n -cube and z 1 .
So the same follows for the path with possible repetitions
z i (cid:12) y 1 for1 ≤ i ≤ k .
The second path z k (cid:12) y j for 1 ≤ j ≤ r is of the same form and hence the concatenationof these two paths is also shortest with repetitions and the theorem proof is complete.
Extremal intersection-closed classes
In this section, we start with a useful way of ﬁnding extremal intersection-closed classes, namelyby checking they are shortest-path and intersection-closed.
We use this to establish the main resultabout intersection-closed classes, namely that they can be embedded into extremal intersection closedclasses, with the VC dimension d increasing to at most 11 d .
We then use a result of Chalopin et al.
(2018) to conclude this gives unlabelled compression schemes of size 11 d for an intersection-closedclass of VC dimension d .
Theorem 4.1.
Suppose that C is shortest-path and intersection closed.
Then C is extremal.
Proof.
Suppose a projection p : { 0 , 1 } n → { 0 , 1 } k shatters C .
We will prove there is a k -subcube cof C which witnesses the shattering.
Consider all the vertices of C which map to the vertex w = (1 , 1 , . .
, 1) in the k -cube.
Choose avertex v in this set which has smallest distance to the origin.
(Since C is shortest-path closed andintersection closed, the origin is in C and distance in C is the same as distance in the n -cube.)
Our claim is that there are vertices v i ∈ C , i ∈
[ k ] , so that the distance from v to each v i is one andp ( v i ) =
, 1) =
w i the vertex in the k -cube with all coordinates 1 except a single 0 atthe i th coordinate.
Once this claim is established, it follows that since C is intersection closed, takingthe intersections of subcollections of the vertices { v i : i ∈ [ k ] } generates the required k -subcube cof C .
Note c witnesses the shattering, since projection commutes with intersection and the verticesw, w i generate c under the operation of taking intersections.
To prove the claim, choose any vertex u ∈ C
so that p ( u ) =
w
i .
Since C is intersection closed,u (cid:48)
= v (cid:12) u ∈ C .
Then p ( u (cid:48) )
=
p ( v (cid:12) u ))
=
p ( v ) (cid:12) p ( u ) = w (cid:12) w i = w i .
Note that u (cid:48) is a vertexbetween v and the origin, i.e. , for any coordinate where u (cid:48) has value 1 then v must also.
Since C isshortest-path closed, a shortest path λ exists in C from v to u (cid:48) .
Moreover λ has vertices in order withdistances decreasing to the origin, since u (cid:48) is between v and the origin.
Let v (cid:48) be the ﬁrst vertex on λ after v .
Since p ( λ ) is a shortest path in the k -cube from w to w i , itfollows that p ( v (cid:48) ) is either w or w i .
If the former, this contradicts our choice of v as a closest vertexin C to the origin which projects to w .
If the latter then we have found our vertex v i with distance 1to v and which projects to w i .
Remark 4.2 .
Chalopin et al.
(2018) discusses the signiﬁcance of properties of extremal intersectionclosed classes, also known as conditional antimatroids .
Remark 4.3 .
There is a partial order on the binary n -cube given by w ≤ v when w is between v andthe origin or equivalently there is a shortest path from v to the origin passing through w .
Choose anarbitrary ordering on [ n ] .
We will denote by λ ( v, w ) the shortest path between v, w with verticesv =
v 1 , v 2 , . . .
v k = w where v i .v
i +1 differ in the ﬁrst coordinate x j in the ordering of indices jamongst all coordinates where v i , w differ.
The key idea of the next result is to enlarge an intersection closed class C by adding all the vertices inthe shortest paths λ ( v, w ) between all pairs v, w ∈ C , giving a class which is both intersection-closedand shortest-path closed.
Theorem 4.4.
Any intersection-closed class C embeds in a shortest-path closed and intersectionclosed class C ∗ so that d ∗ ≤ 11 d where d, d ∗ are the VC dimensions of C , C ∗ respectively.
Toconstruct C ∗ , for every pair of vertices v, w ∈ C with w < v , add all the vertices in λ ( v, w ) to C .
Lemma 4.5.
If C is intersection closed then C ∗ as constructed by Theorem 4.4 is intersection closed.
Proof.
By deﬁnition, for any vertices u, u (cid:48)
∈ C ∗ , there are vertices v, w, v (cid:48) , w (cid:48) ∈ C
so that u ∈λ ( v, w ) , u (cid:48) ∈ λ ( v (cid:48) , w (cid:48) ) .
Moreover, w ≤ v, w (cid:48) ≤ v (cid:48) .
Since C is intersection closed, both v (cid:12) v (cid:48) and w (cid:12) w (cid:48) are in C .
We claim that u (cid:12) u (cid:48) is on the pathλ ( v (cid:12) v (cid:48) , w (cid:12) w (cid:48) ) —note that as part of this claim, we require that w (cid:12) w (cid:48) ≤ v (cid:12) v (cid:48) .
We begin with the latter claim.
This is easy— w (cid:12) w (cid:48) has entry 1 at precisely the coordinates whereboth w, w (cid:48) have entry 1 .
But since w ≤ v, w (cid:48) ≤ v (cid:48) , v, v (cid:48) has entry 1 at all the coordinates wherew, w (cid:48) respectively has entry 1 .
So for any coordinate where w (cid:12) w (cid:48) has entry 1 , it follows immediatelythat both v, v (cid:48) have entry 1 and so does v (cid:12) v (cid:48) .
Next, the set of coordinates where v has entry 1 and u has entry 0 forms an initial segment of thecoordinates where v has entries 1 , using the ordering of the coordinates, since u ∈ λ ( v, w ) .
A similaranalysis applies to the set of coordinates of u (cid:48) which are 0 but the entries for v (cid:48) are 1 .
To complete the proof that C ∗ is intersection closed, we show that the set of coordinates where theentries of v (cid:12) v (cid:48) are 1 and the entries of u (cid:12) u (cid:48) are 0 , forms an initial segment of the ordering of thecoordinates where the entries of v (cid:12) v (cid:48) are 1 .
It will then follow
that u (cid:12) u (cid:48) ∈
λ ( v (cid:12) v (cid:48) , w (cid:12) w (cid:48) ) .
Now the set of coordinates where v (cid:12) v (cid:48) has entries 1 are exactly those coordinates where both v, v (cid:48)have entries 1 .
Moreover u (cid:12) u (cid:48) has entry 0 at a coordinate if either u or u (cid:48) has entry 0 .
We completethe proof using an argument by contradiction.
Suppose that there is a coordinate x i such that u (cid:12) u (cid:48)has entry 1 but this occurs before a coordinate x j where u (cid:12) u (cid:48) has entry 0 and moreover v (cid:12) v (cid:48) hasentry 1 .
Then at x j , either u or u (cid:48) has entry 0 .
Without loss of generality assume the former.
Sinceu (cid:12) u (cid:48) has entry 1 at x i it follows that u has entry 1 at x i .
Note also that v has entry 1 at both x i , x j .
But this is a contradiction since the coordinates where v has entry 1 but u has entry 0 should form aninitial segment of the set of coordinates where v has entry 1 .
But we have found j > i where u hasentry 1 at x i and 0 at x j .
This establishes that C ∗ is intersection closed.
Lemma 4.6.
If C is intersection closed then C ∗ as constructed by Theorem 4.4 is shortest-path closed.
Proof.
By Lemma 4.5, it sufﬁces to show that if v, w ∈ C ∗ then there are shortest paths from v, w tov (cid:12) w , since putting these together will give a shortest path from v to w .
Next, it sufﬁces to showthere is a shortest path in C ∗ between any pair of vertices v, u in C ∗ , where u ≤ v , for we can thenapply this to the pairs v, v (cid:12) w and w, v (cid:12) w .
We claim that there is a vertex v 1 ∈ C so that u ≤ v 1 , v ≤ v 1 and there are shortest paths in C ∗ joiningboth v 1 , u and v 1 , v .
Firstly, since v ∈ C ∗ , there are vertices v 1 , v 2 ∈ C so that v is in λ ( v 1 , v 2 ) .
Hence v ≤ v 1 and there is a shortest path in C ∗ from v 1 to v , namely a segment of λ ( v 1 , v 2 ) .
Next, since u ≤ v , then u ≤ v 1 .
Moreover u lies on a path λ
( u 1 , u 2 ) where u 1 , u 2 ∈ C , since u ∈ C ∗ .
Then v 1 (cid:12) u 1 ∈ C and v 1 (cid:12) u 1 ≤ u 1 .
Since u ≤ v 1 , u ≤ u 1 , it follows that u ≤ v 1 (cid:12) u 1 .
Notice that u must lie on λ ( v 1 (cid:12) u 1 , u 2 ) .
To verify this, observe all the coordinates where entriesof v 1 (cid:12) u 1 differ from entries of u are also coordinates where u 1 entries differ from u entries.
Consequently, coordinates where entries of u and u 2 differ, are lower in the coordinate ordering thanthose where entries of v 1 (cid:12) u 1 and u differ.
So this conﬁrms that u ∈ λ ( v 1 (cid:12) u 1 , u 2 ) .
Combining λ ( v 1 , v 1 (cid:12) u 1 ) and λ ( v 1 (cid:12) u 1 , u ) gives a shortest path from v 1 to u in C ∗ .
So thiscompletes the proof of the claim about v 1 .
To recapitulate, we have constructed shortest paths in C from v 1 to both u, v in C ∗ , where v ≤ v 1 , u ≤v 1 , u ≤ v .
To complete the proof that there is a shortest path from v to u in C , we use induction onthe Hamming distance from v 1 to v .
To begin the induction, suppose this distance is 0 .
Then clearlywe are done, since there is a shortest path in C ∗ from v to u .
Assume the distance from v 1 to v is k and the result is true for v 1 ∗ , v ∗ , u ∗ as above, where thedistance from v 1 ∗ to v ∗ is less than k .
Let v (cid:48) be the ﬁrst vertex in the shortest path from v 1 to v .
Ouraim is to show that there is a shortest path in C ∗ from v (cid:48) to u .
By induction, it then follows there is ashortest path in C ∗ from v to u .
The argument is similar to that in Theorem 3.5.
Let v 1 = y 1 , y 2 , . . .
y r
= u be a shortest path in
C ∗ .
We can construct a shortest path v (cid:48)
= v (cid:48) (cid:12) y 1 , v (cid:48) (cid:12) y 2 , . .
, v (cid:48) (cid:12) y r =
u where there is a singlerepetition.
Namely, since one of the edges joining y j , y j +1 must have the same colour as the edgebetween v 1 , v (cid:48) , then v (cid:48) (cid:12) y
j
= v (cid:48) (cid:12) y
j +1 .
All other pairs of vertices are distance one apart and since
C ∗ is intersection closed, this shortest path is in C ∗ .
So this completes the proof.
Proof.
(Theorem 4.4)
It remains to prove that the VC dimension of C ∗ is at most 11 d .
Choose any projection p : { 0 , 1 } n → { 0 , 1 } 11 d .
By Sauer’s Lemma, since the VC dimension of p ( C )is at most d , the cardinality | p ( C ) | is at most F ( d ; 11 d, 12 ) × 2 11 d where F ( k ; n, p ) is the cumulativebinomial probability function.
Algorithm 1 constructing C ∗ involves ﬁnding and adjoining shortest paths connecting suitable orderedpairs of points in C .
So |C ∗ | ≤ n × |C| 2 .
Similarly we see that | p ( C ∗ ) | ≤ 11 d × | p ( C ) | 2 , sincethe process of adjoining shortest paths is preserved under projection.
Hence | p ( C ∗ | ≤ 11 d ×
F ( d ; 11 d, 12 ) 2 × 2 22 d × 12 .
Note that as pairs ( v, w ) are ordered by requiring that w ≤ v , we get afactor of 12 .
We claim that this upper bound is strictly less than 2 11 d .
Then p cannot shatter and the
VC dimension of C ∗ is at most 11 d − 1 .
The key idea is to use Chernoff’s inequality to estimate F ( d ; 11 d, 12 ) .
This is:
[Equation]
where D ( 111 || 12 ) = 111 log 211 + (1 − 111 ) log 2(1 − 111 ) .
Substituting this into the previous upper bound and simplifying, we obtain | p ( C ∗ ) | ≤ 11 d × 2 22 dexp(22 d log 11 − 2 d log 2 − 20 d log 20) .
We can further simplify this to | p ( C ∗ ) | ≤ 11 d × 11 22 d × 10 − 20 d × 12 .
To complete the argument, we need to show the right side is strictly less than 2 11 d .
Hence it sufﬁces to show that
[Equation]
Now if d ≥ 5 then 11 1 d < 1 .
7 and d 1 d < 1 . 4 .
So we can compute the left side is at most1 .
7 × 1 .
× 121 × 111020 < 1937 whereas the right side is 2048 .
(Here we leave out the factor 2 − 1 d .)
If d = 4 then 112 14 < 1 . 6 and 4 14 < 1 .
So we can compute the left side is as at most 1 . 6 × 1 . 42 ×121 × 111020 =
1810 whereas the right side is 2048 .
So this completes the case when d ≥ 4 .
For theother cases, it is convenient to work directly, as Chernoff’s inequality is not sharp.
For d = 1 , | p ( C ) | ≤ 12 , as it has VC dimension at most 1 in the 11 -cube.
Hence | p ( C ∗ ) | ≤11 × 12 2 × 12 = 792 which is less than 2048 .
Next suppose d = 2 .
| p ( C ) | ≤ 254 , as it has VC dimension at most 2 in the 22 -cube.
Hence | p ( C ∗ ) |is bounded above by 22 × 254 2 × 12 = 709 , 676 which is less than 2048 2 = 4 , 194 , 304 .
Finally assume d = 3 .
| p ( C ) | ≤ 6018 , as it has VC dimension at most 3 in the 33 -cube.
Hence | p ( C ∗ ) |is bounded above by 33 × 6018
× 12 = 597 , 569 , 346 which is less than 2048 3 = 8 , 589 , 934592 .
Corollary 4.7.
If C is an intersection-closed class, then C has a (corner peeling) unlabelled compression scheme of size at most 11 d .
Proof.
This follows immediately by combining Theorem 4.4 and (Chalopin et al., 2018, Theorem4.9).
Unlabelled compression schemes for extremal classes
The aim is to give a condition on extremal classes which implies that they have unlabelled compressionschemes.
We then give some situations where this criterion is satisﬁed.
We conjecture that it is alwaysvalid.
For a maximum class C of VC dimension d , a reduction is a maximum subclass of VC dimensiond − 1 .
So a reduction contains cubes with the same set of colours as any non-maximal cube of C .
Itwill turn out this is the key property we require of extremal classes to have unlabelled compressionschemes of size d , namely they have proper extremal subclasses which have the same properties asreductions of maximum classes.
5.1 High-level strategy
The idea is inspired by Chalopin et al.
Start with a pair ( C , D ) consisting of an extremal classand a proper extremal subclass.
The condition is that for every cube in C which is not maximal, thereis a cube with the same colours in D .
We call this the cubical colour condition of the pair C , D andabbreviate it by ccc .
We will construct a bijection φ between the set S of maximal cubes c of C not in D and C \ D , so thatv = φ ( c ) ∈ c .
Note the cardinalities of these two sets are the same, by the Sandwich Lemma 2.5applied to the two extremal classes C , D .
Then φ − 1 : C \D → S gives a representation map for C \D ,which also separates the vertices of C \ D from the vertices of D .
Therefore, any representation of Dcan be combined with the representation of C \ D to give a representation of C .
Suppose we can ﬁnd a sequence of proper pairs of extremal subclasses ( C , D ) , ( D , E ) , . . .
, ( X , Y ) ,all satisfying ccc , ending with VC ( Y ) ≤ 2 .
Then we can start with a representation map of the VC-2 class Y (Chalopin et al., 2018) and extend to a representation of X , then extending the representationall the way to C using this strategy.
5.2 Detailed construction
We construct φ : C \ D → S and show it is a bijection.
Firstly, denote the extremal class which is thecomplement of D by D ∗ .
Then every vertex of C \ D is in at least one maximal cube of D ∗ .
Deﬁne two cubes c, c ∗ in { 0 , 1 } n to be complementary if the dimensions d, d ∗ of c, c ∗ satisfyd + d ∗ = n and c ∩ c ∗ is a single vertex.
Recall that S is the set of maximal cubes c of C not in D .
Denote the set of maximal cubes of D ∗ which intersect C by S ∗ .
Lemma 5.1.
Assume ( C , D ) is a proper pair of extremal classes satisfying ccc .
If c ∈ S , then there isa complementary maximal cube c ∗ of D ∗ .
Conversely, if c ∗ is a maximal cube of S ∗ , then there isa complementary cube c ∈ S .
A maximal cube c ∗ of S ∗ , cannot have two different complementarycubes in S .
Given Lemma 5.1, we will call v = c
∩ c ∗ the special vertex of c .
We now note properties of v ,assuming Lemma 5.1.
Corollary 5.2.
Assume that ( C , D ) is a proper pair of extremal classes satisfying ccc .
Let v =
c ∩ c ∗where c ∈ S and c
∗ ∈ S ∗ are complementary maximal cubes.
Then
• The set of vertices v = c ∩ c ∗ , for all pairs of complementary maximal cubes c ∈ S andc ∗ ∈ S ∗ , is C \ D .
• The mappings between c, c ∗ , v are bijections between S , S ∗ and C \ D .
Proof.
Lemma 5.1 gives a bijection between S and S ∗ , by the mapping from cubes c to complementarycubes c ∗ .
Moreover by the Sandwich Lemma applied to extremal classes C , D , the cardinalities of
C \ D , S ∗ are the same.
Since the map from maximal cubes in S to special vertices is an injection by
Lemma 5.1, it must be a bijection onto C \ D .
Proof of Lemma 5.1.
Let p : { 0 , 1 } n → { 0 , 1 } d be the projection to the colours of c ∈ S .
Recall cccfor the pair C , D says that for any cube of C which is not maximal, there is a cube of D with thesame set of colours.
So p ( D ) = { 0 , 1 } d \ { w } for some vertex w .
For the only other case is thatp ( D ) = { 0 , 1 } d .
But since D is extremal, this would imply D contained a cube with the same coloursas c .
This contradicts the assumption that c is maximal in C , since for an extremal class, maximalcubes are unique for their sets of colours.
Deﬁne v, c ∗ by v = p − 1 ( w ) ∩ c, c ∗ =
p − 1 ( w ) respectively.
Clearly c ∩ c ∗ = v .
It remains to showthat c ∗ is maximal in D ∗ and c ∗ cannot intersect some other cube of S in a single vertex.
Suppose that c ∗ was properly contained in a larger cube ˜ c in D ∗ .
Then p (˜ c ) would properly contain{ w } .
Moreover ˜ c = p − 1 p (˜ c ) .
But this contradicts our observation above that p ( D ) = { 0 , 1 | d \ { w } .
So this completes the argument that c ∗ is maximal.
Next, assume that c ∗ is complementary to a second maximal cube ˆ c ∈ S .
Consider the projection ˆ pof { 0 , 1 } n to { 0 , 1 } ˆ d given by the ˆ d colours of ˆ c .
As above, ˆ p ( D ) consists of { 0 , 1 } ˆ d with a singlevertex ˆ w removed.
Moreover, ˆ p − 1 ( ˆ w ) is a maximal cube c (cid:48) of D ∗ .
Now as c ∗ ∩ ˆ c = u ∗ is a single vertex, ˆ p ( c ∗ )
= ˆ p ( u ∗ ) = u .
As the cubes c ∗ , ˆ c are complementary,c ∗ =
ˆ p − 1 ( u ) .
But now we see that the colours of the anchor of c ∗ are both the colours of c and of ˆ c .
This gives a contradiction, since two maximal cubes of C cannot have the same set of colours.
Thiscompletes the proof of Lemma 5.1.
Finally we show that the map from a special vertex v to the colours of the corresponding maximalcube c of S gives a representation mapping of C \ D which also separates C \ D from D .
The second property follows immediately.
For recall that if p is the projection given by the d coloursof c , then p ( D ) = { 0 , 1 } d \ { p ( v ) } .
So the colours of c clearly separate v from all the vertices of D .
By this we mean that given any vertex w ∈ D , v, w differ on at least one coordinate from the coloursof c .
Next suppose two vertices v, v (cid:48) have the same values at all the coordinates corresponding to thecolours of their corresponding cubes c, c (cid:48)
∈ S of dimensions d, d (cid:48) respectively.
Project { 0 , 1 } n tothe union of the colours of c, c (cid:48) by a mapping p (cid:48)(cid:48) .
The image of p (cid:48)(cid:48) is a cube { 0 , 1 } d (cid:48)(cid:48) of dimensiond (cid:48)(cid:48) ≤ d + d (cid:48) .
In this cube, C , D map to extremal classes p (cid:48)(cid:48)
( C ) =
C (cid:48)(cid:48) , p (cid:48)(cid:48)
( D ) =
D (cid:48)(cid:48) .
We want to check that the pair of extremal classes ( C (cid:48)(cid:48) , D (cid:48)(cid:48) ) satisfy ccc .
But this follows easily.
Choose any non maximal cube ˜ c ∈ C (cid:48)(cid:48) .
As C is extremal, there is a cube c 0 in C which projectsone-to-one to ˜ c and therefore has the same colours as ˜ c .
Since c 0 is not maximal, there is a cube c 1 in
D with the same colours as c 0 .
Then p (cid:48)(cid:48) ( c 1 ) is the required cube in D (cid:48)(cid:48) with the same colours as ˜ c asrequired.
Next, c, c (cid:48) project one-to-one by p (cid:48)(cid:48) to cubes with colours spanning all of { 0 , 1 } d (cid:48)(cid:48) .
Finally we claimthat p (cid:48)(cid:48) ( v ) (cid:54) =
p (cid:48)(cid:48)
( v (cid:48) ) .
Since p (cid:48)(cid:48) ( v ) , p (cid:48)(cid:48) ( v (cid:48) ) are the special points for p (cid:48)(cid:48)
( c ) , p (cid:48)(cid:48)
( c (cid:48) ) in { 0 , 1 } d (cid:48) (cid:48) , theycannot coincide, by Lemma 5.1.
But then p (cid:48)(cid:48) ( v ) , p (cid:48)(cid:48) ( v (cid:48) ) must have at least one different value of theircoordinates at the colours of c, c (cid:48) and hence the same is true for v, v (cid:48) .
So this gives a representationfor C \ D , as we have veriﬁed the no-clashing condition.
Applications
We give two results where the construction in Section 5 can be applied to special collections ofextremal classes.
Theorem 6.1.
Suppose that C is an extremal class of VC dimension d containing a maximum class Dof VC dimension d − 1 .
Then C has a d -size unlabelled compression scheme.
Proof.
This follows immediately by the construction in Section 5.
We only have to check that thereis a sequence of pairs of extremal classes starting at C , D , all satisfying ccc .
But this is easy sinceany maximum class of VC dimension i is a union of a complete collection of maximal cubes ofdimension i .
Moreover any maximum class of VC dimension i has reductions which are products ofmaximum classes of VC dimension i − 1 with { 0 , 1 } .
Theorem 6.2.
Suppose that C is an extremal class of VC dimension 2 .
Then C contains a properextremal class D so that for every edge of C , there is an edge of D of the same colour.
Hence C has acompression scheme of size 2 .
Proof.
Rubinstein and Rubinstein (2008) develops the idea of splitting classes along a reduction(associated to a coordinate projection).
Choose a reduction C 1 of C which splits off a component Ewith a smallest number of vertices, amongst all choices of reductions and complementary components.
Let D =
C \ E .
We claim that D is extremal and contains edges of each colour in C .
The reason is that for any other reduction C 2 of C , if C 2 ∩ E (cid:54) = ∅ , then C 2 ∩ C 1 (cid:54) = ∅ .
For if
C 2 ∩ C 1 = ∅ , there is a smaller component which C 2 splits off from E , which is a contradiction.
Butthe condition C 2 ∩ C 1 (cid:54) = ∅ implies that D has edges of each colour of C .
Showing D is extremal isstraightforward.
Remark 6.3 .
Chalopin et al.
(2018) showed that VC- 2 extremal classes have unlabelled 2 -compressionschemes.
Theorem 6.2 gives an alternate proof using our criterion for extremal classes to have suchschemes in Section 5.
Conclusion
Chalopin et al.
(2018) constructed unlabelled compression schemes for maximum classes and extremalintersection-closed classes, while Moran and Warmuth (2016) developed labelled compressionschemes for extremal classes.
Here we have shown that intersection-closed classes can be embeddedinto extremal intersection-closed classes with an increase in VC dimension from d to 11 d and so have11 d -size unlabelled compression schemes.
We simplify and extend the d -size unlabelled compressionschemes for maximum classes due to Chalopin et al.
(2018), to apply to extremal classes of VCdimension d which contain maximum classes of VC dimension d − 1 .
We also give a sufﬁcientcondition for extremal classes to admit such compression schemes.
A Proof of Theorem 3.4
Suppose ﬁrst that d achieves the smallest VC dimension of the intersection closure of C over allchoices of origin.
Let C denote the corresponding intersection closure of C .
Then any projection p of
C to a ( d + 1) -cube is not onto.
Assume p ( C ) contained the vertex w = (1 , 1 , . .
. , 1) and all verticesw i =
. , 1) with 0 in the i th position and all other entries 1 .
Then since p ( C ) isintersection closed, the image is the whole ( d + 1) -cube contrary to assumption.
The reason is thattaking intersections of combinations of the vertices w i gives all the vertices in the ( d + 1) -cube.
So atleast one of the vertices w, w 1 , . . .
, w d +1 must not be in p ( C ) .
But then there is an ( n − d − 1) -cubewith
anchor of this form in the complement of C by taking the inverse image under p of this vertex.
Bychoosing v as the vertex with all entries 1 in the n -cube, we have shown that C satisﬁes the d -closecube condition.
For clearly each of our ( n − d − 1) -cubes in the complement of C has distance atmost one from v .
So this establishes that d ≥ k , where k is the smallest value so that C satisﬁes thek -close cube condition.
Conversely suppose that C has a collection of ( n − k − 1) -cubes in its complement, validating that
C satisﬁes the k -close cube condition.
We claim that d ≤ k (cid:48) , where d is the VC dimension of theintersection closure of C for a suitable choice of origin.
This shows that the smallest possible VCdimension d of the intersection closure of C over all choices of origin is not larger than the smallestpossible k where C satisﬁes the k -close cube condition.
For suppose that any projection p of C to the ( k + 1) -cube is selected.
By Lemma 3.2, p ( C ) =
p ( C ) .
But now by assumption, if we choose the origin so that v is the vertex with all entries 1 , then there isan ( n − k − 1) -cube in the complement of C which has anchor with entries all 1 ’s except for at mostone entry 0 and the anchor projects to a vertex of the ( k + 1) -cube.
Hence the projection p of C is notonto and d ≤ k .