Unlabelled Sample Compression Schemes for Intersection-Closed Classes and Extremal Classes
Abstract
The sample compressibility of concept classes plays an important role in learning theory, as a sufﬁcient condition for PAC learnability, and more recently as an avenue for robust generalisation in adaptive data analysis.
Whether compression schemes of size O ( d ) must necessarily exist for all classes of VC dimension d is unknown, but conjectured to be true by Warmuth.
Recently Chalopin, Chepoi, Moran, and
Warmuth (2018) gave a beautiful unlabelled sample compression scheme of size
VC dimension for all maximum classes: classes that meet the Sauer-Shelah-Perles
Lemma with equality.
They also offered a counterexample to compression schemes based on a promising approach known as corner peeling.
In this paper we simplify and extend their proof technique to deal with so-called extremal classes of VC dimension d which contain maximum classes of VC dimension d − 1 .
A criterion is given which would imply that all extremal classes admit unlabelled compression schemes of size d .
We also prove that all intersection-closed classes with VC dimension d admit unlabelled compression schemes of size at most 11 d .
Introduction
Littlestone and Warmuth (1986) initiated the study of sample compression schemes as an alternative characterisation of probably approximately correct (PAC) learnability, to ﬁnite Vapnik-Chervonenkis (VC) dimension.
Informally a sample compression scheme for a concept class comprises a compressor and reconstructor: while the compressor represents samples labelled by the class as a labelled or unlabelled subsample, the reconstructor must be able to recover a concept consistent with the original labelled sample given only this representation.
Littlestone and Warmuth (1986) demonstrated a remarkably simple proof of PAC learnability for any concept classes with labelled compression schemes provided compressed representations have bounded size, constant in the original sample size.
They conjectured that bounded labelled compression schemes should also be necessary for
PAC learnability—completing the characterisation—speciﬁcally that all VC- d concept classes (those having VC dimension d ) have labelled compression schemes of size d .
This conjecture, for size
O ( d ) , remains open now more than 35 years later, and stands as one of the oldest open problems in learning theory.
In addition to proving PAC generalisation bounds (von Luxburg, Bousquet, and
Schölkopf, 2004; Langford, 2005), compression has found important connections to generalisation in the adaptive data analysis setting (Cummings, Ligett, Nissim, Roth, and Wu, 2016).
While Littlestone and Warmuth (1986) conjectured the existence of labelled schemes of size d for all VC- d classes, Warmuth (2003) later relaxed this to O ( d ) , while Kuzmin and Warmuth (2007) conjectured size O ( d ) unlabelled compression schemes.
(In unlabelled schemes, the compressor’s representations omit labels; such schemes of size k imply labelled schemes of size k .)
Indeed VC- d classes are known that cannot be unlabelled compressed to size d (Pálvölgyi and Tardos, 2020).
Notable progress towards resolving the sample compression conjecture has involved compressing families of concept classes.
Shortly after the original conjecture Floyd (1989) proved that all maximum concept classes C (those meeting the Sauer-Shelah-Perles Lemma with equality) can be label compressed to size VC ( C ) (Floyd and Warmuth, 1995).
Chalopin et al.
(2018) recently proved the same bound but with unlabelled compression, while also providing an elegant counter example to the same result for so-called corner peeling schemes.
This shows that the constructions of corner peeling schemes by Kuzmin and Warmuth (2007) and Rubinstein and Rubinstein (2012) are incorrect.
Moran and Yehudayoff (2016) proved that all VC- d classes can be label compressed to size exponential in d .
Recently Pálvölgyi and Tardos (2020) offered examples of VC classes which cannot admit unlabelled compression schemes of size d .
Moran and Warmuth (2016) discover labelled compression schemes for extremal classes, which contain cubes for any coordinates shattered by the class, and generalise maximum classes.
It is unknown if all extremal classes admit unlabelled compression schemes of size O ( d ) .
In this paper, we give a criterion on extremal classes to admit unlabelled compression schemes of size VC dimension
Note that in Chalopin et al.
(2018), a beautiful counterexample is given to the possibility of all maximum classes having corner peeling unlabelled compression schemes of size VC dimension.
This shows that the constructions of corner peeling schemes in Kuzmin and Warmuth (2007) and
Rubinstein and Rubinstein (2012) are incorrect.
A promising approach to the general conjecture—and a strong motivation for compressing special families of concept classes–is via embedding, as a compression scheme restricts to any subclass.
So, there is interest in embedding general VC-classes into larger classes with good compression properties.
Rubinstein, Rubinstein, and Bartlett (2015) proved that VC- d classes cannot always be embedded in maximum classes of VC dimension less than 2 d − 1 .
Embedding VC- d classes into maximum classes of VC-dimension O ( d ) could still achieve compression to size O ( d ) in general.
Intersection-closed classes—those closed under coordinate-wise multiplication—are another promising focus for compression, as any class can be embedded into an intersection-closed class.
In this paper, we show that any intersection-closed class can be embedded into an extremal intersectionclosed class with the VC dimension increasing from d to 11 d .
These extremal intersection-closed classes have VC-sized unlabelled compression schemes by Chalopin et al.
As such, we establish for the ﬁrst time that intersection-closed classes can be unlabelled compressed to size linear in their VC dimension.
We offer a short proof of the result by Chalopin et al.
(2018) that maximum classes have VC dimension
d -size unlabelled compression schemes and extend this to extremal classes which contain maximum classes of VC dimension d − 1 .
Finally we give an efﬁcient method to compute the VC dimension of the intersection closure of a class.
In particular, this could be useful in searching for collections of classes for which the VC dimension does not increase by much when passing to the intersection closure and then constructing the above unlabeled compression scheme.
Preliminaries
Given n ∈ N we call a subset of the binary n -cube C ⊆ { 0 , 1 } n , a concept class with elements v ∈ C concepts .
These come from restrictions of classiﬁer families, on samples of n instances.
1 We equate each instance in [ n ] = { 1 , . . . , n } with coordinates , axes or colours of the n -cube.
An important combinatorial parameter is the VC dimension, which is used to bound concept class cardinality.
Deﬁnition 2.1 (Vapnik and Chervonenkis, 1971) .
The VC dimension of concept class C ⊆ { 0 , 1 } n is deﬁned as VC ( C ) =
max (cid:8) | I | : I ⊆
[ n ] , proj I ( C )
= { 0 , 1 } | I | (cid:9) , where proj I ( C ) = (cid:8) ( v i ) i ∈ I : v ∈ C (cid:9) is the set of coordinate projections of the concepts of C on coordinates I ⊆
[ n ] .
C is said to shatter I when the projection proj I ( C ) = { 0 , 1 } | I | is onto the | I | -cube.
Lemma 2.2 (Sauer-Shelah-Perles Lemma, Vapnik and Chervonenkis, 1971; Shelah, 1972; Sauer, 1972) .
The cardinality of any concept class C ⊆ { 0 , 1 } n is bounded by |C| ≤
[Equation]
Concept classes meeting equality in Sauer’s Lemma are called maximum (Welzl, 1987).
A cubical– complex structure can be put on concept classes C ⊆ { 0 , 1 } n .
To begin, each v ∈ C is a vertex, and pairs of Hamming-1 separated vertices are edges, of the one-inclusion graph of C (Haussler, Littlestone, and Warmuth, 1994).
Each higher-order cube is all points varying over a set of coordinates, the cube’s colours ; a cube’s common values on non-colour coordinates make up its anchor .
(Equivalently d -cubes are pre-images of points under a projection from the binary ( m + d ) -cube to the binary m -cube.)
Any d -maximum class has a special structure as a complete collection of d -cubes:
Deﬁnition 2.3 (Rubinstein, Bartlett, and Rubinstein, 2009) .
A set of d -cubes S of cardinality (cid:0) nd (cid:1) is called d -complete if for all I ⊆
[ n ] of cardinality d , there exists c I ∈ S with proj
I ( c I ) = { 0 , 1 } d .
Another property of a concept class C is of being shortest-path closed —also known as isometric (Chalopin et al., 2018)—where every pair of points in C has a shortest path contained in C (Kuzmin and Warmuth, 2007; Rubinstein and Rubinstein, 2012).
We will be especially interested in extremal classes—also known as ample classes (Chalopin et al., 2018).
Deﬁnition 2.4.
A class C ⊆ { 0 , 1 } n is extremal if whenever C shatters some I ⊆
[ n ] , then there is a subcube c ⊆ C which maps bijectively to the | I | -cube via proj I ( · ) .
That is, proj I ( c ) = { 0 , 1 } | I | .
All maximum classes are extremal and extremal classes are shortest-path closed (Moran and Warmuth, 2016), while the complements of extremal classes are themselves extremal (Bandelt, Chepoi, Dress, and Koolen, 2006; Bollobás and Radcliffe, 1995; Lawrence, 1983).
As an example, a VC one dimensional maximum (respectively extremal) class is a tree with one (respectively at most one) edge for each colour.
An extremal counter-part to Sauer’s Lemma was independently discovered by Pajor (1985), Bollobás and Radcliffe (1995), Dress (1997), and Anstee, Rónyai, and Sali (2002).
Lemma 2.5 (Sandwich Lemma) .
For any extremal class C , its cardinality is equal to both the number of shatterings of C and the number of cube types (an equivalence class of cubes which all have the same set of colours) in C .
Extremal and maximum classes have an elegant recursive structure that is best studied by inductive arguments.
One such tool is the projection operator proj I ( C ) .
The reduction C I ⊆
proj
I ( C ) is the subset of points with non-unique pre-images under the projection, while the tail is the subset of proj
I ( C ) with unique pre-image under the projection (Welzl, 1987; Kuzmin and Warmuth, 2007).
Unlabelled compression schemes are deﬁned by representation mappings .
A party known as the compressor uses a representation mapping to summarise a C -labelled sample of any size with a small unlabelled subset of instances.
A second party, the reconstructor , can invert this representation thanks to condition 1, to a concept that is consistent with the original labelled sample, thanks to condition 2, (Chalopin et al., 2018).
Deﬁnition 2.6 (Kuzmin and Warmuth, 2007) .
Mapping r is a representation mapping of size k < n , also known as an unlabelled compression scheme of size k , of class C ⊆ { 0 , 1 } n if it satisﬁes:
r is an injection between C and subsets of [ n ] of size at most k ; and 2.
(non-clashing) proj r ( v ) ∪ r ( v (cid:48) )
( v ) (cid:54) =
proj r ( v ) ∪ r ( v (cid:48) )
( v (cid:48) ) for all v (cid:54) = v (cid:48) ∈ C .
The intersection closure operator
Examples of intersection-closed classes include axis-aligned hyperrectangles in R m , monomials, which are unions of orthogonal subcubes, and any union of orthogonal intersection-closed subclasses (Moran and Warmuth, 2016).
Closed-below classes (Rubinstein et al., 2009) are intersection closed and extremal.
All extremal classes of VC dimension 1 are intersection closed, so long as the origin is located in the class (Moran and Warmuth, 2016).
Note the property of being intersection closed depends on the choice of origin.
We will use the notation v (cid:12) w to denote the intersection of the vertices v, w .
This is just the operation of bitwise Boolean multiplication v (cid:12) w = ( v 1 w 1 , . .
, v n w n ) .
We can likewise let u 1 (cid:12) . . .
(cid:12) u k
=
( (cid:81)
ki =1 u i 1 , . . .
, (cid:81)
ki =1 u in ) denote the (iterated) intersection of vertices u 1 , . . .
, u k .
Deﬁnition 3.1.
A class C in the binary n -cube is called intersection closed if v (cid:12) w ∈ C , whenever v, w ∈ C .
Let C =
(cid:12) ( C ) =
{ u 1 (cid:12) . . .
(cid:12) u k ) : k ≤ n, { u 1 , . . .
, u k } ⊆ C} denote the intersection closure of any class C obtained via intersections of all ﬁnite subsets of C ’s concepts.
Lemma 3.2.
For any concept class C ⊂ { 0 , 1 } n , the intersection closure C is the unique smallest intersection-closed class containing C , and it is contained in any intersection-closed class containing
C .
Moreover for any J ⊆ [ n ] , we must have proj J ( C ) =
proj J (cid:0) C (cid:1) .
Next is a useful result on computing the VC dimension of the intersection closure of a class.
The key idea is that to verify proj J (cid:0) C (cid:1) shatters, it is necessary and sufﬁcient to show C has | J | + 1 vertices which project to the vertices with at most one coordinate 0 .
Deﬁnition 3.3.
Suppose C is a class in the binary n -cube with VC dimension d .
We will say that
C satisﬁes the k -close cube condition if there is a vertex v ∈ { 0 , 1 } n and a complete union of ( n − k − 1) -cubes in the complement of C so that each cube is distance at most one from v .
For any k (cid:48) > k , C then also satisﬁes the k (cid:48) -close
cube condition.
Theorem 3.4.
Suppose C is a class in the binary n -cube.
The smallest VC dimension d for the intersection closure of C over all choices of origin, is the same as the smallest k for which C satisﬁes the k -close cube condition.
Proof.
Suppose ﬁrst that d achieves the smallest VC dimension of the intersection closure of C over all choices of origin. Let C denote the corresponding intersection closure of C . Then any projection p of C to a ( d + 1) -cube is not onto. Assume p ( C ) contained the vertex w = (1 , 1 , . .
. , 1) and all vertices w i
=
. , 1) with 0 in the i th position and all other entries 1 .
Then since p ( C ) is intersection closed, the image is the whole ( d + 1) -cube contrary to assumption.
The reason is that taking intersections of combinations of the vertices w i gives all the vertices in the ( d +1) -cube.
So at least one of the vertices w, w 1 , . . .
, w d +1 must not be in p ( C ) .
But then there is an ( n − d − 1) -cube with anchor of this form in the complement of C by taking the inverse image under p of this vertex.
By choosing v as the vertex with all entries 1 in the n -cube, we have shown that C satisﬁes the d -close cube condition.
For clearly each of our ( n − d − 1) -cubes in the complement of
C has distance at most one from v .
So this establishes that d ≥ k , where k is the smallest value so that C satisﬁes the k -close cube condition.
Conversely suppose that C has a collection of ( n − k − 1) -cubes in its complement, validating that
C satisﬁes the k -close cube condition.
We claim that d ≤ k (cid:48) , where d is the VC dimension of the intersection closure of C for a suitable choice of origin.
This shows that the smallest possible VC dimension d of the intersection closure of C over all choices of origin is not larger than the smallest possible k where C satisﬁes the k -close cube condition.
For suppose that any projection p of C to the ( k +1) -cube is selected.
By Lemma 3.2, p ( C ) =
p ( C ) .
In fact the intersection operator commutes with the projection operator, i.e. , p ( I ( u, w )) =
I ( p ( u ) , p ( w ))
so the claim follows by Lemma 3.2.
But now by assumption, if we choose the origin so that v is the vertex with all entries 1 , then there is an ( n − k − 1) -cube in the complement of C which has anchor with entries all 1 ’s except for at most one entry 0 and the anchor projects to a vertex of the ( k + 1) -cube.
Hence the projection p of C is not onto and d ≤ k .
We end with a useful property of the intersection closure operator.
We will abuse notation by saying a class A is shortest path closed inside a class B , if A ⊂ B and any two vertices of A are connected by a shortest path inside B .
The main idea in the following theorem is that if C is shortest-path closed, then X = { v : v = u (cid:12) w, u, w ∈ C} is shortest path closed inside C .
Theorem 3.5.
If C is shortest-path closed then its intersection closure C is also shortest-path closed.
Proof.
We ﬁrst observe the easy fact that if λ is a shortest path in C consisting of vertices v 1 , v 2 , . .
. , v k and u ∈ C then u (cid:12) λ := { u (cid:12) v i : i ∈ [ k ] } is a shortest path in C possibly with repetitions.
For an edge between successive vertices v i , v i +1 is transformed into u (cid:12) v i , u (cid:12) v i + which is either an edge of the same colour or a single vertex.
Our main claim that if C is shortest-path closed then
X
=
{ v : v = u (cid:12) w, u, w ∈ C} is shortest path closed inside C .
Once this claim is established, the proof follows easily by repeating this step.
For we can apply the claim to the class X to prove that the class { v : v = u (cid:12) w, u, w ∈ X} consisting of all elements of C together with intersections of up to 4 elements of C is shortest-path closed.
inside
X = C .
Repeating at most log 2 n times, C is obtained and is shortest-path closed.
(Note that since v = v (cid:12) v , elements of C can be written as intersections of pairs of elements of C ).
The ﬁrst step in proving the main claim is to observe that there is a shortest path in X connecting any element u ∈ C to any element u (cid:12) v ∈ X .
To construct this path, start with a shortest path λ between u, v in C and then form u (cid:12) λ .
As above this is a shortest path, possibly with repetitions, between u, u (cid:12) v in X .
We conclude that given two vertices u, v ∈ C , there is a shortest path in X from u to v passing through u (cid:12) v , by concatenating shortest paths in X from u to u (cid:12) v and u (cid:12) v to v .
To prove the main claim, choose elements v, v (cid:48)
∈ X .
We want to ﬁnd a shortest path in C connecting v, v (cid:48) .
Let v = u (cid:12) w, v (cid:48)
=
u (cid:48) (cid:12) w (cid:48) where u, w, u (cid:48) , w (cid:48) ∈ C .
By the ﬁrst step, there are shortest paths λ, λ (cid:48) in X connecting u, u (cid:48) and w, w (cid:48) and passing through u (cid:12) u (cid:48) , w (cid:12) w (cid:48) respectively.
Next, observe that v (cid:12) v (cid:48) =
u (cid:12) w (cid:12) u (cid:48) (cid:12) w (cid:48) =
u (cid:12) u (cid:48) (cid:12) w (cid:12) w (cid:48) .
Split λ, λ (cid:48) into λ i , λ (cid:48)
i for i = 1 ,so that λ 1 , λ (cid:48) 1 join u, w and u (cid:12) u (cid:48) , w (cid:12) w (cid:48) respectively and λ 2 , λ (cid:48) 2 join u (cid:12) u (cid:48) , w (cid:12) w (cid:48) and u (cid:48) , w (cid:48) respectively.
To establish the main claim, it clearly sufﬁces to construct shortest paths in C from v to v (cid:12) v (cid:48) and from v (cid:12) v (cid:48) to v (cid:48) .
By symmetry, we will focus on the ﬁrst of these.
We will prove a slightly stronger claim.
Namely assume λ 1 consists of ordered vertices
u =
z 1 , z 2 , . .
z k = u
(cid:12) u (cid:48) and λ (cid:48) 1 has ordered vertices w = y 1 , y 2 , . . .
, y r = w (cid:12) w (cid:48) .
We will show that a shortest path in X connecting v, v (cid:12) v (cid:48) can be chosen to have all vertices of the form z i (cid:12) y j .
To complete the proof, consider the rectangular grid of intersections
z i (cid:12) y j , for 1 ≤ i ≤ k and 1 ≤ j ≤ r with k, r ≥ 1 .
Each small square of this grid is either a single vertex, one edge or a 2 -cube in X by the previous paragraph.
We claim that there is a shortest path between the diagonally opposite entries v =
z 1 (cid:12) y 1 , v (cid:12) v (cid:48)
=
z k (cid:12)
y r using vertices in this grid.
It sufﬁces to use the boundary of the grid to do this.
So for example, consider the ﬁrst path
z i (cid:12) y 1 for 1 ≤ i ≤ k , followed by a second path z k (cid:12) y j for 1 ≤ j ≤ r .
By the ﬁrst observation in this proof, the ﬁrst path is shortest with possible repetitions, as is the second path.
To complete the argument, we need to show that the concatenation of these two paths is also shortest with possible repetitions.
Notice that the path λ
1 =
z 1 , z 2 , . .
z k runs between ends z 1 , z k where z k is between the origin of the binary n -cube and z 1 .
So the same follows for the path with possible repetitions
z i (cid:12) y 1 for 1 ≤ i ≤ k .
The second path z k (cid:12) y j for 1 ≤ j ≤ r is of the same form and hence the concatenation of these two paths is also shortest with repetitions and the theorem proof is complete.
Extremal intersection-closed classes
In this section, we start with a useful way of ﬁnding extremal intersection-closed classes, namely by checking they are shortest-path and intersection-closed.
We use this to establish the main result about intersection-closed classes, namely that they can be embedded into extremal intersection closed classes, with the VC dimension d increasing to at most 11 d .
We then use a result of Chalopin et al.
(2018) to conclude this gives unlabelled compression schemes of size 11 d for an intersection-closed class of VC dimension d .
Theorem 4.1.
Suppose that C is shortest-path and intersection closed.
Then C is extremal.
Proof.
Suppose a projection p : { 0 , 1 } n → { 0 , 1 } k shatters C .
We will prove there is a k -subcube c of C which witnesses the shattering.
Consider all the vertices of C which map to the vertex w = (1 , 1 , . .
, 1) in the k -cube.
Choose a vertex v in this set which has smallest distance to the origin.
(Since C is shortest-path closed and intersection closed, the origin is in C and distance in C is the same as distance in the n -cube.)
Our claim is that there are vertices v i ∈ C , i ∈
[ k ] , so that the distance from v to each v i is one and p ( v i ) =
, 1) =
w i the vertex in the k -cube with all coordinates 1 except a single 0 at the i th coordinate.
Once this claim is established, it follows that since C is intersection closed, taking the intersections of subcollections of the vertices { v i : i ∈ [ k ] } generates the required k -subcube c of C .
Note c witnesses the shattering, since projection commutes with intersection and the vertices w, w i generate c under the operation of taking intersections.
To prove the claim, choose any vertex u ∈ C
so that p ( u ) =
w
i .
Since C is intersection closed, u (cid:48)
= v (cid:12) u ∈ C .
Then p ( u (cid:48) )
=
p ( v (cid:12) u ))
=
p ( v ) (cid:12) p ( u ) = w (cid:12) w i = w i .
Note that u (cid:48) is a vertex between v and the origin, i.e. , for any coordinate where u (cid:48) has value 1 then v must also.
Since C is shortest-path closed, a shortest path λ exists in C from v to u (cid:48) .
Moreover λ has vertices in order with distances decreasing to the origin, since u (cid:48) is between v and the origin.
Let v (cid:48) be the ﬁrst vertex on λ after v .
Since p ( λ ) is a shortest path in the k -cube from w to w i , it follows that p ( v (cid:48) ) is either w or w i .
If the former, this contradicts our choice of v as a closest vertex in C to the origin which projects to w .
If the latter then we have found our vertex v i with distanceto v and which projects to w i .
So this completes the proof of the claim and the result.
Remark 4.2 .
Chalopin et al.
(2018) discusses the signiﬁcance of properties of extremal intersectionclosed classes, also known as conditional antimatroids .
Remark 4.3 .
There is a partial order on the binary n -cube given by w ≤ v when w is between v and the origin or equivalently there is a shortest path from v to the origin passing through w .
Choose an arbitrary ordering on [ n ] .
We will denote by λ ( v, w ) the shortest path between v, w with vertices v = v 1 , v 2 , . . .
v k = w where v i .v
i +1 differ in the ﬁrst coordinate x j in the ordering of indices j amongst all coordinates where v i , w differ.
In the next result, we enlarge an intersection closed class C to become intersection-closed and shortestpath closed by adding all the vertices in the shortest paths λ ( v, w ) between all pairs v, w ∈ C .
Theorem 4.4.
Any intersection-closed class C embeds in a shortest-path closed and intersectionclosed class C ∗ given by Algorithm 1 so that d ∗ ≤ 11 d where d, d ∗ are the VC dimensions of C , C ∗ respectively.
To construct C ∗ , for every pair of vertices v, w ∈ C with w < v , add all the vertices in λ ( v, w ) to C .
Lemma 4.5.
If C is intersection closed then C ∗ as constructed by Theorem 4.4 is intersection closed.
Proof.
By deﬁnition, for any vertices u, u (cid:48)
∈ C ∗ , there are vertices v, w, v (cid:48) , w (cid:48) ∈ C
so that u ∈ λ ( v, w ) , u (cid:48) ∈ λ ( v (cid:48) , w (cid:48) ) .
Moreover, w ≤ v, w (cid:48) ≤ v (cid:48) .
Since C is intersection closed, both v (cid:12) v (cid:48) and w (cid:12) w (cid:48) are in C .
We claim that u (cid:12) u (cid:48) is on the path λ ( v (cid:12) v (cid:48) , w (cid:12) w (cid:48) ) —note that as part of this claim, we require that w (cid:12) w (cid:48) ≤ v (cid:12) v (cid:48) .
We begin with the latter claim.
This is easy— w (cid:12) w (cid:48) has entry 1 at precisely the coordinates where both w, w (cid:48) have entry 1 .
But since w ≤ v, w (cid:48) ≤ v (cid:48) , v, v (cid:48) has entry 1 at all the coordinates where w, w (cid:48) respectively has entry 1 .
So for any coordinate where w (cid:12) w (cid:48) has entry 1 , it follows immediately that both v, v (cid:48) have entry 1 and so does v (cid:12) v (cid:48) .
Next, the set of coordinates where v has entry 1 and u has entry 0 forms an initial segment of the coordinates where v has entries 1 , using the ordering of the coordinates, since u ∈ λ ( v, w ) .
A similar analysis applies to the set of coordinates of u (cid:48) which are 0 but the entries for v (cid:48) are 1 .
To complete the proof that C ∗ is intersection closed, we show that the set of coordinates where the entries of v (cid:12) v (cid:48) are 1 and the entries of u (cid:12) u (cid:48) are 0 , forms an initial segment of the ordering of the coordinates where the entries of v (cid:12) v (cid:48) are 1 .
It will then follow
that u (cid:12) u (cid:48) ∈
λ ( v (cid:12) v (cid:48) , w (cid:12) w (cid:48) ) .
Now the set of coordinates where v (cid:12) v (cid:48) has entries 1 are exactly those coordinates where both v, v (cid:48) have entries 1 .
Moreover u (cid:12) u (cid:48) has entry 0 at a coordinate if either u or u (cid:48) has entry 0 .
We complete the proof using an argument by contradiction.
Suppose that there is a coordinate x i such that u (cid:12) u (cid:48) has entry 1 but this occurs before a coordinate x j where u (cid:12) u (cid:48) has entry 0 and moreover v (cid:12) v (cid:48) has entry 1 .
Then at x j , either u or u (cid:48) has entry 0 .
Without loss of generality assume the former.
Since u (cid:12) u (cid:48) has entry 1 at x i it follows that u has entry 1 at x i .
Note also that v has entry 1 at both x i , x j .
But this is a contradiction since the coordinates where v has entry 1 but u has entry 0 should form an initial segment of the set of coordinates where v has entry 1 .
But we have found j > i where u has entry 1 at x i and 0 at x j .
This establishes that C ∗ is intersection closed.
Lemma 4.6.
If C is intersection closed then C ∗ as constructed by Theorem 4.4 is shortest-path closed.
Proof.
By Lemma 4.5, it sufﬁces to show that if v, w ∈ C ∗ then there are shortest paths from v, w to v (cid:12) w , since putting these together will give a shortest path from v to w .
Next, it sufﬁces to show there is a shortest path in C ∗ between any pair of vertices v, u in C ∗ , where u ≤ v , for we can then apply this to the pairs v, v (cid:12) w and w, v (cid:12) w .
We claim that there is a vertex v 1 ∈ C so that u ≤ v 1 , v ≤ v 1 and there are shortest paths in C ∗ joining both v 1 , u and v 1 , v .
Firstly, since v ∈ C ∗ , there are vertices v 1 , v 2 ∈ C so that v is in λ ( v 1 , v 2 ) .
Hence v ≤ v 1 and there is a shortest path in C ∗ from v 1 to v , namely a segment of λ ( v 1 , v 2 ) .
Next, since u ≤ v , then u ≤ v 1 .
Moreover u lies on a path λ
( u 1 , u 2 ) where u 1 , u 2 ∈ C , since u ∈ C ∗ .
Then v 1 (cid:12) u 1 ∈ C and v 1 (cid:12) u 1 ≤ u 1 .
Since u ≤ v 1 , u ≤ u 1 , it follows that u ≤ v 1 (cid:12) u 1 .
Notice that u must lie on λ ( v 1 (cid:12) u 1 , u 2 ) .
To verify this, observe all the coordinates where entries of v 1 (cid:12) u 1 differ from entries of u are also coordinates where u 1 entries differ from u entries.
Consequently, coordinates where entries of u and u 2 differ, are lower in the coordinate ordering than those where entries of v 1 (cid:12) u 1 and u differ.
So this conﬁrms that u ∈ λ ( v 1 (cid:12) u 1 , u 2 ) .
Combining λ ( v 1 , v 1 (cid:12) u 1 ) and λ ( v 1 (cid:12) u 1 , u ) gives a shortest path from v 1 to u in C ∗ .
So this completes the proof of the claim about v 1 .
To recapitulate, we have constructed shortest paths in C from v 1 to both u, v in C ∗ , where v ≤ v 1 , u ≤ v 1 , u ≤ v .
To complete the proof that there is a shortest path from v to u in C , we use induction on the Hamming distance from v 1 to v .
To begin the induction, suppose this distance is 0 .
Then clearly we are done, since there is a shortest path in C ∗ from v to u .
Assume the distance from v 1 to v is k and the result is true for v 1 ∗ , v ∗ , u ∗ as above, where the distance from v 1 ∗ to v ∗ is less than k .
Let v (cid:48) be the ﬁrst vertex in the shortest path from v 1 to v .
Our aim is to show that there is a shortest path in C ∗ from v (cid:48) to u .
By induction, it then follows there is a shortest path in C ∗ from v to u .
The argument is similar to that in Theorem 3.5.
Let v 1 = y 1 , y 2 , . . .
y r
= u be a shortest path in
C ∗ .
We can construct a shortest path v (cid:48)
= v (cid:48) (cid:12) y 1 , v (cid:48) (cid:12) y 2 , . .
, v (cid:48) (cid:12) y r =
u where there is a single repetition.
Namely, since one of the edges joining y j , y j +1 must have the same colour as the edge between v 1 , v (cid:48) , then v (cid:48) (cid:12) y
j
= v (cid:48) (cid:12) y
j +1 .
All other pairs of vertices are distance one apart and since
C ∗ is intersection closed, this shortest path is in C ∗ .
So this completes the proof.
Proof.
(Theorem 4.4)
It remains to prove that the VC dimension of C ∗ is at most 11 d .
Choose any projection p : { 0 , 1 } n → { 0 , 1 } 11 d .
By Sauer’s Lemma, since the VC dimension of p ( C ) is at most d , the cardinality | p ( C ) | is at most F ( d ; 11 d, 12 ) × 2 11 d where F ( k ; n, p ) is the cumulative binomial probability function.
Algorithm 1 constructing C ∗ involves ﬁnding and adjoining shortest paths connecting suitable ordered pairs of points in C .
So |C ∗ | ≤ n × |C| 2 .
Similarly we see that | p ( C ∗ ) | ≤ 11 d × | p ( C ) | 2 , since the process of adjoining shortest paths is preserved under projection.
Hence | p ( C ∗ | ≤ 11 d ×
F ( d ; 11 d, 12 ) 2 × 2 22 d × 12 .
Note that as pairs ( v, w ) are ordered by requiring that w ≤ v , we get a factor of 12 .
We claim that this upper bound is strictly less than 2 11 d .
Then p cannot shatter and the
VC dimension of C ∗ is at most 11 d − 1 .
The key idea is to use Chernoff’s inequality to estimate F ( d ; 11 d, 12 ) .
This is:
[Equation]
where D ( 111 || 12 ) = 111 log 211 + (1 − 111 ) log 2(1 − 111 ) .
Substituting this into the previous upper bound and simplifying, we obtain | p ( C ∗ ) | ≤ 11 d × 2 22 dexp(22 d log 11 − 2 d log 2 − 20 d log 20) .
We can further simplify this to | p ( C ∗ ) | ≤ 11 d × 11 22 d × 10 − 20 d × 12 .
To complete the argument, we need to show the right side is strictly less than 2 11 d .
Hence it sufﬁces to show that
[Equation]
Now if d ≥ 5 then 11 1 d < 1 .
7 and d 1 d < 1 . 4 .
So we can compute the left side is at most 1 .
7 × 1 .
× 121 × 111020 < 1937 whereas the right side is 2048 .
(Here we leave out the factor 2 − 1 d .)
If d = 4 then 112 14 < 1 . 6 and 4 14 < 1 .
So we can compute the left side is as at most 1 . 6 × 1 . 42 × 121 × 111020 = 1810 whereas the right side is 2048 .
So this completes the case when d ≥ 4 .
For the other cases, it is convenient to work directly, as Chernoff’s inequality is not sharp.
For d = 1 , | p ( C ) | ≤ 12 , as it has VC dimension at most 1 in the 11 -cube.
Hence | p ( C ∗ ) | ≤ 11 × 12 2 × 12 = 792 which is less than 2048 .
Next suppose d = 2 .
| p ( C ) | ≤ 254 , as it has VC dimension at most 2 in the 22 -cube.
Hence | p ( C ∗ ) | is bounded above by 22 × 254 2 × 12 = 709 , 676 which is less than 2048 2 = 4 , 194 , 304 .
Finally assume d = 3 .
| p ( C ) | ≤ 6018 , as it has VC dimension at most 3 in the 33 -cube.
Hence | p ( C ∗ ) | is bounded above by 33 × 6018
× 12 = 597 , 569 , 346 which is less than 2048 3 = 8 , 589 , 934592 .
Corollary 4.7.
If C is an intersection-closed class, then C has a (corner peeling) unlabelled compression scheme of size at most 11 d .
Proof.
This follows immediately by combining Theorem 4.4 and (Chalopin et al., 2018, Theorem 4.9).
Unlabelled compression schemes for extremal classes
The aim is to give a condition on extremal classes which implies that they have unlabelled compression schemes.
We then give some situations where this criterion is satisﬁed.
We conjecture that it is always valid.
For a maximum class C of VC dimension d , a reduction is a maximum subclass of VC dimension d − 1 .
So a reduction contains cubes with the same set of colours as any non-maximal cube of C .
It will turn out this is the key property we require of extremal classes to have unlabelled compression schemes of size d , namely they have proper extremal subclasses which have the same properties as reductions of maximum classes.
5.1 High-level strategy
The idea is inspired by Chalopin et al.
Start with a pair ( C , D ) consisting of an extremal class and a proper extremal subclass.
The condition is that for every cube in C which is not maximal, there is a cube with the same colours in D .
We call this the cubical colour condition of the pair C , D and abbreviate it by ccc .
We will construct a bijection φ between the set S of maximal cubes c of C not in D and C \ D , so that v = φ ( c ) ∈ c .
Note the cardinalities of these two sets are the same, by the Sandwich Lemma 2. applied to the two extremal classes C , D .
Then φ − 1 : C \D → S gives a representation map for C \D , which also separates the vertices of C \ D from the vertices of D .
Therefore, any representation of D can be combined with the representation of C \ D to give a representation of C .
Suppose we can ﬁnd a sequence of proper pairs of extremal subclasses ( C , D ) , ( D , E ) , . . .
, ( X , Y ) , all satisfying ccc , ending with VC ( Y ) ≤ 2 .
Then we can start with a representation map of the VC-  class Y (Chalopin et al., 2018) and extend to a representation of X , then extending the representation all the way to C using this strategy.
5.2 Detailed construction
We construct φ : C \ D → S and show it is a bijection.
Firstly, denote the extremal class which is the complement of D by D ∗ .
Then every vertex of C \ D is in at least one maximal cube of D ∗ .
Deﬁne two cubes c, c ∗ in { 0 , 1 } n to be complementary if the dimensions d, d ∗ of c, c ∗ satisfy d + d ∗ = n and c ∩ c ∗ is a single vertex.
Recall that S is the set of maximal cubes c of C not in D .
Denote the set of maximal cubes of D ∗ which intersect C by S ∗ .
Lemma 5.1.
Assume ( C , D ) is a proper pair of extremal classes satisfying ccc .
If c ∈ S , then there is a complementary maximal cube c ∗ of D ∗ .
Conversely, if c ∗ is a maximal cube of S ∗ , then there is a complementary cube c ∈ S .
A maximal cube c ∗ of S ∗ , cannot have two different complementary cubes in S .
Given Lemma 5.1, we will call v = c
∩ c ∗ the special vertex of c .
We now note properties of v , assuming Lemma 5.1.
Corollary 5.2.
Assume that ( C , D ) is a proper pair of extremal classes satisfying ccc .
Let v =
c ∩ c ∗ where c ∈ S and c
∗ ∈ S ∗ are complementary maximal cubes.
Then
• The set of vertices v = c ∩ c ∗ , for all pairs of complementary maximal cubes c ∈ S and c ∗ ∈ S ∗ , is C \ D .
• The mappings between c, c ∗ , v are bijections between S , S ∗ and C \ D .
Proof.
Lemma 5.1 gives a bijection between S and S ∗ , by the mapping from cubes c to complementary cubes c ∗ .
Moreover by the Sandwich Lemma applied to extremal classes C , D , the cardinalities of
C \ D , S ∗ are the same.
Since the map from maximal cubes in S to special vertices is an injection by
Lemma 5.1, it must be a bijection onto C \ D .
Proof of Lemma 5.1.
Let p : { 0 , 1 } n → { 0 , 1 } d be the projection to the colours of c ∈ S .
Recall ccc for the pair C , D says that for any cube of C which is not maximal, there is a cube of D with the same set of colours.
So p ( D ) = { 0 , 1 } d \ { w } for some vertex w .
For the only other case is that p ( D ) = { 0 , 1 } d .
But since D is extremal, this would imply D contained a cube with the same colours as c .
This contradicts the assumption that c is maximal in C , since for an extremal class, maximal cubes are unique for their sets of colours.
Deﬁne v, c ∗ by v = p − 1 ( w ) ∩ c, c ∗ =
p − 1 ( w ) respectively.
Clearly c ∩ c ∗ = v .
It remains to show that c ∗ is maximal in D ∗ and c ∗ cannot intersect some other cube of S in a single vertex.
Suppose that c ∗ was properly contained in a larger cube ˜ c in D ∗ .
Then p (˜ c ) would properly contain { w } .
Moreover ˜ c = p − 1 p (˜ c ) .
But this contradicts our observation above that p ( D ) = { 0 , 1 | d \ { w } .
So this completes the argument that c ∗ is maximal.
Next, assume that c ∗ is complementary to a second maximal cube ˆ c ∈ S .
Consider the projection ˆ p of { 0 , 1 } n to { 0 , 1 } ˆ d given by the ˆ d colours of ˆ c .
As above, ˆ p ( D ) consists of { 0 , 1 } ˆ d with a single vertex ˆ w removed.
Moreover, ˆ p − 1 ( ˆ w ) is a maximal cube c (cid:48) of D ∗ .
Now as c ∗ ∩ ˆ c = u ∗ is a single vertex, ˆ p ( c ∗ )
= ˆ p ( u ∗ ) = u .
As the cubes c ∗ , ˆ c are complementary, c ∗ =
ˆ p − 1 ( u ) .
But now we see that the colours of the anchor of c ∗ are both the colours of c and of ˆ c .
This gives a contradiction, since two maximal cubes of C cannot have the same set of colours.
This completes the proof of Lemma 5.1.
Finally we show that the map from a special vertex v to the colours of the corresponding maximal cube c of S gives a representation mapping of C \ D which also separates C \ D from D .
The second property follows immediately.
For recall that if p is the projection given by the d colours of c , then p ( D ) = { 0 , 1 } d \ { p ( v ) } .
So the colours of c clearly separate v from all the vertices of D .
By this we mean that given any vertex w ∈ D , v, w differ on at least one coordinate from the colours of c .
Next suppose two vertices v, v (cid:48) have the same values at all the coordinates corresponding to the colours of their corresponding cubes c, c (cid:48)
∈ S of dimensions d, d (cid:48) respectively.
Project { 0 , 1 } n to the union of the colours of c, c (cid:48) by a mapping p (cid:48)(cid:48) .
The image of p (cid:48)(cid:48) is a cube { 0 , 1 } d (cid:48)(cid:48) of dimension d (cid:48)(cid:48) ≤ d + d (cid:48) .
In this cube, C , D map to extremal classes p (cid:48)(cid:48)
( C ) =
C (cid:48)(cid:48) , p (cid:48)(cid:48)
( D ) =
D (cid:48)(cid:48) .
We want to check that the pair of extremal classes ( C (cid:48)(cid:48) , D (cid:48)(cid:48) ) satisfy ccc .
But this follows easily.
Choose any non maximal cube ˜ c ∈ C (cid:48)(cid:48) .
As C is extremal, there is a cube c 0 in C which projects one-to-one to ˜ c and therefore has the same colours as ˜ c .
Since c 0 is not maximal, there is a cube c 1 in D with the same colours as c 0 .
Then p (cid:48)(cid:48) ( c 1 ) is the required cube in D (cid:48)(cid:48) with the same colours as ˜ c as required.
Finally, c, c (cid:48) project one-to-one by p (cid:48)(cid:48) to cubes with colours spanning all of { 0 , 1 } d (cid:48)(cid:48) .
We claim that p (cid:48)(cid:48) ( v ) (cid:54) =
p (cid:48)(cid:48)
( v (cid:48) ) .
Since p (cid:48)(cid:48) ( v ) , p (cid:48)(cid:48) ( v (cid:48) ) are the special points for p (cid:48)(cid:48)
( c ) , p (cid:48)(cid:48)
( c (cid:48) ) in { 0 , 1 } d (cid:48) (cid:48) , they cannot coincide, by Lemma 5.1.
But then p (cid:48)(cid:48) ( v ) , p (cid:48)(cid:48) ( v (cid:48) ) must have at least one different value of their coordinates at the colours of c, c (cid:48) and hence the same is true for v, v (cid:48) .
So this gives a representation for C \ D , as we have veriﬁed the no-clashing condition.
Applications
We apply the construction in Section 5 to two special collections of extremal classes.
Theorem 6.1.
Suppose that C is an extremal class of VC dimension d containing a maximum class D of VC dimension d − 1 .
Then C has a d -size unlabelled compression scheme.
Proof.
This follows immediately by the construction in Section 5.
We only have to check that there is a sequence of pairs of extremal classes starting at C , D , all satisfying ccc .
But this is easy since any maximum class of VC dimension i is a union of a complete collection of maximal cubes of dimension i .
Moreover any maximum class of VC dimension i has reductions which are products of maximum classes of VC dimension i − 1 with { 0 , 1 } .
Theorem 6.2.
Suppose that C is an extremal class of VC dimension 2 .
Then C contains a proper extremal class D so that for every edge of C , there is an edge of D of the same colour.
Hence C has a compression scheme of size 2 .
Proof.
Rubinstein and Rubinstein (2008) develops the idea of splitting classes along a reduction (associated to a coordinate projection).
Choose a reduction C 1 of C which splits off a component E with a smallest number of vertices, amongst all choices of reductions and complementary components.
Let D =
C \ E .
We claim that D is extremal and contains edges of each colour in C .
The reason is that for any other reduction C 2 of C , if C 2 ∩E (cid:54) = ∅ , then C 2 ∩C 1 (cid:54) = ∅ .
For if C 2 ∩C 1 = ∅ , there is a smaller component which C 2 splits off from E , which is a contradiction.
But C 2 ∩ C 1 (cid:54) = ∅ implies that D has edges of each colour of C .
Showing D is extremal is straightforward.
Remark 6.3 .
Chalopin et al.
(2018) showed that VC- 2 extremal classes have unlabelled 2 -compression schemes.
Theorem 6.2 gives an alternate proof using our criterion for extremal classes to have such schemes in Section 5.
Conclusion
Chalopin et al.
(2018) constructed unlabelled compression schemes for maximum classes and extremal intersection-closed classes, while Moran and Warmuth (2016) developed labelled compression schemes for extremal classes.
Here we have shown that intersection-closed classes can be embedded into extremal intersection-closed classes with an increase in VC dimension from d to 11 d and so have 11 d -size unlabelled compression schemes.
We simplify and extend the d -size unlabelled compression schemes for maximum classes due to Chalopin et al.
(2018), to apply to extremal classes of VC dimension d which contain maximum classes of VC dimension d − 1 .
We also give a sufﬁcient condition for extremal classes to admit such compression schemes.
Acknowledgments and Disclosure of Funding
We acknowledge support from the Australian Research Council Discovery Project DP220102269.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See throughout for precise theoretical claims, and Section 7 for further directions addressing limitations.
(c) Did you discuss any potential negative societal impacts of your work? [No] The work seeks constructions of sample compression schemes for PAC learnable concept classes. The focus of the work is on learning theory. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2. If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See the assumptions written in each of our results. (b) Did you include complete proofs of all theoretical results? [Yes] All theoretical results are accompanied by complete proofs. If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] No experiments were run for this work.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [N/A] We are not using existing assets or curating/releasing new assets.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL? [N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] This work involved no crowdsourcing or human subject research. (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]