On Batch Teaching with Sample Complexity Bounded by VCD
Abstract
In machine teaching, a concept is represented by (and inferred from) a small number of labeled examples.
Various teaching models in the literature cast the interaction between teacher and learner in a way to obtain a small complexity (in terms of the number of examples required for teaching a concept) while obeying certain constraints that are meant to prevent unfair collusion between teacher and learner.
In recent years, one major research goal has been to show interesting relationships between teaching complexity and the VC-dimension ( VCD ).
So far, the only interesting relationship known from batch teaching settings is an upper bound quadratic in the VCD , on a parameter called recursive teaching dimension.
The only known upper bound on teaching complexity that is linear in VCD was obtained in a model of teaching with sequences rather than batches.
This paper is the first to provide an upper bound of VCD on a batch teaching complexity parameter.
This parameter, called STD min , is introduced here as a model of teaching that intuitively incorporates a notion of “importance” of an example for a concept.
In designing the STD min teaching model, we argue that the standard notion of collusion-freeness from the literature may be inadequate for certain applications; we hence propose three desirable properties of teaching complexity and demonstrate that they are satisfied by STD min .
Introduction
The notion of machine teaching refers to the selection of helpful training examples that aid the learner in identifying a target concept [Zhu et al., 2018].
Such processes can make machine learning feasible or economical in situations when it is difficult or expensive to acquire large amounts of training data.
Human-in-the-loop settings [Wang et al., 2020] or inverse reinforcement learning [Kamalaruban et al., 2019] are potential application areas of machine teaching.
Formal studies of teaching consider various teaching models, each of which has a corresponding complexity parameter referring to the worst-case number of examples required to teach any concept in a given concept class [Goldman and Kearns, 1995, Shinohara and Miyano, 1991, Zilles et al., 2011, Gao et al., 2017b, Kirkpatrick et al., 2019,
Mansouri et al., 2019].
Each model has to adhere to rules that prevent teacher and learner from using unwanted “coding tricks”, such as, for instance, agreeing on an indexing of concepts and an indexing of data points, and then allowing the teacher to simply teach the concept indexed i using the data point with index i .
While no general notion of coding trick is agreed upon in the literature, Goldman and Mathias [1996] provided a notion of collusion-freeness (which we call GM-collusion-freeness for short) that is often used as an underlying constraint in teaching models.
One major line of research addresses the question of how teaching complexity parameters relate to one another and to other parameters of interest in computational learning theory, such as, for instance, the VC-dimension ( VCD , Vapnik and Chervonenkis, 1971), the self-directed learning complexity [Goldman and Sloan, 1994], the optimal mistake bound from online learning [Littlestone, 1988], or the size of sample compression schemes [Littlestone and Warmuth, 1986].
Of particular interest is the question how teaching complexity relates to the complexity of learning from randomly chosen examples, and thus to the VCD [Blumer et al., 1989].
Answers to this question provide insights into structural properties that make a concept class easy or hard to teach, and have implications on the design and analysis of teaching and learning algorithms.
For example, it was established that the
Recursive Teaching Dimension ( RTD , Zilles et al., 2011) is upper-bounded by a function quadratic in VCD [Hu et al., 2017].
While RTD may exceed VCD , it remains open whether it is in O (VCD) .
Connections between RTD and the size of an optimal sample compression scheme were also made [Doliwa et al., 2014, Darnstädt et al., 2016].
More recently, Kirkpatrick et al.
[2019] introduced the No-Clash Teaching Dimension ( NCTD ), which is the optimal complexity of teachers satisfying
GM-collusion-freeness.
Since the RTD model also satisfies GM-collusion-freeness, NCTD is upperbounded by RTD and thus trivially inherits Hu et al.
’s upper bound quadratic in VCD .
It remains unresolved whether or not NCTD ≤ VCD holds in general; no counter-example has been found yet.
The models of RTD and NCTD represent teaching with batches (sets of examples).
Naturally, a smaller complexity can be obtained when teachers can encode concepts in sequences of examples.
So far, the only known teaching complexity upper-bounded by VCD stems from a sequential model [Mansouri et al., 2019].
The main problem we address in this paper is to design a reasonable batch model of teaching that has a complexity upper-bounded by VCD .
While GM-collusion-freeness is often considered a very natural condition on teaching, it has recently been argued that there are application settings in which this condition is not natural [Ferri et al., 2022].
In addition, since the question whether NCTD ≤ VCD is not yet resolved, and NCTD is the optimal complexity of GM-collusion-free teaching, our approach is to consider batch models that do not satisfy GM-collusion-freeness, but instead satisfy minimum postulates of teaching without “coding tricks”.
Our first contribution is to formally define three such postulates and to analyze some teaching models from the literature with respect to these postulates.
We then define a new batch teaching model that is not GM-collusion-free but provably satisfies our three postulates of teaching without coding tricks.
Our main results are that the corresponding complexity parameter is upper-bounded by both RTD and VCD .
This makes our paper the first to provide an upper bound of VCD or O (VCD) on a complexity parameter for batch teaching.
All proofs or parts of proofs omitted from the paper are provided in the appendix.
Preliminaries
Throughout this paper, the symbol X is used to denote any finite domain over which concepts, i.e., subsets of X , are defined.
A concept class over X is then a finite set of concepts over X .
In the literature, any kind of finite or infinite set system can serve as a concept class; examples of finite concept classes are given in the form of tables throughout this paper.
Often the concept classes studied are highly structured classes of Boolean functions, such as, for instance the class of all monomials over a fixed set of variables [Zilles et al., 2011].
Infinite classes of geometric concepts, such as hyperplanes in the Euclidean space, are often studied as concept classes as well; see, e.g., [Gao et al., 2017a,b].
If C is any concept over X and x ∈ X , then C ( x ) is the label of C for x , i.e., C ( x )
= 1 if x ∈ C , while C ( x ) =
if x / ∈ C .
A labeled example for C is a pair ( x, C ( x )) for x ∈ X .
As this paper focuses on batch teaching rather than sequential teaching, teachers and learners are designed to operate with sets of labeled examples.
Any set of labeled examples for C is called a sample set for C .
If C is a concept class, C ∈ C , and S is a sample set for C , we say that S is a teaching set for C (wrt C ) if it is not a sample set for any C ′ ̸
= C , C ′ ∈ C .
A teacher for C is a mapping T that assigns to every concept C ∈ C a sample set for C .
Its order ord( T ) is the size of the largest set in its range, i.e., ord( T ) =
max C ∈C | T ( C ) | .
A learner is a (not necessarily total) mapping that assigns a concept to each sample set in its domain.
A learner L and a teacher T are said to match on a concept class C , if L ( T ( C ))
=
C for every concept C ∈ C .
If C is a concept class over X and
X ′ ⊆
X is any subset of the domain, then we use C↓ X ′ to denote the restriction of C to X ′ , i.e.,
C↓ X ′ =
{ C ↓ X ′ | C ∈ C} , where C ↓ X ′ is the projection of C onto X ′ .
We say that X ′ preserves C if
C 1 ̸
= C 2 implies C 1 ↓ X ′
̸
= C 2 ↓ X ′ whenever C 1 , C 2 ∈ C .
A concept class over X is succinct if it is not preserved by any proper subset of X .
We now define various notions of batch teaching that have been studied in the literature, beginning with the classical notion of teaching dimension.
Definition 1 (Shinohara and Miyano [1991], Goldman and Kearns [1995]) Let C be a concept class over a domain X and C ∈ C be a concept.
The teaching dimension of C in C , denoted by
TD( C, C ) , is the size of the smallest teaching set for C wrt C .
The Teaching Dimension of C is defined as TD( C ) :
= max { TD( C, C ) | C ∈ C} .
Moreover, TD min ( C ) := min { TD( C, C ) | C ∈ C} .
A
TD -teacher for C is a teacher that maps any C ∈ C to a smallest teaching set for C .
A TD -teacher matches every learner that outputs concepts consistent with the given sample set.
The more recent literature proposes models in which “smarter” learners can be used.
In recursive teaching, one exploits a canonical partial order over the concepts in the class: first, the concepts with smallest teaching dimension wrt C are encoded with their minimum teaching sets, then one removes these concepts from C and proceeds recursively with the remaining concepts, always encoding and removing the concepts that are “easiest to teach”.
Definition 2 (Zilles et al.
Let C be a concept class over a domain X .
Let C
= C and recursively define C min
i = { C ∈ C
i | TD( C, C i ) =
TD min ( C i ) } as well as C i +
=
C i \ C min i .
An
RTD -teacher for C is a teacher T with the following property: for C ∈ C min i , the set T ( C ) is a teaching set for C wrt C i .
Then the Recursive Teaching Dimension of C is defined as RTD( C ) :
= max { TD min ( C i )
| i ≥ 0 } , which equals the smallest possible order of any RTD -teacher for C .
A recursive teaching plan for C is any sequence
[Equation]
where T is an RTD -teacher, C min
i =
{ C i, 1 , . . .
, C i,n i } for all i , and C = C min1 ∪ . . .
∪ C min k .
For finite concept classes, the notion of RTD coincides with the Preference-Based Teaching Dimension ( PBTD )
[Gao et al., 2017b], so our results will similarly apply to PBTD .
No-clash teaching is strictly more powerful than recursive teaching; here the only constraint on the teacher is that no two concepts agree on the union of the two sample sets encoding them.
Definition 3 (Kirkpatrick et al.
[2019]) Let C be a concept class over a domain X and T be a teacher for C .
We call T a non-clashing teacher ( NCTD -teacher) on C iff there are no two distinct
C, C ′ ∈ C such that T ( C ) is a sample set for C ′ and T ( C ′ ) is a sample set for C .
The No-Clash
Teaching Dimension of C is defined as NCTD( C ) :
= min { ord( T ) | T is an NCTD -teacher on C} .
It is not hard to show that TD ≥ RTD ≥ NCTD
[Kirkpatrick et al., 2019].
A batch teaching parameter that is incomparable to both RTD and NCTD is the so-called subset teaching dimension.
In subset teaching, one considers all minimum-size teaching sets for each concept, and iteratively reduces them to minimum-size subsets that are not contained in any of the sets assigned to other concepts.
The underlying idea is that a learner can recognize a concept even by a subset S of a smallest teaching set if S is not a subset of a set used for teaching another concept.
Definition 4 (Zilles et al.
Let C be a concept class over a domain X , and C ∈ C .
Define
STS
( C, C ) :
= {{ ( x, C ( x ))
| x ∈ X }} .
For k ∈ N , iteratively define a collection STS k
+1 ( C, C ) of subset teaching sets as the collection that contains all smallest-size sets S that satisfy the following:
S ⊆ S ′ for some S ′ ∈ STS k
( C, C ) ; 2.
S ̸⊆ S ′ for all S ′ ∈ STS k
( C ′ , C ) where C ′ ∈ C ,
C ′ ̸ = C .
Let k ∗ be minimal with
STS k
( ˆ C, C ) =
STS k ∗ ( ˆ C, C ) for all k > k ∗ , ˆ C ∈ C .
An STD -teacher for C is any teacher T for C with T (
ˆ C ) ∈
STS k ∗ ( ˆ C, C ) for all ˆ C ∈ C .
The Subset Teaching Dimension of
C , denoted by STD( C ) , is defined as STD( C ) :
= ord( T ) where T is any STD -teacher for C .
STD is well-defined since every STD -teacher for a class C has the same order, with all sets in
STS k ∗ ( C, C ) being of the same size for any fixed C .
Note that STS 1 ( C, C ) consists of all minimumsize teaching sets for C wrt C .
In general, TD ≥ STD , while STD and RTD are incomparable and either one of them can be larger than the other by an arbitrary factor [Zilles et al., 2011].
STD is well-defined since every STD -teacher for a class C has the same order, with all sets in
STS k ∗ ( C, C ) being of the same size for any fixed C .
Note that STS 1 ( C, C ) consists of all minimumsize teaching sets for C wrt C .
In general, TD ≥ STD , while STD and RTD are incomparable and either one of them can be larger than the other by an arbitrary factor [Zilles et al., 2011].
These teaching notions are illustrated with the help of Table 1, which shows a concept class C pairwith 16 concepts over a domain of size 11.
We claim that TD( C pair) = 4 , RTD( C pair)
= 3 ,
NCTD( C pair) = 2 , STD( C pair) = 1 , and TD min ( C pair) = 1 .
Note that the smallest teaching set for any even-numbered concept C 2 i contains only one example, namely for the instance x i ; thus
TD min ( C pair) = 1 .
By contrast, the smallest teaching set for any odd-numbered concept C 2 i −contains labeled examples for the instances x i , x 9 , x 10 , and x 11 .
The last three instances are needed to distinguish C 2 i − 1 from all other odd-numbered concepts, and x i is needed to distinguish C 2 i −from C 2 i .
Hence TD( C pair) = 4 .
To see that RTD( C pair) = 3 , observe that a recursive teaching plan will first list all even-numbered concepts (whose TD is 1), and then has only the odd-numbered ones left, which correspond to the powerset on { x 9 , x 10 , x 11 } ; clearly these will require a teacher to use sets of size 3.
A non-clashing teacher of order 2 is indicated by the boldface labels in the table.
One can argue (see the proof of Proposition 17) that no non-clashing teacher of smaller order exists, i.e., NCTD( C pair) = 2 .
Finally, STD( C pair) = 1 was shown by Zilles et al.
[2011]; in the first iteration of subset teaching, even-numbered concepts are assigned their unique teaching sets of size 1, while odd-numbered concepts C 2 i − 1 are assigned four-element teaching sets containing the example ( x i , 0) .
Since the latter is not contained in a minimum teaching set of any other concept, in the next iteration, C 2 i − 1 is assigned the subset teaching set { ( x i , 0) } . [Table] C [Table]
C [Table] C [Table]
C
[Table] C [Table]
C [Table]
C
[Table]
C
[Table]
C
[Table]
C
[Table]
C C
C
C C
Desirable Properties of Teaching Models
Our goal is to find a “reasonable” batch teaching complexity measure that is upper-bounded by VCD .
Even an upper bound linear in VCD would be of significance; the best known upper bound on batch teaching complexity, in terms of VCD , is the quadratic upper bound on RTD (and thus on NCTD ) that was established by Hu et al.
One important aspect in the design of “reasonable” teaching models is how much to constrain the information exchange between teacher and learner.
In an attempt to prevent unfair collusion between a teacher and a learner, one limits their interaction by certain constraints.
While there is no general definition of what constitutes collusion, Goldman and Mathias [1996] proposed a notion of collusion-freeness that is often adopted in the literature:
Definition 5 (Goldman and Mathias [1996])
A teacher T and a learner L are Goldman-Mathias (GM) collusion-free on a concept class C if, for every C ∈ C , and every sample set S for C , we have
L ( T ( C ) ∪ S ) = C .
The intuition behind this notion is that the learner is not distracted when presented with additional information on the target concept, beyond the minimum information needed to identify the target concept.
At the same time, this prevents certain “unfair coding tricks”.
For every concept class
C , all TD -teachers, RTD -teachers, and NCTD -teachers can be matched with a learner in a GMcollusion-free way [Zilles et al., 2011, Kirkpatrick et al., 2019].
In particular, NCTD( C ) corresponds to the smallest order of any teacher for C that can be matched with a learner without GM-collusion [Kirkpatrick et al., 2019].
By contrast, teachers and learners using subset teaching usually are not in general GM-collusion-free
[Zilles et al., 2011].
As long as it is open whether NCTD is upper-bounded by (a function linear in) VCD , any quest for a measure of batch teaching complexity that is upper-bounded by (a function linear in) VCD is forced to take a new route: it must consider batch teaching notions that violate the definition of
GM-collusion-freeness, since NCTD is the optimal complexity of GM-collusion-free teaching.
Ignoring GM-collusion-freeness has some practical motivation.
The notion of GM-collusion-freeness is inherently limited in that it excludes learners that expect teachers to use only “important” information (for some notion of importance).
If T ( C ) contains only “important” examples for C , while the sample S for C contains “unimportant” examples, then a learner may be confused by the presence of S in its input T ( C ) ∪ S .
If we want to permit superfluous information to have adverse effects in teaching, then we should not require GM-collusion-freeness.
As an illustration of the naturalness of such effects, consider a typical form of information exchange between a human teacher and learner in a classroom.
Suppose a teacher asks a student to prove Theorem X under premises A, B, and C, and suppose further the student comes up with a proof of Theorem X that uses only premises A
and
B, but not the (consistent yet unnecessary) premise C.
The student may then doubt their own proof, since they assume all the information provided by the teacher to be “important”.
This is in line with recent research on machine teaching.
Ferri et al.
[2022] argued that it is not in general unnatural for a learner to change its mind when receiving data that is consistent with the previous hypothesis.
Their model allows additional information to decrease the probability that the current hypothesis is really correct, and to make an alternate consistent hypothesis appear more likely.
This point of view challenges the widely-held belief that natural teaching and learning should be GM-collusion-free.
Consider for example the class in Table 1.
Here, C 1 and C 2 differ only in x 1 .
The smallest teaching set for C 1 uses four instances ( x 1 , x 9 , x 10 , and x 11 ), but a learner expecting the teacher to use “important” examples might infer C 1 from the information ( x 1 , 0) : while there are 15 concepts consistent with this example, C 1 is the only one for which this information is “important”, as it is the only concept in the class for which this example occurs in a smallest teaching set.
An STD -teacher will use the singleton set { ( x 1 , 0) } to teach C 1 , and has an order of 1 on this concept class.
By contrast, no GM-collusion-free teacher/learner pair can succeed with just a single example, since the
NCTD , which is known to be optimal in terms of GM-collusion-free teaching, is greater than 1.
Rather than defining a weaker notion of collusion-freeness, we propose to consider desirable properties of teachers and learners, which could be used as postulates for reasonable models of teaching and learning.
Depending on the application setting, one may require additional properties.
Two of our intuitive postulates concern monotonicity of a teaching complexity measure.
First, for every concept class C one might want every superset of C to be at least as hard to teach as C , i.e., the teaching complexity of a concept class should be no smaller than that of any of its subclasses.
Second, one might require that removing instances from X without reducing the number of concepts should not decrease teaching complexity, because the removed instances may have been helpful but not harmful in teaching.
Both these monotonicity postulates are captured in the following definition.
Definition 6 (Postulate 1 (Class-Monotonicity) and Postulate 2 (Domain-Monotonicity))
Let Z be any function that assigns a non-negative integer to any concept class.
We say that Z is classmonotonic, if Z ( C ′ ) ≥ Z ( C )
whenever C , C ′ are concept classes over the same domain X , and
C ⊆ C ′ .
We say that Z is domain-monotonic, if Z ( C↓ X ′ ) ≥ Z ( C ) whenever C is a concept class over a domain X , and X ′ ⊆
X is a subset that preserves C .
Our third postulate at first glance appears similar to GM-collusion-freeness.
It states that a teaching set for a concept C should not be contained in a teaching set for another concept C ′ .
Intuitively, a teaching set T ( C ) for C must contain enough “important” examples for C such that any concept
C ′ ̸
= C that is consistent with T ( C ) will be missing at least one of these “important” examples in its own teaching set.
In other words, for every C ′ ̸
= C at least one example in the teaching set for C is “unimportant” for C ′ or inconsistent with C ′ .
Definition 7 (Postulate 3 (Antichain Property))
A teacher T has the antichain property (is an antichain teacher) for a concept class C
if C, C ′ ∈ C and C ̸ = C ′ ⇒ T ( C ) ̸⊆ T ( C ′ ) .
We will sometimes say that a teaching complexity notion satisfies the antichain property if all teachers following the corresponding model are antichain teachers.
For example, since every TD -teacher is an antichain teacher, we say that TD satisfies the antichain property.
It is easy to see that also STD satisfies the antichain property.
Moreover, every RTD -teacher ( NCTD -teacher) can be normalized to an RTD -teacher ( NCTD -teacher, resp.)
of the same order that also satisfies the antichain property; see the supplementary material for a proof.
Proposition 8 Let C be any concept class, Z ∈ { RTD , NCTD } , and T any Z -teacher for C .
Then there is a Z -teacher T ′ for C with ord( T ′ )
= ord( T ) such that T ′ has the antichain property.
An interesting natural (yet non-trivial) property of optimal antichain teachers is that their order for the powerset P n on n instances is linear in n .
(The proof, given in the supplementary material, relies on the theory of bipartite matching.)
Theorem 9 Let T n be an antichain teacher for P n and suppose ord( T n ) ≤ ord( T ) for all antichain teachers T for P n .
Then, for all but finitely many n , we have 0 .
22 · n < ord( T n ) < 0 .
23 · n .
When teaching with sequences, the sample complexity for P n may be as low as Θ( n log n ) [Mansouri et al., 2019]; “natural” batch teaching models thus need strictly larger sample sets.
To sum up, our goal is to design a measure of batch teaching complexity that is upper-bounded by (a function linear in) VCD and satisfies the postulates of (i) domain-monotonicity, (ii) classmonotonicity and (iii) the antichain property.
Since we need to focus on complexity notions violating
GM-collusion-freeness, a first natural candidate is STD .
However, STD is not even upper-bounded by a polynomial in VCD [Zilles et al., 2011].
Moreover, while it satisfies Postulate 3, it violates both Postulates 1 and 2.
The class in Table 1 witnesses that STD is not class-monotonic: it satisfies
STD = 1 , while its subclass of odd-numbered concepts has STD = 3 .
This generalizes as follows:
Proposition 10 (Zilles et al.
[2011]) STD is not class-monotonic.
In particular, for every n > 1 , there is a concept class C with a sub-class C ′ such that STD( C ) =
1 , while STD( C ′ )
= n .
Proposition 11 STD is not domain-monotonic.
In particular, for every n > 3 , there is a concept class C over a domain X = X ′ ∪ X ′′ such that STD( C ) =
n − 1 , while STD( C↓ X ′ ) = 2 .
A Variant of STD
Our main contribution is to introduce a new teaching complexity parameter called STD min , which is built on the idea of STD , yet satisfies the two postulates defined in Section 3.
Moreover, we will show that it is upper-bounded by VCD .
This makes STD min the first known complexity parameter for batch teaching that is proven to be upper-bounded by (a function linear in) VCD .
In calculating STD( C, C ) , at stage k + 1 one maintains the collection of all minimum-size subsets of sets for C from stage k
that are not subsets of sets for any other C ′ at stage k .
In STD min , at every stage we pick only one subset, but do not require it to be minimal.
Definition 12 Let C be a concept class over a domain X .
A sequence T =
( T k ) k ∈ N of teachers for
C is called a subset teaching sequence for C if, for all C ∈ C and all k ∈ N :
[Equation]
Let k ∗ ∈ N be minimal with T k ( C )
=
T k ∗ ( C ) for all k > k ∗ and all C ∈ C .
Then T k ∗ is an
STD min -teacher for C
and we define the order of T on C as ord( T ) =
ord( T k ∗ ) .
Finally, define
STD min ( C ) =
min { ord( T ) | T is a subset teaching sequence for C} .
Definition 13 Let C be a concept class over a domain X .
A sequence T =
( T k ) k ∈ N of teachers for C is called a subset teaching sequence for C if, for all C ∈ C and all k ∈ N : [Equation] Let k ∗ ∈ N be minimal with T k ( C )
=
T k ∗ ( C ) for all k > k ∗ and all C ∈ C .
Then T k ∗ is an
STD min -teacher for C
and we define the order of T on C as ord( T ) = ord( T k ∗ ) . Finally, define STD min ( C ) =
min { ord( T ) | T is a subset teaching sequence for C} .
Observation 1 Every subset teaching sequence of order d can be transformed into a normalized sequence ( T k ) k ∈ N of the same order, where a normalized subset teaching sequence has the property that, for every k < k ∗ and every C ∈ C , we have (i) T k +1 differs from T k on exactly one concept, (ii)
| T k +1 ( C ) | ∈ {| T k ( C ) | − 1 , | T k ( C ) |} , (iii) | T k ( C )
| ≥ d , which implies that | T k ∗ ( C ) | = d .
Before showing that STD min satisfies our desired properties, we claim that STD min is upper-bounded by STD and can be arbitrarily smaller than STD (the proof is in the supplementary material):
Proposition 14 STD min ( C ) ≤ STD( C ) , and for all n ∈ N there is some succinct C n such that
STD min ( C n ) = 2 and STD( C ) = n .
Since STD min ≤ STD and STD can be smaller than NCTD , which is the best possible complexity of GM-collusion-free teaching and learning, we immediately obtain:
Corollary 15 There are concept classes for which no teacher of order STD min can be matched with a learner in a GM-collusion-free way.
Next, we claim that STD min also meets Postulates 1 and 2, and thus greatly improves on STD in terms of the postulates defined in Section 3; see the supplementary material for a proof.
Proposition 16 STD min is class-monotonic, domain-monotonic, and satisfies the antichain property.
Relationship of STD min to Other Complexity Parameters
We already know that STD min is upper-bounded by TD and STD .
Subsequently, we will show that, for finite concept classes, it is also upper-bounded by RTD and by VCD , but not by NCTD .
Most importantly, our results make STD min the first known parameter for batch teaching complexity that is known to be upper-bounded by VCD (or even by a function linear in VCD ), for finite classes.
We first show that STD min can be arbitrarily smaller than NCTD , but also by a factor of 2 larger than
NCTD .
The proof, which is detailed in Appendix ?? , uses a generalization of C pairfrom Table 1:
Proposition 17 For every n ∈ N there is (i) a concept class C with STD( C ) =
STD min ( C ) = 1 and
NCTD
Theorem 18 Let C be any concept class.
Then STD min ( C ) ≤ RTD( C ) .
Proof.
Let (( C 1 , T ( C 1 )) , ( C 2 , T ( C 2 )) , . . .
, ( C n , T ( C n ))) be a recursive teaching plan for C .
In particular, we have C
=
{ C 1 , . .
C n } , |C| = n , and, for all i ∈ { 1 , . . . , n } , the set T ( C i ) is a teaching set for C i with respect to { C j | j ≥ i } .
Moreover, max i | T ( C i ) | =
RTD( C ) .
We will use this recursive teaching plan to construct a subset teaching sequence of order at most RTD( C ) for C .
First, for all C ∈ C , define T 0 ( C ) as per Definition 13.
Second, define T 1 by setting T 1 ( C 1 )
=
T ( C 1 ) and T 1 ( C j )
=
T 0 ( C j ) for all j > 1 .
Since T ( C 1 ) is a teaching set for C 1 wrt { C j | j ≥ 1 } , we have that T 1 ( C 1 ) ̸⊆ T 0 ( C j ) for all j ̸ = 1 , so that T 1 satisfies the requirements of Definition 13.
Moreover, | T 1 ( C 1 ) | ≤ RTD( C ) .
Next, we define T k for k > 1 by the following algorithm.
For j ̸ = k , let T k ( C j )
=
T k − 1 ( C j ) .
To define T k ( C k ) , initialize T k ( C k ) :
= T ( C k ) .
Let J k :=
{ j ̸ = k | T k ( C k ) ⊆
T k − 1 ( C j ) } . 4.
If J k = ∅ , then stop and output T k ( C k ) .
Let j ∗ :
= min J k .
Pick any ( x, l ) ∈ T k − 1 ( C j ∗ ) such that C k ( x ) ̸ = l . 6.
Let T k
( C k ) :
= T k ( C k ) ∪ { ( x, C k
( x )) } .
Goto step 3.
Obviously, T k ( C k ) ⊆
T k − 1 ( C k ) .
To complete the proof of Theorem 18, we show that T k ( C k ) ̸⊆
T k − 1 ( C ) for all C ̸ = C k , C ∈ C and that | T k ( C k ) | ≤ RTD( C ) .
Note that | T ( C j ) | ≤ RTD( C ) for all j ; thus | T 1 ( C 1 ) | ≤ RTD( C ) .
If J k is initially empty, then also | T k ( C k ) | ≤ RTD( C ) , and, by definition of J k we have T k ( C k )
̸⊆ T k − 1 ( C ) for all C ̸ = C k .
So suppose J k is not initially empty.
Since T k ( C k ) ⊇ T ( C k ) at any point in time during the construction, and T ( C k ) is a teaching set for C k wrt { C j | j ≥ k } , we have j < k for all j ∈ J k at any point in time.
Now let j ∗ :
= min J k .
By construction, T k − 1 ( C j ∗ ) ⊇ T ( C j ∗ ) ; in particular, T k − 1 ( C j ∗ ) is a teaching set for C j ∗ wrt { C j | j ≥ j ∗ } .
Hence there exists some ( x, l ) ∈ T k − 1 ( C j ∗ ) such that
C k ( x ) ̸ = l .
Thus, step 6 guarantees that T k ( C k ) ̸⊆ T k − 1 ( C j ∗ ) , and at least j ∗ is removed from J k at the next iteration of step 3.
This way, eventually T k ( C k )
̸⊆
T k − 1 ( C ) for all C ̸ = C k .
All that remains to be shown is that | T k ( C k ) | ≤ RTD( C ) .
We do this by proving inductively for all k that | T k
( C j ) | ≤ RTD( C ) for all j ≤ k .
Note that the latter holds true for k = 1 .
Assuming we have proven | T k − 1 ( C j ) | ≤ RTD( C ) for all j ≤ k − 1 , we claim that step 6 will only add an element to
T k ( C k ) if | T k ( C k ) | ≤ RTD( C ) − 1 .
For step 6 to add an element to T k ( C k ) , there must be (at that point in time) a j ∗ < k such that T k ( C k ) ⊆
T k − 1 ( C j ∗ ) .
Since T k − 1 ( C j ∗ ) is a teaching set for C j ∗ wrt { C j | j ≥ j ∗ } , the set T k − 1 ( C j ∗ ) is inconsistent with C k .
In particular, T k ( C k ) is a proper subset of T k − 1 ( C j ∗ ) .
By induction hypothesis, | T k − 1 ( C j ∗ ) | ≤ RTD( C ) .
Hence, | T k ( C k ) | ≤ RTD( C ) −before step 6 adds an element to T k ( C k ) .
In the sum, we have verified that ( T k ) k ∈ N is a subset teaching sequence for C and that its order is at most RTD( C ) .
Thus STD min ( C ) ≤ RTD( C ) .
□
The main result of this section establishes STD min as the first notion of batch teaching complexity for which VCD is provably an upper bound.
Our proof of this result relies on the recursive decomposition of a concept class, as exploited by [Floyd and Warmuth, 1995] and in the proof that VCD can upper-bound the sample complexity of sequential teaching [Mansouri et al., 2019].
If C is a concept class over X and x ∈ X any element of the domain, then we use C x to denote C↓ X \{ x } , and C x to denote { C ∈ C x | C ∪{ x } ∈ C and C ∈ C} .
Floyd and Warmuth call C x
the restriction and C x the reduction of C wrt x .
It is easy to see that
VCD( C x ) ≤ VCD( C ) and VCD( C x ) ≤ VCD( C ) − 1 [Floyd and Warmuth, 1995].
Theorem 19 Let C be any concept class.
Then STD min ( C ) ≤ VCD( C ) .
Proof.
Let d = VCD( C ) ≥ 1 .
Pick some x ∗ ∈ X .
We may assume inductively that the assertion of the theorem is true for the classes C x ∗ and C x ∗ .
Let T rest =
( T rest k ) k ≤ k rest be any subset teaching sequence for C x ∗ with ord( T rest )
=
d .
Here k rest denotes the maximum number of iterations T rest takes to converge on any concept in C x ∗ .
Further, let T red =
( T red k ) k ≤ k red be any subset teaching sequence for C x ∗ with ord( T red ) =
d − 1 .
Here k red denotes the maximum number of iterations
T red takes to converge.
Since any subset teaching sequence can be normalized, it can be assumed in the sequel that the sequences T rest and T red have the normalization properties (i)—(iii) as in
Observation 1.
Now define a subset teaching sequence T =
( T k ) k ∈ N for C as follows.
For all C ∈ C initialize T 0 ( C ) := { ( x, C ( x )) | x ∈ X } .
From now on, we use C x ∗ ,X to denote the set of concepts C ∈ C for which there is some C ′ ∈ C such that C and C ′ differ only in the value of x ∗ .
(Note that C x ∗ ,X has twice as many concepts as C x ∗ and its restriction wrt x ∗ yields C x ∗ .)
Stage 1.
Let k ∈
[1 , k red + 1] .
For all C ∈ C x ∗ ,X set T k ( C ) :
= T red k − 1 ( C ↓ X \{ x ∗ } ) ∪ { ( x ∗ , C ( x ∗ )) } .
For all C ∈ C \ C x ∗ ,X set T k ( C ) :
= { ( x, C ( x )) | x ∈ X \ { x ∗ }} .
Stage 2.
Let k ∈
[ k red + 2 , k red + k rest + 1] .
For all C ∈ C x ∗ ,X , let T k ( C ) :
= T k − 1 ( C ) .
For all
C ∈ C \ C x ∗ ,X , let T k ( C ) :
= T rest k − k red − 1 ( C ↓ X \{ x ∗ } ) .
We claim that ( T k ) k ≤ k red + k rest +1 is a subset teaching sequence for C and has order at most d .
First, we show that the partial sequence defined in Stage 1 is valid, i.e., for all k ≤ k red + 1 and all
C, C ′ ∈ C with C ̸ = C
′ we have T k ( C ) ⊆
T k − 1 ( C ) and T k ( C ) ̸⊆ T k − 1 ( C ′ ) .
The fact T k ( C ) ⊆
T k − 1 ( C ) is inherited from the same fact for the sequence T red .
To show that T k ( C )
̸⊆ T k − 1 ( C ′ ) , consider multiple cases.
If C, C ′ ∈ C x ∗ ,X , this property is again inherited from T red .
If C ∈ C x ∗ ,X and C ′ / ∈ C x ∗ ,X , then ( x ∗ , C ( x ∗ ))
∈ T k ( C ) \ T k − 1 ( C ′ )
when k > 1 , while for k = 1 the property
T k ( C ) ̸⊆ T k − 1 ( C ′ ) is valid because T 1 ( C )
=
T 0 ( C ) .
If C / ∈ C x ∗ ,X , then C and C ′ must differ on some instance x ̸
= x ∗ ; since T k ( C ) contains ( x, C ( x )) , again we obtain T k ( C ) ̸⊆ T k − 1 ( C ′ ) .
Second, we show that the partial sequence defined in Stage 2 is a valid extension of that defined in Stage 1, i.e., for k red + 2 ≤ k ≤ k red + k rest + 1 and all C, C ′ ∈ C with C ′ ̸
=
C we have
T k ( C ) ⊆
T k − 1 ( C ) and T k ( C ) ̸⊆ T k − 1 ( C ′ ) .
So, fix C and C ′ ̸ = C .
For C ∈ C x ∗ ,X , the sets defined in Stage 1 do not get updated in Stage 2.
Hence we may safely assume that C / ∈ C x ∗ ,X .
T k ( C ) ⊆
T k − 1 ( C ) is inherited from the same property for T rest .
If C ′ / ∈ C x ∗ ,X , then the analogous remark applies to T k ( C ) ̸⊆ T k − 1 ( C ′ ) .
Suppose now that C ′ ∈ C x ∗ ,X .
Here we make use of the normalization properties (i)—(iii).
In Stage 2, all sets T k − 1 ( C ′ ) with C ′ ∈ C x ∗ ,X are of size d while all sets T k ( C ) with C / ∈ C x ∗ ,X are of size at least d .
Since T k − 1 ( C ′ ) contains ( x ∗ , C ( x ∗ )) , it is not a subset of T k ( C ) .
Together with | T k ( C ) | ≥ | T k − 1 ( C ′ ) | , it now follows that T k ( C )
̸⊆ T k − 1 ( C ′ ) .
Consequently, T is a subset teaching sequence for C .
The identity ord( T ) =
d is obvious.
□
A Note on Unambiguity in Teaching
While STD min meets our objectives outlined earlier, it loses one potentially desirable property of
STD that has not been addressed in the literature before.
For every teaching model defined above, there are concept classes which can be taught with more than one teacher satisfying the requirements of the model.
Suppose Teacher 1 teaches concept C 1 ∈ C using sample set S , while Teacher 2 teaches concept C 2 ∈ C ,
C 2 ̸
= C 1 , with S .
If the learner knows that she is interacting with some teacher of a given model, but doesn’t know which one, she then cannot disambiguate between C 1 and C 2 when given input S .
In unambiguous teaching models, the learner can infer the target concept from the given sample set without being told with which of the multiple valid teachers she is interacting.
Definition 20 Let Z ∈ { TD , RTD , NCTD , STD , STD min } be a teaching complexity parameter.
We say that Z is unambiguous on a concept class C , if, for every sample set S , there is at most one concept C ∈ C such that some Z -teacher T of order Z ( C ) satisfies T ( C )
=
S . If Z is ambiguous on a concept class, then one would have to normalize Z -teachers or use disambiguating information.
One disadvantage of STD min compared to STD is the loss of unambiguity.
Theorem 21 TD , RTD , and STD are unambiguous on every concept class, while there are concept classes on which NCTD and STD min are not unambiguous.
Proof.
Let C be any concept class and S a sample set.
If a TD -teacher T maps C to S , then S is a teaching set for C and thus cannot be mapped to any other concept by any TD -teacher T ′ .
If an
RTD -teacher T maps C to S and C ∈ C min i , then S is inconsistent with the concepts in C ∈ C i \{ C } and thus cannot be mapped to the latter.
The concepts in C \ C i cannot be mapped to S by any
RTD -teacher either, since they must be mapped to sample sets that are inconsistent with all concepts in C ∈ C i , which includes C .
If an STD -teacher T maps C to S , then S ∈ STS k ∗ ( C, C ) .
Since the latter is disjoint from S ∈ STS k ∗ ( C ′ , C ) for all C ′ ̸
= C , no STD -teacher can map such C ′ to S .
It is easy to see that there are two NCTD -teachers of order 1 for P 2 that even have the same range, one maps ∅ to { ( x 1 , 0) } and { x 1 } to { ( x 2 , 0) } , the other one does the opposite.
A concept class on which STD min is ambiguous is Warmuth’s class
[Doliwa et al., 2014], which is known as the smallest class on which RTD exceeds VCD .
For details, we refer to the supplementary material.
□
The ambiguity observed for Z ∈ { NCTD , STD min } is stronger than the negation of the condition in
Definition 20.
Ambiguity means that multiple optimal Z -teachers map two different concepts to the same teaching set.
But it even happens that two optimal Z -teachers T 1 ̸ = T 2 have exactly the same range R =
{ T 1 ( C ) | C ∈ C} =
{ T 2 ( C ) | C ∈ C} , while assigning the sample sets in R differently to the concepts in C .
The ambiguity of STD min can take extreme forms for artificially created concept classes that have many redundant instances, as discussed in the supplementary material.
Conclusions
In an abstract setting, devoid of any specific application, it is not straightforward to define what makes a teaching/learning model “natural”.
We addressed this problem by considering various properties of teaching that one might find desirable; one can then design models that possess (some of) these properties as seen fit in an underlying application setting.
We have introduced some such properties, here called “postulates”, which are fulfilled by STD min .
Another property of STD min , that is not fulfilled by TD , RTD or NCTD , is that it allows the learner to discard a hypothesis because the given data is not important for that hypothesis (even if it is consistent).
We argue that a learner who knows that data comes from a helpful teacher, has reasons to reject hypotheses for which the data is not important/helpful.
Of course, there may be other natural properties that are not fulfilled by
STD min .
An interesting research question is what natural properties can (or cannot) be fulfilled by teaching models whose parameter is upper-bounded by VCD .
This paper is the first to provide an upper bound of VCD on batch teaching complexity.
Our results and proofs may have implications on the design and analysis of teaching and learning algorithms.
For an overview of most of our results, see Table 2.
While we diverted from the notion of GMcollusion-freeness, we hope that the postulates on teaching that we defined, as well as our notion of unambiguity, will be useful for future studies on machine teaching.
In particular, noting that neither NCTD nor STD min satisfy the notion of unambiguity, one may ask the question whether ambiguity is inherent in “natural” teaching models that yield a complexity upper-bounded by VCD .
All unambiguous models studied in this paper have a complexity that sometimes exceeds VCD .
[Table]
Acknowledgments and Disclosure of Funding
This research was conducted while S. Zilles was a visiting scientist at the Max Planck Institute for Software Systems.
Further, S. Zilles acknowledges financial support by the Natural Sciences and Engineering Research Council (NSERC) of Canada, both through the Canada Research Chairs program and the Discovery Grants program.
She was also supported through the Canada CIFAR AI
[-Then STD min ( C ) = 
 and there are at least |C| ! pairwise distinct STD min -teachers for C with the same range on C . In particular, every permutation σ of C yields-]Chairs program as an Affiliate Chair with the Alberta Machine Intelligence Institute (Amii).