Original: Universal Rates for Interactive Learning
Revised: Universal Rates for Interactive Learning

Original: Abstract
Revised: Abstract

Original: Consider the task of learning an unknown concept from a given concept class; to what extent does interacting with a domain expert accelerate the learning process?
Revised: Consider the task of learning an unknown concept from a given concept class; to what extent does interacting with a domain expert accelerate the learning process?

Original: It is common to measure the effectiveness of learning algorithms by plotting the "learning curve", that is, the decay of the error rate as a function of the algorithm’s resources (examples, queries, etc).
Revised: It is common to measure the effectiveness of learning algorithms by plotting the "learning curve", that is, the decay of the error rate as a function of the algorithm’s resources (examples, queries, etc).

Original: Thus, the overarching question in this work is whether (and which kind of) interaction accelerates the learning curve.
Revised: Thus, the overarching question in this work is whether (and which kind of) interaction accelerates the learning curve.

Original: Previous work in interactive learning focused on uniform bounds on the learning rates which only capture the upper envelope of the learning curves over families of data distributions.
Revised: Previous work in interactive learning focused on uniform bounds on the learning rates which only capture the upper envelope of the learning curves over families of data distributions.

Original: We thus formalize our overarching question within the distribution dependent framework of universal learning, which aims to understand the performance of learning algorithms on every data distribution, but without requiring a single upper bound which applies uniformly to all distributions.
Revised: We thus formalize our overarching question within the distribution dependent framework of universal learning, which aims to understand the performance of learning algorithms on every data distribution, but without requiring a single upper bound which applies uniformly to all distributions.

Original: Our main result reveals a fundamental trichotomy of interactive learning rates, thus providing a complete characterization of universal interactive learning.
Revised: Our main result reveals a fundamental trichotomy of interactive learning rates, thus providing a complete characterization of universal interactive learning.

Original: As a corollary we deduce a strong afﬁrmative answer to our overarching question, showing that interaction is beneﬁcial.
Revised: As a corollary we deduce a strong afﬁrmative answer to our overarching question, showing that interaction is beneﬁcial.

Original: Remarkably, we show that in important cases such beneﬁts are realized with label queries, that is, by active learning algorithms.
Revised: Remarkably, we show that in important cases such beneﬁts are realized with label queries, that is, by active learning algorithms.

Original: On the other hand, our lower bounds apply to arbitrary binary queries and, hence, they hold in any interactive learning setting.
Revised: On the other hand, our lower bounds apply to arbitrary binary queries and, hence, they hold in any interactive learning setting.

Original:  Introduction
Revised:  Introduction

Original: In supervised learning, arguably the most commonly studied variation of machine learning, we consider a learning algorithm that is typically given access to a training set of n labeled examples, sampled from an unknown distribution.
Revised: In supervised learning, arguably the most commonly studied variation of machine learning, we consider a learning algorithm that is typically given access to a training set of n labeled examples, sampled from an unknown distribution.

Original: Based on this training set, the goal of the algorithm is to learn to output a concept/function that maps data points to labels so that the probability of making a mistake on unseen (unlabeled) data, drawn from the same distribution, is minimized.
Revised: Based on this training set, the goal of the algorithm is to learn to output a concept/function that maps data points to labels so that the probability of making a mistake on unseen (unlabeled) data, drawn from the same distribution, is minimized.

Original: However, this passive setting does not capture a lot of applications in which there is an abundance of unlabeled data where labels are not readily available but the learner can interact with the domain expert (such as a human annotator) to acquire more information about the data.
Revised: However, this passive setting does not capture a lot of applications in which there is an abundance of unlabeled data where labels are not readily available but the learner can interact with the domain expert (such as a human annotator) to acquire more information about the data.

Original: Consider, for example, the task of email spam ﬁltering.
Revised: Consider, for example, the task of email spam ﬁltering.

Original: Service providers, such as Google, have easily access to billions of emails.
Revised: Service providers, such as Google, have easily access to billions of emails.

Original: However, obtaining labels (i.e., spam or not) usually requires a human to read the content of those emails, a process that is quite costly.
Revised: However, obtaining labels (i.e., spam or not) usually requires a human to read the content of those emails, a process that is quite costly.

Original: One way to model such tasks is through the interactive learning framework where the learning algorithm is given access to a large (potentially inﬁnite) stream of unlabeled examples from which it can then submit n queries to an expert in order to gain information about these examples.
Revised: One way to model such tasks is through the interactive learning framework where the learning algorithm is given access to a large (potentially inﬁnite) stream of unlabeled examples from which it can then submit n queries to an expert in order to gain information about these examples.

Original: Such queries may include, but are not limited to, asking the labels of those examples (e.g., is this email spam?) or comparing examples with one another (e.g., between these two emails which one looks more like spam?).
Revised: Such queries may include, but are not limited to, asking the labels of those examples (e.g., is this email spam?) or comparing examples with one another (e.g., between these two emails which one looks more like spam?).

Original: In this paper we would like to understand to what extent interacting with an expert who knows the target concept can accelerate the learning process.
Revised: In this paper we would like to understand to what extent interacting with an expert who knows the target concept can accelerate the learning process.

Original: It is common to measure the effectiveness of passive supervised learning processes by plotting the "learning curve", that is, the decay of the error rate as a function of the number of training examples.
Revised: It is common to measure the effectiveness of passive supervised learning processes by plotting the "learning curve", that is, the decay of the error rate as a function of the number of training examples.

Original: Analogously, for interactive learning algorithms one measures the decay of the error rate as a function of the number of queries the algorithm submitted to the domain expert.
Revised: Analogously, for interactive learning algorithms one measures the decay of the error rate as a function of the number of queries the algorithm submitted to the domain expert.

Original: The setting in which the learner is limited to asking label queries is called active learning.
Revised: The setting in which the learner is limited to asking label queries is called active learning.

Original: The overarching question in this work is whether (and which kinds of) interaction accelerates the learning curve?
Revised: The overarching question in this work is whether (and which kinds of) interaction accelerates the learning curve?

Original: Previous work on interactive learning has given rise to a large collection of general theories [KMT93, Das05, BBL09, Han07b, Han07a, DHM07, Han09, YHC10, BH12, Han14, HY15,
Revised: Previous work on interactive learning has given rise to a large collection of general theories [KMT93, Das05, BBL09, Han07b, Han07a, DHM07, Han09, YHC10, BH12, Han14, HY15,

Original: BHLZ16, KLMZ17] establishing guarantees on the error rate as a function of the number of queries of various types.
Revised: BHLZ16, KLMZ17] establishing guarantees on the error rate as a function of the number of queries of various types.

Original: However, most of these theories are primarily concerned with capturing a uniform (i.e., minimax ) guarantee on the rates of convergence: that is, error bounds holding in the worst case over some family of distributions.
Revised: However, most of these theories are primarily concerned with capturing a uniform (i.e., minimax ) guarantee on the rates of convergence: that is, error bounds holding in the worst case over some family of distributions.

Original: Even though these theories give rise to a clean and elegant mathematical framework, they do not fully explain the behavior of learning curves: due to the minimax nature of the guarantees they consider, they can only capture the upper envelope of the learning curves over a family of data distributions.
Revised: Even though these theories give rise to a clean and elegant mathematical framework, they do not fully explain the behavior of learning curves: due to the minimax nature of the guarantees they consider, they can only capture the upper envelope of the learning curves over a family of data distributions.

Original: The minimax perspective does not match practical and experimental machine learning, where the target distribution is ﬁxed and the number of training examples and queries varies according to the learner’s resources and desired accuracy [CT90, CT92].
Revised: The minimax perspective does not match practical and experimental machine learning, where the target distribution is ﬁxed and the number of training examples and queries varies according to the learner’s resources and desired accuracy [CT90, CT92].

Original: In contrast, in order to observe the learning curve that corresponds to uniform/minimax guarantees one would have to vary the data-generating distribution as the constraint on the number of queries the learner can make increases.
Revised: In contrast, in order to observe the learning curve that corresponds to uniform/minimax guarantees one would have to vary the data-generating distribution as the constraint on the number of queries the learner can make increases.

Original: We therefore formalize our overarching question within the distribution dependent framework of universal learning [BHM + 21], a framework designed to understand optimal rates of convergence for learning curves.
Revised: We therefore formalize our overarching question within the distribution dependent framework of universal learning [BHM + 21], a framework designed to understand optimal rates of convergence for learning curves.

Original: This framework aims to understand the best possible asymptotic rate of convergence that can hold for every data distribution, but without requiring an upper bound which applies uniformly to all of these distributions.
Revised: This framework aims to understand the best possible asymptotic rate of convergence that can hold for every data distribution, but without requiring an upper bound which applies uniformly to all of these distributions.

Original: While there have been a few past works to study universal rates for the special case of active learning (i.e., label queries) [BHW08, Han12, YH13], none have yet provided a complete characterization, and none have considered more general forms of interactive learning.
Revised: While there have been a few past works to study universal rates for the special case of active learning (i.e., label queries) [BHW08, Han12, YH13], none have yet provided a complete characterization, and none have considered more general forms of interactive learning.

Original: The main result of this paper is a complete characterization of universal interactive learning rates, for learners able to make arbitrary yes/no queries.
Revised: The main result of this paper is a complete characterization of universal interactive learning rates, for learners able to make arbitrary yes/no queries.

Original: Speciﬁcally, for such learners, we prove there is a fundamental trichotomy of possible optimal universal learning rates: any given concept class has an optimal rate decaying at either (i) arbitrarily fast rates, (ii) exponential rates, or (iii) arbitrarily slow rates.
Revised: Speciﬁcally, for such learners, we prove there is a fundamental trichotomy of possible optimal universal learning rates: any given concept class has an optimal rate decaying at either (i) arbitrarily fast rates, (ii) exponential rates, or (iii) arbitrarily slow rates.

Original: Moreover, the optimal rates for classes of type (i) can be attained by learning rules whose interaction is restricted to querying the labels of points in the (unlabeled) input sample.
Revised: Moreover, the optimal rates for classes of type (i) can be attained by learning rules whose interaction is restricted to querying the labels of points in the (unlabeled) input sample.

Original: In other words, active learning with arbitrarily fast rates is possible for a variety of classes including, and rather surprisingly, for some classes with inﬁnite VC dimension.
Revised: In other words, active learning with arbitrarily fast rates is possible for a variety of classes including, and rather surprisingly, for some classes with inﬁnite VC dimension.

Original: Furthermore, we pinpoint complexity measures to precisely categorize any concept class into one of the above three categories of optimal rate.
Revised: Furthermore, we pinpoint complexity measures to precisely categorize any concept class into one of the above three categories of optimal rate.

Original: Indeed, since these same complexity measures were also found by [BHM + 21] to characterize a trichotomy of rates for passive supervised learning, our results are directly comparable, and reveal a strong beneﬁt from interactive learning: namely, classes with optimal rate 1 n for supervised learning have optimal rate e  n for interactive learning (category ii), while classes with optimal rate e  n for supervised learning have optimal rates that are arbitrarily fast for interactive learning (category i).
Revised: Indeed, since these same complexity measures were also found by [BHM + 21] to characterize a trichotomy of rates for passive supervised learning, our results are directly comparable, and reveal a strong beneﬁt from interactive learning: namely, classes with optimal rate 1 n for supervised learning have optimal rate e  n for interactive learning (category ii), while classes with optimal rate e  n for supervised learning have optimal rates that are arbitrarily fast for interactive learning (category i).

Original: In addition to these strong positive results, including for learning with simple label queries, we also note that our lower bounds on the achievable universal rates are quite powerful, as they apply to any interactive learning setting with binary-valued responses: for instance, in addition to label queries and general membership queries, they also apply to learners based on comparison queries [KLMZ17]
Revised: In addition to these strong positive results, including for learning with simple label queries, we also note that our lower bounds on the achievable universal rates are quite powerful, as they apply to any interactive learning setting with binary-valued responses: for instance, in addition to label queries and general membership queries, they also apply to learners based on comparison queries [KLMZ17]

Original: Finally, novel algorithmic principles arise from our universal interactive learning framework.
Revised: Finally, novel algorithmic principles arise from our universal interactive learning framework.

Original: Recall that passive learning in the distribution-free PAC setting boils down to the Empirical Risk Minimization (ERM) principle via uniform convergence.
Revised: Recall that passive learning in the distribution-free PAC setting boils down to the Empirical Risk Minimization (ERM) principle via uniform convergence.

Original: Analogously, traditional interactive learning gives rise to simple querying policies such as the disagreement-based CAL algorithm [CAL94] algorithm for label queries (which queries the labels of points that are uncertain), or general binary search strategies in a cover of the class [KMT93] in the case of general yes/
Revised: Analogously, traditional interactive learning gives rise to simple querying policies such as the disagreement-based CAL algorithm [CAL94] algorithm for label queries (which queries the labels of points that are uncertain), or general binary search strategies in a cover of the class [KMT93] in the case of general yes/

Original: no queries.
Revised: no queries.

Original: Similarly to the ERM principle, these strategies achieve near-optimal distribution-free uniform rates, for their respective types of queries.
Revised: Similarly to the ERM principle, these strategies achieve near-optimal distribution-free uniform rates, for their respective types of queries.

Original: In contrast, in the universal setting the algorithmic landscape is richer and the design of learning algorithms able to achieve the optimal rates requires great care.
Revised: In contrast, in the universal setting the algorithmic landscape is richer and the design of learning algorithms able to achieve the optimal rates requires great care.

Original: Indeed, the learning rules we use to prove our main results rely on techniques only recently introduced to the learning theory literature, such as Gale-Stewart sequential games and adversarial online learning with inﬁnite ordinal-valued Littlestone dimensions [GS53, BHM + 21].
Revised: Indeed, the learning rules we use to prove our main results rely on techniques only recently introduced to the learning theory literature, such as Gale-Stewart sequential games and adversarial online learning with inﬁnite ordinal-valued Littlestone dimensions [GS53, BHM + 21].

Original: Thus, the universal learning setting motivates the development of a richer variety of algorithmic techniques.
Revised: Thus, the universal learning setting motivates the development of a richer variety of algorithmic techniques.

Original: Uniform vs. Universal Rates.
Revised: Uniform vs. Universal Rates.

Original: To make the distinction between uniform and universal rates more concrete, we ﬁrst recall what uniform learnability means.
Revised: To make the distinction between uniform and universal rates more concrete, we ﬁrst recall what uniform learnability means.

Original: (The notation will be introduced formally below.)
Revised: (The notation will be introduced formally below.)

Original: A concept class H is uniformly learnable at rate R ( n ) if there exists a learning rule ˆ h n such that
Revised: A concept class H is uniformly learnable at rate R ( n ) if there exists a learning rule ˆ h n such that

Original: [Equation]
Revised: [Equation]

Original: In words, there exists a learning rule ˆ h n and distribution-independent constants C, c > 

Revised: In words, there exists a learning rule ˆ h n and distribution-independent constants C, c >

Original: so that for every realizable data-generating distribution P 2 RE( H ) the expected error of the classiﬁer
Revised: so that for every realizable data-generating distribution P 2 RE( H ) the expected error of the classiﬁer

Original: E [er(ˆ h n )] is bounded by CR ( cn ) .
Revised: E [er(ˆ h n )] is bounded by CR ( cn ) .

Original: The deﬁnition of universal learnability is obtained by a simple rearrangement of the quantiﬁers of the previous deﬁnition.
Revised: The deﬁnition of universal learnability is obtained by a simple rearrangement of the quantiﬁers of the previous deﬁnition.

Original: A concept class H is universally learnable at some rate R ( n ) if there exists a learning rule ˆ h n such that
Revised: A concept class H is universally learnable at some rate R ( n ) if there exists a learning rule ˆ h n such that

Original: [Equation]
Revised: [Equation]

Original: 2 RE( H )  9 C, c > such that E [er(ˆ h n )]  CR ( cn ) , 8 n 2 N .
Revised: 2 RE( H )  9 C, c > such that E [er(ˆ h n )]  CR ( cn ) , 8 n 2 N .

Original: Importantly, this subtle change in the deﬁnition allows the constants C, c to be distribution- dependent .
Revised: Importantly, this subtle change in the deﬁnition allows the constants C, c to be distribution- dependent .

Original: As is evident from our main result below, this, seemingly minor, change makes the landscape of the admissible learning rates of interactive learning algorithms vastly different.
Revised: As is evident from our main result below, this, seemingly minor, change makes the landscape of the admissible learning rates of interactive learning algorithms vastly different.

Original: 1.1 Main Results
Revised: 1.1 Main Results

Original: We next present the key deﬁnitions and summarize the main results of this work.
Revised: We next present the key deﬁnitions and summarize the main results of this work.

Original: The learning problem consists of a domain X , where we assume X to be a Polish space, and a nonempty concept class H ✓ { 0 , 1 } X .
Revised: The learning problem consists of a domain X , where we assume X to be a Polish space, and a nonempty concept class H ✓ { 0 , 1 } X .

Original: We assume that H satisﬁes standard measurability assumptions (see Appendix A.1).
Revised: We assume that H satisﬁes standard measurability assumptions (see Appendix A.1).

Original: A classiﬁer is a universally measurable function h : X ! { 0 , 1 } , whose error rate is deﬁned to be er( h )
Revised: A classiﬁer is a universally measurable function h : X ! { 0 , 1 } , whose error rate is deﬁned to be er( h )

Original: =
Revised: =

Original: er P ( h ) :
Revised: er P ( h ) :

Original: = P { ( x, y ) : h ( x ) 6 = y } , where P is a probability distribution over X ⇥ { 0 , 1 } .
Revised: = P { ( x, y ) : h ( x ) 6 = y } , where P is a probability distribution over X ⇥ { 0 , 1 } .

Original: We say that P is realizable with respect to H if inf h 2 H er( h ) = 0 .
Revised: We say that P is realizable with respect to H if inf h 2 H er( h ) = 0 .

Original: We denote with P X its marginal distribution on
Revised: We denote with P X its marginal distribution on

Original: X .
Revised: X .

Original: We deﬁne an interactive learning algorithm to be a sequence of universally measurable functions which, given access to an inﬁnite stream of unlabeled data points from X that are drawn i.i.d.
Revised: We deﬁne an interactive learning algorithm to be a sequence of universally measurable functions which, given access to an inﬁnite stream of unlabeled data points from X that are drawn i.i.d.

Original: from
Revised: from

Original: P X and a query budget n , output a classiﬁer ˆ h n : X ⇥ { 0 , 1 } ; we restrict the algorithm to depend on only a ﬁnite (though unbounded) number of the i.i.d.
Revised: P X and a query budget n , output a classiﬁer ˆ h n : X ⇥ { 0 , 1 } ; we restrict the algorithm to depend on only a ﬁnite (though unbounded) number of the i.i.d.

Original: unlabeled examples.
Revised: unlabeled examples.

Original: The goal of this algorithm is to come up with classiﬁers whose expected error E [er(ˆ h n )] decays as fast as possible as a function of the number of queries n.
Revised: The goal of this algorithm is to come up with classiﬁers whose expected error E [er(ˆ h n )] decays as fast as possible as a function of the number of queries n.

Original: The aim of this paper is to characterize the learning rates achievable by a learning algorithm in a general interactive setting, where the algorithm can ask arbitrary binary-valued queries : i.e., any yes/no questions.
Revised: The aim of this paper is to characterize the learning rates achievable by a learning algorithm in a general interactive setting, where the algorithm can ask arbitrary binary-valued queries : i.e., any yes/no questions.

Original: These include, but are not limited to, label queries and comparison queries for data points, membership queries for any x 2 X , or any other query having a binary answer.
Revised: These include, but are not limited to, label queries and comparison queries for data points, membership queries for any x 2 X , or any other query having a binary answer.

Original: The formal abstraction for this notion of query is provided in Appendix A.1.
Revised: The formal abstraction for this notion of query is provided in Appendix A.1.

Original: We now formalize the notion of achievable learning rates in the universal learning model [BHM + 21].
Revised: We now formalize the notion of achievable learning rates in the universal learning model [BHM + 21].

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: [BHM + 21] Fix a concept class H , and let R : N !
Revised: [BHM + 21] Fix a concept class H , and let R : N !

Original: [0 , 1] , R ( n ) n !
Revised: [0 , 1] , R ( n ) n !

Original: 1 ! 0 be a rate function.
Revised: 1 ! 0 be a rate function.

Original: • H is learnable at rate R if there is a learning algorithm
Revised: • H is learnable at rate R if there is a learning algorithm

Original: ˆ h n such that for every realizable distribution P , there exist distribution-dependent c, C > 0 , for which E [er(ˆ h n )]  CR ( cn ) , 8 n 2 N .
Revised: ˆ h n such that for every realizable distribution P , there exist distribution-dependent c, C > 0 , for which E [er(ˆ h n )]  CR ( cn ) , 8 n 2 N .

Original: • H is not learnable at rate faster than R if for all learning algorithms ˆ h n there exists a realizable distribution P and c, C > 0 , for which E [er(ˆ h n )]  CR ( cn ) , for inﬁnitely many n 2 N .
Revised: • H is not learnable at rate faster than R if for all learning algorithms ˆ h n there exists a realizable distribution P and c, C > 0 , for which E [er(ˆ h n )]  CR ( cn ) , for inﬁnitely many n 2 N .

Original: • H is learnable with optimal rate R if it is learnable at rate R and it is not learnable at rate faster than R .
Revised: • H is learnable with optimal rate R if it is learnable at rate R and it is not learnable at rate faster than R .

Original: • H admits arbitrarily fast rates if for all rate functions R , it is learnable at rate R .
Revised: • H admits arbitrarily fast rates if for all rate functions R , it is learnable at rate R .

Original: • H requires arbitrarily slow rates if for all rate functions R , it is not learnable at rate faster than R .
Revised: • H requires arbitrarily slow rates if for all rate functions R , it is not learnable at rate faster than R .

Original: It is known that, unlike the PAC model, in the universal learning setting every concept class H is learnable [Han21, HKSW21, BHM + 21].
Revised: It is known that, unlike the PAC model, in the universal learning setting every concept class H is learnable [Han21, HKSW21, BHM + 21].

Original: Nevertheless, H might require arbitrarily slow learning rates in some cases [DGL96].
Revised: Nevertheless, H might require arbitrarily slow learning rates in some cases [DGL96].

Original: While most of the above deﬁnitions parallel the deﬁnitions from the work of [BHM + 21] on supervised learning, the case of arbitrarily fast rates is new to the present work.
Revised: While most of the above deﬁnitions parallel the deﬁnitions from the work of [BHM + 21] on supervised learning, the case of arbitrarily fast rates is new to the present work.

Original: While technically it would also arise in passive learning in the trivial cases of | H | = 1 or
Revised: While technically it would also arise in passive learning in the trivial cases of | H | = 1 or

Original: H =
Revised: H =

Original: { h, 1  h } , it arises as a highly non-trivial and important case in interactive learning, with many interesting classes, even of inﬁnite cardinality.
Revised: { h, 1  h } , it arises as a highly non-trivial and important case in interactive learning, with many interesting classes, even of inﬁnite cardinality.

Original: Moreover, our proofs establishing arbitrarily fast rates develop a novel active learning technique (based purely on label queries).
Revised: Moreover, our proofs establishing arbitrarily fast rates develop a novel active learning technique (based purely on label queries).

Original: We are now ready to state one of the main results of this work: a fundamental trichotomy of optimal learning rates.
Revised: We are now ready to state one of the main results of this work: a fundamental trichotomy of optimal learning rates.

Original: That is, we show there are three possible optimal learning rates that a class H can admit in the interactive learning setting.
Revised: That is, we show there are three possible optimal learning rates that a class H can admit in the interactive learning setting.

Original: Theorem 1.
Revised: Theorem 1.

Original: For every concept class H , exactly one of the following cases holds.
Revised: For every concept class H , exactly one of the following cases holds.

Original: • H is interactively learnable with arbitrarily fast rates.
Revised: • H is interactively learnable with arbitrarily fast rates.

Original: • H is interactively learnable at an optimal rate e  n .
Revised: • H is interactively learnable at an optimal rate e  n .

Original: • H requires arbitrarily slow rates for interactive learning.
Revised: • H requires arbitrarily slow rates for interactive learning.

Original: Our next result characterizes exactly when these rates occur by specifying combinatorial measures of H that give rise to each one of these cases.
Revised: Our next result characterizes exactly when these rates occur by specifying combinatorial measures of H that give rise to each one of these cases.

Original: Before we state this result, we need to discuss these combinatorial measures.
Revised: Before we state this result, we need to discuss these combinatorial measures.

Original: We refer the reader to Appendix A.2 for the formal deﬁnitions.
Revised: We refer the reader to Appendix A.2 for the formal deﬁnitions.

Original: We ﬁrst describe inﬁnite Littlestone trees , which were introduced in [BHM + 21].
Revised: We ﬁrst describe inﬁnite Littlestone trees , which were introduced in [BHM + 21].

Original: (Finite versions were studied previously by [Lit88]).
Revised: (Finite versions were studied previously by [Lit88]).

Original: Deﬁnition 2 (Informal (see Deﬁnition 5)) .
Revised: Deﬁnition 2 (Informal (see Deﬁnition 5)) .

Original: A Littlestone tree for H ✓ { 0 , 1 } X is a complete binary tree of depth
Revised: A Littlestone tree for H ✓ { 0 , 1 } X is a complete binary tree of depth

Original: d  1 whose nodes are labeled by elements of X and the edges to the left, right child are labeled by 0 , 1 , respectively.
Revised: d  1 whose nodes are labeled by elements of X and the edges to the left, right child are labeled by 0 , 1 , respectively.

Original: We require that for every level n < d and every path from the root to a node at level n + 1 there is some h 2 H that realizes this path.
Revised: We require that for every level n < d and every path from the root to a node at level n + 1 there is some h 2 H that realizes this path.

Original: We say that H has an inﬁnite
Revised: We say that H has an inﬁnite

Original: Littlestone tree if it has a Littlestone tree of depth d
Revised: Littlestone tree if it has a Littlestone tree of depth d

Original: = 1 .
Revised: = 1 .

Original: For some intuition, we refer the reader to Figure 1a.
Revised: For some intuition, we refer the reader to Figure 1a.

Original: We remark that this notion is closely related to the Littlestone dimension of H .
Revised: We remark that this notion is closely related to the Littlestone dimension of H .

Original: The Littlestone dimension is deﬁned to be the largest d 2 N such that H has a Littlestone tree of depth d and it is 1 if there are Littlestone trees of arbitrary depth.
Revised: The Littlestone dimension is deﬁned to be the largest d 2 N such that H has a Littlestone tree of depth d and it is 1 if there are Littlestone trees of arbitrary depth.

Original: However, this does not necessarily imply that there is a single tree whose depth is inﬁnite, so having inﬁnite Littlestone dimension is not the same as having an inﬁnite Littlestone tree.
Revised: However, this does not necessarily imply that there is a single tree whose depth is inﬁnite, so having inﬁnite Littlestone dimension is not the same as having an inﬁnite Littlestone tree.

Original: We next discuss the notion of a VCL tree introduced by [BHM + 21].
Revised: We next discuss the notion of a VCL tree introduced by [BHM + 21].

Original: Deﬁnition 3 (Informal (see Deﬁnition 6)) .
Revised: Deﬁnition 3 (Informal (see Deﬁnition 6)) .

Original: A VCL tree for H ✓ { 0 , 1 } X is a complete tree of depth d  1 such that every level 0  n < d has nodes that are labeled by X n +1 with branching factor 2 n +1 and whose 2 n +1 edges connecting a node to its children are labeled by the elements of { 0 , 1 } n +1 .
Revised: A VCL tree for H ✓ { 0 , 1 } X is a complete tree of depth d  1 such that every level 0  n < d has nodes that are labeled by X n +1 with branching factor 2 n +1 and whose 2 n +1 edges connecting a node to its children are labeled by the elements of { 0 , 1 } n +1 .

Original: We require that for every node at any ﬁnite level 1  n  d, the path from the root to this node is realized by some h 2 H .
Revised: We require that for every node at any ﬁnite level 1  n  d, the path from the root to this node is realized by some h 2 H .

Original: We say that H has an inﬁnite VCL tree if it has a VCL tree of depth d = 1 .
Revised: We say that H has an inﬁnite VCL tree if it has a VCL tree of depth d = 1 .

Original: For a pictorial representation of a VCL tree, we refer the reader to Figure 1b.
Revised: For a pictorial representation of a VCL tree, we refer the reader to Figure 1b.

Original: Intuitively, the difference between a Littlestone tree and a VCL tree is that in the latter, the size of the nodes is increasing
Revised: Intuitively, the difference between a Littlestone tree and a VCL tree is that in the latter, the size of the nodes is increasing

Original: 0 x ???
Revised: 0 x ???

Original: xxxxxx

Revised: xxxxxx

Original: h 2 H : h ( x ? )
Revised: h 2 H : h ( x ? )

Original: =
Revised: =

Original: h ( x 1 )
Revised: h ( x 1 )

Original: =
Revised: =

Original: h ( x 10 ) =(a) 000 001 0 1 0100 101 110  00 01 1 11 10 x ???
Revised: h ( x 10 ) =(a) 000 001 0 1 0100 101 110  00 01 1 11 10 x ???

Original: ( x 00 ,x 10 ) ( x 01 ,x 11 ) ( x 01 , 01 ,x 11 , 01 ,x 21 , 01 ) ( x 01 , 10 ,x 11 , 10 ,x 21 , 10 ) ( x 01 , 11 ,x 11 , 11 ,x 21 , 11 ) ( x 01 , 00 ,x 11 , 00 ,x 21 , 00 )
Revised: ( x 00 ,x 10 ) ( x 01 ,x 11 ) ( x 01 , 01 ,x 11 , 01 ,x 21 , 01 ) ( x 01 , 10 ,x 11 , 10 ,x 21 , 10 ) ( x 01 , 11 ,x 11 , 11 ,x 21 , 11 ) ( x 01 , 00 ,x 11 , 00 ,x 21 , 00 )

Original: ( x 00 , 00 ,x 10 , 00 ,x 20 , 00 ) ( x 00 , 01 ,x 10 , 01 ,x 20 , 01 ) ( x 00 , 10 ,x 10 , 10 ,x 20 , 10 ) ( x 00 , 11 ,x 10 , 11 ,x 20 , 11 ) 

Revised: ( x 00 , 00 ,x 10 , 00 ,x 20 , 00 ) ( x 00 , 01 ,x 10 , 01 ,x 20 , 01 ) ( x 00 , 10 ,x 10 , 10 ,x 20 , 10 ) ( x 00 , 11 ,x 10 , 11 ,x 20 , 11 )

Original: h 2 H h ( x ? )
Revised: h 2 H h ( x ? )

Original: = 1 h ( x 01 )
Revised: = 1 h ( x 01 )

Original: = 0 , h ( x 11 )
Revised: = 0 , h ( x 11 )

Original: =
Revised: =

Original: h ( x 01 , 00 )
Revised: h ( x 01 , 00 )

Original: = 1 , h ( x 11 , 00 )
Revised: = 1 , h ( x 11 , 00 )

Original: = 0 , h ( x 21 , 00 ) =(b) linearly with the depth of the tree and the branching factor is increasing exponentially.
Revised: = 0 , h ( x 21 , 00 ) =(b) linearly with the depth of the tree and the branching factor is increasing exponentially.

Original: Recall that in a Littlestone tree both quantities are constant.
Revised: Recall that in a Littlestone tree both quantities are constant.

Original: Intuitively, as we move further down the VCL tree along any path, the class H needs to be able to shatter sets of increasing size, while respecting label constraints imposed by all previous levels of the path.
Revised: Intuitively, as we move further down the VCL tree along any path, the class H needs to be able to shatter sets of increasing size, while respecting label constraints imposed by all previous levels of the path.

Original: We now state our characterization on the optimal rates achievable by interactive learning algorithms.
Revised: We now state our characterization on the optimal rates achievable by interactive learning algorithms.

Original: Theorem 2.
Revised: Theorem 2.

Original: For every concept class H the following hold:
Revised: For every concept class H the following hold:

Original: • If H does not have an inﬁnite Littlestone tree, then it is interactively learnable at an arbitrarily fast rate.
Revised: • If H does not have an inﬁnite Littlestone tree, then it is interactively learnable at an arbitrarily fast rate.

Original: • If H has an inﬁnite Littlestone tree but does not have an inﬁnite VCL tree, then it is interactively learnable with optimal rate e  n .
Revised: • If H has an inﬁnite Littlestone tree but does not have an inﬁnite VCL tree, then it is interactively learnable with optimal rate e  n .

Original: • If H has an inﬁnite VCL tree, then it requires arbitrarily slow rates.
Revised: • If H has an inﬁnite VCL tree, then it requires arbitrarily slow rates.

Original: The proof of this theorem is divided into two parts.
Revised: The proof of this theorem is divided into two parts.

Original: In Appendix B we provide the bounds relating to
Revised: In Appendix B we provide the bounds relating to

Original: Littlestone trees and in Appendix C
Revised: Littlestone trees and in Appendix C

Original: we prove the bounds relating to VCL trees.
Revised: we prove the bounds relating to VCL trees.

Original: It is easy to see that
Revised: It is easy to see that

Original: Theorem 2 implies Theorem 1.
Revised: Theorem 2 implies Theorem 1.

Original: We remark that when the learner has access to P X and not just to an inﬁnite stream of i.i.d.
Revised: We remark that when the learner has access to P X and not just to an inﬁnite stream of i.i.d.

Original: data from it, then we can improve the ﬁrst rate of the trichotomy.
Revised: data from it, then we can improve the ﬁrst rate of the trichotomy.

Original: Remark 1.
Revised: Remark 1.

Original: If the learner has access to the marginal distribution P X and H does not have an inﬁnite
Revised: If the learner has access to the marginal distribution P X and H does not have an inﬁnite

Original: Littlestone tree, then it can achieve E [er(ˆ h n )]
Revised: Littlestone tree, then it can achieve E [er(ˆ h n )]

Original: = 0 , after a ﬁnite ( P -dependent) number n 2 N .
Revised: = 0 , after a ﬁnite ( P -dependent) number n 2 N .

Original: As mentioned above, the algorithm which achieves the arbitrarily fast rates requires only label queries, which are, arguably, the simplest type of queries: that is, queries that select any unlabeled example
Revised: As mentioned above, the algorithm which achieves the arbitrarily fast rates requires only label queries, which are, arguably, the simplest type of queries: that is, queries that select any unlabeled example

Original: X i from the i.i.d.
Revised: X i from the i.i.d.

Original: stream and request the target concept’s label for X i .
Revised: stream and request the target concept’s label for X i .

Original: Hence, this result applies to the traditional active learning setting.
Revised: Hence, this result applies to the traditional active learning setting.

Original: Remark 2.
Revised: Remark 2.

Original: If H does not have an inﬁnite Littlestone tree, then there exists an algorithm that achieves arbitrarily fast rates using only label queries.
Revised: If H does not have an inﬁnite Littlestone tree, then there exists an algorithm that achieves arbitrarily fast rates using only label queries.

Original: Interactive vs. Passive Learning.
Revised: Interactive vs. Passive Learning.

Original: We remark that our results can be directly compared to the optimal universal rates for passive supervised learning, as characterized by [BHM + 21].
Revised: We remark that our results can be directly compared to the optimal universal rates for passive supervised learning, as characterized by [BHM + 21].

Original: Speciﬁcally, [BHM + 21] provide the following characterization for the optimal rates achievable by a learner that has access to n labeled examples that are drawn i.i.d.
Revised: Speciﬁcally, [BHM + 21] provide the following characterization for the optimal rates achievable by a learner that has access to n labeled examples that are drawn i.i.d.

Original: from P .
Revised: from P .

Original: Theorem 3.
Revised: Theorem 3.

Original: [BHM + 21]
Revised: [BHM + 21]

Original: For every concept class H (with | H |  3 ) the following hold:
Revised: For every concept class H (with | H |  3 ) the following hold:

Original: • If H does not have an inﬁnite Littlestone tree, then it is passively learnable with optimal rate e  n • If H has an inﬁnite Littlestone tree but does not have an inﬁnite VCL tree, then it is passively learnable with optimal rate 1 n .
Revised: • If H does not have an inﬁnite Littlestone tree, then it is passively learnable with optimal rate e  n • If H has an inﬁnite Littlestone tree but does not have an inﬁnite VCL tree, then it is passively learnable with optimal rate 1 n .

Original: • If H has an inﬁnite VCL tree, then it requires arbitrarily slow rates.
Revised: • If H has an inﬁnite VCL tree, then it requires arbitrarily slow rates.

Original: Our results illustrate the power of interactive learning algorithms compared to their passive counterparts.
Revised: Our results illustrate the power of interactive learning algorithms compared to their passive counterparts.

Original: Notably, if H does not have an inﬁnite Littlestone tree, the rate is improved from exponential to arbitrarily fast; again, this is even achieved by an algorithm based purely on label queries.
Revised: Notably, if H does not have an inﬁnite Littlestone tree, the rate is improved from exponential to arbitrarily fast; again, this is even achieved by an algorithm based purely on label queries.

Original: Moreover, using an interactive learning algorithm we achieve an exponential improvement in the case where H does not have an inﬁnite VCL tree but has an inﬁnite Littlestone tree: reducing a 1 n rate for passive learning to an e  n rate for interactive learning.
Revised: Moreover, using an interactive learning algorithm we achieve an exponential improvement in the case where H does not have an inﬁnite VCL tree but has an inﬁnite Littlestone tree: reducing a 1 n rate for passive learning to an e  n rate for interactive learning.

Original: Finally, when H has an inﬁnite VCL tree we cannot achieve any improvement even in the interactive setting.
Revised: Finally, when H has an inﬁnite VCL tree we cannot achieve any improvement even in the interactive setting.

Original: Another important comparison is between the rates achievable in the universal vs uniform setting.
Revised: Another important comparison is between the rates achievable in the universal vs uniform setting.

Original: Speciﬁcally, results of [KMT93] imply that for interactive learning with arbitrary binary-valued queries, the optimal uniform rate is (i) 0 after a bounded number of queries if | H | < 1 , (ii) e  n when | H | = 1 but the VC dimension of H is ﬁnite, or (iii) does not converge to 0 when the VC dimension of H is inﬁnite.
Revised: Speciﬁcally, results of [KMT93] imply that for interactive learning with arbitrary binary-valued queries, the optimal uniform rate is (i) 0 after a bounded number of queries if | H | < 1 , (ii) e  n when | H | = 1 but the VC dimension of H is ﬁnite, or (iii) does not converge to 0 when the VC dimension of H is inﬁnite.

Original: Comparing to our result in Theorem 2, we note that the optimal universal rates allow richer families in each of the cases: that is, the category of any H in the universal rates trichotomy is never worse than its category in the uniform rates trichotomy, and is often better.
Revised: Comparing to our result in Theorem 2, we note that the optimal universal rates allow richer families in each of the cases: that is, the category of any H in the universal rates trichotomy is never worse than its category in the uniform rates trichotomy, and is often better.

Original: For instance, ﬁnite VC dimension certainly implies there is no inﬁnite VCL tree, but the opposite is not true, and indeed there are classes of inﬁnite VC dimension which do not even have an inﬁnite Littlestone tree [BHM + 21], so that the uniform rates are vaccuous while the universal rates are arbitrarily fast.
Revised: For instance, ﬁnite VC dimension certainly implies there is no inﬁnite VCL tree, but the opposite is not true, and indeed there are classes of inﬁnite VC dimension which do not even have an inﬁnite Littlestone tree [BHM + 21], so that the uniform rates are vaccuous while the universal rates are arbitrarily fast.

Original: 1.2 Examples
Revised: 1.2 Examples

Original: Here we present examples of various hypothesis classes that illustrate the three possible optimal rates.
Revised: Here we present examples of various hypothesis classes that illustrate the three possible optimal rates.

Original: Example 1 (Finite classes) .
Revised: Example 1 (Finite classes) .

Original: Let H be a ﬁnite hypothesis class over some domain X .
Revised: Let H be a ﬁnite hypothesis class over some domain X .

Original: It follows immediately that this class has no inﬁnite Littlestone tree.
Revised: It follows immediately that this class has no inﬁnite Littlestone tree.

Original: Thus, arbitrarily fast rates are achievable.
Revised: Thus, arbitrarily fast rates are achievable.

Original: Recall that for passive learning, the optimal rate was e  n for this scenario.
Revised: Recall that for passive learning, the optimal rate was e  n for this scenario.

Original: This illustrates the improvement compared to the passive universal learning setting, where the best achievable universal rate is exponential [BHM + 21].
Revised: This illustrates the improvement compared to the passive universal learning setting, where the best achievable universal rate is exponential [BHM + 21].

Original: We then consider threshold classiﬁers over the natural numbers.
Revised: We then consider threshold classiﬁers over the natural numbers.

Original: Example 2 (Threshold classiﬁers over N ) .
Revised: Example 2 (Threshold classiﬁers over N ) .

Original: Let X =
Revised: Let X =

Original: N and consider H
Revised: N and consider H

Original: = { h t : t 2 N , h t
Revised: = { h t : t 2 N , h t

Original: = 1 x  t } .
Revised: = 1 x  t } .

Original: Notice that even though there are Littlestone trees of arbitrary depth, there is no inﬁnite Littlestone tree [BHM + 21].
Revised: Notice that even though there are Littlestone trees of arbitrary depth, there is no inﬁnite Littlestone tree [BHM + 21].

Original: Hence, it is learnable at an abitrarily fast rate in the interactive setting.
Revised: Hence, it is learnable at an abitrarily fast rate in the interactive setting.

Original: In the next example we consider threshold classiﬁers over the real numbers.
Revised: In the next example we consider threshold classiﬁers over the real numbers.

Original: Example 3 (Threshold classiﬁers over R . ) .
Revised: Example 3 (Threshold classiﬁers over R . ) .

Original: Let X =
Revised: Let X =

Original: R and H be the class of all threshold classiﬁers over the real line, i.e. H = { h t : t 2 R , h t = 1 x  t } .
Revised: R and H be the class of all threshold classiﬁers over the real line, i.e. H = { h t : t 2 R , h t = 1 x  t } .

Original: This class has an inﬁnite Littlestone tree.
Revised: This class has an inﬁnite Littlestone tree.

Original: However, since its VC dimension is ﬁnite, we know there is no inﬁnite VCL tree, and hence the optimal rate is e  n .
Revised: However, since its VC dimension is ﬁnite, we know there is no inﬁnite VCL tree, and hence the optimal rate is e  n .

Original: Next, an example with inﬁnite VC dimension, which nevertheless is interactively learnable at an arbitrarily fast rate:
Revised: Next, an example with inﬁnite VC dimension, which nevertheless is interactively learnable at an arbitrarily fast rate:

Original: Example 4 (Unions of ﬁnite sets.) .
Revised: Example 4 (Unions of ﬁnite sets.) .

Original: Let X =
Revised: Let X =

Original: [ k X k be the disjoint union of the sets X k , where |X k
Revised: [ k X k be the disjoint union of the sets X k , where |X k

Original: | =
Revised: | =

Original: k .
Revised: k .

Original: We also let H =
Revised: We also let H =

Original: [ k H k , where H k = { 1 S : S ✓ X k } .
Revised: [ k H k , where H k = { 1 S : S ✓ X k } .

Original: The VC dimension of H is inﬁnite.
Revised: The VC dimension of H is inﬁnite.

Original: However, notice that this class does not have an inﬁnite Littlestone tree [BHM + 21].
Revised: However, notice that this class does not have an inﬁnite Littlestone tree [BHM + 21].

Original: Indeed, let x 2 X k be the root of the tree.
Revised: Indeed, let x 2 X k be the root of the tree.

Original: Then, only h 2 H k can have h ( x ) = 1 .
Revised: Then, only h 2 H k can have h ( x ) = 1 .

Original: Thus, there are ﬁnitely many hypotheses consistent with this branch from the root, implying the depth is always limited by the choice of root node.
Revised: Thus, there are ﬁnitely many hypotheses consistent with this branch from the root, implying the depth is always limited by the choice of root node.

Original: Thus, this class is actively learnable at an arbitrarily fast rate.
Revised: Thus, this class is actively learnable at an arbitrarily fast rate.

Original: Finally, we present an example that requires arbitrarily slow rates.
Revised: Finally, we present an example that requires arbitrarily slow rates.

Original: Example 5.
Revised: Example 5.

Original: Let H be the class of all measurable functions over X = N .
Revised: Let H be the class of all measurable functions over X = N .

Original: This class admits an inﬁnite
Revised: This class admits an inﬁnite

Original: VCL tree [BHM + 21], hence the learning rate in our setting is arbitrarily slow.
Revised: VCL tree [BHM + 21], hence the learning rate in our setting is arbitrarily slow.

Original: 1.3 Techniques
Revised: 1.3 Techniques

Original: In this section we brieﬂy highlight some techniques that are important for our results.
Revised: In this section we brieﬂy highlight some techniques that are important for our results.

Original: For a more extensive discussion, we refer the reader to Appendix A.3.
Revised: For a more extensive discussion, we refer the reader to Appendix A.3.

Original: Gale-Stewart Games.
Revised: Gale-Stewart Games.

Original: An important tool that our algorithms build upon is the theory of Gale-Stewart games [GS53].
Revised: An important tool that our algorithms build upon is the theory of Gale-Stewart games [GS53].

Original: These games consist of two players, a learner P L and an adversary P A , that interact over an inﬁnite sequence of discrete timesteps.
Revised: These games consist of two players, a learner P L and an adversary P A , that interact over an inﬁnite sequence of discrete timesteps.

Original: In every round t  1 , the adversary presents an element x t 2 X t to the learner and, subsequently, she picks an element y t 2 Y t , where X t , Y t are some sets.
Revised: In every round t  1 , the adversary presents an element x t 2 X t to the learner and, subsequently, she picks an element y t 2 Y t , where X t , Y t are some sets.

Original: If some predetermined condition is met at some ﬁnite round t 2 N (e.g. a set of functions that are consistent with the choices of the players becomes empty), then player P L wins.
Revised: If some predetermined condition is met at some ﬁnite round t 2 N (e.g. a set of functions that are consistent with the choices of the players becomes empty), then player P L wins.

Original: If the interactions repeat for an inﬁnite number of steps, then player P A wins.
Revised: If the interactions repeat for an inﬁnite number of steps, then player P A wins.

Original: Thus, we can see that the winning strategy of P L is ﬁnitely decidable , which means that the learner knows she has won the game after a ﬁnite number of steps.
Revised: Thus, we can see that the winning strategy of P L is ﬁnitely decidable , which means that the learner knows she has won the game after a ﬁnite number of steps.

Original: The fundamental property that characterizes these games is that either the learner or the adversary has a winning strategy, i.e., playing according to this strategy makes them win the game irrespective of their opponent’s moves [GS53, HW + 93, Kec12].
Revised: The fundamental property that characterizes these games is that either the learner or the adversary has a winning strategy, i.e., playing according to this strategy makes them win the game irrespective of their opponent’s moves [GS53, HW + 93, Kec12].

Original: In this work, we use
Revised: In this work, we use

Original: Gale-Stewart games where P A chooses subsets of data and P L chooses labels for the points, such that the possible play-outs correspond to either Littlestone or VCL trees.
Revised: Gale-Stewart games where P A chooses subsets of data and P L chooses labels for the points, such that the possible play-outs correspond to either Littlestone or VCL trees.

Original: One-Inclusion Graph Predictor [HLW94].
Revised: One-Inclusion Graph Predictor [HLW94].

Original: Another important component that we use is the oneinclusion graph predictor.
Revised: Another important component that we use is the oneinclusion graph predictor.

Original: This is a passive learning algorithm that takes as input n labeled points that are drawn i.i.d.
Revised: This is a passive learning algorithm that takes as input n labeled points that are drawn i.i.d.

Original: from P and an unlabeled example drawn from P X and returns the correct label with probability 1  O (d /n ) , where d is the VC dimension of a given function class F for which P is realizable; indeed, F may in fact be any partial concept class (see below).
Revised: from P and an unlabeled example drawn from P X and returns the correct label with probability 1  O (d /n ) , where d is the VC dimension of a given function class F for which P is realizable; indeed, F may in fact be any partial concept class (see below).

Original: Partial Concept Classes.
Revised: Partial Concept Classes.

Original: The ﬁnal building block that we use is a subroutine that interactively learns partial concept classes.
Revised: The ﬁnal building block that we use is a subroutine that interactively learns partial concept classes.

Original: A partial concept class F is a set of partial functions f : X ! { 0 , 1 , ? } , where f ( x ) = ?
Revised: A partial concept class F is a set of partial functions f : X ! { 0 , 1 , ? } , where f ( x ) = ?

Original: means that f is undeﬁned at point x .
Revised: means that f is undeﬁned at point x .

Original: Essentially, partial concepts provide a framework to express data-dependent assumptions [AHHM21].
Revised: Essentially, partial concepts provide a framework to express data-dependent assumptions [AHHM21].

Original: We design an algorithm that achieves exponential rates when learning a partial concept class F whose VC dimension is bounded (cf.
Revised: We design an algorithm that achieves exponential rates when learning a partial concept class F whose VC dimension is bounded (cf.

Original: Appendix C.2).
Revised: Appendix C.2).

Original: In our proofs of the universal rates, this class F is actually constructed algorithmically given a class
Revised: In our proofs of the universal rates, this class F is actually constructed algorithmically given a class

Original: H and the i.i.d.
Revised: H and the i.i.d.

Original: unlabeled data.
Revised: unlabeled data.

Original:  Overview of the Proofs of the Main Result
Revised:  Overview of the Proofs of the Main Result

Original: In this section we provide an outline of the proof of the main results.
Revised: In this section we provide an outline of the proof of the main results.

Original: We refer the reader to
Revised: We refer the reader to

Original: Appendix B, C for the full proofs.
Revised: Appendix B, C for the full proofs.

Original: Our lower bounds rely on establishing a connection between paths in the inﬁnite tree and a family of possible target concepts.
Revised: Our lower bounds rely on establishing a connection between paths in the inﬁnite tree and a family of possible target concepts.

Original: Since the paths can be speciﬁed by binary strings, our lower bound results follow from lower bounds we prove for lossy coding, representing a universal variant of rate-distortion theory.
Revised: Since the paths can be speciﬁed by binary strings, our lower bound results follow from lower bounds we prove for lossy coding, representing a universal variant of rate-distortion theory.

Original: We leave the details for the appendix, and spend the remainder of this section on outlining the upper bounds, to focus on the novel algorithmic aspects of this work.
Revised: We leave the details for the appendix, and spend the remainder of this section on outlining the upper bounds, to focus on the novel algorithmic aspects of this work.

Original: Arbitrarily Fast Rates.
Revised: Arbitrarily Fast Rates.

Original: We ﬁrst design an algorithm that achieves arbitrarily fast rates whenever H does not have an inﬁnite Littlestone tree.
Revised: We ﬁrst design an algorithm that achieves arbitrarily fast rates whenever H does not have an inﬁnite Littlestone tree.

Original: A key subroutine of our algorithm is the ordinal Standard
Revised: A key subroutine of our algorithm is the ordinal Standard

Original: Optimal Algorithm (SOA)
Revised: Optimal Algorithm (SOA)

Original: [BHM + 21].
Revised: [BHM + 21].

Original: Essentially, this algorithm generalizes the SOA [Lit88] to handle the case of ﬁnite, but not uniformly bounded, Littlestone trees.
Revised: Essentially, this algorithm generalizes the SOA [Lit88] to handle the case of ﬁnite, but not uniformly bounded, Littlestone trees.

Original: For a more extensive discussion we kindly refer to Appendix B. Importantly, when this algorithm is executed on a (realizable) 2 sequence of labeled data ( x 1 , y 1 , x 2 , y 2 , . . . )
Revised: For a more extensive discussion we kindly refer to Appendix B. Importantly, when this algorithm is executed on a (realizable) 2 sequence of labeled data ( x 1 , y 1 , x 2 , y 2 , . . . )

Original: it will stop making mistakes after a ﬁnite time t ⇤ .
Revised: it will stop making mistakes after a ﬁnite time t ⇤ .

Original: However, t ⇤ depends on the sequence and there is no bound on how big it is.
Revised: However, t ⇤ depends on the sequence and there is no bound on how big it is.

Original: Our approach that achieves the arbitrarily fast rates is outlined in Figure 2.
Revised: Our approach that achieves the arbitrarily fast rates is outlined in Figure 2.

Original: The intuition behind our algorithm is the following.
Revised: The intuition behind our algorithm is the following.

Original: If we take a large enough number of unlabeled points m 1 and consider all of their 2 m 1 possible classiﬁcations, there will be one that is correct.
Revised: If we take a large enough number of unlabeled points m 1 and consider all of their 2 m 1 possible classiﬁcations, there will be one that is correct.

Original: Let S ⇤ 1 be the correctly labeled set.
Revised: Let S ⇤ 1 be the correctly labeled set.

Original: For large enough m 1 , we know that the ordinal SOA trained on S ⇤ 1 will output a classiﬁer that is always correct.
Revised: For large enough m 1 , we know that the ordinal SOA trained on S ⇤ 1 will output a classiﬁer that is always correct.

Original: The issues are that (i) we do not have access to the correct classiﬁcation, and (ii) we do not know how large m 1 needs to be.
Revised: The issues are that (i) we do not have access to the correct classiﬁcation, and (ii) we do not know how large m 1 needs to be.

Original: Since the data are generated by some distribution, the time of the last mistake also follows a distribution which can, potentially, have heavy tails.
Revised: Since the data are generated by some distribution, the time of the last mistake also follows a distribution which can, potentially, have heavy tails.

Original: However, there is some (distribution-dependent) number m ⇤ 2 N for which, when the ordinal SOA is trained on a sequence of size m ⇤ , with probability at least 1 / 2 , it will output a classiﬁer that is correct on every point x 2 X .
Revised: However, there is some (distribution-dependent) number m ⇤ 2 N for which, when the ordinal SOA is trained on a sequence of size m ⇤ , with probability at least 1 / 2 , it will output a classiﬁer that is correct on every point x 2 X .

Original: Thus, if we consider a lot of batches that have size at least m ⇤ and train the ordinal SOA using these batches, at least 1 / 3 of the executions will produce an always correct classiﬁer, with high probability.
Revised: Thus, if we consider a lot of batches that have size at least m ⇤ and train the ordinal SOA using these batches, at least 1 / 3 of the executions will produce an always correct classiﬁer, with high probability.

Original: To overcome the issue that we only have unlabeled data, we consider all the possible classiﬁcations of these batches and run the ordinal
Revised: To overcome the issue that we only have unlabeled data, we consider all the possible classiﬁcations of these batches and run the ordinal

Original: SOA on each one of them.
Revised: SOA on each one of them.

Original: Moreover, we know that all the correct classiﬁers produce the same output when executed on a new set of unlabeled data.
Revised: Moreover, we know that all the correct classiﬁers produce the same output when executed on a new set of unlabeled data.

Original: This is why we use points from S 2 and create equivalence classes of classiﬁers that have the same output on S 2 ; this guarantees that all the correct classiﬁers will be in the same class F ` ⇤ , with probability one.
Revised: This is why we use points from S 2 and create equivalence classes of classiﬁers that have the same output on S 2 ; this guarantees that all the correct classiﬁers will be in the same class F ` ⇤ , with probability one.

Original: By taking S 2 to be large enough, we can make the probability that two classiﬁers which are in the same equivalence class differ on some x 2 X arbitrarily small.
Revised: By taking S 2 to be large enough, we can make the probability that two classiﬁers which are in the same equivalence class differ on some x 2 X arbitrarily small.

Original: Thus, it sufﬁces to consider one representative f ` from every equivalence class F ` .
Revised: Thus, it sufﬁces to consider one representative f ` from every equivalence class F ` .

Original: Finally, we show that there is an ordering of these representatives which guarantees that, with arbitrarily high probability, the representative f ` ⇤ of the class F ` ⇤ (which the correct classiﬁers belong to) has ranking at most i ⇤ , where i ⇤ is a distribution-dependent constant.
Revised: Finally, we show that there is an ordering of these representatives which guarantees that, with arbitrarily high probability, the representative f ` ⇤ of the class F ` ⇤ (which the correct classiﬁers belong to) has ranking at most i ⇤ , where i ⇤ is a distribution-dependent constant.

Original: Thus, by requesting the labels ofthe points on which these f i ,  1  i  i ⇤ , disagree, we can just output the one that is correct on all these points. Hence, if bp n c  i ⇤ we output the classiﬁer f ` ⇤  with arbitrarily high probability.
Revised: Thus, by requesting the labels of points on which pairs f i , f j , 1  i, j  p n, disagree, we can just output the one that is correct on all these points, and if bp n c  i ⇤ , such an f i will exist and will be f ` ⇤ , with arbitrarily high probability.

Original: We emphasize that this algorithm uses only label queries, so it also applies to the traditional active learning setting.
Revised: We emphasize that this algorithm uses only label queries, so it also applies to the traditional active learning setting.

Original: Moreover, if the learner has access to the marginal distribution P X , then all the steps in the algorithm can be implemented exactly, i.e. with probability one, so the output will be a classiﬁer that has zero error rate with probability one.
Revised: Moreover, if the learner has access to the marginal distribution P X , then all the steps in the algorithm can be implemented exactly, i.e. with probability one, so the output will be a classiﬁer that has zero error rate with probability one.

Original: Exponential Rates.
Revised: Exponential Rates.

Original: We now move to the algorithm that achieves exponential rates whenever H does not have an inﬁnite VCL tree.
Revised: We now move to the algorithm that achieves exponential rates whenever H does not have an inﬁnite VCL tree.

Original: As we discussed before, the fact that H does not have an inﬁnite
Revised: As we discussed before, the fact that H does not have an inﬁnite

Original: Littlestone tree gives rise to an algorithm that makes a ﬁnite number of mistakes on any realizable sequence ( x 1 , y 1 , . . . ) .
Revised: Littlestone tree gives rise to an algorithm that makes a ﬁnite number of mistakes on any realizable sequence ( x 1 , y 1 , . . . ) .

Original: In the case of ﬁnite VCL trees, [BHM + 21] provide an algorithm, which given such a sequence, learns after a ﬁnite number of steps to rule out patterns in the data.
Revised: In the case of ﬁnite VCL trees, [BHM + 21] provide an algorithm, which given such a sequence, learns after a ﬁnite number of steps to rule out patterns in the data.

Original: This means that, for some n ⇤ 2 N there is a function g : X n ⇤ !
Revised: This means that, for some n ⇤ 2 N there is a function g : X n ⇤ !

Original: { 0 , 1 } n ⇤ such that g ( x 1 , . . .
Revised: { 0 , 1 } n ⇤ such that g ( x 1 , . . .

Original: , x n ⇤ ) are not the correct labels of the points ( x 1 , . . .
Revised: , x n ⇤ ) are not the correct labels of the points ( x 1 , . . .

Original: , x n ⇤ ) .
Revised: , x n ⇤ ) .

Original: [BHM + 21] obtain this function by deﬁning an appropriate Gale-Stewart game, called the VCL game.
Revised: [BHM + 21] obtain this function by deﬁning an appropriate Gale-Stewart game, called the VCL game.

Original: In every round t  1 of this game, the adversary selects a tuple ( x 0 t , x 1 t , . . .
Revised: In every round t  1 of this game, the adversary selects a tuple ( x 0 t , x 1 t , . . .

Original: , x t  1 t ) 2 X t of t unlabeled points and the learner picks their labels ( y 0 t , y 1 t , . . .
Revised: , x t  1 t ) 2 X t of t unlabeled points and the learner picks their labels ( y 0 t , y 1 t , . . .

Original: , y t  1 t ) 2 { 0 , 1 } t .
Revised: , y t  1 t ) 2 { 0 , 1 } t .

Original: The goal of the learner is to make the set of concepts from H that are consistent with the execution of the game empty.
Revised: The goal of the learner is to make the set of concepts from H that are consistent with the execution of the game empty.

Original: [BHM + 21] show that if H does not have an inﬁnite VCL tree, then the learner has a winning strategy and this strategy gives rise to the pattern avoidance function g we described earlier.
Revised: [BHM + 21] show that if H does not have an inﬁnite VCL tree, then the learner has a winning strategy and this strategy gives rise to the pattern avoidance function g we described earlier.

Original: Importantly, the pattern avoidance function helps us deﬁne a partial concept class whose VC dimension is ﬁnite .
Revised: Importantly, the pattern avoidance function helps us deﬁne a partial concept class whose VC dimension is ﬁnite .

Original: For details, we refer to Appendix C.1.
Revised: For details, we refer to Appendix C.1.

Original: The ﬁrst step in our algorithm, presented in Figure 4, is to use half of the query budget to ﬁnd the labels of b n/ 2 c points in order to simulate the VCL game.
Revised: The ﬁrst step in our algorithm, presented in Figure 4, is to use half of the query budget to ﬁnd the labels of b n/ 2 c points in order to simulate the VCL game.

Original: The eventually correct pattern avoidance function g we described before, induces some data-dependent constraints.
Revised: The eventually correct pattern avoidance function g we described before, induces some data-dependent constraints.

Original: To make the idea of our approach easier to grasp, let us assume that we have access to both the function g and to the number n ⇤ .
Revised: To make the idea of our approach easier to grasp, let us assume that we have access to both the function g and to the number n ⇤ .

Original: Then, the next step is to consider the partial concept class
Revised: Then, the next step is to consider the partial concept class

Original: F ⇤ =
Revised: F ⇤ =

Original: { f : X ! { 0 , 1 , ? } : 8 ( x 1 , . . .
Revised: { f : X ! { 0 , 1 , ? } : 8 ( x 1 , . . .

Original: , x n ⇤ )
Revised: , x n ⇤ )

Original: 2 X n ⇤ , ( f ( x 1 ) , . . .
Revised: 2 X n ⇤ , ( f ( x 1 ) , . . .

Original: , f ( x n ⇤ ))
Revised: , f ( x n ⇤ ))

Original: 6 = g ( x 1 , . . .
Revised: 6 = g ( x 1 , . . .

Original: , x n ⇤ ) } .
Revised: , x n ⇤ ) } .

Original: Importantly, the VC dimension of F ⇤ is bounded by n ⇤  1 , since it does not shatter any sequence of length n ⇤ .
Revised: Importantly, the VC dimension of F ⇤ is bounded by n ⇤  1 , since it does not shatter any sequence of length n ⇤ .

Original: Thus, we have reduced the problem to interactively learning a partial concept class whose
Revised: Thus, we have reduced the problem to interactively learning a partial concept class whose

Original: VC dimension is bounded.
Revised: VC dimension is bounded.

Original: We provide such an algorithm in Figure 3.
Revised: We provide such an algorithm in Figure 3.

Original: PartialInteractiveLearning : Input is an unlabeled data stream { x 1 , . . .
Revised: PartialInteractiveLearning : Input is an unlabeled data stream { x 1 , . . .

Original: , } and a query budget n.
Revised: , } and a query budget n.

Original: Take a large enough sample of unlabeled data S = { x 1 , . . .
Revised: Take a large enough sample of unlabeled data S = { x 1 , . . .

Original: , x m } .
Revised: , x m } .

Original: Let ˆ H | S to be the total concept class of the functions in F that are not undeﬁned on S , i.e.,
Revised: Let ˆ H | S to be the total concept class of the functions in F that are not undeﬁned on S , i.e.,

Original: [Equation]
Revised: [Equation]

Original: whose domain is restricted to be S.
Revised: whose domain is restricted to be S.

Original: ## this is a total class with
Revised: ## this is a total class with

Original: VC dimension bounded by d .
Revised: VC dimension bounded by d .

Original: If ˆ H = ; , return an arbitrary classiﬁer (e
Revised: If ˆ H = ; , return an arbitrary classiﬁer (e

Original: g. the all-zero classiﬁer).
Revised: g. the all-zero classiﬁer).

Original: Consider the set ˆ S of all the O ( m d ) possible classiﬁcations of S. 5.
Revised: Consider the set ˆ S of all the O ( m d ) possible classiﬁcations of S. 5.

Original: Since the learner is allowed to use arbitrary binary queries, it can ﬁgure out the labels of all points using O (d log m ) queries as follows: • Divide ˆ S into two sets of equal size ˆ S 1 , ˆ S 2 and ask whether the true classiﬁcation is in ˆ S 1 .
Revised: Since the learner is allowed to use arbitrary binary queries, it can ﬁgure out the labels of all points using O (d log m ) queries as follows: • Divide ˆ S into two sets of equal size ˆ S 1 , ˆ S 2 and ask whether the true classiﬁcation is in ˆ S 1 .

Original: Based on the answer of the query, recurse to the appropriate set.
Revised: Based on the answer of the query, recurse to the appropriate set.

Original: The last step is to use these m labeled points as the training set for the one-inclusion graph supervised learning method of [HLW94] for the class F .
Revised: The last step is to use these m labeled points as the training set for the one-inclusion graph supervised learning method of [HLW94] for the class F .

Original: The idea of the approach is to take a large enough sample S of m unlabeled points and consider the total concept class H ⇤ ✓ F ⇤ of the functions that are not undeﬁned on S , whose domain restricted is
Revised: The idea of the approach is to take a large enough sample S of m unlabeled points and consider the total concept class H ⇤ ✓ F ⇤ of the functions that are not undeﬁned on S , whose domain restricted is

Original: S .
Revised: S .

Original: Then, H ⇤ also has VC dimension that is bounded by n ⇤  1 , so there are at most O ( m n ⇤ ) possible classiﬁcations of these points.
Revised: Then, H ⇤ also has VC dimension that is bounded by n ⇤  1 , so there are at most O ( m n ⇤ ) possible classiﬁcations of these points.

Original: The next step is to ﬁnd the correct classiﬁcation of these points using
Revised: The next step is to ﬁnd the correct classiﬁcation of these points using

Original: O ( n ⇤ log m ) binary queries in total.
Revised: O ( n ⇤ log m ) binary queries in total.

Original: Finally, using these exponentially many labeled points as the training set for the one-inclusion graph algorithm (cf.
Revised: Finally, using these exponentially many labeled points as the training set for the one-inclusion graph algorithm (cf.

Original: Theorem 5), we are able to get a classiﬁer whose error rate decreases exponentially fast.
Revised: Theorem 5), we are able to get a classiﬁer whose error rate decreases exponentially fast.

Original: An important difﬁculty we need to overcome is that we do not know how long we need to run the
Revised: An important difﬁculty we need to overcome is that we do not know how long we need to run the

Original: VCL game for in order to get a correct pattern avoidance function g .
Revised: VCL game for in order to get a correct pattern avoidance function g .

Original: Our approach, described in
Revised: Our approach, described in

Original: Figure 5, is to use N = b
Revised: Figure 5, is to use N = b

Original: n/ 2 c of our query budget to request the labels of some points in order to run the VCL game on.
Revised: n/ 2 c of our query budget to request the labels of some points in order to run the VCL game on.

Original: Then, utilizing a result from [BHM + 21], we use b N/ 2 c of the data in order to obtain some estimators ˆ t N such that running the game on ˆ t N points produces a correct g with probability at least 5 / 8 (cf.
Revised: Then, utilizing a result from [BHM + 21], we use b N/ 2 c of the data in order to obtain some estimators ˆ t N such that running the game on ˆ t N points produces a correct g with probability at least 5 / 8 (cf.

Original: Lemma 5).
Revised: Lemma 5).

Original: Having obtained these estimators, we split the remaining b N/ 2 c into ˆ N = b N/ (2 ˆ t N ) c batches of size ˆ t N .
Revised: Having obtained these estimators, we split the remaining b N/ 2 c into ˆ N = b N/ (2 ˆ t N ) c batches of size ˆ t N .

Original: Subsequently, we run the VCL game on each batch and obtain the functions
Revised: Subsequently, we run the VCL game on each batch and obtain the functions

Original: ˆ
Revised: ˆ

Original: y i
Revised: y i

Original: ˆ N , 1  i  b N/ (2ˆ t N ) c .
Revised: ˆ N , 1  i  b N/ (2ˆ t N ) c .

Original: The choice of ˆ t n guarantees that, with high probability, at least 9 / 16 of these functions will be correct pattern avoidance functions.
Revised: The choice of ˆ t n guarantees that, with high probability, at least 9 / 16 of these functions will be correct pattern avoidance functions.

Original: Then, we consider the partial concept classes F i that are induced by these functions, in a similar manner as we described before.
Revised: Then, we consider the partial concept classes F i that are induced by these functions, in a similar manner as we described before.

Original: Finally, we aggregate all these classes into a majority class
Revised: Finally, we aggregate all these classes into a majority class

Original: F ˆ Nm = ⇢ f : X ! { 0 , 1 , ? } : 8 ` 2 N , 8 ( x 1 , . .
Revised: F ˆ Nm = ⇢ f : X ! { 0 , 1 , ? } : 8 ` 2 N , 8 ( x 1 , . .

Original: , x ` ) 2 X ` , 9 at least (9 / 16) ˆ N classes F j , 1  j  ˆ N s.t.
Revised: , x ` ) 2 X ` , 9 at least (9 / 16) ˆ N classes F j , 1  j  ˆ N s.t.

Original: 9 ˆ f 2 F j with ( ˆ f ( x 1 ) , . .
Revised: 9 ˆ f 2 F j with ( ˆ f ( x 1 ) , . .

Original: , ˆ f ( x ` ))
Revised: , ˆ f ( x ` ))

Original: =
Revised: =

Original: ( f ( x 1 ) , . . . ,
Revised: ( f ( x 1 ) , . . . ,

Original: f ( x ` ))  .
Revised: f ( x ` ))  .

Original: The motivation behind aggregating the classes is that (i) we can show that F ˆ Nm has bounded VC dimension, and (ii) if F ˆ N m cannot produce a labeling for an unlabeled tuple, then this is not the correct labeling (cf.
Revised: The motivation behind aggregating the classes is that (i) we can show that F ˆ Nm has bounded VC dimension, and (ii) if F ˆ N m cannot produce a labeling for an unlabeled tuple, then this is not the correct labeling (cf.

Original: Lemma 7).
Revised: Lemma 7).

Original: An equivalent way to describe F ˆ Nm is through a sequence of universally-measurable functions
Revised: An equivalent way to describe F ˆ Nm is through a sequence of universally-measurable functions

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: o .
Revised: o .

Original: This leads us to the last step in our algorithm.
Revised: This leads us to the last step in our algorithm.

Original: So far, we have produced a partial concept class
Revised: So far, we have produced a partial concept class

Original: F ˆ Nm with bounded VC dimension, that produces the correct labelings for any (ﬁnite) tuple.
Revised: F ˆ Nm with bounded VC dimension, that produces the correct labelings for any (ﬁnite) tuple.

Original: Thus, it sufﬁces to use the interactive learning algorithm with the b n/ 2 c remaining queries that learns partial concept classes at an exponential rate, which we described before (cf.
Revised: Thus, it sufﬁces to use the interactive learning algorithm with the b n/ 2 c remaining queries that learns partial concept classes at an exponential rate, which we described before (cf.

Original: Figure 3).
Revised: Figure 3).

Original:  Conclusion and Future Directions
Revised:  Conclusion and Future Directions

Original: In this paper we provide a complete characterization of universal interactive learning and reveal a fundamental trichotomy of interactive learning rates.
Revised: In this paper we provide a complete characterization of universal interactive learning and reveal a fundamental trichotomy of interactive learning rates.

Original: Moreover, we specify exactly which properties of the hypothesis class give rise to each case in the trichotomy.
Revised: Moreover, we specify exactly which properties of the hypothesis class give rise to each case in the trichotomy.

Original: We emphasize that the general interactive model we consider makes our lower bounds particularly strong.
Revised: The general interactive model we consider makes our lower bounds particularly strong.

Original: We believe that there are important questions in this line of work that are beyond the scope of our paper and need to be addressed, such as characterizing the unlabeled sample complexity, studying weaker types of interactions such as label queries and comparison queries, and designing efﬁcient and practical algorithms for natural classes.
Revised: We believe that there are important questions in this line of work that are beyond the scope of our paper and need to be addressed, such as characterizing the unlabeled sample complexity, studying weaker types of interactions such as label queries and comparison queries, and designing efﬁcient and practical algorithms for natural classes.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: Amin Karbasi acknowledges funding in direct support of this work from NSF (IIS-1845032), ONR (N00014- 19-1-2406), and the AI Institute for Learning-Enabled Optimization at Scale (TILOS).
Revised: Amin Karbasi acknowledges funding in direct support of this work from NSF (IIS-1845032), ONR (N00014- 19-1-2406), and the AI Institute for Learning-Enabled Optimization at Scale (TILOS).

Original: Shay Moran is a Robert J. Shillman Fellow; he acknowledges support byERC grant 802599, by ISF grant 1225/20, by BSF grant 2018385, by an Azrieli Faculty Fellowship, by Israel PBC-VATAT,and by the Technion Center for Machine Learning and Intelligent Systems (MLIS).
Revised: Shay Moran is a Robert J. Shillman Fellow; he acknowledges support by ISF grant 1225/20, by BSF grant 2018385, by an Azrieli Faculty Fellowship, by Israel PBC-VATAT, by the Technion Center for Machine Learning and Intelligent Systems (MLIS), and by the the European Union (ERC, GENERALIZATION, 101039692).

Revised: Views and opinions expressed are however those of the author(s) only and do not necessarily reﬂect those of the European Union or the European Research Council

Revised: Executive Agency.

Original:  Grigoris Velegkas is supported by NSF (IIS-1845032), an Onassis Foundation PhD Fellowship and a Bodossaki Foundation PhD Fellowship.The authors would like to thank the anonymous reviewers for their helpful comments and suggestions.
Revised: Neither the European Union nor the granting authority can be held responsible for them. Grigoris Velegkas is supported by NSF (IIS-1845032), an Onassis Foundation PhD Fellowship and a Bodossaki Foundation PhD Fellowship.