Original: Active Labeling: Streaming Stochastic Gradients
Revised: Active Labeling: Streaming Stochastic Gradients

Original: Abstract
Revised: Abstract

Original: The workhorse of machine learning is stochastic gradient descent.
Revised: The workhorse of machine learning is stochastic gradient descent.

Original: To access stochastic gradients, it is common to consider iteratively input/output pairs of a training dataset.
Revised: To access stochastic gradients, it is common to consider iteratively input/output pairs of a training dataset.

Original: Interestingly, it appears that one does not need full supervision to access stochastic gradients, which is the main motivation of this paper.
Revised: Interestingly, it appears that one does not need full supervision to access stochastic gradients, which is the main motivation of this paper.

Original: After formalizing the “active labeling” problem, which focuses on active learning with partial supervision, we provide a streaming technique that provably minimizes the ratio of generalization error over the number of samples.
Revised: After formalizing the "active labeling" problem, which focuses on active learning with partial supervision, we provide a streaming technique that provably minimizes the ratio of generalization error over the number of samples.

Original: We illustrate our technique in depth for robust regression
Revised: We illustrate our technique in depth for robust regression

Original:  Introduction
Revised:  Introduction

Original: A large amount of the current hype around artificial intelligence was fueled by the recent successes of supervised learning.
Revised: A large amount of the current hype around artificial intelligence was fueled by the recent successes of supervised learning.

Original: Supervised learning consists in designing an algorithm that maps inputs to outputs by learning from a set of input/output examples.
Revised: Supervised learning consists in designing an algorithm that maps inputs to outputs by learning from a set of input/output examples.

Original: When accessing many samples, and given enough computation power, this framework is able to tackle complex tasks.
Revised: When accessing many samples, and given enough computation power, this framework is able to tackle complex tasks.

Original: Interestingly, many of the difficulties arising in practice do not emerge from choosing the right statistical model to solve the supervised learning problem, but from the problem of collecting and cleaning enough data (see
Revised: Interestingly, many of the difficulties arising in practice do not emerge from choosing the right statistical model to solve the supervised learning problem, but from the problem of collecting and cleaning enough data (see

Original: Chapters 1 and 2 of Géron, 2017, for example).
Revised: Chapters 1 and 2 of Géron, 2017, for example).

Original: Those difficulties are not disjoint from the current trends toward data privacy regulations (Council of European Union, 2016).
Revised: Those difficulties are not disjoint from the current trends toward data privacy regulations (Council of European Union, 2016).

Original: This fact motivates this work, where we focus on how to efficiently collect information to carry out the learning process.
Revised: This fact motivates this work, where we focus on how to efficiently collect information to carry out the learning process.

Original: In this paper, we formalize the “active labeling” problem for weak supervision, where the goal is to learn a target function by acquiring the most informative dataset given a restricted budget for annotation.
Revised: In this paper, we formalize the “active labeling” problem for weak supervision, where the goal is to learn a target function by acquiring the most informative dataset given a restricted budget for annotation.

Original: We focus explicitly on weak supervision that comes as a set of label candidates for each input, aiming to partially supervise input data in the most efficient way to guide a learning algorithm.
Revised: We focus explicitly on weak supervision that comes as a set of label candidates for each input, aiming to partially supervise input data in the most efficient way to guide a learning algorithm.

Original: We also restrict our study to the streaming variant where, for each input, only a single partial information can be collected about its corresponding output.
Revised: We also restrict our study to the streaming variant where, for each input, only a single partial information can be collected about its corresponding output.

Original: The crux of this work is to leverage the fact that full supervision is not needed to acquire unbiased stochastic gradients
Revised: The crux of this work is to leverage the fact that full supervision is not needed to acquire unbiased stochastic gradients, and perform stochastic

Original: and perform stochastic gradient descent.
Revised: gradient descent.

Original: The following summarizes our contributions.
Revised: The following summarizes our contributions.

Original: First, we introduce the “active labeling” problem, which is a relevant theoretical framework that encompasses many useful problems encountered by practitioners trying to annotate their data in the most efficient fashion, as well as its streaming variation, in order to deal with privacy preserving issues.
Revised: First, we introduce the “active labeling” problem, which is a relevant theoretical framework that encompasses many useful problems encountered by practitioners trying to annotate their data in the most efficient fashion, as well as its streaming variation, in order to deal with privacy preserving issues.

Original: This is the focus of Section 2. 2.
Revised: This is the focus of Section 2. 2.

Original: Then, in Section 3, we give a high-level framework to access unbiased stochastic gradients with weak information only.
Revised: Then, in Section 3, we give a high-level framework to access unbiased stochastic gradients with weak information only.

Original: This provides a simple solution to the streaming “active labeling” problem.
Revised: This provides a simple solution to the streaming “active labeling” problem.

Original: Finally, we detail this framework for a robust regression task in Section 4, and provide an algorithm whose optimality is proved in Section 5.
Revised: Finally, we detail this framework for a robust regression task in Section 4, and provide an algorithm whose optimality is proved in Section 5.

Original: As a proof of concept, we provide numerical simulations in Section 6.
Revised: As a proof of concept, we provide numerical simulations in Section 6.

Original: We conclude with a high-level discussion around our methods in Section 7.
Revised: We conclude with a high-level discussion around our methods in Section 7.

Original: Related work.
Revised: Related work.

Original: Active query of information is relevant to many settings.
Revised: Active query of information is relevant to many settings.

Original: The most straightforward applications are searching games, such as Bar Kokhba or twenty questions (Walsorth, 1882).
Revised: The most straightforward applications are searching games, such as Bar Kokhba or twenty questions (Walsorth, 1882).

Original: We refer to Pelc (2002) for an in-depth survey of such games, especially when liars introduce uncertainty, and their relations with coding on noisy channels.
Revised: We refer to Pelc (2002) for an in-depth survey of such games, especially when liars introduce uncertainty, and their relations with coding on noisy channels.

Original: But applications are much more diverse, e.g. for numerical simulation (Chevalier et al., 2014), database search (Qarabaqi and Riedewald, 2014), or shape recognition (Geman and Jedynak, 1993), to name a few.
Revised: But applications are much more diverse, e.g. for numerical simulation (Chevalier et al., 2014), database search (Qarabaqi and Riedewald, 2014), or shape recognition (Geman and Jedynak, 1993), to name a few.

Original: In terms of motivations, many streams of research can be related to this problem, such as experimental design (Chernoff, 1959), statistical queries (Kearns, 1998; Fotakis et al., 2021), crowdsourcing (Doan et al., 2011), or aggregation methods in weak supervision (Ratner et al., 2020).
Revised: In terms of motivations, many streams of research can be related to this problem, such as experimental design (Chernoff, 1959), statistical queries (Kearns, 1998; Fotakis et al., 2021), crowdsourcing (Doan et al., 2011), or aggregation methods in weak supervision (Ratner et al., 2020).

Original: More precisely, “active labeling” 2 consists in having several inputs and querying partial information on the labels.
Revised: More precisely, “active labeling” 2 consists in having several inputs and querying partial information on the labels.

Original: It is close to active learning (Settles, 2010; Dasgupta, 2011; Hanneke, 2014), where there are several inputs, but exact outputs are queried; and to active ranking (Valiant, 1975; Ailon, 2011; Braverman et al., 2019), where partial information is queried, but there is only one input.
Revised: It is close to active learning (Settles, 2010; Dasgupta, 2011; Hanneke, 2014), where there are several inputs, but exact outputs are queried; and to active ranking (Valiant, 1975; Ailon, 2011; Braverman et al., 2019), where partial information is queried, but there is only one input.

Original: The streaming variant introduces privacy preserving constraints, a problem that is usually tackled through the notion of differential privacy (Dwork et al., 2006).
Revised: The streaming variant introduces privacy preserving constraints, a problem that is usually tackled through the notion of differential privacy (Dwork et al., 2006).

Original: In terms of formalization, we build on the partial supervision formalization of Cabannes et al.
Revised: In terms of formalization, we build on the partial supervision formalization of Cabannes et al.

Original: (2020), which casts weak supervision as sets of label candidates and generalizes semi-supervised learning (Chapelle et al., 2006).
Revised: (2020), which casts weak supervision as sets of label candidates and generalizes semi-supervised learning (Chapelle et al., 2006).

Original: Finally, our sequential setting with a unique final reward is similar to combinatorial bandits in a pure-exploration setting (Garivier and Kaufmann, 2016; Fiez et al., 2019).
Revised: Finally, our sequential setting with a unique final reward is similar to combinatorial bandits in a pure-exploration setting (Garivier and Kaufmann, 2016; Fiez et al., 2019).

Original:  The “active labeling” problem
Revised:  The “active labeling” problem

Original: Supervised learning is traditionally modeled in the following manner.
Revised: Supervised learning is traditionally modeled in the following manner.

Original: Consider X an input space, Y an output space, ℓ : Y × Y → R a loss function, and 𝜌 ∈ Δ X × Y a joint probability distribution.
Revised: Consider X an input space, Y an output space, ℓ : Y × Y → R a loss function, and 𝜌 ∈ Δ X × Y a joint probability distribution.

Original: The goal is to recover the function
Revised: The goal is to recover the function

Original: [Equation]
Revised: [Equation]

Original: yet, without accessing 𝜌 , but a dataset of independent samples distributed according to 𝜌 , D 𝑛 = ( 𝑋 𝑖 ,𝑌 𝑖 ) 𝑖 ≤ 𝑛 ∼ 𝜌 ⊗ 𝑛 .
Revised: yet, without accessing 𝜌 , but a dataset of independent samples distributed according to 𝜌 , D 𝑛 = ( 𝑋 𝑖 ,𝑌 𝑖 ) 𝑖 ≤ 𝑛 ∼ 𝜌 ⊗ 𝑛 .

Original: In practice, accessing data comes at a cost, and it is valuable to understand the cheapest way to collect a dataset allowing to discriminate 𝑓 ∗ .
Revised: In practice, accessing data comes at a cost, and it is valuable to understand the cheapest way to collect a dataset allowing to discriminate 𝑓 ∗ .

Original: We shall suppose that the input data ( 𝑋 𝑖 ) 𝑖 ≤ 𝑛 are easy to collect, yet that labeling those inputs to get outputs ( 𝑌 𝑖 ) 𝑖 ≤ 𝑛 demands a high amount of work.
Revised: We shall suppose that the input data ( 𝑋 𝑖 ) 𝑖 ≤ 𝑛 are easy to collect, yet that labeling those inputs to get outputs ( 𝑌 𝑖 ) 𝑖 ≤ 𝑛 demands a high amount of work.

Original: For example, it is relatively easy to scrap the web or medical databases to access radiography images, but labeling them by asking radiologists to recognize tumors on zillions of radiographs will be both time-consuming and expensive.
Revised: For example, it is relatively easy to scrap the web or medical databases to access radiography images, but labeling them by asking radiologists to recognize tumors on zillions of radiographs will be both time-consuming and expensive.

Original: As a consequence, we assume the ( 𝑋 𝑖 ) 𝑖 ≤ 𝑛 given but the ( 𝑌 𝑖 ) 𝑖 ≤ 𝑛 unknown .
Revised: As a consequence, we assume the ( 𝑋 𝑖 ) 𝑖 ≤ 𝑛 given but the ( 𝑌 𝑖 ) 𝑖 ≤ 𝑛 unknown .

Original: As getting information on the labels comes at a cost ( e.g. , paying a pool of label workers, or spending your own time), given a budget constraint, what information should we query on the labels?
Revised: As getting information on the labels comes at a cost ( e.g. , paying a pool of label workers, or spending your own time), given a budget constraint, what information should we query on the labels?

Original: To quantify this problem, we will assume that we can sequentially and adaptively query 𝑇 information of the type 1 𝑌 𝑖𝑡 ∈ 𝑆 𝑡 , for any index 𝑖 𝑡 ∈ { 1 , · · · , 𝑛 } and any set of labels 𝑆 𝑡 ⊂ Y (belonging to a specified set of subsets of Y ).
Revised: To quantify this problem, we will assume that we can sequentially and adaptively query 𝑇 information of the type 1 𝑌 𝑖𝑡 ∈ 𝑆 𝑡 , for any index 𝑖 𝑡 ∈ { 1 , · · · , 𝑛 } and any set of labels 𝑆 𝑡 ⊂ Y (belonging to a specified set of subsets of Y ).

Original: Here, 𝑡 ∈ { 1 , · · · ,𝑇 } indexes the query sequence, and 𝑇 ∈ N is a fixed budget.
Revised: Here, 𝑡 ∈ { 1 , · · · ,𝑇 } indexes the query sequence, and 𝑇 ∈ N is a fixed budget.

Original: The goal is to optimize the design of the sequence ( 𝑖 𝑡 , 𝑆 𝑡 ) in order to get the best estimate of 𝑓 ∗ in terms of risk minimization (1).
Revised: The goal is to optimize the design of the sequence ( 𝑖 𝑡 , 𝑆 𝑡 ) in order to get the best estimate of 𝑓 ∗ in terms of risk minimization (1).

Original: In the following, we give some examples to make this setting more concrete.
Revised: In the following, we give some examples to make this setting more concrete.

Original: Example 1 (Classification with attributes) .
Revised: Example 1 (Classification with attributes) .

Original: Suppose that a labeler is asked to provide fine-grained classes on images (Krause et al., 2016; Zheng et al., 2019), such as the label “caracal” in Figure 6.
Revised: Suppose that a labeler is asked to provide fine-grained classes on images (Krause et al., 2016; Zheng et al., 2019), such as the label “caracal” in Figure 6.

Original: This would be difficult for many people.
Revised: This would be difficult for many people.

Original: Yet, it is relatively easy to recognize that the image depicts a “feline” with “tufted-ears” and “sandy color”.
Revised: Yet, it is relatively easy to recognize that the image depicts a “feline” with “tufted-ears” and “sandy color”.

Original: As such, a labeler can give the weak information that 𝑌 belongs to the set “feline”, 𝑆 1 = { “cat”, “lion”, “tiger” , . . . } , and the set “tufted ears”, 𝑆 2 = { “Great horned owl”, “Aruacana chicken” , . . . } .
Revised: As such, a labeler can give the weak information that 𝑌 belongs to the set “feline”, 𝑆 1 = { “cat”, “lion”, “tiger” , . . . } , and the set “tufted ears”, 𝑆 2 = { “Great horned owl”, “Aruacana chicken” , . . . } .

Original: This is enough to recognize that 𝑌 ∈ 𝑆 1 ∩ 𝑆 2 = { “caracal” } .
Revised: This is enough to recognize that 𝑌 ∈ 𝑆 1 ∩ 𝑆 2 = { “caracal” } .

Original: The question 1 𝑌 ∈ 𝑆 1 , corresponds to asking if the image depicts a feline.
Revised: The question 1 𝑌 ∈ 𝑆 1 , corresponds to asking if the image depicts a feline.

Original: Literature on hierarchical classification and autonomic taxonomy construction provides interesting ideas for this problem ( e.g. , Cesa-Bianchi et al., 2006; Gangaputra and Geman, 2006).
Revised: Literature on hierarchical classification and autonomic taxonomy construction provides interesting ideas for this problem ( e.g. , Cesa-Bianchi et al., 2006; Gangaputra and Geman, 2006).

Original: Example 2 (Ranking with partial ordering) .
Revised: Example 2 (Ranking with partial ordering) .

Original: Consider a problem where for a given input 𝑥 , characterizing a user, we are asked to deduce their preferences over 𝑚 items.
Revised: Consider a problem where for a given input 𝑥 , characterizing a user, we are asked to deduce their preferences over 𝑚 items.

Original: Collecting such a label requires knowing the exact ordering of the 𝑚 items induced by a user.
Revised: Collecting such a label requires knowing the exact ordering of the 𝑚 items induced by a user.

Original: This might be hard to ask for.
Revised: This might be hard to ask for.

Original: Instead, one can easily ask the user which items they prefer in a collection of a few items.
Revised: Instead, one can easily ask the user which items they prefer in a collection of a few items.

Original: The user’s answer will give weak information about the labels, which can be modeled as knowing 1 𝑌 𝑖 ∈ 𝑆 = 1 , for 𝑆 the set of total orderings that satisfy this partial ordering.
Revised: The user’s answer will give weak information about the labels, which can be modeled as knowing 1 𝑌 𝑖 ∈ 𝑆 = 1 , for 𝑆 the set of total orderings that satisfy this partial ordering.

Original: We refer the curious reader to active ranking and dueling bandits for additional contents (Jamieson and Nowak, 2011; Bengs et al., 2021).
Revised: We refer the curious reader to active ranking and dueling bandits for additional contents (Jamieson and Nowak, 2011; Bengs et al., 2021).

Original: Example 3 (Pricing a product) .
Revised: Example 3 (Pricing a product) .

Original: Suppose that we want to sell a product to a consumer characterized by some features 𝑥 , this consumer is ready to pay a price 𝑦 ∈ R for this product.
Revised: Suppose that we want to sell a product to a consumer characterized by some features 𝑥 , this consumer is ready to pay a price 𝑦 ∈ R for this product.

Original: We price it 𝑓 ( 𝑥 ) ∈ R , and we observe 1 𝑓 ( 𝑥 ) <
Revised: We price it 𝑓 ( 𝑥 ) ∈ R , and we observe 1 𝑓 ( 𝑥 ) <

Original: 𝑦 , that is if the consumer is willing to buy this product at this price tag or not (Cesa-Bianchi et al., 2019; Liu et al., 2021).
Revised: 𝑦 , that is if the consumer is willing to buy this product at this price tag or not (Cesa-Bianchi et al., 2019; Liu et al., 2021).

Original: Although, in this setting, the goal is often to minimize the regret, which contrasts with our pure exploration setting.
Revised: Although, in this setting, the goal is often to minimize the regret, which contrasts with our pure exploration setting.

Original: As a counter-example, our assumptions are not set to deal with missing data, i.e. if some coordinates of some input feature vectors 𝑋 𝑖 are missing (Rubin, 1976).
Revised: As a counter-example, our assumptions are not set to deal with missing data, i.e. if some coordinates of some input feature vectors 𝑋 𝑖 are missing (Rubin, 1976).

Original: Typically, this happens when input data comes from different sources ( e.g. , when trying to predict economic growth from country information that is self-reported).
Revised: Typically, this happens when input data comes from different sources ( e.g. , when trying to predict economic growth from country information that is self-reported).

Original: Streaming variation.
Revised: Streaming variation.

Original: The special case of the active labeling problem we shall consider consists in its variant without resampling.
Revised: The special case of the active labeling problem we shall consider consists in its variant without resampling.

Original: This corresponds to the online setting where one can only ask one question by sample, formally 𝑖 𝑡 = 𝑡 .
Revised: This corresponds to the online setting where one can only ask one question by sample, formally 𝑖 𝑡 = 𝑡 .

Original: This setting is particularly appealing for privacy concerns, in settings where the labels ( 𝑌 𝑖 ) contain sensitive information that should not be revealed totally.
Revised: This setting is particularly appealing for privacy concerns, in settings where the labels ( 𝑌 𝑖 ) contain sensitive information that should not be revealed totally.

Original: For example, some people might be more comfortable giving a range over a salary rather than the exact value; or in the context of polling, one might not call back a previous respondent characterized by some features 𝑋 𝑖 to ask them again about their preferences captured by 𝑌 𝑖 .
Revised: For example, some people might be more comfortable giving a range over a salary rather than the exact value; or in the context of polling, one might not call back a previous respondent characterized by some features 𝑋 𝑖 to ask them again about their preferences captured by 𝑌 𝑖 .

Original: Similarly, the streaming setting is relevant for web marketing, where inputs model new users visiting a website, queries model sets of advertisements chosen by an advertising company, and one observes potential clicks.
Revised: Similarly, the streaming setting is relevant for web marketing, where inputs model new users visiting a website, queries model sets of advertisements chosen by an advertising company, and one observes potential clicks.

Original:  Weak information as stochastic gradients
Revised:  Weak information as stochastic gradients

Original: In this section, we discuss how unbiased stochastic gradients can be accessed through weak information.
Revised: In this section, we discuss how unbiased stochastic gradients can be accessed through weak information.

Original: Suppose that we model 𝑓 = 𝑓 𝜃 for some Hilbert space Θ ∋ 𝜃 .
Revised: Suppose that we model 𝑓 = 𝑓 𝜃 for some Hilbert space Θ ∋ 𝜃 .

Original: With some abuse of notations, let us denote ℓ ( 𝑥, 𝑦, 𝜃 ) : =
Revised: With some abuse of notations, let us denote ℓ ( 𝑥, 𝑦, 𝜃 ) : =

Original: ℓ ( 𝑓 𝜃 ( 𝑥 ) , 𝑦 ) .
Revised: ℓ ( 𝑓 𝜃 ( 𝑥 ) , 𝑦 ) .

Original: We aim to minimize R ( 𝜃 ) =
Revised: We aim to minimize R ( 𝜃 ) =

Original: E ( 𝑋,𝑌 ) [ ℓ ( 𝑋,𝑌, 𝜃 )] .
Revised: E ( 𝑋,𝑌 ) [ ℓ ( 𝑋,𝑌, 𝜃 )] .

Original: Assume that R is differentiable (or sub-differentiable) and denote its gradients by ∇ 𝜃 R .
Revised: Assume that R is differentiable (or sub-differentiable) and denote its gradients by ∇ 𝜃 R .

Original: Definition 1 (Stochastic gradient) .
Revised: Definition 1 (Stochastic gradient) .

Original: A stochastic gradient of R is any random function 𝐺 : Θ → Θ such that E [ 𝐺 ( 𝜃 )]
Revised: A stochastic gradient of R is any random function 𝐺 : Θ → Θ such that E [ 𝐺 ( 𝜃 )]

Original: = ∇ 𝜃 R ( 𝜃 ) .
Revised: = ∇ 𝜃 R ( 𝜃 ) .

Original: Given some step size function 𝛾 : N → R ∗ , a stochastic gradient descent (SGD) is a procedure, ( 𝜃 𝑡 )
Revised: Given some step size function 𝛾 : N → R ∗ , a stochastic gradient descent (SGD) is a procedure, ( 𝜃 𝑡 )

Original: ∈
Revised: ∈

Original: Θ N , initialized with some 𝜃 0 and updated as 𝜃 𝑡 + 1 =
Revised: Θ N , initialized with some 𝜃 0 and updated as 𝜃 𝑡 + 1 =

Original: 𝜃 𝑡 − 𝛾 ( 𝑡 ) 𝐺 ( 𝜃 𝑡 ) , where the realization of 𝐺 ( 𝜃 𝑡 ) given 𝜃 𝑡 is independent of the previous realizations of 𝐺 ( 𝜃 𝑠 ) given 𝜃 𝑠 .
Revised: 𝜃 𝑡 − 𝛾 ( 𝑡 ) 𝐺 ( 𝜃 𝑡 ) , where the realization of 𝐺 ( 𝜃 𝑡 ) given 𝜃 𝑡 is independent of the previous realizations of 𝐺 ( 𝜃 𝑠 ) given 𝜃 𝑠 .

Original: In supervised learning, SGD is usually performed with the stochastic gradients ∇ 𝜃 ℓ ( 𝑋,𝑌, 𝜃 ) .
Revised: In supervised learning, SGD is usually performed with the stochastic gradients ∇ 𝜃 ℓ ( 𝑋,𝑌, 𝜃 ) .

Original: More generally, stochastic gradients are given by
Revised: More generally, stochastic gradients are given by

Original: [Equation]
Revised: [Equation]

Original: for 𝜏 : T → Θ with T ⊂ 2 Θ a set of subsets of Θ , and 𝑇 a random variable on T , such that
Revised: for 𝜏 : T → Θ with T ⊂ 2 Θ a set of subsets of Θ , and 𝑇 a random variable on T , such that

Original: [Equation]
Revised: [Equation]

Original: Stated otherwise, if you have a way to image a vector 𝜃 from partial measurements 1 𝜃 ∈ 𝑇 such that you can reconstruct this vector in a linear fashion (3), then it provides you a generic strategy to get an unbiased stochastic estimate of this vector from a partial measurement (2).
Revised: Stated otherwise, if you have a way to image a vector 𝜃 from partial measurements 1 𝜃 ∈ 𝑇 such that you can reconstruct this vector in a linear fashion (3), then it provides you a generic strategy to get an unbiased stochastic estimate of this vector from a partial measurement (2).

Original: For 𝜓 : Y → Θ a function from Y to Θ ( e.g. , 𝜓 = ∇ 𝜃 ( 𝑋, · , 𝜃 ) ), a question 1 𝜓 ( 𝑌 )∈ 𝑇 translates into a question 1 𝑌 ∈ 𝑆 for some set 𝑆 = 𝜓 − 1 ( 𝑇 ) ⊂
Revised: For 𝜓 : Y → Θ a function from Y to Θ ( e.g. , 𝜓 = ∇ 𝜃 ( 𝑋, · , 𝜃 ) ), a question 1 𝜓 ( 𝑌 )∈ 𝑇 translates into a question 1 𝑌 ∈ 𝑆 for some set 𝑆 = 𝜓 − 1 ( 𝑇 ) ⊂

Original: Y , meaning that the stochastic gradient (2) can be evaluated from a single query.
Revised: Y , meaning that the stochastic gradient (2) can be evaluated from a single query.

Original: As a proof of concept, we derive a generic implementation for 𝑇 and 𝜏 in
Revised: As a proof of concept, we derive a generic implementation for 𝑇 and 𝜏 in

Original: Appendix B. This provides a generic SGD scheme to learn functions from weak queries when there are no constraints on the sets to query.
Revised: Appendix B. This provides a generic SGD scheme to learn functions from weak queries when there are no constraints on the sets to query.

Original: Remark 2 (Cutting plane methods) .
Revised: Remark 2 (Cutting plane methods) .

Original: While we provide here a descent method, one could also develop cutting-plane/ellipsoid methods to localize 𝜃 ∗ according to weak information, which corresponds to the techniques developed for pricing by Cohen et al.
Revised: While we provide here a descent method, one could also develop cutting-plane/ellipsoid methods to localize 𝜃 ∗ according to weak information, which corresponds to the techniques developed for pricing by Cohen et al.

Original: (2020) and related literature.
Revised: (2020) and related literature.

Original:  Median regression
Revised:  Median regression

Original: In this section, we focus on efficiently acquiring weak information providing stochastic gradients for regression problems.
Revised: In this section, we focus on efficiently acquiring weak information providing stochastic gradients for regression problems.

Original: In particular, we motivate and detail our methods for the absolute deviation loss.
Revised: In particular, we motivate and detail our methods for the absolute deviation loss.

Original: Motivated by seminal works on censored data (Tobin, 1958), we shall suppose that we query halfspaces .
Revised: Motivated by seminal works on censored data (Tobin, 1958), we shall suppose that we query halfspaces .

Original: For an output 𝑦 ∈ Y
Revised: For an output 𝑦 ∈ Y

Original: = R 𝑚 , and any hyper-plane 𝑧 +
Revised: = R 𝑚 , and any hyper-plane 𝑧 +

Original: 𝑢 ⊥ ⊂ R 𝑚 for 𝑧 ∈ R 𝑚 , 𝑢 ∈ S 𝑚 − 1 , we can ask a labeler to tell us which half-space 𝑦 belongs to.
Revised: 𝑢 ⊥ ⊂ R 𝑚 for 𝑧 ∈ R 𝑚 , 𝑢 ∈ S 𝑚 − 1 , we can ask a labeler to tell us which half-space 𝑦 belongs to.

Original: Formally, we access the quantity sign (⟨ 𝑦 − 𝑧, 𝑢 ⟩) for a given unit cost .
Revised: Formally, we access the quantity sign (⟨ 𝑦 − 𝑧, 𝑢 ⟩) for a given unit cost .

Original: Such an imaging scheme where one observes summations of its components rather than a vector itself bears similarity with compressed sensing.
Revised: Such an imaging scheme where one observes summations of its components rather than a vector itself bears similarity with compressed sensing.

Original: To provide further illustration, this setting could help to price products while selling bundles: where the context 𝑥 characterizes some users, web-pages or/and advertisement companies; the label 𝑦 ∈ R 𝑚 corresponds to the value associated to 𝑚 different items, such as stocks composing an index, or advertisement spots; and the observation sign (⟨ 𝑦, 𝑢 ⟩ − 𝑐 ) (with 𝑐 = ⟨ 𝑧, 𝑢 ⟩ ) captures if the user 𝑥 buys the basket with weights 𝑢 ∈ S 𝑚 − 1 when it is priced 𝑐 .
Revised: To provide further illustration, this setting could help to price products while selling bundles: where the context 𝑥 characterizes some users, web-pages or/and advertisement companies; the label 𝑦 ∈ R 𝑚 corresponds to the value associated to 𝑚 different items, such as stocks composing an index, or advertisement spots; and the observation sign (⟨ 𝑦, 𝑢 ⟩ − 𝑐 ) (with 𝑐 = ⟨ 𝑧, 𝑢 ⟩ ) captures if the user 𝑥 buys the basket with weights 𝑢 ∈ S 𝑚 − 1 when it is priced 𝑐 .

Original: Least-squares.
Revised: Least-squares.

Original: For regression problems, it is common to look at the mean square loss ℓ
Revised: For regression problems, it is common to look at the mean square loss ℓ

Original: [Equation]
Revised: [Equation]

Original: where 𝐷 𝑓 𝜃 ( 𝑥 ) ∈ Y ⊗ Θ denotes the Jacobian of 𝜃 → 𝑓 𝜃 ( 𝑥 ) .
Revised: where 𝐷 𝑓 𝜃 ( 𝑥 ) ∈ Y ⊗ Θ denotes the Jacobian of 𝜃 → 𝑓 𝜃 ( 𝑥 ) .

Original: In rich parametric models, it is preferable to ask questions on 𝑌 ∈ Y rather than on gradients in Θ which is a potentially much bigger space.
Revised: In rich parametric models, it is preferable to ask questions on 𝑌 ∈ Y rather than on gradients in Θ which is a potentially much bigger space.

Original: If we assume that 𝑌 and 𝑓 𝜃 ( 𝑋 ) are bounded in ℓ 2 -norm by 𝑀 ∈ R + , we can adapt (2) and (3) through the fact that for any 𝑧 ∈ Y , such that ∥ 𝑧 ∥ ≤ 2 𝑀 , as proven in Appendix B,
Revised: If we assume that 𝑌 and 𝑓 𝜃 ( 𝑋 ) are bounded in ℓ 2 -norm by 𝑀 ∈ R + , we can adapt (2) and (3) through the fact that for any 𝑧 ∈ Y , such that ∥ 𝑧 ∥ ≤ 2 𝑀 , as proven in Appendix B,

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: for 𝑈 uniform on the sphere S 𝑚 − 1 and 𝑉 uniform on [ 0 , 2 𝑀 ] .
Revised: for 𝑈 uniform on the sphere S 𝑚 − 1 and 𝑉 uniform on [ 0 , 2 𝑀 ] .

Original: Applied to 𝑧 = 𝑓 𝜃 ( 𝑋 ) − 𝑌 , it designs an
Revised: Applied to 𝑧 = 𝑓 𝜃 ( 𝑋 ) − 𝑌 , it designs an

Original: SGD procedure by querying information of the type 1 ⟨ 𝑌,𝑈 ⟩ < ⟨ 𝑓 𝜃 ( 𝑋 ) ,𝑈 ⟩− 𝑉 .
Revised: SGD procedure by querying information of the type 1 ⟨ 𝑌,𝑈 ⟩ < ⟨ 𝑓 𝜃 ( 𝑋 ) ,𝑈 ⟩− 𝑉 .

Original: A case for median regression.
Revised: A case for median regression.

Original: Motivated by robustness purposes, we will rather expand on median regression.
Revised: Motivated by robustness purposes, we will rather expand on median regression.

Original: In general, we would like to learn a function that, given an input, replicates the output of I/O samples generated by the joint probability 𝜌 .
Revised: In general, we would like to learn a function that, given an input, replicates the output of I/O samples generated by the joint probability 𝜌 .

Original: In many instances, 𝑋 does not characterize all the sources of variations of 𝑌 , i.e. input features are not rich enough to characterize a unique output, leading to randomness in the conditional distributions ( 𝑌 | 𝑋 ) .
Revised: In many instances, 𝑋 does not characterize all the sources of variations of 𝑌 , i.e. input features are not rich enough to characterize a unique output, leading to randomness in the conditional distributions ( 𝑌 | 𝑋 ) .

Original: When many targets can be linked to a vector 𝑥 ∈ X , how to define a consensual 𝑓 ( 𝑥 ) ?
Revised: When many targets can be linked to a vector 𝑥 ∈ X , how to define a consensual 𝑓 ( 𝑥 ) ?

Original: For analytical reasons, statisticians tend to use the least-squares error which corresponds to asking for 𝑓 ( 𝑥 ) to be the mean of the distribution ( 𝑌 | 𝑋 = 𝑥 ) .
Revised: For analytical reasons, statisticians tend to use the least-squares error which corresponds to asking for 𝑓 ( 𝑥 ) to be the mean of the distribution ( 𝑌 | 𝑋 = 𝑥 ) .

Original: Yet, means are known to be too sensitive to rare but large outputs (see e.g. , Huber, 1981), and cannot be defined as good and robust consensus in a world of heavy-tailed distributions.
Revised: Yet, means are known to be too sensitive to rare but large outputs (see e.g. , Huber, 1981), and cannot be defined as good and robust consensus in a world of heavy-tailed distributions.

Original: This contrasts with the median, which, as a consequence, is often much more valuable to summarize a range of values.
Revised: This contrasts with the median, which, as a consequence, is often much more valuable to summarize a range of values.

Original: For instance, median income is preferred over mean income as a population indicator (see e.g. , US
Revised: For instance, median income is preferred over mean income as a population indicator (see e.g. , US

Original: Census Bureau, 2021).
Revised: Census Bureau, 2021).

Original: Median regression.
Revised: Median regression.

Original: The geometric median is variationally defined through the absolute deviation loss, leading to
Revised: The geometric median is variationally defined through the absolute deviation loss, leading to

Original: [Equation]
Revised: [Equation]

Original: Similarly to the least-squares case, we can access weakly supervised stochastic gradients through the fact that for 𝑧 ∈ S 𝑚 − 1 , as shown in Appendix B,
Revised: Similarly to the least-squares case, we can access weakly supervised stochastic gradients through the fact that for 𝑧 ∈ S 𝑚 − 1 , as shown in Appendix B,

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: where 𝑈 is uniformly drawn on the sphere S 𝑚 − 1 , and Γ is the gamma function.
Revised: where 𝑈 is uniformly drawn on the sphere S 𝑚 − 1 , and Γ is the gamma function.

Original: This suggests
Revised: This suggests

Original: Algorithm 1.
Revised: Algorithm 1.

Original: Data: A model 𝑓 𝜃 for 𝜃 ∈ Θ , some data ( 𝑋 𝑖 ) 𝑖 ≤ 𝑛 , a labeling budget 𝑇 , a step size rule 𝛾 : N → R +
Revised: Data: A model 𝑓 𝜃 for 𝜃 ∈ Θ , some data ( 𝑋 𝑖 ) 𝑖 ≤ 𝑛 , a labeling budget 𝑇 , a step size rule 𝛾 : N → R +

Original: Result:
Revised: Result:

Original: A learned parameter ˆ 𝜃 and the predictive function ˆ 𝑓 = 𝑓 ˆ 𝜃 .
Revised: A learned parameter ˆ 𝜃 and the predictive function ˆ 𝑓 = 𝑓 ˆ 𝜃 .

Original: Initialize 𝜃 0 .
Revised: Initialize 𝜃 0 .

Original: for 𝑡 ← 1 to 𝑇 do
Revised: for 𝑡 ← 1 to 𝑇 do

Original: Sample 𝑈 𝑡 uniformly on S 𝑚 − 1 .
Revised: Sample 𝑈 𝑡 uniformly on S 𝑚 − 1 .

Original: Query 𝜀 = sign (⟨ 𝑌 𝑡 − 𝑧,𝑈 𝑡 ⟩) for 𝑧 = 𝑓 𝜃 𝑡 − 1 ( 𝑋 𝑡 ) .
Revised: Query 𝜀 = sign (⟨ 𝑌 𝑡 − 𝑧,𝑈 𝑡 ⟩) for 𝑧 = 𝑓 𝜃 𝑡 − 1 ( 𝑋 𝑡 ) .

Original: Update the parameter 𝜃 𝑡 = 𝜃 𝑡 − 1 + 𝛾 ( 𝑡 ) 𝜀 · 𝑈 ⊤ 𝑡 ( 𝐷 𝑓 𝜃 𝑡 − 1 ( 𝑋 𝑡 )) .
Revised: Update the parameter 𝜃 𝑡 = 𝜃 𝑡 − 1 + 𝛾 ( 𝑡 ) 𝜀 · 𝑈 ⊤ 𝑡 ( 𝐷 𝑓 𝜃 𝑡 − 1 ( 𝑋 𝑡 )) .

Original: Output ˆ 𝜃 = 𝜃 𝑇 , or some average, e.g. , ˆ 𝜃 = 𝑇 − 1 (cid:205)
Revised: Output ˆ 𝜃 = 𝜃 𝑇 , or some average, e.g. , ˆ 𝜃 = 𝑇 − 1 (cid:205)

Original: 𝑇𝑡 = 1 𝜃 𝑡 .
Revised: 𝑇𝑡 = 1 𝜃 𝑡 .

Original:  Statistical analysis
Revised:  Statistical analysis

Original: In this section, we quantify the performance of Algorithm 1 by proving optimal rates of convergence when the median regression problem is approached with (reproducing) kernels.
Revised: In this section, we quantify the performance of Algorithm 1 by proving optimal rates of convergence when the median regression problem is approached with (reproducing) kernels.

Original: For simplicity, we will assume that 𝑓 ∗ can be parametrized by a linear model (potentially of infinite dimension).
Revised: For simplicity, we will assume that 𝑓 ∗ can be parametrized by a linear model (potentially of infinite dimension).

Original: Assumption 1.
Revised: Assumption 1.

Original: Assume that the solution 𝑓 ∗ : X → R 𝑚 of the median regression problem (1) and (4) can be parametrized by some separable Hilbert space H , and a bounded feature map 𝜑 : X → H , such that, for any 𝑖 ∈ [ 𝑚 ] , there exists some 𝜃 ∗ 𝑖 ∈
Revised: Assume that the solution 𝑓 ∗ : X → R 𝑚 of the median regression problem (1) and (4) can be parametrized by some separable Hilbert space H , and a bounded feature map 𝜑 : X → H , such that, for any 𝑖 ∈ [ 𝑚 ] , there exists some 𝜃 ∗ 𝑖 ∈

Original: H such that ⟨ 𝑓 ∗ (·) , 𝑒 𝑖 ⟩ Y
Revised: H such that ⟨ 𝑓 ∗ (·) , 𝑒 𝑖 ⟩ Y

Original: =
Revised: =

Original: (cid:10) 𝜃 ∗ 𝑖 , 𝜑 (·) (cid:11) H , where ( 𝑒 𝑖 ) is the canonical basis of R 𝑚 .
Revised: (cid:10) 𝜃 ∗ 𝑖 , 𝜑 (·) (cid:11) H , where ( 𝑒 𝑖 ) is the canonical basis of R 𝑚 .

Original: Written into matrix form, there exists 𝜃 ∗ ∈ Y ⊗ H , such that 𝑓 ∗ (·)
Revised: Written into matrix form, there exists 𝜃 ∗ ∈ Y ⊗ H , such that 𝑓 ∗ (·)

Original: =
Revised: =

Original: 𝜃 ∗ 𝜑 (·) .
Revised: 𝜃 ∗ 𝜑 (·) .

Original: The curious reader can easily relax this assumption in the realm of reproducing kernel Hilbert spaces following the work of Pillaud-Vivien et al.
Revised: The curious reader can easily relax this assumption in the realm of reproducing kernel Hilbert spaces following the work of Pillaud-Vivien et al.

Original: (2018a).
Revised: (2018a).

Original: Under the linear model of Assumption 1,
Revised: Under the linear model of Assumption 1,

Original: Algorithm 1 is specified with 𝑢 ⊤ 𝐷 𝑓 𝜃 ( 𝑥 ) =
Revised: Algorithm 1 is specified with 𝑢 ⊤ 𝐷 𝑓 𝜃 ( 𝑥 ) =

Original: 𝑢 ⊗ 𝜑 ( 𝑥 ) .
Revised: 𝑢 ⊗ 𝜑 ( 𝑥 ) .

Original: Note that rather than working with Θ = Y ⊗ H which is potentially infinite-dimensional, empirical estimates can be represented in the finitedimensional space Y ⊗ Span { 𝜑 ( 𝑋 𝑖 )} 𝑖 ≤ 𝑛 , and well approximated by small-dimensional spaces to ensure efficient computations (Williams and Seeger, 2000; Meanti et al., 2020).
Revised: Note that rather than working with Θ = Y ⊗ H which is potentially infinite-dimensional, empirical estimates can be represented in the finitedimensional space Y ⊗ Span { 𝜑 ( 𝑋 𝑖 )} 𝑖 ≤ 𝑛 , and well approximated by small-dimensional spaces to ensure efficient computations (Williams and Seeger, 2000; Meanti et al., 2020).

Original: One of the key points of SGD is that gradient descent is so gradual that one can use noisy or stochastic gradients without loosing statistical guarantees while speeding up computations.
Revised: One of the key points of SGD is that gradient descent is so gradual that one can use noisy or stochastic gradients without loosing statistical guarantees while speeding up computations.

Original: This is especially true when minimizing convex functions that are nor strongly-convex, i.e. , bounded below by a quadratic, nor smooth, i.e. , with Lipschitz-continuous gradient (see, e.g. , Bubeck, 2015).
Revised: This is especially true when minimizing convex functions that are nor strongly-convex, i.e. , bounded below by a quadratic, nor smooth, i.e. , with Lipschitz-continuous gradient (see, e.g. , Bubeck, 2015).

Original: In particular, the following theorem, proven in Appendix A.1, states that Algorithm 1 minimizes the population risk at a speed at least proportional to 𝑂 ( 𝑇 − 1 / 2 ) .
Revised: In particular, the following theorem, proven in Appendix A.1, states that Algorithm 1 minimizes the population risk at a speed at least proportional to 𝑂 ( 𝑇 − 1 / 2 ) .

Original: Theorem 1 (Convergence rates) .
Revised: Theorem 1 (Convergence rates) .

Original: Under Assumption 1, and under the knowledge of 𝜅 and 𝑀 two real values such that E [∥ 𝜑 ( 𝑋 )∥ 2 ] ≤ 𝜅 2 and ∥ 𝜃 ∗ ∥ ≤ 𝑀 , with a budget 𝑇 ∈ N , a constant step size 𝛾 = 𝑀𝜅 √ 𝑇 and the average estimate ˆ 𝜃 = 1 𝑇 (cid:205)
Revised: Under Assumption 1, and under the knowledge of 𝜅 and 𝑀 two real values such that E [∥ 𝜑 ( 𝑋 )∥ 2 ] ≤ 𝜅 2 and ∥ 𝜃 ∗ ∥ ≤ 𝑀 , with a budget 𝑇 ∈ N , a constant step size 𝛾 = 𝑀𝜅 √ 𝑇 and the average estimate ˆ 𝜃 = 1 𝑇 (cid:205)

Original: 𝑇 − 1 𝑡 = 0 𝜃 𝑡 , Algorithm 1 leads to an estimate 𝑓 that suffers from an excess of risk
Revised: 𝑇 − 1 𝑡 = 0 𝜃 𝑡 , Algorithm 1 leads to an estimate 𝑓 that suffers from an excess of risk

Original: [Equation]
Revised: [Equation]

Original: where the expectation is taken with respect to the randomness of ˆ 𝜃 that depends on the dataset ( 𝑋 𝑖 ,𝑌 𝑖 ) as well as the questions ( 𝑖 𝑡 , 𝑆 𝑡 ) 𝑡 ≤ 𝑇 .
Revised: where the expectation is taken with respect to the randomness of ˆ 𝜃 that depends on the dataset ( 𝑋 𝑖 ,𝑌 𝑖 ) as well as the questions ( 𝑖 𝑡 , 𝑆 𝑡 ) 𝑡 ≤ 𝑇 .

Original: While we give here a result for a fixed step size, one could retake the extensive literature on SGD to prove similar results for decaying step sizes that do not require to know the labeling budget in advance ( e.g. setting 𝛾 ( 𝑡 ) ∝ 𝑡 − 1 / 2 at the expense of an extra term in log ( 𝑇 ) in front of the rates), as well as different averaging strategies (see e.g. , Bach, 2023).
Revised: While we give here a result for a fixed step size, one could retake the extensive literature on SGD to prove similar results for decaying step sizes that do not require to know the labeling budget in advance ( e.g. setting 𝛾 ( 𝑡 ) ∝ 𝑡 − 1 / 2 at the expense of an extra term in log ( 𝑇 ) in front of the rates), as well as different averaging strategies (see e.g. , Bach, 2023).

Original: In practice, one might not know a priori the parameter 𝑀 but could nonetheless find the right scaling for 𝛾 based on cross-validation.
Revised: In practice, one might not know a priori the parameter 𝑀 but could nonetheless find the right scaling for 𝛾 based on cross-validation.

Original: The rate in 𝑂 ( 𝑇 − 1 / 2 ) applies more broadly to all the strategies described in Section 3 as long as the loss ℓ and the parametric model
Revised: The rate in 𝑂 ( 𝑇 − 1 / 2 ) applies more broadly to all the strategies described in Section 3 as long as the loss ℓ and the parametric model

Original: 𝑓 𝜃 ensure that R ( 𝜃 ) is convex and Lipschitz-continuous.
Revised: 𝑓 𝜃 ensure that R ( 𝜃 ) is convex and Lipschitz-continuous.

Original: Although the constants appearing in front of rates depend on the complexity to reconstruct the full gradient ∇ 𝜃 ℓ ( 𝑓 𝜃 ( 𝑋 𝑖 ,𝑌 𝑖 )) from the reconstruction scheme (3).
Revised: Although the constants appearing in front of rates depend on the complexity to reconstruct the full gradient ∇ 𝜃 ℓ ( 𝑓 𝜃 ( 𝑋 𝑖 ,𝑌 𝑖 )) from the reconstruction scheme (3).

Original: Those constants correspond to the second moment of the stochastic gradient.
Revised: Those constants correspond to the second moment of the stochastic gradient.

Original: For example, for the least-squares technique described earlier one would have to replace 𝑐 2 by 𝑐 1 in (6).
Revised: For example, for the least-squares technique described earlier one would have to replace 𝑐 2 by 𝑐 1 in (6).

Original: Theorem 2, proven in Appendix A.3, states that any algorithm that accesses a fully supervised learning dataset of size 𝑇 cannot beat the rates in 𝑂 ( 𝑇 − 1 / 2 ) , hence any algorithm that collects weaker information on ( 𝑌 𝑖 ) 𝑖 ≤ 𝑇 cannot display better rates than the ones verified by Algorithm 1.
Revised: Theorem 2, proven in Appendix A.3, states that any algorithm that accesses a fully supervised learning dataset of size 𝑇 cannot beat the rates in 𝑂 ( 𝑇 − 1 / 2 ) , hence any algorithm that collects weaker information on ( 𝑌 𝑖 ) 𝑖 ≤ 𝑇 cannot display better rates than the ones verified by Algorithm 1.

Original: This proves minimax optimality of our algorithm up to constants.
Revised: This proves minimax optimality of our algorithm up to constants.

Original: Theorem 2 (Minimax optimality) .
Revised: Theorem 2 (Minimax optimality) .

Original: Under Assumption 1 and the knowledge of an upper bound on ∥ 𝜃 ∗ ∥ ≤ 𝑀 , assuming that 𝜑 is bounded by 𝜅 , there exists a universal constant 𝑐 3 such that for any algorithm A that takes as input D 𝑇 =
Revised: Under Assumption 1 and the knowledge of an upper bound on ∥ 𝜃 ∗ ∥ ≤ 𝑀 , assuming that 𝜑 is bounded by 𝜅 , there exists a universal constant 𝑐 3 such that for any algorithm A that takes as input D 𝑇 =

Original: ( 𝑋 𝑖 ,𝑌 𝑖 ) 𝑖 ≤ 𝑇 ∼ 𝜌 ⊗ 𝑇 for any 𝑇 ∈ N and output a parameter 𝜃 ,
Revised: ( 𝑋 𝑖 ,𝑌 𝑖 ) 𝑖 ≤ 𝑇 ∼ 𝜌 ⊗ 𝑇 for any 𝑇 ∈ N and output a parameter 𝜃 ,

Original: [Equation]
Revised: [Equation]

Original: The supremum over 𝜌 ∈ M 𝑀 has to be understood as the supremum over all distributions 𝜌
Revised: The supremum over 𝜌 ∈ M 𝑀 has to be understood as the supremum over all distributions 𝜌

Original: ∈
Revised: ∈

Original: Δ X × Y such that the problem defined through the risk R ( 𝑓 ; 𝜌 ) : =
Revised: Δ X × Y such that the problem defined through the risk R ( 𝑓 ; 𝜌 ) : =

Original: E 𝜌 [ ℓ ( 𝑓 ( 𝑋 ) ,𝑌 )] is minimized for 𝑓 𝜌 that verifies Assumption 1 with ∥ 𝜃 ∗ ∥ bounded by a constant 𝑀 .
Revised: E 𝜌 [ ℓ ( 𝑓 ( 𝑋 ) ,𝑌 )] is minimized for 𝑓 𝜌 that verifies Assumption 1 with ∥ 𝜃 ∗ ∥ bounded by a constant 𝑀 .

Original: The same theorem applies for least-squares with a different universal constant.
Revised: The same theorem applies for least-squares with a different universal constant.

Original: It should be noted that minimax lower bounds are in essence quantifying worst cases of a given class of problems.
Revised: It should be noted that minimax lower bounds are in essence quantifying worst cases of a given class of problems.

Original: In particular, to prove Theorem 2, we consider distributions that lead to hard problems; more specifically, we assumed the variance of the conditional distribution ( 𝑌 | 𝑋 ) to be high.
Revised: In particular, to prove Theorem 2, we consider distributions that lead to hard problems; more specifically, we assumed the variance of the conditional distribution ( 𝑌 | 𝑋 ) to be high.

Original: The practitioner should keep in mind that it is possible to add additional structure on the solution, leverage active learning or semi-supervised strategy such as uncertainty sampling (Nguyen et al., 2021), or Laplacian regularization (Zhu et al., 2003; Cabannes et al., 2021a), and reduce the optimal rates of convergence.
Revised: The practitioner should keep in mind that it is possible to add additional structure on the solution, leverage active learning or semi-supervised strategy such as uncertainty sampling (Nguyen et al., 2021), or Laplacian regularization (Zhu et al., 2003; Cabannes et al., 2021a), and reduce the optimal rates of convergence.

Original: To conclude this section, let us remark that most of our derivations could easily be refined for practitioners facing a slightly different cost model for annotation.
Revised: To conclude this section, let us remark that most of our derivations could easily be refined for practitioners facing a slightly different cost model for annotation.

Original: In particular, they might prefer to perform batches of annotations before updating 𝜃 rather than modifying the question strategy after each input annotation.
Revised: In particular, they might prefer to perform batches of annotations before updating 𝜃 rather than modifying the question strategy after each input annotation.

Original: This would be similar to mini-batching in gradient descent.
Revised: This would be similar to mini-batching in gradient descent.

Original: Indeed, the dependency of our result on the annotation cost model and on Assumption 1 should not be seen as a limitation but rather as a proof of concept.
Revised: Indeed, the dependency of our result on the annotation cost model and on Assumption 1 should not be seen as a limitation but rather as a proof of concept.

Original:  Numerical analysis
Revised:  Numerical analysis

Original: In this section, we illustrate the differences between our active method versus a classical passive method, for regression and classification problems.
Revised: In this section, we illustrate the differences between our active method versus a classical passive method, for regression and classification problems.

Original: Further discussions are provided in Appendix E.
Revised: Further discussions are provided in Appendix E.

Original: Our code is available online at https://github.com/VivienCabannes/active-labeling .
Revised: Our code is available online at https://github.com/VivienCabannes/active-labeling .

Original: Let us begin with the regression problem that consists in estimating the function 𝑓 ∗ that maps 𝑥 ∈ [ 0 , 1 ] to sin ( 2 𝜋𝑥 ) ∈ R .
Revised: Let us begin with the regression problem that consists in estimating the function 𝑓 ∗ that maps 𝑥 ∈ [ 0 , 1 ] to sin ( 2 𝜋𝑥 ) ∈ R .

Original: Such a regular function, which belongs to any Hölder or Sobolev classes of functions, can be estimated with the Gaussian kernel, which would ensure Assumption 1, and that corresponds to a feature map 𝜑
Revised: Such a regular function, which belongs to any Hölder or Sobolev classes of functions, can be estimated with the Gaussian kernel, which would ensure Assumption 1, and that corresponds to a feature map 𝜑

Original: such that 𝑘 ( 𝑥, 𝑥 ′ ) :
Revised: such that 𝑘 ( 𝑥, 𝑥 ′ ) :

Original: = ⟨ 𝜑 ( 𝑥 ) , 𝜑 ( 𝑥 ′ )⟩ = exp (− | 𝑥 − 𝑥 ′ | /( 2 𝜎 2 ))
Revised: = ⟨ 𝜑 ( 𝑥 ) , 𝜑 ( 𝑥 ′ )⟩ = exp (− | 𝑥 − 𝑥 ′ | /( 2 𝜎 2 ))

Original: for any bandwidth parameter 𝜎 > 0.
Revised: for any bandwidth parameter 𝜎 > 0.

Original: On Figure 1, we focus on estimating 𝑓 ∗ given data ( 𝑋 𝑖 )
Revised: On Figure 1, we focus on estimating 𝑓 ∗ given data ( 𝑋 𝑖 )

Original: 𝑖 ∈
Revised: 𝑖 ∈

Original: [ 𝑇 ] that are uniform on [ 0 , 1 ] in the noiseless setting where 𝑌 𝑖 = 𝑓 ∗ ( 𝑋 𝑖 ) , based on the minimization of the absolute deviation loss.
Revised: [ 𝑇 ] that are uniform on [ 0 , 1 ] in the noiseless setting where 𝑌 𝑖 = 𝑓 ∗ ( 𝑋 𝑖 ) , based on the minimization of the absolute deviation loss.

Original: The passive baseline consists in randomly choosing a threshold 𝑈 𝑖 ∼ N ( 0 , 1 ) and acquiring the observations ( 1 𝑌 𝑖 >𝑈 𝑖 )
Revised: The passive baseline consists in randomly choosing a threshold 𝑈 𝑖 ∼ N ( 0 , 1 ) and acquiring the observations ( 1 𝑌 𝑖 >𝑈 𝑖 )

Original: 𝑖 ∈
Revised: 𝑖 ∈

Original: [ 𝑇 ] that can be cast as the observation of the half-space 𝑆 𝑖 = (cid:8) 𝑦 ∈ Y (cid:12)(cid:12) 1 𝑦>𝑈 𝑖
Revised: [ 𝑇 ] that can be cast as the observation of the half-space 𝑆 𝑖 = (cid:8) 𝑦 ∈ Y (cid:12)(cid:12) 1 𝑦>𝑈 𝑖

Original: = 1 𝑌 𝑖 >𝑈 𝑖 (cid:9) = :
Revised: = 1 𝑌 𝑖 >𝑈 𝑖 (cid:9) = :

Original: 𝑠 ( 𝑌 𝑖 ,𝑈 𝑖 ) .
Revised: 𝑠 ( 𝑌 𝑖 ,𝑈 𝑖 ) .

Original: In this noiseless setting, a good baseline to learn 𝑓 ∗ from the data ( 𝑋 𝑖 , 𝑆 𝑖 ) is provided by the infimum loss characterization (see Cabannes et al., 2020)
Revised: In this noiseless setting, a good baseline to learn 𝑓 ∗ from the data ( 𝑋 𝑖 , 𝑆 𝑖 ) is provided by the infimum loss characterization (see Cabannes et al., 2020)

Original: [Equation]
Revised: [Equation]

Original: where the distribution over 𝑋 corresponds to the marginal of 𝜌 over X , and the distribution over ( 𝑆 | 𝑋 = 𝑥 ) is the pushforward of 𝑈 ∼ N ( 0 , 1 ) under 𝑠 ( 𝑓 ∗ ( 𝑥 ) , ·) .
Revised: where the distribution over 𝑋 corresponds to the marginal of 𝜌 over X , and the distribution over ( 𝑆 | 𝑋 = 𝑥 ) is the pushforward of 𝑈 ∼ N ( 0 , 1 ) under 𝑠 ( 𝑓 ∗ ( 𝑥 ) , ·) .

Original: The left plot on Figure 1 corresponds to an instance of SGD on such an objective based on the data ( 𝑋 𝑖 , 𝑆 𝑖 ) , while the right plot corresponds to Algorithm 1.
Revised: The left plot on Figure 1 corresponds to an instance of SGD on such an objective based on the data ( 𝑋 𝑖 , 𝑆 𝑖 ) , while the right plot corresponds to Algorithm 1.

Original: We take the same hyperparameters for both plots, a bandwidth 𝜎 = 0 .
Revised: We take the same hyperparameters for both plots, a bandwidth 𝜎 = 0 .

Original: 2 and an SGD step size 𝛾 = 0 .
Revised: 2 and an SGD step size 𝛾 = 0 .

Original: We refer the curious reader to Figure 7 in Appendix E for plots illustrating the streaming history, and to Figure 10 for “real-world” experiments.
Revised: We refer the curious reader to Figure 7 in Appendix E for plots illustrating the streaming history, and to Figure 10 for “real-world” experiments.

Original: To illustrate the versatility of our method, we approach a classification problem through the median surrogate technique presented in Proposition 3.
Revised: To illustrate the versatility of our method, we approach a classification problem through the median surrogate technique presented in Proposition 3.

Original: To do so, we consider the classification problem
Revised: To do so, we consider the classification problem

Original: [Figure]
Revised: [Figure]

Original: Above
Revised: Above

Original: Below
Revised: Below

Original: [Figure]
Revised: [Figure]

Original: with 𝑚 ∈ N classes
Revised: with 𝑚 ∈ N classes

Original: [Figure]
Revised: [Figure]

Original: X = [ 0 , 1 ] and the conditional distribution ( 𝑌 | 𝑋 ) linearly interpolating between
Revised: X = [ 0 , 1 ] and the conditional distribution ( 𝑌 | 𝑋 ) linearly interpolating between

Original: Dirac in 𝑦 1 , 𝑦 2 and 𝑦 3 respectively for 𝑥 = 0, 𝑥 = 1 / 2 and 𝑥 = 1 and the uniform distribution for 𝑥 = 1 / 4 and 𝑥 = 3 / 4; and 𝑋 uniform on X \ ([ 1 / 4 − 𝜀, 1 / 4 + 𝜀 ] ∪ [ 3 / 4 − 𝜀, 3 / 4 + 𝜀 ]) .
Revised: Dirac in 𝑦 1 , 𝑦 2 and 𝑦 3 respectively for 𝑥 = 0, 𝑥 = 1 / 2 and 𝑥 = 1 and the uniform distribution for 𝑥 = 1 / 4 and 𝑥 = 3 / 4; and 𝑋 uniform on X \ ([ 1 / 4 − 𝜀, 1 / 4 + 𝜀 ] ∪ [ 3 / 4 − 𝜀, 3 / 4 + 𝜀 ]) .

Original: [Figure]
Revised: [Figure]

Original:  Discussion 7.1 Discrete output problems
Revised:  Discussion 7.1 Discrete output problems

Original: In this section, we discuss casting Algorithm 1 into a procedure to tackle discrete-output problems, by leveraging surrogate regression tasks.
Revised: In this section, we discuss casting Algorithm 1 into a procedure to tackle discrete-output problems, by leveraging surrogate regression tasks.

Original: Learning problems with discrete output spaces are not as well understood as regression problems.
Revised: Learning problems with discrete output spaces are not as well understood as regression problems.

Original: This is a consequence of the complexity of dealing with combinatorial structures in contrast with continuous metric spaces.
Revised: This is a consequence of the complexity of dealing with combinatorial structures in contrast with continuous metric spaces.

Original: In particular, gradients are not defined for discrete output models.
Revised: In particular, gradients are not defined for discrete output models.

Original: The current state-of-the-art framework to deal with discrete output problems is to introduce a continuous surrogate problem whose solution can be decoded as a solution on the original problem (Bartlett et al.,
Revised: The current state-of-the-art framework to deal with discrete output problems is to introduce a continuous surrogate problem whose solution can be decoded as a solution on the original problem (Bartlett et al.,

Original: For example, one could solve a classification task with a median regression surrogate problem, which is the object of the next proposition, proven in Appendix C.
Revised: For example, one could solve a classification task with a median regression surrogate problem, which is the object of the next proposition, proven in Appendix C.

Original: Proposition 3 (Consistency of median surrogate) .
Revised: Proposition 3 (Consistency of median surrogate) .

Original: The classification setting where Y is a finite space, and ℓ : Y × Y → R is the zero-one loss ℓ ( 𝑦, 𝑧 )
Revised: The classification setting where Y is a finite space, and ℓ : Y × Y → R is the zero-one loss ℓ ( 𝑦, 𝑧 )

Original: = 1 𝑦 ≠ 𝑧 can be solved as a regression task through the simplex embedding of Y in R Y with the orthonormal basis ( 𝑒 𝑦 ) 𝑦 ∈ Y .
Revised: = 1 𝑦 ≠ 𝑧 can be solved as a regression task through the simplex embedding of Y in R Y with the orthonormal basis ( 𝑒 𝑦 ) 𝑦 ∈ Y .

Original: More precisely, if 𝑔 ∗ : X → R Y is the minimizer of the median surrogate risk R 𝑆 ( 𝑔 ) =
Revised: More precisely, if 𝑔 ∗ : X → R Y is the minimizer of the median surrogate risk R 𝑆 ( 𝑔 ) =

Original: E [∥ 𝑔 ( 𝑋 ) − 𝑒 𝑌 ∥] , then 𝑓 ∗ : X → Y defined as 𝑓 ∗ ( 𝑥 ) =
Revised: E [∥ 𝑔 ( 𝑋 ) − 𝑒 𝑌 ∥] , then 𝑓 ∗ : X → Y defined as 𝑓 ∗ ( 𝑥 ) =

Original: arg max 𝑦 ∈ Y 𝑔 ∗ 𝑦 ( 𝑥 ) minimizes the original risk R ( 𝑓 ) =
Revised: arg max 𝑦 ∈ Y 𝑔 ∗ 𝑦 ( 𝑥 ) minimizes the original risk R ( 𝑓 ) =

Original: E [ ℓ ( 𝑓 ( 𝑋 ) ,𝑌 )] .
Revised: E [ ℓ ( 𝑓 ( 𝑋 ) ,𝑌 )] .

Original: More generally, any discrete output problem can be solved by reusing the consistent least-squares surrogate of Ciliberto et al.
Revised: More generally, any discrete output problem can be solved by reusing the consistent least-squares surrogate of Ciliberto et al.

Original: Algorithm 1 can be adapted to the least-squares problem based on specifications at the beginning of Section 4.
Revised: Algorithm 1 can be adapted to the least-squares problem based on specifications at the beginning of Section 4.

Original: This allows using our method in an off-the-shelve fashion for all discrete output problems.
Revised: This allows using our method in an off-the-shelve fashion for all discrete output problems.

Original: For example, a problem consisting in ranking preferences over 𝑚 items can be approached with the Kendall correlation loss ℓ ( 𝑦, 𝑧 ) =
Revised: For example, a problem consisting in ranking preferences over 𝑚 items can be approached with the Kendall correlation loss ℓ ( 𝑦, 𝑧 ) =

Original: − 𝜑 ( 𝑦 ) ⊤ 𝜑 ( 𝑧 ) with 𝜑 ( 𝑦 ) =
Revised: − 𝜑 ( 𝑦 ) ⊤ 𝜑 ( 𝑧 ) with 𝜑 ( 𝑦 ) =

Original: ( 1 𝑦 ( 𝑖 ) >𝑦 ( 𝑗 ) ) for 𝑖 < 𝑗 ≤ 𝑚 , where 𝑦 and 𝑧 are permutations over [ 𝑚 ] that encode the rank of each element in terms of user preferences.
Revised: ( 1 𝑦 ( 𝑖 ) >𝑦 ( 𝑗 ) ) for 𝑖 < 𝑗 ≤ 𝑚 , where 𝑦 and 𝑧 are permutations over [ 𝑚 ] that encode the rank of each element in terms of user preferences.

Original: In this setting, the surrogate task introduced by Ciliberto et al.
Revised: In this setting, the surrogate task introduced by Ciliberto et al.

Original: (2020) consists in learning 𝑔 ( 𝑥 ) =
Revised: (2020) consists in learning 𝑔 ( 𝑥 ) =

Original: E [ 𝜑 ( 𝑌 )| 𝑋 = 𝑥 ] as a least-squares problem.
Revised: E [ 𝜑 ( 𝑌 )| 𝑋 = 𝑥 ] as a least-squares problem.

Original: The half-space surrogate queries translate directly into the questions (cid:205)
Revised: The half-space surrogate queries translate directly into the questions (cid:205)

Original: 𝑖<𝑗 ≤ 𝑚 𝑤 ( 𝑖, 𝑗 )
Revised: 𝑖<𝑗 ≤ 𝑚 𝑤 ( 𝑖, 𝑗 )

Original: 1 𝑦 ( 𝑖 ) >𝑦 ( 𝑗 ) > 𝑐 for some ( 𝑤 ( 𝑖, 𝑗 )) , 𝑐 in R .
Revised: 1 𝑦 ( 𝑖 ) >𝑦 ( 𝑗 ) > 𝑐 for some ( 𝑤 ( 𝑖, 𝑗 )) , 𝑐 in R .

Original: In particular, if 𝑈 is chosen to be uniform on the canonical basis (rather than on the sphere), those questions translate into pairwise orderings ( e.g. , does user 𝑥 prefer movie 𝑖 or movie 𝑗 ?).
Revised: In particular, if 𝑈 is chosen to be uniform on the canonical basis (rather than on the sphere), those questions translate into pairwise orderings ( e.g. , does user 𝑥 prefer movie 𝑖 or movie 𝑗 ?).

Original: In terms of guarantee akin to Theorem 1, retaking the calibration inequality of Ciliberto et al.
Revised: In terms of guarantee akin to Theorem 1, retaking the calibration inequality of Ciliberto et al.

Original: (2020), we get convergence rates of the form 𝑚 3 / 2 𝑇 − 1 / 4 .
Revised: (2020), we get convergence rates of the form 𝑚 3 / 2 𝑇 − 1 / 4 .

Original: In terms of guarantee akin to Theorem 2, since we need as least log 2 ( 𝑚 ! )
Revised: In terms of guarantee akin to Theorem 2, since we need as least log 2 ( 𝑚 ! )

Original: ≃ 𝑚 log ( 𝑚 ) binary queries to discriminate between 𝑚 !
Revised: ≃ 𝑚 log ( 𝑚 ) binary queries to discriminate between 𝑚 !

Original: permutations, we can expect a lower bound in 𝑚 1 / 2 log ( 𝑚 ) 1 / 2 𝑇 − 1 / 2 .
Revised: permutations, we can expect a lower bound in 𝑚 1 / 2 log ( 𝑚 ) 1 / 2 𝑇 − 1 / 2 .

Original: More generally, many ranking problems can be approached with correlation losses and tackled through surrogate regression problems on the convex hulls of some well-known polytopes such as the Birkhoff polytope or the permutohedron ( e.g. , Ailon, 2014).
Revised: More generally, many ranking problems can be approached with correlation losses and tackled through surrogate regression problems on the convex hulls of some well-known polytopes such as the Birkhoff polytope or the permutohedron ( e.g. , Ailon, 2014).

Original: Although their descriptions is out-of-scope of this paper, linear cuts of those polytopes form well-structured queries sets – e.g. , the faces of all dimensions of the permutohedron correspond, in a one-to-one fashion, to strict weak orderings (Ziegler, 1995).
Revised: Although their descriptions is out-of-scope of this paper, linear cuts of those polytopes form well-structured queries sets – e.g. , the faces of all dimensions of the permutohedron correspond, in a one-to-one fashion, to strict weak orderings (Ziegler, 1995).

Original: In those discrete settings, Theorem 1 can be refined under low noise conditions.
Revised: In those discrete settings, Theorem 1 can be refined under low noise conditions.

Original: In particular, under generalization of the Massart noise condition, our approach could even exhibit exponential convergence rates as illustrated on Figure 2.
Revised: In particular, under generalization of the Massart noise condition, our approach could even exhibit exponential convergence rates as illustrated on Figure 2.

Original: For classification problems, this condition can be expressed as the existence of a threshold 𝛿 > 0 such that for almost all 𝑥 ∈ X and 𝑧 ∈ Y , we have P ( 𝑌 = 𝑓 ( 𝑥 )| 𝑋 = 𝑥 ) − P
Revised: For classification problems, this condition can be expressed as the existence of a threshold 𝛿 > 0 such that for almost all 𝑥 ∈ X and 𝑧 ∈ Y , we have P ( 𝑌 = 𝑓 ( 𝑥 )| 𝑋 = 𝑥 ) − P

Original: ( 𝑌 = 𝑧 | 𝑋 = 𝑥 ) ∉ ( 0 , 𝛿 ) .
Revised: ( 𝑌 = 𝑧 | 𝑋 = 𝑥 ) ∉ ( 0 , 𝛿 ) .

Original: Arguably, this assumption is met on well-curated images dataset such as ImageNet or CIFAR10, where for each input 𝑋 the most probable class has always more than e.g. 60% of chance to be the target 𝑌 .
Revised: Arguably, this assumption is met on well-curated images dataset such as ImageNet or CIFAR10, where for each input 𝑋 the most probable class has always more than e.g. 60% of chance to be the target 𝑌 .

Original: When this assumption holds together with
Revised: When this assumption holds together with

Original: Assumption 1 (when the surrogate target 𝑔 ∗ belongs to the RKHS and the kernel is bounded), then the right hand-side of equation (6) can be replaced by exp (− 𝑐𝑇 ) for some constant 𝑐 .
Revised: Assumption 1 (when the surrogate target 𝑔 ∗ belongs to the RKHS and the kernel is bounded), then the right hand-side of equation (6) can be replaced by exp (− 𝑐𝑇 ) for some constant 𝑐 .

Original: The proof would be a simple adaptation of Pillaud-Vivien et al.
Revised: The proof would be a simple adaptation of Pillaud-Vivien et al.

Original: (2018a)
Revised: (2018a)

Original: ; Cabannes et al.
Revised: ; Cabannes et al.

Original: (2021b) to our case.
Revised: (2021b) to our case.

Original: 7.2 Supervised learning baseline with resampling
Revised: 7.2 Supervised learning baseline with resampling

Original: In this section, we discuss simple supervised learning baselines that compete with Algorithm 1 when resampling is allowed.
Revised: In this section, we discuss simple supervised learning baselines that compete with Algorithm 1 when resampling is allowed.

Original: When resampling is allowed a simple baseline for the active labeling problem is provided by supervised learning.
Revised: When resampling is allowed a simple baseline for the active labeling problem is provided by supervised learning.

Original: In regression problems with the query of any half-space, a method that consists in annotating each ( 𝑌 𝑖 ) 𝑖 ≤ 𝑛 ( 𝑇,𝜀 ) up to precision 𝜀 , before using any supervised learning method to learn 𝑓 from ( 𝑋 𝑖 ,𝑌 𝑖 ) 𝑖 ≤ 𝑛 ( 𝑇,𝜀 ) could acquire 𝑛 ( 𝑇, 𝜀 ) ≃ 𝑇 / 𝑚 log 2 ( 𝜀 − 1 ) data points with a dichotomic search along all directions, assuming 𝑌 𝑖 bounded or sub-Gaussian.
Revised: In regression problems with the query of any half-space, a method that consists in annotating each ( 𝑌 𝑖 ) 𝑖 ≤ 𝑛 ( 𝑇,𝜀 ) up to precision 𝜀 , before using any supervised learning method to learn 𝑓 from ( 𝑋 𝑖 ,𝑌 𝑖 ) 𝑖 ≤ 𝑛 ( 𝑇,𝜀 ) could acquire 𝑛 ( 𝑇, 𝜀 ) ≃ 𝑇 / 𝑚 log 2 ( 𝜀 − 1 ) data points with a dichotomic search along all directions, assuming 𝑌 𝑖 bounded or sub-Gaussian.

Original: In terms of minimax rates, such a procedure cannot perform better than in 𝑛 ( 𝑇, 𝜀 ) − 1 / 2 + 𝜀 , the first term being due to the statistical limit in Theorem 2, the second due to the incertitude 𝜀 on each 𝑌 𝑖 that transfers to the same level of incertitude on 𝑓 .
Revised: In terms of minimax rates, such a procedure cannot perform better than in 𝑛 ( 𝑇, 𝜀 ) − 1 / 2 + 𝜀 , the first term being due to the statistical limit in Theorem 2, the second due to the incertitude 𝜀 on each 𝑌 𝑖 that transfers to the same level of incertitude on 𝑓 .

Original: Optimizing with respect to 𝜀 yields a bound in 𝑂 ( 𝑇 − 1 / 2 log ( 𝑇 ) 1 / 2 ) .
Revised: Optimizing with respect to 𝜀 yields a bound in 𝑂 ( 𝑇 − 1 / 2 log ( 𝑇 ) 1 / 2 ) .

Original: Therefore, this not-so-naive baseline is only suboptimal by a factor log ( 𝑇 ) 1 / 2 .
Revised: Therefore, this not-so-naive baseline is only suboptimal by a factor log ( 𝑇 ) 1 / 2 .

Original: In the meanwhile, Algorithm 1 can be rewritten with resampling, as well as Theorem 1, which we prove in Appendix A.2.
Revised: In the meanwhile, Algorithm 1 can be rewritten with resampling, as well as Theorem 1, which we prove in Appendix A.2.

Original: Hence, our technique will still achieve minimax optimality for the problem “with resampling”.
Revised: Hence, our technique will still achieve minimax optimality for the problem “with resampling”.

Original: In other terms, by deciding to acquire more imprecise information, our algorithm reduces annotation cost for a given level of generalization error (or equivalently reduces generalization error for a given annotation budget) by a factor log ( 𝑇 ) 1 / 2 when compared to this baseline.
Revised: In other terms, by deciding to acquire more imprecise information, our algorithm reduces annotation cost for a given level of generalization error (or equivalently reduces generalization error for a given annotation budget) by a factor log ( 𝑇 ) 1 / 2 when compared to this baseline.

Original: The picture is slightly different for discrete-output problems.
Revised: The picture is slightly different for discrete-output problems.

Original: If one can ask any question 𝑠 ∈ 2 Y then with a dichotomic search, one can retrieve any label with log 2 ( 𝑚 ) questions.
Revised: If one can ask any question 𝑠 ∈ 2 Y then with a dichotomic search, one can retrieve any label with log 2 ( 𝑚 ) questions.

Original: Hence, to theoretically beat the fully supervised baseline with the SGD method described in Section 3, one would have to derive a gradient strategy (2) with a small enough second moment ( e.g. , for convex losses that are non-smooth nor strongly convex, the increase in the second moment compared to the usual stochastic gradients should be no greater than log 2 ( 𝑚 ) 1 / 2 ).
Revised: Hence, to theoretically beat the fully supervised baseline with the SGD method described in Section 3, one would have to derive a gradient strategy (2) with a small enough second moment ( e.g. , for convex losses that are non-smooth nor strongly convex, the increase in the second moment compared to the usual stochastic gradients should be no greater than log 2 ( 𝑚 ) 1 / 2 ).

Original: How to best refine our technique to better take into account the discrete structure of the output space is an open question.
Revised: How to best refine our technique to better take into account the discrete structure of the output space is an open question.

Original: Introducing bias that does not modify convergence properties while reducing variance eventually thanks to importance sampling is a potential way to approach this problem.
Revised: Introducing bias that does not modify convergence properties while reducing variance eventually thanks to importance sampling is a potential way to approach this problem.

Original: A simpler idea would be to remember information of the type 𝑌 𝑖 ∈ 𝑠 to restrict the questions asked in order to locate 𝑓 𝜃 𝑡 ( 𝑋 𝑖 ) − 𝑌 𝑖 when performing stochastic gradient descent with resampling.
Revised: A simpler idea would be to remember information of the type 𝑌 𝑖 ∈ 𝑠 to restrict the questions asked in order to locate 𝑓 𝜃 𝑡 ( 𝑋 𝑖 ) − 𝑌 𝑖 when performing stochastic gradient descent with resampling.

Original: Combinatorial bandits might also provide helpful insights on the matter.
Revised: Combinatorial bandits might also provide helpful insights on the matter.

Original: Ultimately, we would like to build an understanding of the whole distribution ( 𝑌 | 𝑋 ) and not only of 𝑓 ∗ ( 𝑋 ) as we explore labels in order to refine this exploration.
Revised: Ultimately, we would like to build an understanding of the whole distribution ( 𝑌 | 𝑋 ) and not only of 𝑓 ∗ ( 𝑋 ) as we explore labels in order to refine this exploration.

Original: 7.3 Min-max approaches
Revised: 7.3 Min-max approaches

Original: In this section, we discuss potential extensions of our SGD procedure, based on min-max variational objectives.
Revised: In this section, we discuss potential extensions of our SGD procedure, based on min-max variational objectives.

Original: Min-max approaches have been popularized for searching games and active learning, where one searches for the question that minimizes the size of the space where a potential guess could lie under the worst possible answer to that question.
Revised: Min-max approaches have been popularized for searching games and active learning, where one searches for the question that minimizes the size of the space where a potential guess could lie under the worst possible answer to that question.

Original: A particularly well illustrative example is the solution of the Mastermind game proposed by Knuth (1977).
Revised: A particularly well illustrative example is the solution of the Mastermind game proposed by Knuth (1977).

Original: While our work leverages plain SGD, one could build on the vector field point-of-view of gradient descent (see, e.g. , Bubeck, 2015) to tackle min-max convex concave problems with similar guarantees.
Revised: While our work leverages plain SGD, one could build on the vector field point-of-view of gradient descent (see, e.g. , Bubeck, 2015) to tackle min-max convex concave problems with similar guarantees.

Original: In particular, we could design weakly supervised losses 𝐿 ( 𝑓 ( 𝑥 ) , 𝑠 ;
Revised: In particular, we could design weakly supervised losses 𝐿 ( 𝑓 ( 𝑥 ) , 𝑠 ;

Original: 1 𝑦 ∈ 𝑠 ) and min-max games where a prediction player aims at minimizing such a loss with respect to the prediction 𝑓 , while the query player aims at maximizing it with respect to the question 𝑠 , that is querying information that best elicit mistakes made by the prediction player.
Revised: 1 𝑦 ∈ 𝑠 ) and min-max games where a prediction player aims at minimizing such a loss with respect to the prediction 𝑓 , while the query player aims at maximizing it with respect to the question 𝑠 , that is querying information that best elicit mistakes made by the prediction player.

Original: For example, the dual norm characterization of the norm leads to the following min-max approach to the median regression
Revised: For example, the dual norm characterization of the norm leads to the following min-max approach to the median regression

Original: [Equation]
Revised: [Equation]

Original: Such min-max formulations would be of interest if they lead to improvement of computational and statistical efficiencies, similarly to the work of Babichev et al.
Revised: Such min-max formulations would be of interest if they lead to improvement of computational and statistical efficiencies, similarly to the work of Babichev et al.

Original: For classification problems, the following proposition introduces such a game and suggests its suitability.
Revised: For classification problems, the following proposition introduces such a game and suggests its suitability.

Original: Its proof can be found in
Revised: Its proof can be found in

Original: Appendix D.
Revised: Appendix D.

Original: Proposition 4.
Revised: Proposition 4.

Original: Consider the classification problem of learning 𝑓 ∗ : X → Y where Y is of finite cardinality, with the 0-1 loss ℓ ( 𝑧, 𝑦 )
Revised: Consider the classification problem of learning 𝑓 ∗ : X → Y where Y is of finite cardinality, with the 0-1 loss ℓ ( 𝑧, 𝑦 )

Original: = 1 𝑧 ≠ 𝑦 , minimizing the risk (1) under a distribution 𝜌 on X × Y .
Revised: = 1 𝑧 ≠ 𝑦 , minimizing the risk (1) under a distribution 𝜌 on X × Y .

Original: Introduce the surrogate score functions 𝑔 : X → Δ Y ; 𝑥
Revised: Introduce the surrogate score functions 𝑔 : X → Δ Y ; 𝑥

Original: → 𝑣 where 𝑣 = ( 𝑣 𝑦 ) 𝑦
Revised: → 𝑣 where 𝑣 = ( 𝑣 𝑦 ) 𝑦

Original: ∈ Y is a family of non-negative weights that sum to one, as well as the surrogate loss function 𝐿 : Δ Y × S × {− 1 , 1 } → R ; ( 𝑣, 𝑆, 𝜀 ) =
Revised: ∈ Y is a family of non-negative weights that sum to one, as well as the surrogate loss function 𝐿 : Δ Y × S × {− 1 , 1 } → R ; ( 𝑣, 𝑆, 𝜀 ) =

Original: 𝜀 ( 1 − 2 (cid:205) 𝑦 ∈ 𝑆 𝑣 𝑦 ) , and the min-max game
Revised: 𝜀 ( 1 − 2 (cid:205) 𝑦 ∈ 𝑆 𝑣 𝑦 ) , and the min-max game

Original: [Equation]
Revised: [Equation]

Original: When S contains the singletons and with the low-noise condition that P ( 𝑌 ≠ 𝑓 ∗ ( 𝑥 ) | 𝑋 = 𝑥 ) < 1 /almost everywhere, then 𝑓 ∗ can be learned through the relation 𝑓 ∗ ( 𝑥 ) =
Revised: When S contains the singletons and with the low-noise condition that P ( 𝑌 ≠ 𝑓 ∗ ( 𝑥 ) | 𝑋 = 𝑥 ) < 1 /almost everywhere, then 𝑓 ∗ can be learned through the relation 𝑓 ∗ ( 𝑥 ) =

Original: arg min 𝑦 ∈ Y 𝑔 ∗ ( 𝑥 ) 𝑦 for the unique minimizer 𝑔 ∗ of (8) .
Revised: arg min 𝑦 ∈ Y 𝑔 ∗ ( 𝑥 ) 𝑦 for the unique minimizer 𝑔 ∗ of (8) .

Original: Moreover, the minimization of the empirical version of this objective with the stochastic gradient updates for saddle point problems provides a natural “active labeling” scheme to find this 𝑔 ∗ .
Revised: Moreover, the minimization of the empirical version of this objective with the stochastic gradient updates for saddle point problems provides a natural “active labeling” scheme to find this 𝑔 ∗ .

Original: On the one hand, this min-max formulation could help to easily incorporate restrictions on the sets to query.
Revised: On the one hand, this min-max formulation could help to easily incorporate restrictions on the sets to query.

Original: On the other hand, it is not completely clear how to best update (or derive an unbiased stochastic gradient strategy for) the adversarial query strategy 𝜇 based on partial information.
Revised: On the other hand, it is not completely clear how to best update (or derive an unbiased stochastic gradient strategy for) the adversarial query strategy 𝜇 based on partial information.

Original:  Conclusion
Revised:  Conclusion

Original: We have introduced the “active labeling” problem, which corresponds to “active partially supervised learning”.
Revised: We have introduced the “active labeling” problem, which corresponds to “active partially supervised learning”.

Original: We provided a solution to this problem based on stochastic gradient descent.
Revised: We provided a solution to this problem based on stochastic gradient descent.

Original: Although our method can be used for any discrete output problem, we detailed how it works for median regression, where we show that it optimizes the generalization error for a given annotation budget.
Revised: Although our method can be used for any discrete output problem, we detailed how it works for median regression, where we show that it optimizes the generalization error for a given annotation budget.

Original: In a near future, we would like to focus on better exploiting the discrete structure of classification problems, eventually with resampling strategies.
Revised: In a near future, we would like to focus on better exploiting the discrete structure of classification problems, eventually with resampling strategies.

Original: Understanding more precisely the key issues in applications concerned with privacy, and studying how weak gradients might provide a good trade-off between learning efficiently and revealing too much information also provide interesting follow-ups.
Revised: Understanding more precisely the key issues in applications concerned with privacy, and studying how weak gradients might provide a good trade-off between learning efficiently and revealing too much information also provide interesting follow-ups.

Original: Finally, regarding dataset annotation, exploring different paradigms of weakly supervised learning would lead to different active weakly supervised learning frameworks.
Revised: Finally, regarding dataset annotation, exploring different paradigms of weakly supervised learning would lead to different active weakly supervised learning frameworks.

Original: While this work is based on partial labeling, similar formalization could be made based on other weak supervision models, such as aggregation ( e.g. , Ratner et al., 2020), or group statistics (Dietterich et al., 1997).
Revised: While this work is based on partial labeling, similar formalization could be made based on other weak supervision models, such as aggregation ( e.g. , Ratner et al., 2020), or group statistics (Dietterich et al., 1997).

Original: In particular, annotating a huge dataset is often done by bagging inputs according to predicted labels and correcting errors that can be spotted on those bags of inputs (Deng et al., 2009).
Revised: In particular, annotating a huge dataset is often done by bagging inputs according to predicted labels and correcting errors that can be spotted on those bags of inputs (Deng et al., 2009).

Original: We left for future work the study of variants of the “active labeling” problem that model those settings.
Revised: We left for future work the study of variants of the “active labeling” problem that model those settings.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: This work was funded in part by the French government under management of Agence Nationale de la Recherche as part of the “Investissements d’avenir” program, reference ANR-19-P3IA- (PRAIRIE 3IA Institute).
Revised: While at INRIA / ENS / PSL, VC was funded in part by the French government under management of Agence Nationale de la Recherche as part of the “Investissements d’avenir” program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA Institute).

Original: We also acknowledge support of the EuropeanResearch Council (grants
Revised: FR and AR also acknowledges support of the European

Original:  SEQUOIA 724063 and REAL 947908).
Revised: Research Council (grants SEQUOIA 724063 and REAL 947908).

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See discussion section.
Revised: [Yes] See discussion section.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] This work aims at developping advanced techniques to learn without too much supervision.
Revised: [N/A] This work aims at developping advanced techniques to learn without too much supervision.

Original: Such a quest of increasing AI systems capability at a reduced human labor cost is associated with broad societal issues.
Revised: Such a quest of increasing AI systems capability at a reduced human labor cost is associated with broad societal issues.

Original: Those questions being really generic, we did not mention them in the main text.
Revised: Those questions being really generic, we did not mention them in the main text.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] 3.
Revised: [Yes] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [N/A]
Revised: [N/A]

Original: The experiments were run on a personal laptop and did not require many charges.
Revised: The experiments were run on a personal laptop and did not require many charges.

Original: Indeed, the amount of compute for experiments were similar to the amount used to write this paper.
Revised: Indeed, the amount of compute for experiments were similar to the amount used to write this paper.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] Although we have not cited the creators of some L A TEX libraries we used such as Michael Sharpe and the newtx package which we used for fonts in our text.
Revised: [Yes] Although we have not cited the creators of some L A TEX libraries we used such as Michael Sharpe and the newtx package which we used for fonts in our text.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A] Numpy and LIBSVM are under Berkeley Software Distribution licenses (respectively the liberal and revised ones), Python and matplotlib are under the Python Software Foundation license.
Revised: [N/A] Numpy and LIBSVM are under Berkeley Software Distribution licenses (respectively the liberal and revised ones), Python and matplotlib are under the Python Software Foundation license.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Proofs of the statistical analysis
Revised: A Proofs of the statistical analysis

Original: In the following proofs, we assume X to be Polish and Y
Revised: In the following proofs, we assume X to be Polish and Y

Original: =
Revised: =

Original: R 𝑚 , so to define the joint probability 𝜌 ∈ Δ
Revised: R 𝑚 , so to define the joint probability 𝜌 ∈ Δ

Original: X × Y .
Revised: X × Y .

Original: Moreover, we assume that E [∥ 𝑌 ∥] < +∞ in order to define the risk of median regression.
Revised: Moreover, we assume that E [∥ 𝑌 ∥] < +∞ in order to define the risk of median regression.

Original: We consider H to be a Hilbert space that is separable ( i.e. only the origin is in all the neighborhood of the origin), and 𝜑 to be a measurable mapping from X to H .
Revised: We consider H to be a Hilbert space that is separable ( i.e. only the origin is in all the neighborhood of the origin), and 𝜑 to be a measurable mapping from X to H .

Original: In terms of notations, we denote { 1 , 2 , · · · , 𝑛 } by [ 𝑛 ] for any 𝑛 ∈ N ∗ , and by ( 𝑥 𝑖 ) 𝑖 ≤ 𝑛 the family ( 𝑥 1 , · · · , 𝑥 𝑛 ) for any sequence ( 𝑥 𝑖 ) .
Revised: In terms of notations, we denote { 1 , 2 , · · · , 𝑛 } by [ 𝑛 ] for any 𝑛 ∈ N ∗ , and by ( 𝑥 𝑖 ) 𝑖 ≤ 𝑛 the family ( 𝑥 1 , · · · , 𝑥 𝑛 ) for any sequence ( 𝑥 𝑖 ) .

Original: The unit sphere in R 𝑚 is denoted by S 𝑚 − 1 .
Revised: The unit sphere in R 𝑚 is denoted by S 𝑚 − 1 .

Original: The symbol ⊗ denotes tensors, and is extended to product measures in the notation 𝜌 ⊗ 𝑛 = 𝜌 × 𝜌 × · · · × 𝜌 .
Revised: The symbol ⊗ denotes tensors, and is extended to product measures in the notation 𝜌 ⊗ 𝑛 = 𝜌 × 𝜌 × · · · × 𝜌 .

Original: We have used the isometry between trace-class linear mappings from H to Y and the tensor space Y ⊗ H , which generalizes the matrix representation of linear map between two finite-dimensional vector spaces.
Revised: We have used the isometry between trace-class linear mappings from H to Y and the tensor space Y ⊗ H , which generalizes the matrix representation of linear map between two finite-dimensional vector spaces.

Original: This space inherits from the Hilbertian structure of H and Y and we denote by ∥·∥ the Hilbertian norm that generalizes the Frobenius norm on linear maps between Euclidean spaces.
Revised: This space inherits from the Hilbertian structure of H and Y and we denote by ∥·∥ the Hilbertian norm that generalizes the Frobenius norm on linear maps between Euclidean spaces.

Original: A.1 Upper bound for stochastic gradient descent
Revised: A.1 Upper bound for stochastic gradient descent

Original: This subsection is devoted to the proof of Theorem 1.
Revised: This subsection is devoted to the proof of Theorem 1.

Original: For simplicity, we will work with the rescaled step size 𝛾 𝑡 : = 𝑐 2 𝛾 ( 𝑡 ) rather than the step size described in the main text 𝛾 ( 𝑡 ) .
Revised: For simplicity, we will work with the rescaled step size 𝛾 𝑡 : = 𝑐 2 𝛾 ( 𝑡 ) rather than the step size described in the main text 𝛾 ( 𝑡 ) .

Original: Convergence of stochastic gradient descent for non-smooth problems is a known result.
Revised: Convergence of stochastic gradient descent for non-smooth problems is a known result.

Original: For completeness, we reproduce and adapt a usual proof to our setting.
Revised: For completeness, we reproduce and adapt a usual proof to our setting.

Original: For 𝑡 ∈ N , let us introduce the random functions
Revised: For 𝑡 ∈ N , let us introduce the random functions

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: for ( 𝑋 𝑡 ,𝑌 𝑡 ) ∼ 𝜌 , 𝑈 𝑡 uniform on the sphere S 𝑚 − 1 ⊂ Y .
Revised: for ( 𝑋 𝑡 ,𝑌 𝑡 ) ∼ 𝜌 , 𝑈 𝑡 uniform on the sphere S 𝑚 − 1 ⊂ Y .

Original: Those random functions all average to
Revised: Those random functions all average to

Original: R ( 𝜃 ) =
Revised: R ( 𝜃 ) =

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: E 𝑈 [ 𝑐
Revised: E 𝑈 [ 𝑐

Original: [Equation]
Revised: [Equation]

Original: 12 |
Revised: 12 |

Original: [Equation]
Revised: [Equation]

Original: 𝜃𝜑 ( 𝑋 ) − 𝑌,𝑈 ⟩|] =
Revised: 𝜃𝜑 ( 𝑋 ) − 𝑌,𝑈 ⟩|] =

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: [∥ 𝜃𝜑 ( 𝑋 ) − 𝑌 ∥] .
Revised: [∥ 𝜃𝜑 ( 𝑋 ) − 𝑌 ∥] .

Original: After a random initialization 𝜃 0 ∈ Θ , the stochastic gradient update rule can be written for any 𝑡 ∈ N as
Revised: After a random initialization 𝜃 0 ∈ Θ , the stochastic gradient update rule can be written for any 𝑡 ∈ N as

Original: [Equation]
Revised: [Equation]

Original: where ∇ R 𝑡 denotes any sub-gradients of R 𝑡 .
Revised: where ∇ R 𝑡 denotes any sub-gradients of R 𝑡 .

Original: We can compute
Revised: We can compute

Original: [Equation]
Revised: [Equation]

Original: This corresponds to the gradient written in Algorithm 1.
Revised: This corresponds to the gradient written in Algorithm 1.

Original: Let us now express the recurrence relation on ∥ 𝜃 𝑡 + 1 − 𝜃 ∗ ∥ .
Revised: Let us now express the recurrence relation on ∥ 𝜃 𝑡 + 1 − 𝜃 ∗ ∥ .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Because R 𝑡 is convex, it is above its tangents
Revised: Because R 𝑡 is convex, it is above its tangents

Original: [Equation]
Revised: [Equation]

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: This allows bounding the excess of risk as
Revised: This allows bounding the excess of risk as

Original: [Equation]
Revised: [Equation]

Original: where we used the fact that ∥∇ R 𝑡 ∥ = 𝑐 − 12 ∥ 𝜑 ( 𝑋 𝑡 )∥ .
Revised: where we used the fact that ∥∇ R 𝑡 ∥ = 𝑐 − 12 ∥ 𝜑 ( 𝑋 𝑡 )∥ .

Original: Let us multiply this inequality by 𝜂 𝑡 > 0 and sum from 𝑡 = 0 to 𝑡 = 𝑇 − 1, we get
Revised: Let us multiply this inequality by 𝜂 𝑡 > 0 and sum from 𝑡 = 0 to 𝑡 = 𝑇 − 1, we get

Original: [Equation]
Revised: [Equation]

Original: From here, there is several options to obtain a convergence result, either one assume ∥ 𝜃 𝑡 − 𝜃 ∗ ∥ bounded and take 𝜂 𝑡 𝛾 𝑡 − 1 ≥ 𝜂 𝑡 − 1 𝛾 𝑡 ; or one take 𝜂 𝑡 = 𝛾 𝑡 but at the price of paying an extra log ( 𝑇 ) factor in the bound; or one take 𝛾 𝑡 and 𝜂 𝑡 independent of 𝑡 .
Revised: From here, there is several options to obtain a convergence result, either one assume ∥ 𝜃 𝑡 − 𝜃 ∗ ∥ bounded and take 𝜂 𝑡 𝛾 𝑡 − 1 ≥ 𝜂 𝑡 − 1 𝛾 𝑡 ; or one take 𝜂 𝑡 = 𝛾 𝑡 but at the price of paying an extra log ( 𝑇 ) factor in the bound; or one take 𝛾 𝑡 and 𝜂 𝑡 independent of 𝑡 .

Original: Since we suppose the annotation budget given, we will choose 𝛾 𝑡 and 𝜂 𝑡 independent of 𝑡 , only depending on 𝑇 .
Revised: Since we suppose the annotation budget given, we will choose 𝛾 𝑡 and 𝜂 𝑡 independent of 𝑡 , only depending on 𝑇 .

Original: [Equation]
Revised: [Equation]

Original: Let now take the expectation with respect to all the random variables, for the risk
Revised: Let now take the expectation with respect to all the random variables, for the risk

Original: [Equation]
Revised: [Equation]

Original: For the variance, E [∥ 𝜑 ( 𝑋 𝑠 )∥ 2 ] =
Revised: For the variance, E [∥ 𝜑 ( 𝑋 𝑠 )∥ 2 ] =

Original: E [∥ 𝜑 ( 𝑋 )∥ 2 ] = 𝜅 2 .
Revised: E [∥ 𝜑 ( 𝑋 )∥ 2 ] = 𝜅 2 .

Original: Let us fix 𝑇 and consider 𝜂 𝑡 = 1 / 𝑇 , by Jensen we can bound the following averaging
Revised: Let us fix 𝑇 and consider 𝜂 𝑡 = 1 / 𝑇 , by Jensen we can bound the following averaging

Original: [Equation]
Revised: [Equation]

Original: Initializing 𝜃 0 to zero, we can optimize the resulting quantity to get the desired result.
Revised: Initializing 𝜃 0 to zero, we can optimize the resulting quantity to get the desired result.

Original: A.2 Upper bound for resampling strategy
Revised: A.2 Upper bound for resampling strategy

Original: For resampling strategies, the proof is built on classical statistical learning theory considerations.
Revised: For resampling strategies, the proof is built on classical statistical learning theory considerations.

Original: Let us decompose the risk between estimation and optimization errors.
Revised: Let us decompose the risk between estimation and optimization errors.

Original: Recall the expression of the risk
Revised: Recall the expression of the risk

Original: R , the function taking as inputs measurable functions from X to Y and outputting a real number
Revised: R , the function taking as inputs measurable functions from X to Y and outputting a real number

Original: [Equation]
Revised: [Equation]

Original: Let us denote by F the class of functions from X to Y we are going to work with.
Revised: Let us denote by F the class of functions from X to Y we are going to work with.

Original: Let 𝑓 𝑛 be our estimate of 𝑓 ∗ which maps almost every 𝑥 ∈ X to the geometric median of ( 𝑌 | 𝑋 ) .
Revised: Let 𝑓 𝑛 be our estimate of 𝑓 ∗ which maps almost every 𝑥 ∈ X to the geometric median of ( 𝑌 | 𝑋 ) .

Original: Denote by R ∗ D 𝑛 the best value that can be achieved by our class of functions to minimize the empirical average absolute deviation
Revised: Denote by R ∗ D 𝑛 the best value that can be achieved by our class of functions to minimize the empirical average absolute deviation

Original: [Equation]
Revised: [Equation]

Original: Assumption 1 states that we have a well-specified model F to estimate the median, i.e. 𝑓 ∗ ∈ F .
Revised: Assumption 1 states that we have a well-specified model F to estimate the median, i.e. 𝑓 ∗ ∈ F .

Original: Hence, the excess of risk can be decomposed as an estimation and an optimization error, without approximation error (it is not difficult to add an approximation error, but it will make the derivations longer and the convergence rates harder to parse for the reader).
Revised: Hence, the excess of risk can be decomposed as an estimation and an optimization error, without approximation error (it is not difficult to add an approximation error, but it will make the derivations longer and the convergence rates harder to parse for the reader).

Original: Using the fact that R D 𝑛 ( 𝑓 ∗ ) ≥ R ∗ D 𝑛 by definition of the infimum, we have
Revised: Using the fact that R D 𝑛 ( 𝑓 ∗ ) ≥ R ∗ D 𝑛 by definition of the infimum, we have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: optimization error .
Revised: optimization error .

Original: [Equation]
Revised: [Equation]

Original: Estimation error.
Revised: Estimation error.

Original: Let us begin by controlling the estimation error.
Revised: Let us begin by controlling the estimation error.

Original: We have two terms in it.
Revised: We have two terms in it.

Original: R D 𝑛 ( 𝑓 ∗ ) − R ( 𝑓 ∗ ) can be controlled with a concentration inequality on the empirical average of ∥ 𝑓 ∗ ( 𝑋 ) − 𝑌 ∥ around its population mean.
Revised: R D 𝑛 ( 𝑓 ∗ ) − R ( 𝑓 ∗ ) can be controlled with a concentration inequality on the empirical average of ∥ 𝑓 ∗ ( 𝑋 ) − 𝑌 ∥ around its population mean.

Original: Assuming sub-Gaussian moments of 𝑌 , it can be done with Bernstein inequality.
Revised: Assuming sub-Gaussian moments of 𝑌 , it can be done with Bernstein inequality.

Original: R D 𝑛 ( 𝑓 𝑛 ) − R ( 𝑓 𝑛 ) is harder to control as 𝑓 𝑛 depends on D 𝑛 , so we can not use the same technique.
Revised: R D 𝑛 ( 𝑓 𝑛 ) − R ( 𝑓 𝑛 ) is harder to control as 𝑓 𝑛 depends on D 𝑛 , so we can not use the same technique.

Original: The classical technique consists in going for the brutal uniform majoration,
Revised: The classical technique consists in going for the brutal uniform majoration,

Original: [Equation]
Revised: [Equation]

Original: (10) where F denotes the set of functions that 𝑓 𝑛 could be in concordance with our algorithm.
Revised: (10) where F denotes the set of functions that 𝑓 𝑛 could be in concordance with our algorithm.

Original: While this bound could seem highly suboptimal, when the class of functions is well-behaved, we can indeed control the deviation R ( 𝑓 ) − R D 𝑛 ( 𝑓 ) uniformly over this class without losing much (indeed for any class of functions, it is possible to build some really adversarial distribution 𝜌 so that this supremum behaves similarly to the concentration we are looking for (Vapnik, 1995; Anthony and Bartlett, 1999)).
Revised: While this bound could seem highly suboptimal, when the class of functions is well-behaved, we can indeed control the deviation R ( 𝑓 ) − R D 𝑛 ( 𝑓 ) uniformly over this class without losing much (indeed for any class of functions, it is possible to build some really adversarial distribution 𝜌 so that this supremum behaves similarly to the concentration we are looking for (Vapnik, 1995; Anthony and Bartlett, 1999)).

Original: This is particularly the case for our model linked with Assumption 1.
Revised: This is particularly the case for our model linked with Assumption 1.

Original: Expectations of supremum processes have been extensively studied, allowing to get satisfying upper bounds (note that when the ∥ 𝑓 ( 𝑋 ) − 𝑌 ∥ is bounded, deviation of the quantity of interest around its expectation can be controlled through McDiarmid inequality).
Revised: Expectations of supremum processes have been extensively studied, allowing to get satisfying upper bounds (note that when the ∥ 𝑓 ( 𝑋 ) − 𝑌 ∥ is bounded, deviation of the quantity of interest around its expectation can be controlled through McDiarmid inequality).

Original: In the statistical learning literature, it is usual to proceed with
Revised: In the statistical learning literature, it is usual to proceed with

Original: Rademacher complexity.
Revised: Rademacher complexity.

Original: Lemma 5 (Uniform control of functions deviation with Rademacher complexity) .
Revised: Lemma 5 (Uniform control of functions deviation with Rademacher complexity) .

Original: The expectation of the excess of risk can be bounded as
Revised: The expectation of the excess of risk can be bounded as

Original: [Equation]
Revised: [Equation]

Original: where ( 𝜎 𝑖 ) 𝑖 ≤ 𝑛 is defined as a family of Bernoulli independent variables taking value one or minus one with equal probability, and R 𝑛 ( F , ℓ, 𝜌 ) is called Rademacher complexity.
Revised: where ( 𝜎 𝑖 ) 𝑖 ≤ 𝑛 is defined as a family of Bernoulli independent variables taking value one or minus one with equal probability, and R 𝑛 ( F , ℓ, 𝜌 ) is called Rademacher complexity.

Original: Proof.
Revised: Proof.

Original: This results from the reduction to larger supremum and a symmetrization trick,
Revised: This results from the reduction to larger supremum and a symmetrization trick,

Original: E D 𝑛
Revised: E D 𝑛

Original: [Equation]
Revised: [Equation]

Original: which ends the proof.
Revised: which ends the proof.

Original: □
Revised: □

Original: In our case, we want to compute the Rademacher complexity for ℓ given by the norm of Y , and
Revised: In our case, we want to compute the Rademacher complexity for ℓ given by the norm of Y , and

Original: F = { 𝑥 → 𝜃𝜑 ( 𝑥 ) | 𝜃 ∈ Y ⊗ H , ∥ 𝜃 ∥ < 𝑀 } , for 𝑀 > 0 a parameter to specify in order to make sure that ∥ 𝜃 ∗ ∥ <
Revised: F = { 𝑥 → 𝜃𝜑 ( 𝑥 ) | 𝜃 ∈ Y ⊗ H , ∥ 𝜃 ∥ < 𝑀 } , for 𝑀 > 0 a parameter to specify in order to make sure that ∥ 𝜃 ∗ ∥ <

Original: 𝑀 , where the norm has to be understood as the ℓ 2 -product norm on Y ⊗ H ≃ H 𝑚 .
Revised: 𝑀 , where the norm has to be understood as the ℓ 2 -product norm on Y ⊗ H ≃ H 𝑚 .

Original: Working with linear models and Lipschitz losses is a well-known setting, allowing to derive directly the following bound.
Revised: Working with linear models and Lipschitz losses is a well-known setting, allowing to derive directly the following bound.

Original: Lemma 6 (Rademacher complexity of linear models with Lipschitz losses) .
Revised: Lemma 6 (Rademacher complexity of linear models with Lipschitz losses) .

Original: The complexity of the linear class of vector-valued function F = { 𝑥 → 𝜃𝜑 ( 𝑥 ) | 𝜃 ∈ Y ⊗ H , ∥ 𝜃 ∥ < 𝑀 } is bounded as
Revised: The complexity of the linear class of vector-valued function F = { 𝑥 → 𝜃𝜑 ( 𝑥 ) | 𝜃 ∈ Y ⊗ H , ∥ 𝜃 ∥ < 𝑀 } is bounded as

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: This proposition is usually split in two.
Revised: This proposition is usually split in two.

Original: First using the fact that the composition of a space of functions with a Lipschitz function does not increase the entropy of the subsequent space (Vitushkin, 1954).
Revised: First using the fact that the composition of a space of functions with a Lipschitz function does not increase the entropy of the subsequent space (Vitushkin, 1954).

Original: Then bounding the Rademacher complexity of linear models.
Revised: Then bounding the Rademacher complexity of linear models.

Original: We refer to Maurer (2016) for a self-contained proof of this result (stated in its Section 4.3).
Revised: We refer to Maurer (2016) for a self-contained proof of this result (stated in its Section 4.3).

Original: □
Revised: □

Original: Adding all the pieces together we have proven the following proposition, using the fact that the previous bound also applies to sup 𝑓 ∈ F R D 𝑛 ( 𝑓 ) − R ( 𝑓 ) by symmetry, hence it can be used for the deviation of R D 𝑛 ( 𝑓 ∗ ) − R ( 𝑓 ∗ ) .
Revised: Adding all the pieces together we have proven the following proposition, using the fact that the previous bound also applies to sup 𝑓 ∈ F R D 𝑛 ( 𝑓 ) − R ( 𝑓 ) by symmetry, hence it can be used for the deviation of R D 𝑛 ( 𝑓 ∗ ) − R ( 𝑓 ∗ ) .

Original: Proposition 7 (Control of the estimation error) .
Revised: Proposition 7 (Control of the estimation error) .

Original: Under Assumption 1, with the model of computation
Revised: Under Assumption 1, with the model of computation

Original: F = {
Revised: F = {

Original: 𝑥 ∈ X → 𝜃𝜑 ( 𝑥 ) ∈ Y | ∥ 𝜃 ∥ ≤ 𝑀 } , the generalization error of 𝑓 𝑛 is controlled by a term in 𝑛 − 1 /plus an optimization error on the empirical risk minimization
Revised: 𝑥 ∈ X → 𝜃𝜑 ( 𝑥 ) ∈ Y | ∥ 𝜃 ∥ ≤ 𝑀 } , the generalization error of 𝑓 𝑛 is controlled by a term in 𝑛 − 1 /plus an optimization error on the empirical risk minimization

Original: [Equation]
Revised: [Equation]

Original: as long as 𝑓 ∗ ∈ F .
Revised: as long as 𝑓 ∗ ∈ F .

Original: Note that this result can be refined using regularized risk (Sridharan et al., 2008), which would be useful under richer (stronger or weaker) source assumptions ( e.g. , Caponnetto and De Vito, 2006).
Revised: Note that this result can be refined using regularized risk (Sridharan et al., 2008), which would be useful under richer (stronger or weaker) source assumptions ( e.g. , Caponnetto and De Vito, 2006).

Original: Such a refinement would allow switching from a constraint ∥ 𝜃 ∥ <
Revised: Such a refinement would allow switching from a constraint ∥ 𝜃 ∥ <

Original: 𝑀 to define F to a regularization parameter 𝜆 ∥ 𝜃 ∥ 2 added in the risk without restrictions on ∥ 𝜃 ∥ , which would be better aligned with the current practice of machine learning.
Revised: 𝑀 to define F to a regularization parameter 𝜆 ∥ 𝜃 ∥ 2 added in the risk without restrictions on ∥ 𝜃 ∥ , which would be better aligned with the current practice of machine learning.

Original: Under Assumption 1, this will not fundamentally change the result.
Revised: Under Assumption 1, this will not fundamentally change the result.

Original: The estimation error can be controlled with the derivation in Appendix A.1, where stochastic gradients correspond to random sampling of a coefficient 𝑖 𝑡 ≤ 𝑛 plus the choice of a random 𝑈 𝑡 .
Revised: The estimation error can be controlled with the derivation in Appendix A.1, where stochastic gradients correspond to random sampling of a coefficient 𝑖 𝑡 ≤ 𝑛 plus the choice of a random 𝑈 𝑡 .

Original: For the option without resampling, there exists an acceleration scheme specific to different losses in order to benefit from the strong convexity ( e.g. , Bach and Moulines, 2013).
Revised: For the option without resampling, there exists an acceleration scheme specific to different losses in order to benefit from the strong convexity ( e.g. , Bach and Moulines, 2013).

Original: A.3 Lower bound
Revised: A.3 Lower bound

Original: In this section, we prove Theorem 2.
Revised: In this section, we prove Theorem 2.

Original: Let us consider any algorithm A : ∪ 𝑛 ∈ N ( X × Y ) 𝑛 → Θ that matches a dataset D 𝑛 to an estimate 𝜃 D 𝑛 ∈ Θ .
Revised: Let us consider any algorithm A : ∪ 𝑛 ∈ N ( X × Y ) 𝑛 → Θ that matches a dataset D 𝑛 to an estimate 𝜃 D 𝑛 ∈ Θ .

Original: Let us consider jointly a distribution 𝜌 and a parameter 𝜃 such that Assumption 1 holds, that is 𝑓 𝜌 : =
Revised: Let us consider jointly a distribution 𝜌 and a parameter 𝜃 such that Assumption 1 holds, that is 𝑓 𝜌 : =

Original: arg min 𝑓 : X → Y E 𝜌 [ ℓ ( 𝑓 ( 𝑋 ) ,𝑌 )]
Revised: arg min 𝑓 : X → Y E 𝜌 [ ℓ ( 𝑓 ( 𝑋 ) ,𝑌 )]

Original: = 𝑓 𝜃 .
Revised: = 𝑓 𝜃 .

Original: We are interested in characterizing for each algorithm the worst excess of risk it can achieve with respect to an adversarial distribution.
Revised: We are interested in characterizing for each algorithm the worst excess of risk it can achieve with respect to an adversarial distribution.

Original: The best worst performance that can be achieved by algorithms matching datasets to parameter can be written as
Revised: The best worst performance that can be achieved by algorithms matching datasets to parameter can be written as

Original: [Equation]
Revised: [Equation]

Original: This provides a lower bound to upper bounds such as (6) that can be derived for any algorithm.
Revised: This provides a lower bound to upper bounds such as (6) that can be derived for any algorithm.

Original: There are many ways to get lower bounds on this quantity.
Revised: There are many ways to get lower bounds on this quantity.

Original: Ultimately, we want to quantify the best certainty one can have on an estimate 𝜃 based on some observations ( 𝑋 𝑖 ,𝑌 𝑖 ) 𝑖 ≤ 𝑛 .
Revised: Ultimately, we want to quantify the best certainty one can have on an estimate 𝜃 based on some observations ( 𝑋 𝑖 ,𝑌 𝑖 ) 𝑖 ≤ 𝑛 .

Original: In particular, the algorithms A can be seen as rules to discriminate a model 𝜃 from observations D 𝑛 made under 𝜌 𝜃 , and where the error is measured through the excess of risk R ( 𝑓 ˆ 𝜃 , 𝜌 𝜃 ) − R ( 𝑓 𝜃 ; 𝜌 𝜃 ) where R ( 𝑓 ; 𝜌 ) =
Revised: In particular, the algorithms A can be seen as rules to discriminate a model 𝜃 from observations D 𝑛 made under 𝜌 𝜃 , and where the error is measured through the excess of risk R ( 𝑓 ˆ 𝜃 , 𝜌 𝜃 ) − R ( 𝑓 𝜃 ; 𝜌 𝜃 ) where R ( 𝑓 ; 𝜌 ) =

Original: E 𝜌 [ ℓ ( 𝑓 ( 𝑋 ) ,𝑌 )] and 𝜌 𝜃 is a distribution parametrized by 𝜃 such that 𝑓 𝜃 = 𝑓 𝜌 .
Revised: E 𝜌 [ ℓ ( 𝑓 ( 𝑋 ) ,𝑌 )] and 𝜌 𝜃 is a distribution parametrized by 𝜃 such that 𝑓 𝜃 = 𝑓 𝜌 .

Original: Let us first characterize the measure of error.
Revised: Let us first characterize the measure of error.

Original: Surprisingly, when in presence of Gaussian noise or uniform noise, the excess of risk behaves like a quadratic metric between parameters.
Revised: Surprisingly, when in presence of Gaussian noise or uniform noise, the excess of risk behaves like a quadratic metric between parameters.

Original: Lemma 8 (Quadratic behavior of the median regression excess of risk with Gaussian noise) .
Revised: Lemma 8 (Quadratic behavior of the median regression excess of risk with Gaussian noise) .

Original: Consider the random variable 𝑌 ∼ N ( 𝜇, 𝜎 2 𝐼 𝑚 ) , denote by ˆ 𝜇 an estimate of 𝜇 , the excess of risk can be developed as
Revised: Consider the random variable 𝑌 ∼ N ( 𝜇, 𝜎 2 𝐼 𝑚 ) , denote by ˆ 𝜇 an estimate of 𝜇 , the excess of risk can be developed as

Original: [Equation]
Revised: [Equation]

Original: where 𝑐 4 =
Revised: where 𝑐 4 =

Original: Γ ( 𝑚 + 12 )/( 2 √ 2 Γ ( 𝑚 + 22 )) ≥ ( 𝑚 + 2 ) − 1 / 2 / 2 .
Revised: Γ ( 𝑚 + 12 )/( 2 √ 2 Γ ( 𝑚 + 22 )) ≥ ( 𝑚 + 2 ) − 1 / 2 / 2 .

Original: Proof.
Revised: Proof.

Original: With this specific noise model, one can do the following derivations.
Revised: With this specific noise model, one can do the following derivations.

Original: [Equation]
Revised: [Equation]

Original: ˆ 𝜇 − 𝜇 𝜎 − 𝑌 (cid:13)(cid:13)(cid:13)(cid:13)(cid:21) .
Revised: ˆ 𝜇 − 𝜇 𝜎 − 𝑌 (cid:13)(cid:13)(cid:13)(cid:13)(cid:21) .

Original: We recognize the mean of a non-central 𝜒 -distribution of parameter 𝑘 = 𝑚 and 𝜆 = (cid:13)(cid:13)(cid:13) ˆ 𝜇 − 𝜇𝜎 (cid:13)(cid:13)(cid:13) .
Revised: We recognize the mean of a non-central 𝜒 -distribution of parameter 𝑘 = 𝑚 and 𝜆 = (cid:13)(cid:13)(cid:13) ˆ 𝜇 − 𝜇𝜎 (cid:13)(cid:13)(cid:13) .

Original: It can be expressed through the generalized Laguerre functions, which allows us to get the following Taylor expansion
Revised: It can be expressed through the generalized Laguerre functions, which allows us to get the following Taylor expansion

Original: [Equation]
Revised: [Equation]

Original: Hence, the following expression of the excess of risk,
Revised: Hence, the following expression of the excess of risk,

Original: [Equation]
Revised: [Equation]

Original: Note that in dimension one, the calculation can be done explicitly by computing integrals with the error function.
Revised: Note that in dimension one, the calculation can be done explicitly by computing integrals with the error function.

Original: [Equation]
Revised: [Equation]

Original: where we used the error function
Revised: where we used the error function

Original: [Equation]
Revised: [Equation]

Original: which is the symmetric function defined for 𝑥 ∈ R + as
Revised: which is the symmetric function defined for 𝑥 ∈ R + as

Original: [Equation]
Revised: [Equation]

Original: Developing those two functions in the Taylor series leads to the same quadratic behavior.
Revised: Developing those two functions in the Taylor series leads to the same quadratic behavior.

Original: □
Revised: □

Original: Let us now add a context variable.
Revised: Let us now add a context variable.

Original: Lemma 9 (Reduction to least-squares) .
Revised: Lemma 9 (Reduction to least-squares) .

Original: For Y = R 𝑚 , there exists a 𝜎 𝑚 > 0 , such that if 𝜑 is bounded by 𝜅 , and 𝑓 ∗ belongs to the class of functions F = { 𝑥 → 𝜃𝜑 ( 𝑥 ) | 𝜃 ∈ Y ⊗ H , ∥ 𝜃 ∥ ≤ 𝑀 } , and the conditional distribution are distributed as ( 𝑌 | 𝑋 ) ∼ N ( 𝑓 ∗ ( 𝑥 ) , 𝜎 2 𝐼 𝑚 ) , with 𝜎 > 2 𝑀𝜅𝜎 𝑚 ,
Revised: For Y = R 𝑚 , there exists a 𝜎 𝑚 > 0 , such that if 𝜑 is bounded by 𝜅 , and 𝑓 ∗ belongs to the class of functions F = { 𝑥 → 𝜃𝜑 ( 𝑥 ) | 𝜃 ∈ Y ⊗ H , ∥ 𝜃 ∥ ≤ 𝑀 } , and the conditional distribution are distributed as ( 𝑌 | 𝑋 ) ∼ N ( 𝑓 ∗ ( 𝑥 ) , 𝜎 2 𝐼 𝑚 ) , with 𝜎 > 2 𝑀𝜅𝜎 𝑚 ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: According to the precedent lemma, there exists 𝜎 𝑚 such that ∥ ˆ 𝜇 − 𝜇 ∥ 𝜎 − 1 ≤ 𝜎 − 1 𝑚 leads to
Revised: According to the precedent lemma, there exists 𝜎 𝑚 such that ∥ ˆ 𝜇 − 𝜇 ∥ 𝜎 − 1 ≤ 𝜎 − 1 𝑚 leads to

Original: [Equation]
Revised: [Equation]

Original: Let 𝑓 and 𝑓 ∗ ∈ F be parametrized by 𝜃 and 𝜃 ∗ .
Revised: Let 𝑓 and 𝑓 ∗ ∈ F be parametrized by 𝜃 and 𝜃 ∗ .

Original: For a given 𝑥 , setting ˆ 𝜇 = 𝑓 𝜃 ( 𝑥 ) = 𝜃𝜑 ( 𝑥 ) and 𝜇 = 𝑓 𝜃 ∗ ( 𝑥 ) , we get that, using the operator norm,
Revised: For a given 𝑥 , setting ˆ 𝜇 = 𝑓 𝜃 ( 𝑥 ) = 𝜃𝜑 ( 𝑥 ) and 𝜇 = 𝑓 𝜃 ∗ ( 𝑥 ) , we get that, using the operator norm,

Original: [Equation]
Revised: [Equation]

Original: Hence, as soon as 2 𝑀𝜅 ≤ 𝜎𝜎 − 1 𝑚 , we have that for almost all 𝑥 ∈ X
Revised: Hence, as soon as 2 𝑀𝜅 ≤ 𝜎𝜎 − 1 𝑚 , we have that for almost all 𝑥 ∈ X

Original: [Equation]
Revised: [Equation]

Original: The result follows from integration over X .
Revised: The result follows from integration over X .

Original: □
Revised: □

Original: We now have a characterization of the excess of risk that will allow us to reuse lower bounds for least-squares regression.
Revised: We now have a characterization of the excess of risk that will allow us to reuse lower bounds for least-squares regression.

Original: We will follow the exposition of Bach (2023) that we reproduce and comment here for completeness.
Revised: We will follow the exposition of Bach (2023) that we reproduce and comment here for completeness.

Original: It is based on the generalized Fano’s method (Ibragimov and Khas’minskii, 1977; Birgé, 1983).
Revised: It is based on the generalized Fano’s method (Ibragimov and Khas’minskii, 1977; Birgé, 1983).

Original: Learnability over a class of functions depends on the size of this class of functions.
Revised: Learnability over a class of functions depends on the size of this class of functions.

Original: For least-squares regression with a Hilbert class of functions, the right notion of size is given by the Kolmogorov entropy.
Revised: For least-squares regression with a Hilbert class of functions, the right notion of size is given by the Kolmogorov entropy.

Original: Let us call 𝜀 -packing of F with a metric 𝑑 any family ( 𝑓 𝑖 ) 𝑖 ≤ 𝑁 ∈ F 𝑁 such that 𝑑 ( 𝑓 𝑖 , 𝑓 𝑗 ) > 𝜀 .
Revised: Let us call 𝜀 -packing of F with a metric 𝑑 any family ( 𝑓 𝑖 ) 𝑖 ≤ 𝑁 ∈ F 𝑁 such that 𝑑 ( 𝑓 𝑖 , 𝑓 𝑗 ) > 𝜀 .

Original: The logarithm of the maximum cardinality of an 𝜀 -packing defines the 𝜀 -capacity of the class of functions F .
Revised: The logarithm of the maximum cardinality of an 𝜀 -packing defines the 𝜀 -capacity of the class of functions F .

Original: We refer the interested reader to Theorem 6 in Kolmogorov and Tikhomirov (1959) to make a link between the notions of capacity and entropy of a space.
Revised: We refer the interested reader to Theorem 6 in Kolmogorov and Tikhomirov (1959) to make a link between the notions of capacity and entropy of a space.

Original: To be perfectly rigorous, the least-squares error in not a norm on the space of 𝐿 2 functions, but we will call it a quasi-distance as it verifies symmetry, positive definiteness and the inequality 𝑑 ( 𝑥, 𝑦 )
Revised: To be perfectly rigorous, the least-squares error in not a norm on the space of 𝐿 2 functions, but we will call it a quasi-distance as it verifies symmetry, positive definiteness and the inequality 𝑑 ( 𝑥, 𝑦 )

Original: ≤ 𝐾 ( 𝑑 ( 𝑥, 𝑧 ) + 𝑑 ( 𝑧, 𝑦 )) for 𝐾 ≥ 1.
Revised: ≤ 𝐾 ( 𝑑 ( 𝑥, 𝑧 ) + 𝑑 ( 𝑧, 𝑦 )) for 𝐾 ≥ 1.

Original: Let us define an 𝜀 -packing with respect to a quasi-distance similarly as before.
Revised: Let us define an 𝜀 -packing with respect to a quasi-distance similarly as before.

Original: The 𝜀 -capacity of a space F gives a lower bound on the number of information to transmit in order to recover a function in F up to precision 𝜀 .
Revised: The 𝜀 -capacity of a space F gives a lower bound on the number of information to transmit in order to recover a function in F up to precision 𝜀 .

Original: We will leverage this fact in order to show our lower bound.
Revised: We will leverage this fact in order to show our lower bound.

Original: Let us first reduce the problem to a statistical test.
Revised: Let us first reduce the problem to a statistical test.

Original: Lemma 10 (Reduction to statistical testing) .
Revised: Lemma 10 (Reduction to statistical testing) .

Original: Let us consider a class of functions F and
Revised: Let us consider a class of functions F and

Original: an 𝜀 -packing ( 𝑓 𝑖 ) 𝑖 ≤ 𝑁 of F with respect to a quasi-distance 𝑑 (· , ·) verifying the triangular inequality up to a multiplicative factor 𝐾 .
Revised: an 𝜀 -packing ( 𝑓 𝑖 ) 𝑖 ≤ 𝑁 of F with respect to a quasi-distance 𝑑 (· , ·) verifying the triangular inequality up to a multiplicative factor 𝐾 .

Original: Then the minimax optimality of an algorithm A that takes as input the dataset
Revised: Then the minimax optimality of an algorithm A that takes as input the dataset

Original: D 𝑛 =
Revised: D 𝑛 =

Original: ( 𝑋 𝑖 ,𝑌 𝑖 ) 𝑖 ≤ 𝑛 and output a function in F can be related to the minimax optimality of an algorithm
Revised: ( 𝑋 𝑖 ,𝑌 𝑖 ) 𝑖 ≤ 𝑛 and output a function in F can be related to the minimax optimality of an algorithm

Original: C that takes an input the dataset D 𝑛 and output an index 𝑗 ∈ [ 𝑚 ] through
Revised: C that takes an input the dataset D 𝑛 and output an index 𝑗 ∈ [ 𝑚 ] through

Original: [Equation]
Revised: [Equation]

Original: (17) where the supremum over 𝜌 has to be understood as taken over all measures whose marginals can be written N ( 𝑓 ∗ ( 𝑥 ) , 𝜎 ) for 𝜎 bigger than a threshold 𝜎 𝑚 and 𝑓 ∗ ∈ F , and the supremum over 𝜌 𝑖 taken over the same type of measures with 𝑓 ∗ ∈ ( 𝑓 𝑖 ) 𝑖 ≤ 𝑁 .
Revised: (17) where the supremum over 𝜌 has to be understood as taken over all measures whose marginals can be written N ( 𝑓 ∗ ( 𝑥 ) , 𝜎 ) for 𝜎 bigger than a threshold 𝜎 𝑚 and 𝑓 ∗ ∈ F , and the supremum over 𝜌 𝑖 taken over the same type of measures with 𝑓 ∗ ∈ ( 𝑓 𝑖 ) 𝑖 ≤ 𝑁 .

Original: Proof.
Revised: Proof.

Original: Consider an algorithm A that takes as input a dataset D 𝑛
Revised: Consider an algorithm A that takes as input a dataset D 𝑛

Original: =
Revised: =

Original: ( 𝑋 𝑗 ,𝑌 𝑗 ) 𝑗 ≤ 𝑛 and output a function 𝑓 ∈
Revised: ( 𝑋 𝑗 ,𝑌 𝑗 ) 𝑗 ≤ 𝑛 and output a function 𝑓 ∈

Original: F .
Revised: F .

Original: We would like to see A as deriving from a classification rule and relate the classification and regression errors.
Revised: We would like to see A as deriving from a classification rule and relate the classification and regression errors.

Original: The natural classification rule associated with the algorithm A can be defined through 𝜋 the projection from F to [ 𝑁 ] that minimizes 𝑑 (cid:0) 𝑓 , 𝑓 𝜋 ( 𝑓 ) (cid:1) .
Revised: The natural classification rule associated with the algorithm A can be defined through 𝜋 the projection from F to [ 𝑁 ] that minimizes 𝑑 (cid:0) 𝑓 , 𝑓 𝜋 ( 𝑓 ) (cid:1) .

Original: The classification error and regression error made by 𝜋 ◦ A can be related thanks to the 𝜀 -packing property.
Revised: The classification error and regression error made by 𝜋 ◦ A can be related thanks to the 𝜀 -packing property.

Original: For any index 𝑗 ∈ [ 𝑁 ]
Revised: For any index 𝑗 ∈ [ 𝑁 ]

Original: [Equation]
Revised: [Equation]

Original: The error made by 𝑓 A ( D 𝑛 ) relates to the one made by 𝑓 𝜋 ◦ A ( D 𝑛 ) thanks to the modified triangular inequality, using the definition of the projection
Revised: The error made by 𝑓 A ( D 𝑛 ) relates to the one made by 𝑓 𝜋 ◦ A ( D 𝑛 ) thanks to the modified triangular inequality, using the definition of the projection

Original: [Equation]
Revised: [Equation]

Original: Finally,
Revised: Finally,

Original: [Equation]
Revised: [Equation]

Original: Assuming that the data were generated by a 𝜌 𝑖 and taking the expectation, the supremum over 𝜌 𝑖 and the infimum over A leads to
Revised: Assuming that the data were generated by a 𝜌 𝑖 and taking the expectation, the supremum over 𝜌 𝑖 and the infimum over A leads to

Original: [Equation]
Revised: [Equation]

Original: Because 𝜋 ◦ A are part of classification rules (indeed it parametrizes all the classification rules, simply consider A that matches a dataset to one of the functions ( 𝑓 𝑖 ) 𝑖 ≤ 𝑁 ), and because the distributions 𝜌 𝑖 are part of the distributions 𝜌 defined in the lemma, this last equation implies the stated result.
Revised: Because 𝜋 ◦ A are part of classification rules (indeed it parametrizes all the classification rules, simply consider A that matches a dataset to one of the functions ( 𝑓 𝑖 ) 𝑖 ≤ 𝑁 ), and because the distributions 𝜌 𝑖 are part of the distributions 𝜌 defined in the lemma, this last equation implies the stated result.

Original: □
Revised: □

Original: One of the harshest inequalities in the last proof is due to the usage of the 𝜀 -packing condition without considering error made by 𝑑 (cid:0) 𝑓 𝜋 ◦
Revised: One of the harshest inequalities in the last proof is due to the usage of the 𝜀 -packing condition without considering error made by 𝑑 (cid:0) 𝑓 𝜋 ◦

Original: A ( D 𝑛 ) , 𝑓 𝑗 (cid:1) that might be much worse than 𝜀 .
Revised: A ( D 𝑛 ) , 𝑓 𝑗 (cid:1) that might be much worse than 𝜀 .

Original: We will later add a condition on the 𝜀 -packings to ensure that the ( 𝑓 𝑖 ) are not too far from each other.
Revised: We will later add a condition on the 𝜀 -packings to ensure that the ( 𝑓 𝑖 ) are not too far from each other.

Original: This will not be a major problem when considering small balls in big dimension spaces.
Revised: This will not be a major problem when considering small balls in big dimension spaces.

Original: A.3.1 Results from statistical testing
Revised: A.3.1 Results from statistical testing

Original: In this section, we expand on lower bounds for statistical testing.
Revised: In this section, we expand on lower bounds for statistical testing.

Original: We refer the curious reader to Cover and Thomas (1991).
Revised: We refer the curious reader to Cover and Thomas (1991).

Original: We begin by relaxing the supremum by an average
Revised: We begin by relaxing the supremum by an average

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The last quantity can be seen as the best measure of error that can be achieved by a decoder C of a signal 𝑖 ∈ [ 𝑁 ] based on noisy observations D 𝑛 of the signal.
Revised: The last quantity can be seen as the best measure of error that can be achieved by a decoder C of a signal 𝑖 ∈ [ 𝑁 ] based on noisy observations D 𝑛 of the signal.

Original: A lower bound on such a similar quantity is the object of Fano’s inequality (Fano, 1968).
Revised: A lower bound on such a similar quantity is the object of Fano’s inequality (Fano, 1968).

Original: Lemma 11 (Fano’s inequality) .
Revised: Lemma 11 (Fano’s inequality) .

Original: Let ( 𝑋,𝑌 ) be a couple of random variables in X × Y with X , Y finite, and ˆ 𝑋 :
Revised: Let ( 𝑋,𝑌 ) be a couple of random variables in X × Y with X , Y finite, and ˆ 𝑋 :

Original: Y → X be a classification rule.
Revised: Y → X be a classification rule.

Original: Then, the error 𝑒 = 𝑒 ( 𝑋,𝑌 ) = 1 𝑋 ≠ ˆ 𝑋 ( 𝑌 ) verifies
Revised: Then, the error 𝑒 = 𝑒 ( 𝑋,𝑌 ) = 1 𝑋 ≠ ˆ 𝑋 ( 𝑌 ) verifies

Original: [Equation]
Revised: [Equation]

Original: Where for ( 𝑋,𝑌 ) ∈ Δ X × Y , 𝐻 ( 𝑋 ) and 𝐻 ( 𝑋 | 𝑌 ) denotes the entropy and conditional entropy, defined as, with the convention 0 log 0 = 0 ,
Revised: Where for ( 𝑋,𝑌 ) ∈ Δ X × Y , 𝐻 ( 𝑋 ) and 𝐻 ( 𝑋 | 𝑌 ) denotes the entropy and conditional entropy, defined as, with the convention 0 log 0 = 0 ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: This lemma is actually the result of two properties.
Revised: This lemma is actually the result of two properties.

Original: The first part of the proof is due to some manipulation of the entropy, consisting in showing that
Revised: The first part of the proof is due to some manipulation of the entropy, consisting in showing that

Original: [Equation]
Revised: [Equation]

Original: Let us first recall the following additive property of entropy
Revised: Let us first recall the following additive property of entropy

Original: [Equation]
Revised: [Equation]

Original: 𝑥,𝑌 = 𝑦, 𝑍
Revised: 𝑥,𝑌 = 𝑦, 𝑍

Original: [Equation]
Revised: [Equation]

Original: ) log
Revised: ) log

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: ) − ∑︁ 𝑥
Revised: ) − ∑︁ 𝑥

Original: [Equation]
Revised: [Equation]

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original: 𝑦
Revised: 𝑦

Original: [Equation]
Revised: [Equation]

Original: P (
Revised: P (

Original: [Equation]
Revised: [Equation]

Original: 𝑥
Revised: 𝑥

Original: [Equation]
Revised: [Equation]

Original: 𝑌
Revised: 𝑌

Original: [Equation]
Revised: [Equation]

Original: 𝑦
Revised: 𝑦

Original: [Equation]
Revised: [Equation]

Original: 𝑍
Revised: 𝑍

Original: [Equation]
Revised: [Equation]

Original: ) log (
Revised: ) log (

Original: [Equation]
Revised: [Equation]

Original: ( 𝑋
Revised: ( 𝑋

Original: [Equation]
Revised: [Equation]

Original: 𝑥 | 𝑍
Revised: 𝑥 | 𝑍

Original: [Equation]
Revised: [Equation]

Original: 𝑧 ))
Revised: 𝑧 ))

Original: =
Revised: =

Original: 𝐻 ( 𝑌 | 𝑋, 𝑍 ) + 𝐻 ( 𝑋 | 𝑍 ) .
Revised: 𝐻 ( 𝑌 | 𝑋, 𝑍 ) + 𝐻 ( 𝑋 | 𝑍 ) .

Original: Using this chain rule, we get
Revised: Using this chain rule, we get

Original: [Equation]
Revised: [Equation]

Original: Because 𝑒 is a function of ˆ 𝑋 and 𝑋 one can check that 𝐻 (cid:0) 𝑒 (cid:12)(cid:12) 𝑋, ˆ 𝑋 (cid:1) =
Revised: Because 𝑒 is a function of ˆ 𝑋 and 𝑋 one can check that 𝐻 (cid:0) 𝑒 (cid:12)(cid:12) 𝑋, ˆ 𝑋 (cid:1) =

Original: 0, 𝐻 (cid:0) 𝑒 (cid:12) (cid:12) 𝑋, ˆ 𝑋 (cid:1) =
Revised: 0, 𝐻 (cid:0) 𝑒 (cid:12) (cid:12) 𝑋, ˆ 𝑋 (cid:1) =

Original: − ∑︁ 𝑒,𝑋
Revised: − ∑︁ 𝑒,𝑋

Original: [Equation]
Revised: [Equation]

Original: ˆ 𝑋
Revised: ˆ 𝑋

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: log
Revised: log

Original: [Equation]
Revised: [Equation]

Original: 𝑒
Revised: 𝑒

Original: [Equation]
Revised: [Equation]

Original: 𝑋
Revised: 𝑋

Original: [Equation]
Revised: [Equation]

Original: ˆ 𝑋 (cid:1) )
Revised: ˆ 𝑋 (cid:1) )

Original: = − ∑︁ 𝑒,𝑋, ˆ 𝑋
Revised: = − ∑︁ 𝑒,𝑋, ˆ 𝑋

Original: P ( 𝑋
Revised: P ( 𝑋

Original: [Equation]
Revised: [Equation]

Original: ˆ
Revised: ˆ

Original: [Equation]
Revised: [Equation]

Original: 1 𝑒
Revised: 1 𝑒

Original: [Equation]
Revised: [Equation]

Original: 1 𝑋 ≠ ˆ 𝑋 log
Revised: 1 𝑋 ≠ ˆ 𝑋 log

Original: [Equation]
Revised: [Equation]

Original: 1 𝑒
Revised: 1 𝑒

Original: [Equation]
Revised: [Equation]

Original: 𝑋 ≠ ˆ 𝑋
Revised: 𝑋 ≠ ˆ 𝑋

Original: [Equation]
Revised: [Equation]

Original: = − ∑︁ 𝑒,𝑋, ˆ 𝑋
Revised: = − ∑︁ 𝑒,𝑋, ˆ 𝑋

Original: P ( 𝑋
Revised: P ( 𝑋

Original: [Equation]
Revised: [Equation]

Original: ˆ 𝑋 ) · 0 =
Revised: ˆ 𝑋 ) · 0 =

Original: Using Jensen inequality for the logarithm, we get 𝐻 (cid:0) 𝑋 (cid:12)(cid:12) 𝑒, ˆ 𝑋 (cid:1) =
Revised: Using Jensen inequality for the logarithm, we get 𝐻 (cid:0) 𝑋 (cid:12)(cid:12) 𝑒, ˆ 𝑋 (cid:1) =

Original: − ∑︁ 𝑋,𝑒, ˆ 𝑋
Revised: − ∑︁ 𝑋,𝑒, ˆ 𝑋

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: 𝑒, ˆ 𝑋
Revised: 𝑒, ˆ 𝑋

Original: [Equation]
Revised: [Equation]

Original: ) =
Revised: ) =

Original: [Equation]
Revised: [Equation]

Original: ∑︁ 𝑥,𝑥 ′
Revised: ∑︁ 𝑥,𝑥 ′

Original: P
Revised: P

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 0 , ˆ
Revised: 0 , ˆ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: 𝑥
Revised: 𝑥

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ˆ 𝑋
Revised: ˆ 𝑋

Original: [Equation]
Revised: [Equation]

Original: 𝑥
Revised: 𝑥

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: P
Revised: P

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 𝑥 ′ )
Revised: 𝑥 ′ )

Original: [Equation]
Revised: [Equation]

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: ) =
Revised: ) =

Original: [Equation]
Revised: [Equation]

Original: ∑︁ 𝑥
Revised: ∑︁ 𝑥

Original: [Equation]
Revised: [Equation]

Original: P
Revised: P

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Using that conditioning reduces the entropy
Revised: Using that conditioning reduces the entropy

Original: [Equation]
Revised: [Equation]

Original: which follows again from Jensen inequality,
Revised: which follows again from Jensen inequality,

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: we have proven that
Revised: we have proven that

Original: [Equation]
Revised: [Equation]

Original: The rest of the proof follows from the so-called data processing inequality, that is
Revised: The rest of the proof follows from the so-called data processing inequality, that is

Original: [Equation]
Revised: [Equation]

Original: We will not derive it here, since it will not be used in the following.
Revised: We will not derive it here, since it will not be used in the following.

Original: □
Revised: □

Original: In our case, a slight modification of the proof of Fano’s inequality leads to the following Proposition.
Revised: In our case, a slight modification of the proof of Fano’s inequality leads to the following Proposition.

Original: Lemma 12 (Generalized Fano’s method) .
Revised: Lemma 12 (Generalized Fano’s method) .

Original: For any family of distributions ( 𝜌 𝑖 ) 𝑖 ≤ 𝑁 on X × Y with 𝑁 ∈ N ∗ , any classification rule
Revised: For any family of distributions ( 𝜌 𝑖 ) 𝑖 ≤ 𝑁 on X × Y with 𝑁 ∈ N ∗ , any classification rule

Original: C : D 𝑛 → [ 𝑁 ] cannot beat the following average lower bound
Revised: C : D 𝑛 → [ 𝑁 ] cannot beat the following average lower bound

Original: [Equation]
Revised: [Equation]

Original: where 𝐾 ( 𝑝 || 𝑞 ) is the Kullback-Leibler divergence defined for any measure 𝑝 absolutely continuous with respect to a measure 𝑞 as
Revised: where 𝐾 ( 𝑝 || 𝑞 ) is the Kullback-Leibler divergence defined for any measure 𝑝 absolutely continuous with respect to a measure 𝑞 as

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let us consider the joint variable ( 𝑋,𝑌 ) where 𝑋 is a uniform variable on [ 𝑁 ] and ( 𝑌 | 𝑋 ) is distributed according to 𝜌 ⊗ 𝑛𝑋 .
Revised: Let us consider the joint variable ( 𝑋,𝑌 ) where 𝑋 is a uniform variable on [ 𝑁 ] and ( 𝑌 | 𝑋 ) is distributed according to 𝜌 ⊗ 𝑛𝑋 .

Original: For any classification rule ˆ 𝑋 : D 𝑛 → [ 𝑁 ] , using (20) we get
Revised: For any classification rule ˆ 𝑋 : D 𝑛 → [ 𝑁 ] , using (20) we get

Original: [Equation]
Revised: [Equation]

Original: We should work on 𝐻 (cid:0) 𝑋 (cid:12)(cid:12)
Revised: We should work on 𝐻 (cid:0) 𝑋 (cid:12)(cid:12)

Original: ˆ 𝑋 (cid:12)(cid:12) 𝑋 (cid:1) with similar ideas to the data processing inequality.
Revised: ˆ 𝑋 (cid:12)(cid:12) 𝑋 (cid:1) with similar ideas to the data processing inequality.

Original: First of all, using the chain rule for entropy
Revised: First of all, using the chain rule for entropy

Original: [Equation]
Revised: [Equation]

Original: where 𝐼 is the mutual information defined as, for 𝑋 and 𝑍 discrete
Revised: where 𝐼 is the mutual information defined as, for 𝑋 and 𝑍 discrete

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Similarly, one can define the mutual information for continuous variables.
Revised: Similarly, one can define the mutual information for continuous variables.

Original: In particular, we are interested in the case where 𝑋 is discrete and 𝑌 is continuous, denote by 𝜇 Y the marginal of ( 𝑋,𝑌 ) over 𝑌 and by 𝜇 | 𝑥 the conditional ( 𝑌 | 𝑋 = 𝑥 ) .
Revised: In particular, we are interested in the case where 𝑋 is discrete and 𝑌 is continuous, denote by 𝜇 Y the marginal of ( 𝑋,𝑌 ) over 𝑌 and by 𝜇 | 𝑥 the conditional ( 𝑌 | 𝑋 = 𝑥 ) .

Original: [Equation]
Revised: [Equation]

Original: Let us show the following version of the data processing inequality
Revised: Let us show the following version of the data processing inequality

Original: [Equation]
Revised: [Equation]

Original: To do so
Revised: To do so

Original: [Equation]
Revised: [Equation]

Original: we will use the conditional independence of 𝑋 and ˆ 𝑋 given 𝑌
Revised: we will use the conditional independence of 𝑋 and ˆ 𝑋 given 𝑌

Original: [Equation]
Revised: [Equation]

Original: which leads to
Revised: which leads to

Original: [Equation]
Revised: [Equation]

Original: Hence, using Jensen inequality,
Revised: Hence, using Jensen inequality,

Original: [Equation]
Revised: [Equation]

Original: We continue by computing the value of 𝐼 ( 𝑋,𝑌 )
Revised: We continue by computing the value of 𝐼 ( 𝑋,𝑌 )

Original: [Equation]
Revised: [Equation]

Original: by definition and using Jensen inequality, we get
Revised: by definition and using Jensen inequality, we get

Original: [Equation]
Revised: [Equation]

Original: We conclude from the fact that for 𝑝 and 𝑞 two distributions on a space Z
Revised: We conclude from the fact that for 𝑝 and 𝑞 two distributions on a space Z

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This explains the result.
Revised: This explains the result.

Original: □
Revised: □

Original: Let us assemble all the results proven thus far.
Revised: Let us assemble all the results proven thus far.

Original: In order to reduce our excess risk to a quadratic metric, we have assumed that the conditional distribution 𝜌 𝑖 | 𝑥 to be Gaussian noise.
Revised: In order to reduce our excess risk to a quadratic metric, we have assumed that the conditional distribution 𝜌 𝑖 | 𝑥 to be Gaussian noise.

Original: In order to integrate this constraint into the precedent derivations
Revised: In order to integrate this constraint into the precedent derivations

Original: [Equation]
Revised: [Equation]

Original: we leverage the following lemma.
Revised: we leverage the following lemma.

Original: Lemma 13 (Kullback-Leibler divergence with Gaussian noise) .
Revised: Lemma 13 (Kullback-Leibler divergence with Gaussian noise) .

Original: If 𝜌 𝑖 and 𝜌 𝑗 are two different distributions on X × Y such that there marginal over X are equal and the conditional distributions ( 𝑌 | 𝑋 = 𝑥 ) are respectively equal to
Revised: If 𝜌 𝑖 and 𝜌 𝑗 are two different distributions on X × Y such that there marginal over X are equal and the conditional distributions ( 𝑌 | 𝑋 = 𝑥 ) are respectively equal to

Original: N ( 𝑓 𝑖 ( 𝑥 ) , 𝜎𝐼 𝑚 ) and N ( 𝑓 𝑗 ( 𝑥 ) , 𝜎𝐼 𝑚 ) , then
Revised: N ( 𝑓 𝑖 ( 𝑥 ) , 𝜎𝐼 𝑚 ) and N ( 𝑓 𝑗 ( 𝑥 ) , 𝜎𝐼 𝑚 ) , then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We proceed with
Revised: We proceed with

Original: [Equation]
Revised: [Equation]

Original: where we have used the fact that the mean of a non-central 𝜒 -square variable of parameter ( 𝑚, 𝜇 2 ) is 𝑚 + 𝜇 2 .
Revised: where we have used the fact that the mean of a non-central 𝜒 -square variable of parameter ( 𝑚, 𝜇 2 ) is 𝑚 + 𝜇 2 .

Original: One could also develop the first two squared norms and use the fact that for any vector 𝑢 ∈ R 𝑚 , E [⟨ 𝑌 − 𝑓 𝑖 ( 𝑥 ) , 𝑢 ⟩] = 0 to get the result.
Revised: One could also develop the first two squared norms and use the fact that for any vector 𝑢 ∈ R 𝑚 , E [⟨ 𝑌 − 𝑓 𝑖 ( 𝑥 ) , 𝑢 ⟩] = 0 to get the result.

Original: □
Revised: □

Original: Combining the different results leads to the following proposition.
Revised: Combining the different results leads to the following proposition.

Original: Lemma 14.
Revised: Lemma 14.

Original: Under Assumption 1 with F = { 𝑥 ∈ X → 𝜃𝜑 ( 𝑥 ) ∈ Y | ∥ 𝜃 ∥ ≤ 𝑀 } and 𝜑 bounded by 𝜅 , for any family ( 𝑓 𝑖 ) 𝑖 ≤ 𝑁 𝜀 ∈ F 𝑁 and
Revised: Under Assumption 1 with F = { 𝑥 ∈ X → 𝜃𝜑 ( 𝑥 ) ∈ Y | ∥ 𝜃 ∥ ≤ 𝑀 } and 𝜑 bounded by 𝜅 , for any family ( 𝑓 𝑖 ) 𝑖 ≤ 𝑁 𝜀 ∈ F 𝑁 and

Original: any 𝜎 > 2 𝑀𝜅𝜎 𝑚
Revised: any 𝜎 > 2 𝑀𝜅𝜎 𝑚

Original: [Equation]
Revised: [Equation]

Original: for any algorithm A that maps a dataset D 𝑛 ∈ ( X × Y )
Revised: for any algorithm A that maps a dataset D 𝑛 ∈ ( X × Y )

Original: 𝑛 to a parameter 𝜃 ∈ Θ .
Revised: 𝑛 to a parameter 𝜃 ∈ Θ .

Original: A.3.2 Covering number for linear model
Revised: A.3.2 Covering number for linear model

Original: We are left with finding a good packing of the space induced by Assumption 1.
Revised: We are left with finding a good packing of the space induced by Assumption 1.

Original: To do so, we shall recall some property of reproducing kernel methods.
Revised: To do so, we shall recall some property of reproducing kernel methods.

Original: Lemma 15 (Linear models are ellipsoids) .
Revised: Lemma 15 (Linear models are ellipsoids) .

Original: For H a separable Hilbert space and 𝜑 : X → H bounded, the class of functions
Revised: For H a separable Hilbert space and 𝜑 : X → H bounded, the class of functions

Original: [Equation]
Revised: [Equation]

Original: 𝑥 ) ∈ Y | ∥ 𝜃 ∥ ≤ 𝑀 } can be characterized by
Revised: 𝑥 ) ∈ Y | ∥ 𝜃 ∥ ≤ 𝑀 } can be characterized by

Original: [Equation]
Revised: [Equation]

Original: where 𝜌 X is any distribution on X and 𝐾 is the operator on 𝐿 2 ( 𝜌 X ) that map 𝑓 to
Revised: where 𝜌 X is any distribution on X and 𝐾 is the operator on 𝐿 2 ( 𝜌 X ) that map 𝑓 to

Original: [Equation]
Revised: [Equation]

Original: whose image is assumed to be dense in 𝐿 2 .
Revised: whose image is assumed to be dense in 𝐿 2 .

Original: Proof.
Revised: Proof.

Original: This follows for isometry between elements in H and elements in 𝐿 2 .
Revised: This follows for isometry between elements in H and elements in 𝐿 2 .

Original: More precisely, let us define
Revised: More precisely, let us define

Original: [Equation]
Revised: [Equation]

Original: The adjoint of 𝑆 is characterized by
Revised: The adjoint of 𝑆 is characterized by

Original: [Equation]
Revised: [Equation]

Original: which follows from the fact that for 𝜃 ∈ Y ⊗ H , 𝑓 ∈ 𝐿 2 we have
Revised: which follows from the fact that for 𝜃 ∈ Y ⊗ H , 𝑓 ∈ 𝐿 2 we have

Original: [Equation]
Revised: [Equation]

Original: When 𝑆𝑆 ∗ is compact and dense in 𝐿
Revised: When 𝑆𝑆 ∗ is compact and dense in 𝐿

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: The compactness allows considering spectral decomposition hence fractional powers.
Revised: The compactness allows considering spectral decomposition hence fractional powers.

Original: We continue by observing that 𝑆𝑆 ∗ =
Revised: We continue by observing that 𝑆𝑆 ∗ =

Original: 𝐾 , which follows from
Revised: 𝐾 , which follows from

Original: [Equation]
Revised: [Equation]

Original: The compactness of 𝐾 derives from the fact that
Revised: The compactness of 𝐾 derives from the fact that

Original: [Equation]
Revised: [Equation]

Original: Hence, ∥ 𝐾 ∥ op ≤ 𝜅 2 .
Revised: Hence, ∥ 𝐾 ∥ op ≤ 𝜅 2 .

Original: Indeed, it is not hard to prove that the trace of 𝐾 is bounded by 𝑚𝜅 2 , hence 𝐾 is not only compact but trace-class.
Revised: Indeed, it is not hard to prove that the trace of 𝐾 is bounded by 𝑚𝜅 2 , hence 𝐾 is not only compact but trace-class.

Original: □
Revised: □

Original: It should be noted that the condition on 𝐾 being dense in 𝐿 2 ( 𝜌 X ) is not restrictive, as indeed all the problem is only seen through the lens of 𝜑 and 𝜌 X : one can replace X by supp 𝜌 X and 𝐿 2 ( 𝜌 X ) by the closure of the range of 𝐾 in 𝐿 2 ( 𝜌 X ) without modifying nor the analysis, nor the original problem.
Revised: It should be noted that the condition on 𝐾 being dense in 𝐿 2 ( 𝜌 X ) is not restrictive, as indeed all the problem is only seen through the lens of 𝜑 and 𝜌 X : one can replace X by supp 𝜌 X and 𝐿 2 ( 𝜌 X ) by the closure of the range of 𝐾 in 𝐿 2 ( 𝜌 X ) without modifying nor the analysis, nor the original problem.

Original: We should study packing in the ellipsoid F = (cid:8) 𝑓 ∈ 𝐿 2 (cid:12)(cid:12)
Revised: We should study packing in the ellipsoid F = (cid:8) 𝑓 ∈ 𝐿 2 (cid:12)(cid:12)

Original: ∥
Revised: ∥

Original: 𝐾 − 1 / 2 𝑓 ∥ 𝐿 2 ( 𝜌 X ) ≤ 𝑀 (cid:9) .
Revised: 𝐾 − 1 / 2 𝑓 ∥ 𝐿 2 ( 𝜌 X ) ≤ 𝑀 (cid:9) .

Original: It is useful to split the ellipsoid between a projection on a finite dimensional space that is isomorphic to the Euclidean space R 𝑘 and on a residual space 𝑅 where the energies (∥ 𝑓 | 𝑅 ∥
Revised: It is useful to split the ellipsoid between a projection on a finite dimensional space that is isomorphic to the Euclidean space R 𝑘 and on a residual space 𝑅 where the energies (∥ 𝑓 | 𝑅 ∥

Original: 𝐿 2 ( 𝜌 X ) ) 2 𝑓 ∈ F are uniformly small.
Revised: 𝐿 2 ( 𝜌 X ) ) 2 𝑓 ∈ F are uniformly small.

Original: We begin with the following packing lemma, sometimes referred to as Gilbert-Varshamov bound (Gilbert, 1952; Varshamov, 1957) which corresponds to a more generic result in coding theory.
Revised: We begin with the following packing lemma, sometimes referred to as Gilbert-Varshamov bound (Gilbert, 1952; Varshamov, 1957) which corresponds to a more generic result in coding theory.

Original: Lemma 16 ( ℓ 22 -packing of the hypercube) .
Revised: Lemma 16 ( ℓ 22 -packing of the hypercube) .

Original: For any 𝑘 ∈ N ∗ , there exists a 𝑘 / 4 -packing of the hypercube { 0 , 1 } 𝑘 , with respect to Hamming distance, of cardinality 𝑁 = exp ( 𝑘 / 8 ) .
Revised: For any 𝑘 ∈ N ∗ , there exists a 𝑘 / 4 -packing of the hypercube { 0 , 1 } 𝑘 , with respect to Hamming distance, of cardinality 𝑁 = exp ( 𝑘 / 8 ) .

Original: Proof.
Revised: Proof.

Original: Let us consider 𝜀 > 0, and a maximal 𝜀 -packing ( 𝑥 𝑖 )
Revised: Let us consider 𝜀 > 0, and a maximal 𝜀 -packing ( 𝑥 𝑖 )

Original: 𝑖 ≤ 𝑁 of the hypercube with respect to the distance 𝑑 ( 𝑥, 𝑦 ) =
Revised: 𝑖 ≤ 𝑁 of the hypercube with respect to the distance 𝑑 ( 𝑥, 𝑦 ) =

Original: (cid:205)
Revised: (cid:205)

Original: 𝑖 ∈[ 𝑘 ] 1 𝑥 𝑖 ≠ 𝑦 𝑖 =
Revised: 𝑖 ∈[ 𝑘 ] 1 𝑥 𝑖 ≠ 𝑦 𝑖 =

Original: ∥ 𝑥 − 𝑦 ∥ 1 = ∥ 𝑥 − 𝑦 ∥ 22 .
Revised: ∥ 𝑥 − 𝑦 ∥ 1 = ∥ 𝑥 − 𝑦 ∥ 22 .

Original: By maximality, we have { 0 , 1 } 𝑘 ⊂ ∪ 𝑖
Revised: By maximality, we have { 0 , 1 } 𝑘 ⊂ ∪ 𝑖

Original: ∈
Revised: ∈

Original: [ 𝑁 ] 𝐵 𝑑 ( 𝑥 𝑖 , 𝜀 ) , hence
Revised: [ 𝑁 ] 𝐵 𝑑 ( 𝑥 𝑖 , 𝜀 ) , hence

Original: [Equation]
Revised: [Equation]

Original: This inequality can be rewritten with 𝑍 a binomial variable of parameter ( 𝑘, 1 / 2 ) as 1 ≤ 𝑁 P ( 𝑍 ≤ 𝜀 ) .
Revised: This inequality can be rewritten with 𝑍 a binomial variable of parameter ( 𝑘, 1 / 2 ) as 1 ≤ 𝑁 P ( 𝑍 ≤ 𝜀 ) .

Original: Using Hoeffding inequality (Hoeffding, 1963), when 𝜀 = 𝑘 / 4 we get
Revised: Using Hoeffding inequality (Hoeffding, 1963), when 𝜀 = 𝑘 / 4 we get

Original: [Equation]
Revised: [Equation]

Original: This is the desired result.
Revised: This is the desired result.

Original: □
Revised: □

Original: Lemma 17 (Packing of infinite-dimensional ellipsoids) .
Revised: Lemma 17 (Packing of infinite-dimensional ellipsoids) .

Original: Let F be the function in 𝐿 2 ( 𝜌 X ) such that (cid:13)(cid:13)
Revised: Let F be the function in 𝐿 2 ( 𝜌 X ) such that (cid:13)(cid:13)

Original: 𝐾 − 1 / 2 𝑓
Revised: 𝐾 − 1 / 2 𝑓

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: 𝐿 2 ( 𝜌 X ) ≤ 𝑀 for 𝐾 a compact operator and 𝑀 any positive number.
Revised: 𝐿 2 ( 𝜌 X ) ≤ 𝑀 for 𝐾 a compact operator and 𝑀 any positive number.

Original: For any 𝑘 ∈ N ∗ , it is possible to find a family of 𝑁 ≥ exp ( 𝑘 / 8 ) elements ( 𝑓 𝑖 )
Revised: For any 𝑘 ∈ N ∗ , it is possible to find a family of 𝑁 ≥ exp ( 𝑘 / 8 ) elements ( 𝑓 𝑖 )

Original: 𝑖 ∈[ 𝑁 ] in F such that for any 𝑖 ≠ 𝑗 ,
Revised: 𝑖 ∈[ 𝑁 ] in F such that for any 𝑖 ≠ 𝑗 ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ( 𝜆 𝑖 ) 𝑖 ∈ N are the ordered (with repetition) eigenvalues of 𝐾 .
Revised: where ( 𝜆 𝑖 ) 𝑖 ∈ N are the ordered (with repetition) eigenvalues of 𝐾 .

Original: Proof.
Revised: Proof.

Original: Let us denote by ( 𝜆 𝑖 ) 𝑖 ∈ N the eigenvalues of 𝐾 and ( 𝑢 𝑖 ) 𝑖 ∈ N in 𝐿 2 the associated eigenvectors.
Revised: Let us denote by ( 𝜆 𝑖 ) 𝑖 ∈ N the eigenvalues of 𝐾 and ( 𝑢 𝑖 ) 𝑖 ∈ N in 𝐿 2 the associated eigenvectors.

Original: Consider ( 𝑎 𝑠 ) 𝑠 ∈
Revised: Consider ( 𝑎 𝑠 ) 𝑠 ∈

Original: [ 𝑁 ] a 𝑘 -packing of the hypercube {− 1 , 1 } 𝑘 for 𝑁 ≥ exp ( 𝑘 / 8 ) with respect to the ℓquasi-distance and define for any 𝑎 ∈ { 𝑎 𝑠 }
Revised: [ 𝑁 ] a 𝑘 -packing of the hypercube {− 1 , 1 } 𝑘 for 𝑁 ≥ exp ( 𝑘 / 8 ) with respect to the ℓquasi-distance and define for any 𝑎 ∈ { 𝑎 𝑠 }

Original: [Equation]
Revised: [Equation]

Original: with 𝑐 2 = (cid:205) 𝑘𝑖 = 1 𝜆 − 1 𝑖 .
Revised: with 𝑐 2 = (cid:205) 𝑘𝑖 = 1 𝜆 − 1 𝑖 .

Original: We verify that
Revised: We verify that

Original: [Equation]
Revised: [Equation]

Original: This is the object of the lemma.
Revised: This is the object of the lemma.

Original: □
Revised: □

Original: So far, we have proven the following lower bound.
Revised: So far, we have proven the following lower bound.

Original: Lemma 18.
Revised: Lemma 18.

Original: Under Assumption 1 with F = { 𝑥 ∈ X → 𝜃𝜑 ( 𝑥 ) ∈ Y | ∥ 𝜃 ∥ ≤ 𝑀 } and 𝜑 bounded by 𝜅 , for any family ( 𝑓 𝑖 ) 𝑖 ≤ 𝑁 𝜀 ∈ F 𝑁 and any 𝜎 > 2 𝑀𝜅𝜎 𝑚 and 𝑘𝑚 > 10 ,
Revised: Under Assumption 1 with F = { 𝑥 ∈ X → 𝜃𝜑 ( 𝑥 ) ∈ Y | ∥ 𝜃 ∥ ≤ 𝑀 } and 𝜑 bounded by 𝜅 , for any family ( 𝑓 𝑖 ) 𝑖 ≤ 𝑁 𝜀 ∈ F 𝑁 and any 𝜎 > 2 𝑀𝜅𝜎 𝑚 and 𝑘𝑚 > 10 ,

Original: [Equation]
Revised: [Equation]

Original: for any algorithm A that maps a dataset D 𝑛 ∈ ( X × Y )
Revised: for any algorithm A that maps a dataset D 𝑛 ∈ ( X × Y )

Original: 𝑛 to a parameter 𝜃 ∈ Θ , and where ( 𝜆 𝑖 ) are the ordered eigenvalue of the operator 𝐾 on 𝐿 2 ( X , R , 𝜌 X ) that maps any function 𝑓 to the function 𝐾 𝑓 defines for 𝑥 ′ ∈ X as
Revised: 𝑛 to a parameter 𝜃 ∈ Θ , and where ( 𝜆 𝑖 ) are the ordered eigenvalue of the operator 𝐾 on 𝐿 2 ( X , R , 𝜌 X ) that maps any function 𝑓 to the function 𝐾 𝑓 defines for 𝑥 ′ ∈ X as

Original: [Equation]
Revised: [Equation]

Original: In particular, when 𝜆 𝑖 = 𝜅 2 𝑖 − 𝑎 / 𝜁 ( 𝛼 ) , where 𝜁 denotes the Riemann zeta function, we get the following bounds.
Revised: In particular, when 𝜆 𝑖 = 𝜅 2 𝑖 − 𝑎 / 𝜁 ( 𝛼 ) , where 𝜁 denotes the Riemann zeta function, we get the following bounds.

Original: If we optimize with respect to 𝜎 , there exists 𝑛 𝛼 ∈ N such that for any 𝑛 > 𝑛 𝛼 .
Revised: If we optimize with respect to 𝜎 , there exists 𝑛 𝛼 ∈ N such that for any 𝑛 > 𝑛 𝛼 .

Original: [Equation]
Revised: [Equation]

Original: If we fix 𝜎 = 𝛽𝑀𝜅 with 𝛽 ≥ 2 , and we optimize with respect to 𝑘 , there exists a constant 𝑐 𝛽 and an integer 𝑛 0 such that for 𝑛 > 𝑛 0 we have
Revised: If we fix 𝜎 = 𝛽𝑀𝜅 with 𝛽 ≥ 2 , and we optimize with respect to 𝑘 , there exists a constant 𝑐 𝛽 and an integer 𝑛 0 such that for 𝑛 > 𝑛 0 we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Reusing Lemma 14, with the same notations, we have the lower bound in
Revised: Reusing Lemma 14, with the same notations, we have the lower bound in

Original: [Equation]
Revised: [Equation]

Original: Let 𝐾 and 𝐾 Y be the self-adjoint operators on 𝐿 2 ( X , R , 𝜌 X ) and 𝐿 2 ( X , Y , 𝜌 X ) respectively, both defined through the formula
Revised: Let 𝐾 and 𝐾 Y be the self-adjoint operators on 𝐿 2 ( X , R , 𝜌 X ) and 𝐿 2 ( X , Y , 𝜌 X ) respectively, both defined through the formula

Original: [Equation]
Revised: [Equation]

Original: When 𝐾 is compact, it admits an eigenvalue decomposition 𝐾 = (cid:205)
Revised: When 𝐾 is compact, it admits an eigenvalue decomposition 𝐾 = (cid:205)

Original: 𝑖 ∈ N 𝜆 𝑖 𝑢 𝑖 ⊗ 𝑢 𝑖 where the equality as to be understood as the convergence of operator with respect to the operator norm based on the 𝐿 2 -topology.
Revised: 𝑖 ∈ N 𝜆 𝑖 𝑢 𝑖 ⊗ 𝑢 𝑖 where the equality as to be understood as the convergence of operator with respect to the operator norm based on the 𝐿 2 -topology.

Original: It follows from the product structure of 𝐿 2 ( X , Y , 𝜌 X ) ≃ 𝐿 2 ( X , R , 𝜌 X ) 𝑚 that 𝐾
Revised: It follows from the product structure of 𝐿 2 ( X , Y , 𝜌 X ) ≃ 𝐿 2 ( X , R , 𝜌 X ) 𝑚 that 𝐾

Original: Y = (cid:205) 𝑖 ∈ R ,𝑗 ∈[ 𝑚 ] (cid:205) 𝑖 ∈ N ,𝑗 ∈[ 𝑚 ] 𝜆 𝑖 ( 𝑒 𝑖 ⊗ 𝑦 𝑗 ) ⊗ ( 𝑒 𝑖 ⊗ 𝑢 𝑗 ) with ( 𝑒 𝑗 ) the canonical basis of Y
Revised: Y = (cid:205) 𝑖 ∈ R ,𝑗 ∈[ 𝑚 ] (cid:205) 𝑖 ∈ N ,𝑗 ∈[ 𝑚 ] 𝜆 𝑖 ( 𝑒 𝑖 ⊗ 𝑦 𝑗 ) ⊗ ( 𝑒 𝑖 ⊗ 𝑢 𝑗 ) with ( 𝑒 𝑗 ) the canonical basis of Y

Original: = R 𝑚 .
Revised: = R 𝑚 .

Original: As a consequence, if ( 𝜆 𝑖 ) 𝑖 ∈ N are the ordered eigenvalues of 𝐾 then ( 𝜆 ⌊ 𝑖 / 𝑚 ⌋ ) are the ordered eigenvalues of 𝐾
Revised: As a consequence, if ( 𝜆 𝑖 ) 𝑖 ∈ N are the ordered eigenvalues of 𝐾 then ( 𝜆 ⌊ 𝑖 / 𝑚 ⌋ ) are the ordered eigenvalues of 𝐾

Original: Y .
Revised: Y .

Original: Hence, with Lemmas 15 and 17, it is possible to find 𝑁 =
Revised: Hence, with Lemmas 15 and 17, it is possible to find 𝑁 =

Original: exp ( 𝑘𝑚 / 8 ) functions in F such that
Revised: exp ( 𝑘𝑚 / 8 ) functions in F such that

Original: [Equation]
Revised: [Equation]

Original: If we multiply those functions by 𝜂 ∈ [ 0 , 1 ] we get a lower bound in 𝜂 2 𝑀
Revised: If we multiply those functions by 𝜂 ∈ [ 0 , 1 ] we get a lower bound in 𝜂 2 𝑀

Original: [Equation]
Revised: [Equation]

Original: Making sure that the last two terms are smaller than one fourth and one half respectively we get the following conditions on 𝑘 and 𝜂 , with Λ 𝑘 = (cid:205)
Revised: Making sure that the last two terms are smaller than one fourth and one half respectively we get the following conditions on 𝑘 and 𝜂 , with Λ 𝑘 = (cid:205)

Original: 𝑖 ≤ 𝑘 ( 𝑘𝜆 𝑖
Revised: 𝑖 ≤ 𝑘 ( 𝑘𝜆 𝑖

Original: [Equation]
Revised: [Equation]

Original: − 1 ,
Revised: − 1 ,

Original: [Equation]
Revised: [Equation]

Original: Using the fact that 𝜂 < 1, the lower bound becomes
Revised: Using the fact that 𝜂 < 1, the lower bound becomes

Original: [Equation]
Revised: [Equation]

Original: as long as 𝑘𝑚 > 10.
Revised: as long as 𝑘𝑚 > 10.

Original: When 𝜆 − 1 𝑖 = 𝑖 𝛼 𝜁 ( 𝛼 )/ 𝜅 2 , since Λ 𝑘 ≤ 𝜆 − 1 𝑘 , we simplify the last expression as
Revised: When 𝜆 − 1 𝑖 = 𝑖 𝛼 𝜁 ( 𝛼 )/ 𝜅 2 , since Λ 𝑘 ≤ 𝜆 − 1 𝑘 , we simplify the last expression as

Original: [Equation]
Revised: [Equation]

Original: Optimizing with respect to 𝜎 leads to
Revised: Optimizing with respect to 𝜎 leads to

Original: [Equation]
Revised: [Equation]

Original: 𝜅 2 𝜎
Revised: 𝜅 2 𝜎

Original: [Equation]
Revised: [Equation]

Original: This gives
Revised: This gives

Original: [Equation]
Revised: [Equation]

Original: The dependency of 𝑛 𝛼 to 𝑚 can be removed since any problem with Y =
Revised: The dependency of 𝑛 𝛼 to 𝑚 can be removed since any problem with Y =

Original: R 𝑚 can be cast as a problem in R 𝑚 + 1 by adding a spurious coordinate.
Revised: R 𝑚 can be cast as a problem in R 𝑚 + 1 by adding a spurious coordinate.

Original: Taking 𝑘 = 1 and 𝑚 = 10 leads to the result stated in the lemma.
Revised: Taking 𝑘 = 1 and 𝑚 = 10 leads to the result stated in the lemma.

Original: When 𝑛 < 𝑛 𝛼 , one can artificially multiply the bound by 𝑛 1 / 2 𝛼 , since an optimal algorithm can not do better with fewer data.
Revised: When 𝑛 < 𝑛 𝛼 , one can artificially multiply the bound by 𝑛 1 / 2 𝛼 , since an optimal algorithm can not do better with fewer data.

Original: After checking that one can take 𝜎 1 ≥ 1, this leads to a bound in
Revised: After checking that one can take 𝜎 1 ≥ 1, this leads to a bound in

Original: [Equation]
Revised: [Equation]

Original: Optimizing with respect to 𝑘 leads to 𝑘 𝛼 + 1 = 32 𝑀 2 𝜅 2 𝑛 /( 𝜎 2 𝑚𝜁 ( 𝛼 )) and a bound in ( 𝜎𝑚 1 / 2 )
Revised: Optimizing with respect to 𝑘 leads to 𝑘 𝛼 + 1 = 32 𝑀 2 𝜅 2 𝑛 /( 𝜎 2 𝑚𝜁 ( 𝛼 )) and a bound in ( 𝜎𝑚 1 / 2 )

Original: 𝛼 − 1 𝛼 + 1 ( 𝑀𝜅 ) 2 𝛼 +128 ( 32 𝑛 ) 𝛼𝛼 + 1 𝜁 ( 𝛼 ) 1 𝛼 +.
Revised: 𝛼 − 1 𝛼 + 1 ( 𝑀𝜅 ) 2 𝛼 +128 ( 32 𝑛 ) 𝛼𝛼 + 1 𝜁 ( 𝛼 ) 1 𝛼 +.

Original: The condition 𝑘 > min (cid:8) 10 𝑚 − 1 , 1 (cid:9) and 𝜎 ≥ 2 𝑀𝜅𝜎 𝑚 translates into the condition
Revised: The condition 𝑘 > min (cid:8) 10 𝑚 − 1 , 1 (cid:9) and 𝜎 ≥ 2 𝑀𝜅𝜎 𝑚 translates into the condition

Original: [Equation]
Revised: [Equation]

Original: We deduce that 𝜎 𝑚 = 𝑂 ( 𝑚 − 1 / 2 ) , otherwise we would not respect the upper bound derived with
Revised: We deduce that 𝜎 𝑚 = 𝑂 ( 𝑚 − 1 / 2 ) , otherwise we would not respect the upper bound derived with

Original: Rademacher complexity (or have made a mistake somewhere).
Revised: Rademacher complexity (or have made a mistake somewhere).

Original: Once again we can remove the dependency to 𝑚 .
Revised: Once again we can remove the dependency to 𝑚 .

Original: Considering 𝜎 = 𝛽𝑀𝜅 leads to the result stated in the lemma.
Revised: Considering 𝜎 = 𝛽𝑀𝜅 leads to the result stated in the lemma.

Original: □
Revised: □

Original: A.3.3 Controlling eigenvalues decay
Revised: A.3.3 Controlling eigenvalues decay

Original: Based on Lemma 18, in order to prove Theorem 2, we only need to show that there exists a mapping 𝜑 , an input space X and a distribution 𝜌 X such that the integral operator 𝐾 introduced in the lemma verifies the assumption on its eigenvalues.
Revised: Based on Lemma 18, in order to prove Theorem 2, we only need to show that there exists a mapping 𝜑 , an input space X and a distribution 𝜌 X such that the integral operator 𝐾 introduced in the lemma verifies the assumption on its eigenvalues.

Original: Notice that we show in the proof of Lemma 18 that the universal constant 𝑐 3 can be taken as 𝑐 3 = 2 − 11 .
Revised: Notice that we show in the proof of Lemma 18 that the universal constant 𝑐 3 can be taken as 𝑐 3 = 2 − 11 .

Original: To proceed, let us consider any infinite dimensional Hilbert space H with a basis ( 𝑒 𝑖 ) 𝑖 ∈ N , X = N
Revised: To proceed, let us consider any infinite dimensional Hilbert space H with a basis ( 𝑒 𝑖 ) 𝑖 ∈ N , X = N

Original: and 𝜑 : N → H ; 𝑖 → 𝜅𝑒 𝑖 .
Revised: and 𝜑 : N → H ; 𝑖 → 𝜅𝑒 𝑖 .

Original: For 𝑎 : N → R we have
Revised: For 𝑎 : N → R we have

Original: [Equation]
Revised: [Equation]

Original: Hence, the eigenvalues of 𝐾 are ( 𝜅 2 𝜌 X ( 𝑖 ))
Revised: Hence, the eigenvalues of 𝐾 are ( 𝜅 2 𝜌 X ( 𝑖 ))

Original: 𝑖 ≤ 𝑛 .
Revised: 𝑖 ≤ 𝑛 .

Original: It suffices to consider 𝜌 X ( 𝑖 ) =
Revised: It suffices to consider 𝜌 X ( 𝑖 ) =

Original: 𝑖 − 𝛼 / 𝜁 ( 𝛼 ) to conclude.
Revised: 𝑖 − 𝛼 / 𝜁 ( 𝛼 ) to conclude.

Original: The eigenvalue decay in 𝑂 ( 𝑖 − 𝛼 ) can also be witnessed in many regression problems.
Revised: The eigenvalue decay in 𝑂 ( 𝑖 − 𝛼 ) can also be witnessed in many regression problems.

Original: One way to build those cases is to turn a sequence of non-negative real values into a one-periodic function ℎ from R 𝑑 to R thanks to the inverse Fourier transform.
Revised: One way to build those cases is to turn a sequence of non-negative real values into a one-periodic function ℎ from R 𝑑 to R thanks to the inverse Fourier transform.

Original: Using Bochner (1933), one can construct a map 𝜑 such that the convolution operator linked with ℎ corresponds to the operator 𝐾 .
Revised: Using Bochner (1933), one can construct a map 𝜑 such that the convolution operator linked with ℎ corresponds to the operator 𝐾 .

Original: When 𝜌 is uniform on [ 0 , 1 ] 𝑑 , diagonalizing this convolution operator with the Fourier functions and using the property in Lemma 15 shows that the class of functions F are akin to Sobolev spaces.
Revised: When 𝜌 is uniform on [ 0 , 1 ] 𝑑 , diagonalizing this convolution operator with the Fourier functions and using the property in Lemma 15 shows that the class of functions F are akin to Sobolev spaces.

Original: Similar behavior can be proven when X = R 𝑑 and 𝜌 X is absolutely continuous with respect to the Lebesgue measure and has bounded density (Widom, 1963).
Revised: Similar behavior can be proven when X = R 𝑑 and 𝜌 X is absolutely continuous with respect to the Lebesgue measure and has bounded density (Widom, 1963).

Original: We refer the curious reader to Scholkopf and Smola (2001) or Bach (2023) for details.
Revised: We refer the curious reader to Scholkopf and Smola (2001) or Bach (2023) for details.

Original: B Unbiased weakly supervised stochastic gradients
Revised: B Unbiased weakly supervised stochastic gradients

Original: In this section, we provide a generic scheme to acquire unbiased weakly supervised stochastic gradients, as well as specifications of the formula given in the main text for least-squares and median regression.
Revised: In this section, we provide a generic scheme to acquire unbiased weakly supervised stochastic gradients, as well as specifications of the formula given in the main text for least-squares and median regression.

Original: B.1 Generic implementation
Revised: B.1 Generic implementation

Original: Suppose that Θ is finite dimensional, or that it can be approximated by a finite dimensional space without too much approximation error.
Revised: Suppose that Θ is finite dimensional, or that it can be approximated by a finite dimensional space without too much approximation error.

Original: For example, in the realm of scalar-valued kernel methods, it is usual to consider either the random finite dimensional space Span { 𝜑 ( 𝑥 𝑖 )} 𝑖 ≤ 𝑛 for ( 𝑥 𝑖 ) the data points, or the finite dimension space linked to the first eigenspaces of the operator E [ 𝜑 ( 𝑋 ) ⊗ 𝜑 ( 𝑋 )] .
Revised: For example, in the realm of scalar-valued kernel methods, it is usual to consider either the random finite dimensional space Span { 𝜑 ( 𝑥 𝑖 )} 𝑖 ≤ 𝑛 for ( 𝑥 𝑖 ) the data points, or the finite dimension space linked to the first eigenspaces of the operator E [ 𝜑 ( 𝑋 ) ⊗ 𝜑 ( 𝑋 )] .

Original: In the context of neural networks, the parameter space is always finite-dimensional.
Revised: In the context of neural networks, the parameter space is always finite-dimensional.

Original: Suppose also that, given 𝜃 , we know an upper bound 𝑀 𝜃 on the amplitude of ∇ 𝜃 ℓ ( 𝑓 𝜃 ( 𝑥 ) , 𝑦 ) , or that we know how to handle clipped gradients at amplitude
Revised: Suppose also that, given 𝜃 , we know an upper bound 𝑀 𝜃 on the amplitude of ∇ 𝜃 ℓ ( 𝑓 𝜃 ( 𝑥 ) , 𝑦 ) , or that we know how to handle clipped gradients at amplitude

Original: 𝑀 𝜃 for SGD.
Revised: 𝑀 𝜃 for SGD.

Original: Then, similarly to the least-squares method proposed in the main text, we can access weakly supervised gradient through the formula
Revised: Then, similarly to the least-squares method proposed in the main text, we can access weakly supervised gradient through the formula

Original: [Equation]
Revised: [Equation]

Original: where 𝐵 Θ is the unit ball of Θ .
Revised: where 𝐵 Θ is the unit ball of Θ .

Original: This scheme is really generic, and we do not advocate for it in practice as one may hope to leverage specific structure of the loss function and the parametric model in a more efficient way.
Revised: This scheme is really generic, and we do not advocate for it in practice as one may hope to leverage specific structure of the loss function and the parametric model in a more efficient way.

Original: This formula is rather a proof of concept to illustrate that our technique can be applied generically, and is not specific to least-squares or median regression.
Revised: This formula is rather a proof of concept to illustrate that our technique can be applied generically, and is not specific to least-squares or median regression.

Original: B.2 Specific implementations
Revised: B.2 Specific implementations

Original: Let us prove the two formulas to get stochastic gradients for both least-squares and median regression.
Revised: Let us prove the two formulas to get stochastic gradients for both least-squares and median regression.

Original: We begin with median regression.
Revised: We begin with median regression.

Original: Consider 𝑧 ∈ S 𝑚 − 1 , and let us denote 𝑥
Revised: Consider 𝑧 ∈ S 𝑚 − 1 , and let us denote 𝑥

Original: =
Revised: =

Original: E 𝑈 [ sign (⟨ 𝑧,𝑈 ⟩) 𝑈 ] .
Revised: E 𝑈 [ sign (⟨ 𝑧,𝑈 ⟩) 𝑈 ] .

Original: The direction 𝑥 /∥ 𝑥
Revised: The direction 𝑥 /∥ 𝑥

Original: ∥ ∈ S 𝑚 − 1 is characterized by the argmax over the sphere of the linear form
Revised: ∥ ∈ S 𝑚 − 1 is characterized by the argmax over the sphere of the linear form

Original: [Equation]
Revised: [Equation]

Original: , 𝑦 ⟩] .
Revised: , 𝑦 ⟩] .

Original: This linear form has a unique maximizer on S 𝑚 − 1 and by invariance by symmetry over the axis 𝑧 , this maximizer is aligned with 𝑧 , hence 𝑥 = 𝑐 𝑥 · 𝑧 .
Revised: This linear form has a unique maximizer on S 𝑚 − 1 and by invariance by symmetry over the axis 𝑧 , this maximizer is aligned with 𝑧 , hence 𝑥 = 𝑐 𝑥 · 𝑧 .

Original: We compute the amplitude with the formula, because 𝑧 is a unit vector 𝑐 𝑥 = ⟨ 𝑥, 𝑧 ⟩ = E 𝑈 [ sign (⟨ 𝑧,𝑈 ⟩) ⟨ 𝑈, 𝑧 ⟩] .
Revised: We compute the amplitude with the formula, because 𝑧 is a unit vector 𝑐 𝑥 = ⟨ 𝑥, 𝑧 ⟩ = E 𝑈 [ sign (⟨ 𝑧,𝑈 ⟩) ⟨ 𝑈, 𝑧 ⟩] .

Original: By invariance by rotation of both the uniform distribution and the scalar product, 𝑐 𝑥 is actually a constant
Revised: By invariance by rotation of both the uniform distribution and the scalar product, 𝑐 𝑥 is actually a constant

Original: The same type of reasoning applies for the least-squares case.
Revised: The same type of reasoning applies for the least-squares case.

Original: Consider 𝑧 ∈ R 𝑚 , and denote
Revised: Consider 𝑧 ∈ R 𝑚 , and denote

Original: [Equation]
Revised: [Equation]

Original: For the same reasons as before 𝑥 = 𝑐 𝑥 · 𝑢 for 𝑢
Revised: For the same reasons as before 𝑥 = 𝑐 𝑥 · 𝑢 for 𝑢

Original: = 𝑧 /∥ 𝑧 ∥ , and 𝑐 𝑥 verifies
Revised: = 𝑧 /∥ 𝑧 ∥ , and 𝑐 𝑥 verifies

Original: [Equation]
Revised: [Equation]

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: This explains the formula for least-squares.
Revised: This explains the formula for least-squares.

Original: Lemma 19 (Constant for the uniform strategy) .
Revised: Lemma 19 (Constant for the uniform strategy) .

Original: Under the uniform distribution on the sphere
Revised: Under the uniform distribution on the sphere

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let us compute 𝑐 2 =
Revised: Let us compute 𝑐 2 =

Original: E 𝑢 ∼ S 𝑚 − 1 [|⟨ 𝑢, 𝑒 1 ⟩|] .
Revised: E 𝑢 ∼ S 𝑚 − 1 [|⟨ 𝑢, 𝑒 1 ⟩|] .

Original: This constant can be written explicitly as
Revised: This constant can be written explicitly as

Original: [Equation]
Revised: [Equation]

Original: Remark that for any function 𝑓 : R → R , we have ∫
Revised: Remark that for any function 𝑓 : R → R , we have ∫

Original: [Equation]
Revised: [Equation]

Original: By denoting 𝑆 𝑚 the surface of the 𝑚 -sphere, the last integral is nothing but 𝑆 𝑚 − 2 .
Revised: By denoting 𝑆 𝑚 the surface of the 𝑚 -sphere, the last integral is nothing but 𝑆 𝑚 − 2 .

Original: By setting 𝑓 ( 𝑥 ) = 1, we can retrieve by recurrence the expression of 𝑆 𝑚 .
Revised: By setting 𝑓 ( 𝑥 ) = 1, we can retrieve by recurrence the expression of 𝑆 𝑚 .

Original: In our case, 𝑓 ( 𝑥 ) = | 𝑥 | , so we compute, with 𝑢 = 1 − 𝑥
Revised: In our case, 𝑓 ( 𝑥 ) = | 𝑥 | , so we compute, with 𝑢 = 1 − 𝑥

Original: [Equation]
Revised: [Equation]

Original: This leads to
Revised: This leads to

Original: [Equation]
Revised: [Equation]

Original: The ratio 𝑆 𝑚 − 2 / 𝑆 𝑚 − 1 can be expressed with the integral corresponding to 𝑓 = 1, but it is common knowledge that 𝑆 𝑚 − 1 = 2 𝜋 𝑚 / 2 / Γ ( 𝑚 / 2 ) .
Revised: The ratio 𝑆 𝑚 − 2 / 𝑆 𝑚 − 1 can be expressed with the integral corresponding to 𝑓 = 1, but it is common knowledge that 𝑆 𝑚 − 1 = 2 𝜋 𝑚 / 2 / Γ ( 𝑚 / 2 ) .

Original: □
Revised: □

Original: Lemma 20 (Constant for least-squares) .
Revised: Lemma 20 (Constant for least-squares) .

Original: Under the uniform distributions on [ 0 , 𝑀 ] and the sphere
Revised: Under the uniform distributions on [ 0 , 𝑀 ] and the sphere

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Similarly to the previous case, this constant can be written explicitly as
Revised: Similarly to the previous case, this constant can be written explicitly as

Original: [Equation]
Revised: [Equation]

Original: We continue as before with ∫
Revised: We continue as before with ∫

Original: [Equation]
Revised: [Equation]

Original: This leads to
Revised: This leads to

Original: [Equation]
Revised: [Equation]

Original: This is the result stated in the lemma.
Revised: This is the result stated in the lemma.

Original: □
Revised: □

Original: C Median surrogate
Revised: C Median surrogate

Original: Let us begin this section by proving Proposition 3.
Revised: Let us begin this section by proving Proposition 3.

Original: This result is actually the integration over 𝑥 ∈ X of a pointwise result, so let us fix 𝑥 ∈ X .
Revised: This result is actually the integration over 𝑥 ∈ X of a pointwise result, so let us fix 𝑥 ∈ X .

Original: Consider a probability distribution 𝑝 ∈ Δ Y over Y , and its median Θ ∗ ⊂ R Y defined as the minimizer of R 𝑆 ( 𝜃 ) =
Revised: Consider a probability distribution 𝑝 ∈ Δ Y over Y , and its median Θ ∗ ⊂ R Y defined as the minimizer of R 𝑆 ( 𝜃 ) =

Original: E 𝑝 [∥ 𝜃 − 𝑒 𝑌 ∥] .
Revised: E 𝑝 [∥ 𝜃 − 𝑒 𝑌 ∥] .

Original: We will to prove that ∪ 𝜃 ∈ Θ ∗ arg max 𝑦 ∈ Y 𝜃 𝑦 = arg max 𝑦 ∈ Y 𝑝 ( 𝑦 ) .
Revised: We will to prove that ∪ 𝜃 ∈ Θ ∗ arg max 𝑦 ∈ Y 𝜃 𝑦 = arg max 𝑦 ∈ Y 𝑝 ( 𝑦 ) .

Original: Let us begin by the inclusion arg max 𝑦 ∈ Y 𝑝 ( 𝑦 ) ⊂ ∪ 𝜃 ∈ Θ ∗ arg max 𝑦 ∈ Y 𝜃 𝑦 .
Revised: Let us begin by the inclusion arg max 𝑦 ∈ Y 𝑝 ( 𝑦 ) ⊂ ∪ 𝜃 ∈ Θ ∗ arg max 𝑦 ∈ Y 𝜃 𝑦 .

Original: To do so, consider 𝜃 ∈ R Y and 𝜎 ∈ S Y the transposition of two elements 𝑦 and 𝑧 in Y .
Revised: To do so, consider 𝜃 ∈ R Y and 𝜎 ∈ S Y the transposition of two elements 𝑦 and 𝑧 in Y .

Original: Denote by 𝜃 𝜎 ∈ R Y , the vector such that ( 𝜃 𝜎 ) 𝑦 ′
Revised: Denote by 𝜃 𝜎 ∈ R Y , the vector such that ( 𝜃 𝜎 ) 𝑦 ′

Original: =
Revised: =

Original: 𝜃 𝜎 ( 𝑦 ′ ) for any 𝑦 ′ ∈ Y .
Revised: 𝜃 𝜎 ( 𝑦 ′ ) for any 𝑦 ′ ∈ Y .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Because, for any 𝑎 ∈ R +
Revised: Because, for any 𝑎 ∈ R +

Original: [Equation]
Revised: [Equation]

Original: the function 𝑥 → √ 𝑎 − 2 𝑥 is increasing, if 𝑝 ( 𝑦 ) > 𝑝 ( 𝑧 ) , then to minimize
Revised: the function 𝑥 → √ 𝑎 − 2 𝑥 is increasing, if 𝑝 ( 𝑦 ) > 𝑝 ( 𝑧 ) , then to minimize

Original: R , we should make sure that 𝜃 𝑦 ≥ 𝜃 𝑧 i. As a consequence, because of symmetry, the modes of 𝑝 do correspond to argmax of ( 𝜃 ∗ 𝑦 ) 𝑦 ∈ Y for some 𝜃 ∗ ∈ Θ ∗ .
Revised: R , we should make sure that 𝜃 𝑦 ≥ 𝜃 𝑧 i. As a consequence, because of symmetry, the modes of 𝑝 do correspond to argmax of ( 𝜃 ∗ 𝑦 ) 𝑦 ∈ Y for some 𝜃 ∗ ∈ Θ ∗ .

Original: Let us now prove the second inclusion.
Revised: Let us now prove the second inclusion.

Original: To do so, suppose that 𝑝 ( 1 ) > 𝑝 ( 2 ) , and let us show that 𝜃 ∗ 1 > 𝜃 ∗ 2 .
Revised: To do so, suppose that 𝑝 ( 1 ) > 𝑝 ( 2 ) , and let us show that 𝜃 ∗ 1 > 𝜃 ∗ 2 .

Original: Let us parametrize 𝜃 1 =
Revised: Let us parametrize 𝜃 1 =

Original: 𝑎 + 𝜀
Revised: 𝑎 + 𝜀

Original: and 𝜃 

Revised: and 𝜃

Original: = 𝑎 − 𝜀 for a given 𝑎 , and show that 𝜀 = 0 is not optimal in order to minimize the risk R 𝑆 seen as a function of 𝜀 .
Revised: = 𝑎 − 𝜀 for a given 𝑎 , and show that 𝜀 = 0 is not optimal in order to minimize the risk R 𝑆 seen as a function of 𝜀 .

Original: To do so, we can use the Taylor expansion of √ 1 + 𝑥 = 1 + 𝑥 / 2.
Revised: To do so, we can use the Taylor expansion of √ 1 + 𝑥 = 1 + 𝑥 / 2.

Original: Hence, with 𝐴 = (cid:205) 𝑦> 2 ( 𝜃 ∗ 𝑦 ) 2 , retaking the last derivations
Revised: Hence, with 𝐴 = (cid:205) 𝑦> 2 ( 𝜃 ∗ 𝑦 ) 2 , retaking the last derivations

Original: [Equation]
Revised: [Equation]

Original: This shows that taking 𝜃 ∗ 1 =
Revised: This shows that taking 𝜃 ∗ 1 =

Original: 𝜃 ∗ 2 , that is 𝜀 = 0, is not optimal, hence we have the second inclusion, which ends the proof.
Revised: 𝜃 ∗ 2 , that is 𝜀 = 0, is not optimal, hence we have the second inclusion, which ends the proof.

Original: Note that we have proven a much stronger result, we have shown that ( 𝜃 𝑦 ) and 𝑝 ( 𝑦 ) are order in the exact same fashion (with respect to the strict comparison 𝑝 ( 𝑦 ) > 𝑝 ( 𝑧 ) ⇒ 𝜃 ∗ 𝑦 >
Revised: Note that we have proven a much stronger result, we have shown that ( 𝜃 𝑦 ) and 𝑝 ( 𝑦 ) are order in the exact same fashion (with respect to the strict comparison 𝑝 ( 𝑦 ) > 𝑝 ( 𝑧 ) ⇒ 𝜃 ∗ 𝑦 >

Original: 𝜃 ∗ 𝑧 for any 𝜃
Revised: 𝜃 ∗ 𝑧 for any 𝜃

Original: ∗ ∈ Θ ∗ ).
Revised: ∗ ∈ Θ ∗ ).

Original: C.1 Discussion around
Revised: C.1 Discussion around

Original: the median surrogate
Revised: the median surrogate

Original: The median surrogate have some nice properties for a surrogate method, in particular it does not fully characterize the distribution 𝑝 ( 𝑦 ) in the sense that there is no one-to-one mapping from 𝑝 to 𝜃 ∗ .
Revised: The median surrogate have some nice properties for a surrogate method, in particular it does not fully characterize the distribution 𝑝 ( 𝑦 ) in the sense that there is no one-to-one mapping from 𝑝 to 𝜃 ∗ .

Original: For example, when Y = { 1 , 2 , 3 } if 𝑝 ( 𝑦 = 𝑒 1 ) , 𝑝 ( 𝑦 = 𝑒 2 ) , 𝑝 ( 𝑦 = 𝑒 3 ) ∝ ( 1 , 1 , 2 cos ( 𝜋 / 6 )) , then the geometric median correspond to 𝜃 ∗ =
Revised: For example, when Y = { 1 , 2 , 3 } if 𝑝 ( 𝑦 = 𝑒 1 ) , 𝑝 ( 𝑦 = 𝑒 2 ) , 𝑝 ( 𝑦 = 𝑒 3 ) ∝ ( 1 , 1 , 2 cos ( 𝜋 / 6 )) , then the geometric median correspond to 𝜃 ∗ =

Original: 𝑒 3 .
Revised: 𝑒 3 .

Original: This differs from smooth surrogates, such as logistic regression or least-squares, that implicitly learn the full distribution 𝑝 , which should be seen as a waste of resources.
Revised: This differs from smooth surrogates, such as logistic regression or least-squares, that implicitly learn the full distribution 𝑝 , which should be seen as a waste of resources.

Original: Non-smooth surrogates tend to exhibit faster rates of convergence (in terms of decrease of the original risk as a function of the number of samples) than smooth surrogates when rates are derived through calibration inequalities (Nowak-Vila, 2021).
Revised: Non-smooth surrogates tend to exhibit faster rates of convergence (in terms of decrease of the original risk as a function of the number of samples) than smooth surrogates when rates are derived through calibration inequalities (Nowak-Vila, 2021).

Original: It would be nice to derive generic calibration inequality for the median surrogate for multiclass, and see how to derive a median surrogate for more structured problems such as ranking problems.
Revised: It would be nice to derive generic calibration inequality for the median surrogate for multiclass, and see how to derive a median surrogate for more structured problems such as ranking problems.

Original: [Figure]
Revised: [Figure]

Original: Median bias
Revised: Median bias

Original: [Figure]
Revised: [Figure]

Original: Median
Revised: Median

Original: [Figure]
Revised: [Figure]

Original: In this
Revised: In this

Original: , if we classification problem by of through median will converge exponentially fast toward through will never converge to the
Revised: , if we classification problem by of through median will converge exponentially fast toward through will never converge to the

Original: D Classification with a min-max game
Revised: D Classification with a min-max game

Original: In this section we prove and extend on Proposition 4.
Revised: In this section we prove and extend on Proposition 4.

Original: First of all
Revised: First of all

Original: [Equation]
Revised: [Equation]

Original: let us consider the average loss, for ( 𝑣 𝑦 ) ∈
Revised: let us consider the average loss, for ( 𝑣 𝑦 ) ∈

Original: R Y summing to one
Revised: R Y summing to one

Original: [Equation]
Revised: [Equation]

Original: Consider now this loss conditioned on the observation 1 𝑦 ∈ 𝑠
Revised: Consider now this loss conditioned on the observation 1 𝑦 ∈ 𝑠

Original: [Equation]
Revised: [Equation]

Original: we have plenty of characterizations of 𝐿 ,
Revised: we have plenty of characterizations of 𝐿 ,

Original: [Equation]
Revised: [Equation]

Original: Minimizing this loss or the loss 2 𝐿 − 1 as defined in Proposition 4 is equivalent
Revised: Minimizing this loss or the loss 2 𝐿 − 1 as defined in Proposition 4 is equivalent

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: three elements, when 𝑌 is deterministic, the optimal query strategy consists in considering 𝑠 = { 𝑦 } , while surrogate strategies, such as least-squares and median regression, that learn 𝑧 ∗ = 𝑒 𝑦 , would only make such a query only two third of the time (which is the ratio of the solid angle of [ 𝑒 2 , 𝑒 3 ] from 𝑒 1 divided by 𝜋 ).
Revised: three elements, when 𝑌 is deterministic, the optimal query strategy consists in considering 𝑠 = { 𝑦 } , while surrogate strategies, such as least-squares and median regression, that learn 𝑧 ∗ = 𝑒 𝑦 , would only make such a query only two third of the time (which is the ratio of the solid angle of [ 𝑒 2 , 𝑒 3 ] from 𝑒 1 divided by 𝜋 ).

Original: This shows that those surrogate strategies do not fully leverage the specific structure of the output.
Revised: This shows that those surrogate strategies do not fully leverage the specific structure of the output.

Original: D.1 Consistency
Revised: D.1 Consistency

Original: Let us consider the loss as defined in this proposition, we have the characterization
Revised: Let us consider the loss as defined in this proposition, we have the characterization

Original: [Equation]
Revised: [Equation]

Original: Let us rewrite (8) based on this previous characterization of the loss, we have
Revised: Let us rewrite (8) based on this previous characterization of the loss, we have

Original: [Equation]
Revised: [Equation]

Original: Hence, without any context variable, the min-max game (8) can be rewritten as
Revised: Hence, without any context variable, the min-max game (8) can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: We will analyze this problem through the lens of a mix-actions zero-sum game.
Revised: We will analyze this problem through the lens of a mix-actions zero-sum game.

Original: We know from von
Revised: We know from von

Original: Neumann and Morgenstern (1944) that a solution to this min-max problem exists, and that one can switch the min-max to a max-min without modifying the value of the solution.
Revised: Neumann and Morgenstern (1944) that a solution to this min-max problem exists, and that one can switch the min-max to a max-min without modifying the value of the solution.

Original: Let us denote by ( 𝑣 ∗ , 𝜇 ∗ ) the argument of a solution.
Revised: Let us denote by ( 𝑣 ∗ , 𝜇 ∗ ) the argument of a solution.

Original: To minimize the value of this game, the player 𝑣 should play such that
Revised: To minimize the value of this game, the player 𝑣 should play such that

Original: [Equation]
Revised: [Equation]

Original: which allows this player to ensure a negative value to the game.
Revised: which allows this player to ensure a negative value to the game.

Original: Stated otherwise
Revised: Stated otherwise

Original: [Equation]
Revised: [Equation]

Original: As a consequence, if there exists any set such that P ( 𝑌 ∈ 𝑠 ) = 1 / 2, the best strategy of player 𝜇 is to play only those sets to ensure the value zero, and any 𝑣 that satisfies (31) is optimal.
Revised: As a consequence, if there exists any set such that P ( 𝑌 ∈ 𝑠 ) = 1 / 2, the best strategy of player 𝜇 is to play only those sets to ensure the value zero, and any 𝑣 that satisfies (31) is optimal.

Original: It should be noted that (31) does not generally imply that ( 𝑣 𝑦 ) 𝑦 ∈ Y has the same ordering as ( P ( 𝑌 = 𝑦 ))
Revised: It should be noted that (31) does not generally imply that ( 𝑣 𝑦 ) 𝑦 ∈ Y has the same ordering as ( P ( 𝑌 = 𝑦 ))

Original: 𝑦 ∈ Y .
Revised: 𝑦 ∈ Y .

Original: When { 𝑦 ∗ } ∈ S and P ( 𝑌 = 𝑦 ∗ ) > 1 / 2, if 𝑣 = 𝛿 𝑦 ∗ , the prediction player is able to ensure a value of max 𝑠 ∈ S − | 2 P ( 𝑌 ∈ 𝑠 ) − 1 | , which is maximized by the query player with 𝑠 = { 𝑦 ∗ } ∪ 𝑠 ′ for any 𝑠 ′ such that P ( 𝑌 ∈ 𝑠 ′ )
Revised: When { 𝑦 ∗ } ∈ S and P ( 𝑌 = 𝑦 ∗ ) > 1 / 2, if 𝑣 = 𝛿 𝑦 ∗ , the prediction player is able to ensure a value of max 𝑠 ∈ S − | 2 P ( 𝑌 ∈ 𝑠 ) − 1 | , which is maximized by the query player with 𝑠 = { 𝑦 ∗ } ∪ 𝑠 ′ for any 𝑠 ′ such that P ( 𝑌 ∈ 𝑠 ′ )

Original: = 0.
Revised: = 0.

Original: Other strategies for 𝑣 will only increase this value, hence 𝑣 ∗ = 𝛿 𝑦 ∗ which implies the first part of Proposition 4.
Revised: Other strategies for 𝑣 will only increase this value, hence 𝑣 ∗ = 𝛿 𝑦 ∗ which implies the first part of Proposition 4.

Original: A counter example.
Revised: A counter example.

Original: While we hope that the solution ( 𝑣 ∗ , 𝜇 ∗ ) does characterize the original solution 𝑦 ∗ , it should be noted that 𝑣 ∗ alone does not characterize 𝑦 ∗ .
Revised: While we hope that the solution ( 𝑣 ∗ , 𝜇 ∗ ) does characterize the original solution 𝑦 ∗ , it should be noted that 𝑣 ∗ alone does not characterize 𝑦 ∗ .

Original: Indeed, it is even possible to have 𝑣 ∗ uniquely defined without having 𝑦 ∗ =
Revised: Indeed, it is even possible to have 𝑣 ∗ uniquely defined without having 𝑦 ∗ =

Original: arg max 𝑦 ∈ Y 𝑣 ∗ 𝑦 .
Revised: arg max 𝑦 ∈ Y 𝑣 ∗ 𝑦 .

Original: For example, consider the case where
Revised: For example, consider the case where

Original: Y = { 1 , 2 , 3 } and ( P ( 𝑌 = 𝑖 )) 𝑖 ∈[ 3 ] =
Revised: Y = { 1 , 2 , 3 } and ( P ( 𝑌 = 𝑖 )) 𝑖 ∈[ 3 ] =

Original: By symmetry, the player 𝜇 only has to play on
Revised: By symmetry, the player 𝜇 only has to play on

Original: S = {{ 1 } , { 2 } , { 3 }} , which leads to the min-max game
Revised: S = {{ 1 } , { 2 } , { 3 }} , which leads to the min-max game

Original: [Equation]
Revised: [Equation]

Original: The value of this game is − .
Revised: The value of this game is − .

Original: and is achieved for 𝜇 ∗ = ( . 5 , .
Revised: and is achieved for 𝜇 ∗ = ( . 5 , .

Original: 25 ) , 𝑣 ∗ = ( . 25 , .
Revised: 25 ) , 𝑣 ∗ = ( . 25 , .

Original: D.2 Optimization procedure
Revised: D.2 Optimization procedure

Original: Let us rewrite the problem through the objective
Revised: Let us rewrite the problem through the objective

Original: [Equation]
Revised: [Equation]

Original: We want to solve the min-max problem min 𝑔 max 𝜇 E ( 𝑔, 𝜇 ) .
Revised: We want to solve the min-max problem min 𝑔 max 𝜇 E ( 𝑔, 𝜇 ) .

Original: This problem can be solved efficiently based on the vector field point of view of gradient descent (Bubeck, 2015) if:
Revised: This problem can be solved efficiently based on the vector field point of view of gradient descent (Bubeck, 2015) if:

Original: • we can parametrize the function 𝑔 : X → Δ Y such that E is convex with respect to the parametrization of 𝑔 ; • we can access unbiased stochastic gradients of E with respect to 𝑔 that have a small second moment; • we can parametrize the function 𝜇 : X → Δ S such that E is concave with respect to the parametrization of 𝜇 ; • we can access unbiased stochastic gradients of E with respect to 𝜇 that have a small second moment.
Revised: • we can parametrize the function 𝑔 : X → Δ Y such that E is convex with respect to the parametrization of 𝑔 ; • we can access unbiased stochastic gradients of E with respect to 𝑔 that have a small second moment; • we can parametrize the function 𝜇 : X → Δ S such that E is concave with respect to the parametrization of 𝜇 ; • we can access unbiased stochastic gradients of E with respect to 𝜇 that have a small second moment.

Original: The first two points are no problems, 𝑔 can be parametrized with softmax regression, and since 𝐿 is linear with respect to the scores, it will keep the problem convex.
Revised: The first two points are no problems, 𝑔 can be parametrized with softmax regression, and since 𝐿 is linear with respect to the scores, it will keep the problem convex.

Original: Moreover, to access a stochastic gradient of E , one can sample 𝑋 𝑖 ∼ 𝜌 X and 𝑆 𝑖 ∼ 𝜇 ( 𝑋 𝑖 ) before querying 1 𝑌 𝑖 ∈ 𝑆 𝑖 and computing the gradient of 𝐿 ( 𝑔 ( 𝑋 𝑖 ) , 𝑆 𝑖 , 1 𝑌 𝑖 ∈ 𝑆 𝑖 − 1 𝑌 𝑖 ∉ 𝑆 𝑖 ) with respect to the parametrization of 𝑔 .
Revised: Moreover, to access a stochastic gradient of E , one can sample 𝑋 𝑖 ∼ 𝜌 X and 𝑆 𝑖 ∼ 𝜇 ( 𝑋 𝑖 ) before querying 1 𝑌 𝑖 ∈ 𝑆 𝑖 and computing the gradient of 𝐿 ( 𝑔 ( 𝑋 𝑖 ) , 𝑆 𝑖 , 1 𝑌 𝑖 ∈ 𝑆 𝑖 − 1 𝑌 𝑖 ∉ 𝑆 𝑖 ) with respect to the parametrization of 𝑔 .

Original: The third point is slightly harder to tackle.
Revised: The third point is slightly harder to tackle.

Original: Since E is linear with respect to 𝜇 , one way to proceed is to find a linear parametrization of 𝜇 .
Revised: Since E is linear with respect to 𝜇 , one way to proceed is to find a linear parametrization of 𝜇 .

Original: In particular, one can take a family ( 𝑔 𝑖 )
Revised: In particular, one can take a family ( 𝑔 𝑖 )

Original: 𝑖 ∈[ 𝑁 ] of linearly independent functions from X to Δ S and search for 𝑔 under the form (cid:205)
Revised: 𝑖 ∈[ 𝑁 ] of linearly independent functions from X to Δ S and search for 𝑔 under the form (cid:205)

Original: 𝑖 ∈[ 𝑁 ] 𝑐 𝑖 𝑔 𝑖 for ( 𝑐 𝑖 ) positive summing to one.
Revised: 𝑖 ∈[ 𝑁 ] 𝑐 𝑖 𝑔 𝑖 for ( 𝑐 𝑖 ) positive summing to one.

Original: To build such a family, one can eventually use “atom functions” and simple operations such as symmetry with respect to Y and S , rescaling, translation, rotations with respect to X .
Revised: To build such a family, one can eventually use “atom functions” and simple operations such as symmetry with respect to Y and S , rescaling, translation, rotations with respect to X .

Original: For example if
Revised: For example if

Original: X is a Banach space, one could define atom functions as, for 𝑦 𝑖 ∈ Y
Revised: X is a Banach space, one could define atom functions as, for 𝑦 𝑖 ∈ Y

Original: [Equation]
Revised: [Equation]

Original: Those functions could be rescaled and translated as 𝑔 𝜎,𝜏,𝑖 ( 𝑥 )
Revised: Those functions could be rescaled and translated as 𝑔 𝜎,𝜏,𝑖 ( 𝑥 )

Original: =
Revised: =

Original: 𝑔 𝑖 ( 𝜎 ( 𝑥 − 𝜏 )) , in order to specify a family ( 𝑔 𝜎,𝜏,𝑖 ) from few values for 𝜏 and 𝜎 .
Revised: 𝑔 𝑖 ( 𝜎 ( 𝑥 − 𝜏 )) , in order to specify a family ( 𝑔 𝜎,𝜏,𝑖 ) from few values for 𝜏 and 𝜎 .

Original: The last point is the most difficult one.
Revised: The last point is the most difficult one.

Original: Without context variables, and with no-parametrization for 𝜇 , a naive unbiased gradient strategy for 𝜇 consists in asking random questions to update the full knowledge of ( P ( 𝑌 ∈ 𝑠 ))
Revised: Without context variables, and with no-parametrization for 𝜇 , a naive unbiased gradient strategy for 𝜇 consists in asking random questions to update the full knowledge of ( P ( 𝑌 ∈ 𝑠 ))

Original: 𝑠 ∈ S .
Revised: 𝑠 ∈ S .

Original: But such a strategy will be much worse than our median surrogate technique with queries 1 𝑌 ∈{ 𝑦 } for 𝑦 sampled uniformly at random in Y .
Revised: But such a strategy will be much worse than our median surrogate technique with queries 1 𝑌 ∈{ 𝑦 } for 𝑦 sampled uniformly at random in Y .

Original: Eventually, one should go for a biased gradient strategy, while making sure to update 𝜇 coherently to avoid getting stalled on bad estimates as a result of biases.
Revised: Eventually, one should go for a biased gradient strategy, while making sure to update 𝜇 coherently to avoid getting stalled on bad estimates as a result of biases.

Original: E Experimental details
Revised: E Experimental details

Original: Our experiments are done in Python .
Revised: Our experiments are done in Python .

Original: We leverage the C implementation of high-level array instructions by Harris et al.
Revised: We leverage the C implementation of high-level array instructions by Harris et al.

Original: (2020), as well as the visualization library of Hunter (2007).
Revised: (2020), as well as the visualization library of Hunter (2007).

Original: Randomness in experiments is controlled by choosing explicitly the seed of a pseudo-random number generator.
Revised: Randomness in experiments is controlled by choosing explicitly the seed of a pseudo-random number generator.

Original: E.1 Comparison with fully supervised SGD
Revised: E.1 Comparison with fully supervised SGD

Original: In this section, we investigate the difference between weakly and fully supervised SGD.
Revised: In this section, we investigate the difference between weakly and fully supervised SGD.

Original: According to
Revised: According to

Original: Theorem 1, we only lost a constant factor of order 𝑚 3 / 2 in our rates compared to fully supervised (or plain) SGD.
Revised: Theorem 1, we only lost a constant factor of order 𝑚 3 / 2 in our rates compared to fully supervised (or plain) SGD.

Original: This behavior can be checked by adding the plain SGD curve on Figure 2.
Revised: This behavior can be checked by adding the plain SGD curve on Figure 2.

Original: On the left side of Figure 8, we do observe that the risk of both Algorithm 1 and plain SGD decrease with same exponent with respect to number of iteration but with a different constant in front of the rates: that is we observe the same slopes on the logarithm scaled plot, but different intercepts.
Revised: On the left side of Figure 8, we do observe that the risk of both Algorithm 1 and plain SGD decrease with same exponent with respect to number of iteration but with a different constant in front of the rates: that is we observe the same slopes on the logarithm scaled plot, but different intercepts.

Original: Going one step further to check the tightness of our bound, one can plot the intercept, or the error achieved by both
Revised: Going one step further to check the tightness of our bound, one can plot the intercept, or the error achieved by both

Original: Algorithm 1 and plain SGD as a function of the output space dimension 𝑚 .
Revised: Algorithm 1 and plain SGD as a function of the output space dimension 𝑚 .

Original: The right side of Figureshows evidence that this error grows as 𝑚 𝜀 for some 𝜀 ∈ [ 1 , 3 / 2 ] , which is coherent with our upper bound.
Revised: The right side of Figureshows evidence that this error grows as 𝑚 𝜀 for some 𝜀 ∈ [ 1 , 3 / 2 ] , which is coherent with our upper bound.

Original: Similarly to Figure 2, this figure was computed after cross validation to find the best scaling of the step sizes for each dimension 𝑚 . 
Revised: Similarly to Figure 2, this figure was computed after cross validation to find the best scaling of the step sizes for each dimension 𝑚 . T = T = T =

Revised: T =

Revised: T =

Original: E.2 Passive strategies for classification
Revised: T = T = T = T =

Revised: T =

Original:  A simple passive strategy for classification based on median surrogate consists in using the active strategy with coordinates sampling, that is 𝑢 being uniform on (cid:8) 𝑒 𝑦 (cid:9) 𝑦 ∈ Y , where ( 𝑒 𝑦 )
Revised: T = T = E.2 Passive strategies for classification A simple passive strategy for classification based on median surrogate consists in using the active strategy with coordinates sampling, that is 𝑢 being uniform on (cid:8) 𝑒 𝑦 (cid:9) 𝑦 ∈ Y , where ( 𝑒 𝑦 )

Original: 𝑦 ∈ Y is the canonical basis of R Y used to define the simplex Δ Y as the convex hull of this basis.
Revised: 𝑦 ∈ Y is the canonical basis of R Y used to define the simplex Δ Y as the convex hull of this basis.

Original: Querying 1 ⟨ 𝑔 𝜃 ( 𝑥 )− 𝑒 𝑦 ,𝑒 𝑦 ⟩ > 0 is formally equivalent to the query of 1 𝑌 = 𝑦 when 𝑔 𝜃 ( 𝑥 ) ∈ Δ Y .
Revised: Querying 1 ⟨ 𝑔 𝜃 ( 𝑥 )− 𝑒 𝑦 ,𝑒 𝑦 ⟩ > 0 is formally equivalent to the query of 1 𝑌 = 𝑦 when 𝑔 𝜃 ( 𝑥 ) ∈ Δ Y .

Original: This is the baseline we plot on Figure 2.T = T = T =
Revised: This is the baseline we plot on Figure 2.

Original: T =

Original: T =

Original: T =

Original: T =

Original: T =

Original: T =

Original: T = T = T = A more advanced passive baseline is provided by the infimum loss (Cour et al., 2011; Cabannes et al., 2020).
Revised: A more advanced passive baseline is provided by the infimum loss (Cour et al., 2011; Cabannes et al., 2020).

Original: It consists in solving
Revised: It consists in solving

Original: [Equation]
Revised: [Equation]

Original:  where 𝑆 is a random subset of Y and 𝐿 is defined from the original loss ℓ : Y × Y → R as, for 𝑧 ∈ Y , 𝑠 ⊂
Revised: [Figure] [Figure] [Figure] where 𝑆 is a random subset of Y and 𝐿 is defined from the original loss ℓ : Y × Y → R as, for 𝑧 ∈ Y , 𝑠 ⊂

Original: Y and 𝑦 ∈ Y ,
Revised: Y and 𝑦 ∈ Y ,

Original: [Equation]
Revised: [Equation]

Original:  Random subsets 𝑆 could be generated by making sure that the variable ( 𝑦 ∈ 𝑆 ) 𝑦 ∈ Y are independent balanced Bernoulli variables; and by removing the trivial sets 𝑆 = ∅ and 𝑆
Revised: if [Equation] otherwise. Random subsets 𝑆 could be generated by making sure that the variable ( 𝑦 ∈ 𝑆 ) 𝑦 ∈ Y are independent balanced Bernoulli variables; and by removing the trivial sets 𝑆 = ∅ and 𝑆

Original: =
Revised: =

Original: Y from the subsequent distribution.
Revised: Y from the subsequent distribution.

Original: In order to optimize this risk in practice, one can use a parametric model and a surrogate [Figure] [Figure] [Figure]
Revised: In order to optimize this risk in practice, one can use a parametric model and a surrogate differentiable loss together with stochastic gradient descent on the empirical risk.

Original: differentiable loss together with stochastic gradient descent on the empirical risk. For classification with the 0-1 loss, we can reuse the surrogate introduced in Proposition 3 and minimize, assuming that we always observed 
Revised: For classification with the 0-1 loss, we can reuse the surrogate introduced in Proposition 3 and minimize, assuming that we always observed 1 𝑌 𝑖 ∈ 𝑆 𝑖 = 1 for simplicity,

Original: [Equation]
Revised: [Equation]

Original: for simplicity,
Revised: Stochastic gradients are then given by, assuming ties have no probability to happen,

Original: [Equation]
Revised: [Equation]

Original: Stochastic gradients are then given by, assuming ties have no probability to happen, ∇ 𝜃 inf 𝑦 ∈ 𝑆 𝑡 (cid:13)(cid:13) 𝑔 𝜃 ( 𝑋 𝑡 ) − 𝑒 𝑦 (cid:13)(cid:13) = (cid:32) 𝑔 𝜃 ( 𝑋 𝑡 ) − 𝑒 𝑦 ∗ (cid:13)(cid:13) 𝑔 𝜃 ( 𝑋 𝑡 ) − 𝑒 𝑦 ∗ (cid:13)(cid:13) (cid:33)
Revised: with

Original: ⊤ 𝐷𝑔 𝜃 ( 𝑋 𝑡 ) with 𝑦 ∗ : = arg max 𝑦 ∈ 𝑆 𝑡 (cid:10) 𝑔 𝜃 ( 𝑋 𝑡 ) , 𝑒 𝑦 (cid:11) .
Revised: [Equation]

Original: This gives a good passive baseline to compare our active strategy with.
Revised: This gives a good passive baseline to compare our active strategy with.

Original: In our experiments with the
Revised: In our experiments with the

Original: Gaussian kernel, see Figure 9 for an example, we witness that this baseline is highly competitive.
Revised: Gaussian kernel, see Figure 9 for an example, we witness that this baseline is highly competitive.

Original: Although we find that it is slightly harder to properly tune the step size for SGD, and that the need to compute an argmax for each gradient slows-down the computations.
Revised: Although we find that it is slightly harder to properly tune the step size for SGD, and that the need to compute an argmax for each gradient slows-down the computations.

Original: E.3 Real-world classification datasets
Revised: E.3 Real-world classification datasets

Original:  In Figure 10, we compare the “well-conditioned” passive baseline with our active strategy on the real-world problems of LIBSVM (Chang and Lin, 2011).
Revised: [Figure] In Figure 10, we compare the “well-conditioned” passive baseline with our active strategy on the real-world problems of LIBSVM (Chang and Lin, 2011).

Original: We choose the “USPS” and “pen digits”iteration T 0 .0 .0 . R i s k o f ¯ θ T “usps” dataset active passive random      iteration T 0 .0 .0 .0 . R i s k o f ¯ θ T “pendigits” dataset active passive random datasets as they contain 𝑚 = 10 classes each with 𝑛 = 7291 and 𝑛 = 7494 samples respectively, with 𝑑 = 50 and 𝑑 = 16 features each.
Revised: We choose the “USPS” and “pen digits” datasets as they contain 𝑚 = 10 classes each with 𝑛 = 7291 and 𝑛 = 7494 samples respectively, with 𝑑 = 50 and 𝑑 = 16 features each.

Original: We have chosen those datasets as they present enough classes that leads to many different sets 𝑆 to query, and they are made of the right number of samples to do some experiments on a laptop without the need for “advanced” computational techniques such as caching or low-rank approximation (Meanti et al., 2020).
Revised: We have chosen those datasets as they present enough classes that leads to many different sets 𝑆 to query, and they are made of the right number of samples to do some experiments on a laptop without the need for “advanced” computational techniques such as caching or low-rank approximation (Meanti et al., 2020).

Original: On Figure 10, we use the same linear model as for Figure 2, that is a Gaussian kernel.
Revised: On Figure 10, we use the same linear model as for Figure 2, that is a Gaussian kernel.

Original: We choose the bandwidth to be 𝜎 = 𝑑 / 5, and we normalize the features beforehand to make sure that they are all centered with unit variance.
Revised: We choose the bandwidth to be 𝜎 = 𝑑 / 5, and we normalize the features beforehand to make sure that they are all centered with unit variance.

Original: We report error by taking two thirds of the samples for training and one third for testing, and averaging over one hundred different ways of splitting the datasets.
Revised: We report error by taking two thirds of the samples for training and one third for testing, and averaging over one hundred different ways of splitting the datasets.

Original: We observe that the active strategy leads to important gains on the “USPS” dataset, yet is not that useful for the “pen digits” dataset.
Revised: We observe that the active strategy leads to important gains on the “USPS” dataset, yet is not that useful for the “pen digits” dataset.

Original: We have not dug in to understand those two different behaviors.
Revised: We have not dug in to understand those two different behaviors.

Original: E.4 Real-world regression dataset & Nyström method
Revised: E.4 Real-world regression dataset & Nyström method

Original: In this section, we provide two experiments on real-world datasets.
Revised: In this section, we provide two experiments on real-world datasets.

Original: In order to deal with big regression datasets, it is useful to approximate the parameter space Y ⊗ H in
Revised: In order to deal with big regression datasets, it is useful to approximate the parameter space Y ⊗ H in

Original: Assumption 1 with a small dimensional space.
Revised: Assumption 1 with a small dimensional space.

Original: To do so, let us remark that given samples ( 𝑋 𝑖 ) 𝑖 ≤ 𝑛 ∈ X 𝑛 for 𝑛 ∈ N , we know that our estimate 𝑓 𝜃 𝑛 can be represented as
Revised: To do so, let us remark that given samples ( 𝑋 𝑖 ) 𝑖 ≤ 𝑛 ∈ X 𝑛 for 𝑛 ∈ N , we know that our estimate 𝑓 𝜃 𝑛 can be represented as

Original: [Equation]
Revised: [Equation]

Original: for some ( 𝑎 𝑖𝑗 ) ∈ R 𝑝 × 𝑚 and where ( 𝑒 𝑗 ) 𝑗 ≤ 𝑚 is the canonical basis of Y = R 𝑚 .
Revised: for some ( 𝑎 𝑖𝑗 ) ∈ R 𝑝 × 𝑚 and where ( 𝑒 𝑗 ) 𝑗 ≤ 𝑚 is the canonical basis of Y = R 𝑚 .

Original: For large datasets, that is when 𝑛 is large, it is smart to approximate this representation through the parameterization
Revised: For large datasets, that is when 𝑛 is large, it is smart to approximate this representation through the parameterization

Original: [Equation]
Revised: [Equation]

Original: where 𝑝 ≤ 𝑛 is the rank of our approximation, and 𝑘 is the kernel defined as 𝑘 ( 𝑥, 𝑥 ′ )
Revised: where 𝑝 ≤ 𝑛 is the rank of our approximation, and 𝑘 is the kernel defined as 𝑘 ( 𝑥, 𝑥 ′ )

Original: = ⟨ 𝜑 ( 𝑥 ) , 𝜑 ( 𝑥 ′ )⟩ .
Revised: = ⟨ 𝜑 ( 𝑥 ) , 𝜑 ( 𝑥 ′ )⟩ .

Original: Stated with words, we only use a small number 𝑝 , instead of 𝑛 , of vectors 𝜑 ( 𝑥 𝑖 ) to parameterize 𝑓 .
Revised: Stated with words, we only use a small number 𝑝 , instead of 𝑛 , of vectors 𝜑 ( 𝑥 𝑖 ) to parameterize 𝑓 .

Original: This allows to only keep a matrix of size 𝑝 × 𝑚 in memory instead of 𝑛 × 𝑚 , while not fundamentally changing the statistical guarantee of the method (Rudi et al., 2015).
Revised: This allows to only keep a matrix of size 𝑝 × 𝑚 in memory instead of 𝑛 × 𝑚 , while not fundamentally changing the statistical guarantee of the method (Rudi et al., 2015).

Original: In this setting, the stochastic gradients are specified from the fact that
Revised: In this setting, the stochastic gradients are specified from the fact that

Original: [Equation]
Revised: [Equation]

Original: In other terms, in order to update the parameter 𝑎 with respect to the observation made at ( 𝑥, 𝑢 ) , we check how much each coordinate of 𝑎 determines the value of 𝑢 ⊤ 𝑓 𝑎 ( 𝑥 ) .
Revised: In other terms, in order to update the parameter 𝑎 with respect to the observation made at ( 𝑥, 𝑢 ) , we check how much each coordinate of 𝑎 determines the value of 𝑢 ⊤ 𝑓 𝑎 ( 𝑥 ) .

Original: In the following, we experiment with two real-world datasets.
Revised: In the following, we experiment with two real-world datasets.

Original: In order to learn the relation between inputs and outputs, we use a Gaussian kernel after normalizing input features so that each of them has zero mean and unit variance.
Revised: In order to learn the relation between inputs and outputs, we use a Gaussian kernel after normalizing input features so that each of them has zero mean and unit variance.

Original: To keep computational cost, we sample 𝑝 random (Nyström) representers among the training inputs which are used to parameterize functions.
Revised: To keep computational cost, we sample 𝑝 random (Nyström) representers among the training inputs which are used to parameterize functions.

Original: To avoid overfitting, we add a small regularization to the empirical objective.
Revised: To avoid overfitting, we add a small regularization to the empirical objective.

Original: It reads 𝜆 ∥ 𝜃 ∥ 2 H with our notations and corresponds to the Hilbertian norm inherited from the reproducing kernel 𝑘 of the function 𝑓 𝜃 (Scholkopf and Smola, 2001).
Revised: It reads 𝜆 ∥ 𝜃 ∥ 2 H with our notations and corresponds to the Hilbertian norm inherited from the reproducing kernel 𝑘 of the function 𝑓 𝜃 (Scholkopf and Smola, 2001).

Original: [Figure]
Revised: [Figure]

Original: Our first experiment is based on the data collected by the California Cooperative Oceanic Fisheries
Revised: Our first experiment is based on the data collected by the California Cooperative Oceanic Fisheries

Original: Investigation between March 1949 and November 2016.
Revised: Investigation between March 1949 and November 2016.