Original: Generalization Analysis on Learning with a Concurrent Verifier
Revised: Generalization Analysis on Learning with a Concurrent Verifier

Original: Author s)

Original: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: Machine learning technologies have been used in a wide range of practical systems.
Revised: Machine learning technologies have been used in a wide range of practical systems.

Original: In practical situations, it would be natural to expect  input-output pairs of a machinelearning model to satisfy some requirements.
Revised: In practical situations, it is natural to expect the input-output pairs of a machine learning model to satisfy some requirements.

Original: However, it is hard to obtain a modelsatisfying the requirements by just learning from examples.
Revised: However, it is difficult to obtain a model that satisfies requirements by just learning from examples.

Original: A simple solution isto add a module that checks whether the input-output pairs meet the requirementsand then modifies the model’s outputs.
Revised: A simple solution is to add a module that checks whether the input-output pairs meet the requirements and then modifies the model’s outputs.

Original: Such a module,  we call  concurrent verifier(CV), can give a certification, but how the generalizability of the machine learningmodel changesby using a CV is unclear.
Revised: Such a module, which we call a concurrent verifier (CV), can give a certification, although how the generalizability of the machine learning model changes using a CV is unclear.

Original: This paper gives a generalization analysisof learning with a CV.
Revised: This paper gives a generalization analysis of learning with a CV.

Original: We analyze how the learnability of a machine learningmodel changes when we use a CV, and show the condition where we can obtain aguaranteed hypothesis using a verifier only in the inference time.
Revised: We analyze how the learnability of a machine learning model changes with a CV and show a condition where we can obtain a guaranteed hypothesis using a verifier only in the inference time.

Original: We also showthat typical error bounds based onthe Rademacher complexity will be no largerthan that of the original model when using a CV in multi-class classification andstructured prediction settings.
Revised: We also show that typical error bounds based on Rademacher complexity will be no larger than that of the original model when using a CV in multi-class classification and structured prediction settings.

Original:  Introduction
Revised:  Introduction

Original: As machine learning technology matures, many systems exploiting machine learning models havebeen developed.
Revised: As machine learning technology matures many systems have been developed that exploit machine learning models.

Original: When developing a system that uses a machine learning model, a model with merelya small prediction error would notbe satisfactory due to real-field requirements.
Revised: When developing a system that uses a machine learning model a model with merely small prediction error is not satisfactory due to real-field requirements.

Original: For example, anobject recognition model  sensitive to slight noise would cause security issues [4, 26] or a modelwith unexpected output would increase the system’s cost for dealing with them.
Revised: For example, an object recognition model that is sensitive to slight noise would cause security issues [4, 28] or a model with unexpected output would increase a system’s cost for dealing with it.

Original: Thus, we wantinput-output pairs of a machine learning model to satisfy some requirements .
Revised: Thus, we want the input-output pairs of a machine learning model to satisfy some requirements .

Original: However, it is hardto obtain a model satisfying the requirements by just learning from examples.
Revised: However, it is difficult to obtain a model that satisfies the requirements by just learning from examples.

Original: Moreover, since thelearned models tend to be complex and the input domain tends to be quite large, it would not berealistic to certify that every input-output pair satisfies  requirements.
Revised: Moreover, since the learned models tend to be complex and the input domain tends to be quite large, it is unrealistic to certify that every input-output pair satisfies the requirements.

Original: In addition, even if we find aninput-output pair that does not satisfy the requirements, it would not be easy to modify a model sincewe have to re-estimate it from  training examples.
Revised: In addition, even if we find an input-output pair that does not satisfy the requirements, modifying a model is difficult since we have to re-estimate it from the training examples.

Original: This paper considers a way to obtain a machine learning model that makes its input-output pairssatisfy required properties.
Revised: This paper considers a way to obtain a machine learning model whose input-output pairs satisfy the required properties.

Original: Here, we consider the following assumptions on the situation where amachine learning model is used.
Revised: We address the following assumptions for a situation where a machine learning model is used.

Original: First, we cancertainly judge whetheran input-output pair ( x, h ( x ))satisfies the requirements or not, where h : X → Y is a machine learning model or a hypothesis.
Revised: First, we can judge whether input-output pair ( x, h ( x )) satisfies the requirements, where h : X → Y is a machine learning model or a hypothesis.

Original: Aswe will show later, important use cases fit this setting.
Revised: As we show below, important use cases fit this setting.

Original: Second,there already exists a machine learningmodel whose prediction error is small enough, but its input-output pairs are not guaranteed to satisfythe requirements.
Revised: Second, a machine learning model already exists whose prediction error is small enough, although its input-output pairs are not guaranteed to satisfy the requirements.

Original: This second assumption is also reasonable since modern machine learning modelsshow high enough prediction accuracy in various tasks.
Revised: This second assumption is also reasonable since modern machine learning models show sufficient prediction accuracy in various tasks.

Original: Under these assumptions, a practical choicefor addressing this problem is not to change the machine learning model but adding a module thatchecks input-output pairs of machine learning model h .
Revised: Under these assumptions, a practical choice for addressing this problem isn’t changing the machine learning model but adding a module that checks the input-output pairs of machine learning model h .

Original: We call the module a concurrent verifier (CV).
Revised: We call this module a concurrent verifier (CV).

Original: [Figure]
Revised: Fig.

Original: Figure 1 shows the system configuration of a machine learning model with a CV.
Revised: 1 shows the system configuration of a machine learning model with a CV.

Original: The verifier checkswhether the input-output pair (x, h ( x )) satisfies the required properties.
Revised: The verifier checks whether the input-output 36th Neural Processing Systems (

Original:  If it satisfies requirements,the verifier outputs h ( x ) .
Revised: [Figure] pair ( x, h ( x )) satisfies the required properties. If it satisfies the requirements, it outputs h ( x ) .

Original: If not, then it rejects h ( x ) and modifies or asks the learning model to modifyits output.
Revised: If not, then it rejects h ( x ) and modifies or requests the learning model to modify its output.

Original: Apair of a machine learning model anda verifier  can be seen as another machine learningmodel whose input-output pairs are guaranteed to satisfy the required conditions.
Revised: A machine learning model and verifier pair can be seen as another machine learning model whose input-output pairs are guaranteed to satisfy the required conditions.

Original: Although a model with a verifier can guarantee that its input-output pairs satisfy requirements, itseffect on prediction error is unclear.
Revised: Although a model with a verifier can guarantee that its input-output pairs satisfy requirements, its effect on prediction error is unclear.

Original: This paper gives theoretical analyses of  generalization errors ofa machine learning model with a CV.
Revised: This paper gives theoretical analyses of the generalization errors of a machine learning model with a CV.

Original: We focus on how the learnability of the original model, denotedas hypothesis class H , can change by using the verifier.
Revised: We focus on how the learnability of the original model, denoted as hypothesis class H , can change by using the verifier.

Original: First, we consider the situation where we usea CV only in the inference phase.
Revised: First, we consider a situation where we use a CV only in the inference phase.

Original: This setting corresponds to the case where  required properties areunknown when we are in the training phase.
Revised: This setting corresponds to a case where the required properties are unknown when we are in the training phase.

Original: We show that if the hypothesis class is PAC learnable,we can obtain a guaranteed hypothesisby using a verifier only in the inference time.
Revised: If the hypothesis class is PAC-learnable, we can obtain a guaranteed hypothesis using a verifier only in the inference time.

Original: Second, we consider the situation where we know the requirements when learning the model.
Revised: Second, we consider a situation where we know the requirements when learning the model.

Original: Thissituation corresponds to viewing the learnability of hypothesis set H c , which is obtained by modifyingevery hypothesis h ∈ H to satisfy the requirements.
Revised: This situation corresponds to viewing the learnability of hypothesis set H c , which is obtained by modifying every hypothesis h ∈ H to satisfy the requirements.

Original: Hence we compare the generalization errorupper bounds of H c with those of H .
Revised: Hence we compare the generalization error upper bounds of H c with those of H .

Original: On the multi-class classification setting, we show that existingerror bounds [13, 16] based on the Rademacher complexity of H are also bounds of the modifiedhypothesis H c for any input-output requirements.
Revised: On the multi-class classification setting, we show that existing error bounds [15, 18] based on the Rademacher complexity of H are also bounds of modified hypothesis H c for any input-output requirements.

Original: Moreover, we give similar analyses for thestructured prediction task. A structured prediction  is a kind of multi-class classification where theset of classes Y can be decomposed into substructures, and it is worth analyzing the task sincethereare many works considering constraints in structured prediction.
Revised: Moreover, we give similar analyses for a structured prediction task, which is a kind of multi-class classification where set of classes Y can be decomposed into substructures. It is worth analyzing the task since many works address the constraints in structured prediction.

Original: Some works give error boundsfor structured prediction tasks, which are tighter than simply applying the bound for multi-classclassification task [14, 6, 17].
Revised: Some works give error bounds for structured prediction tasks, which are tighter than simply applying the bound for multi-class classification tasks [16, 6, 19].

Original: Similar to the case of multi-class classification, we show that existing Rademacher complexity-based bounds for  structured prediction of H are also the bounds for H c .
Revised: Similar to the case of multiclass classification, we show that existing Rademacher complexity-based bounds for the structured prediction of H are also the bounds for H c .

Original: Our main contributions are as follows: a) We introduce a concurrent verifier,  a model-agnostic wayto guarantee  machine learning models satisfy  required properties.
Revised: Our main contributions are as follows: a) We introduce a concurrent verifier, which is a modelagnostic way to guarantee that machine learning models satisfy the required properties.

Original: Although a similar mechanismwas used in some existing models, our model gives a generalization analysis that does not dependon a specific model.
Revised: Although a similar mechanism was used in some existing models, our model gives a generalization analysis that does not depend on a specific model.

Original: b) We show that if hypothesis class H is PAC-learnable, then using a verifierat inference time can give a hypothesis with a guarantee in its generalization error.
Revised: b) We show that if hypothesis class H is PAC-learnable, then using a verifier at the inference time can give a hypothesis with a guarantee in its generalization error.

Original: Interestingly, if H is not PAC learnable, we might fail to obtain a guaranteed hypothesis even if  requirements areconsistent with distribution D .
Revised: Interestingly, if H is not PAC-learnable, we might fail to obtain a guaranteed hypothesis even if the requirements are consistent with distribution D .

Original: c)
Revised: c)

Original: We show that if we use a CV in a learning phase of multi-classclassification tasks, then the theoretical error bounds of H based on the Rademacher complexity willnot increase with any input-output requirements.
Revised: We show that if we use a CV in a learning phase of multi-class classification tasks, then the theoretical error bounds of H based on the Rademacher complexity will not increase with any input-output requirements.

Original: We also give similar results for structured predictiontasks.
Revised: We also give similar results for structured prediction tasks.

Original: 1.1 Use Cases of a Concurrent Verifier
Revised: 1.1 Use Cases of a Concurrent Verifier

Original: We show some typical use cases of CVs.
Revised: The following are some typical use cases for CVs.

Original: Error-sensitive applications A typical situation where we want to use a verifier is that someprediction errors might cause severe effects, and we want to avoid them.
Revised: Error-sensitive applications: A typical situation where we want to use a verifier is that some prediction errors might cause severe effects, which we want to avoid.

Original: For example, a recommendersystem might limit the set of candidate items depending on user attributes.
Revised: For example, a recommender system might limit the set of candidate items depending on user attributes.

Original: Although such a rulemight degrade the prediction accuracy, practically a safer model is preferable.
Revised: Although such a rule might degrade the prediction accuracy, practically a safer model is preferable.

Original: Controlling outputs of structured prediction Constraints are frequently used in structured predictiontasks for improving the performance orimproving the controllability of  outputs.
Revised: Controlling outputs of structured prediction: Constraints are frequently used in structured prediction tasks for improving the performance or the controllability of the outputs.

Original: For example, [21, 5]exploits constraints on sequence labeling tasks for reflecting background knowledge to improve theprediction results.
Revised: For example, some works [23, 5] exploited the constraints on sequence labeling tasks for reflecting background knowledge to improve the prediction results.

Original: More recently, there are works [9, 2] exploiting constraints in language generationtasks, including image captioning and machine translation, where they restrict a model to output a sentence that includes given keywords.
Revised: More recently, some works [9, 2] exploited the constraints in language generation tasks, including image captioning and machine translation, and restricted a model to output a sentence that includes given keywords.

Original: Since the constraints used in this previous work can bewritten as a logical formula, our CV model can represent them as requirements.
Revised: Since the constraints used in this previous work can be written as a logical formula, our CV model can represent them as requirements.

Original: Robustness against input perturbations If a machine learning model changes its output if we modifyits input  x to x ′ that is very close to x , then the model is called sensitive against a small change [25].
Revised: Robustness against input perturbations: If a machine learning model changes its output because we modified its input from x to x ′ , which is very close to x , then the model is described as sensitive against a small change [27].

Original: It might be a security risk if a model is sensitive since its behavior is unpredictable.
Revised: It might be a security risk if a model is sensitive since its behavior is unpredictable.

Original: Therefore, somemethods evaluate and verify the robustness of neural networks against small perturbations [26, 4].
Revised: Therefore, some methods evaluate and verify the robustness of neural networks against small perturbations [28, 4].

Original: Existing verification method checks a machine learning model’s robustness around input x bysearching whetherthere exists x ′  that is close to x and the model f gives different outputs, i.e.,h ( x ) ̸ = h ( x ′ ) , for verification samples x 1 , . . .
Revised: Existing verification methods check a machine learning model’s robustness around input x by determining whether x ′ exists that is close to x and whether model f gives different outputs, i.e., h ( x ) ̸ = h ( x ′ ) , for verification samples x 1 , . . .

Original: , x n .
Revised: , x n .

Original: However, although these verification methodscan test a model, they do not directly show a way to obtain a robust model.
Revised: Although these verification methods can test a model, they do not directly show how to obtain a robust model.

Original: A CV canbe used to fix a model to achieve robustness around samples x 1 , . .
Revised: A CV can fix a model to achieve robustness around samples x 1 , . .

Original: , x n by setting a ruleof the form: “ h ( x ′ ) mustbe equalto h ( x i ) if x ′ is close to x i ”.
Revised: , x n by setting a rule of form: “ h ( x ′ ) must equal h ( x i ) if x ′ is close to x i .”

Original: Although this solution might notguarantee robustness where samples are scarce, adding enough non-labeled verification samples canbe a reasonable choice.
Revised: Although this solution might not guarantee robustness where samples are scarce, adding enough non-labeled verification samples is often a reasonable choice.

Original:  Related work
Revised:  Related Work

Original: Machine learning models that can exploit constraints have been investigated in many researchfields, including statistical symbolic learning and structured prediction.
Revised: Machine learning models that can exploit constraints have been investigated in many research fields, including statistical symbolic learning and structured prediction.

Original: For example, Markov logicnetworks [20], Problogs [8], and probabilistic circuit models [11] integrate statistical models withsymbolic logic formulations.
Revised: For example, Markov logic networks [22], Problogs [8], and probabilistic circuit models [11] integrate statistical models with symbolic logic formulations.

Original: Since these models can incorporate hard constraints represented bysymbolic logic, they can guarantee input-output pairs.
Revised: Since these models can incorporate hard constraints represented by symbolic logic, they can guarantee input-output pairs.

Original: However, previous research focused on itspractical performance and gave little theoretical analysis of its learnability when hard constraints areused.
Revised: However, previous research focused on their practical performance and gave little theoretical analysis of their learnability when hard constraints are used.

Original: Moreover, previous works integrated the ability to exploit constraints into specific models.
Revised: Moreover, previous works integrated the ability to exploit constraints into specific models.

Original: Incontrast, our CV is model-agnostic and can be used in combination with a wide range of machinelearning models.
Revised: In contrast, our CV is model-agnostic and can be used in combination with a wide range of machine learning models.

Original: Recently,  verification of machine learning models has gathered attention.
Revised: Recently, the verification of machine learning models has been gathering more attention.

Original: There are attempts to checkwhether a machine learning model has the desired property[4, 26, 10, 24].
Revised: Attempts have verfified whether a machine learning model has the desired properties [4, 28, 10, 26].

Original: There are exact verificationmethods using integer programming (MIP) [26], constraint satisfaction (SAT) [18], and satisfiablemodule theory (SMT) solver [10] to assess the robustness of a neural network model against inputnoise.
Revised: Exact verification methods use integer programming (MIP) [28], constraint satisfaction (SAT) [20], and a satisfiable module theory (SMT) solver [10] to assess the robustness of a neural network model against input noise.

Original: These approaches aim at obtaining models that fulfill  required properties.
Revised: These approaches aim to obtain models that fulfill the required properties.

Original: However, verificationmethods cannot help modify  models if the model does not satisfy the requirements.
Revised: However, verification methods cannot help modify the models if they do not satisfy the requirements.

Original: If we want MLmodels to meet requirements, post-processing would be needed as our concurrent verification model.
Revised: If we want ML models to meet requirements, post-processing is needed as our concurrent verification model.

Original: There are methodsthat can give upper bounds on generalization error, including VC-dimension [27]and its extensions [7, 19], Rademacher complexity [3, 12], stability [23], and PAC-Bayes [15, 1].
Revised: Other methods can give upper bounds on generalization error, including VC-dimension [29] and its extensions [7, 21], Rademacher complexity [3, 12], stability [25], and PAC-Bayes [17, 1].

Original: Weuse Rademacher complexity in the following analysis since they are among the most popular toolsfor giving theoretical upper bounds onthe generalization error.
Revised: We use Rademacher complexity in the following analysis since it is among the most popular tools for giving theoretical upper bounds on generalization error.

Original: Moreover, Rademacher complexityhas some extensions, including local Rademacher complexity [13] and factor graph Rademachercomplexity [6].
Revised: Rademacher complexity also has some extensions, including local Rademacher complexity [15] and factor graph Rademacher complexity [6].

Original: We can provide theoretical guarantees on these extended measures.
Revised: We can provide theoretical guarantees on these extended measures.

Original:  Preliminaries
Revised:  Preliminaries

Original: Our notation follows [22].
Revised: Our notation follows a previous work [24].

Original: We first introduce the notations used in the following sections.
Revised: We first introduce the notations used in the following sections.

Original: Let Xdenote the domain of  inputs,  Y be the domain of label, and  Z be the domain of  examples definedas Z := X × Y .
Revised: Let X denote the domain of the inputs, let Y be the domain of the labels, and let Z be the domain of the examples defined as Z := X ×Y .

Original: Let H be a hypothesis class, and  ℓ : H × Z → R + be  loss function.
Revised: Let H be a hypothesis class, and let ℓ : H× Z → R + be a loss function.

Original: A sample S=
Revised: Training data S

Original:  ( z 1 , . . .
Revised: = ( z 1 , . . .

Original: , z m ) ∈
Revised: , z m ) ∈

Original: Z m is a finite sequence of size m drawn i.i.d.
Revised: Z m is a finite sequence of size m drawn i.i.d.

Original: from a fixed but unknownprobability distribution D on Z .
Revised: from a fixed but unknown probability distribution D on Z .

Original: A learning algorithm
Revised: Learning algorithm

Original: A maps a sample toa hypothesis h .
Revised: A maps training data S to hypothesis h .

Original: We usethe notation A ( S ) to denote the hypothesis thatthe learning algorithm
Revised: We use notation A ( S ) to denote the hypothesis that learning algorithm

Original: A returns upon receiving S .
Revised: A returns upon receiving S .

Original: We represent set { 1 , . .
Revised: We represent set { 1 , . .

Original: , K } as [ K ] .
Revised: , K } as [ K ] .

Original: Given distribution D on Z , we denote by L D ( h ) the generalization error and by L S ( h ) the empiricalerror of h over S  defined by:
Revised: Given distribution D on Z , we denote by L D ( h ) the generalization error and by L S ( h ) the empirical error of h over S , defined by

Original: [Equation]
Revised: [Equation]

Original: 129 PAC learnability We introduce PAC learnability and agnostic PAC learnability as follows.
Revised: PAC learnability: We introduce PAC learnability and agnostic PAC learnability as follows.

Original: Definition 3.1.
Revised: Definition 3.1.

Original: (Agnostic PAC learnability)
Revised: (Agnostic PAC learnability)

Original: Hypothesis class H is agnostic PAC learnable if thereexists a function m H : (0 , 1) [-
-] → N anda learning algorithm A with the following property: For everyϵ, δ ∈ (0 , 1) and distribution D over Z , if Sis a sample consists of m ≥ m H ( ϵ, δ ) i.i.d.
Revised: Hypothesis class H is agnostic PAC-learnable if there exists function m H : (0 , 1) [-
-]2 → N and learning algorithm A with the following property: For every ϵ, δ ∈ (0 , 1) and distribution D over Z , if S consists of m ≥ m H ( ϵ, δ ) i.i.d.

Original: examplesgenerated by D , then, with at least probability 1 − δ , the following holds:
Revised: examples generated by D , then with at least probability 1 − δ , the following holds:

Original: [Equation]
Revised: [Equation]

Original: A distribution D is realizable by hypothesis set H ifthere exists h ∗ ∈ H  such that L D ( h ∗ )
Revised: Distribution D is realizable by hypothesis set H if h ∗ ∈ H exists such that L D ( h ∗ )

Original: = 0 .
Revised: = 0 .

Original: If Dis realizable by agnostic PAC learnable hypothesis H , thenwe say H is PAC learnable .
Revised: If D is realizable by agnostic PAC-learnable hypothesis H , then H is PAC-learnable .

Original: If H is PAClearnable, then  (2) becomes: L D ( A ( S ))
Revised: If H is PAC-learnable, then Eq. (2) becomes L D ( A ( S ))

Original: ≤ ϵ since min h ′ ∈H L D ( h ′ )
Revised: ≤ ϵ since min h ′ ∈H L D ( h ′ )

Original: = 0 .
Revised: = 0 .

Original: Rademacher complexity In the following sections, we use Rademacher complexity for derivinggeneralization bounds.
Revised: Rademacher complexity: In the following sections, we use Rademacher complexity for deriving the generalization bounds.

Original: Given loss function ℓ ( h, z ) and hypothesis class H , let us denote G as
Revised: Given loss function ℓ ( h, z ) and hypothesis class H , we denote G as

Original: [Equation]
Revised: [Equation]

Original: Definition 3.2.
Revised: Definition 3.2.

Original: (Empirical Rademacher complexity) Let G be a family of functions mapping from Z to R  and  S =
Revised: (Empirical Rademacher complexity) Let G be a family of functions mapping from Z to R , and let S =

Original: ( z 1 , . . .
Revised: ( z 1 , . . .

Original: , z m ) ∈
Revised: , z m ) ∈

Original: Z m be a sample of size m .
Revised: Z m be the training data of size m .

Original: Then, the empirical Rademachercomplexity of G with respect tosample S is defined as:
Revised: Then the empirical Rademacher complexity of G with respect to S is defined:

Original: [Equation]
Revised: [Equation]

Original: where σ = ( σ 1 , . .
Revised: where σ = ( σ 1 , . .

Original: , σ m ) ∈ {± 1 } m are random variables distributed i.i.d.
Revised: , σ m ) ∈ {± 1 } m are random variables distributed i.i.d.

Original: according to P [ σ i =1] = P [ σ i = − 1] = 1 / 2 .
Revised: according to P [ σ i = 1] = P [ σ i = − 1] = 1 / 2 .

Original: The Rademacher complexity of G is defined as the expected value of theempirical Rademacher complexity, defined as:
Revised: The Rademacher complexity of G is defined as the expected value of the empirical Rademacher complexity:

Original: [Equation]
Revised: [Equation]

Original:  Concurrent Verifier
Revised:  Concurrent Verifier

Original: We give a formal definition of a CV.
Revised: Next we give a formal definition of a CV.

Original: A CV works with a machine learning model, which is a functionh : X → Y .
Revised: A CV works with a machine learning model, which is function h : X → Y .

Original: If x is given to the model and the model outputs h ( x ) , then the verifier checks whether( x, h ( x )) satisfies the required property.
Revised: If x is given to the model, which outputs h ( x ) , then the verifier checks whether ( x, h ( x )) satisfies the required property.

Original: We assume that the required property can be representedas requirement function c : ( X × Y ) → { 0 , 1 } .
Revised: We assume that the required property can be represented as requirement function c : ( X × Y ) → { 0 , 1 } .

Original: If c ( x, h ( x ))
Revised: If c ( x, h ( x ))

Original: = 1  then the pair satisfies the propertyand if c ( x, h ( x ))
Revised: = 1 , then the pair satisfies the property; if c ( x, h ( x ))

Original: = 0  then  does not satisfy.
Revised: = 0 , then it does not.

Original: Requirement function c can be represented by a set ofdeterministic rules.
Revised: Requirement function c can be represented by a set of deterministic rules.

Original: For example, if X = R and Y = { 0 , 1 } , then  requirements can be in the form:“if x > 0 , then y ̸ = 0 ”.
Revised: For example, if X = R and Y = { 0 , 1 } , then the requirements can be in the following form: “if x > 0 , then y ̸ = 0 .”

Original: We assume that for all possible input x ∈ X , there exists y ∈ Y such thatc ( x, y )
Revised: We assume that for all possible input x ∈ X , there exists y ∈ Y such that c ( x, y )

Original: = 1 for avoiding the situation that the requirements are unsatisfiable for any output y .
Revised: = 1 for avoiding the situation where the requirements are unsatisfiable for any output y .

Original: It is notdifficult to relax this assumption  if we allow a machine learning model to reject unsatisfiable input x .
Revised: This assumption can be easily relaxed if we allow a machine learning model to reject unsatisfiable input x .

Original: After checking the input-output pair, a verifier modifies output h ( x ) depending on the value ofc ( x, h ( x )) .
Revised: After checking the input-output pair, a verifier modifies output h ( x ) depending on the value of c ( x, h ( x )) .

Original: If c ( x, h ( x ))
Revised: If c ( x, h ( x ))

Original: = 1 , the verifier outputs h ( x ) since it satisfies  requirements.
Revised: = 1 , the verifier outputs h ( x ) since it satisfies the requirements.

Original: If c ( x, h ( x ))
Revised: If c ( x, h ( x ))

Original: =0 , then the verifier will modify h ( x ) to some y ∈ Y satisfying c ( x, y )
Revised: = 0 , then the verifier modifies h ( x ) to some y ∈ Y that satisfies c ( x, y )

Original: = 1 .
Revised: = 1 .

Original: Therefore, if we use averifier with a machine learning model corresponding to h , then the combination of the model andthe verifier can be seen asa function h c : X → Y  defined as:
Revised: If we use a verifier with a machine learning model that corresponds to h , then the combination of the model and the verifier can be seen as function h c : X → Y , defined as

Original: [Equation]
Revised: [Equation]

Original: (3) where y c ∈ Y satisfies c ( x, y c )
Revised: 3) where y c ∈ Y satisfies c ( x, y c )

Original: = 

Revised: =

Original: and is selected deterministically.
Revised: and is selected deterministically.

Original: When Y =
Revised: When Y =

Original: [ K ] , an examplefor selecting minimum i ∈
Revised: [ K ] , an example for selecting minimum i ∈

Original: [ K ] satisfying c ( x, i )
Revised: [ K ] satisfying c ( x, i )

Original: = 1 as y c would be a reasonable choice.
Revised: = 1 as y c is a reasonable choice.

Original: When Y =
Revised: When Y =

Original: [ K ] and h ( x ) is made by scoring functions
Revised: [ K ] and h ( x ) is made by scoring functions

Original: h ( x, y ) : ( X × Y ) → R , it would be also reasonableto select y ∗ such that y ∗ =
Revised: h ( x, y ) : ( X × Y ) → R , it is also reasonable to select y ∗ such that y ∗ =

Original: argmax y ∈Y ,c ( x,y )
Revised: argmax y ∈Y ,c ( x,y )

Original: =1 h ( x, y ) .
Revised: =1 h ( x, y ) .

Original: Learning a model corresponds to selectinga hypothesis h from hypothesis class H .
Revised: Learning a model corresponds to selecting hypothesis h from hypothesis class H .

Original: Therefore, learning a model with  CV corresponds tochoosing a hypothesis from the modified hypothesis class: H c = { h c : h ∈ H} .
Revised: Therefore, learning a model with a CV corresponds to choosing a hypothesis from the modified hypothesis class: H c = { h c : h ∈ H} .

Original: By definition,every hypothesis in H c satisfies the requirements, and thus we can guarantee  the model satisfiesthe condition if we select a hypothesis from H c .
Revised: By definition, every hypothesis in H c satisfies the requirements, and thus we can guarantee that the model satisfies the condition if we select a hypothesis from H c .

Original: In the following sections, wewill analyze thelearnability of H c by comparing it with that of H .
Revised: In the following sections, we analyze the learnability of H c by comparing it with that of H .

Original:  Inference Time Verification
Revised:  Inference Time Verification

Original: We first analyze the change of  generalization errors when we use a verifier only in an inference phase.
Revised: We first analyze the change of the generalization errors when we use a verifier only in an inference phase.

Original: In other words, requirements are unknown in the learning phase and we estimate hypothesis ˆ h
Revised: In other words, requirements are unknown in the learning phase,and we estimate hypothesis ˆ h

Original: = A ( S )from hypothesis class H by using training sample S and algorithm A .
Revised: = A ( S ) from hypothesis class H by using training data S and algorithm A .

Original: In the inference phase, we usea CV to modify ˆ h to ˆ h c .
Revised: In the inference phase, we use a CV to modify ˆ h to ˆ h c .

Original: We call this setting  inference time verification (ITV).
Revised: We call this setting the inference time verification (ITV).

Original: This class of situationscontains many exciting settings. Typical settings are 1) pre-trained machine learning models used ina wide range of applications, and 2) models that are hard to replace, which might encounter differentrequirements from ones at the learning time in the long run.
Revised: This class of situations contains many exciting settings: 1) pre-trained machine learning models used in a wide range of applications, and 2) models that are hard to replace, which might encounter different requirements from those at the learning time in the long run.

Original: In this section, wewill give analyses on a multi-class classification setting.
Revised: In this section, we give analyses on a multi-class classification setting.

Original: We set Y =
Revised: We set Y =

Original: [ K ] andhypothesis class H isa set of mappings h : X → [ K ] .
Revised: [ K ] , and hypothesis class H is set of mappings h : X → [ K ] .

Original: We also assume thatthe loss function ℓ is 0-1loss defined as ℓ 0-1 ( h, ( x, y )) :
Revised: We also assume that loss function ℓ is 0-1 loss defined as ℓ 0-1 ( h, ( x, y )) :

Original: = 1 h ( x ) ̸
Revised: = 1 h ( x ) ̸

Original: = y , where 1 is an indicator function.
Revised: = y , where 1 is an indicator function.

Original: The following theorem shows the situation where ITV works well; In other words, a situation wherethe generalization error of ˆ h c is not larger than that of  other hypotheses in H c with high probability.
Revised: The following theorem shows a situation where ITV works well: a situation where the generalization error of ˆ h c does not exceed that of the other hypotheses in H c with high probability.

Original: Theorem 5.1.
Revised: Theorem 5.1.

Original: If Y =
Revised: If Y =

Original: [ K ]  and hypothesis class H is PAC learnable with 0-1 loss ℓ 0-1 , trainingsample S , and algorithm A . Suppose ˆ h
Revised: [ K ] , and hypothesis class H is PAC-learnable with 0-1 loss ℓ 0-1 , training data S , and algorithm A , then suppose that ˆ h

Original: =
Revised: =

Original: A ( S ) be the hypothesis estimated form S satisfying
Revised: A ( S ) is a hypothesis estimated form S satisfying

Original: L D ( ˆ h ) ≤ ϵ for some parameter ϵ ∈ (0 , 1) .
Revised: L D (ˆ h ) ≤ ϵ for some parameter ϵ ∈ (0 , 1) .

Original: Then, for any requirement c ,the hypothesis ˆ h c obtainedby modifying ˆ h with a CV satisfies
Revised: Then for any requirement c , hypothesis ˆ h c obtained by modifying ˆ h with a CV satisfies

Original: [Equation]
Revised: [Equation]

Original: We give a proof in Appendix A.
Revised: We give a proof in Appendix A.

Original: The proof bounds L D (ˆ h c )by using the fact that it is close to L D ( f c ) ,where f c is obtained by modifying f : X → Y satisfying L D ( f ) = 0 .
Revised: The proof bounds L D (ˆ h c ) using the fact that it is close to L D ( f c ) , where f c is obtained by modifying f : X → Y to satisfy L D ( f ) = 0 .

Original: The theorem suggests thatif H is PAC-learnable,  inference time verification is sufficient to obtain a hypothesis with a smallgeneralization error in H c .
Revised: The theorem suggests that if H is PAC-learnable, then inference time verification is sufficient to obtain a hypothesis with small generalization error in H c .

Original: We should note that the generalization error might increase by using a verifier, and the amount ofincrease is always larger than L D ( f c ) .
Revised: Note that the generalization error might increase with a verifier, and the amount of the increase is always larger than L D ( f c ) .

Original: Therefore, L D ( f c ) can be seen as representing the discrepancybetween the  data distributionD and requirement c .
Revised: Therefore, L D ( f c ) represents the discrepancy between data distribution

Original: We say requirement c  is consistent with fif c ( x, f ( x ))
Revised: D and requirement c , which is consistent with f if c ( x, f ( x ))

Original: = 1 for all x .
Revised: = 1 for all x .

Original: If c is consistent with f , then L D ( f c ) = [-
-] and we can certify that L D (ˆ h c ) ≤ ϵ .
Revised: If c is consistent with f , then L D ( f c ) = [-
-]0 , and we can certify that L D (ˆ h c ) ≤ ϵ .

Original: The above theorem shows  ITV works when H is PAC-learnable.
Revised: The above theorem shows that ITV works when H is PAC-learnable.

Original: However, this will not hold if D isnot realizable with H , i.e, H is not PAC-learnable.
Revised: However, this will not hold if D is not realizable with H , i.e, H is not PAC-learnable.

Original: Theorem 5.2.
Revised: Theorem 5.2.

Original: If Y =
Revised: If Y =

Original: [ K ] , the loss function is 0-1 loss ℓ 0-1 and hypothesis class H is not realizablewith D ,  then there exists training sample S , algorithm A , requirements c  and ϵ ∈ (0 , 1) such thatˆ h
Revised: [ K ] , the loss function is 0-1 loss ℓ 0-1 and hypothesis class H is not realizable with D , and then there exists training data S , algorithm A , requirements c , and ϵ ∈ (0 , 1) such that ˆ h

Original: = A ( S ) satisfies L D (ˆ h ) ≤
Revised: = A ( S ) satisfies L D (ˆ h ) ≤

Original: min h ∈H L D ( h ) + ε but L D (ˆ h c ) > min h c ∈H c L D ( h c )
Revised: min h ∈H L D ( h ) + ε but L D (ˆ h c ) > min h c ∈H c L D ( h c )

Original: + ϵ .
Revised: + ϵ .

Original: We givea proof in Appendix B. The proof  shows a counterexample even if c is consistent with groundtruth f .
Revised: We give in Appendix B a proof that shows a counterexample even if c is consistent with ground truth f .

Original: The above theorems show that the realizability of H is the key factor that distinguishes thecases where ITV works well.
Revised: The above theorems show that the realizability of H is the key factor that distinguishes among the cases where ITV works well.

Original: Moreover, different from the realizable case, Theorem 5.2 holds evenif requirement c is consistent with distribution D .
Revised: Moreover, unlike the realizable case, Theorem 5.2 holds even if requirement c is consistent with distribution D .

Original: Let f D : X → Y be  the Bayes optimal predictordefined as:
Revised: Let f D : X → Y be defined as the Bayes optimal predictor:

Original: [Equation]
Revised: [Equation]

Original: The Bayes optimal predictor is optimal, in the sense that for every other classifier g : X → Y ,
Revised: The Bayes optimal predictor is optimal, in the sense that for every other classifier g : X → Y ,

Original: L D ( f D ) ≤ L D ( g ) .
Revised: L D ( f D ) ≤ L D ( g ) .

Original: Theorem 5.2 holds if c is consistent with f D .
Revised: Theorem 5.2 holds if c is consistent with f D .

Original: These results show that existingmethods [9, 2] using constraints only in the inference time might fail to select the best hypothesis.
Revised: These results show that existing methods [9, 2] using constraints only in the inference time might fail to select the best hypothesis.

Original: Running time analysis Using a CV will increase the time needed for inference.
Revised: Running time analysis: Using a CV increases the time needed for inference.

Original: Suppose thata verifier is an oracle that can answer the query of asking the value of c ( x, y ) .
Revised: Suppose that a verifier is an oracle that can answer the query about the value of c ( x, y ) .

Original: To achieve themodification procedure shown in (3), we need at most K queries.
Revised: To achieve a previously shown modification procedure (3), we need at most K queries.

Original:  Learning Time Verification
Revised:  Learning Time Verification

Original: In Section 5, we show that if H is PAC learnable with 0-1 loss, then modifying a hypothesis at aninference time is enough to obtain a hypothesis with the smallest generalization error while satisfying  requirements.
Revised: In Section 5, we show that if H is PAC-learnable with 0-1 loss, then modifying a hypothesis at the inference time is sufficient to obtain a hypothesis with the smallest generalization error while satisfying the requirements.

Original: If H is not PAC learnable, then the ITV scheme might fail to obtain a hypothesis witha small generalization error.
Revised: If H is not PAC-learnable, then the ITV scheme might fail to obtain a hypothesis with small generalization error.

Original: In this section, we show  the generalization error can be bounded whenwe use a CV in the learning phase.
Revised: Here we show that the generalization error can be bounded when we use a CV in the learning phase.

Original: We call this setting learning time verification (LTV).
Revised: We call this setting learning time verification (LTV).

Original: We can see that the LTV scheme corresponds to the learning task where the hypothesis class is H c . Therefore, we analyze the learnability of H c using the standard tools for generalization analyses.
Revised: Since the LTV scheme corresponds to a learning task where the hypothesis class is H c , we analyze the learnability of H c using the standard tools for generalization analyses.

Original: This paper provides analyses based onthe Rademacher complexity since they are widely used toolsand can give tight bounds for both data-dependent and data-independent cases.
Revised: This paper provides analyses based on Rademacher complexity since its a widely used tools that can give tight bounds for both data-dependent and data-independent cases.

Original: Moreover, someprevious work gives bounds of structured prediction tasks usingthe Rademacher complexity.
Revised: Moreover, some previous work gives bounds of structured prediction tasks using Rademacher complexity.

Original: In theliterature, constraints are actively used in structured prediction tasks, including language generationand sequence labeling.
Revised: In the literature, constraints are actively used in structured prediction tasks, including language generation and sequence labeling.

Original: Therefore, analyzing the generalization error  when using a CV on structuredprediction tasks is important.
Revised: Therefore, analyzing the generalization error is important when using a CV on structured prediction tasks.

Original: In the following, we first showthat the upper bounds of generation error based on the Rademachercomplexity of H c in a multi-class classification task (§6.1, 6.2) and a structured prediction setting(§6.3).
Revised: In the following, we first show the upper bounds of generation error based on the Rademacher complexity of H c in a multi-class classification task (§6.1, 6.2) and a structured prediction setting (§6.3).

Original: Our main finding is that the upper bounds based on the Rademacher complexity of H c isalways less than or equal to those of H .
Revised: Our main finding is that the upper bounds based on the Rademacher complexity of H c are always less than or equal to those of H .

Original: Therefore, adding a CV to a machine learning model will notdegrade its learnability.
Revised: Therefore, adding a CV to a machine learning model will not degrade its learnability.

Original: 6.1 Multi-class Classification
Revised: 6.1 Multi-class Classification

Original: We first give the Rademacher complexity-based error bounds on a multi-class classification task, i.e., Y =
Revised: We first give the Rademacher complexity-based error bounds on a multi-class classification task, i.e., Y =

Original: [ K ] .
Revised: [ K ] .

Original: In this section, we show that a standard upper bound [16] based on  Rademacher complexityof H can be used as an upper bound of H c for any requirement c .
Revised: In this section, we show that a standard upper bound [18] based on the Rademacher complexity of H can be used as an upper bound of H c for any requirement c .

Original: In the next section, wewill show astate-of-the-art error bound,which is based onthe local Rademacher complexity H ,which can alsobe used as an upper bound of H c .
Revised: In the next section, we show that a state-of-the-art error bound, based on local Rademacher complexity H , can also be used as an upper bound of H c .

Original: Following previous works, let h : ( X × Y )
Revised: Following previous works, let h : ( X × Y )

Original: → R be a scoring function, and  hypothesis class Hisdefined as a set of scoring functions.
Revised: → R be a scoring function, and define hypothesis class H as a set of scoring functions.

Original: A scoring function defines a mapping from X to Y :
Revised: A scoring function defines a mapping from X to Y :

Original: [Equation]
Revised: [Equation]

Original: argmax

Original: [Equation] h [Equation] Let ρ h ( x, y ) be the margin of function of h defined as:
Revised: Let ρ h ( x, y ) be the margin of function of h :

Original: [Equation]
Revised: [Equation]

Original: A hypothesis h misclassifies the labeled example ( x, y ) if ρ h ( x, y ) ≤ 0 .
Revised: Hypothesis h misclassifies the labeled example ( x, y ) if ρ h ( x, y ) ≤ 0 .

Original: Thus, by using a marginfunction, the 0-1 loss can be represented as ℓ 0-1 ( h, z ) = 1 ρ h ( x,y ) ≤ 0 .
Revised: Thus, by using a margin function, the 0-1 loss can be represented as ℓ 0-1 ( h, z ) = 1 ρ h ( x,y ) ≤ 0 .

Original: Since 0-1 loss is hard to handlein learning, we use margin loss ℓ ρ ( h, ( x, y ))
Revised: Since 0-1 loss is hard to handle during learning, we use margin loss ℓ ρ ( h, ( x, y ))

Original: =
Revised: =

Original: Φ ρ ( ρ h ( x, y )) , where Φ ρ ( t ) is defined as:
Revised: Φ ρ ( ρ h ( x, y )) , where Φ ρ ( t ) is defined as

Original: [Equation]
Revised: [Equation]

Original: A function f : R → R is said to be µ -Lipschitz if | f ( t ) − f ( t ′ )
Revised: Function f : R → R is said to be µ -Lipschitz if | f ( t ) − f ( t ′ )

Original: | ≤ µ | t − t ′ | for any t, t ′ ∈ R .
Revised: | ≤ µ | t − t ′ | for any t, t ′ ∈ R .

Original: Φ ρ isan 1 /ρ -Lipschitz function.
Revised: Φ ρ is an 1 /ρ -Lipschitz function.

Original: The empirical margin loss of hypothesis h is defined as:
Revised: The empirical margin loss of hypothesis h is defined as

Original: [Equation]
Revised: [Equation]

Original: As same in the case of ITV, introducing a CV to a machine learning model corresponds to modify itscorresponding hypothesis class H to hypothesis class H c that is consistent with requirement c .
Revised: Identical to the case of ITV, introducing a CV to a machine learning model corresponds to modifying its corresponding hypothesis class H to hypothesis class H c that is consistent with requirement c .

Original: If h isa score function, then we definethe consistent function h c as:
Revised: If h is a score function, then we define consistent function h c :

Original: [Equation]
Revised: [Equation]

Original: where M is a positive constant satisfying M > | max ( x
Revised: where M is a positive constant satisfying M > | max ( x

Original: ,y ) ∈
Revised: ,y ) ∈

Original: Z h ( x, y ) | .
Revised: Z h ( x, y ) | .

Original: As stated in Section 4, weassume that there exists y ∈ Y satisfying c ( x, y )
Revised: As described in Section 4, we assume that there exists y ∈ Y that satisfies c ( x, y )

Original: =
Revised: =

Original: 1 for all x ∈ X .
Revised: 1 for all x ∈ X .

Original: Therefore, we can guaranteethat ρ h c ( x, y ) < [-
-] if c ( x, y ) = 0 .
Revised: Therefore, we can guarantee that ρ h c ( x, y ) < [-
-]0 if c ( x, y ) = 0 .

Original: Followings are the main results of the general multi-class learning problem, which is based on themargin bound shown in Theorem 9.2 of [16].
Revised: The following are the main results of the general multi-class learning problem, which is based on the margin bound shown in Theorem 9.2 of Mohri et al.

Original: Our main finding is that the generalization error of anyhypothesis, h c  is bounded by the Rademacher complexity ofthe hypothesis set H , which suggeststhat if we have a tight bound for hypothesis class H , then we can expect to find a good hypothesisfrom H c under any requirements c .
Revised: Our main finding is that the generalization error of any hypothesis, h c , is bounded by the Rademacher complexity of hypothesis set H , which suggests that if we have a tight bound for hypothesis class H , then we can expect to find a good hypothesis from H c under any requirements c .

Original: Theorem 6.1.
Revised: Theorem 6.1.

Original: Let H ⊆
Revised: Let H ⊆

Original: R X×Y be a hypothesis class with Y =
Revised: R X×Y be a hypothesis class with Y =

Original: [ K ] , and  c be a requirement.
Revised: [ K ] , and let c be a requirement.

Original: Fixρ > 0 .
Revised: Fix ρ > 0 .

Original: Then, for any δ > 0 , with probability at least 1 − δ , the following bound holds for all h c ∈ H c :
Revised: Then for any δ > 0 , with probability at least 1 − δ , the following bound holds for all h c ∈ H c :

Original: [Equation]
Revised: [Equation]

Original: where Π 1 ( H ) is defined as:
Revised: where Π 1 ( H ) is defined as

Original: [Equation]
Revised: [Equation]

Original: We give a proof in Appendix C. We obtain the results by showing  upper bounds of the Rademachercomplexity of H c are bounded by some upper bounds of the Rademacher complexity of H .
Revised: We give a proof in Appendix C. We obtain the results by showing that the upper bounds of the Rademacher complexity of H c are bounded by some upper bounds of the Rademacher complexity of H .

Original: All theproofs of the theorems in this section use similar techniques.
Revised: All the proofs of the theorems in this section use similar techniques.

Original: The parameter ρ sets the margin value.
Revised: Parameter ρ sets the margin value.

Original: Following the technique shown in [16], wecan obtain a generalized bound that holds uniformly forall ρ > 0 .
Revised: Following a previously shown technique [18], we obtain a generalized bound that holds uniformly for all ρ > 0 .

Original: The above theorem suggests that using a CV at a learning phase would not worsen theerror bound for any requirement c .
Revised: The above theorem suggests that using a CV at a learning phase does not worsen the error bound for any requirement c .

Original: Intuitively, the theorem seems reasonable since requirements cimpose a restriction on H  and thus the complexity of H c is not larger than H .
Revised: Intuitively, the theorem seems reasonable since requirements c imposes a restriction on H , and thus the complexity of H c is not larger than H .

Original: However,we shouldnote that it is not so trivial since H c ⊆ H is not always true.
Revised: However, it is not so trivial since H c ⊆ H is not always true.

Original: Running time analysis We analyze the number of evaluations c ( x, y ) required for learning with a CV.
Revised: Running time analysis: We analyze the number of evaluations c ( x, y ) required for learning with a CV.

Original: Let S 1 be a sub-sequence of training example S such that c ( x i , y i )
Revised: Let S 1 be a sub-sequence of training example S such that c ( x i , y i )

Original: = 1 , and  S 0 be a sub-sequencesuch that c ( x i , y i )
Revised: = 1 , and let S 0 be a sub-sequence such that c ( x i , y i )

Original: = 0 .
Revised: = 0 .

Original: If we use a 0-1 loss function, then the empirical loss of hypothesis h c is:
Revised: If we use a 0-1 loss function, then the empirical loss of hypothesis h c is

Original: [Equation]
Revised: [Equation]

Original: since every h c misclassifies  examples in S 0 .
Revised: since every h c misclassifies the examples in S 0 .

Original: Therefore, we need at most K | S 1 | + | S 0 | queries forthe learning process.
Revised: Therefore, we need at most K | S 1 | + | S 0 | queries for the learning process.

Original: It is also true when we use a margin loss function.
Revised: This is also true when we use a margin loss function.

Original: On the other hand, theproblem of estimating the best hypothesis can be harder than the original problem depending on therequirement c .
Revised: On the other hand, the problem of estimating the best hypothesis might be more difficult than the original problem depending on requirement c .

Original: 6.2A Tighter Bound Based on Local Rademacher Complexity
Revised: 6.2 Tighter Bound Based on Local Rademacher Complexity

Original: The bound for H c we showed in the previous section is relatively simple, and tighter bounds of Hbased on the Rademacher complexity have been developed in the literature.
Revised: The bound for H c shown in the previous section is relatively simple, and tighter bounds of H based on the Rademacher complexity have been developed in the literature.

Original: In this section, we showthe state-of-the-art error bound for H based on the local Rademacher complexity can be used as abound for H c for any requirements c .
Revised: In this section, we show that the state-of-the-art error bound for H based on the local Rademacher complexity can be used as a bound for H c for any requirements c .

Original: Definition 6.2.
Revised: Definition 6.2.

Original: Let G be a family of functions from Z to R , and  S be a sample of size m. Then, forany r > 0 , the empirical local Rademacher complexity of G is defined as:
Revised: Let G be a family of functions from Z to R , and let S be training data of size m. Then for any r > 0 , the empirical local Rademacher complexity of G is defined as

Original: [Equation]
Revised: [Equation]

Original: [13] shows a tighter generalization bound for a multi-class classification problem using the local Rademacher complexity when a hypothesis class is the ℓ p norm hypothesis space with kernel κdefined as:
Revised: Li et al. [15] showed a tighter generalization bound for a multi-class classification problem using the local Rademacher complexity when the hypothesis class is a ℓ p norm hypothesis space with kernel κ , defined as

Original: [Equation]
Revised: [Equation]

Original: where h is represented as a vector valued function ( h 1 , .
Revised: where h is represented as a vector valued function ( h 1 , .

Original: . . , h K ) with h j ( x )
Revised: . . , h K ) with h j ( x )

Original: = h ( x, j ) ,
Revised: = h ( x, j ) ,

Original: ∀ j =1 , . .
Revised: ∀ j = 1 , . .

Original: , K , and κ : X × X
Revised: , K , and κ : X × X

Original: → R is a Mercer kernel withthe associated feature map ϕ , i.e., κ ( x, x ′ )
Revised: → R is a Mercer kernel with associated feature map ϕ , i.e., κ ( x, x ′ )

Original: =⟨ ϕ ( x ) , ϕ ( x ′ ) ⟩ .
Revised: = ⟨ ϕ ( x ) , ϕ ( x ′ ) ⟩ .

Original: w = ( w 1 , . . . , w K )  and
Revised: w = ( w 1 , . . . , w K ) , and

Original: ∥ w ∥ = (cid:104)(cid:80) Ki =1 ∥ w ∥ p 2 (cid:105) 1 p is the ℓ 2 ,p -norm.
Revised: ∥ w ∥ = (cid:104)(cid:80) Ki =1 ∥ w ∥ p 2 (cid:105) 1 p is the ℓ 2 ,p -norm.

Original: For any p ≥ 1 , letq be the dual exponent of p satisfying 1 /p
Revised: For any p ≥ 1 , let q be the dual exponent of p satisfying 1 /p

Original: + 1 /q
Revised: + 1 /q

Original: = 1 .
Revised: = 1 .

Original: Let Φ : R → R be a loss function satisfying:1) 1 t ≤ 0 ( t ) ≤ Φ( t ) for all t ; 2) Φ( t) is decreasing
Revised: Let Φ : R → R be a loss function satisfying the following: [Equation] ) ≤ Φ ( t

Revised: ) for all

Revised: t

Original:  andit hasa zero point c Φ ; 3) Φ is ζ -smooth, that is,| Φ ′ ( t ) − Φ ′ ( t′ )
Revised: ; 2) Φ ( t ) is decreasing and has zero point c Φ ; 3) Φ is ζ -smooth, that is, | Φ ′ ( t

Original:  | ≤ ζ | t − t ′ | .
Revised: − Φ ′ ( t | ≤ ζ | t − t ′ | .

Revised: Let H p,κ,c be the hypothesis class obtained by modifying hypothesis H p,κ to satisfy requirements c , and L c := { ( x, y )

Original: We let L c := { ( x, y ) (cid:55)→ Φ( ρ h c ( x, y )) : 
Revised: (cid:55)→ Φ( ρ h c ( x, y )) : h c ∈ H p,κ,c } . The following theorem gives a bound of the local

Original: h c∈ H p,κ,c } . 
Revised: Rademacher complexity of L c . [Figure]

Original: Theorem 6.3.
Revised: Theorem 6.3.

Original: Let H p,κ,c be the set of hypothesis obtained by modifying hypothesis h ∈ H p,κ withrequirement c .
Revised: Let H p,κ,c be the set of hypotheses obtained by modifying hypothesis h ∈ H p,κ with requirement c .

Original: For any δ > 0 , with probability at least 1 − δ , the following bound holds:
Revised: For any δ > 0 , with probability at least 1 − δ , the following bound holds:

Original: [Equation]
Revised: [Equation]

Original: where ϑ = sup x ∈X κ ( x, x ) < ∞ , d = sup t ∈ R Φ( t ) < ∞ , and C d,ϑ is a constant.
Revised: where ϑ = sup x ∈X κ ( x, x ) < ∞ , d = sup t ∈ R Φ( t ) < ∞ , and C d,ϑ is a constant.

Original: ξ ( K ) is:
Revised: ξ ( K ) is

Original: [Equation]
Revised: [Equation]

Original: otherwise .

Original: [Figure]

Original: We give a proof in Appendix D.
Revised: We give a proof in Appendix D.

Original: The bound equals that of R m
Revised: The bound equals that of R m

Original: ( L ; r ) (Theorem 1 of [13]) for anyrequirement c and any hypothesis H ρ,κ .
Revised: ( L ; r ) (Theorem 1 of [15]) for any requirement c and any hypothesis H ρ,κ .

Original: Therefore, the generalization error boundsin [13] based on 
Revised: Therefore, the generalization error bounds based on Theorem 1 of Li et al.

Original: Theorem 1 of the paper holds for any requirement c .
Revised: [15] holds for any requirement c .

Original: 6.3 Analyses of Structured Prediction
Revised: 6.3 Analyses of Structured Prediction

Original: Structured prediction is a kind of multi-class classification task, wherethe label set Y might be a setof sequences, images, graphs, trees, or other objects admitting some possibly overlapping structure.
Revised: Structured prediction is a kind of multi-class classification task, where label set Y might be a set of sequences, images, graphs, trees, or other objects admitting some possibly overlapping structure.

Original: As mentioned in Section 1, previous works try to imposesome constraints on the output of structuredprediction tasks.
Revised: As mentioned in Section 1, previous works try to impose constraints on the output of structured prediction tasks.

Original: Thus it is also useful to derive error bounds for structured prediction tasks whenwe use a CV.
Revised: Thus it is also useful to derive error bounds for structured prediction tasks when we use a CV.

Original: In the following, we show that the Rademacher complexity-based generalization errorbounds derived in the seminal work  [6] also hold if we use a CV.
Revised: In the following, we show that the Rademacher complexity-based generalization error bounds derived in a seminal work of Cortes et al. [6] also hold if we use a CV.

Original: Although tighter bounds are givenin more recent work [17, 14], we give bounds based on  [6] due to their simplicity.
Revised: Although tighter bounds are given in a more recent work [19, 16], we give bounds based on Cortes et al. [6] due to their simplicity.

Original: We give some definitions for the structured prediction task.
Revised: We give some definitions for the structured prediction task.

Original: Following previous work, we assumethat Y is decomposable along with substructures:
Revised: Following previous work, we assume that Y is decomposable along with substructures:

Original: Y =
Revised: Y =

Original: Y 1 × · · · × Y l .
Revised: Y 1 × · · · × Y l .

Original: Here, Y k is the set of possiblelabels that can be assigned to the k -th substructure.
Revised: Here Y k is a set of possible labels that can be assigned to the k -th substructure.

Original: We denote by L : Y × Y → R + a loss functionmeasuring the dissimilarity of two elements ofthe output space Y .
Revised: We denote by L : Y × Y → R + a loss function that measures the dissimilarity of two elements of output space Y .

Original: L is definite , that is, L ( y, y ′ )
Revised: L is definite , that is, L ( y, y ′ )

Original: = {+
+}
Revised: = {+
+}

Original: 0iff y = y ′ .
Revised: iff y = y ′ .

Original: A typical definite loss function for a structured prediction task is the Hamming loss definedby L ( y, y ′ )
Revised: A typical definite loss function for a structured prediction task is the Hamming loss defined by L ( y, y ′ )

Original: = 1 l (cid:80) lk =1 1 y k ̸ = y ′ k
Revised: = 1 l (cid:80) l k =1 1 y k ̸ = y ′ k

Original: for all y =
Revised: for all y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y l ) and y ′ =
Revised: , y l ) and y ′ =

Original: ( y ′ 1 , . . .
Revised: ( y ′ 1 , . . .

Original: , y ′ l ) , with y k ,
Revised: , y ′ l ) , with y k ,

Original: y ′ k ∈
Revised: y ′ k ∈

Original: Y k .
Revised: Y k .

Original: Other typical examples of loss functions can be seen in [6].
Revised: Other typical examples of loss functions can be seen in Cortes et al.

Original: Using loss function L , the generalizationerror andthe empirical error of h is defined as:
Revised: Using loss function L , the generalization and empirical error of h are defined:

Original: [Equation]
Revised: [Equation]

Original: Same with the multi-class classification task, hypothesis class H can be represented as a set ofscoring function h : X × Y → R .
Revised: As with the multi-class classification task, hypothesis class H can be represented as a set of scoring function h : X × Y → R .

Original: We use h ( x ) to represent the predictor defined by h ∈ H :h ( x ) :=
Revised: We use h ( x ) to represent the predictor defined by h ∈ H : h ( x ) :=

Original: argmax y ∈Y h ( x, y ) for all x ∈ X .
Revised: argmax y ∈Y h ( x, y ) for all x ∈ X .

Original: Following the previous work, we assume that eachscoring function can be decomposed as a sum, and such decomposition follows a factor graph .
Revised: Following the previous work, we assume that each scoring function can be decomposed as a sum, and such decomposition follows a factor graph .

Original: Afactor graph G is a tuple G =
Revised: Factor graph G is a tuple G =

Original: ( V, F, E ) , where V is a set of variable nodes, F is a set of factornodes, and E is a set of undirected edges between a variable node and a factor node.
Revised: ( V, F, E ) , where V is a set of variable nodes, F is a set of factor nodes, and E is a set of undirected edges between a variable node and a factor node.

Original: Every node in V corresponds to a substructure index, that is, V = { 1 , . . .
Revised: Every node in V corresponds to a substructure index, where V = { 1 , . . .

Original: , l } .
Revised: , l } .

Original: For any factor node f , we denote by N ( f ) ⊆
Revised: For any factor node f , we denote by N ( f ) ⊆

Original: V the set of variable nodes connected to f anddefine Y f asthe substructure set cross-product Y f
Revised: V a set of variable nodes connected to f and define Y f as substructure set cross-product Y f

Original: = (cid:81) k ∈N ( f ) Y k .
Revised: = (cid:81) k ∈N ( f ) Y k .

Original: Then, h admits the followingdecomposition as a sum of functions
Revised: Then h admits the following decomposition as a sum of functions

Original: h f , each taking as  argument a pair of ( x, y f ) ∈
Revised: h f , each taking as an argument a pair of ( x, y f ) ∈

Original: X × Y f :
Revised: X × Y f :

Original: [Equation]
Revised: [Equation]

Original: Figure 2 shows examples of decompositions based on factor graphs.
Revised: Figure 2 shows examples of decompositions based on factor graphs.

Original: We conventionally assume thatthe structure of  factor graphs may change depending on a particular example ( x i , y i ) :
Revised: We conventionally assume that the structure of the factor graphs may change depending on a particular example ( x i , y i ) :

Original: G ( x i , y i ) =
Revised: G ( x i , y i ) =

Original: G i =
Revised: G i =

Original: ([ l i ] , F i , E i ) .
Revised: ([ l i ] , F i , E i ) .

Original: A special case of this setting is whenthe size l i of each example is allowed tovary.
Revised: A special case of this setting is when size l i of each example is allowed to vary.

Original: In such a case, the number of possible labels Y is potentially infinite.
Revised: In such a case, the number of possible labels Y is potentially infinite.

Original: Followingthe multi-class classification, our CV maps hypothesis h to h c to satisfy the requirements.
Revised: Following multi-class classification, our CV maps hypothesis h to h c to satisfy the requirements.

Original: The definition of h c follows (4).
Revised: The definition of h c follows Eq.

Original: We should note that the definition does not require h c to have afactored representation.
Revised: This definition does not require h c to have a factored representation.

Original: For analyzing the complexity,  [6] introduces the empirical factor graph Rademacher complexity R GS ( H ) ofa hypothesis class H fora sample S =
Revised: For analyzing the complexity, Cortes et al. [6] introduced empirical factor graph Rademacher complexity R GS ( H ) of hypothesis class H for S =

Original: ( x 1 , . . .
Revised: ( x 1 , . . .

Original: , x m ) and factor graph G as follows:
Revised: , x m ) and factor graph G :

Original: [Equation]
Revised: [Equation]

Original: where ϵ =
Revised: where ϵ =

Original: ( ϵ i,f,y ) i
Revised: ( ϵ i,f,y ) i

Original: ∈
Revised: ∈

Original: [ m ] ,f ∈ F i ,y ∈Y f and every ϵ i,f,y is i.i.d.
Revised: [ m ] ,f ∈ F i ,y ∈Y f and every ϵ i,f,y is i.i.d.

Original:  Rademacher random variable.
Revised: a Rademacher random variable.

Original: The factor 331 graph Rademacher complexity of H fora factor graph G is defined as the expectation:
Revised: Factor graph Rademacher complexity of H for factor graph G is defined as expectation

Original: [Equation]
Revised: [Equation]

Original: By using the factor graph Rademacher complexity,  [6] gives bounds for  structured prediction taskwith the following additive and multiplicative empirical losses:
Revised: By using the factor graph Rademacher complexity, Cortes et al. [6] gives bounds for a structured prediction task with the following additive and multiplicative empirical losses:

Original: [Equation]
Revised: [Equation]

Original: where Φ ∗ ( t ) = min( B, max(0 , t )) for all t , with B = max y,y ′
Revised: where Φ ∗ ( t ) = min( B, max(0 , t )) for all t , with B = max y,y ′

Original: L ( y, y ′ ) .
Revised: L ( y, y ′ ) .

Original: As shown in [6], these lossfunctions cover typical surrogate loss function used in structured prediction tasks.
Revised: As shown in [6], these loss functions cover typical surrogate loss functions used in structured prediction tasks.

Original: We show thefollowing bound for structured prediction.
Revised: We show the following bound for structured predictions.

Original: Theorem 6.4.
Revised: Theorem 6.4.

Original: Fix ρ > 0 .
Revised: Fix ρ > 0 .

Original: For any δ > 0 and requirement c , with probability at least 1 − δ over thedraw ofa sample S of size m from distribution D , the following holds for all h c ∈ H c :
Revised: For any δ > 0 and requirement c , with probability at least 1 − δ over the draw of sample S of size m from distribution D , the following holds for all h c ∈ H c :

Original: [Equation]
Revised: [Equation]

Original: We give a proof in Appendix E. ρ is a parameter that determines the margin.
Revised: We give a proof in Appendix E. ρ is a parameter that determines the margin.

Original: Similar to the case ofthe multi-class classification, we can derive a bound that holds for any ρ > 0 following the derivationof [6].
Revised: Similar to the case of multi-class classification, we can derive a bound that holds for any ρ > 0 following a previous derivation [6].

Original: The above result indicates that the bound will not change if we use a CV for any requirementc .
Revised: The above result indicates that the bound will not change if we use a CV for any requirement c .

Original: This is interesting since the above result holds even if we do not have a factored representation ofh c ( x, y )  similar to  (5), while the derived bound depends on the factor graph Rademacher complexity,which depends on the factored representation of h ( x, y ) .
Revised: This is interesting since the above result holds even if we do not have a factored representation of h c ( x, y ) , similar to Eq. (5), although the derived bound depends on the factor graph Rademacher complexity, which depends on the factored representation of h ( x, y ) .

Original: We analyze the overhead in running time for evaluating loss function L add S,ρ ( h c ) and L mult S,ρ ( h c ) forhypothesis h c .
Revised: We analyzed the overhead of the running time for evaluating loss function L add S,ρ ( h c ) and L mult S,ρ ( h c ) for hypothesis h c .

Original: Different from the multi-class classification case, both the number of queries andthe overhead in running time for the loss evaluation when we use a CV depends onboth the modeland the type of requirements for structured prediction.
Revised: Different from the multi-class classification case, both the number of queries and the overhead of the running time for the loss evaluation when we use a CV depend on the model and the type of requirements for structured predictions.

Original: This result is consistent with the literaturereporting that forthe structured prediction task, it is known that originally tractable optimizationproblems can be intractable if we put additional constraints [21].
Revised: This result is consistent with the literature, which reports that for structured prediction tasks, original tractable optimization problems can be intractable if we put additional constraints [23].

Original:  Conclusion
Revised:  Conclusion

Original: This paper gives a generalization analysis when there are requirements that  input-output pairs ofa machine learning model must satisfy.
Revised: This paper gives a generalization analysis when there are requirements that the input-output pairs of a machine learning model must satisfy.

Original: We introduce a concurrent verifier, a simple module thatenables us to guarantee that the input-output pairs of a machine learning model satisfy  requirements.
Revised: We introduce a concurrent verifier, a simple module that enables us to guarantee that the input-output pairs of a machine learning model satisfy the requirements.

Original: We show the situation where we can obtain a hypothesis witha small error when we use a verifier onlyin an inference phase.
Revised: We show a situation where we can obtain a hypothesis with small error when we use a verifier only in the inference phase.

Original: Interestingly, if H is not PAC learnable, we might fail to obtain a guaranteedhypothesis even if  requirements are consistent with distribution D .
Revised: Interestingly, if H is not PAC-learnable, we might fail to obtain a guaranteed hypothesis even if the requirements are consistent with distribution D .

Original: We also give generalizationbounds based onthe Rademacher complexity when we use a verifier in a learning phase. We find thatthe obtained bounds are less than or equal to  existing bounds, independent of the machine learningmodel and  type of requirements.
Revised: We also give the generalization bounds based on Rademacher complexity when we use a verifier in a learning phase and find that the obtained bounds are less than or equal to the existing ones, independent of the machine learning model and the type of requirements Acknowledgements

Revised: The authors thank the anonymous reviewers for their valuable feedback, corrections and suggestions.

Revised: This work was supported by JST PRESTO (Grant Number JPMJPR20C7, Japan) and JSPS KAKENHI (Grant Number JP20H05963, Japan)

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes]
Revised: [Yes]

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.•
Revised: [No] The code and the data are proprietary.

Original:  Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] We discuss conditions a verifier does not work well in Section 5.
Revised: [Yes] We discuss conditions a verifier does not work well in Section 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] We give proof inAppendix3.
Revised: [Yes] We give proof in Appendix 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [N/A] (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
Revised: [N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A] (d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
Revised: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [N/A] 
Revised: [N/A] 4.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A] (b) Did you mention the license of the assets?
Revised: [N/A] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL? [N/A] 
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL? [N/A] (d) Did you discuss whether

Original: d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating? 
Revised: and how consent was obtained from people whose data you’re using/curating? [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 
Revised: [N/A] 5.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: information or offensive content?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Revised: A Proof of Theorem 5.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: Proof. Since D is realizable, f : X → Y exists such that L D ( f ) = 0 . We show that L D ( f c ) ≤ L D (ˆ h c ) ≤ L D ( f c ) + ϵ .

Revised: We first prove L D ( f c )

Revised: ≤ L D (ˆ h c ) .

Revised: From the definition of f c , if f ( x ) ̸ = f c ( x ) , then c ( x, f ( x ))

Revised: = 0 , and thus h c ( x ) ̸

Revised: = f ( x ) for all h c ∈ H c .

Revised: Therefore, for all h c ∈ H c ,

Revised: L D ( f c ) = P [ f c ( x ) ̸ = f ( x )] ≤ P [ h c ( x ) ̸ = f ( x )] =

Revised: L D ( h c ) .

Revised: Next we prove L D (ˆ h c )

Revised: ≤ L D ( f c ) + ϵ .

Revised: For all h ∈ H , we show that L D ( h c ) − L D ( h ) is bounded:

Revised: L D ( h c ) − L D ( h ) ≤ P [ h c ( x )

Revised: ̸ = h ( x ) and h ( x ) = f

Revised: ( x )] ≤ P [ f c ( x ) ̸ = f ( x )]

Revised: =

Revised: L D ( f c ) , where the first inequality uses the fact that the error increases if we modify the output at x and satisfy f ( x ) = h ( x ) .

Original: {+Since c ( x, f ( x )) = 
+}
Revised: {+Since c ( x, f ( x )) = 
+}

Revised: for such x , the probability is less than error L D ( f c ) .

Revised: Thus,

Revised: L D (ˆ h c ) ≤ L D (ˆ h ) + L D ( f c ) ≤ L D ( f c ) + ϵ .

Revised: B Proof of Theorem 5.

Revised: Proof.

Revised: Suppose that H = { h 0 , h 1 } , and f ̸∈ H exists, satisfying L D ( f ) = 0 .

Revised: Suppose partition

Revised: X 0 , X 1 of X exists such that h 0 ( x ) ̸ = f ( x ) iff x ∈ X 0 and h 1 ( x ) ̸ = f ( x ) iff x ∈ X 1 .

Revised: Suppose that | L D ( h 0 ) − L D ( h 1 ) | ≤ ϵ for some ϵ ∈ (0 , 1) .

Original: {+If we design c such that c ( x, y ) = 
+}
Revised: {+If we design c such that c ( x, y ) = 
+}

Revised: iff x ∈ Xand y

Revised: = h 0 ( x ) , otherwise c ( x, y )

Revised: = 1 .

Revised: Then the generalization error of modified hypothesis h c 1 becomes L D ( h c 0 )

Revised: = 0 , and L D ( h c 1 )

Revised: =

Revised: L D ( h 1 ) .

Revised: Thus if L D ( h 1 ) > ϵ , and then difference | L D ( h c 0 ) − L D ( h c 1 ) | becomes larger than ϵ .

Revised: Note that the above proof holds for c , which is consistent with f ; that is, c ( x, f ( x ))

Revised: = 1 for all x ∈ X .

Revised: C Proof of Theorem 6.

Revised: We first introduce Talagrand’s lemma with which we prove the main theorem.

Revised: Lemma C.1 (Talagrand’s lemma, [13, 18] ) .

Revised: Let Φ be the µ -Lipschitz function from R to R , and let σ 1 , . . .

Revised: , σ m be Rademacher random variables.

Revised: Then for any hypothesis set H of real-valued functions, the following inequality holds:

Revised: [Equation]

Revised: We also use the following lemma.

Revised: Lemma C.2 (Lemma 9.1 of Mohri et al.

Revised: Let F 1 , . .

Revised: . , F l be l hypothesis sets in R X , l ≥ 1 , and let G = { max { h 1 , . .

Revised: , h l } : h i ∈ F i , i ∈

Revised: [ l ] } .

Revised: Then for any training data S of size m , the empirical

Revised: Rademacher complexity of G can be upper bounded:

Revised: [Equation]

Revised: We use max { h 1 , . . .

Revised: , h l } to represent the mapping from X to R that maps x ∈ X to max { h 1 ( x ) , . . .

Revised: , h l ( x ) } .

Revised: The above inequality holds if we use min instead of max in the definition of G .

Revised: The following lemma shows the relationship between the Rademacher complexities of H and H c .

Revised: Lemma C.3.

Revised: Let H be a hypothesis set in R X×Y , and let c : X × Y → { 0 , 1 } be the requirements.

Revised: Then for any training data S =

Revised: (( x 1 , y 1 ) , . .

Revised: , ( x m , y m )) of size m , the following inequality holds:

Revised: [Equation]

Revised: Proof.

Revised: Let b 0 ( x, y ) :

Revised: X × Y → R be the following function:

Revised: [Equation]

Revised: ( x, y ) = (cid:26) M if c ( x, y )

Revised: =− M

Revised: if c ( x, y ) = 0 .

Original: {+Let B 
+}
Revised: {+Let B 
+}

Revised: = { b 0 } .

Revised: Then H c can be represented as H c = { min { h, b 0 } : h ∈ H , b 0 ∈ B 0 } .

Revised: From

Revised: Lemma C.2,

Revised: [Equation]

Revised: Since the empirical Rademacher complexity of singleton hypothesis class B 0 is zero for any S ,

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: R S ( H c ) ≤ R S ( H ) holds. Proof of Theorem 6.1. Let us define two sets of mappings, H c 0 and H c 1 :

Revised: [Equation]

Revised: where we define ρ θ,h ( x, y ) :

Revised: [Equation]

Revised: where θ > 0 is an arbitrary constant.

Revised: ρ θ,h satisfies E [ 1 ρ h ( x,y ) ≤ 0 ] ≤ E [ 1 ρ θ,h ( x,y ) ≤ 0 ] since ρ θ,h ( x, y ) ≤

Revised: ρ h ( x, y ) holds for all ( x, y ) ∈

Revised: X × Y .

Revised: Following the proof of Theorem 9.2 of

Revised: Mohri et al.

Revised: [18] with a probability of at least 1 − δ , for all h c ∈ H c :

Revised: [Equation]

Revised: Thus, to complete the proof it suffices to show R m ( H c 0 ) ≤ 2 KR m

Revised: (Π 1 ( H )) .

Revised: We can upper bound

Revised: R m ( H c 0 ) :

Revised: [Equation]

Revised: By applying lemma C.3, the first term is bounded by R m ( H ) .

Revised: Then we follow a previous proof of

Revised: Mohri et al.

Revised: [18] to see that R m ( H ) ≤

Revised: KR m

Revised: (Π 1 ( H )) .

Revised: We bound the second term:

Revised: [Equation]

Revised: We use lemma C.2 to derive the second line and lemma C.3 to derive the fourth line.

Revised: D Proof of Theorem 6.

Revised: Our proof is made by changing a part of a previous proof of theorem 1 of Li et al.

Revised: We first introduce empirical Gaussian complexity and a lemma.

Revised: Definition D.1.

Revised: (Empirical Gaussian complexity) Let G be a family of functions mapping from Z to R , and let S =

Revised: ( z 1 , . . .

Revised: , z m ) ∈

Revised: Z m be training data of size m .

Revised: Then the empirical Gaussian complexity of G with respect to S is defined:

Revised: [Equation]

Revised: where g 1 , . .

Revised: , g m are independent N (0 , 1) random variables.

Revised: We need the following lemma, which is based on lemma 4 of Lei et al.

Revised: Lemma D.2.

Revised: Let H be a hypothesis class of mappings X × Y

Revised: → R , where Y =

Revised: [ K ] .

Revised: h ∈ H is represented as vector h =

Revised: ( h 1 , . . .

Revised: , h K ) .

Revised: Let c be a requirement, and let g 1 , . .

Revised: . , g mK be N (0 , 1) random variables.

Revised: Then for any training data S = ( x 1 , . . .

Original: applicable?
Revised: , x m ) of size m , we have: [Equation] [Equation] Proof.

Revised: We make a proof by modifying the proof for lemma 4 of Lei et al.

Revised: Define two Gaussian processes indexed by H c and H :

Revised: [Equation]

Revised: For any h = ( h 1 , . . .

Revised: , h K ) and ˜ h

Revised: = (˜ h 1 , . . .

Revised: , ˜ h K ) ∈ H , the independence of g i and equalities E [ g 2 i ]

Revised: = 1 imply that

Revised: [Equation]

Revised: For any a = ( a 1 , . . . , a K ) and b = ( b 1 , . . .

Revised: , b K ) ∈ R K , it can be directly checked that

Revised: [Equation]

Revised: Using the above inequality, we have the following bounds between X h c and Y h for all h, ˜ h ∈ H :

Revised: [Equation]

Revised: Finally, we can prove the lemma using this inequality and lemma A.1 of Lei et al.

Revised: Note that the lemma holds if we substitute h j ( x i ) with h j ( x i ) + a ij , and h cj ( x i ) with h cj ( x i ) + a ij in Eq.

Revised: (6), where ( a 11 , . . .

Revised: , a mK ) are constants.

Revised: We substitute lemma 1 of Li et al.

Revised: [15] with the following lemma.

Revised: Lemma D.3.

Revised: The empirical Rademacher complexity of L c with example S of size m satisfies the following:

Revised: [Equation]

Revised: where g 1 , . .

Revised: . , g nK are independent random variables following Gaussian distribution N (0 , 1) .

Revised: Proof.

Revised: Following the proof of lemma 1 of Li et al.

Revised: [15], we have

Revised: [Equation]

Revised: Following the proof

Revised: [Equation]

Revised: the first term can be bounded:

Revised: [Equation]

Revised: By using lemma D

Revised: [Equation]

Revised: [Equation]

Revised: we can bound the second term:

Revised: [Equation]

Revised: Adding these bounds can prove the lemma.

Revised: We use the above lemma instead of lemma 1 of Li et al.

Revised: [15] for proving Theorem 1 Li et al.

Revised: [15], which gives a proof for our theorem.

Revised: E Proof of Theorem 6.

Revised: Proof.

Revised: We first prove the bound for L add S,ρ ( h ) .

Revised: Following the proof of Theorem 1 in Cortes et al.[6], we can prove that L D ( h c ) <

Revised: L add D ,ρ ( h c ) and

Revised: L add D ,ρ ( h c ) ≤ L add S

Revised: [Equation]

Revised: where H c 1 is defined:

Revised: [Equation]

Revised: We give a bound on the empirical Rademacher complexity of H c 1 .

Revised: Due to the sub-additivity of the supremum, the following holds:

Revised: [Equation]

Revised: sup h c ∈H c m (cid:88) i

Revised: = σ i max y ′ ̸

Revised: = y i (cid:18)

Revised: L ( y ′ , y i ) + h c ( x i , y ′ )

Revised: ρ (cid:19)(cid:35) +m E σ (cid:34) sup h c ∈H c m (cid:88) i =

Revised: σ i h c ( x i , y i ) ρ (cid:35) .

Revised: We first bound the first term with the Lipschitzness of h (cid:55)→ max y ′ ̸

Revised: = y (cid:16) L ( y ′ , y i )

Revised: + h c

Revised: ( x i ,y ′ )

Revised: ρ (cid:17) for any requirement c .

Revised: For any h, ˜ h ∈ H , (cid:12)(cid:12)(cid:12)(cid:12)

Revised: [Equation]

Revised: max y ̸

Revised: = y i

Revised: [Equation]

Revised: where we use the fact that | h c ( x, y ) − ˜ h c ( x, y ) | ≤ | h ( x, y ) − ˜ h ( x, y ) | for any h, ˜ h ∈ H and

Revised: ( x, y ) ∈

Revised: X ×Y since | h c ( x, y ) − ˜

Revised: h c ( x, y ) | = 0 or equals | h ( x, y )

Revised: − ˜ h

Revised: ( x, y ) | , depending on requirement c .

Revised: Following the proof of Theorem 1 in Cortes et al.

Revised: [6], we have

Revised: [Equation]

Revised: We can apply Lemma 5 of Cortes et al.

Revised: [6], which yields:

Revised: [Equation]

Revised: Similarly, for the second term, the following Lipschitz property holds:

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [N/A]
Revised: Therefore, we can also obtain bound [Equation] Taking the expectation over S of the two inequalities shows that R m ( H c 1 ) ≤ 2 √ 2 ρ R Gm ( H ) , which completes the proof of the first statement. For the second statement, we follow a proof of Cortes et al.

Original: b) Did you describe any potential participant risks, with links to Institutional Review
Revised: [6] to obtain: [Equation] δ

Original: Board (IRB) approvals, if applicable?
Revised: [Equation] m , where [Equation] We can see the following inequality holds:

Original: [N/A]
Revised: [Equation]

Original: c) Did you include the estimated hourly wage paid to participants and  the total amount
Revised: where we use the fact | h c ( x, y ) − ˜ h c ( x, y ) | ≤ | h ( x, y ) − ˜ h ( x, y ) | for any h, ˜ h ∈ H and ( x, y ) ∈ X ×Y . The reminder of the proof is identical as in the previous argument.

Original: spent on participant compensation?
Revised: Table 1 shows the notations used in the paper. [Table] hypothesis h modified to satisfy requirement c

Original: [N/A]
Revised: [Table]

Revised: set of modified hypotheses defined as H c :

Revised: = { h c : h ∈ H}

Revised: [Table]