Original: Generalization Analysis on Learning with a Concurrent Verifier
Revised: Generalization Analysis on Learning with a Concurrent Verifier

Original: Author s)

Original: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: Machine learning technologies have been used in a wide range of practical systems.
Revised: Machine learning technologies have been used in a wide range of practical systems.

Original: In practical situations, it would be natural to expect input-output pairs of a machinelearning model to satisfy some requirements.
Revised: In practical situations, it would be natural to expect input-output pairs of a machinelearning model to satisfy some requirements.

Original: However, it is hard to obtain a modelsatisfying the requirements by just learning from examples.
Revised: However, it is hard to obtain a modelsatisfying the requirements by just learning from examples.

Original: A simple solution isto add a module that checks whether the input-output pairs meet the requirementsand then modifies the model’s outputs.
Revised: A simple solution isto add a module that checks whether the input-output pairs meet the requirementsand then modifies the model’s outputs.

Original: Such a module, we call concurrent verifier(CV), can give a certification, but how the generalizability of the machine learningmodel changes by using a CV is unclear.
Revised: Such a module, we call concurrent verifier(CV), can give a certification, but how the generalizability of the machine learningmodel changes by using a CV is unclear.

Original: This paper gives a generalization analysisof learning with a CV.
Revised: This paper gives a generalization analysisof learning with a CV.

Original: We analyze how the learnability of a machine learningmodel changes when we use a CV, and show the condition where we can obtain aguaranteed hypothesis using a verifier only in the inference time.
Revised: We analyze how the learnability of a machine learningmodel changes when we use a CV, and show the condition where we can obtain aguaranteed hypothesis using a verifier only in the inference time.

Original: We also showthat typical error bounds based on the Rademacher complexity will be no largerthan that of the original model when using a CV in multi-class classification andstructured prediction settings.
Revised: We also showthat typical error bounds based on the Rademacher complexity will be no largerthan that of the original model when using a CV in multi-class classification andstructured prediction settings.

Revised: Therefore, using a verifier in a learning phase willnot hurt the generalizability of the model.

Original:  Introduction
Revised:  Introduction

Original: As machine learning technology matures, many systems exploiting machine learning models havebeen developed.
Revised: As machine learning technology matures, many systems exploiting machine learning models havebeen developed.

Original: When developing a system that uses a machine learning model, a model with merelya small prediction error would not be satisfactory due to real-field requirements.
Revised: When developing a system that uses a machine learning model, a model with merelya small prediction error would not be satisfactory due to real-field requirements.

Original: For example, anobject recognition model sensitive to slight noise would cause security issues [4, 26] or a modelwith unexpected output would increase the system’s cost for dealing with them.
Revised: For example, anobject recognition model sensitive to slight noise would cause security issues [4, 26] or a modelwith unexpected output would increase the system’s cost for dealing with them.

Original: Thus, we wantinput-output pairs of a machine learning model to satisfy some requirements .
Revised: Thus, we wantinput-output pairs of a machine learning model to satisfy some requirements .

Original: However, it is hardto obtain a model satisfying the requirements by just learning from examples.
Revised: However, it is hardto obtain a model satisfying the requirements by just learning from examples.

Original: Moreover, since thelearned models tend to be complex and the input domain tends to be quite large, it would not berealistic to certify that every input-output pair satisfies requirements.
Revised: Moreover, since thelearned models tend to be complex and the input domain tends to be quite large, it would not berealistic to certify that every input-output pair satisfies requirements.

Original: In addition, even if we find aninput-output pair that does not satisfy the requirements, it would not be easy to modify a model sincewe have to re-estimate it from training examples.
Revised: In addition, even if we find aninput-output pair that does not satisfy the requirements, it would not be easy to modify a model sincewe have to re-estimate it from training examples.

Original: This paper considers a way to obtain a machine learning model that makes its input-output pairssatisfy required properties.
Revised: This paper considers a way to obtain a machine learning model that makes its input-output pairssatisfy required properties.

Original: Here, we consider the following assumptions on the situation where amachine learning model is used.
Revised: Here, we consider the following assumptions on the situation where amachine learning model is used.

Original: First, we can certainly judge whether an input-output pair ( x, h ( x ))satisfies the requirements or not, where h : X → Y is a machine learning model or a hypothesis.
Revised: First, we can certainly judge whether an input-output pair ( x, h ( x ))satisfies the requirements or not, where h : X → Y is a machine learning model or a hypothesis.

Original: Aswe will show later, important use cases fit this setting.
Revised: Aswe will show later, important use cases fit this setting.

Original: Second, there already exists a machine learningmodel whose prediction error is small enough, but its input-output pairs are not guaranteed to satisfythe requirements.
Revised: Second, there already exists a machine learningmodel whose prediction error is small enough, but its input-output pairs are not guaranteed to satisfythe requirements.

Original: This second assumption is also reasonable since modern machine learning modelsshow high enough prediction accuracy in various tasks.
Revised: This second assumption is also reasonable since modern machine learning modelsshow high enough prediction accuracy in various tasks.

Original: Under these assumptions, a practical choicefor addressing this problem is not to change the machine learning model but adding a module thatchecks input-output pairs of machine learning model h .
Revised: Under these assumptions, a practical choicefor addressing this problem is not to change the machine learning model but adding a module that [Figure] checks input-output pairs of machine learning model h .

Original: We call the module a concurrent verifier (CV).
Revised: We call the module a concurrent verifier (CV).

Original: [Figure]

Original: Figure 1 shows the system configuration of a machine learning model with a CV.
Revised: Figure 1 shows the system configuration of a machine learning model with a CV.

Original: The verifier checkswhether the input-output pair ( x, h ( x )) satisfies the required properties.
Revised: The verifier checkswhether the input-output pair ( x, h ( x )) satisfies the required properties.

Original: If it satisfies requirements,the verifier outputs h ( x ) .
Revised: If it satisfies requirements,the verifier outputs h ( x ) .

Original: If not, then it rejects h ( x ) and modifies or asks the learning model to modifyits output.
Revised: If not, then it rejects h ( x ) and modifies or asks the learning model to modifyits output.

Original: A pair of a machine learning model and a verifier can be seen as another machine learningmodel whose input-output pairs are guaranteed to satisfy the required conditions.
Revised: A pair of a machine learning model and a verifier can be seen as another machine learningmodel whose input-output pairs are guaranteed to satisfy the required conditions.

Original: Although a model with a verifier can guarantee that its input-output pairs satisfy requirements, itseffect on prediction error is unclear.
Revised: Although a model with a verifier can guarantee that its input-output pairs satisfy requirements, itseffect on prediction error is unclear.

Original: This paper gives theoretical analyses of generalization errors ofa machine learning model with a CV.
Revised: This paper gives theoretical analyses of generalization errors ofa machine learning model with a CV.

Original: We focus on how the learnability of the original model, denotedas hypothesis class H , can change by using the verifier.
Revised: We focus on how the learnability of the original model, denotedas hypothesis class H , can change by using the verifier.

Original: First, we consider the situation where we usea CV only in the inference phase.
Revised: First, we consider the situation where we usea CV only in the inference phase.

Original: This setting corresponds to the case where required properties areunknown when we are in the training phase.
Revised: This setting corresponds to the case where required properties areunknown when we are in the training phase.

Original: We show that if the hypothesis class is PAC learnable,we can obtain a guaranteed hypothesis by using a verifier only in the inference time.
Revised: We show that if the hypothesis class is PAC learnable,we can obtain a guaranteed hypothesis by using a verifier only in the inference time.

Original: Second, we consider the situation where we know the requirements when learning the model.
Revised: Second, we consider the situation where we know the requirements when learning the model.

Original: Thissituation corresponds to viewing the learnability of hypothesis set H c , which is obtained by modifyingevery hypothesis h ∈ H to satisfy the requirements.
Revised: Thissituation corresponds to viewing the learnability of hypothesis set H c , which is obtained by modifyingevery hypothesis h ∈ H to satisfy the requirements.

Original: Hence we compare the generalization errorupper bounds of H c with those of H .
Revised: Hence we compare the generalization errorupper bounds of H c with those of H .

Original: On the multi-class classification setting, we show that existingerror bounds [13, 16] based on the Rademacher complexity of H are also bounds of the modifiedhypothesis H c for any input-output requirements.
Revised: On the multi-class classification setting, we show that existingerror bounds [13, 16] based on the Rademacher complexity of H are also bounds of the modifiedhypothesis H c for any input-output requirements.

Original: Moreover, we give similar analyses for thestructured prediction task.
Revised: Moreover, we give similar analyses for thestructured prediction task.

Original: A structured prediction is a kind of multi-class classification where theset of classes Y can be decomposed into substructures, and it is worth analyzing the task since thereare many works considering constraints in structured prediction.
Revised: A structured prediction is a kind of multi-class classification where theset of classes Y can be decomposed into substructures, and it is worth analyzing the task since thereare many works considering constraints in structured prediction.

Original: Some works give error boundsfor structured prediction tasks, which are tighter than simply applying the bound for multi-classclassification task [14, 6, 17].
Revised: Some works give error boundsfor structured prediction tasks, which are tighter than simply applying the bound for multi-classclassification task [14, 6, 17].

Original: Similar to the case of multi-class classification, we show that existing
Revised: Similar to the case of multi-class classification, we show that existing

Original: Rademacher complexity-based bounds for structured prediction of H are also the bounds for H c .
Revised: Rademacher complexity-based bounds for structured prediction of H are also the bounds for H c .

Original: Our main contributions are as follows: a) We introduce a concurrent verifier, a model-agnostic wayto guarantee machine learning models satisfy required properties.
Revised: Our main contributions are as follows: a) We introduce a concurrent verifier, a model-agnostic wayto guarantee machine learning models satisfy required properties.

Original: Although a similar mechanismwas used in some existing models, our model gives a generalization analysis that does not dependon a specific model.
Revised: Although a similar mechanismwas used in some existing models, our model gives a generalization analysis that does not dependon a specific model.

Original: b) We show that if hypothesis class H is PAC-learnable, then using a verifierat inference time can give a hypothesis with a guarantee in its generalization error.
Revised: b) We show that if hypothesis class H is PAC-learnable, then using a verifierat inference time can give a hypothesis with a guarantee in its generalization error.

Original: Interestingly, if
Revised: Interestingly, if

Original: H is not PAC learnable, we might fail to obtain a guaranteed hypothesis even if requirements areconsistent with distribution D .
Revised: H is not PAC learnable, we might fail to obtain a guaranteed hypothesis even if requirements areconsistent with distribution D .

Original: c)
Revised: c)

Original: We show that if we use a CV in a learning phase of multi-classclassification tasks, then the theoretical error bounds of H based on the Rademacher complexity willnot increase with any input-output requirements.
Revised: We show that if we use a CV in a learning phase of multi-classclassification tasks, then the theoretical error bounds of H based on the Rademacher complexity willnot increase with any input-output requirements.

Original: We also give similar results for structured predictiontasks.
Revised: We also give similar results for structured predictiontasks.

Original: 1.1 Use Cases of a Concurrent Verifier
Revised: 1.1 Use Cases of a Concurrent Verifier

Original: We show some typical use cases of CVs.
Revised: We show some typical use cases of CVs.

Original: Error-sensitive applications A typical situation where we want to use a verifier is that someprediction errors might cause severe effects, and we want to avoid them.
Revised: Error-sensitive applications A typical situation where we want to use a verifier is that someprediction errors might cause severe effects, and we want to avoid them.

Original: For example, a recommendersystem might limit the set of candidate items depending on user attributes.
Revised: For example, a recommendersystem might limit the set of candidate items depending on user attributes.

Original: Although such a rulemight degrade the prediction accuracy, practically a safer model is preferable.
Revised: Although such a rulemight degrade the prediction accuracy, practically a safer model is preferable.

Original: Controlling outputs of structured prediction Constraints are frequently used in structured predictiontasks for improving the performance or improving the controllability of outputs.
Revised: Controlling outputs of structured prediction Constraints are frequently used in structured predictiontasks for improving the performance or improving the controllability of outputs.

Original: For example, [21, 5]exploits constraints on sequence labeling tasks for reflecting background knowledge to improve theprediction results.
Revised: For example, [21, 5]exploits constraints on sequence labeling tasks for reflecting background knowledge to improve theprediction results.

Original: More recently, there are works [9, 2] exploiting constraints in language generationtasks, including image captioning and machine translation, where they restrict a model to output a sentence that includes given keywords.
Revised: More recently, there are works [9, 2] exploiting constraints in language generation tasks, including image captioning and machine translation, where they restrict a model to outputa sentence that includes given keywords.

Original: Since the constraints used in this previous work can bewritten as a logical formula, our CV model can represent them as requirements.
Revised: Since the constraints used in this previous work can bewritten as a logical formula, our CV model can represent them as requirements.

Original: Robustness against input perturbations If a machine learning model changes its output if we modifyits input x to x ′ that is very close to x , then the model is called sensitive against a small change [25].
Revised: Robustness against input perturbations If a machine learning model changes its output if we modifyits input x to x ′ that is very close to x , then the model is called sensitive against a small change [25].

Original: It might be a security risk if a model is sensitive since its behavior is unpredictable.
Revised: It might be a security risk if a model is sensitive since its behavior is unpredictable.

Original: Therefore, somemethods evaluate and verify the robustness of neural networks against small perturbations [26, 4].
Revised: Therefore, somemethods evaluate and verify the robustness of neural networks against small perturbations [26, 4].

Original: Existing verification method checks a machine learning model’s robustness around input x bysearching whether there exists x ′ that is close to x and the model f gives different outputs, i.e.,h ( x ) ̸ = h ( x ′ ) , for verification samples x 1 , . . .
Revised: Existing verification method checks a machine learning model’s robustness around input x bysearching whether there exists x ′ that is close to x and the model f gives different outputs, i.e.,h ( x ) ̸ = h ( x ′ ) , for verification samples x 1 , . . .

Original: , x n .
Revised: , x n .

Original: However, although these verification methodscan test a model, they do not directly show a way to obtain a robust model.
Revised: However, although these verification methodscan test a model, they do not directly show a way to obtain a robust model.

Original: A CV can be used to fix a model to achieve robustness around samples x 1 , . .
Revised: A CV can be used to fix a model to achieve robustness around samples x 1 , . .

Original: , x n by setting a ruleof the form: “ h ( x ′ ) must be equal to h ( x i ) if x ′ is close to x i ”.
Revised: , x n by setting a ruleof the form: “ h ( x ′ ) must be equal to h ( x i ) if x ′ is close to x i ”.

Original: Although this solution might notguarantee robustness where samples are scarce, adding enough non-labeled verification samples canbe a reasonable choice.
Revised: Although this solution might notguarantee robustness where samples are scarce, adding enough non-labeled verification samples canbe a reasonable choice.

Original:  Related work
Revised:  Related work

Original: Machine learning models that can exploit constraints have been investigated in many researchfields, including statistical symbolic learning and structured prediction.
Revised: Machine learning models that can exploit constraints have been investigated in many researchfields, including statistical symbolic learning and structured prediction.

Original: For example, Markov logicnetworks [20], Problogs [8], and probabilistic circuit models [11] integrate statistical models withsymbolic logic formulations.
Revised: For example, Markov logicnetworks [20], Problogs [8], and probabilistic circuit models [11] integrate statistical models withsymbolic logic formulations.

Original: Since these models can incorporate hard constraints represented bysymbolic logic, they can guarantee input-output pairs.
Revised: Since these models can incorporate hard constraints represented bysymbolic logic, they can guarantee input-output pairs.

Original: However, previous research focused on itspractical performance and gave little theoretical analysis of its learnability when hard constraints areused.
Revised: However, previous research focused on itspractical performance and gave little theoretical analysis of its learnability when hard constraints areused.

Original: Moreover, previous works integrated the ability to exploit constraints into specific models.
Revised: Moreover, previous works integrated the ability to exploit constraints into specific models.

Original: Incontrast, our CV is model-agnostic and can be used in combination with a wide range of machinelearning models.
Revised: Incontrast, our CV is model-agnostic and can be used in combination with a wide range of machinelearning models.

Original: Recently, verification of machine learning models has gathered attention.
Revised: Recently, verification of machine learning models has gathered attention.

Original: There are attempts to checkwhether a machine learning model has the desired property[4, 26, 10, 24].
Revised: There are attempts to checkwhether a machine learning model has the desired property[4, 26, 10, 24].

Original: There are exact verificationmethods using integer programming (MIP) [26], constraint satisfaction (SAT)
Revised: There are exact verificationmethods using integer programming (MIP) [26], constraint satisfaction (SAT)

Original: [18], and satisfiablemodule theory (SMT) solver [10] to assess the robustness of a neural network model against inputnoise.
Revised: [18], and satisfiablemodule theory (SMT) solver [10] to assess the robustness of a neural network model against inputnoise.

Original: These approaches aim at obtaining models that fulfill required properties.
Revised: These approaches aim at obtaining models that fulfill required properties.

Original: However, verificationmethods cannot help modify models if the model does not satisfy the requirements.
Revised: However, verificationmethods cannot help modify models if the model does not satisfy the requirements.

Original: If we want MLmodels to meet requirements, post-processing would be needed as our concurrent verification model.
Revised: If we want MLmodels to meet requirements, post-processing would be needed as our concurrent verification model.

Original: There are methods that can give upper bounds on generalization error, including VC-dimension [27]and its extensions [7, 19], Rademacher complexity [3, 12], stability [23], and PAC-Bayes [15, 1].
Revised: There are methods that can give upper bounds on generalization error, including VC-dimension [27]and its extensions [7, 19], Rademacher complexity [3, 12], stability [23], and PAC-Bayes [15, 1].

Original: Weuse Rademacher complexity in the following analysis since they are among the most popular toolsfor giving theoretical upper bounds on the generalization error.
Revised: Weuse Rademacher complexity in the following analysis since they are among the most popular toolsfor giving theoretical upper bounds on the generalization error.

Original: Moreover, Rademacher complexityhas some extensions, including local Rademacher complexity [13] and factor graph Rademachercomplexity [6].
Revised: Moreover, Rademacher complexityhas some extensions, including local Rademacher complexity [13] and factor graph Rademachercomplexity [6].

Original: We can provide theoretical guarantees on these extended measures.
Revised: We can provide theoretical guarantees on these extended measures.

Original:  Preliminaries
Revised:  Preliminaries

Original: Our notation follows [22].
Revised: Our notation follows [22].

Original: We first introduce the notations used in the following sections.
Revised: We first introduce the notations used in the following sections.

Original: Let Xdenote the domain of inputs, Y be the domain of label, and Z be the domain of examples definedas
Revised: Let Xdenote the domain of inputs, Y be the domain of label, and Z be the domain of examples definedas

Original: Z := X
Revised: Z := X

Original: × Y .
Revised: × Y .

Original: Let H be a hypothesis class, and ℓ : H ×
Revised: Let H be a hypothesis class, and ℓ : H ×

Original: Z
Revised: Z

Original: → R + be loss function.
Revised: → R + be loss function.

Original: A sample
Revised: A sample

Original: S =
Revised: S =

Original: ( z 1 , . . .
Revised: ( z 1 , . . .

Original: , z m ) ∈
Revised: , z m ) ∈

Original: Z m is a finite sequence of size m drawn i.i.d.
Revised: Z m is a finite sequence of size m drawn i.i.d.

Original: from a fixed but unknownprobability distribution D on Z .
Revised: from a fixed but unknownprobability distribution D on Z .

Original: A learning algorithm
Revised: A learning algorithm

Original: A maps a sample to a hypothesis h .
Revised: A maps a sample to a hypothesis h .

Original: We usethe notation
Revised: We usethe notation

Original: A ( S ) to denote the hypothesis that the learning algorithm
Revised: A ( S ) to denote the hypothesis that the learning algorithm

Original: A returns upon receiving S .
Revised: A returns upon receiving S .

Original: We represent set { 1 , . .
Revised: We represent set { 1 , . .

Original: , K } as [ K ] .
Revised: , K } as [ K ] .

Original: Given distribution D on Z , we denote by L D ( h ) the generalization error and by L S ( h ) the empiricalerror of h over S defined by:
Revised: Given distribution D on Z , we denote by L D ( h ) the generalization error and by L S ( h ) the empiricalerror of h over S defined by:

Original: [Equation]
Revised: [Equation]

Original: 129 PAC learnability We introduce PAC learnability and agnostic PAC learnability as follows.
Revised: 130 PAC learnability We introduce PAC learnability and agnostic PAC learnability as follows.

Original: Definition 3.1.
Revised: Definition 3.1.

Original: (Agnostic PAC learnability)
Revised: (Agnostic PAC learnability)

Original: Hypothesis class H is agnostic PAC learnable if thereexists a function m H : (0 , 1) 

Revised: Hypothesis class H is agnostic PAC learnable if thereexists a function m H : (0 , 1)

Original: → N and a learning algorithm A with the following property: For everyϵ, δ ∈ (0 , 1) and distribution D over Z , if S is a sample consists of m ≥ m H ( ϵ, δ ) i.i.d.
Revised: → N and a learning algorithm A with the following property: For everyϵ, δ ∈ (0 , 1) and distribution D over Z , if S is a sample consists of m ≥ m H ( ϵ, δ ) i.i.d.

Original: examplesgenerated by D , then, with at least probability 1 − δ , the following holds:
Revised: examplesgenerated by D , then, with at least probability 1 − δ , the following holds:

Original: [Equation]
Revised: [Equation]

Original: A distribution D is realizable by hypothesis set H if there exists h ∗ ∈ H such that L D ( h ∗ )
Revised: A distribution D is realizable by hypothesis set H if there exists h ∗ ∈ H such that L D ( h ∗ )

Original: = 0 .
Revised: = 0 .

Original: If Dis realizable by agnostic PAC learnable hypothesis H , then we say H is PAC learnable .
Revised: If Dis realizable by agnostic PAC learnable hypothesis H , then we say H is PAC learnable .

Original: If H is PAClearnable, then (2) becomes: L D ( A ( S ))
Revised: If H is PAClearnable, then (2) becomes: L D ( A ( S ))

Original: ≤ ϵ since min h ′ ∈H L D ( h ′ )
Revised: ≤ ϵ since min h ′ ∈H L D ( h ′ )

Original: = 0 .
Revised: = 0 .

Original: Rademacher complexity
Revised: Rademacher complexity

Original: In the following sections, we use Rademacher complexity for derivinggeneralization bounds.
Revised: In the following sections, we use Rademacher complexity for derivinggeneralization bounds.

Original: Given loss function ℓ ( h, z ) and hypothesis class H , let us denote G as
Revised: Given loss function ℓ ( h, z ) and hypothesis class H , let us denote G as

Original: [Equation]
Revised: [Equation]

Original: Definition 3.2.
Revised: Definition 3.2.

Original: (Empirical Rademacher complexity) Let G be a family of functions mapping from
Revised: (Empirical Rademacher complexity) Let G be a family of functions mapping from

Original: Z to R and S =
Revised: Z to R and S =

Original: ( z 1 , . . .
Revised: ( z 1 , . . .

Original: , z m ) ∈
Revised: , z m ) ∈

Original: Z m be a sample of size m .
Revised: Z m be a sample of size m .

Original: Then, the empirical Rademachercomplexity of G with respect to sample S is defined as:
Revised: Then, the empirical Rademachercomplexity of G with respect to sample S is defined as:

Original: [Equation]
Revised: [Equation]

Original: where σ = ( σ 1 , . .
Revised: where σ = ( σ 1 , . .

Original: , σ m ) ∈ {± 1 } m are random variables distributed i.i.d.
Revised: , σ m ) ∈ {± 1 } m are random variables distributed i.i.d.

Original: according to P [ σ i =1] = P [ σ i = − 1] = 1 / 2 .
Revised: according to P [ σ i =1] = P [ σ i = − 1] = 1 / 2 .

Original: The Rademacher complexity of G is defined as the expected value of theempirical Rademacher complexity, defined as:
Revised: The Rademacher complexity of G is defined as the expected value of theempirical Rademacher complexity, defined as:

Original: [Equation]
Revised: [Equation]

Original:  Concurrent Verifier
Revised:  Concurrent Verifier

Original: We give a formal definition of a CV.
Revised: We give a formal definition of a CV.

Original: A CV works with a machine learning model, which is a functionh :
Revised: A CV works with a machine learning model, which is a functionh :

Original: X → Y .
Revised: X → Y .

Original: If x is given to the model and the model outputs h ( x ) , then the verifier checks whether( x, h ( x )) satisfies the required property.
Revised: If x is given to the model and the model outputs h ( x ) , then the verifier checks whether( x, h ( x )) satisfies the required property.

Original: We assume that the required property can be representedas requirement function c : ( X × Y ) → { 0 , 1 } .
Revised: We assume that the required property can be representedas requirement function c : ( X × Y ) → { 0 , 1 } .

Original: If c ( x, h ( x ))
Revised: If c ( x, h ( x ))

Original: =
Revised: =

Original: 1 then the pair satisfies the propertyand if c ( x, h ( x ))
Revised: 1 then the pair satisfies the propertyand if c ( x, h ( x ))

Original: = 0 then does not satisfy.
Revised: = 0 then does not satisfy.

Original: Requirement function c can be represented by a set ofdeterministic rules.
Revised: Requirement function c can be represented by a set ofdeterministic rules.

Original: For example, if X = R and Y = { 0 , 1 } , then requirements can be in the form:“if x > 0 , then y ̸ = 0 ”.
Revised: For example, if X = R and Y = { 0 , 1 } , then requirements can be in the form:“if x > 0 , then y ̸ = 0 ”.

Original: We assume that for all possible input x ∈ X , there exists y ∈ Y such thatc ( x, y )
Revised: We assume that for all possible input x ∈ X , there exists y ∈ Y such thatc ( x, y )

Original: = 1 for avoiding the situation that the requirements are unsatisfiable for any output y .
Revised: = 1 for avoiding the situation that the requirements are unsatisfiable for any output y .

Original: It is notdifficult to relax this assumption if we allow a machine learning model to reject unsatisfiable input x .
Revised: It is notdifficult to relax this assumption if we all a machine learning model to reject unsatisfiable input x .

Original: After checking the input-output pair, a verifier modifies output h ( x ) depending on the value ofc ( x, h ( x )) .
Revised: After checking the input-output pair, a verifier modifies output h ( x ) depending on the value ofc ( x, h ( x )) .

Original: If c ( x, h ( x ))
Revised: If c ( x, h ( x ))

Original: = 1 , the verifier outputs h ( x ) since it satisfies requirements.
Revised: = 1 , the verifier outputs h ( x ) since it satisfies requirements.

Original: If c ( x, h ( x ))
Revised: If c ( x, h ( x ))

Original: =0 , then the verifier will modify h ( x ) to some y ∈ Y satisfying c ( x, y )
Revised: =0 , then the verifier will modify h ( x ) to some y ∈ Y satisfying c ( x, y )

Original: = 1 .
Revised: = 1 .

Original: Therefore, if we use averifier with a machine learning model corresponding to h , then the combination of the model andthe verifier can be seen as a function h c :
Revised: Therefore, if we use averifier with a machine learning model corresponding to h , then the combination of the model andthe verifier can be seen as a function h c :

Original: X → Y defined as:
Revised: X → Y defined as:

Original: [Equation]
Revised: [Equation]

Original: (3) where y c ∈ Y satisfies c ( x, y c )
Revised: where y c ∈ Y satisfies c ( x, y c )

Original: = 

Revised: =

Original: and is selected deterministically.
Revised: and is selected deterministically.

Original: When Y =
Revised: When Y =

Original: [ K ] , an examplefor selecting minimum i ∈
Revised: [ K ] , an examplefor selecting minimum i ∈

Original: [ K ] satisfying c ( x, i )
Revised: [ K ] satisfying c ( x, i )

Original: = 1 as y c would be a reasonable choice.
Revised: = 1 as y c would be a reasonable choice.

Original: When
Revised: When

Original: Y =
Revised: Y =

Original: [ K ] and h ( x ) is made by scoring functions
Revised: [ K ] and h ( x ) is made by scoring functions

Original: h ( x, y ) : ( X × Y ) → R , it would be also reasonableto select y ∗ such that y ∗ =
Revised: h ( x, y ) : ( X × Y ) → R , it would be also reasonableto select y ∗ such that y ∗ =

Original: argmax y ∈Y ,c ( x,y )
Revised: argmax y ∈Y ,c ( x,y )

Original: =1 h ( x, y ) .
Revised: =1 h ( x, y ) .

Original: Learning a model corresponds to selectinga hypothesis h from hypothesis class H .
Revised: Learning a model corresponds to selectinga hypothesis h from hypothesis class H .

Original: Therefore, learning a model with CV corresponds tochoosing a hypothesis from the modified hypothesis class: H c = { h c : h ∈ H} .
Revised: Therefore, learning a model with CV corresponds tochoosing a hypothesis from the modified hypothesis class: H c = { h c : h ∈ H} .

Original: By definition,every hypothesis in H c satisfies the requirements, and thus we can guarantee the model satisfiesthe condition if we select a hypothesis from H c .
Revised: By definition,every hypothesis in H c satisfies the requirements, and thus we can guarantee the model satisfiesthe condition if we select a hypothesis from H c .

Original: In the following sections, we will analyze thelearnability of H c by comparing it with that of H .
Revised: In the following sections, we will analyze thelearnability of H c by comparing it with that of H .

Original:  Inference Time Verification
Revised:  Inference Time Verification

Original: We first analyze the change of generalization errors when we use a verifier only in an inference phase.
Revised: We first analyze the change of generalization errors when we use a verifier only in an inference phase.

Original: In other words, requirements are unknown in the learning phase and we estimate hypothesis ˆ h
Revised: In other words, requirements are unknown in the learning phase and we estimate hypothesis ˆ h

Original: = A ( S )from hypothesis class H by using training sample S and algorithm A .
Revised: = A ( S )from hypothesis class H by using training sample S and algorithm A .

Original: In the inference phase, we usea CV to modify ˆ h to ˆ h c .
Revised: In the inference phase, we usea CV to modify ˆ h to ˆ h c .

Original: We call this setting inference time verification (ITV).
Revised: We call this setting inference time verification (ITV).

Original: This class of situationscontains many exciting settings.
Revised: This class of situationscontains many exciting settings.

Original: Typical settings are 1) pre-trained machine learning models used ina wide range of applications, and 2) models that are hard to replace, which might encounter differentrequirements from ones at the learning time in the long run.
Revised: Typical settings are 1) pre-trained machine learning models used ina wide range of applications, and 2) models that are hard to replace, which might encounter differentrequirements from ones at the learning time in the long run.

Original: In this section, we will give analyses on a multi-class classification setting.
Revised: In this section, we will give analyses on a multi-class classification setting.

Original: We set Y =
Revised: We set Y =

Original: [ K ] andhypothesis class H is a set of mappings h : X → [ K ] .
Revised: [ K ] andhypothesis class H is a set of mappings h : X → [ K ] .

Original: We also assume that the loss function ℓ is 0-1loss defined as ℓ 0-1 ( h, ( x, y )) :
Revised: We also assume that the loss function ℓ is 0-1loss defined as ℓ 0-1 ( h, ( x, y )) :

Original: = 1 h ( x ) ̸
Revised: = 1 h ( x ) ̸

Original: = y , where 1 is an indicator function.
Revised: = y , where 1 is an indicator function.

Original: The following theorem shows the situation where ITV works well; In other words, a situation wherethe generalization error of ˆ h c is not larger than that of other hypotheses in H c with high probability.
Revised: The following theorem shows the situation where ITV works well; In other words, a situation wherethe generalization error of ˆ h c is not larger than that of other hypotheses in H c with high probability.

Original: Theorem 5.1.
Revised: Theorem 5.1.

Original: If Y =
Revised: If Y =

Original: [ K ] and hypothesis class H is PAC learnable with 0-1 loss ℓ 0-1 , trainingsample S , and algorithm A .
Revised: [ K ] and hypothesis class H is PAC learnable with 0-1 loss ℓ 0-1 , trainingexample S , and algorithm A .

Original: Suppose ˆ h
Revised: Suppose ˆ h

Original: =
Revised: =

Original: A ( S ) be the hypothesis estimated form S satisfying
Revised: A ( S ) be the hypothesis estimated form S satisfying

Original: L D ( ˆ h ) ≤ ϵ for some parameter ϵ ∈ (0 , 1) .
Revised: L D (ˆ h ) ≤ ϵ for some parameter ϵ ∈ (0 , 1) .

Original: Then, for any requirement c , the hypothesis ˆ h c obtainedby modifying ˆ h with a CV satisfies
Revised: Then, for any requirement c , the hypothesis ˆ h c obtainedby modifying ˆ h with a CV satisfies

Original: [Equation]
Revised: [Equation]

Original: We give a proof in Appendix A.
Revised: We give a proof in Appendix A.

Original: The proof bounds L D (ˆ h c ) by using the fact that it is close to L D ( f c ) ,where f c is obtained by modifying f : X → Y satisfying L D ( f ) = 0 .

Original: The theorem suggests thatif H is PAC-learnable, inference time verification is sufficient to obtain a hypothesis with a smallgeneralization error in H c .
Revised: The theorem suggests that if H is PAC-learnable, inference timeverification is sufficient to obtain a hypothesis with a small generalization error in H c .

Original: We should note that the generalization error might increase by using a verifier, and the amount ofincrease is always larger than L D ( f c ) .
Revised: We should notethat the generalization error might increase by using a verifier, and the amount of increase is alwayslarger than L D ( f c ) .

Original: Therefore, L D ( f c ) can be seen as representing the discrepancybetween the data distribution D and requirement c .
Revised: Therefore, L D ( f c ) can be seen as representing the discrepancy between the datadistribution D and requirement c .

Original: We say requirement c is consistent with fif c ( x, f ( x ))
Revised: We say requirement c is consistent with f if c ( x, f ( x ))

Original: = 1 for all x .
Revised: = 1 for allx .

Original: If c is consistent with f , then L D ( f c ) = 

Revised: If c is consistent with f , then L D ( f c ) =

Original: and we can certify that L D (ˆ h c ) ≤ ϵ .
Revised: and we can certify that L D (ˆ h c ) ≤ ϵ .

Original: The above theorem shows ITV works when H is PAC-learnable.
Revised: The above theorem shows ITV works when H is PAC-learnable.

Original: However, this will not hold if D isnot realizable with H , i.e, H is not PAC-learnable.
Revised: However, this will not hold if D isnot realizable with H .

Original: Theorem 5.2.
Revised: Theorem 5.2.

Original: If Y = [ K ] , the loss function is 0-1 loss ℓ 0-1 and hypothesis class H is not realizablewith D , then there exists training sample S , algorithm A , requirements c and ϵ ∈ (0 , 1) 
Revised: Suppose D is not realizable with hypothesis class H . Then, there exists D , trainingexample S , algorithm A , requirements c and ϵ ∈ (0 , 1) such that ˆ h

Original: such thatˆ h
Revised: =

Original: = A ( S ) satisfies L D (ˆ h ) ≤
Revised: A ( S ) satisfies L D (ˆ h ) ≤min h ∈H L D ( h )

Original: min h ∈H L D ( h ) + ε but L D (ˆ h c ) > min h c ∈H c L D ( h c ) + ϵ .
Revised: + ε but L D (ˆ h c ) > min h c ∈H c L D ( h c ) + ϵ .

Original: We give a proof in Appendix B.
Revised: We give a proof in Appendix B.

Original: The proof shows a counterexample even if c is consistent with groundtruth f .

Original: The above theorems show that the realizability of H is the key factor that distinguishes thecases where ITV works well.
Revised: The above theorems show that the realizability of H is the key factorthat distinguishes the cases where ITV works well.

Original: Moreover, different from the realizable case, Theorem 5.2 holds evenif requirement c is consistent with distribution D .
Revised: Moreover, different from the realizable case, Theorem 5.2 holds even if requirement c is consistent with distribution D .

Original: Let f D : X → Y be the Bayes optimal predictordefined as:
Revised: Let f D : X → Y be the Bayes optimal predictor defined as:

Original: [Equation]
Revised: [Equation]

Original: The Bayes optimal predictor is optimal, in the sense that for every other classifier g : X → Y ,
Revised: The Bayes optimal predictor is optimal, in the sense that for every other classifier g : X → Y ,

Original: L D ( f D ) ≤ L D ( g ) .
Revised: L D ( f D ) ≤ L D ( g ) .

Original: Theorem 5.2 holds if c is consistent with f D .
Revised: Theorem 5.2 holds if c is consistent with f D .

Original: These results show that existingmethods [9, 2] using constraints only in the inference time might fail to select the best hypothesis.
Revised: These results show that existingmethods [9, 2] using constraints only in the inference time might fail to select the best hypothesis.

Original: Running time analysis Using a CV will increase the time needed for inference.
Revised: Running time analysis Using a CV will increase the time needed for inference.

Original: Suppose thata verifier is an oracle that can answer the query of asking the value of c ( x, y ) .
Revised: Suppose thata verifier is an oracle that can answer the query of asking the value of c ( x, y ) .

Original: To achieve themodification procedure shown in (3), we need at most K queries.
Revised: To achieve themodification procedure shown in (3), we need at most K queries.

Original:  Learning Time Verification
Revised:  Learning Time Verification

Original: In Section 5, we show that if H is PAC learnable with 0-1 loss, then modifying a hypothesis at aninference time is enough to obtain a hypothesis with the smallest generalization error while satisfying requirements.
Revised: In Section 5, we show that if H is PAC learnable with 0-1 loss, then modifying a hypothesis at aninference time is enough to obtain a hypothesis with the smallest generalization error while satisfyingrequirements.

Original: If H is not PAC learnable, then the ITV scheme might fail to obtain a hypothesis witha small generalization error.
Revised: If H is not PAC learnable, then the ITV scheme might fail to obtain a hypothesis witha small generalization error.

Original: In this section, we show the generalization error can be bounded whenwe use a CV in the learning phase.
Revised: In this section, we show the generalization error can be bounded whenwe use a CV in the learning phase.

Original: We call this setting learning time verification (LTV).
Revised: We call this setting learning time verification (LTV).

Original: We can see that the LTV scheme corresponds to the learning task where the hypothesis class is H c .
Revised: We can see that the LTV scheme corresponds to the learning task where the hypothesis class is H c .

Original: Therefore, we analyze the learnability of H c using the standard tools for generalization analyses.
Revised: Therefore, we analyze the learnability of H c using the standard tools for generalization analyses.

Original: This paper provides analyses based on the Rademacher complexity since they are widely used toolsand can give tight bounds for both data-dependent and data-independent cases.
Revised: This paper provides analyses based on the Rademacher complexity since they are widely used toolsand can give tight bounds for both data-dependent and data-independent cases.

Original: Moreover, someprevious work gives bounds of structured prediction tasks using the Rademacher complexity.
Revised: Moreover, someprevious work gives bounds of structured prediction tasks using the Rademacher complexity.

Original: In theliterature, constraints are actively used in structured prediction tasks, including language generationand sequence labeling.
Revised: In theliterature, constraints are actively used in structured prediction tasks, including language generationand sequence labeling.

Original: Therefore, analyzing the generalization error when using a CV on structuredprediction tasks is important.
Revised: Therefore, analyzing the generalization error when using a CV on structuredprediction tasks is important.

Original: In the following, we first show that the upper bounds of generation error based on the Rademachercomplexity of H c in a multi-class classification task (§6.1, 6.2) and a structured prediction setting(§6.3).
Revised: In the following, we first show that the upper bounds of generation error based on the Rademachercomplexity of H c in a multi-class classification task (§6.1, 6.2) and a structured prediction setting(§6.3).

Original: Our main finding is that the upper bounds based on the Rademacher complexity of H c isalways less than or equal to those of H .
Revised: Our main finding is that the upper bounds based on the Rademacher complexity of H c isalways less than or equal to those of H .

Original: Therefore, adding a CV to a machine learning model will notdegrade its learnability.
Revised: Therefore, adding a CV to a machine learning model will notdegrade its learnability.

Original: 6.1 Multi-class Classification
Revised: 6.1 Multi-class Classification

Original: We first give the Rademacher complexity-based error bounds on a multi-class classification task, i.e.,
Revised: We first give the Rademacher complexity-based error bounds on a multi-class classification task, i.e.,

Original: Y =
Revised: Y =

Original: [ K ] .
Revised: [ K ] .

Original: In this section, we show that a standard upper bound [16] based on Rademacher complexityof H can be used as an upper bound of H c for any requirement c .
Revised: In this section, we show that a standard upper bound [16] based on Rademacher complexityof H can be used as an upper bound of H c for any requirement c .

Original: In the next section, we will show astate-of-the-art error bound, which is based on the local Rademacher complexity H , which can alsobe used as an upper bound of H c .
Revised: In the next section, we will show astate-of-the-art error bound, which is based on the local Rademacher complexity H , which can alsobe used as an upper bound of H c .

Original: Following previous works, let h : ( X × Y )
Revised: Following previous works, let h : ( X × Y )

Original: → R be a scoring function, and hypothesis class H isdefined as a set of scoring functions.
Revised: → R be a scoring function, and hypothesis class H isdefined as a set of scoring functions.

Original: A scoring function defines a mapping from X to Y :
Revised: A scoring function defines a mapping from X to Y :

Original: [Equation]
Revised: [Equation]

Original: argmax

Original: [Equation] h [Equation] Let ρ h ( x, y ) be the margin of function of h defined as:
Revised: Let ρ h ( x, y ) be the margin of function of h defined as:

Original: [Equation]
Revised: [Equation]

Original: A hypothesis h misclassifies the labeled example ( x, y ) if ρ h ( x, y ) ≤ 0 .
Revised: A hypothesis h misclassifies the labeled example ( x, y ) if ρ h ( x, y ) ≤ 0 .

Original: Thus, by using a marginfunction, the 0-1 loss can be represented as ℓ 0-1 ( h, z ) = 1 ρ h ( x,y ) ≤ 0 .
Revised: Thus, by using a marginfunction, the 0-1 loss can be represented as ℓ 0-1 ( h, z ) = 1 ρ h ( x,y ) ≤ 0 .

Original: Since 0-1 loss is hard to handlein learning, we use margin loss ℓ ρ ( h, ( x, y ))
Revised: Since 0-1 loss is hard to handlein learning, we use margin loss ℓ ρ ( h, ( x, y ))

Original: =
Revised: =

Original: Φ ρ ( ρ h ( x, y )) , where Φ ρ ( t ) is defined as:
Revised: Φ ρ ( ρ h ( x, y )) , where Φ ρ ( t ) is defined as:

Original: [Equation]
Revised: [Equation]

Original: A function f : R → R is said to be µ -Lipschitz if | f ( t ) − f ( t ′ )
Revised: A function f : R → R is said to be µ -Lipschitz if | f ( t ) − f ( t ′ )

Original: | ≤ µ | t − t ′ | for any t, t ′ ∈ R .
Revised: | ≤ µ | t − t ′ | for any t, t ′ ∈ R .

Original: Φ ρ isan 1 /ρ -Lipschitz function.
Revised: Φ ρ isan 1 /ρ -Lipschitz function.

Original: The empirical margin loss of hypothesis h is defined as:
Revised: The empirical margin loss of hypothesis h is defined as:

Original: [Equation]
Revised: [Equation]

Original: As same in the case of ITV, introducing a CV to a machine learning model corresponds to modify itscorresponding hypothesis class H to hypothesis class H c that is consistent with requirement c .
Revised: As same in the case of ITV, introducing a CV to a machine learning model corresponds to modify itscorresponding hypothesis class H to hypothesis class H c that is consistent with requirement c .

Original: If h isa score function, then we define the consistent function h c as:
Revised: If h isa score function, then we define the consistent function h c as:

Original: [Equation]
Revised: [Equation]

Original: where M is a positive constant satisfying M > | max ( x
Revised: where M is a positive constant satisfying M > | max ( x

Original: ,y ) ∈
Revised: ,y ) ∈

Original: Z h ( x, y ) | .
Revised: Z h ( x, y ) | .

Original: As stated in Section 4, weassume that there exists y ∈ Y satisfying c ( x, y )
Revised: As stated in Section 4, weassume that there exists y ∈ Y satisfying c ( x, y )

Original: =
Revised: =

Original: 1 for all x ∈ X .
Revised: 1 for all x ∈ X .

Original: Therefore, we can guaranteethat ρ h c ( x, y ) < 

Revised: Therefore, we can guaranteethat ρ h c ( x, y ) <

Original: if c ( x, y ) = 0 .
Revised: if c ( x, y ) = 0 .

Original: Followings are the main results of the general multi-class learning problem, which is based on themargin bound shown in Theorem 9.2 of [16].
Revised: Followings are the main results of the general multi-class learning problem, which is based on themargin bound shown in Theorem 9.2 of [16].

Original: Our main finding is that the generalization error of anyhypothesis, h c is bounded by the Rademacher complexity of the hypothesis set H , which suggeststhat if we have a tight bound for hypothesis class H , then we can expect to find a good hypothesisfrom H c under any requirements c .
Revised: Our main finding is that the generalization error of anyhypothesis, h c is bounded by the Rademacher complexity of the hypothesis set H , which suggeststhat if we have a tight bound for hypothesis class H , then we can expect to find a good hypothesisfrom H c under any requirements c .

Original: Theorem 6.1.
Revised: Theorem 6.1.

Original: Let H ⊆
Revised: Let H ⊆

Original: R X×Y be a hypothesis class with Y =
Revised: R X×Y be a hypothesis class with Y =

Original: [ K ] , and c be a requirement.
Revised: [ K ] , and c be a requirement.

Original: Fixρ > 0 .
Revised: Fixρ > 0 .

Original: Then, for any δ > 0 , with probability at least 1 − δ , the following bound holds for all h c ∈ H c :
Revised: Then, for any δ > 0 , with probability at least 1 − δ , the following bound holds for all h c ∈ H c :

Original: [Equation]
Revised: [Equation]

Original: where Π 1 ( H ) is defined as:
Revised: where Π 1 ( H ) is defined as:

Original: [Equation]
Revised: [Equation]

Original: We give a proof in Appendix C.We obtain the results by showing upper bounds of the Rademachercomplexity of H c are bounded by some upper bounds of the Rademacher complexity of H . All theproofs of the theorems in this section use similar techniques.
Revised: We give a proof in Appendix C.

Original: The parameter ρ sets the margin value.
Revised: The parameter ρ sets the margin value.

Original: Following the technique shown in [16], we can obtain a generalized bound that holds uniformly forall ρ > 0 .
Revised: Following the techniqueshown in [16], we can obtain a generalized bound that holds uniformly for all ρ > 0 .

Original: The above theorem suggests that using a CV at a learning phase would not worsen theerror bound for any requirement c .
Revised: The abovetheorem suggests that using a CV at a learning phase would not worsen the error bound for anyrequirement c .

Original: Intuitively, the theorem seems reasonable since requirements cimpose a restriction on H and thus the complexity of H c is not larger than H .
Revised: Intuitively, the theorem seems reasonable since requirements c impose a restrictionon H and thus the complexity of H c is not larger than H .

Original: However, we shouldnote that it is not so trivial since H c ⊆ H is not always true.
Revised: However, we should note that it is not sotrivial since H c ⊆ H is not always true.

Original: Running time analysis We analyze the number of evaluations c ( x, y ) required for learning with a CV.
Revised: Running time analysis We analyze the number of evaluations c ( x, y ) required for learning with a CV.

Original: Let S 1 be a sub-sequence of training example S such that c ( x i , y i )
Revised: Let S 1 be a sub-sequence of training example S such that c ( x i , y i )

Original: = 1 , and S 0 be a sub-sequencesuch that c
Revised: = 1 , and S 0 be a sub-sequencesuch that c

Original: ( x i , y i )
Revised: ( x i , y i )

Original: = 0 .
Revised: = 0 .

Original: If we use a 0-1 loss function, then the empirical loss of hypothesis h c is:
Revised: If we use a 0-1 loss function, then the empirical loss of hypothesis h c is:

Original: [Equation]
Revised: [Equation]

Original: since every h c misclassifies examples in S 0 .
Revised: since every h c misclassifies examples in S 0 .

Original: Therefore, we need at most K | S 1 | + | S 0 | queries forthe learning process.
Revised: Therefore, we need at most K | S 1 | + | S 0 | queries forthe learning process.

Original: It is also true when we use a margin loss function.
Revised: It is also true when we use a margin loss function.

Original: On the other hand, theproblem of estimating the best hypothesis can be harder than the original problem depending on therequirement c .
Revised: On the other hand, theproblem of estimating the best hypothesis can be harder than the original problem depending on therequirement c .

Original: 6.2 A Tighter Bound Based on Local Rademacher Complexity
Revised: 6.2 A Tighter Bound Based on Local Rademacher Complexity

Original: The bound for H c we showed in the previous section is relatively simple, and tighter bounds of Hbased on the Rademacher complexity have been developed in the literature.
Revised: The bound for H c we showed in the previous section is relatively simple, and tighter bounds of Hbased on the Rademacher complexity have been developed in the literature.

Original: In this section, we showthe state-of-the-art error bound for H based on the local Rademacher complexity can be used as abound for H c for any requirements c .
Revised: In this section, we showthe state-of-the-art error bound for H based on the local Rademacher complexity can be used as abound for H c for any requirements c .

Original: Definition 6.2.
Revised: Definition 6.2.

Original: Let G be a family of functions from Z to R , and S be a sample of size m. Then, forany r > 0 , the empirical local Rademacher complexity of G is defined as:
Revised: Let G be a family of functions from Z to R , and S be a sample of size m. Then, forany r > 0 , the empirical local Rademacher complexity of G is defined as:

Original: [Equation]
Revised: [Equation]

Original: [13] shows a tighter generalization bound for a multi-class classification problem using the local
Revised: [13] shows a tighter generalization bound for a multi-class classification problem using the local

Original: Rademacher complexity when a hypothesis class is the ℓ p norm hypothesis space with kernel κdefined as: 
Revised: Rademacher complexity when a hypothesis class is the ℓ p norm hypothesis space with kernel κdefined as: [Equation] ≤

Original: [Equation]
Revised: [Equation]

Original: where h is represented as a vector valued function ( h 1 , .
Revised: where h is represented as a vector valued function ( h 1 , .

Original: . . , h K ) with h j ( x )
Revised: . . , h K ) with h j ( x )

Original: = h ( x, j ) ,
Revised: = h ( x, j ) ,

Original: ∀ j =1 , . .
Revised: ∀ j =1 , . .

Original: , K , and κ : X × X
Revised: , K , and κ : X × X

Original: → R is a Mercer kernel with the associated feature map ϕ , i.e., κ ( x, x ′ )
Revised: → R is a Mercer kernel with the associated feature map ϕ , i.e., κ ( x, x ′ )

Original: =⟨ ϕ ( x ) , ϕ ( x ′ ) ⟩ .
Revised: =⟨ ϕ ( x ) , ϕ ( x ′ ) ⟩ .

Original: w = ( w 1 , . . . , w K ) and
Revised: w = ( w 1 , . . . , w K ) and

Original: ∥ w ∥ = (cid:104)(cid:80) Ki =1 ∥ w ∥ p 2 (cid:105) 1 p is the ℓ 2 ,p -norm.
Revised: ∥ w ∥ = (cid:104)(cid:80) Ki =1 ∥ w ∥ p 2 (cid:105) 1 p is the ℓ 2 ,p -norm.

Original: For any p ≥ 1 , letq be the dual exponent of p satisfying 1 /p
Revised: For any p ≥ 1 , letq be the dual exponent of p satisfying 1 /p

Original: + 1 /q
Revised: + 1 /q

Original: = 1 .
Revised: = 1 .

Original: Let Φ : R → R be a loss function satisfying:1) 1 t ≤ 0 ( t ) ≤ Φ( t ) for all t ; 2) Φ( t ) is decreasing
Revised: Let Φ : R → R be a loss function satisfying: 1) 1 t ≤ 0 ( t ) ≤ Φ( t ) for all t ; 2) Φ( t ) is decreasing

Original: and it has a zero point c Φ ; 3) Φ is ζ -smooth, that is,| Φ ′ ( t ) − Φ ′ ( t ′ )
Revised: and it has a zero point c Φ ; 3) Φ is ζ -smooth , that is,| Φ ′ ( t ) − Φ ′ ( t ′ )

Original: | ≤ ζ | t − t ′ | .
Revised: | ≤ ζ | t − t ′ | .

Original: We let L c := { ( x, y ) (cid:55)→ Φ( ρ h c ( x, y )) :
Revised: We let L c := { ( x, y ) (cid:55)→ Φ( ρ h c ( x, y )) :

Original: h c ∈ H p,κ,c } .
Revised: h c ∈ H p,κ,c } .

Original: Theorem 6.3.
Revised: Theorem 6.3.

Original: Let H p,κ,c be the set of hypothesis obtained by modifying hypothesis h ∈ H p,κ withrequirement c .
Revised: Let H p,κ,c be the set of hypothesis obtained by modifying hypothesis h ∈ H p,κ withrequirement c .

Original: For any δ > 0 , with probability at least 1 − δ , the following bound holds:
Revised: For any δ > 0 , with probability at least 1 − δ , the following bound holds:

Original: [Equation]
Revised: [Equation]

Original: where ϑ = sup x ∈X κ ( x, x ) < ∞ , d = sup t ∈ R Φ( t ) < ∞ , and C d,ϑ is a constant.
Revised: where ϑ = sup x ∈X κ ( x, x ) < ∞ , d = sup t ∈ R Φ( t ) < ∞ , and C d,ϑ is a constant.

Original: ξ ( K ) is:
Revised: ξ ( K ) is:

Original: [Equation]
Revised: [Equation]

Original: otherwise .
Revised: otherwise .

Original: [Figure]
Revised: [Figure]

Original: We give a proof in Appendix D.
Revised: We give a proof in Appendix D.

Original: The bound equals that of R m
Revised: The bound equals that of R m

Original: ( L ; r ) (Theorem 1 of [13]) for anyrequirement c and any hypothesis H ρ,κ .
Revised: ( L ; r ) (Theorem 1 of [13]) for anyrequirement c and any hypothesis H ρ,κ .

Original: Therefore, the generalization error bounds in [13] based on
Revised: Therefore, the generalization error bounds in [13] based on

Original: Theorem 1 of the paper holds for any requirement c .
Revised: Theorem 1 of the paper holds for any requirement c .

Original: 6.3 Analyses of Structured Prediction
Revised: 6.3 Analyses of Structured Prediction

Original: Structured prediction is a kind of multi-class classification task, where the label set Y might be a setof sequences, images, graphs, trees, or other objects admitting some possibly overlapping structure.
Revised: Structured prediction is a kind of multi-class classification task, where the label set Y might be a setof sequences, images, graphs, trees, or other objects admitting some possibly overlapping structure.

Original: As mentioned in Section 1, previous works try to impose some constraints on the output of structuredprediction tasks.
Revised: As mentioned in Section 1, previous works try to impose some constraints on the output of structuredprediction tasks.

Original: Thus it is also useful to derive error bounds for structured prediction tasks whenwe use a CV.
Revised: Thus it is also useful to derive error bounds for structured prediction tasks whenwe use a CV.

Original: In the following, we show that the Rademacher complexity-based generalization errorbounds derived in the seminal work [6] also hold if we use a CV.
Revised: In the following, we show that the Rademacher complexity-based generalization errorbounds derived in the seminal work [6] also hold if we use a CV.

Original: Although tighter bounds are givenin more recent work [17, 14], we give bounds based on [6] due to their simplicity.
Revised: Although tighter bounds are givenin more recent work [17, 14], we give bounds based on [6] due to their simplicity.

Original: We give some definitions for the structured prediction task.
Revised: We give some definitions for the structured prediction task.

Original: Following previous work, we assumethat Y is decomposable along with substructures:
Revised: Following previous work, we assumethat Y is decomposable along with substructures:

Original: Y =
Revised: Y =

Original: Y 1 × · · · × Y l .
Revised: Y 1 × · · · × Y l .

Original: Here, Y k is the set of possiblelabels that can be assigned to the k -th substructure.
Revised: Here, Y k is the set of possiblelabels that can be assigned to the k -th substructure.

Original: We denote by L : Y × Y → R + a loss functionmeasuring the dissimilarity of two elements of the output space Y .
Revised: We denote by L : Y × Y → R + a loss functionmeasuring the dissimilarity of two elements of the output space Y .

Original: L is definite , that is, L ( y, y ′ )
Revised: L is definite , that is, L ( y, y ′ )

Original: =
Revised: =

Original: 0iff y
Revised: 0iff y

Original: =
Revised: =

Original: y ′ .
Revised: y ′ .

Original: A typical definite loss function for a structured prediction task is the Hamming loss definedby L ( y, y ′ )
Revised: A typical definite loss function for a structured prediction task is the Hamming loss definedby L ( y, y ′ )

Original: = 1 l (cid:80) lk =1 1 y k ̸ = y ′ k
Revised: = 1 l (cid:80) lk =1 1 y k ̸ = y ′ k

Original: for all y =
Revised: for all y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y l ) and y ′ =
Revised: , y l ) and y ′ =

Original: ( y ′ 1 , . . .
Revised: ( y ′ 1 , . . .

Original: , y ′ l ) , with y k ,
Revised: , y ′ l ) , with y k ,

Original: y ′ k ∈
Revised: y ′ k ∈

Original: Y k .
Revised: Y k .

Original: Other typical examples of loss functions can be seen in [6].
Revised: Other typical examples of loss functions can be seen in [6].

Original: Using loss function L , the generalizationerror and the empirical error of h is defined as:
Revised: Using loss function L , the generalizationerror and the empirical error of h is defined as:

Original: [Equation]
Revised: [Equation]

Original: Same with the multi-class classification task, hypothesis class H can be represented as a set ofscoring function h : X × Y
Revised: Same with the multi-class classification task, hypothesis class H can be represented as a set ofscoring function h : X × Y

Original: → R .
Revised: → R .

Original: We use h ( x ) to represent the predictor defined by h ∈ H :h ( x ) :=
Revised: We use h ( x ) to represent the predictor defined by h ∈ H :h ( x ) :=

Original: argmax y ∈Y h ( x, y ) for all x ∈ X .
Revised: argmax y ∈Y h ( x, y ) for all x ∈ X .

Original: Following the previous work, we assume that eachscoring function can be decomposed as a sum, and such decomposition follows a factor graph .
Revised: Following the previous work, we assume that eachscoring function can be decomposed as a sum, and such decomposition follows a factor graph .

Original: Afactor graph G is a tuple G =
Revised: Afactor graph G is a tuple G =

Original: ( V, F, E ) , where V is a set of variable nodes, F is a set of factornodes, and E is a set of undirected edges between a variable node and a factor node.
Revised: ( V, F, E ) , where V is a set of variable nodes, F is a set of factornodes, and E is a set of undirected edges between a variable node and a factor node.

Original: Every node in
Revised: Every node in

Original: V corresponds to a substructure index, that is, V = { 1 , . . .
Revised: V corresponds to a substructure index, that is, V = { 1 , . . .

Original: , l } .
Revised: , l } .

Original: For any factor node f , we denote by N ( f ) ⊆
Revised: For any factor node f , we denote by N ( f ) ⊆

Original: V the set of variable nodes connected to f anddefine Y f as the substructure set cross-product Y f
Revised: V the set of variable nodes connected to f anddefine Y f as the substructure set cross-product Y f

Original: = (cid:81) k ∈N ( f ) Y k .
Revised: = (cid:81) k ∈N ( f ) Y k .

Original: Then, h admits the followingdecomposition as a sum of functions
Revised: Then, h admits the followingdecomposition as a sum of functions

Original: h f , each taking as argument a pair of ( x, y f ) ∈
Revised: h f , each taking as argument a pair of ( x, y f ) ∈

Original: X × Y f :
Revised: X × Y f :

Original: [Equation]
Revised: [Equation]

Original: Figure 2 shows examples of decompositions based on factor graphs.
Revised: Figure 2 shows examples of decompositions based on factor graphs.

Original: We conventionally assume thatthe structure of factor graphs may change depending on a particular example ( x i , y i ) :
Revised: We conventionally assume thatthe structure of factor graphs may change depending on a particular example ( x i , y i ) :

Original: G ( x i , y i ) =
Revised: G ( x i , y i ) =

Original: G i =
Revised: G i =

Original: ([ l i ] , F i , E i ) .
Revised: ([ l i ] , F i , E i ) .

Original: A special case of this setting is when the size l i of each example is allowed tovary.
Revised: A special case of this setting is when the size l i of each example is allowed tovary.

Original: In such a case, the number of possible labels Y is potentially infinite.
Revised: In such a case, the number of possible labels Y is potentially infinite.

Original: Following the multi-class classification, our CV maps hypothesis h to h c to satisfy the requirements.
Revised: Following the multi-class classification, our CV maps hypothesis h to h c to satisfy the requirements.

Original: The definition of h c follows (4).
Revised: The definition of h c follows (4).

Original: We should note that the definition does not require h c to have afactored representation.
Revised: We should note that the definition does not require h c to have afactored representation.

Original: For analyzing the complexity, [6] introduces the empirical factor graph Rademacher complexity
Revised: For analyzing the complexity, [6] introduces the empirical factor graph Rademacher complexity

Original: R GS ( H ) of a hypothesis class H for a sample S =
Revised: R GS ( H ) of a hypothesis class H for a sample S =

Original: ( x 1 , . . .
Revised: ( x 1 , . . .

Original: , x m ) and factor graph G as follows:
Revised: , x m ) and factor graph G as follows:

Original: [Equation]
Revised: [Equation]

Original: where ϵ =
Revised: where ϵ =

Original: ( ϵ i,f,y ) i
Revised: ( ϵ i,f,y ) i

Original: ∈
Revised: ∈

Original: [ m ] ,f ∈ F i ,y ∈Y f and every ϵ i,f,y is i.i.d.
Revised: [ m ] ,f ∈ F i ,y ∈Y f and every ϵ i,f,y is i.i.d.

Original: Rademacher random variable.
Revised: Rademacher random variable.

Original: The factor 331 graph Rademacher complexity of H for a factor graph G is defined as the expectation:
Revised: The factor 326 graph Rademacher complexity of H for a factor graph G is defined as the expectation:

Original: [Equation]
Revised: [Equation]

Original: By using the factor graph Rademacher complexity, [6] gives bounds for structured prediction taskwith the following additive and multiplicative empirical losses:
Revised: By using the factor graph Rademacher complexity, [6] gives bounds for structured prediction taskwith the following additive and multiplicative empirical losses:

Original: [Equation]
Revised: [Equation]

Original: where Φ ∗ ( t ) = min( B, max(0 , t )) for all t , with B = max y,y ′
Revised: where Φ ∗ ( t ) = min( B, max(0 , t )) for all t , with B = max y,y ′

Original: L ( y, y ′ ) .
Revised: L ( y, y ′ ) .

Original: As shown in [6], these lossfunctions cover typical surrogate loss function used in structured prediction tasks.
Revised: As shown in [6], these lossfunctions cover typical surrogate loss function used in structured prediction tasks.

Original: We show thefollowing bound for structured prediction.
Revised: We show thefollowing bound for structured prediction.

Original: Theorem 6.4.
Revised: Theorem 6.4.

Original: Fix ρ > 0 .
Revised: Fix ρ > 0 .

Original: For any δ > 0 and requirement c , with probability at least 1 − δ over thedraw of a sample S of size m from distribution D , the following holds for all h c ∈ H c :
Revised: For any δ > 0 and requirement c , with probability at least 1 − δ over thedraw of a sample S of size m from distribution D , the following holds for all h c ∈ H c :

Original: [Equation]
Revised: [Equation]

Original: We give a proof in Appendix E. ρ is a parameter that determines the margin.
Revised: We give a proof in Appendix E. ρ is a parameter that determines the margin.

Original: Similar to the case ofthe multi-class classification, we can derive a bound that holds for any ρ > 0 following the derivationof [6].
Revised: Similar to the case ofthe multi-class classification, we can derive a bound that holds for any ρ > 0 following the derivationof [6].

Original: The above result indicates that the bound will not change if we use a CV for any requirementc .
Revised: The above result indicates that the bound will not change if we use a CV for any requirementc .

Original: This is interesting since the above result holds even if we do not have a factored representation ofh c ( x, y ) similar to (5), while the derived bound depends on the factor graph Rademacher complexity,which depends on the factored representation of h ( x, y ) .
Revised: This is interesting since the above result holds even if we do not have a factored representation ofh c ( x, y ) similar to (5), while the derived bound depends on the factor graph Rademacher complexity,which depends on the factored representation of h ( x, y ) .

Original: We analyze the overhead in running time for evaluating loss function L add S,ρ ( h c ) and L mult S,ρ ( h c ) forhypothesis h c .
Revised: We analyze the overhead in running time for evaluating loss function L add S,ρ ( h c ) and L mult S,ρ ( h c )for hypothesis h c .

Revised: Evaluating them needs to solve an optimization problem of finding y ∈

Original: Different from the multi-class classification case, both the number of queries andthe overhead in running time for the loss evaluation when we use a CV depends on both the modeland the type of requirements for structured prediction. This result is consistent with the literaturereporting that for the structured prediction task, it is known that originally tractable optimizationproblems can be intractable if we put additional constraints [21]. 
Revised: Ythat maximizes h c ( x, y ) . For the structured prediction task, it is known that originally tractableoptimization problems can be intractable if we put additional constraints [21]. Similarly, the overheadin running time for evaluating loss function when we use a CV depends on both the model and thetype of requirements.

Original:  Conclusion
Revised:  Conclusion

Original: This paper gives a generalization analysis when there are requirements that input-output pairs ofa machine learning model must satisfy.
Revised: This paper gives a generalization analysis when there are requirements that input-output pairs ofa machine learning model must satisfy.

Original: We introduce a concurrent verifier, a simple module thatenables us to guarantee that the input-output pairs of a machine learning model satisfy requirements.
Revised: We introduce a concurrent verifier, a simple module thatenables us to guarantee that the input-output pairs of a machine learning model satisfy requirements.

Original: We show the situation where we can obtain a hypothesis with a small error when we use a verifier onlyin an inference phase.
Revised: We show the situation where we can obtain a hypothesis with a small error when we use a verifier onlyin an inference phase.

Original: Interestingly, if H is not PAC learnable, we might fail to obtain a guaranteedhypothesis even if requirements are consistent with distribution D .
Revised: Interestingly, if H is not PAC learnable, we might fail to obtain a guaranteedhypothesis even if requirements are consistent with distribution D .

Original: We also give generalizationbounds based on the Rademacher complexity when we use a verifier in a learning phase.
Revised: We also give generalizationbounds based on the Rademacher complexity when we use a verifier in a learning phase.

Original: We find thatthe obtained bounds are less than or equal to existing bounds, independent of the machine learningmodel and type of requirements.
Revised: We find thatthe obtained bounds are less than or equal to existing bounds, independent of the machine learningmodel and type of requirements.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No]  or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes]
Revised: [Yes]

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.•
Revised: [No] The code and the data areproprietary.•

Original: Did you include the license to the code and datasets?
Revised: Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] We discuss conditions a verifier
Revised: [Yes] We discuss conditions a verifier

Original: does not work well in Section 5.
Revised: does not work well in Section 5.(c)

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]
Revised: [Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] We give proof inAppendix3.
Revised: [Yes] We give proof inAppendix3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi- ments multiple times)?
Revised: g., with respect to the random seed after running experiments multiple times)?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you include the total amount of compute and the type of resources used (e
Revised: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., type of GPUs, internal cluster, or cloud provider)?
Revised: g., typeof GPUs, internal cluster, or cloud provider)?

Original: [N/A]
Revised: [N/A]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you mention the license of the assets?
Revised: ) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: ) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: b) Did you describe any potential participant risks, with links to Institutional Review
Revised: Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board ( ) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: c) Did youinclude the estimated hourly wage paid to participants and the total amount
Revised: Did you the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]