Original: Generalization Analysis on Learning with a Concurrent Verifier
Revised: Generalization Analysis on Learning with a Concurrent Verifier

Original: Abstract
Revised: Abstract

Original: Machine learning technologies have been used in a wide range of practical systems.
Revised: Machine learning technologies have been used in a wide range of practical systems.

Original: In practical situations, it is natural to expect the input-output pairs of a machine learning model to satisfy some requirements.
Revised: In practical situations, it is natural to expect the input-output pairs of a machine learning model to satisfy some requirements.

Original: However, it is difficult to obtain a model that satisfies requirements by just learning from examples.
Revised: However, it is difficult to obtain a model that satisfies requirements by just learning from examples.

Original: A simple solution is to add a module that checks whether the input-output pairs meet the requirements and then modifies the model’s outputs.
Revised: A simple solution is to add a module that checks whether the input-output pairs meet the requirements and then modifies the model’s outputs.

Original: Such a module, which we call a concurrent verifier (CV), can give a certification, although how the generalizability of the machine learning model changes using a CV is unclear.
Revised: Such a module, which we call a concurrent verifier (CV), can give a certification, although how the generalizability of the machine learning model changes using a CV is unclear.

Original: This paper gives a generalization analysis of learning with a CV.
Revised: This paper gives a generalization analysis of learning with a CV.

Original: We analyze how the learnability of a machine learning model changes with a CV and show a condition where we can obtain a guaranteed hypothesis using a verifier only in the inference time.
Revised: We analyze how the learnability of a machine learning model changes with a CV and show a condition where we can obtain a guaranteed hypothesis using a verifier only in the inference time.

Original: We also show that typical error bounds based on Rademacher complexity will be no larger than that of the original model when using a CV in multi-class classification and structured prediction settings.
Revised: We also show that typical error bounds based on Rademacher complexity will be no larger than that of the original model when using a CV in multi-class classification and structured prediction settings.

Original:  Introduction
Revised:  Introduction

Original: As machine learning technology matures many systems have been developed that exploit machine learning models.
Revised: As machine learning technology matures many systems have been developed that exploit machine learning models.

Original: When developing a system that uses a machine learning model a model with merely small prediction error is not satisfactory due to real-field requirements.
Revised: When developing a system that uses a machine learning model a model with merely small prediction error is not satisfactory due to real-field requirements.

Original: For example, an object recognition model that is sensitive to slight noise would cause security issues [4, 28] or a model with unexpected output would increase a system’s cost for dealing with it.
Revised: For example, an object recognition model that is sensitive to slight noise would cause security issues [4, 26] or a model with unexpected output would increase a system’s cost for dealing with it.

Original: Thus, we want the input-output pairs of a machine learning model to satisfy some requirements .
Revised: Thus, we want the input-output pairs of a machine learning model to satisfy some requirements .

Original: However, it is difficult to obtain a model that satisfies the requirements by just learning from examples.
Revised: However, it is difficult to obtain a model that satisfies the requirements by just learning from examples.

Original: Moreover, since the learned models tend to be complex and the input domain tends to be quite large, it is unrealistic to certify that every input-output pair satisfies the requirements.
Revised: Moreover, since the learned models tend to be complex and the input domain tends to be quite large, it is unrealistic to certify that every input-output pair satisfies the requirements.

Original: In addition, even if we find an input-output pair that does not satisfy the requirements, modifying a model is difficult since we have to re-estimate it from the training examples.
Revised: In addition, even if we find an input-output pair that does not satisfy the requirements, modifying a model is difficult since we have to re-estimate it from the training examples.

Original: This paper considers a way to obtain a machine learning model whose input-output pairs satisfy the required properties.
Revised: This paper considers a way to obtain a machine learning model whose input-output pairs satisfy the required properties.

Original: We address the following assumptions for a situation where a machine learning model is used.
Revised: We address the following assumptions for a situation where a machine learning model is used.

Original: First, we can judge whether input-output pair ( x, h ( x )) satisfies the requirements, where h : X → Y is a machine learning model or a hypothesis.
Revised: First, we can judge whether input-output pair ( x, h ( x )) satisfies the requirements, where h : X → Y is a machine learning model or a hypothesis.

Original: As we show below, important use cases fit this setting.
Revised: As we show below, important use cases fit this setting.

Original: Second, a machine learning model already exists whose prediction error is small enough, although its input-output pairs are not guaranteed to satisfy the requirements.
Revised: Second, a machine learning model already exists whose prediction error is small enough, although its input-output pairs are not guaranteed to satisfy the requirements.

Original: This second assumption is also reasonable since modern machine learning models show sufficient prediction accuracy in various tasks.
Revised: This second assumption is also reasonable since modern machine learning models show sufficient prediction accuracy in various tasks.

Original: Under these assumptions, a practical choice for addressing this problem isn’t changing the machine learning model but adding a module that checks the input-output pairs of machine learning model h .
Revised: Under these assumptions, a practical choice for addressing this problem isn’t changing the machine learning model but adding a module that checks the input-output pairs of machine learning model h .

Original: We call this module a concurrent verifier (CV).
Revised: We call this module a concurrent verifier (CV).

Original: Fig.
Revised: Fig.

Original: 1 shows the system configuration of a machine learning model with a CV.
Revised: 1 shows the system configuration of a machine learning model with a CV.

Original: The verifier checks whether the input-output 36th
Revised: The verifier checks whether the input-output 36th

Original: Neural
Revised: Neural

Original: Processing Systems (
Revised: Processing Systems (

Original: [Figure]
Revised: [Figure]

Original: pair ( x, h ( x )) satisfies the required properties.
Revised: pair ( x, h ( x )) satisfies the required properties.

Original: If it satisfies the requirements, it outputs h ( x ) .
Revised: If it satisfies the requirements, it outputs h ( x ) .

Original: If not, then it rejects h ( x ) and modifies or requests the learning model to modify its output.
Revised: If not, then it rejects h ( x ) and modifies or requests the learning model to modify its output.

Original: A machine learning model and verifier pair can be seen as another machine learning model whose input-output pairs are guaranteed to satisfy the required conditions.
Revised: A machine learning model and verifier pair can be seen as another machine learning model whose input-output pairs are guaranteed to satisfy the required conditions.

Original: Although a model with a verifier can guarantee that its input-output pairs satisfy requirements, its effect on prediction error is unclear.
Revised: Although a model with a verifier can guarantee that its input-output pairs satisfy requirements, its effect on prediction error is unclear.

Original: This paper gives theoretical analyses of the generalization errors of a machine learning model with a CV.
Revised: This paper gives theoretical analyses of the generalization errors of a machine learning model with a CV.

Original: We focus on how the learnability of the original model, denoted as hypothesis class H , can change by using the verifier.
Revised: We focus on how the learnability of the original model, denoted as hypothesis class H , can change by using the verifier.

Original: First, we consider a situation where we use a CV only in the inference phase.
Revised: First, we consider a situation where we use a CV only in the inference phase.

Original: This setting corresponds to a case where the required properties are unknown when we are in the training phase.
Revised: This setting corresponds to a case where the required properties are unknown when we are in the training phase.

Original: If the hypothesis class is PAC-learnable, we can obtain a guaranteed hypothesis using a verifier only in the inference time.
Revised: If the hypothesis class is PAC-learnable, we can obtain a guaranteed hypothesis using a verifier only in the inference time.

Original: Second, we consider a situation where we know the requirements when learning the model.
Revised: Second, we consider a situation where we know the requirements when learning the model.

Original: This situation corresponds to viewing the learnability of hypothesis set H c , which is obtained by modifying every hypothesis h ∈ H to satisfy the requirements.
Revised: This situation corresponds to viewing the learnability of hypothesis set H c , which is obtained by modifying every hypothesis h ∈ H to satisfy the requirements.

Original: Hence we compare the generalization error upper bounds of H c with those of H .
Revised: Hence we compare the generalization error upper bounds of H c with those of H .

Original: On the multi-class classification setting, we show that existing error bounds [15, 18] based on the Rademacher complexity of H are also bounds of modified hypothesis
Revised: On the multi-class classification setting, we show that existing error bounds [13, 16] based on the Rademacher complexity of H are also bounds of modified hypothesis

Original: H c for any input-output requirements.
Revised: H c for any input-output requirements.

Original: Moreover, we give similar analyses for a structured prediction task, which is a kind of multi-class classification where set of classes Y can be decomposed into substructures.
Revised: Moreover, we give similar analyses for a structured prediction task, which is a kind of multi-class classification where set of classes Y can be decomposed into substructures.

Original: It is worth analyzing the task since many works address the constraints in structured prediction.
Revised: It is worth analyzing the task since many works address the constraints in structured prediction.

Original: Some works give error bounds for structured prediction tasks, which are tighter than simply applying the bound for multi-class classification tasks [16, 6, 19].
Revised: Some works give error bounds for structured prediction tasks, which are tighter than simply applying the bound for multi-class classification tasks [14, 6, 17].

Original: Similar to the case of multiclass classification, we show that existing Rademacher complexity-based bounds for the structured prediction of H are also the bounds for H c .
Revised: Similar to the case of multiclass classification, we show that existing Rademacher complexity-based bounds for the structured prediction of H are also the bounds for H c .

Original: Our main contributions are as follows: a) We introduce a concurrent verifier, which is a modelagnostic way to guarantee that machine learning models satisfy the required properties.
Revised: Our main contributions are as follows: a) We introduce a concurrent verifier, which is a modelagnostic way to guarantee that machine learning models satisfy the required properties.

Original: Although a similar mechanism was used in some existing models, our model gives a generalization analysis that does not depend on a specific model.
Revised: Although a similar mechanism was used in some existing models, our model gives a generalization analysis that does not depend on a specific model.

Original: b) We show that if hypothesis class H is PAC-learnable, then using a verifier at the inference time can give a hypothesis with a guarantee in its generalization error.
Revised: b) We show that if hypothesis class H is PAC-learnable, then using a verifier at the inference time can give a hypothesis with a guarantee in its generalization error.

Original: Interestingly, if H is not PAC-learnable, we might fail to obtain a guaranteed hypothesis even if the requirements are consistent with distribution D .
Revised: Interestingly, if H is not PAC-learnable, we might fail to obtain a guaranteed hypothesis even if the requirements are consistent with distribution D .

Original: c)
Revised: c)

Original: We show that if we use a CV in a learning phase of multi-class classification tasks, then the theoretical error bounds of H based on the Rademacher complexity will not increase with any input-output requirements.
Revised: We show that if we use a CV in a learning phase of multi-class classification tasks, then the theoretical error bounds of H based on the Rademacher complexity will not increase with any input-output requirements.

Original: We also give similar results for structured prediction tasks.
Revised: We also give similar results for structured prediction tasks.

Original: 1.1 Use Cases of a Concurrent Verifier
Revised: 1.1 Use Cases of a Concurrent Verifier

Original: The following are some typical use cases for CVs.
Revised: The following are some typical use cases for CVs.

Original: Error-sensitive applications: A typical situation where we want to use a verifier is that some prediction errors might cause severe effects, which we want to avoid.
Revised: Error-sensitive applications: A typical situation where we want to use a verifier is that some prediction errors might cause severe effects, which we want to avoid.

Original: For example, a recommender system might limit the set of candidate items depending on user attributes.
Revised: For example, a recommender system might limit the set of candidate items depending on user attributes.

Original: Although such a rule might degrade the prediction accuracy, practically a safer model is preferable.
Revised: Although such a rule might degrade the prediction accuracy, practically a safer model is preferable.

Original: Controlling outputs of structured prediction: Constraints are frequently used in structured prediction tasks for improving the performance or the controllability of the outputs.
Revised: Controlling outputs of structured prediction: Constraints are frequently used in structured prediction tasks for improving the performance or the controllability of the outputs.

Original: For example, some works [23, 5] exploited the constraints on sequence labeling tasks for reflecting background knowledge to improve the prediction results.
Revised: For example, some works [21, 5] exploited the constraints on sequence labeling tasks for reflecting background knowledge to improve the prediction results.

Original: More recently, some works [9, 2] exploited the constraints in language generation tasks, including image captioning and machine translation, and restricted a model to output a sentence that includes given keywords.
Revised: More recently, some works [9, 2] exploited the constraints in language generation tasks, including image captioning and machine translation, and restricted a model to output a sentence that includes given keywords.

Original: Since the constraints used in this previous work can be written as a logical formula, our CV model can represent them as requirements.
Revised: Since the constraints used in this previous work can be written as a logical formula, our CV model can represent them as requirements.

Original: Robustness against input perturbations: If a machine learning model changes its output because we modified its input from x to x ′ , which is very close to x , then the model is described as sensitive against a small change [27].
Revised: Robustness against input perturbations: If a machine learning model changes its output because we modified its input from x to x ′ , which is very close to x , then the model is described as sensitive against a small change [25].

Original: It might be a security risk if a model is sensitive since its behavior is unpredictable.
Revised: It might be a security risk if a model is sensitive since its behavior is unpredictable.

Original: Therefore, some methods evaluate and verify the robustness of neural networks against small perturbations [28, 4].
Revised: Therefore, some methods evaluate and verify the robustness of neural networks against small perturbations [26, 4].

Original: Existing verification methods check a machine learning model’s robustness around input x by determining whether x ′ exists that is close to x and whether model f gives different outputs, i.e., h ( x ) ̸ = h ( x ′ ) , for verification samples x 1 , . . .
Revised: Existing verification methods check a machine learning model’s robustness around input x by determining whether x ′ exists that is close to x and whether model f gives different outputs, i.e., h ( x ) ̸ = h ( x ′ ) , for verification samples x 1 , . . .

Original: , x n .
Revised: , x n .

Original: Although these verification methods can test a model, they do not directly show how to obtain a robust model.
Revised: Although these verification methods can test a model, they do not directly show how to obtain a robust model.

Original: A CV can fix a model to achieve robustness around samples x 1 , . .
Revised: A CV can fix a model to achieve robustness around samples x 1 , . .

Original: , x n by setting a rule of form: “ h ( x ′ ) must equal h ( x i ) if x ′ is close to x i .”
Revised: , x n by setting a rule of form: “ h ( x ′ ) must equal h ( x i ) if x ′ is close to x i .”

Original: Although this solution might not guarantee robustness where samples are scarce, adding enough non-labeled verification samples is often a reasonable choice.
Revised: Although this solution might not guarantee robustness where samples are scarce, adding enough non-labeled verification samples is often a reasonable choice.

Original:  Related Work
Revised:  Related Work

Original: Machine learning models that can exploit constraints have been investigated in many research fields, including statistical symbolic learning and structured prediction.
Revised: Machine learning models that can exploit constraints have been investigated in many research fields, including statistical symbolic learning and structured prediction.

Original: For example, Markov logic networks [22], Problogs [8], and probabilistic circuit models [11] integrate statistical models with symbolic logic formulations.
Revised: For example, Markov logic networks [20], Problogs [8], and probabilistic circuit models [11] integrate statistical models with symbolic logic formulations.

Original: Since these models can incorporate hard constraints represented by symbolic logic, they can guarantee input-output pairs.
Revised: Since these models can incorporate hard constraints represented by symbolic logic, they can guarantee input-output pairs.

Original: However, previous research focused on their practical performance and gave little theoretical analysis of their learnability when hard constraints are used.
Revised: However, previous research focused on their practical performance and gave little theoretical analysis of their learnability when hard constraints are used.

Original: Moreover, previous works integrated the ability to exploit constraints into specific models.
Revised: Moreover, previous works integrated the ability to exploit constraints into specific models.

Original: In contrast, our CV is model-agnostic and can be used in combination with a wide range of machine learning models.
Revised: In contrast, our CV is model-agnostic and can be used in combination with a wide range of machine learning models.

Original: Recently, the verification of machine learning models has been gathering more attention.
Revised: Recently, the verification of machine learning models has been gathering more attention.

Original: Attempts have verfified whether a machine learning model has the desired properties [4, 28, 10, 26].
Revised: Attempts have verfified whether a machine learning model has the desired properties [4, 26, 10, 24].

Original: Exact verification methods use integer programming (MIP) [28], constraint satisfaction (SAT) [20], and a satisfiable module theory (SMT) solver [10] to assess the robustness of a neural network model against input noise.
Revised: Exact verification methods use integer programming (MIP) [26], constraint satisfaction (SAT) [18], and a satisfiable module theory (SMT) solver [10] to assess the robustness of a neural network model against input noise.

Original: These approaches aim to obtain models that fulfill the required properties.
Revised: These approaches aim to obtain models that fulfill the required properties.

Original: However, verification methods cannot help modify the models if they do not satisfy the requirements.
Revised: However, verification methods cannot help modify the models if they do not satisfy the requirements.

Original: If we want ML models to meet requirements, post-processing is needed as our concurrent verification model.
Revised: If we want ML models to meet requirements, post-processing is needed as our concurrent verification model.

Original: Other methods can give upper bounds on generalization error, including VC-dimension [29] and its extensions [7, 21], Rademacher complexity [3, 12], stability [25], and PAC-Bayes [17, 1].
Revised: Other methods can give upper bounds on generalization error, including VC-dimension [27] and its extensions [7, 19], Rademacher complexity [3, 12], stability [23], and PAC-Bayes [15, 1].

Original: We use
Revised: We use

Original: Rademacher complexity in the following analysis since it is among the most popular tools for giving theoretical upper bounds on generalization error.
Revised: Rademacher complexity in the following analysis since it is among the most popular tools for giving theoretical upper bounds on generalization error.

Original: Rademacher complexity also has some extensions, including local Rademacher complexity [15] and factor graph Rademacher complexity [6].
Revised: Rademacher complexity also has some extensions, including local Rademacher complexity [13] and factor graph Rademacher complexity [6].

Original: We can provide theoretical guarantees on these extended measures.
Revised: We can provide theoretical guarantees on these extended measures.

Original:  Preliminaries
Revised:  Preliminaries

Original: Our notation follows a previous work [24].
Revised: Our notation follows a previous work [22].

Original: We first introduce the notations used in the following sections.
Revised: We first introduce the notations used in the following sections.

Original: Let X denote the domain of the inputs, let Y be the domain of the labels, and let Z be the domain of the examples defined as Z := X ×Y .
Revised: Let X denote the domain of the inputs, let Y be the domain of the labels, and let Z be the domain of the examples defined as Z := X ×Y .

Original: Let H be a hypothesis class, and let ℓ : H× Z → R + be a loss function.
Revised: Let H be a hypothesis class, and let ℓ : H× Z → R + be a loss function.

Original: Training data S
Revised: Training data S

Original: = ( z 1 , . . .
Revised: = ( z 1 , . . .

Original: , z m ) ∈
Revised: , z m ) ∈

Original: Z m is a finite sequence of size m drawn i.i.d.
Revised: Z m is a finite sequence of size m drawn i.i.d.

Original: from a fixed but unknown probability distribution D on Z .
Revised: from a fixed but unknown probability distribution D on Z .

Original: Learning algorithm
Revised: Learning algorithm

Original: A maps training data
Revised: A maps training data

Original: S to hypothesis h .
Revised: S to hypothesis h .

Original: We use notation A ( S ) to denote the hypothesis that learning algorithm
Revised: We use notation A ( S ) to denote the hypothesis that learning algorithm

Original: A returns upon receiving S .
Revised: A returns upon receiving S .

Original: We represent set { 1 , . .
Revised: We represent set { 1 , . .

Original: , K } as [ K ] .
Revised: , K } as [ K ] .

Original: Given distribution D on Z , we denote by L D ( h ) the generalization error and by L S ( h ) the empirical error of h over S , defined by
Revised: Given distribution D on Z , we denote by L D ( h ) the generalization error and by L S ( h ) the empirical error of h over S , defined by

Original: [Equation]
Revised: [Equation]

Original: PAC learnability: We introduce PAC learnability and agnostic PAC learnability as follows.
Revised: PAC learnability: We introduce PAC learnability and agnostic PAC learnability as follows.

Original: Definition 3.1.
Revised: Definition 3.1.

Original: (Agnostic PAC learnability)
Revised: (Agnostic PAC learnability)

Original: Hypothesis class H is agnostic PAC-learnable if there exists function m H : (0 , 1) 2 → N and learning algorithm A with the following property: For every ϵ, δ ∈ (0 , 1) and distribution D over Z , if S consists of m ≥ m H ( ϵ, δ ) i.i.d.
Revised: Hypothesis class H is agnostic PAC-learnable if there exists function m H : (0 , 1) 2 → N and learning algorithm A with the following property: For every ϵ, δ ∈ (0 , 1) and distribution D over Z , if S consists of m ≥ m H ( ϵ, δ ) i.i.d.

Original: examples generated by
Revised: examples generated by

Original: D , then with at least probability 1 − δ , the following holds:
Revised: D , then with at least probability 1 − δ , the following holds:

Original: [Equation]
Revised: [Equation]

Original: Distribution D is realizable by hypothesis set H if h ∗ ∈ H exists such that L D ( h ∗ )
Revised: Distribution D is realizable by hypothesis set H if h ∗ ∈ H exists such that L D ( h ∗ )

Original: = 0 .
Revised: = 0 .

Original: If D is realizable by agnostic PAC-learnable hypothesis H , then H is PAC-learnable .
Revised: If D is realizable by agnostic PAC-learnable hypothesis H , then H is PAC-learnable .

Original: If H is PAC-learnable, then Eq.
Revised: If H is PAC-learnable, then Eq.

Original: (2) becomes L D ( A ( S ))
Revised: (2) becomes L D ( A ( S ))

Original: ≤ ϵ since min h ′ ∈H L D ( h ′ )
Revised: ≤ ϵ since min h ′ ∈H L D ( h ′ )

Original: = 0 .
Revised: = 0 .

Original: Rademacher complexity: In the following sections, we use Rademacher complexity for deriving the generalization bounds.
Revised: Rademacher complexity: In the following sections, we use Rademacher complexity for deriving the generalization bounds.

Original: Given loss function ℓ ( h, z ) and hypothesis class H , we denote G as
Revised: Given loss function ℓ ( h, z ) and hypothesis class H , we denote G as

Original: [Equation]
Revised: [Equation]

Original: Definition 3.2.
Revised: Definition 3.2.

Original: (Empirical Rademacher complexity) Let G be a family of functions mapping from Z to R , and let S =
Revised: (Empirical Rademacher complexity) Let G be a family of functions mapping from Z to R , and let S =

Original: ( z 1 , . . .
Revised: ( z 1 , . . .

Original: , z m ) ∈
Revised: , z m ) ∈

Original: Z m be the training data of size m .
Revised: Z m be the training data of size m .

Original: Then the empirical Rademacher complexity of G with respect to S is defined:
Revised: Then the empirical Rademacher complexity of G with respect to S is defined:

Original: [Equation]
Revised: [Equation]

Original: where σ = ( σ 1 , . .
Revised: where σ = ( σ 1 , . .

Original: , σ m ) ∈ {± 1 } m are random variables distributed i.i.d.
Revised: , σ m ) ∈ {± 1 } m are random variables distributed i.i.d.

Original: according to P [ σ i = 1] = P [ σ i = − 1] = 1 / 2 .
Revised: according to P [ σ i = 1] = P [ σ i = − 1] = 1 / 2 .

Original: The Rademacher complexity of G is defined as the expected value of the empirical Rademacher complexity:
Revised: The Rademacher complexity of G is defined as the expected value of the empirical Rademacher complexity:

Original: [Equation]
Revised: [Equation]

Original:  Concurrent Verifier
Revised:  Concurrent Verifier

Original: Next we give a formal definition of a CV.
Revised: Next we give a formal definition of a CV.

Original: A CV works with a machine learning model, which is function h : X → Y .
Revised: A CV works with a machine learning model, which is function h : X → Y .

Original: If x is given to the model, which outputs h ( x ) , then the verifier checks whether ( x, h ( x )) satisfies the required property.
Revised: If x is given to the model, which outputs h ( x ) , then the verifier checks whether ( x, h ( x )) satisfies the required property.

Original: We assume that the required property can be represented as requirement function c : ( X × Y ) → { 0 , 1 } .
Revised: We assume that the required property can be represented as requirement function c : ( X × Y ) → { 0 , 1 } .

Original: If c ( x, h ( x ))
Revised: If c ( x, h ( x ))

Original: = 1 , then the pair satisfies the property; if c ( x, h ( x ))
Revised: = 1 , then the pair satisfies the property; if c ( x, h ( x ))

Original: = 0 , then it does not.
Revised: = 0 , then it does not.

Original: Requirement function c can be represented by a set of deterministic rules.
Revised: Requirement function c can be represented by a set of deterministic rules.

Original: For example, if X = R and Y = { 0 , 1 } , then the requirements can be in the following form: “if x > 0 , then y ̸ = 0 .”
Revised: For example, if X = R and Y = { 0 , 1 } , then the requirements can be in the following form: “if x > 0 , then y ̸ = 0 .”

Original: We assume that for all possible input x ∈ X , there exists y ∈ Y such that c ( x, y )
Revised: We assume that for all possible input x ∈ X , there exists y ∈ Y such that c ( x, y )

Original: = 1 for avoiding the situation where the requirements are unsatisfiable for any output y .
Revised: = 1 for avoiding the situation where the requirements are unsatisfiable for any output y .

Original: This assumption can be easily relaxed if we allow a machine learning model to reject unsatisfiable input x .
Revised: This assumption can be easily relaxed if we allow a machine learning model to reject unsatisfiable input x .

Original: After checking the input-output pair, a verifier modifies output h ( x ) depending on the value of c ( x, h ( x )) .
Revised: After checking the input-output pair, a verifier modifies output h ( x ) depending on the value of c ( x, h ( x )) .

Original: If c ( x, h ( x ))
Revised: If c ( x, h ( x ))

Original: = 1 , the verifier outputs h ( x ) since it satisfies the requirements.
Revised: = 1 , the verifier outputs h ( x ) since it satisfies the requirements.

Original: If c ( x, h ( x ))
Revised: If c ( x, h ( x ))

Original: = 0 , then the verifier modifies h ( x ) to some y ∈ Y that satisfies c ( x, y )
Revised: = 0 , then the verifier modifies h ( x ) to some y ∈ Y that satisfies c ( x, y )

Original: = 1 .
Revised: = 1 .

Original: If we use a verifier with a machine learning model that corresponds to h , then the combination of the model and the verifier can be seen as function h c : X → Y , defined as
Revised: If we use a verifier with a machine learning model that corresponds to h , then the combination of the model and the verifier can be seen as function h c : X → Y , defined as

Original: [Equation]
Revised: [Equation]

Original: 3) where y c ∈ Y satisfies c ( x, y c )
Revised: 3) where y c ∈ Y satisfies c ( x, y c )

Original: = 

Revised: =

Original: and is selected deterministically.
Revised: and is selected deterministically.

Original: When Y =
Revised: When Y =

Original: [ K ] , an example for selecting minimum i ∈
Revised: [ K ] , an example for selecting minimum i ∈

Original: [ K ] satisfying c ( x, i )
Revised: [ K ] satisfying c ( x, i )

Original: = 1 as y c is a reasonable choice.
Revised: = 1 as y c is a reasonable choice.

Original: When Y =
Revised: When Y =

Original: [ K ] and h ( x ) is made by scoring functions
Revised: [ K ] and h ( x ) is made by scoring functions

Original: h ( x, y ) : ( X × Y ) → R , it is also reasonable to select y ∗ such that y ∗ =
Revised: h ( x, y ) : ( X × Y ) → R , it is also reasonable to select y ∗ such that y ∗ =

Original: argmax y ∈Y ,c ( x,y )
Revised: argmax y ∈Y ,c ( x,y )

Original: =1 h ( x, y ) .
Revised: =1 h ( x, y ) .

Original: Learning a model corresponds to selecting hypothesis h from hypothesis class H .
Revised: Learning a model corresponds to selecting hypothesis h from hypothesis class H .

Original: Therefore, learning a model with a CV corresponds to choosing a hypothesis from the modified hypothesis class: H c = { h c : h ∈ H} .
Revised: Therefore, learning a model with a CV corresponds to choosing a hypothesis from the modified hypothesis class: H c = { h c : h ∈ H} .

Original: By definition, every hypothesis in H c satisfies the requirements, and thus we can guarantee that the model satisfies the condition if we select a hypothesis from H c .
Revised: By definition, every hypothesis in H c satisfies the requirements, and thus we can guarantee that the model satisfies the condition if we select a hypothesis from H c .

Original: In the following sections, we analyze the learnability of H c by comparing it with that of H .
Revised: In the following sections, we analyze the learnability of H c by comparing it with that of H .

Original:  Inference Time Verification
Revised:  Inference Time Verification

Original: We first analyze the change of the generalization errors when we use a verifier only in an inference phase.
Revised: We first analyze the change of the generalization errors when we use a verifier only in an inference phase.

Original: In other words, requirements are unknown in the learning phase,and we estimate hypothesis ˆ h
Revised: In other words, requirements are unknown in the learning phase,and we estimate hypothesis ˆ h

Original: =
Revised: =

Original: A ( S ) from hypothesis class H by using training data S and algorithm A .
Revised: A ( S ) from hypothesis class H by using training data S and algorithm A .

Original: In the inference phase, we use a CV to modify ˆ h to ˆ h c .
Revised: In the inference phase, we use a CV to modify ˆ h to ˆ h c .

Original: We call this setting the inference time verification (ITV).
Revised: We call this setting the inference time verification (ITV).

Original: This class of situations contains many exciting settings: 1) pre-trained machine learning models used in a wide range of applications, and 2) models that are hard to replace, which might encounter different requirements from those at the learning time in the long run.
Revised: This class of situations contains many exciting settings: 1) pre-trained machine learning models used in a wide range of applications, and 2) models that are hard to replace, which might encounter different requirements from those at the learning time in the long run.

Original: In this section, we give analyses on a multi-class classification setting.
Revised: In this section, we give analyses on a multi-class classification setting.

Original: We set Y =
Revised: We set Y =

Original: [ K ] , and hypothesis class H is set of mappings h :
Revised: [ K ] , and hypothesis class H is set of mappings h :

Original: X → [ K ] .
Revised: X → [ K ] .

Original: We also assume that loss function ℓ is 0-1 loss defined as ℓ 0-1 ( h, ( x, y )) :
Revised: We also assume that loss function ℓ is 0-1 loss defined as ℓ 0-1 ( h, ( x, y )) :

Original: = 1 h ( x ) ̸
Revised: = 1 h ( x ) ̸

Original: = y , where 1 is an indicator function.
Revised: = y , where 1 is an indicator function.

Original: The following theorem shows a situation where ITV works well: a situation where the generalization error of ˆ h c does not exceed that of the other hypotheses in H c with high probability.
Revised: The following theorem shows a situation where ITV works well: a situation where the generalization error of ˆ h c does not exceed that of the other hypotheses in H c with high probability.

Original: Theorem 5.1.
Revised: Theorem 5.1.

Original: If Y =
Revised: If Y =

Original: [ K ] , and hypothesis class H is PAC-learnable with 0-1 loss ℓ 0-1 , training data S , and algorithm A , then suppose that ˆ h
Revised: [ K ] , and hypothesis class H is PAC-learnable with 0-1 loss ℓ 0-1 , training data S , and algorithm A , then suppose that ˆ h

Original: =
Revised: =

Original: A ( S ) is a hypothesis estimated form S satisfying
Revised: A ( S ) is a hypothesis estimated form S satisfying

Original: L D (ˆ h ) ≤ ϵ for some parameter ϵ ∈ (0 , 1) .
Revised: L D (ˆ h ) ≤ ϵ for some parameter ϵ ∈ (0 , 1) .

Original: Then for any requirement c , hypothesis ˆ h c obtained by modifying ˆ h with a CV satisfies
Revised: Then for any requirement c , hypothesis ˆ h c obtained by modifying ˆ h with a CV satisfies

Original: [Equation]
Revised: [Equation]

Original: We give a proof in Appendix A.
Revised: We give a proof in Appendix A.

Original: The proof bounds L D (ˆ h c ) using the fact that it is close to L D ( f c ) , where f c is obtained by modifying f : X → Y to satisfy L D ( f ) = 0 .
Revised: The proof bounds L D (ˆ h c ) using the fact that it is close to L D ( f c ) , where f c is obtained by modifying f : X → Y to satisfy L D ( f ) = 0 .

Original: The theorem suggests that if
Revised: The theorem suggests that if

Original: H is PAC-learnable, then inference time verification is sufficient to obtain a hypothesis with small generalization error in H c .
Revised: H is PAC-learnable, then inference time verification is sufficient to obtain a hypothesis with small generalization error in H c .

Original: Note that the generalization error might increase with a verifier, and the amount of the increase is always larger than L D ( f c ) .
Revised: Note that the generalization error might increase with a verifier, and the amount of the increase is always larger than L D ( f c ) .

Original: Therefore, L D ( f c ) represents the discrepancy between data distribution
Revised: Therefore, L D ( f c ) represents the discrepancy between data distribution

Original: D and requirement c , which is consistent with f if c ( x, f ( x ))
Revised: D and requirement c , which is consistent with f if c ( x, f ( x ))

Original: = 1 for all x .
Revised: = 1 for all x .

Original: If c is consistent with f , then L D ( f c ) = 0 , and we can certify that L D (ˆ h c ) ≤ ϵ .
Revised: If c is consistent with f , then L D ( f c ) = 0 , and we can certify that L D (ˆ h c ) ≤ ϵ .

Original: The above theorem shows that ITV works when H is PAC-learnable.
Revised: The above theorem shows that ITV works when H is PAC-learnable.

Original: However, this will not hold if D is not realizable with H , i.e, H is not PAC-learnable.
Revised: However, this will not hold if D is not realizable with H , i.e, H is not PAC-learnable.

Original: Theorem 5.2.
Revised: Theorem 5.2.

Original: If Y =
Revised: If Y =

Original: [ K ] , the loss function is 0-1 loss ℓ 0-1 and hypothesis class H is not realizable with D , and then there exists training data S , algorithm A , requirements c , and ϵ ∈ (0 , 1) such that ˆ h
Revised: [ K ] , the loss function is 0-1 loss ℓ 0-1 and hypothesis class H is not realizable with D , and then there exists training data S , algorithm A , requirements c , and ϵ ∈ (0 , 1) such that ˆ h

Original: = A ( S ) satisfies L D (ˆ h ) ≤
Revised: = A ( S ) satisfies L D (ˆ h ) ≤

Original: min h ∈H L D ( h ) + ε but L D (ˆ h c ) > min h c ∈H c L D ( h c )
Revised: min h ∈H L D ( h ) + ε but L D (ˆ h c ) > min h c ∈H c L D ( h c )

Original: + ϵ .
Revised: + ϵ .

Original: We give in Appendix B a proof that shows a counterexample even if c is consistent with ground truth f .
Revised: We give in Appendix B a proof that shows a counterexample even if c is consistent with ground truth f .

Original: The above theorems show that the realizability of H is the key factor that distinguishes among the cases where ITV works well.
Revised: The above theorems show that the realizability of H is the key factor that distinguishes among the cases where ITV works well.

Original: Moreover, unlike the realizable case, Theorem 5.2 holds even if requirement c is consistent with distribution D .
Revised: Moreover, unlike the realizable case, Theorem 5.2 holds even if requirement c is consistent with distribution D .

Original: Let f D : X → Y be defined as the Bayes optimal predictor:
Revised: Let f D : X → Y be defined as the Bayes optimal predictor:

Original: [Equation]
Revised: [Equation]

Original: The Bayes optimal predictor is optimal, in the sense that for every other classifier g : X → Y ,
Revised: The Bayes optimal predictor is optimal, in the sense that for every other classifier g : X → Y ,

Original: L D ( f D ) ≤ L D ( g ) .
Revised: L D ( f D ) ≤ L D ( g ) .

Original: Theorem 5.2 holds if c is consistent with f D .
Revised: Theorem 5.2 holds if c is consistent with f D .

Original: These results show that existing methods [9, 2] using constraints only in the inference time might fail to select the best hypothesis.
Revised: These results show that existing methods [9, 2] using constraints only in the inference time might fail to select the best hypothesis.

Original: Running time analysis: Using a CV increases the time needed for inference.
Revised: Running time analysis: Using a CV increases the time needed for inference.

Original: Suppose that a verifier is an oracle that can answer the query about the value of c ( x, y ) .
Revised: Suppose that a verifier is an oracle that can answer the query about the value of c ( x, y ) .

Original: To achieve a previously shown modification procedure (3), we need at most K queries.
Revised: To achieve a previously shown modification procedure (3), we need at most K queries.

Original:  Learning Time Verification
Revised:  Learning Time Verification

Original: In Section 5, we show that if H is PAC-learnable with 0-1 loss, then modifying a hypothesis at the inference time is sufficient to obtain a hypothesis with the smallest generalization error while satisfying the requirements.
Revised: In Section 5, we show that if H is PAC-learnable with 0-1 loss, then modifying a hypothesis at the inference time is sufficient to obtain a hypothesis with the smallest generalization error while satisfying the requirements.

Original: If H is not PAC-learnable, then the ITV scheme might fail to obtain a hypothesis with small generalization error.
Revised: If H is not PAC-learnable, then the ITV scheme might fail to obtain a hypothesis with small generalization error.

Original: Here we show that the generalization error can be bounded when we use a CV in the learning phase.
Revised: Here we show that the generalization error can be bounded when we use a CV in the learning phase.

Original: We call this setting learning time verification (LTV).
Revised: We call this setting learning time verification (LTV).

Original: Since the LTV scheme corresponds to a learning task where the hypothesis class is H c , we analyze the learnability of H c using the standard tools for generalization analyses.
Revised: Since the LTV scheme corresponds to a learning task where the hypothesis class is H c , we analyze the learnability of H c using the standard tools for generalization analyses.

Original: This paper provides analyses based on Rademacher complexity since its a widely used tools that can give tight bounds for both data-dependent and data-independent cases.
Revised: This paper provides analyses based on Rademacher complexity since its a widely used tools that can give tight bounds for both data-dependent and data-independent cases.

Original: Moreover, some previous work gives bounds of structured prediction tasks using Rademacher complexity.
Revised: Moreover, some previous work gives bounds of structured prediction tasks using Rademacher complexity.

Original: In the literature, constraints are actively used in structured prediction tasks, including language generation and sequence labeling.
Revised: In the literature, constraints are actively used in structured prediction tasks, including language generation and sequence labeling.

Original: Therefore, analyzing the generalization error is important when using a CV on structured prediction tasks.
Revised: Therefore, analyzing the generalization error is important when using a CV on structured prediction tasks.

Original: In the following, we first show the upper bounds of generation error based on the Rademacher complexity of H c in a multi-class classification task (§6.1, 6.2) and a structured prediction setting (§6.3).
Revised: In the following, we first show the upper bounds of generation error based on the Rademacher complexity of H c in a multi-class classification task (§6.1, 6.2) and a structured prediction setting (§6.3).

Original: Our main finding is that the upper bounds based on the Rademacher complexity of H c are always less than or equal to those of H .
Revised: Our main finding is that the upper bounds based on the Rademacher complexity of H c are always less than or equal to those of H .

Original: Therefore, adding a CV to a machine learning model will not degrade its learnability.
Revised: Therefore, adding a CV to a machine learning model will not degrade its learnability.

Original: 6.1 Multi-class Classification
Revised: 6.1 Multi-class Classification

Original: We first give the Rademacher complexity-based error bounds on a multi-class classification task, i.e., Y =
Revised: We first give the Rademacher complexity-based error bounds on a multi-class classification task, i.e., Y =

Original: [ K ] .
Revised: [ K ] .

Original: In this section, we show that a standard upper bound [18] based on the Rademacher complexity of H can be used as an upper bound of H c for any requirement c .
Revised: In this section, we show that a standard upper bound [16] based on the Rademacher complexity of H can be used as an upper bound of H c for any requirement c .

Original: In the next section, we show that a state-of-the-art error bound, based on local Rademacher complexity H , can also be used as an upper bound of H c .
Revised: In the next section, we show that a state-of-the-art error bound, based on local Rademacher complexity H , can also be used as an upper bound of H c .

Original: Following previous works, let h : ( X × Y )
Revised: Following previous works, let h : ( X × Y )

Original: → R be a scoring function, and define hypothesis class
Revised: → R be a scoring function, and define hypothesis class

Original: H as a set of scoring functions.
Revised: H as a set of scoring functions.

Original: A scoring function defines a mapping from X to Y :
Revised: A scoring function defines a mapping from X to Y :

Original: [Equation]
Revised: [Equation]

Original: Let ρ h ( x, y ) be the margin of function of h :
Revised: Let ρ h ( x, y ) be the margin of function of h :

Original: [Equation]
Revised: [Equation]

Original: Hypothesis h misclassifies the labeled example ( x, y ) if ρ h ( x, y ) ≤ 0 .
Revised: Hypothesis h misclassifies the labeled example ( x, y ) if ρ h ( x, y ) ≤ 0 .

Original: Thus, by using a margin function, the 0-1 loss can be represented as ℓ 0-1 ( h, z ) = 1 ρ h ( x,y ) ≤ 0 .
Revised: Thus, by using a margin function, the 0-1 loss can be represented as ℓ 0-1 ( h, z ) = 1 ρ h ( x,y ) ≤ 0 .

Original: Since 0-1 loss is hard to handle during learning, we use margin loss ℓ ρ ( h, ( x, y ))
Revised: Since 0-1 loss is hard to handle during learning, we use margin loss ℓ ρ ( h, ( x, y ))

Original: =
Revised: =

Original: Φ ρ ( ρ h ( x, y )) , where Φ ρ ( t ) is defined as
Revised: Φ ρ ( ρ h ( x, y )) , where Φ ρ ( t ) is defined as

Original: [Equation]
Revised: [Equation]

Original: Function f : R → R is said to be µ -Lipschitz if | f ( t ) − f ( t ′ )
Revised: Function f : R → R is said to be µ -Lipschitz if | f ( t ) − f ( t ′ )

Original: | ≤ µ | t − t ′ | for any t, t ′ ∈ R .
Revised: | ≤ µ | t − t ′ | for any t, t ′ ∈ R .

Original: Φ ρ is an 1 /ρ -Lipschitz function.
Revised: Φ ρ is an 1 /ρ -Lipschitz function.

Original: The empirical margin loss of hypothesis h is defined as
Revised: The empirical margin loss of hypothesis h is defined as

Original: [Equation]
Revised: [Equation]

Original: Identical to the case of ITV, introducing a CV to a machine learning model corresponds to modifying its corresponding hypothesis class H to hypothesis class H c that is consistent with requirement c .
Revised: Identical to the case of ITV, introducing a CV to a machine learning model corresponds to modifying its corresponding hypothesis class H to hypothesis class H c that is consistent with requirement c .

Original: If h is a score function, then we define consistent function h c :
Revised: If h is a score function, then we define consistent function h c :

Original: [Equation]
Revised: [Equation]

Original: where M is a positive constant satisfying M > | max ( x
Revised: where M is a positive constant satisfying M > | max ( x

Original: ,y ) ∈
Revised: ,y ) ∈

Original: Z h ( x, y ) | .
Revised: Z h ( x, y ) | .

Original: As described in Section 4, we assume that there exists y ∈ Y that satisfies c ( x, y )
Revised: As described in Section 4, we assume that there exists y ∈ Y that satisfies c ( x, y )

Original: =
Revised: =

Original: 1 for all x ∈ X .
Revised: 1 for all x ∈ X .

Original: Therefore, we can guarantee that ρ h c ( x, y ) < 0 if c ( x, y )
Revised: Therefore, we can guarantee that ρ h c ( x, y ) < 0 if c ( x, y )

Original: = 0 .
Revised: = 0 .

Original: The following are the main results of the general multi-class learning problem, which is based on the margin bound shown in Theorem 9.2 of Mohri et al.
Revised: The following are the main results of the general multi-class learning problem, which is based on the margin bound shown in Theorem 9.2 of Mohri et al.

Original: Our main finding is that the generalization error of any hypothesis, h c , is bounded by the Rademacher complexity of hypothesis set H , which suggests that if we have a tight bound for hypothesis class H , then we can expect to find a good hypothesis from H c under any requirements c .
Revised: Our main finding is that the generalization error of any hypothesis, h c , is bounded by the Rademacher complexity of hypothesis set H , which suggests that if we have a tight bound for hypothesis class H , then we can expect to find a good hypothesis from H c under any requirements c .

Original: Theorem 6.1.
Revised: Theorem 6.1.

Original: Let H ⊆
Revised: Let H ⊆

Original: R X×Y be a hypothesis class with Y =
Revised: R X×Y be a hypothesis class with Y =

Original: [ K ] , and let c be a requirement.
Revised: [ K ] , and let c be a requirement.

Original: Fix ρ > 0 .
Revised: Fix ρ > 0 .

Original: Then for any δ > 0 , with probability at least 1 − δ , the following bound holds for all h c ∈ H c :
Revised: Then for any δ > 0 , with probability at least 1 − δ , the following bound holds for all h c ∈ H c :

Original: [Equation]
Revised: [Equation]

Original: where Π 1 ( H ) is defined as
Revised: where Π 1 ( H ) is defined as

Original: [Equation]
Revised: [Equation]

Original: We give a proof in Appendix C. We obtain the results by showing that the upper bounds of the
Revised: We give a proof in Appendix C. We obtain the results by showing that the upper bounds of the

Original: Rademacher complexity of H c are bounded by some upper bounds of the Rademacher complexity of H .
Revised: Rademacher complexity of H c are bounded by some upper bounds of the Rademacher complexity of H .

Original: All the proofs of the theorems in this section use similar techniques.
Revised: All the proofs of the theorems in this section use similar techniques.

Original: Parameter ρ sets the margin value.
Revised: Parameter ρ sets the margin value.

Original: Following a previously shown technique [18], we obtain a generalized bound that holds uniformly for all ρ > 0 .
Revised: Following a previously shown technique [16], we obtain a generalized bound that holds uniformly for all ρ > 0 .

Original: The above theorem suggests that using a CV at a learning phase does not worsen the error bound for any requirement c .
Revised: The above theorem suggests that using a CV at a learning phase does not worsen the error bound for any requirement c .

Original: Intuitively, the theorem seems reasonable since requirements c imposes a restriction on H , and thus the complexity of H c is not larger than H .
Revised: Intuitively, the theorem seems reasonable since requirements c imposes a restriction on H , and thus the complexity of H c is not larger than H .

Original: However, it is not so trivial since H c ⊆ H is not always true.
Revised: However, it is not so trivial since H c ⊆ H is not always true.

Original: Running time analysis: We analyze the number of evaluations c ( x, y ) required for learning with a CV.
Revised: Running time analysis: We analyze the number of evaluations c ( x, y ) required for learning with a CV.

Original: Let S 1 be a sub-sequence of training example S such that c ( x i , y i )
Revised: Let S 1 be a sub-sequence of training example S such that c ( x i , y i )

Original: = 1 , and let S 0 be a sub-sequence such that c ( x i , y i )
Revised: = 1 , and let S 0 be a sub-sequence such that c ( x i , y i )

Original: = 0 .
Revised: = 0 .

Original: If we use a 0-1 loss function, then the empirical loss of hypothesis h c is
Revised: If we use a 0-1 loss function, then the empirical loss of hypothesis h c is

Original: [Equation]
Revised: [Equation]

Original: since every h c misclassifies the examples in S 0 .
Revised: since every h c misclassifies the examples in S 0 .

Original: Therefore, we need at most K | S 1 | + | S 0 | queries for the learning process.
Revised: Therefore, we need at most K | S 1 | + | S 0 | queries for the learning process.

Original: This is also true when we use a margin loss function.
Revised: This is also true when we use a margin loss function.

Original: On the other hand, the problem of estimating the best hypothesis might be more difficult than the original problem depending on requirement c .
Revised: On the other hand, the problem of estimating the best hypothesis might be more difficult than the original problem depending on requirement c .

Original: 6.2 Tighter Bound Based on Local Rademacher Complexity
Revised: 6.2 Tighter Bound Based on Local Rademacher Complexity

Original: The bound for H c shown in the previous section is relatively simple, and tighter bounds of H based on the Rademacher complexity have been developed in the literature.
Revised: The bound for H c shown in the previous section is relatively simple, and tighter bounds of H based on the Rademacher complexity have been developed in the literature.

Original: In this section, we show that the state-of-the-art error bound for H based on the local Rademacher complexity can be used as a bound for H c for any requirements c .
Revised: In this section, we show that the state-of-the-art error bound for H based on the local Rademacher complexity can be used as a bound for H c for any requirements c .

Original: Definition 6.2.
Revised: Definition 6.2.

Original: Let G be a family of functions from Z to R , and let S be training data of size m.
Revised: Let G be a family of functions from Z to R , and let S be training data of size m.

Original: Then for any r > 0 , the empirical local Rademacher complexity of G is defined as
Revised: Then for any r > 0 , the empirical local Rademacher complexity of G is defined as

Original: [Equation]
Revised: [Equation]

Original: Li et al.
Revised: Li et al.

Original: [15] showed a tighter generalization bound for a multi-class classification problem using the local Rademacher complexity when the hypothesis class is a ℓ p norm hypothesis space with kernel κ , defined as
Revised: [13] showed a tighter generalization bound for a multi-class classification problem using the local Rademacher complexity when the hypothesis class is a ℓ p norm hypothesis space with kernel κ , defined as

Original: [Equation]
Revised: [Equation]

Original: where h is represented as a vector valued function ( h 1 , .
Revised: where h is represented as a vector valued function ( h 1 , .

Original: . . , h K ) with h j ( x )
Revised: . . , h K ) with h j ( x )

Original: = h ( x, j ) ,
Revised: = h ( x, j ) ,

Original: ∀ j = 1 , . .
Revised: ∀ j = 1 , . .

Original: , K , and κ : X × X
Revised: , K , and κ : X × X

Original: → R is a Mercer kernel with associated feature map ϕ , i.e., κ ( x, x ′ )
Revised: → R is a Mercer kernel with associated feature map ϕ , i.e., κ ( x, x ′ )

Original: = ⟨ ϕ ( x ) , ϕ ( x ′ ) ⟩ .
Revised: = ⟨ ϕ ( x ) , ϕ ( x ′ ) ⟩ .

Original: w = ( w 1 , . . . , w K ) , and
Revised: w = ( w 1 , . . . , w K ) , and

Original: ∥ w ∥ = (cid:104)(cid:80) Ki =1 ∥ w ∥ p 2 (cid:105) 1 p is the ℓ 2 ,p -norm.
Revised: ∥ w ∥ = (cid:104)(cid:80) Ki =1 ∥ w ∥ p 2 (cid:105) 1 p is the ℓ 2 ,p -norm.

Original: For any p ≥ 1 , let q be the dual exponent of p satisfying 1 /p
Revised: For any p ≥ 1 , let q be the dual exponent of p satisfying 1 /p

Original: + 1 /q
Revised: + 1 /q

Original: = 1 .
Revised: = 1 .

Original: Let Φ : R → R be a loss function satisfying the following:
Revised: Let Φ : R → R be a loss function satisfying the following:

Original: [Equation]
Revised: [Equation]

Original: ) ≤ Φ
Revised: ) ≤ Φ

Original: ( t
Revised: ( t

Original: ) for all
Revised: ) for all

Original: t
Revised: t

Original: ; 2) Φ
Revised: ; 2) Φ

Original: ( t )
Revised: ( t )

Original: is decreasing and has zero point c Φ ; 3) Φ is ζ -smooth, that is, | Φ ′
Revised: is decreasing and has zero point c Φ ; 3) Φ is ζ -smooth, that is, | Φ ′

Original: ( t
Revised: ( t

Original: −
Revised: −

Original: Φ ′
Revised: Φ ′

Original: ( t
Revised: ( t

Original: | ≤ ζ | t − t ′ | .
Revised: | ≤ ζ | t − t ′ | .

Original: Let H p,κ,c be the hypothesis class obtained by modifying hypothesis H p,κ to satisfy requirements c , and L c := { ( x, y )
Revised: Let H p,κ,c be the hypothesis class obtained by modifying hypothesis H p,κ to satisfy requirements c , and L c := { ( x, y )

Original: (cid:55)→ Φ( ρ h c ( x, y )) : h c ∈ H p,κ,c } .
Revised: (cid:55)→ Φ( ρ h c ( x, y )) : h c ∈ H p,κ,c } .

Original: The following theorem gives a bound of the local
Revised: The following theorem gives a bound of the local

Original: Rademacher complexity of L c .
Revised: Rademacher complexity of L c .

Original: [Figure]
Revised: [Figure]

Original: Theorem 6.3.
Revised: Theorem 6.3.

Original: Let H p,κ,c be the set of hypotheses obtained by modifying
Revised: Let H p,κ,c be the set of hypotheses obtained by modifying

Original: hypothesis h ∈ H p,κ with requirement c .
Revised: hypothesis h ∈ H p,κ with requirement c .

Original: For any δ > 0 , with probability at least 1 − δ , the following bound holds:
Revised: For any δ > 0 , with probability at least 1 − δ , the following bound holds:

Original: [Equation]
Revised: [Equation]

Original: where ϑ = sup x ∈X κ ( x, x ) < ∞ , d = sup t ∈ R Φ( t ) < ∞ , and C d,ϑ is a constant.
Revised: where ϑ = sup x ∈X κ ( x, x ) < ∞ , d = sup t ∈ R Φ( t ) < ∞ , and C d,ϑ is a constant.

Original: ξ ( K ) is
Revised: ξ ( K ) is

Original: [Equation]
Revised: [Equation]

Original: We give a proof in Appendix D.
Revised: We give a proof in Appendix D.

Original: The bound equals that of R m
Revised: The bound equals that of R m

Original: ( L ; r ) (Theorem 1 of [15]) for any requirement c and any hypothesis H ρ,κ .
Revised: ( L ; r ) (Theorem 1 of [13]) for any requirement c and any hypothesis H ρ,κ .

Original: Therefore, the generalization error bounds based on Theorem 1 of Li et al.
Revised: Therefore, the generalization error bounds based on Theorem 1 of Li et al.

Original: [15] holds for any requirement c .
Revised: [13] holds for any requirement c .

Original: 6.3 Analyses of Structured Prediction
Revised: 6.3 Analyses of Structured Prediction

Original: Structured prediction is a kind of multi-class classification task, where label set Y might be a set of sequences, images, graphs, trees, or other objects admitting some possibly overlapping structure.
Revised: Structured prediction is a kind of multi-class classification task, where label set Y might be a set of sequences, images, graphs, trees, or other objects admitting some possibly overlapping structure.

Original: As mentioned in Section 1, previous works try to impose constraints on the output of structured prediction tasks.
Revised: As mentioned in Section 1, previous works try to impose constraints on the output of structured prediction tasks.

Original: Thus it is also useful to derive error bounds for structured prediction tasks when we use a CV.
Revised: Thus it is also useful to derive error bounds for structured prediction tasks when we use a CV.

Original: In the following, we show that the Rademacher complexity-based generalization error bounds derived in a seminal work of Cortes et al.
Revised: In the following, we show that the Rademacher complexity-based generalization error bounds derived in a seminal work of Cortes et al.

Original: [6] also hold if we use a CV.
Revised: [6] also hold if we use a CV.

Original: Although tighter bounds are given in a more recent work [19, 16], we give bounds based on Cortes et al.
Revised: Although tighter bounds are given in a more recent work [17, 14], we give bounds based on Cortes et al.

Original: [6] due to their simplicity.
Revised: [6] due to their simplicity.

Original: We give some definitions for the structured prediction task.
Revised: We give some definitions for the structured prediction task.

Original: Following previous work, we assume that
Revised: Following previous work, we assume that

Original: Y is decomposable along with substructures:
Revised: Y is decomposable along with substructures:

Original: Y =
Revised: Y =

Original: Y 1 × · · · × Y l .
Revised: Y 1 × · · · × Y l .

Original: Here Y k is a set of possible labels that can be assigned to the k -th substructure.
Revised: Here Y k is a set of possible labels that can be assigned to the k -th substructure.

Original: We denote by L : Y × Y → R + a loss function that measures the dissimilarity of two elements of output space Y .
Revised: We denote by L : Y × Y → R + a loss function that measures the dissimilarity of two elements of output space Y .

Original: L is definite , that is, L ( y, y ′ )
Revised: L is definite , that is, L ( y, y ′ )

Original: = 

Revised: =

Original: iff y = y ′ .
Revised: iff y = y ′ .

Original: A typical definite loss function for a structured prediction task is the Hamming loss defined by L ( y, y ′ )
Revised: A typical definite loss function for a structured prediction task is the Hamming loss defined by L ( y, y ′ )

Original: = 1 l (cid:80) l k =1 1 y k ̸ = y ′ k
Revised: = 1 l (cid:80) l k =1 1 y k ̸ = y ′ k

Original: for all y =
Revised: for all y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y l ) and y ′ =
Revised: , y l ) and y ′ =

Original: ( y ′ 1 , . . .
Revised: ( y ′ 1 , . . .

Original: , y ′ l ) , with y k ,
Revised: , y ′ l ) , with y k ,

Original: y ′ k ∈
Revised: y ′ k ∈

Original: Y k .
Revised: Y k .

Original: Other typical examples of loss functions can be seen in Cortes et al.
Revised: Other typical examples of loss functions can be seen in Cortes et al.

Original: Using loss function L , the generalization and empirical error of h are defined:
Revised: Using loss function L , the generalization and empirical error of h are defined:

Original: [Equation]
Revised: [Equation]

Original: As with the multi-class classification task, hypothesis class H can be represented as a set of scoring function h : X × Y → R .
Revised: As with the multi-class classification task, hypothesis class H can be represented as a set of scoring function h : X × Y → R .

Original: We use h ( x ) to represent the predictor defined by h ∈ H : h ( x ) :=
Revised: We use h ( x ) to represent the predictor defined by h ∈ H : h ( x ) :=

Original: argmax y ∈Y h ( x, y ) for all x ∈ X .
Revised: argmax y ∈Y h ( x, y ) for all x ∈ X .

Original: Following the previous work, we assume that each scoring function can be decomposed as a sum, and such decomposition follows a factor graph .
Revised: Following the previous work, we assume that each scoring function can be decomposed as a sum, and such decomposition follows a factor graph .

Original: Factor graph G is a tuple G =
Revised: Factor graph G is a tuple G =

Original: ( V, F, E ) , where V is a set of variable nodes, F is a set of factor nodes, and E is a set of undirected edges between a variable node and a factor node.
Revised: ( V, F, E ) , where V is a set of variable nodes, F is a set of factor nodes, and E is a set of undirected edges between a variable node and a factor node.

Original: Every node in V corresponds to a substructure index, where V = { 1 , . . .
Revised: Every node in V corresponds to a substructure index, where V = { 1 , . . .

Original: , l } .
Revised: , l } .

Original: For any factor node f , we denote by N ( f ) ⊆
Revised: For any factor node f , we denote by N ( f ) ⊆

Original: V a set of variable nodes connected to f and define Y f as substructure set cross-product Y f
Revised: V a set of variable nodes connected to f and define Y f as substructure set cross-product Y f

Original: = (cid:81) k ∈N ( f ) Y k .
Revised: = (cid:81) k ∈N ( f ) Y k .

Original: Then h admits the following decomposition as a sum of functions
Revised: Then h admits the following decomposition as a sum of functions

Original: h f , each taking as an argument a pair of ( x, y f ) ∈
Revised: h f , each taking as an argument a pair of ( x, y f ) ∈

Original: X × Y f :
Revised: X × Y f :

Original: [Equation]
Revised: [Equation]

Original: Figure 2 shows examples of decompositions based on factor graphs.
Revised: Figure 2 shows examples of decompositions based on factor graphs.

Original: We conventionally assume that the structure of the factor graphs may change depending on a particular example ( x i , y i ) :
Revised: We conventionally assume that the structure of the factor graphs may change depending on a particular example ( x i , y i ) :

Original: G ( x i , y i )
Revised: G ( x i , y i )

Original: =
Revised: =

Original: G i =
Revised: G i =

Original: ([ l i ] , F i , E i ) .
Revised: ([ l i ] , F i , E i ) .

Original: A special case of this setting is when size l i of each example is allowed to vary.
Revised: A special case of this setting is when size l i of each example is allowed to vary.

Original: In such a case, the number of possible labels Y is potentially infinite.
Revised: In such a case, the number of possible labels Y is potentially infinite.

Original: Following multi-class classification, our CV maps hypothesis h to h c to satisfy the requirements.
Revised: Following multi-class classification, our CV maps hypothesis h to h c to satisfy the requirements.

Original: The definition of h c follows Eq.
Revised: The definition of h c follows Eq.

Original: This definition does not require h c to have a factored representation.
Revised: This definition does not require h c to have a factored representation.

Original: For analyzing the complexity, Cortes et al.
Revised: For analyzing the complexity, Cortes et al.

Original: [6] introduced
Revised: [6] introduced

Original: empirical factor graph Rademacher complexity R GS ( H ) of hypothesis class H for S =
Revised: empirical factor graph Rademacher complexity R GS ( H ) of hypothesis class H for S =

Original: ( x 1 , . . .
Revised: ( x 1 , . . .

Original: , x m ) and factor graph G :
Revised: , x m ) and factor graph G :

Original: [Equation]
Revised: [Equation]

Original: where ϵ =
Revised: where ϵ =

Original: ( ϵ i,f,y ) i
Revised: ( ϵ i,f,y ) i

Original: ∈
Revised: ∈

Original: [ m ] ,f ∈ F i ,y ∈Y f and every ϵ i,f,y is i.i.d.
Revised: [ m ] ,f ∈ F i ,y ∈Y f and every ϵ i,f,y is i.i.d.

Original: a Rademacher random variable.
Revised: a Rademacher random variable.

Original: Factor graph Rademacher complexity of H for factor graph G is defined as expectation
Revised: Factor graph Rademacher complexity of H for factor graph G is defined as expectation

Original: [Equation]
Revised: [Equation]

Original: By using the factor graph Rademacher complexity, Cortes et al.
Revised: By using the factor graph Rademacher complexity, Cortes et al.

Original: [6] gives bounds for a structured prediction task with the following additive and multiplicative empirical losses:
Revised: [6] gives bounds for a structured prediction task with the following additive and multiplicative empirical losses:

Original: [Equation]
Revised: [Equation]

Original: where Φ ∗ ( t ) = min( B, max(0 , t )) for all t , with B = max y,y ′
Revised: where Φ ∗ ( t ) = min( B, max(0 , t )) for all t , with B = max y,y ′

Original: L ( y, y ′ ) .
Revised: L ( y, y ′ ) .

Original: As shown in [6], these loss functions cover typical surrogate loss functions used in structured prediction tasks.
Revised: As shown in [6], these loss functions cover typical surrogate loss functions used in structured prediction tasks.

Original: We show the following bound for structured predictions.
Revised: We show the following bound for structured predictions.

Original: Theorem 6.4.
Revised: Theorem 6.4.

Original: Fix ρ > 0 .
Revised: Fix ρ > 0 .

Original: For any δ > 0 and requirement c , with probability at least 1 − δ over the draw of sample S of size m from distribution D , the following holds for all h c ∈ H c :
Revised: For any δ > 0 and requirement c , with probability at least 1 − δ over the draw of sample S of size m from distribution D , the following holds for all h c ∈ H c :

Original: [Equation]
Revised: [Equation]

Original: We give a proof in Appendix E. ρ is a parameter that determines the margin.
Revised: We give a proof in Appendix E. ρ is a parameter that determines the margin.

Original: Similar to the case of multi-class classification, we can derive a bound that holds for any ρ > 0 following a previous derivation [6].
Revised: Similar to the case of multi-class classification, we can derive a bound that holds for any ρ > 0 following a previous derivation [6].

Original: The above result indicates that the bound will not change if we use a CV for any requirement c .
Revised: The above result indicates that the bound will not change if we use a CV for any requirement c .

Original: This is interesting since the above result holds even if we do not have a factored representation of h c ( x, y ) , similar to Eq.
Revised: This is interesting since the above result holds even if we do not have a factored representation of h c ( x, y ) , similar to Eq.

Original: (5), although the derived bound depends on the factor graph
Revised: (5), although the derived bound depends on the factor graph

Original: Rademacher complexity, which depends on the factored representation of h ( x, y ) .
Revised: Rademacher complexity, which depends on the factored representation of h ( x, y ) .

Original: We analyzed the overhead of the running time for evaluating loss function L add S,ρ ( h c ) and L mult S,ρ ( h c ) for hypothesis h c .
Revised: We analyzed the overhead of the running time for evaluating loss function L add S,ρ ( h c ) and L mult S,ρ ( h c ) for hypothesis h c .

Original: Different from the multi-class classification case, both the number of queries and the overhead of the running time for the loss evaluation when we use a CV depend on the model and the type of requirements for structured predictions.
Revised: Different from the multi-class classification case, both the number of queries and the overhead of the running time for the loss evaluation when we use a CV depend on the model and the type of requirements for structured predictions.

Original: This result is consistent with the literature, which reports that for structured prediction tasks, original tractable optimization problems can be intractable if we put additional constraints [23].
Revised: This result is consistent with the literature, which reports that for structured prediction tasks, original tractable optimization problems can be intractable if we put additional constraints [21].

Original:  Conclusion
Revised:  Conclusion

Original: This paper gives a generalization analysis when there are requirements that the input-output pairs of a machine learning model must satisfy.
Revised: This paper gives a generalization analysis when there are requirements that the input-output pairs of a machine learning model must satisfy.

Original: We introduce a concurrent verifier, a simple module that enables us to guarantee that the input-output pairs of a machine learning model satisfy the requirements.
Revised: We introduce a concurrent verifier, a simple module that enables us to guarantee that the input-output pairs of a machine learning model satisfy the requirements.

Original: We show a situation where we can obtain a hypothesis with small error when we use a verifier only in the inference phase.
Revised: We show a situation where we can obtain a hypothesis with small error when we use a verifier only in the inference phase.

Original: Interestingly, if H is not PAC-learnable, we might fail to obtain a guaranteed hypothesis even if the requirements are consistent with distribution D .
Revised: Interestingly, if H is not PAC-learnable, we might fail to obtain a guaranteed hypothesis even if the requirements are consistent with distribution D .

Original: We also give the generalization bounds based on Rademacher complexity when we use a verifier in a learning phase and find that the obtained bounds are less than or equal to the existing ones, independent of the machine learning model and the type of requirements
Revised: We also give the generalization bounds based on Rademacher complexity when we use a verifier in a learning phase and find that the obtained bounds are less than or equal to the existing ones, independent of the machine learning model and the type of requirements.

Original: Acknowledgements
Revised: Acknowledgements

Original: The authors thank the anonymous reviewers for their valuable feedback, corrections and suggestions.
Revised: The authors thank the anonymous reviewers for their valuable feedback, corrections and suggestions.

Original: This work was supported by JST PRESTO (Grant Number JPMJPR20C7,  Japan) and JSPS KAKENHI (Grant Number JP20H05963, Japan)
Revised: This work was supported by JST PRESTO (Grant Number JPMJPR20C7, Japan), JST CREST (Grant Number JPMJCR22D3, Japan) and JSPS KAKENHI (Grant Number JP20H05963, Japan)

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] • Did you include the license to the code and datasets?
Revised: [Yes] • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.
Revised: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] We discuss conditions a verifier does not work well in Section 5.
Revised: [Yes] We discuss conditions a verifier does not work well in Section 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] We give proof in Appendix 3.
Revised: [Yes] We give proof in Appendix 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [N/A] 4.
Revised: [N/A] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A] (b) Did you mention the license of the assets?
Revised: [N/A] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]

Original: A Proof of Theorem 5.

Original: Proof.

Original: Since D is realizable, f : X → Y exists such that L D ( f ) = 0 .

Original: We show that L D ( f c ) ≤

Original: L D (ˆ h c ) ≤ L D ( f c ) + ϵ .

Original: We first prove L D ( f c )

Original: ≤ L D (ˆ h c ) .

Original: From the definition of f c , if f ( x ) ̸ = f c ( x ) , then c ( x, f ( x ))

Original: = 0 , and thus h c ( x ) ̸

Original: = f ( x ) for all h c ∈ H c .

Original: Therefore, for all h c ∈ H c ,

Original: L D ( f c ) = P [ f c ( x ) ̸ = f ( x )] ≤ P [ h c ( x ) ̸ = f ( x )] =

Original: L D ( h c ) .

Original: Next we prove L D (ˆ h c )

Original: ≤ L D ( f c ) + ϵ .

Original: For all h ∈ H , we show that L D ( h c ) − L D ( h ) is bounded:

Original: L D ( h c ) − L D ( h ) ≤ P [ h c ( x )

Original: ̸ = h ( x ) and h ( x ) = f

Original: ( x )] ≤ P [ f c ( x ) ̸ = f ( x )]

Original: =

Original: L D ( f c ) , where the first inequality uses the fact that the error increases if we modify the output at x and satisfy f ( x ) = h ( x ) .

Original: [-Since c ( x, f ( x )) = 
-]
Revised: [-Since c ( x, f ( x )) = 
-]

Original: for such x , the probability is less than error L D ( f c ) .

Original: Thus,

Original: L D (ˆ h c ) ≤ L D (ˆ h ) + L D ( f c ) ≤ L D ( f c ) + ϵ .

Original: B Proof of Theorem 5.

Original: Proof.

Original: Suppose that H = { h 0 , h 1 } , and f ̸∈ H exists, satisfying L D ( f ) = 0 .

Original: Suppose partition

Original: X 0 , X 1 of X exists such that h 0 ( x ) ̸ = f ( x ) iff x ∈ X 0 and h 1 ( x ) ̸ = f ( x ) iff x ∈ X 1 .

Original: Suppose that | L D ( h 0 ) − L D ( h 1 ) | ≤ ϵ for some ϵ ∈ (0 , 1) .

Original: [-If we design c such that c ( x, y ) = 
-]
Revised: [-If we design c such that c ( x, y ) = 
-]

Original: iff x ∈ Xand y

Original: = h 0 ( x ) , otherwise c ( x, y )

Original: = 1 .

Original: Then the generalization error of modified hypothesis h c 1 becomes L D ( h c 0 )

Original: = 0 , and L D ( h c 1 )

Original: =

Original: L D ( h 1 ) .

Original: Thus if L D ( h 1 ) > ϵ , and then difference | L D ( h c 0 ) − L D ( h c 1 ) | becomes larger than ϵ .

Original: Note that the above proof holds for c , which is consistent with f ; that is, c ( x, f ( x ))

Original: = 1 for all x ∈ X .

Original: C Proof of Theorem 6.

Original: We first introduce Talagrand’s lemma with which we prove the main theorem.

Original: Lemma C.1 (Talagrand’s lemma, [13, 18] ) .

Original: Let Φ be the µ -Lipschitz function from R to R , and let σ 1 , . . .

Original: , σ m be Rademacher random variables.

Original: Then for any hypothesis set H of real-valued functions, the following inequality holds:

Original: [Equation]

Original: We also use the following lemma.

Original: Lemma C.2 (Lemma 9.1 of Mohri et al.

Original: Let F 1 , . .

Original: . , F l be l hypothesis sets in R X , l ≥ 1 , and let G = { max { h 1 , . .

Original: , h l } : h i ∈ F i , i ∈

Original: [ l ] } .

Original: Then for any training data S of size m , the empirical

Original: Rademacher complexity of G can be upper bounded:

Original: [Equation]

Original: We use max { h 1 , . . .

Original: , h l } to represent the mapping from X to R that maps x ∈ X to max { h 1 ( x ) , . . .

Original: , h l ( x ) } .

Original: The above inequality holds if we use min instead of max in the definition of G .

Original: The following lemma shows the relationship between the Rademacher complexities of H and H c .

Original: Lemma C.3.

Original: Let H be a hypothesis set in R X×Y , and let c : X × Y → { 0 , 1 } be the requirements.

Original: Then for any training data S =

Original: (( x 1 , y 1 ) , . .

Original: , ( x m , y m )) of size m , the following inequality holds:

Original: [Equation]

Original: Proof.

Original: Let b 0 ( x, y ) :

Original: X × Y → R be the following function:

Original: [Equation]

Original: [-( x, y ) = (cid:26) M if c ( x, y ) =− M if c ( x, y ) = 0 . Let B 
-]
Revised: [-( x, y ) = (cid:26) M if c ( x, y ) =− M if c ( x, y ) = 0 . Let B 
-]If you used crowdsourcing

Original: = { b 0 } .

Original: Then H c can be represented as H c = { min { h, b 0 } : h ∈ H , b 0 ∈ B 0 } .

Original: From

Original: Lemma C.2,

Original: [Equation]

Original: Since the empirical Rademacher complexity of singleton hypothesis class B 0 is zero for any S ,

Original: R S ( H c ) ≤ R S

Original: ( H ) holds.

Original: Proof of Theorem 6.1.

Original: Let us define two sets of mappings, H c 0 and H c 1 :

Original: [Equation]

Original: where we define ρ θ,h ( x, y ) :

Original: [Equation]

Original: where θ > 0 is an arbitrary constant.

Original: ρ θ,h satisfies E [ 1 ρ h ( x,y ) ≤ 0 ] ≤ E [ 1 ρ θ,h ( x,y ) ≤ 0 ] since ρ θ,h ( x, y ) ≤

Original: ρ h ( x, y ) holds for all ( x, y ) ∈

Original: X × Y .

Original: Following the proof of Theorem 9.2 of

Original: Mohri et al.

Original: [18] with a probability of at least 1 − δ , for all h c ∈ H c :

Original: [Equation]

Original: Thus, to complete the proof it suffices to show R m ( H c 0 ) ≤ 2 KR m

Original: (Π 1 ( H )) .

Original: We can upper bound

Original: R m ( H c 0 ) :

Original: [Equation]

Original: By applying lemma C.3, the first term is bounded by R m ( H ) .

Original: Then we follow a previous proof of Mohri et al. [18] to see that R m ( H ) ≤ KR m
Revised: or conducted research with human subjects...

Original: (Π 1 ( H )) . We bound the second term: [Equation] We use lemma C.2 to derive the second line and lemma C.3 to derive the fourth line.
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: D Proof of Theorem 6.

Original: Our proof is made by changing a part of a previous proof of theorem 1 of Li et al.

Original: We first introduce empirical Gaussian complexity and a lemma.

Original: Definition D.1.

Original: (Empirical Gaussian complexity) Let G be a family of functions mapping from Z to R , and let S =

Original: ( z 1 , . . .

Original: , z m ) ∈

Original: Z m be training data of size m .

Original: Then the empirical Gaussian complexity of G with respect to S is defined:

Original: [Equation]

Original: where g 1 , . .

Original: , g m are independent N (0 , 1) random variables.

Original: We need the following lemma, which is based on lemma 4 of Lei et al.

Original: Lemma D.2.

Original: Let H be a hypothesis class of mappings X × Y

Original: → R , where Y =

Original: [ K ] .

Original: h ∈ H is represented as vector h =

Original: ( h 1 , . . .

Original: , h K ) .

Original: Let c be a requirement, and let g 1 , . .

Original: . , g mK be N (0 , 1) random variables.

Original: Then for any training data S = ( x 1 , . . .

Original: , x m ) of size m , we have:

Original: [Equation]

Original: [Equation]

Original: Proof.

Original: We make a proof by modifying the proof for lemma 4 of Lei et al.

Original: Define two Gaussian processes indexed by H c and H :

Original: [Equation]

Original: For any h = ( h 1 , . . .

Original: , h K ) and ˜ h

Original: = (˜ h 1 , . . .

Original: , ˜ h K ) ∈ H , the independence of g i and equalities E [ g 2 i ]

Original: = 1 imply that [Equation] For any a = ( a 1 , . . . , a K ) and b = ( b 1 , . . . , b K ) ∈ R K , it can be directly checked that
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [Equation]

Original: Using the above inequality, we have the following bounds between X h c and Y h for all h, ˜ h ∈ H :

Original: [Equation]

Original: Finally, we can prove the lemma using this inequality and lemma A.1 of Lei et al.

Original: Note that the lemma holds if we substitute h j ( x i ) with h j ( x i ) + a ij , and h cj ( x i ) with h cj ( x i ) + a ij in Eq.

Original: (6), where ( a 11 , . . .

Original: , a mK ) are constants.

Original: We substitute lemma 1 of Li et al.

Original: [15] with the following lemma.

Original: Lemma D.3.

Original: The empirical Rademacher complexity of L c with example S of size m satisfies the following:

Original: [Equation]

Original: where g 1 , . .

Original: . , g nK are independent random variables following Gaussian distribution N (0 , 1) .

Original: Proof.

Original: Following the proof of lemma 1 of Li et al.

Original: [15], we have

Original: [Equation]

Original: Following the proof

Original: [Equation]

Original: the first term can be bounded:

Original: [Equation]

Original: By using lemma D

Original: [Equation]

Original: [Equation]

Original: we can bound the second term:

Original: [Equation]

Original: Adding these bounds can prove the lemma.

Original: We use the above lemma instead of lemma 1 of Li et al.

Original: [15] for proving Theorem 1 Li et al.

Original: [15], which gives a proof for our theorem.

Original: E Proof of Theorem 6.

Original: Proof.

Original: We first prove the bound for L add S,ρ ( h ) .

Original: Following the proof of Theorem 1 in Cortes et al.[6], we can prove that L D ( h c ) <

Original: L add D ,ρ ( h c ) and

Original: L add D ,ρ ( h c ) ≤ L add S

Original: [Equation]

Original: where H c 1 is defined:

Original: [Equation]

Original: We give a bound on the empirical Rademacher complexity of H c 1 .

Original: Due to the sub-additivity of the supremum, the following holds:

Original: [Equation]

Original: sup h c ∈H c m (cid:88) i

Original: = σ i max y ′ ̸

Original: = y i (cid:18)

Original: L ( y ′ , y i ) + h c ( x i , y ′ )

Original: ρ (cid:19)(cid:35) +m E σ (cid:34) sup h c ∈H c m (cid:88) i =

Original: σ i h c ( x i , y i ) ρ (cid:35) .

Original: We first bound the first term with the Lipschitzness of h (cid:55)→ max y ′ ̸

Original: = y (cid:16) L ( y ′ , y i )

Original: + h c

Original: ( x i ,y ′ )

Original: ρ (cid:17) for any requirement c .

Original: For any h, ˜ h ∈ H , (cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]

Original: max y ̸

Original: = y i

Original: [Equation]

Original: where we use the fact that | h c ( x, y ) − ˜ h c ( x, y ) | ≤ | h ( x, y ) − ˜ h ( x, y ) | for any h, ˜ h ∈ H and

Original: ( x, y ) ∈

Original: X ×Y since | h c ( x, y ) − ˜

Original: h c ( x, y ) | = 0 or equals | h ( x, y )

Original: − ˜ h

Original: ( x, y ) | , depending on requirement c .

Original: Following the proof of Theorem 1 in Cortes et al.

Original: [6], we have

Original: [Equation]

Original: We can apply Lemma 5 of Cortes et al.

Original: [6], which yields:

Original: [Equation]

Original: Similarly, for the second term, the following Lipschitz property holds:

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: Therefore, we can also obtain bound

Original: [Equation]

Original: Taking the expectation over S of the two inequalities shows that R m ( H c 1 ) ≤ 2 √ 2 ρ R Gm ( H ) , which completes the proof of the first statement.

Original: For the second statement, we follow a proof of Cortes et al.

Original: [6] to obtain:

Original: [Equation]

Original: δ

Original: [Equation]

Original: m , where

Original: [Equation]

Original: We can see the following inequality holds:

Original: [Equation] where we use the fact | h c ( x, y ) − ˜ h c ( x, y ) | ≤ | h ( x, y ) − ˜ h ( x, y ) | for any h, ˜ h ∈ H and ( x, y ) ∈ X ×Y .
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: The reminder of the proof is identical as in the previous argument.

Original: F Summary of Notations

Original: Table 1 shows the notations used in the paper.

Original: [Table]

Original: hypothesis h modified to satisfy requirement c

Original: [Table]
Revised: [N/A]

Original: set of modified hypotheses defined as H c :

Original: = { h c : h ∈ H}

Original: [Table]