Original: Cost-Sensitive Self-Training for Optimizing Non-Decomposable Metrics
Revised: Cost-Sensitive Self-Training for Optimizing Non-Decomposable Metrics

Original: Abstract
Revised: Abstract

Original: Self-training-based semi-supervised learning algorithms have enabled the learning of highly accurate deep neural networks, using only a fraction of labeled data.
Revised: Self-training based semi-supervised learning algorithms have enabled the learning of highly accurate deep neural networks, using only a fraction of labeled data.

Original: However, the majority of work on self-training has focused on the objective of improving accuracy whereas practical machine learning systems can have complex goals i.e. maximizing the minimum of recall across classes  that are non-decomposable.
Revised: However, the majority of work on self-training has focused on the objective of improving accuracy whereas practical machine learning systems can have complex goals (e.g. maximizing the minimum of recall across classes etc.) that are non-decomposable in nature.

Original: In this work, we introduce the Cost-Sensitive Self-Training ( CSST ) framework which generalizes the self-training-based methods for optimizing non-decomposable metrics.
Revised: In this work, we introduce the Cost-Sensitive SelfTraining ( CSST ) framework which generalizes the self-training-based methods for optimizing non-decomposable metrics.

Original: We prove that our framework can better optimize the desired non-decomposable metric, under similar data distribution assumptions made for the analysis of self-training.
Revised: We prove that our framework can better optimize the desired non-decomposable metric utilizing unlabeled data, under similar data distribution assumptions made for the analysis of self-training.

Original: Using the proposed CSST framework we obtain practical self-training methods (for both vision and NLP tasks) for optimizing different non-decomposable metrics using deep neural networks.
Revised: Using the proposed CSST framework we obtain practical self-training methods (for both vision and NLP tasks) for optimizing different non-decomposable metrics using deep neural networks.

Original: Our results demonstrate that CSST achieves an improvement over the state-of-the-art ina majority of the cases across datasets and objectives.
Revised: Our results demonstrate that CSST achieves an improvement over the state-of-the-art in majority of the cases across datasets and objectives.

Original:  Introduction
Revised:  Introduction

Original: In recent years, semi-supervised learning algorithms are increasingly being used for training deep neural networks [4, 13, 35, 43].
Revised: In recent years, semi-supervised learning algorithms are increasingly being used for training deep neural networks [4, 13, 35, 43].

Original: These algorithms lead to accurate models by leveraging the unlabeled data in addition to the limited labeled data present.
Revised: These algorithms lead to accurate models by leveraging the unlabeled data in addition to the limited labeled data present.

Original: For example, it’s possible to obtain a model with minimal accuracy degradation ( ≤ 1% ) using 5% of labeled data with semi-supervised algorithms compared to supervised models trained using 100% labeled data [35].
Revised: For example, it’s possible to obtain a model with minimal accuracy degradation ( ≤ 1% ) using 5% of labeled data with semi-supervised algorithms compared to supervised models trained using 100% labeled data [35].

Original: Hence, the development of these algorithms has resulted in a vast reduction in the requirement for expensive labeled data.
Revised: Hence, the development of these algorithms has resulted in a vast reduction in the requirement for expensive labeled data.

Original: Self-training is one of the major paradigms for semi-supervised learning.
Revised: Self-training is one of the major paradigms for semi-supervised learning.

Original: It involves obtaining targets ( e .
Revised: It involves obtaining targets ( e .

Original: g .
Revised: g .

Original: pseudo-labels) from a network from the unlabeled data, and using them to train the network further.
Revised: pseudo-labels) from a network from the unlabeled data, and using them to train the network further.

Original: The modern self-training methods also utilize additional regularizers that enforce prediction consistency across input transformations (e.g., adversarial perturbations [22], augmentations [42, 35], etc.), enabling them to achieve high performance using only a tiny fraction of labeled data.
Revised: The modern self-training methods also utilize additional regularizers that enforce prediction consistency across input transformations (e.g., adversarial perturbations [22], augmentations [42, 35], etc.) , enabling them to achieve high performance using only a tiny fraction of labeled data.

Original: Currently, the enhanced variants of self-training with consistency regularization [47, 29] are among the state-ofthe-art (SOTA) methods for semi-supervised learning.
Revised: Currently, the enhanced variants of self-training with consistency regularization [47, 29] are among the state-ofthe-art (SOTA) methods for semi-supervised learning.

Original: Despite the popularity of self-training methods, most of the works [42, 1, 35] have focused on the objective of improving prediction accuracy.
Revised: Despite the popularity of self-training methods, most of the works [42, 1, 35] have focused on the objective of improving prediction accuracy.

Original: However, there are nuanced objectives in real-world based on the application requirements.
Revised: However, there are nuanced objectives in real-world based on the application requirements.

Original: Examples include minimizing the worst-case recall [23] used for federated learning, classifier coverage for minority classes for ensuring fairness [8], etc.
Revised: Examples include minimizing the worst-case recall [23] used for federated learning, classifier coverage for minority classes for ensuring fairness [8], etc.

Original: These objectives are complex and cannot be expressed just by using a loss function on the prediction of a single input (i.e., non-decomposable).
Revised: These objectives are complex and cannot be expressed just by using a loss function on the prediction of a single input (i.e., non-decomposable).

Original: There has been a considerable effort in optimizing nondecomposable objectives for different supervised machine learning models [24, 34].
Revised: There has been a considerable effort in optimizing nondecomposable objectives for different supervised machine learning models [24, 34].

Original: However, as supervision can be expensive in this work we aim to answer the question: Can we optimize nondecomposable objectives using self-training-based methods developed for semi-supervised learning?
Revised: However, as supervision can be expensive, in this work we aim to answer the question: Can we optimize nondecomposable objectives using self-training-based methods developed for semi-supervised learning?

Original: We first demonstrate that vanilla self-training methods (e.g., FixMatch [35], UDA [42], etc.) can produce unsatisfactory results forother non-decomposable metrics (Fig. 1).
Revised: We first demonstrate that vanilla self-training methods (e.g., FixMatch [35], UDA [42], etc.) can produce unsatisfactory results for non-decomposable metrics (Fig. 1).

Original: We then generalize the Cost-Sensitive Loss for Self-Training by introducing a novel weighted consistency regularizer, for a particular non-decomposable metric.
Revised: We then generalize the CostSensitive Loss for Self-Training by introducing a novel weighted consistency regularizer, for a particular non-decomposable metric.

Original: Further, for training neural networks we introduce appropriate loss functions and pseudo label selection (thresholding) mechanisms considering the non-decomposable metric we aim to optimize.
Revised: Further, for training neural networks we introduce appropriate loss functions and pseudo label selection (thresholding) mechanisms considering the non-decomposable metric we aim to optimize.

Original: We also prove that we can achieve better performance on desired non-decomposable metric through our framework utilizing self-training, under similar assumptions on data distributions as made for theoretical analysis of self-training [40].We demonstrate the practical application by optimizing various non-decomposable metrics by plugging existing methods ( e . g .
Revised: We also prove that we can achieve better performance on desired nondecomposable metric through our framework utilizing self-training, under similar assumptions on data distributions as made for theoretical analysis of self-training [40]. We demonstrate the practical application by optimizing various non-decomposable metrics by plugging existing methods ( e . g .

Original: FixMatch [35] etc.) into our framework.
Revised: FixMatch [35] etc.) into our framework.

Original: Our framework leads to a significant average improvement in desired metric of minimizing worst-case recall while maintaining similar accuracy (Fig. 1).
Revised: Our framework leads to a significant average improvement in desired metric of minimizing worst-case recall while maintaining similar accuracy (Fig. 1).

Original: We will release our code and framework publicly to promote reproducible research.

Original: In summary: a) we introduce a Cost-Sensitive Self-Training ( CSST ) framework for optimizing nondecomposable metrics that utilizes unlabeled data in addition to labeled data.
Revised: In summary: a) we introduce a Cost-Sensitive Self-Training ( CSST ) framework for optimizing nondecomposable metrics that utilizes unlabeled data in addition to labeled data.

Original: (Sec. 4) b) we provably demonstrate that our CSST framework can leverage unlabeled data to achieve better performance over baseline on desired non-decomposable metric (Sec. 3) c) we show that by combining CSST with self-training frameworks ( e .
Revised: (Sec. 4) b) we provably demonstrate that our CSST framework can leverage unlabeled data to achieve better performance over baseline on desired non-decomposable metric (Sec. 3) c) we show that by combining CSST with self-training frameworks ( e .

Original: g .
Revised: g .

Original: FixMatch [35], UDA [42] etc.) leads to effective optimization of non-decomposable metrics, resulting in significant improvement over vanilla baselines.
Revised: FixMatch [35], UDA [42] etc.) leads to effective optimization of non-decomposable metrics, resulting in significant improvement over vanilla baselines.

Original: (Sec. 5)
Revised: (Sec. 5)

Original:  Preliminaries 2.1 Non-Decomposable Objectives and Reduction to Cost-Sensitive Learning
Revised:  Preliminaries 2.1 Non-Decomposable Objectives and Reduction to Cost-Sensitive Learning

Original: [Table]
Revised: [Table]

Original: We consider the K -class classification problem with an instance space X and the set of labels Y =
Revised: We consider the K -class classification problem with an instance space X and the set of labels Y =

Original: [ K ] .
Revised: [ K ] .

Original: The data distribution on X × [ K ] is denoted by D .
Revised: The data distribution on X × [ K ] is denoted by D .

Original: For i ∈ [ K ] , we denote by π i the class prior P ( y = i ) .
Revised: For i ∈ [ K ] , we denote by π i the class prior P ( y = i ) .

Original: Notations commonly used across paper are in Table 5 present in Appendix.
Revised: Notations commonly used across paper are in Table 5 present in Appendix.

Original: For a classifier F : X → [ K ] , we define confusion matrix C
Revised: For a classifier F : X → [ K ] , we define confusion matrix C

Original: [ F ] ∈ R K
Revised: [ F ] ∈ R K

Original: × K by
Revised: × K by

Original: C ij [ F ] =
Revised: C ij [ F ] =

Original: E ( x,y ) ∼ D [ 1 ( y = i, F ( x ) = j )] .
Revised: E ( x,y ) ∼ D [ 1 ( y = i, F ( x ) = j )] .

Original: Many metrics relevant to classification can be defined as functions of entries of confusion matrices such as class-coverage, recall and accuracy to name a few.
Revised: Many metrics relevant to classification can be defined as functions of entries of confusion matrices such as class-coverage, recall and accuracy to name a few.

Original: We introduce more complex metrics, which are of practical importance in the case of imbalanced distributions [6] (Tab. 1).
Revised: We introduce more complex metrics, which are of practical importance in the case of imbalanced distributions [6] (Tab. 1).

Original: A clas- sifier often tends to suffer low recalls on tail (minority) classes in such cases.
Revised: A classifier often tends to suffer low recalls on tail (minority) classes in such cases.

Original: Therefore, one may want to maximize the worst case recall,
Revised: Therefore, one may want to maximize the worst case recall,

Original: [Equation]
Revised: [Equation]

Original: Similarly, on long-tailed datasets, the tail classes suffer from low coverage, lower than their respective priors.
Revised: Similarly, on long-tailed datasets, the tail classes suffer from low coverage, lower than their respective priors.

Original: An interesting objective in such circumstances is to maximise the mean recall, subject to the coverage being within a given margin.
Revised: An interesting objective in such circumstances is to maximise the mean recall, subject to the coverage being within a given margin.

Original: [Equation]
Revised: [Equation]

Original: Many of these metrics are non-decomposable , i.e., one cannot compute these metrics by simply calculating the average of scores on individual examples.
Revised: Many of these metrics are non-decomposable , i.e., one cannot compute these metrics by simply calculating the average of scores on individual examples.

Original: Optimizing for these metrics can be regarded as instances of cost-sensitive learning (CSL).
Revised: Optimizing for these metrics can be regarded as instances of cost-sensitive learning (CSL).

Original: More specifically, optimization problems of the form which can be written as a linear combination of G i,j and C ij [ F ] will be our focus in this work where G is a K × K matrix.
Revised: More specifically, optimization problems of the form which can be written as a linear combination of G i,j and C ij [ F ] will be our focus in this work where G is a K × K matrix.

Original: [Equation]
Revised: [Equation]

Original: The entry G ij represents the reward associated with predicting class j when the true class is i .
Revised: The entry G ij represents the reward associated with predicting class j when the true class is i .

Original: The matrix G is called a gain matrix [24].
Revised: The matrix G is called a gain matrix [24].

Original: Some more complex non-decomposable objectives for classification can be reduced to CSL [26, 36, 24].
Revised: Some more complex non-decomposable objectives for classification can be reduced to CSL [26, 36, 24].

Original: For instance, the aforementioned two complex objectives can be reduced to CSL using continuous relaxation or a Lagrange multiplier as bellow.
Revised: For instance, the aforementioned two complex objectives can be reduced to CSL using continuous relaxation or a Lagrange multiplier as bellow.

Original: Let ∆ K − 1 ⊂ R K be the K − 1 -dimensional probability simplex.
Revised: Let ∆ K − 1 ⊂ R K be the K − 1 -dimensional probability simplex.

Original: Then, maximizing the minimum recall is equivalent to the saddle-point optimization problem:
Revised: Then, maximizing the minimum recall is equivalent to the saddle-point optimization problem:

Original: [Equation]
Revised: [Equation]

Original: Thus, for a fixed λ , the corresponding gain matrix is given as a diagonal matrix diag( G 1 , . . .
Revised: Thus, for a fixed λ , the corresponding gain matrix is given as a diagonal matrix diag( G 1 , . . .

Original: , G K ) with G i
Revised: , G K ) with G i

Original: = λ
Revised: = λ

Original: i /π i for 1 ≤ i ≤ K .
Revised: i /π i for 1 ≤ i ≤ K .

Original: Similarly, using Lagrange multipliers λ ∈ R K ≥ 0 , Eq.
Revised: Similarly, using Lagrange multipliers λ ∈ R K ≥ 0 , Eq.

Original: (1) is rewritten as a max-min optimization problem [24, Sec. 2]:
Revised: (1) is rewritten as a max-min optimization problem [24, Sec. 2]:

Original: [Equation]
Revised: [Equation]

Original: In this case, the corresponding gain matrix G is given as G ij =
Revised: In this case, the corresponding gain matrix G is given as G ij =

Original: δ ij
Revised: δ ij

Original: Kπ i + λ j , where δ ij is the Kronecker’s delta.
Revised: Kπ i + λ j , where δ ij is the Kronecker’s delta.

Original: One can solve these max-min problems by alternatingly updating λ (using exponented gradient or projected gradient descent) and 
Revised: One can solve these max-min problems by alternatingly updating λ (using exponented gradient or projected gradient descent) and optimizing the cost-sensitive objectives [24].

Original: 2.2 Loss Functions for Non-Decomposable Objectives
Revised: 2.2 Loss Functions for Non-Decomposable Objectives

Original: The cross entropy loss function is appropriate for optimizing accuracy for deep neural networks, however, learning with CE can suffer low performance for cost-sensitive objectives [24].
Revised: The cross entropy loss function is appropriate for optimizing accuracy for deep neural networks, however, learning with CE can suffer low performance for cost-sensitive objectives [24].

Original: Following [24], we introduce calibrated loss functions for given gain matrix G .
Revised: Following [24], we introduce calibrated loss functions for given gain matrix G .

Original: We let p m : X → ∆ K − 1 ⊂ R K be a prediction function of a model, where ∆ K − 1 is the K − 1 -dimensional probability simplex.
Revised: We let p m : X → ∆ K − 1 ⊂ R K be a prediction function of a model, where ∆ K − 1 is the K − 1 -dimensional probability simplex.

Original: For a gain matrix G , the corresponding loss function is given as a combination of logit adjustment [21] and loss re-weighting [28].
Revised: For a gain matrix G , the corresponding loss function is given as a combination of logit adjustment [21] and loss re-weighting [28].

Original: We decompose the gain matrix G as G = MD , where D = diag( G 11 , . . .
Revised: We decompose the gain matrix G as G = MD , where D = diag( G 11 , . . .

Original: , G KK ) be a diagonal matrix, with D ii > 0 ,
Revised: , G KK ) be a diagonal matrix, with D ii > 0 ,

Original: ∀ i ∈
Revised: ∀ i ∈

Original: [ K ] and M ∈ R K × K .
Revised: [ K ] and M ∈ R K × K .

Original: For y ∈ [ K ] and model prediction p m ( x ) , the hybrid loss is defined as follows:
Revised: For y ∈ [ K ] and model prediction p m ( x ) , the hybrid loss is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: To make the dependence on G explicit, we also denote ℓ hyb ( y, p m ( x )) as ℓ hyb ( y, p m ( x ); G ) .
Revised: To make the dependence of G explicit, we also denote ℓ hyb ( y, p m ( x )) as ℓ hyb ( y, p m ( x ); G ) .

Original: The average loss on training sample S ⊂ X is defined as L hyb ( X ) = 1 | S | (cid:80) x ∈ S ℓ hyb ( u,
Revised: The average loss on training sample S ⊂ X is defined as L hyb ( X ) = 1 | S | (cid:80) x ∈ S ℓ hyb ( u,

Original: p m ( x )) .
Revised: p m ( x )) .

Original: Narasimhan and Menon [24] proved that the hybrid loss is calibrated, that is learning with ℓ hyb gives the Bayes optimal classifier for G (c.f., [24, Proposition 4], of which we provide a formal statement in Sec.
Revised: Narasimhan and Menon [24] proved that the hybrid loss is calibrated, that is learning with ℓ hyb gives the Bayes optimal classifier for G (c.f., [24, Proposition 4], of which we provide a formal statement in Sec.

Original: N.1).
Revised: N.1).

Original: If G is a diagonal matrix (i.e., M = 1 K ), the hybrid loss is called the logit adjusted (LA) loss and ℓ hyb ( y, p m ( x )) is denoted by ℓ LA ( y, p m ( x )) .
Revised: If G is a diagonal matrix (i.e., M = 1 K ), the hybrid loss is called the logit adjusted (LA) loss and ℓ hyb ( y, p m ( x )) is denoted by ℓ LA ( y, p m ( x )) .

Original: 2.3 Consistency Regularizer for Semi-Supervised Learning
Revised: 2.3 Consistency Regularizer for Semi-Supervised Learning

Original: Modern self-training methods not only leverage pseudo labels, but also forces consistent predictions of a classifier on augmented examples or neighbor examples [40, 22, 42, 35].
Revised: Modern self-training methods not only leverage pseudo labels, but also forces consistent predictions of a classifier on augmented examples or neighbor examples [40, 22, 42, 35].

Original: More formally, a classifier F is trained so that the consistent regularizer R ( F ) is small while a supervised loss or a loss between pseudo labeler are minimized [40, 35].
Revised: More formally, a classifier F is trained so that the consistent regularizer R ( F ) is small while a supervised loss or a loss between pseudo labeler are minimized [40, 35].

Original: Here the consistency regularizer R ( F ) is defined as
Revised: Here the consistency regularizer R ( F ) is defined as

Original: E x [ 1 ( F ( x ) ̸ = F ( x ′ ) , ∃ x ′ s.t. x
Revised: E x [ 1 ( F ( x ) ̸ = F ( x ′ ) , ∃ x ′ s.t. x

Original: ′ is a neighbor of an augmentation of x )] .
Revised: ′ is a neighbor of an augmentation of x )] .

Original: In existing works, consistency regularizers are considered for optimization of accuracy.
Revised: In existing works, consistency regularizers are considered for optimization of accuracy.

Original: In the subsequent sections, we consider consistency regularizers for cost-sensitive objectives.
Revised: In the subsequent sections, we consider consistency regularizers for cost-sensitive objectives.

Original:  Cost-Sensitive Self-Training for Non-Decomposable Metrics 3.1 CSL and Weighted Error
Revised:  Cost-Sensitive Self-Training for Non-Decomposable Metrics 3.1 CSL and Weighted Error

Original: In the case of accuracy or 0-1-error, a self-training based SSL algorithm using a consistency regularizer achieves the state-of-the-art performance across a variety of datasets [35] and its effectiveness has been proved theoretically [40].
Revised: In the case of accuracy or 0-1-error, a self-training based SSL algorithm using a consistency regularizer achieves the state-of-the-art performance across a variety of datasets [35] and its effectiveness has been proved theoretically [40].

Original: This section provides theoretical analysis of a self-training based SSL algorithm for non-decomposable objectives by generalizing [40].
Revised: This section provides theoretical analysis of a self-training based SSL algorithm for non-decomposable objectives by generalizing [40].

Original: More precisely, the main result of this section (Theorem 5) states that an SSL method using consistency regularizer improves a given pseudo labeler for non-decomposable objectives.
Revised: More precisely, the main result of this section (Theorem 5) states that an SSL method using consistency regularizer improves a given pseudo labeler for non-decomposable objectives.

Original: We provide all the omitted proofs in Appendix for theoretical results in the paper.
Revised: We provide all the omitted proofs in Appendix for theoretical results in the paper.

Original: In Sec. 2, we considered non-decomposable metrics and their reduction to cost-sensitive learning objectives defined by Eq.
Revised: In Sec. 2, we considered non-decomposable metrics and their reduction to cost-sensitive learning objectives defined by Eq.

Original: (2) using a gain matrix.
Revised: (2) using a gain matrix.

Original: In this section, we consider an equivalent objective using the notion of weighted error.
Revised: In this section, we consider an equivalent objective using the notion of weighted error.

Original: For weight matrix w = ( w ij ) 1 ≤ i,j ≤ K and a classifier
Revised: For weight matrix w = ( w ij ) 1 ≤ i,j ≤ K and a classifier

Original: F : X → [ K ] , a weighted error is defined as follows:
Revised: F : X → [ K ] , a weighted error is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: where, P i ( x ) denotes the class conditional distribution P ( x
Revised: where, P i ( x ) denotes the class conditional distribution P ( x

Original: [Equation]
Revised: [Equation]

Original: = i ) .
Revised: = i ) .

Original: If w = diag(1 /K, . .
Revised: If w = diag(1 /K, . .

Original: , 1 /K ) , then this coincides with the usual balanced error [21].
Revised: , 1 /K ) , then this coincides with the usual balanced error [21].

Original: Since C ij [ F ] =
Revised: Since C ij [ F ] =

Original: E ( x,y ) ∼ D [ 1 ( y = i, F ( x ) = j )]
Revised: E ( x,y ) ∼ D [ 1 ( y = i, F ( x ) = j )]

Original: = P ( y = i ) − P
Revised: = P ( y = i ) − P

Original: ( y = i )
Revised: ( y = i )

Original: E x ∼ P i [ 1 ( F ( x ) ̸ = j )] , we can write:
Revised: E x ∼ P i [ 1 ( F ( x ) ̸ = j )] , we can write:

Original: G ij C ij [ F ] =
Revised: G ij C ij [ F ] =

Original: G ij ( P ( y = i ) − P
Revised: G ij ( P ( y = i ) − P

Original: ( y = i )
Revised: ( y = i )

Original: E x ∼ P i [ 1 ( F ( x ) ̸ = j )])
Revised: E x ∼ P i [ 1 ( F ( x ) ̸ = j )])

Original: =
Revised: =

Original: G ij ( π i − π i E x ∼ P i [ 1 ( F ( x ) ̸ = j )])
Revised: G ij ( π i − π i E x ∼ P i [ 1 ( F ( x ) ̸ = j )])

Original: Here π i is the class prior P
Revised: Here π i is the class prior P

Original: ( y = i ) for 1 ≤ i ≤ K as before.
Revised: ( y = i ) for 1 ≤ i ≤ K as before.

Original: Hence CSL objective (2) i.e. max F (cid:80) i,j G ij C ij [ F ] is equivalent to minimizing the negative term above i.e.
Revised: Hence CSL objective (2) i.e. max F (cid:80) i,j G ij C ij [ F ] is equivalent to minimizing the negative term above i.e.

Original: G ij π i E x ∼ P i [ 1 ( F ( x ) ̸ = j )] which is same as Err w ( F ) with w ij = G ij π i for 1 ≤ i, j ≤ K .
Revised: G ij π i E x ∼ P i [ 1 ( F ( x ) ̸ = j )] which is same as Err w ( F ) with w ij = G ij π i for 1 ≤ i, j ≤ K .

Original: Hence, the notion of weighted error is equivalent to CSL , which we will also use later for deriving loss functions.
Revised: Hence, the notion of weighted error is equivalent to CSL , which we will also use later for deriving loss functions.

Original: We further note that if we add a matrix with the same columns ( c 1 ≥ 0 ) to the gain matrix
Revised: We further note that if we add a matrix with the same columns ( c 1 ≥ 0 ) to the gain matrix

Original: G , still the maximizers of CSL (2) are the same as the original problem.
Revised: G , still the maximizers of CSL (2) are the same as the original problem.

Original: Hence, without loss of generality, we assume w ij ≥ 0 .
Revised: Hence, without loss of generality, we assume w ij ≥ 0 .

Original: We assume w ̸ = 0 , i.e., | w | 1 > 0 for avoiding degenerate solutions.
Revised: We assume w ̸ = 0 , i.e., | w | 1 > 0 for avoiding degenerate solutions.

Original: In the previous work [40], it is assumed that there exists a ground truth classifier F ⋆ : X → [ K ] and the supports of distributions { P i } 1 ≤ i ≤ K are disjoint.
Revised: In the previous work [40], it is assumed that there exists a ground truth classifier F ⋆ : X → [ K ] and the supports of distributions { P i } 1 ≤ i ≤ K are disjoint.

Original: However, if supports of distributions { P i } 1 ≤ i ≤ K are disjoint, a solution of the minimization problem min F Err w ( F ) is independent of w in some cases.
Revised: However, if supports of distributions { P i } 1 ≤ i ≤ K are disjoint, a solution of the minimization problem min F Err w ( F ) is independent of w in some cases.

Original: More precisely, if w = diag( w 1 , . . . , w K ) i.e. a diagonal matrix and w i > 0 , ∀
Revised: More precisely, if w = diag( w 1 , . . . , w K ) i.e. a diagonal matrix and w i > 0 , ∀

Original: i , then the optimal classifier is given as x (cid:55)→ argmax k ∈
Revised: i , then the optimal classifier is given as x (cid:55)→ argmax k ∈

Original: [ K ] w k P k ( x ) (this follows from [24, Proposition 1]).
Revised: [ K ] w k P k ( x ) (this follows from [24, Proposition 1]).

Original: If supports are disjoint, then the optimal classifier is the same as x (cid:55)→ argmax k ∈
Revised: If supports are disjoint, then the optimal classifier is the same as x (cid:55)→ argmax k ∈

Original: [ K ] P k ( x ) , which coincides with the ground truth classifier.
Revised: [ K ] P k ( x ) , which coincides with the ground truth classifier.

Original: Therefore, we do not assume the supports of P i are disjoint nor a ground truth classifier exists unlike [40].
Revised: Therefore, we do not assume the supports of P i are disjoint nor a ground truth classifier exists unlike [40].

Original: See Fig.
Revised: See Fig.

Original: 2 for an intuitive explanation.
Revised: 2 for an intuitive explanation.

Original: 3.2 Weighted Consistency Regularizer
Revised: 3.2 Weighted Consistency Regularizer

Original: For improving accuracy, the consistency in prediction is equally important across the distributions
Revised: For improving accuracy, the consistency in prediction is equally important across the distributions

Original: P i for 1 ≤ i ≤ K [35, 40].
Revised: P i for 1 ≤ i ≤ K [35, 40].

Original: However, for our case of weighted error, if the entries of the i th 0 row of the weight matrix w are larger than the other entries for some i 0 , then the consistency of model predictions on examples drawn from the distribution P i 0 are more important than those on the other
Revised: However, for our case of weighted error, if the entries of the i th 0 row of the weight matrix w are larger than the other entries for some i 0 , then the consistency of model predictions on examples drawn from the distribution P i 0 are more important than those on the other

Original: [Figure]
Revised: [Figure]

Original: examples.
Revised: examples.

Original: In this case, we require more restrictive consistency regularizer for distribution for P i 0 .
Revised: In this case, we require more restrictive consistency regularizer for distribution for P i 0 .

Original: Thus, we require a weighted (or cost-sensitive) consistency regularizer, which we define below.
Revised: Thus, we require a weighted (or cost-sensitive) consistency regularizer, which we define below.

Original: We assume that the instance space X is a normed vector space with norm | · | and T is a set of augmentations, i.e., each T ∈ T is a map from X to itself.
Revised: We assume that the instance space X is a normed vector space with norm | · | and T is a set of augmentations, i.e., each T ∈ T is a map from X to itself.

Original: For a fixed r > 0 , we define B ( x ) by { x ′ ∈ X : ∃ T ∈ T s.t.
Revised: For a fixed r > 0 , we define B ( x ) by { x ′ ∈ X : ∃ T ∈ T s.t.

Original: | x ′ − T ( x ) | ≤ r } .
Revised: | x ′ − T ( x ) | ≤ r } .

Original: For each i ∈ [ K ] , we define conditional consistency regularizer by R B ,i ( F ) =
Revised: For each i ∈ [ K ] , we define conditional consistency regularizer by R B ,i ( F ) =

Original: E x ∼ P i [ 1 (
Revised: E x ∼ P i [ 1 (

Original: ∃ x ′ ∈ B ( x ) s.t. F ( x ) ̸ = F ( x ′ ))] .
Revised: ∃ x ′ ∈ B ( x ) s.t. F ( x ) ̸ = F ( x ′ ))] .

Original: Then, we define the weighted consistency regularizer by R B ,w ( F ) =
Revised: Then, we define the weighted consistency regularizer by R B ,w ( F ) =

Original: (cid:80) i,j ∈
Revised: (cid:80) i,j ∈

Original: [ K ] w ij R B ,i ( F ) .
Revised: [ K ] w ij R B ,i ( F ) .

Original: If the prediction of classifier F is robust to augmentation T ∈ T and small noise, then R B ,w ( F ) is small.
Revised: If the prediction of classifier F is robust to augmentation T ∈ T and small noise, then R B ,w ( F ) is small.

Original: For β > 0 , we consider the following optimization objective for finding a classifier F :
Revised: For β > 0 , we consider the following optimization objective for finding a classifier F :

Original: [Equation]
Revised: [Equation]

Original: A solution of the problem (6) is denoted by F ∗ .
Revised: A solution of the problem (6) is denoted by F ∗ .

Original: We let F pl : X → [ K ] a pseudo labeler (a classifier) with reasonable performance (we elaborate on this in Section 3.4).
Revised: We let F pl : X → [ K ] a pseudo labeler (a classifier) with reasonable performance (we elaborate on this in Section 3.4).

Original: The following mathematically informal assumption below is required to interpret our main theorem.
Revised: The following mathematically informal assumption below is required to interpret our main theorem.

Original: A SSUMPTION 1.
Revised: A SSUMPTION 1.

Original: We assume both β and Err w ( F ∗ ) are sufficiently small so that they are negligible compared to Err w ( F pl ) .
Revised: We assume both β and Err w ( F ∗ ) are sufficiently small so that they are negligible compared to Err w ( F pl ) .

Original: Assumption 1 assumes existence of an optimal classifier F ∗ that minimizes the error Err w ( F ) (i.e.
Revised: Assumption 1 assumes existence of an optimal classifier F ∗ that minimizes the error Err w ( F ) (i.e.

Original: Bayes Optimal) among the class of classifiers which are robust to data augmentation (i.e. low weighted consistency R B ,w ( F ) ).
Revised: Bayes Optimal) among the class of classifiers which are robust to data augmentation (i.e. low weighted consistency R B ,w ( F ) ).

Original: As we operate in case of overparameterized neural networks such a classifier F ∗ is bound to exist, but is unknown in our problem setup.
Revised: As we operate in case of overparameterized neural networks such a classifier F ∗ is bound to exist, but is unknown in our problem setup.

Original: In the case of the balanced error, the validity of this assumption is justified by the fact that the existing work [35] using consistency regularizer on data augmentation obtains classifier F , that achieves high accuracy (i.e., low balanced errors) for balanced datasets.
Revised: In the case of the balanced error, the validity of this assumption is justified by the fact that the existing work [35] using consistency regularizer on data augmentation obtains classifier F , that achieves high accuracy (i.e., low balanced errors) for balanced datasets.

Original: Also in Appendix C.1, we provide an example that supports the validity of the assumption in the case of Gaussian mixtures and diagonal weight matrices.
Revised: Also in Appendix C.1, we provide an example that supports the validity of the assumption in the case of Gaussian mixtures and diagonal weight matrices.

Original: 3.3 Expansion Property
Revised: 3.3 Expansion Property

Original: For x ∈ X , we define the neighborhood N ( x ) of x by { x ′ ∈ X : B ( x ) ∩ B ( x ′ )
Revised: For x ∈ X , we define the neighborhood N ( x ) of x by { x ′ ∈ X : B ( x ) ∩ B ( x ′ )

Original: ̸ = ∅} .
Revised: ̸ = ∅} .

Original: For a subset
Revised: For a subset

Original: S ⊆ X , neighborhood of S is defined as N ( S ) = ∪ x ∈ S N ( x ) .
Revised: S ⊆ X , neighborhood of S is defined as N ( S ) = ∪ x ∈ S N ( x ) .

Original: Similarly to [40], we consider the following property on distributions.
Revised: Similarly to [40], we consider the following property on distributions.

Original: D EFINITION 2.
Revised: D EFINITION 2.

Original: Let c : (0 , 1] → [1 , ∞ ) be a non-increasing function.
Revised: Let c : (0 , 1] → [1 , ∞ ) be a non-increasing function.

Original: For a distribution Q on X we say Q has c -expansion property if Q ( N ( S )) ≥ c ( Q ( S ))
Revised: For a distribution Q on X we say Q has c -expansion property if Q ( N ( S )) ≥ c ( Q ( S ))

Original: Q ( S ) for any measurable S ⊆ X .
Revised: Q ( S ) for any measurable S ⊆ X .

Original: The c -expansion property implies that if Q ( S ) decreases, then the “expansion factor” Q ( N ( S )) /Q
Revised: The c -expansion property implies that if Q ( S ) decreases, then the “expansion factor” Q ( N ( S )) /Q

Original: ( S ) increases.
Revised: ( S ) increases.

Original: This is a natural condition, because it roughly requires that if Q ( S ) is small, then Q ( N ( S )) is large compared to Q ( S ) .
Revised: This is a natural condition, because it roughly requires that if Q ( S ) is small, then Q ( N ( S )) is large compared to Q ( S ) .

Original: For intuition let us consider a ball of radius l depicting S ⊂ R d with volume Q ( S ) and its neighborhood N ( S ) expands to a ball with radius l + 1 .
Revised: For intuition let us consider a ball of radius l depicting S ⊂ R d with volume Q ( S ) and its neighborhood N ( S ) expands to a ball with radius l + 1 .

Original: The expansion factor here would be (( l + 1) /l )
Revised: The expansion factor here would be (( l + 1) /l )

Original: d , hence as l (i.e. Q ( S ) ) increases (1 + 1 /l )
Revised: d , hence as l (i.e. Q ( S ) ) increases (1 + 1 /l )

Original: d (i.e. Q ( N ( S )) /Q
Revised: d (i.e. Q ( N ( S )) /Q

Original: ( S ) ) decreases.
Revised: ( S ) ) decreases.

Original: Hence, it’s natural to expect c to be a non-decreasing function.
Revised: Hence, it’s natural to expect c to be a non-decreasing function.

Original: The c -expansion property (on each P i ) considered here is equivalent to the ( a, (cid:101) c ) -expansion property, which is shown to be realistic for vision and used for theoretical analysis of self-training in [40], where a ∈ (0 , 1) and (cid:101) c > 1 (see Sec.
Revised: The c -expansion property (on each P i ) considered here is equivalent to the ( a, (cid:101) c ) -expansion property, which is shown to be realistic for vision and used for theoretical analysis of self-training in [40], where a ∈ (0 , 1) and (cid:101) c > 1 (see Sec.

Original: N.2 in Appendix).
Revised: N.2 in Appendix).

Original: In addition, we also show that it is also satisfied for mixtures of Gaussians and mixtures of manifolds (see Example 8 in Appendix for more details).
Revised: In addition, we also show that it is also satisfied for mixtures of Gaussians and mixtures of manifolds (see Example 8 in Appendix for more details).

Original: Thus, the c -expansion property is a general property satisfied for a wide class of distributions.
Revised: Thus, the c -expansion property is a general property satisfied for a wide class of distributions.

Original: A SSUMPTION 3.
Revised: A SSUMPTION 3.

Original: For weighted probability measure P w on X by P w ( U ) = (cid:80) i,j ∈
Revised: For weighted probability measure P w on X by P w ( U ) = (cid:80) i,j ∈

Original: [ K ] w ij P i ( U ) (cid:80) i,j ∈
Revised: [ K ] w ij P i ( U ) (cid:80) i,j ∈

Original: [ K ] w ij for
Revised: [ K ] w ij for

Original: U ⊆
Revised: U ⊆

Original: X .
Revised: X .

Original: We assume P w satisfies c -expansion for a non-increasing function c : (0 , 1] → [1 , ∞ ) .
Revised: We assume P w satisfies c -expansion for a non-increasing function c : (0 , 1] → [1 , ∞ ) .

Original: 3.4 Cost-Sensitive Self-Training with Weighted Consistency Regularizer
Revised: 3.4 Cost-Sensitive Self-Training with Weighted Consistency Regularizer

Original: In this section we first introduce the assumptions on the pseudo labeler F pl
Revised: In this section we first introduce the assumptions on the pseudo labeler F pl

Original: and then introduce the theoretical Cost-Sensitive Self-Training ( CSST ) objective.
Revised: and then introduce the theoretical Cost-Sensitive Self-Training ( CSST ) objective.

Original: F pl can be any classifier satisfying the following assumption, however, typically it is a classifier trained on a labeled dataset.
Revised: F pl can be any classifier satisfying the following assumption, however, typically it is a classifier trained on a labeled dataset.

Original: A SSUMPTION 4.
Revised: A SSUMPTION 4.

Original: We assume that Err w ( F pl ) + Err w ( F ∗ ) ≤ | w | 1 .
Revised: We assume that Err w ( F pl ) + Err w ( F ∗ ) ≤ | w | 1 .

Original: Let γ =
Revised: Let γ =

Original: c ( p w ) ,
Revised: c ( p w ) ,

Original: where p w =
Revised: where p w =

Original: Err w ( F pl )+Err w ( F ∗ ) | w |.
Revised: Err w ( F pl )+Err w ( F ∗ ) | w |.

Original: We also assume γ > 3 .
Revised: We also assume γ > 3 .

Original: Since c is non-increasing, γ (as a function of Err w ( F pl ) ) is a non-increasing function of Err w ( F pl ) (and Err w ( F ∗ ) ).
Revised: Since c is non-increasing, γ (as a function of Err w ( F pl ) ) is a non-increasing function of Err w ( F pl ) (and Err w ( F ∗ ) ).

Original: Therefore, the assumption γ > 3 roughly requires that Err w ( F pl ) is “small”.
Revised: Therefore, the assumption γ > 3 roughly requires that Err w ( F pl ) is “small”.

Original: We provide concrete conditions for Err w ( F pl ) that satisfy γ > 3 in the case of mixture of isotropic d -dimenional Gaussians for a region B ( x ) defined by r in Appendix (Example 9).
Revised: We provide concrete conditions for Err w ( F pl ) that satisfy γ > 3 in the case of mixture of isotropic d -dimenional Gaussians for a region B ( x ) defined by r in Appendix (Example 9).

Original: In the example, we show that the condition γ > 3 is satisfied if Err w ( F pl ) < 0 .
Revised: In the example, we show that the condition γ > 3 is satisfied if Err w ( F pl ) < 0 .

Original: 17 in the case when r = 1 / (2 √ d ) and satisfied if Err w ( F pl ) < 0 .
Revised: 17 in the case when r = 1 / (2 √ d ) and satisfied if Err w ( F pl ) < 0 .

Original: 33 in the case when r = 3 / (2 √ d ) , where X ⊆ R d .
Revised: 33 in the case when r = 3 / (2 √ d ) , where X ⊆ R d .

Original: Since we assume
Revised: Since we assume

Original: Err w ( F ∗ ) is negligible compared to Err w ( F pl )
Revised: Err w ( F ∗ ) is negligible compared to Err w ( F pl )

Original: (Assumption 1), the former condition in Assumption 4 is approximately equivalent to Err w ( F pl ) ≤ | w | 1 which is satisfied by the definition of Err w .
Revised: (Assumption 1), the former condition in Assumption 4 is approximately equivalent to Err w ( F pl ) ≤ | w | 1 which is satisfied by the definition of Err w .

Original: We define L ( i ) 0-1 ( F, F ′ )
Revised: We define L ( i ) 0-1 ( F, F ′ )

Original: =
Revised: =

Original: E x ∼ P i [ 1 ( F ( x ) ̸
Revised: E x ∼ P i [ 1 ( F ( x ) ̸

Original: = F ′ ( x ))] .
Revised: = F ′ ( x ))] .

Original: Then, we consider the following objective:
Revised: Then, we consider the following objective:

Original: [Equation]
Revised: [Equation]

Original: Here L w ( F, F pl ) is defined as (cid:80) i,j ∈
Revised: Here L w ( F, F pl ) is defined as (cid:80) i,j ∈

Original: [ K ] w ij L ( i ) 0-1 ( F, F pl ) .
Revised: [ K ] w ij L ( i ) 0-1 ( F, F pl ) .

Original: The above objective corresponds to costsensitive self-training (with F pl ) objective with weighted consistency regularization.
Revised: The above objective corresponds to costsensitive self-training (with F pl ) objective with weighted consistency regularization.

Original: We provide following theorem which relates the weighted error of classifier ˆ F learnt using the above objective to the weighted error of the pseudo labeler ( F pl ).
Revised: We provide following theorem which relates the weighted error of classifier ˆ F learnt using the above objective to the weighted error of the pseudo labeler ( F pl ).

Original: T HEOREM 5.
Revised: T HEOREM 5.

Original: Any learnt classifier (cid:98) F using the loss function L w (i.e., argmin F L w ( F ) ) satisfies:
Revised: Any learnt classifier (cid:98) F using the loss function L w (i.e., argmin F L w ( F ) ) satisfies:

Original: [Equation]
Revised: [Equation]

Original: R EMARK .
Revised: R EMARK .

Original: Since both Err w ( F ∗ ) and R B ,w ( F ∗ ) ≤ β are negligible compared to Err w ( F pl ) and γ > 3 , Theorem 5 asserts that ˆ F learnt by minimizing semi-supervised loss L w ( F, F pl ) with the consistency regularizer R B ,w ( F ) can achieve superior performance than the pseudo labeler F pl in terms of the weighted error Err w .
Revised: Since both Err w ( F ∗ ) and R B ,w ( F ∗ ) ≤ β are negligible compared to Err w ( F pl ) and γ > 3 , Theorem 5 asserts that ˆ F learnt by minimizing semi-supervised loss L w ( F, F pl ) with the consistency regularizer R B ,w ( F ) can achieve superior performance than the pseudo labeler F pl in terms of the weighted error Err w .

Original: The above theorem is a generalization of [40, Theorem 4.3], which provided a similar result for balanced 0-1-error in the case of distributions with disjoint supports.
Revised: The above theorem is a generalization of [40, Theorem 4.3], which provided a similar result for balanced 0-1-error in the case of distributions with disjoint supports.

Original: In
Revised: In

Original: Appendix Sec.
Revised: Appendix Sec.

Original: D, following [40, 39], we also provide a generalization bound for Err w ( F ) using all-layer margin [39] in the case when classifiers are neural networks.
Revised: D, following [40, 39], we also provide a generalization bound for Err w ( F ) using all-layer margin [39] in the case when classifiers are neural networks.

Original:  CSST in Practice
Revised:  CSST in Practice

Original: In the previous section, we proved that  using self-training ( CSST ) we can achieve a superior classifier ˆ F in comparison to pseudo labeler F pl through weighted consistency regularization.
Revised: In the previous section, we proved that by using self-training ( CSST ), we can achieve a superior classifier ˆ F in comparison to pseudo labeler F pl through weighted consistency regularization.

Original: As we have established the equivalance of the weighted error Err w to the CSL objective expressed in terms of G (Sec. 3.1) , we can theoretically optimize a given non-decomposable metric expressed by G better using CSST , utilizing the additional unlabeled data via self-training and weighted consistency regularization.
Revised: As we have established the equivalence of the weighted error Err w to the CSL objective expressed in terms of G (Sec. 3.1) , we can theoretically optimize a given non-decomposable metric expressed by G better using CSST , utilizing the additional unlabeled data via self-training and weighted consistency regularization.

Original: We now show how CSST can be used in practice for optimizing non-decomposable metrics in  case of neural networks.
Revised: We now show how CSST can be used in practice for optimizing non-decomposable metrics in the case of neural networks.

Original: The practical self-training methods utilizing consistency regularization (eg. FixMatch [35], etc.) for semi-supervised learning, have supervised loss L s for labeled and consistency regularization loss for unlabeled samples (i.e. L u ) with a thresholding mechanism to select unlabeled samples.
Revised: The practical self-training methods utilizing consistency regularization (e.g., FixMatch [35], etc.) for semi-supervised learning have supervised loss L s for labeled and consistency regularization loss for unlabeled samples (i.e., L u ) with a thresholding mechanism to select unlabeled samples.

Original: The final loss for training the network is L s + λ u L u  where λ u is the hyperparameter.
Revised: The final loss for training the network is L s + λ u L u , where λ u is the hyperparameter.

Original: The supervised loss L s can be modified easily based on  desired non-decomposable metric by using suitable G (Sec. 
Revised: The supervised loss L s can be modified conveniently based on the desired non-decomposable metric by using suitable G (Sec. 2.1).

Original: We will now introduce the novel weighted consistency loss and its corresponding thresholding mechanism for unlabeled data in CSST , used for optimizing desired non-decomposable metric.
Revised: We will now introduce the novel weighted consistency loss and its corresponding thresholding mechanism for unlabeled data in CSST , used for optimizing desired non-decomposable metric.

Original: Weighted Consistency Regularization.
Revised: Weighted Consistency Regularization.

Original: As the idea of consistency regularization is to enforce consistency between model prediction on different augmentations of input, this is usually achieved by minimizing some kind of divergence D .
Revised: As the idea of consistency regularization is to enforce consistency between model prediction on different augmentations of input, this is usually achieved by minimizing some kind of divergence D .

Original: A lot of recent works [22, 35, 42] in semi-supervised learning use D KL to enforce consistency between the model’s prediction on unlabeled data and its augmentations, p m ( x ) and p m ( A ( x )) .
Revised: A lot of recent works [22, 35, 42] in semi-supervised learning use D KL to enforce consistency between the model’s prediction on unlabeled data and its augmentations, p m ( x ) and p m ( A ( x )) .

Original: Here A usually denotes a form of strong augmentation.
Revised: Here A usually denotes a form of strong augmentation.

Original: Across these works, the distribution of confidence of the model’s prediction is either sharpened or used to get a hard pseudo label to obtain ˆ p m ( x ) .
Revised: Across these works, the distribution of confidence of the model’s prediction is either sharpened or used to get a hard pseudo label to obtain ˆ p m ( x ) .

Original: As we aim to optimized the cost-sensitive learning objective, we aim to match the distribution of normalized distribution (i.e. norm ( G T ˆ p m ( x )) =[Equation] [Equation] i
Revised: As we aim to optimized the cost-sensitive learning objective, we aim to match the distribution of normalized distribution (i.e. norm ( G T ˆ p m ( x )) =

Original: [Equation] G [Equation] ˆ p m 
Revised: G T ˆ p m ( x ) / (cid:80) i ( G T ˆ p m ( x )) i )

Original: [Equation] x )) [Equation] ) (Proposition 2[24] also in Prop. 7 ) with the p m ( A ( x )) by minimizing the KL-Divergence between these.
Revised: (Proposition 2 [24] also in Prop. 7 ) with the p m ( A ( x )) by minimizing the KL-Divergence between these.

Original: We now propose to use the following weighted consistency regularizer loss function for optimizing the same:
Revised: We now propose to use the following weighted consistency regularizer loss function for optimizing the same:

Original: [Equation]
Revised: [Equation]

Original: P ROPOSITION 6.
Revised: P ROPOSITION 6.

Original: The minimizer of L wtu =
Revised: The minimizer of L wtu =

Original: [Equation]
Revised: [Equation]

Revised: | B u | (cid:80)

Original:  ( A ( x )) , G ) leads to minimization of KL Divergence i.e. D KL ( norm ( G T ˆ p m ( x ))
Revised: [Equation] [Equation] ) , p m ( A ( x )) , G ) leads to minimization of KL Divergence i.e. D KL ( norm ( G T ˆ p m ( x ))

Original: ||
Revised: ||

Original: p m ( A ( x )))
Revised: p m ( A ( x )))

Original: ∀ x ∈ X .
Revised: ∀ x ∈ X .

Original: As the above loss is similar in nature to the cost sensitive losses introduced by Narasimhan and
Revised: As the above loss is similar in nature to the cost sensitive losses introduced by Narasimhan and

Original: Menon [24] (Sec. 2.1) we can use the logit-adjusted variants (i.e. ℓ LA and ℓ hyb based on type of
Revised: Menon [24] (Sec. 2.1) we can use the logit-adjusted variants (i.e. ℓ LA and ℓ hyb based on type of

Original: G ) of these in our final loss formulations ( L wtu ) for training overparameterized deep networks.
Revised: G ) of these in our final loss formulations ( L wtu ) for training overparameterized deep networks.

Original: We further show in Appendix Sec.
Revised: We further show in Appendix Sec.

Original: B that the above loss ℓ wt u approximately minimizes the theoretical weighted consistency regularization term R B ,w ( F ) defined in Sec.
Revised: B that the above loss ℓ wt u approximately minimizes the theoretical weighted consistency regularization term R B ,w ( F ) defined in Sec.

Original: Threshold Mechanism for CSST .
Revised: Threshold Mechanism for CSST .

Original: In the usual semi-supervised learning formulation [35] we use the confidence threshold ( max i ( p m ( x ) i ) > τ ) as the function to select samples for which consistency regularization term is non-zero.
Revised: In the usual semi-supervised learning formulation [35] we use the confidence threshold ( max i ( p m ( x ) i ) > τ ) as the function to select samples for which consistency regularization term is non-zero.

Original: We find that this leads to sub-optimal results in particularly the case of non-diagonal G as only a few samples cross the threshold (Fig. 4).
Revised: We find that this leads to sub-optimal results in particularly the case of non-diagonal G as only a few samples cross the threshold (Fig. 4).

Original: As in the case of cost-sensitive loss formulation the samples may not achieve the high confidence to cross the threshold of consistency regularization.
Revised: As in the case of cost-sensitive loss formulation the samples may not achieve the high confidence to cross the threshold of consistency regularization.

Original: This is also theoretically justified by the following proposition:
Revised: This is also theoretically justified by the following proposition:

Original: P ROPOSITION 7 ([24] Proposition 2) .
Revised: P ROPOSITION 7 ([24] Proposition 2) .

Original: Let p optm ( x ) be the optimal softmax model function obtained by optimizing the cost-sensitive objective in Eq.
Revised: Let p optm ( x ) be the optimal softmax model function obtained by optimizing the cost-sensitive objective in Eq.

Original: (2) by averaging weighted loss function ℓ wt ( y, p m ( x ))
Revised: (2) by averaging weighted loss function ℓ wt ( y, p m ( x ))

Original: = − (cid:80)
Revised: = − (cid:80)

Original: Ki
Revised: Ki

Original: =

Revised: =

Original: G y,i log ( p m ( x ) i ) (cid:80)
Revised: G y,i log ( p m ( x ) i ) (cid:80)

Original: j p m ( x ) j .
Revised: j p m ( x ) j .

Original: Then optimal p optm ( x ) is: p optm ( x ) =
Revised: Then optimal p optm ( x ) is: p optm ( x [Equation] =

Original: [Equation]
Revised: [Equation]

Original: Here y is the one-hot representation vector for a label y .
Revised: Here y is the one-hot representation vector for a label y .

Original: This proposition demonstrates that for a particular sample the high confidence p m ( x ) may not be optimal based on G .
Revised: This proposition demonstrates that for a particular sample the high confidence p m ( x ) may not be optimal based on G .

Original: We now propose our novel way of thresholding samples for which consistency regularization is applied in CSST .
Revised: We now propose our novel way of thresholding samples for which consistency regularization is applied in CSST .

Original: Our thresholding method takes into account the objective of optimizing the non-decomposable metric by taking G into account.
Revised: Our thresholding method takes into account the objective of optimizing the non-decomposable metric by taking G into account.

Original: We propose to use the threshold on KL-Divergence of the softmax of the sample p m ( x ) with the optimal softmax (i.e. norm ( G T ˆ p m ( x )) ) for a given G corresponding to the pseudo label (or sharpened)
Revised: We propose to use the threshold on KL-Divergence of the softmax of the sample p m ( x ) with the optimal softmax (i.e. norm ( G T ˆ p m ( x )) ) for a given G corresponding to the pseudo label (or sharpened)

Original: ˆ p m ( x ) , using which we modify the consistency regularization loss term:
Revised: ˆ p m ( x ) , using which we modify the consistency regularization loss term:

Original: [Equation]
Revised: [Equation]

Original: We name this proposed combination of KL-Thresholding and weighted consistency regularization as
Revised: We name this proposed combination of KL-Thresholding and weighted consistency regularization as

Original: CSST in our experimental results.
Revised: CSST in our experimental results.

Original: We find that for non-diagonal gain matrix G the proposed thresholding plays a major role in improving performance over supervised learning.
Revised: We find that for non-diagonal gain matrix G the proposed thresholding plays a major role in improving performance over supervised learning.

Original: This is demonstrated by comparison of CSST and CSST w/o KL-Thresholding (without proposed thresholding mechanism) in Fig.
Revised: This is demonstrated by comparison of CSST and CSST w/o KL-Thresholding (without proposed thresholding mechanism) in Fig.

Original: 4 and Tab. 3.
Revised: 4 and Tab. 3.

Original: We will now empirically incorporate CSST by introducing consistency based losses and thresholding mechanism for unlabeled data, into the popular semi-supervised methods of FixMatch [35] and Unsupervised Data Augmentation for Consistency Training (UDA) [42].
Revised: We will now empirically incorporate CSST by introducing consistency based losses and thresholding mechanism for unlabeled data, into the popular semi-supervised methods of FixMatch [35] and Unsupervised Data Augmentation for Consistency Training (UDA) [42].

Original: The exact expression for the weighted consistency losses utilized for UDA and FixMatch have been provided in the Appendix Sec.
Revised: The exact expression for the weighted consistency losses utilized for UDA and FixMatch have been provided in the Appendix Sec.

Original: G.2 and G.3 .
Revised: G.2 and G.3 .

Original: [Table]
Revised: [Table]

Original:  Experiments
Revised:  Experiments

Original: We demonstrate that the proposed CSST framework shows significant gains in performance on both vision and NLP tasks on imbalanced datasets, with an imbalance ratio defined on the training set as ρ = max i P ( y = i ) min i P ( y = i ) .
Revised: We demonstrate that the proposed CSST framework shows significant gains in performance on both vision and NLP tasks on imbalanced datasets, with an imbalance ratio defined on the training set as ρ = max i P ( y = i ) min i P ( y = i ) .

Original: We assume the labeled and unlabeled samples come from a similar data distribution and the unlabeled samples are much more abundant ( µ times) the labeled.
Revised: We assume the labeled and unlabeled samples come from a similar data distribution and the unlabeled samples are much more abundant ( µ times) the labeled.

Original: The frequency of samples follows an exponentially decaying long-tailed distributed as seen in Fig.
Revised: The frequency of samples follows an exponentially decaying long-tailed distributed as seen in Fig.

Original: 3, which closely imitates the distribution of real-world data [37, 14].
Revised: 3, which closely imitates the distribution of real-world data [37, 14].

Original: For CIFAR-10 [15], IMDb [20] and DBpedia-14 [18], we use ρ = 100 and ρ = 10 for CIFAR-100 [15] and ImageNet-100 [31] 2 datasets.
Revised: For CIFAR-10 [15], IMDb [20] and DBpedia-14 [18], we use ρ = 100 and ρ = 10 for CIFAR-100 [15] and ImageNet-100 [31] 2 datasets.

Original: We compare our method against supervised methods of ERM, Logit Adjustment ( LA ) [21] and Cost Sensitive
Revised: We compare our method against supervised methods of ERM, Logit Adjustment ( LA ) [21] and Cost Sensitive

Original: Learning ( CSL ) [25] trained on the same number of labeled samples as used by semi-supervised learning methods, along with vanilla semi-supervised methods of FixMatch (for vision) and UDA (for
Revised: Learning ( CSL ) [25] trained on the same number of labeled samples as used by semi-supervised learning methods, along with vanilla semi-supervised methods of FixMatch (for vision) and UDA (for

Original: NLP tasks).
Revised: NLP tasks).

Original: We use WideResNets(WRN) [45], specifically WRN-28-2 and WRN-28-8 for CIFARand CIFAR-100 respectively.
Revised: We use WideResNets(WRN) [45], specifically WRN-28-2 and WRN-28-8 for CIFARand CIFAR-100 respectively.

Original: For ImageNet, we use a ResNet-50 [9] network for our experiments and finetuned DistilBERT(base uncased)
Revised: For ImageNet, we use a ResNet-50 [9] network for our experiments and finetuned DistilBERT(base uncased)

Original: [33] for IMDb and DBpedia-14 datasets.
Revised: [33] for IMDb and DBpedia-14 datasets.

Original: We divide the balanced held-out set for each dataset equally into validation and test sets.
Revised: We divide the balanced held-out set for each dataset equally into validation and test sets.

Original: A detailed list of hyper-parameters and additional experiments can be found in the Appendix Tab. 6 and Sec.
Revised: A detailed list of hyper-parameters and additional experiments can be found in the Appendix Tab. 6 and Sec.

Original: O respectively.
Revised: O respectively.

Original: Maximizing Worst-Case Recall.
Revised: Maximizing Worst-Case Recall.

Original: For CIFAR-10, IMDb, and DBpedia-14 datasets, we maximize the minimum recall among all classes (Eq.
Revised: For CIFAR-10, IMDb, and DBpedia-14 datasets, we maximize the minimum recall among all classes (Eq.

Original: Given the low number of samples per class for datasets with larger number of classes like CIFAR-100 and ImageNet-100, we pick objective (10).
Revised: Given the low number of samples per class for datasets with larger number of classes like CIFAR-100 and ImageNet-100, we pick objective (10).

Original: We define the head classes ( H ) and tail classes ( T ) as the first 90 classes and last 10 classes respectively.
Revised: We define the head classes ( H ) and tail classes ( T ) as the first 90 classes and last 10 classes respectively.

Original: The
Revised: The

Original: Min.
Revised: Min.

Original: HT recall objective can be mathematically formulated as:
Revised: HT recall objective can be mathematically formulated as:

Original: [Equation]
Revised: [Equation]

Original: The corresponding gain matrix G is diag( λ H π 1 |H| , λ H π 2 |H| , . . .
Revised: The corresponding gain matrix G is diag( λ H π 1 |H| , λ H π 2 |H| , . . .

Original: [Equation]
Revised: [Equation]

Original: K |T | ).
Revised: K |T | ).

Original: Since G is diagonal here, we use CSST (FixMatch) loss function Eq.
Revised: Since G is diagonal here, we use CSST (FixMatch) loss function Eq.

Original: (9) with the corresponding ℓ wt u being substituted by ℓ LA u as define in Sec. 2.1.
Revised: (9) with the corresponding ℓ wt u being substituted by ℓ LA u as define in Sec. 2.1.

Original: Also for labeled samples we use L LA s as G is diagonal, we then combine the loss and train network using SGD.
Revised: Also for labeled samples we use L LA s as G is diagonal, we then combine the loss and train network using SGD.

Original: Each few steps of SGD, were followed by an update on the λ and G based on the uniform validation set (See Alg. 1 in Appendix).
Revised: Each few steps of SGD, were followed by an update on the λ and G based on the uniform validation set (See Alg. 1 in Appendix).

Original: We find that CSST (FixMatch) significantly outperforms the other baselines in terms of the Min.
Revised: We find that CSST (FixMatch) significantly outperforms the other baselines in terms of the Min.

Original: recall and Min.
Revised: recall and Min.

Original: Head-Tail recall for all datasets, the metrics which we aimed to optimize (Tab. 2), which shows effectiveness of CSST framework.
Revised: Head-Tail recall for all datasets, the metrics which we aimed to optimize (Tab. 2), which shows effectiveness of CSST framework.

Original: Despite optimizing worst-case recall we find that our framework is still able to maintain reasonable average (Avg.) recall in comparison to baseline vanilla (FixMatch), which demonstrates it’s practical applicability.
Revised: Despite optimizing worst-case recall we find that our framework is still able to maintain reasonable average (Avg.) recall in comparison to baseline vanilla (FixMatch), which demonstrates it’s practical applicability.

Original: We find that optimizing Min.
Revised: We find that optimizing Min.

Original: recall across NLP tasks of classification on long-tailed data by plugging UDA into CSST(UDA) framework shows similar improvement in performance (Tab. 4).
Revised: recall across NLP tasks of classification on long-tailed data by plugging UDA into CSST(UDA) framework shows similar improvement in performance (Tab. 4).

Original: This establishes the generality of our framework to even self-training methods across domain of NLP as well.
Revised: This establishes the generality of our framework to even self-training methods across domain of NLP as well.

Original: As the G is a diagonal matrix for this objective, the proposed
Revised: As the G is a diagonal matrix for this objective, the proposed

Original: KL-Based Thresholding here is equivalent to the confidence based threshold of FixMatch in this case.
Revised: KL-Based Thresholding here is equivalent to the confidence based threshold of FixMatch in this case.

Original: Despite the equivalence of thresholding mechanism, we see significant gains in min-recall (Tab. 2) just using the regularization term.
Revised: Despite the equivalence of thresholding mechanism, we see significant gains in min-recall (Tab. 2) just using the regularization term.

Original: We discuss their equivalance in Appendix Sec. H.
Revised: We discuss their equivalance in Appendix Sec. H.

Original: Maximizing Mean Recall Under Coverage Constraints.
Revised: Maximizing Mean Recall Under Coverage Constraints.

Original: Maximizing mean recall under coverage constraints objective seeks to result in a model with good average recall, yet at the same time
Revised: Maximizing mean recall under coverage constraints objective seeks to result in a model with good average recall, yet at the same time

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: vanilla

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: constraints the proportion of predictions for each class to be uniform across classes.
Revised: constraints the proportion of predictions for each class to be uniform across classes.

Original: The ideal target coverage under a balanced evaluation set(or such circumstances) is given as cov i [ F ] = 1 K , ∀ i ∈
Revised: The ideal target coverage under a balanced evaluation set(or such circumstances) is given as cov i [ F ] = 1 K , ∀ i ∈

Original: [ K ] .
Revised: [ K ] .

Original: Along similar lines to objective (10) we modify the objective (4) to maximise the average recall subject to the both the average head and tail class coverage (HT Coverage) being above a given threshold of 0 .
Revised: Along similar lines to objective (10) we modify the objective (4) to maximize the average recall subject to the both the average head and tail class coverage (HT Coverage) being above a given threshold of 0 .

Original: 95 K .
Revised: 95 K .

Original: The G is non diagonal here and G ij = ( 1 i ∈H λ H |H| + 1 i ∈T λ T |T | )
Revised: The G is non-diagonal here and G ij = ( 1 j ∈H λ H |H| + 1 j ∈T λ T |T | )

Original: + δ ij
Revised: + δ ij

Original: Kπ
Revised: Kπ

Original: i 
Revised: i .

Original: [Equation]
Revised: [Equation]

Original: As these objectives corresponds to a non-diagonal G as shown in Eq.
Revised: As these objectives corresponds to a non-diagonal G as shown in Eq.

Original: (4) in Sec. 2.1.
Revised: (4) in Sec. 2.1.

Original: Hence, for introducing CSST into FixMatch we replace first supervised loss L s with L hyb s .
Revised: Hence, for introducing CSST into FixMatch we replace first supervised loss L s with L hyb s .

Original: For the unlabeled data we introduce ℓ hyb in L wt u (Eq.
Revised: For the unlabeled data we introduce ℓ hyb in L wt u (Eq.

Original: Hence, the final objective L is defined as,
Revised: Hence, the final objective L is defined as,

Original: L =
Revised: L =

Original: L hyb s + λ u L wt u .
Revised: L hyb s + λ u L wt u .

Original: We update the parameters of the cost-sensitive loss ( G and λ ) periodically after few of SGD on the model parameters (Alg. 2 in Appendix).
Revised: We update the parameters of the cost-sensitive loss ( G and λ ) periodically after few of SGD on the model parameters (Alg. 2 in Appendix).

Original: In this case our proposed thresholding mechanism in CSST (FixMatch) introduced in Sec. 4, leads to effective utilization of unlabled data resulting in improved performance over the naive CSST (FixMatch) without (w/o) KL-Thresholding (Tab. 3).
Revised: In this case our proposed thresholding mechanism in CSST (FixMatch) introduced in Sec. 4, leads to effective utilization of unlabled data resulting in improved performance over the naive CSST (FixMatch) without (w/o) KL-Thresholding (Tab. 3).

Original: In these experiments, the mean recall of our proposed approach either improves or stays same to the vanilla (FixMatch) implementation but only ours is the one that comes close to satisfying the coverage constraint.
Revised: In these experiments, the mean recall of our proposed approach either improves or stays same to the vanilla (FixMatch) implementation but only ours is the one that comes close to satisfying the coverage constraint.

Original: We further note that among all the methods, the only methods that come close to satisfying the coverage constraints are the ones with ℓ hyb included in them.
Revised: We further note that among all the methods, the only methods that come close to satisfying the coverage constraints are the ones with ℓ hyb included in them.

Original: Ablation on amount of unlabeled data.
Revised: Ablation on amount of unlabeled data.

Original: Here, we ablated the total number of labeled sampled while keeping the number of unlabeled samples constant.
Revised: Here, we ablated the total number of labeled sampled while keeping the number of unlabeled samples constant.

Original: We observe (in Fig.
Revised: We observe (in Fig.

Original: 5a) that as we increase the number of labeled samples the mean recall improved.
Revised: 5a) that as we increase the number of labeled samples the mean recall improved.

Original: This is because more labeled samples helps in better pseudo-labeling on the unlabeled samples and similarly as we decrease the number of labeled sample, the models errors on the pseudo-labels increase causing a reduction in mean recall.
Revised: This is because more labeled samples helps in better pseudo-labeling on the unlabeled samples and similarly as we decrease the number of labeled sample, the models errors on the pseudo-labels increase causing a reduction in mean recall.

Original: Hence, any addtional labeled data can be easily used to improve CSST performance.
Revised: Hence, any addtional labeled data can be easily used to improve CSST performance.

Original: Ablation on τ threshold.
Revised: Ablation on τ threshold.

Original: Fig.
Revised: Fig.

Original: 5b shows that when the KL divergence threshold is too high, a large number of samples with very low degree of distribution match are used for generating the sharpened target (or pseudo-labels), this leads to worsening of mean recall as many targets are incorrect.
Revised: 5b shows that when the KL divergence threshold is too high, a large number of samples with very low degree of distribution match are used for generating the sharpened target (or pseudo-labels), this leads to worsening of mean recall as many targets are incorrect.

Original: We find that keeping a conservative of τ = 0 .
Revised: We find that keeping a conservative of τ = 0 .

Original: 3 works well across multiple experiments.
Revised: 3 works well across multiple experiments.

Original: for CIFAR-100 ( ρ = 10 , µ = 4 ) (Sec. 5).
Revised: for CIFAR-100 ( ρ = 10 , µ = 4 ) (Sec. 5).

Original: [Figure]
Revised: [Figure]

Original: =
Revised: =

Original:  Related Work
Revised:  Related Work

Original: Self-Training.
Revised: Self-Training.

Original: Self-training algorithms have been popularly used for the tasks of semi-supervised learning [1, 43, 35, 17] and unsupervised domain adaptation [32, 48].
Revised: Self-training algorithms have been popularly used for the tasks of semi-supervised learning [1, 43, 35, 17] and unsupervised domain adaptation [32, 48].

Original: In recent years several regularizers which enforce consistency in the neighborhood (either an adversarial perturbation [22] or augmentation [42]) of a given sample have further enhanced the applicability and performance of self-training methods, when used in conjunction.
Revised: In recent years several regularizers which enforce consistency in the neighborhood (either an adversarial perturbation [22] or augmentation [42]) of a given sample have further enhanced the applicability and performance of self-training methods, when used in conjunction.

Original: However, these works have focused mostly on improving the generic metric of accuracy, unlike the general non-decomposable metrics we consider.
Revised: However, these works have focused mostly on improving the generic metric of accuracy, unlike the general non-decomposable metrics we consider.

Original: Cost-Sensitive Learning.
Revised: Cost-Sensitive Learning.

Original: It refers to problem settings where the cost of error differs for a sample based on what class it belongs to.
Revised: It refers to problem settings where the cost of error differs for a sample based on what class it belongs to.

Original: These settings are very important for critical real world applications like disease diagnosis, wherein mistakenly classifying a diseased person as healthy can be disastrous.
Revised: These settings are very important for critical real world applications like disease diagnosis, wherein mistakenly classifying a diseased person as healthy can be disastrous.

Original: There have been a pletheora of techniques proposed for these which can be classified into: importance weighting [19, 44, 7] and adaptive margin [3, 44] based techniques.
Revised: There have been a pletheora of techniques proposed for these which can be classified into: importance weighting [19, 44, 7] and adaptive margin [3, 44] based techniques.

Original: For overparameterized models
Revised: For overparameterized models

Original: Narasimhan et al.
Revised: Narasimhan et al.

Original: [26] show that loss weighting based techniques are ineffective and propose a logit-adjustment based cost-sensitive loss which we also use in our framework.
Revised: [26] show that loss weighting based techniques are ineffective and propose a logit-adjustment based cost-sensitive loss which we also use in our framework.

Original: Complex Metrics for Deep Learning.
Revised: Complex Metrics for Deep Learning.

Original: There has been a prolonged effort on optimizing more complex metrics that take into account practical constraints [25, 30, 27].
Revised: There has been a prolonged effort on optimizing more complex metrics that take into account practical constraints [25, 30, 27].

Original: However most work has focused on linear models leaving scope for works in context of deep neural networks.
Revised: However most work has focused on linear models leaving scope for works in context of deep neural networks.

Original: Sanyal et al.
Revised: Sanyal et al.

Original: [34] train DNN using reweighting strategies for optimizing metrics, Huang et al.
Revised: [34] train DNN using reweighting strategies for optimizing metrics, Huang et al.

Original: [10] use a reinforcement learning strategy to optimize complex metrics, and Kumar et al.
Revised: [10] use a reinforcement learning strategy to optimize complex metrics, and Kumar et al.

Original: [16] optimize complex AUC (Area
Revised: [16] optimize complex AUC (Area

Original: Under Curve) metric for a deep neural network.
Revised: Under Curve) metric for a deep neural network.

Original: However, all these works have primarily worked in supervised learning setup and are not designed to effectively make use of available unlabeled data.
Revised: However, all these works have primarily worked in supervised learning setup and are not designed to effectively make use of available unlabeled data.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we aim to optimizethe practical non-decomposable metrics readily used in machine learning through self-training with consistency regularization a class of semi-supervised learning methods.
Revised: In this work, we aim to optimize practical non-decomposable metrics readily used in machine learning through self-training with consistency regularization, a class of semi-supervised learning methods.

Original: We introduce a cost-sensitive self-training framework ( CSST ) which involves minimizing a cost-sensitive error on pseudo labels along with consistency regularization.
Revised: We introduce a cost-sensitive self-training framework ( CSST ) that involves minimizing a cost-sensitive error on pseudo labels and consistency regularization.

Original: We show theoretically that we can obtain classifiers which can better optimize the desired non-decomposable metric than  original model used for obtaining pseudo labels, under similar data distribution assumptions as used for theoretical analysis of Self-training.
Revised: We show theoretically that we can obtain classifiers that can better optimize the desired non-decomposable metric than the original model used for obtaining pseudo labels, under similar data distribution assumptions as used for theoretical analysis of Self-training.

Original: We then apply CSST to practical and effective self-training method of FixMatch and UDA, incorporating a novel regularizer and thresholding mechanism based on  given non-decomposable objective.
Revised: We then apply CSST to practical and effective self-training method of FixMatch and UDA, incorporating a novel regularizer and thresholding mechanism based on a given non-decomposable objective.

Original: We find that CSST leads to a significant gain in performance of desired non-decomposable metric, in comparison to vanilla self-training based baseline.
Revised: We find that CSST leads to a significant gain in performance of desired non-decomposable metric, in comparison to vanilla self-training-based baseline.

Original: Analyzing the CSST framework when the distribution of unlabeled data significantly differs from labeled data, is a good direction to pursue for future work.
Revised: Analyzing the CSST framework when the distribution of unlabeled data significantly differs from labeled data is a good direction to pursue for future work.

Original: Acknowledgements: This work was supported in part by Fujitsu Research grant.
Revised: Acknowledgements: This work was supported in part by Fujitsu Research grant.

Original: Harsh Rangwani is supported by Prime Minister’s Research Fellowship (PMRF).
Revised: Harsh Rangwani is supported by Prime Minister’s Research Fellowship (PMRF).

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See Section A in Appendix (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] See Section A in Appendix (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See Section A in Appendix (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [Yes] See Section A in Appendix (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Assumption 1, Assumption 3, and Assumption 4.
Revised: [Yes] See Assumption 1, Assumption 3, and Assumption 4.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Section C, D and Section E in Appendix.
Revised: [Yes] See Section C, D and Section E in Appendix.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [No] We will release our code here: https://github.com/val-iisc/CostSensitiveSelfTraining (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [No] We will release our code here: https://github.com/val-iisc/CostSensitiveSelfTraining (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] Refer Section 5, and more detail is available in Section K in supplementary material.
Revised: [Yes] Refer Section 5, and more detail is available in Section K in supplementary material.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] As our experiments are done on neural networks which are expensive to train we haven’t included the error bars in result currently.
Revised: [No] As our experiments are done on neural networks which are expensive to train we haven’t included the error bars in result currently.

Original: However, we do plan to provide error bars for a certain subset of experiments in Appendix Tab.(d)
Revised: However, we do plan to provide error bars for a certain subset of experiments in Appendix Tab.(d)

Original: Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Section L in Appendix 4.
Revised: [Yes] See Section L in Appendix 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section L in Appendix (b) Did you mention the license of the assets?
Revised: [Yes] See Section L in Appendix (b) Did you mention the license of the assets?

Original: [Yes] See Section L in Appendix (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] See Section L in Appendix (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether
Revised: [No] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: Appendix
Revised: Appendix

Original: Table of Contents
Revised: Table of Contents

Original: C
Revised: C

Original: C. a Lemma . . . . . . . . . . . . . . . . . . . . .
Revised: C. a Lemma . . . . . . . . . . . . . . . . . . . . .

Original: E
Revised: E

Original: F Notation
Revised: F Notation

Original: G Objective
Revised: G Objective

Original: G.1 Logit Adjusted Weighted Consistency Regularizer . . . . . . . . . . . . . . . .
Revised: G.1 Logit Adjusted Weighted Consistency Regularizer . . . . . . . . . . . . . . . .

Original: G.2 CSST (FixMatch) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: G.2 CSST (FixMatch) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G.3 CSST (UDA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: G.3 CSST (UDA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: H Threshold mechanism for diagonal Gain Matrix
Revised: H Threshold mechanism for diagonal Gain Matrix

Original: I Dataset
Revised: I Dataset

Original: J Algorithms
Revised: J Algorithms

Original: K Details of Experiments and Hyper-parameters
Revised: K Details of Experiments and Hyper-parameters

Original: L Code, License, Assets and Computation Requirements
Revised: L Code, License, Assets and Computation Requirements

Original: L.1 Code and Licenses of Assets . .
Revised: L.1 Code and Licenses of Assets . .

Original: .2 Computational
Revised: .2 Computational

Original: M Statistical Analysis
Revised: M Statistical Analysis

Original: N Additional Details
Revised: N Additional Details

Original: N.1 Formal Statement Omitted in Sec.
Revised: N.1 Formal Statement Omitted in Sec.

Original: N
Revised: N

Original: O Additional Experiments
Revised: O Additional Experiments

Original: A Limitations and Negative Societal Impacts A.1 Limitations of our Work
Revised: A Limitations and Negative Societal Impacts A.1 Limitations of our Work

Original: At this point we only consider optimizing objectives through CSST which can be written as a linear combination of entries of confusion matrix.
Revised: At this point we only consider optimizing objectives through CSST which can be written as a linear combination of entries of confusion matrix.

Original: Although there are important metrics like Recall,
Revised: Although there are important metrics like Recall,

Original: Coverage etc.
Revised: Coverage etc.

Original: which can be expressed as a linear form of confusion matrix.
Revised: which can be expressed as a linear form of confusion matrix.

Original: However, there do exist important metrics like Intersection over Union (IoU), Q-Mean etc. which we don’t consider in the current work.
Revised: However, there do exist important metrics like Intersection over Union (IoU), Q-Mean etc. which we don’t consider in the current work.

Original: We leave this as an open direction for further work.
Revised: We leave this as an open direction for further work.

Original: Also in this work we considered datasets where unlabeled data distribution doesn’t significantly differ from the labeled data distribution, developing robust methods which can also take into account the distribution shift between unlabeled and labeled is an interesting direction for future work.
Revised: Also in this work we considered datasets where unlabeled data distribution doesn’t significantly differ from the labeled data distribution, developing robust methods which can also take into account the distribution shift between unlabeled and labeled is an interesting direction for future work.

Original: A.2 Negative Societal Impact
Revised: A.2 Negative Societal Impact

Original: Our work has application in fairness domain [6], where it can be used to improve performance of minority sub-groups present in data.
Revised: Our work has application in fairness domain [6], where it can be used to improve performance of minority sub-groups present in data.

Original: These fairness objectives can be practically enforced on neural networks through the proposed CSST framework.
Revised: These fairness objectives can be practically enforced on neural networks through the proposed CSST framework.

Original: However these same algorithms can be tweaked to artifically induce bias in decision making of trained neural networks, for example by ignoring performance of models on certain subgroups.
Revised: However these same algorithms can be tweaked to artifically induce bias in decision making of trained neural networks, for example by ignoring performance of models on certain subgroups.

Original: Hence, we suggest deployment of these models after through testing on all sub groups of data.
Revised: Hence, we suggest deployment of these models after through testing on all sub groups of data.

Original: B Connection between Minimization of Weighted Consistency Regularizer Loss (Eq. (8) ) and Theoretical Weighted Consistency ( R B ,w ( F ) in Sec. 3.4)
Revised: B Connection between Minimization of Weighted Consistency Regularizer Loss (Eq. (8) ) and Theoretical Weighted Consistency ( R B ,w ( F ) in Sec. 3.4)

Original: The theoretical weighted consistency regularization can be approximated under strong augmentation
Revised: The theoretical weighted consistency regularization can be approximated under strong augmentation

Original: A as R B ,w ( F ) ≈
Revised: A as R B ,w ( F ) ≈

Original: (cid:80) i,j ∈
Revised: (cid:80) i,j ∈

Original: [ K ] w ij E x ∼ P i [ 1 ( F ( A ( x ))
Revised: [ K ] w ij E x ∼ P i [ 1 ( F ( A ( x ))

Original: ̸ = F ( x ))] .
Revised: ̸ = F ( x ))] .

Original: Noting that 1 ( F ( A ( x )) ̸ = F ( x )) ≤ 1 ( F ( A ( x ))
Revised: Noting that 1 ( F ( A ( x )) ̸ = F ( x )) ≤ 1 ( F ( A ( x ))

Original: ̸
Revised: ̸

Original: = j ) + 1 ( F ( x ) ̸ = j ) for any class label j , this value of R B ,w ( F ) is bounded as follows:
Revised: = j ) + 1 ( F ( x ) ̸ = j ) for any class label j , this value of R B ,w ( F ) is bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: ] ≤
Revised: ] ≤

Original: [Equation]
Revised: [Equation]

Original: Assuming F ( x ) to be composed of a classifier made of neural network model output p m ( x ) .
Revised: Assuming F ( x ) to be composed of a classifier made of neural network model output p m ( x ) .

Original: For consistency regularization, we choose samples which have a low second term by thresholding on confidence for which we obtain the pseudo-label F ( x ) =
Revised: For consistency regularization, we choose samples which have a low second term by thresholding on confidence for which we obtain the pseudo-label F ( x ) =

Original: ˆ p m ( x ) .
Revised: ˆ p m ( x ) .

Original: In that case the minimization of (the upper bound of) R B ,w ( F ) is approximately equivalent to cost-sensitive learning (CSL):
Revised: In that case the minimization of (the upper bound of) R B ,w ( F ) is approximately equivalent to cost-sensitive learning (CSL):

Original: [Equation]
Revised: [Equation]

Original: The above objective can be equivalently expressed in form of gain matrix G where G i,j = w i,j / ( π i ) .
Revised: The above objective can be equivalently expressed in form of gain matrix G where G i,j = w i,j / ( π i ) .

Original: Hence, the above minimization can be effectively done for different gain matrices based on corresponding non-decomposable metric 2.1.
Revised: Hence, the above minimization can be effectively done for different gain matrices based on corresponding non-decomposable metric 2.1.

Original: Following Narasimhan and Menon [24], we can effectively minimize the R B ,w ( F ) using the calibrated CSL loss function by plugging in the pseudo label ˆ p m ( x ) in place of ground truth label ( y ).
Revised: Following Narasimhan and Menon [24], we can effectively minimize the R B ,w ( F ) using the calibrated CSL loss function by plugging in the pseudo label ˆ p m ( x ) in place of ground truth label ( y ).

Original: The final weighted regularization function can be seen as:
Revised: The final weighted regularization function can be seen as:

Original: [Equation]
Revised: [Equation]

Original: The above regularization loss is same as the weighted consistency regularization loss in Eq.
Revised: The above regularization loss is same as the weighted consistency regularization loss in Eq.

Original: (8), which leads to minimization of D KL ( norm ( G T ˆ p m ( x ))
Revised: (8), which leads to minimization of D KL ( norm ( G T ˆ p m ( x ))

Original: ||
Revised: ||

Original: p m ( A ( x )))
Revised: p m ( A ( x )))

Original: ∀ x ∈ X as seen in Prop. 6.
Revised: ∀ x ∈ X as seen in Prop. 6.

Original: Hence, the minimization of theoretical consistency regularization term is achieved by minimizing the weighted consistency loss.
Revised: Hence, the minimization of theoretical consistency regularization term is achieved by minimizing the weighted consistency loss.

Original: C Additional Examples and Proof of Theorem
Revised: C Additional Examples and Proof of Theorem

Original: In this section, we provide some examples for assumptions introduced in Sec. 3 and a proof of
Revised: In this section, we provide some examples for assumptions introduced in Sec. 3 and a proof of

Original: Theorem 5.
Revised: Theorem 5.

Original: We provide proof of these examples in Sec.
Revised: We provide proof of these examples in Sec.

Original: C.1 Examples for Theoretical Assumptions
Revised: C.1 Examples for Theoretical Assumptions

Original: The following example (Example 8) shows that the c -expansion (Definition 2) property is satisfied for mixtures of Gaussians and mixtures of manifolds.
Revised: The following example (Example 8) shows that the c -expansion (Definition 2) property is satisfied for mixtures of Gaussians and mixtures of manifolds.

Original: E XAMPLE 8.
Revised: E XAMPLE 8.

Original: By [40, Examples 3.4, 3.5], the c -expansion property is satisfied for mixtures of isotropic Gaussian distributions and mixtures of manifolds.
Revised: By [40, Examples 3.4, 3.5], the c -expansion property is satisfied for mixtures of isotropic Gaussian distributions and mixtures of manifolds.

Original: More precisely, in the case of mixtures of isotropic Gaussian distributions, i.e., if Q is given as mixtures of N ( τ i , 1 d I d × d ) for i = 1 , . .
Revised: More precisely, in the case of mixtures of isotropic Gaussian distributions, i.e., if Q is given as mixtures of N ( τ i , 1 d I d × d ) for i = 1 , . .

Original: . , n with some n ∈ Z ≥ 1 and τ i ∈ R d , and B ( x ) is an ℓ 2 -ball with radius r then by [2, (13)] and [40, Section
Revised: . , n with some n ∈ Z ≥ 1 and τ i ∈ R d , and B ( x ) is an ℓ 2 -ball with radius r then by [2, (13)] and [40, Section

Original: B.2], Q satisfies the c -expansion property with c ( p ) =
Revised: B.2], Q satisfies the c -expansion property with c ( p ) =

Original: R h ( p ) /p for p > 0 and h = 2 r √ d (c.f., [40, section B.2]).
Revised: R h ( p ) /p for p > 0 and h = 2 r √ d (c.f., [40, section B.2]).

Original: Here R h ( p ) = Φ(Φ − 1 ( p ) + h ) and Φ is the cumulative distribution function of the standard normal distribution on R .
Revised: Here R h ( p ) = Φ(Φ − 1 ( p ) + h ) and Φ is the cumulative distribution function of the standard normal distribution on R .

Original: In Sec. 3.4, we required the assumption that γ > 3 (Assumption 4) and remarked that it roughly requires Err w ( F pl ) is “small”.
Revised: In Sec. 3.4, we required the assumption that γ > 3 (Assumption 4) and remarked that it roughly requires Err w ( F pl ) is “small”.

Original: The following example provides explicit conditions for Err w ( F pl ) that satisfy the assumption using a toy example.
Revised: The following example provides explicit conditions for Err w ( F pl ) that satisfy the assumption using a toy example.

Original: E XAMPLE 9.
Revised: E XAMPLE 9.

Original: Using a toy example provided in Example 8, we provide conditions that satisfy the assumption γ > 3 approximately.
Revised: Using a toy example provided in Example 8, we provide conditions that satisfy the assumption γ > 3 approximately.

Original: To explain the assumption γ > 3 , we assume that P w is given as a mixture of isotropic Gaussians and B ( x ) is ℓ 2 -ball with radius r as in Example 8.
Revised: To explain the assumption γ > 3 , we assume that P w is given as a mixture of isotropic Gaussians and B ( x ) is ℓ 2 -ball with radius r as in Example 8.

Original: Furthermore, we assume that | w | 1 = 1 and Err w ( F ∗ ) is sufficiently small compared to Err w ( F pl ) .
Revised: Furthermore, we assume that | w | 1 = 1 and Err w ( F ∗ ) is sufficiently small compared to Err w ( F pl ) .

Original: Then, p w =
Revised: Then, p w =

Original: Err w ( F pl ) + Err w ( F ∗ )
Revised: Err w ( F pl ) + Err w ( F ∗ )

Original: ≈ Err w ( F pl ) .
Revised: ≈ Err w ( F pl ) .

Original: Using this approximation, since P w satisfies the c -expansion property with c ( p )
Revised: Using this approximation, since P w satisfies the c -expansion property with c ( p )

Original: =
Revised: =

Original: R 2 r √ d ( p ) /p , if r = 12 √ d then, the condition γ > 3 is satisfied when Err w ( F pl ) < 0 .
Revised: R 2 r √ d ( p ) /p , if r = 12 √ d then, the condition γ > 3 is satisfied when Err w ( F pl ) < 0 .

Original: If r = 32 √ d then, the condition γ > 3 is satisfied when Err w ( F pl ) < 0 .
Revised: If r = 32 √ d then, the condition γ > 3 is satisfied when Err w ( F pl ) < 0 .

Original: In Assumption 1, we assumed that both of Err w ( F ∗ ) and R B ,w ( F ∗ ) are small.
Revised: In Assumption 1, we assumed that both of Err w ( F ∗ ) and R B ,w ( F ∗ ) are small.

Original: The following example suggests the validity of this assumption.
Revised: The following example suggests the validity of this assumption.

Original: E XAMPLE 10.
Revised: E XAMPLE 10.

Original: In this example, we assume w is a diagonal matrix diag( w 1 , . . .
Revised: In this example, we assume w is a diagonal matrix diag( w 1 , . . .

Original: , w K ) .
Revised: , w K ) .

Original: For simplicity, we normalize w so that (cid:80) i ∈
Revised: For simplicity, we normalize w so that (cid:80) i ∈

Original: [ K ] w i = 1 .
Revised: [ K ] w i = 1 .

Original: As in [40, Example 3.4], we assume that P i is given as isotropic Gaussian distribution
Revised: As in [40, Example 3.4], we assume that P i is given as isotropic Gaussian distribution

Original: N ( τ i , 1 d I d × d ) with τ i ∈ R
Revised: N ( τ i , 1 d I d × d ) with τ i ∈ R

Original: d for i = 1 , . . .
Revised: d for i = 1 , . . .

Original: , K and B ( x ) is an ℓ 2 -ball with radius 12 √ d .
Revised: , K and B ( x ) is an ℓ 2 -ball with radius 12 √ d .

Original: Furthermore, we assume inf 1 ≤ i<j ≤ K
Revised: Furthermore, we assume inf 1 ≤ i<j ≤ K

Original: ∥ τ i − τ j ∥ 2 ≳ √ log d √ d
Revised: ∥ τ i − τ j ∥ 2 ≳ √ log d √ d

Original: and sup i,j ∈
Revised: and sup i,j ∈

Original: [ K ] w i w j = o ( d ) , where the latter assumption is valid for high dimensional datasets (e.g., image datasets).
Revised: [ K ] w i w j = o ( d ) , where the latter assumption is valid for high dimensional datasets (e.g., image datasets).

Original: Then it can be proved that there exists a classifier F such that R B ,w ( F ) =
Revised: Then it can be proved that there exists a classifier F such that R B ,w ( F ) =

Original: O ( 1 d c ) and Err w ( F ) =
Revised: O ( 1 d c ) and Err w ( F ) =

Original: O ( 1 d c ) , where c > 0 is a constant (we can take F as the Bayes-optimal classifier for Err w ).
Revised: O ( 1 d c ) , where c > 0 is a constant (we can take F as the Bayes-optimal classifier for Err w ).

Original: Thus, this suggests that Assumption 1 is valid for datasets with high dimensional instances.
Revised: Thus, this suggests that Assumption 1 is valid for datasets with high dimensional instances.

Original: The statement of Example 9 follows from numerical computation of R 2 r √ d ( p ) /p .
Revised: The statement of Example 9 follows from numerical computation of R 2 r √ d ( p ) /p .

Original: We provide proofs of Examples 8 and 10 in Sec.
Revised: We provide proofs of Examples 8 and 10 in Sec.

Original: C.4.
Revised: C.4.

Original: C.2 Proof of Theorem 5 Assuming a Lemma
Revised: C.2 Proof of Theorem 5 Assuming a Lemma

Original: Theorem 5 can be deduced from the following lemma (by taking H
Revised: Theorem 5 can be deduced from the following lemma (by taking H

Original: = F ∗ and L Q,H ( (cid:98) F ) ≤
Revised: = F ∗ and L Q,H ( (cid:98) F ) ≤

Original: L Q,H ( F ∗ ) ), which provides a similar result to [40, Lemma A.8].
Revised: L Q,H ( F ∗ ) ), which provides a similar result to [40, Lemma A.8].

Original: L EMMA 11.
Revised: L EMMA 11.

Original: Let H be a classifier and Q a probability measure on X satisfying c -expansion property.
Revised: Let H be a classifier and Q a probability measure on X satisfying c -expansion property.

Original: We put γ H
Revised: We put γ H

Original: = c ( Q ( { x ∈ X :
Revised: = c ( Q ( { x ∈ X :

Original: F pl ( x ) ̸ = H ( x ) } )) .
Revised: F pl ( x ) ̸ = H ( x ) } )) .

Original: For a classifier F , we define S B ( F ) by
Revised: For a classifier F , we define S B ( F ) by

Original: S B ( F ) = { x ∈ X : F ( x ) =
Revised: S B ( F ) = { x ∈ X : F ( x ) =

Original: F ( x ′ ) ∀ x ′ ∈ B ( x ) } .
Revised: F ( x ′ ) ∀ x ′ ∈ B ( x ) } .

Original: For a classifier F , we define L Q,H ( F ) by γ H +
Revised: For a classifier F , we define L Q,H ( F ) by γ H +

Original: [Equation]
Revised: [Equation]

Original: where S c B ( F ) denotes the complement of S B ( F ) .
Revised: where S c B ( F ) denotes the complement of S B ( F ) .

Original: Then, we have Q ( { x ∈ X : F ( x ) ̸
Revised: Then, we have Q ( { x ∈ X : F ( x ) ̸

Original: = H ( x ) } ) ≤
Revised: = H ( x ) } ) ≤

Original: L Q,H ( F ) for any classifier F .
Revised: L Q,H ( F ) for any classifier F .

Original: In this subsection, we provide a proof of Theorem 5 assuming Lemma 11.
Revised: In this subsection, we provide a proof of Theorem 5 assuming Lemma 11.

Original: We provide a proof of the lemma in the next subsection.
Revised: We provide a proof of the lemma in the next subsection.

Original: For a classifier F , we define M ( F ) as { x ∈ X : F ( x ) ̸ = F ∗ ( x ) } and
Revised: For a classifier F , we define M ( F ) as { x ∈ X : F ( x ) ̸ = F ∗ ( x ) } and

Original: M pl ( F ) as { x ∈ X : F ( x ) ̸ = F pl ( x ) } .
Revised: M pl ( F ) as { x ∈ X : F ( x ) ̸ = F pl ( x ) } .

Original: We define (cid:101)
Revised: We define (cid:101)

Original: L w ( F ) by (cid:101)
Revised: L w ( F ) by (cid:101)

Original: L w ( F ) =
Revised: L w ( F ) =

Original: L w ( F ) + 2 γ γ − 1 R B ,w ( F ∗ ) − P w ( { x ∈ X : F pl ( x ) ̸ = F ∗ ( x ) } ) .
Revised: L w ( F ) + 2 γ γ − 1 R B ,w ( F ∗ ) − P w ( { x ∈ X : F pl ( x ) ̸ = F ∗ ( x ) } ) .

Original: We note that (cid:101)
Revised: We note that (cid:101)

Original: L w ( F ) − L w ( F ) does not depend on F .
Revised: L w ( F ) − L w ( F ) does not depend on F .

Original: Proof of Theorem 5.
Revised: Proof of Theorem 5.

Original: We let Q = P w and H =
Revised: We let Q = P w and H =

Original: F ∗ in Lemma 11 and denote γ H in the lemma by γ ′ .
Revised: F ∗ in Lemma 11 and denote γ H in the lemma by γ ′ .

Original: Since w ij ≥ 0 and E x ∼ P i [ 1 ( F pl ( x ) ̸ = F ∗ ( x ))] ≤ E x ∼ P i [ 1 ( F pl ( x ) ̸ = j )]
Revised: Since w ij ≥ 0 and E x ∼ P i [ 1 ( F pl ( x ) ̸ = F ∗ ( x ))] ≤ E x ∼ P i [ 1 ( F pl ( x ) ̸ = j )]

Original: + E x ∼ P i [ 1 ( F ∗ ( x )
Revised: + E x ∼ P i [ 1 ( F ∗ ( x )

Original: ̸ = j )] for any i, j , we have the following:
Revised: ̸ = j )] for any i, j , we have the following:

Original: [Equation]
Revised: [Equation]

Original: Here, the first equation follows from the definition of P w .
Revised: Here, the first equation follows from the definition of P w .

Original: Thus, we obtain P w ( M ( F pl ))
Revised: Thus, we obtain P w ( M ( F pl ))

Original: ≤ p w .
Revised: ≤ p w .

Original: By definition of γ and γ ′ ( γ = c ( p w ) and γ ′ =
Revised: By definition of γ and γ ′ ( γ = c ( p w ) and γ ′ =

Original: c ( P w ( M ( F pl ))) ) and the assumption that c is non-increasing, we have the following:
Revised: c ( P w ( M ( F pl ))) ) and the assumption that c is non-increasing, we have the following:

Original: [Equation]
Revised: [Equation]

Original: We note that
Revised: We note that

Original: [Equation]
Revised: [Equation]

Original: Here the first inequality holds since 1 ( F ( x ) ̸ = j ) ≤ 1 ( F ( x ) ̸ = F ∗ ( x )) + 1 ( F ∗ ( x ) ̸ = j ) for any x and j .
Revised: Here the first inequality holds since 1 ( F ( x ) ̸ = j ) ≤ 1 ( F ( x ) ̸ = F ∗ ( x )) + 1 ( F ∗ ( x ) ̸ = j ) for any x and j .

Original: By (14) and Lemma 11, the error is upper bounded as follows:
Revised: By (14) and Lemma 11, the error is upper bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: By (13)
Revised: By (13)

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: By definition of L w and letting F = (cid:98) F
Revised: By definition of L w and letting F = (cid:98) F

Original: [Equation]
Revised: [Equation]

Original: we have the following:
Revised: we have the following:

Original: [Equation]
Revised: [Equation]

Original: Here, the second inequality holds since (cid:98) F is a minimizer of L w , the third inequality follows from( F ∗ ( x ) ̸ = F pl ( x )) ≤ 

Revised: Here, the second inequality holds since (cid:98) F is a minimizer of L w , the third inequality follows from( F ∗ ( x ) ̸ = F pl ( x )) ≤

Original: ( F ∗ ( x ) ̸ = j ) + 1 ( F pl ( x ) ̸ = j ) for any j .
Revised: ( F ∗ ( x ) ̸ = j ) + 1 ( F pl ( x ) ̸ = j ) for any j .

Original: Thus, we have the assertion of the theorem.
Revised: Thus, we have the assertion of the theorem.

Original: C.3 Proof of Lemma
Revised: C.3 Proof of Lemma

Original: We decompose M ( F ) ∩ S B ( F ) ∩ S B ( H ) into the following three sets:
Revised: We decompose M ( F ) ∩ S B ( F ) ∩ S B ( H ) into the following three sets:

Original: N 1 = { x ∈ S B ( F ) ∩ S B ( H ) :
Revised: N 1 = { x ∈ S B ( F ) ∩ S B ( H ) :

Original: F ( x ) =
Revised: F ( x ) =

Original: F pl ( x ) , and F pl ( x ) ̸
Revised: F pl ( x ) , and F pl ( x ) ̸

Original: = H ( x ) } ,
Revised: = H ( x ) } ,

Original: N 2 = { x ∈ S B ( F ) ∩ S B ( H ) :
Revised: N 2 = { x ∈ S B ( F ) ∩ S B ( H ) :

Original: F ( x ) ̸ = F pl ( x ) , F pl ( x ) ̸
Revised: F ( x ) ̸ = F pl ( x ) , F pl ( x ) ̸

Original: = H ( x ) , and F ( x ) ̸
Revised: = H ( x ) , and F ( x ) ̸

Original: = H ( x ) } ,
Revised: = H ( x ) } ,

Original: N 3 = { x ∈ S B ( F ) ∩ S B ( H ) :
Revised: N 3 = { x ∈ S B ( F ) ∩ S B ( H ) :

Original: F ( x ) ̸
Revised: F ( x ) ̸

Original: = F pl ( x ) and F pl ( x ) =
Revised: = F pl ( x ) and F pl ( x ) =

Original: H ( x ) } .
Revised: H ( x ) } .

Original: L EMMA 12.
Revised: L EMMA 12.

Original: Let S
Revised: Let S

Original: =
Revised: =

Original: S B ( F ) ∩ S B ( H ) and V =
Revised: S B ( F ) ∩ S B ( H ) and V =

Original: M ( F ) ∩ M ( F pl ) ∩ S .
Revised: M ( F ) ∩ M ( F pl ) ∩ S .

Original: Then, we have N ( V ) ∩
Revised: Then, we have N ( V ) ∩

Original: M c ( F ) ∩ S = ∅ and N ( V ) ∩ M c ( F pl ) ∩ S ⊆
Revised: M c ( F ) ∩ S = ∅ and N ( V ) ∩ M c ( F pl ) ∩ S ⊆

Original: M pl ( F ) .
Revised: M pl ( F ) .

Original: Here M pl ( F ) is defined as { x ∈ X :
Revised: Here M pl ( F ) is defined as { x ∈ X :

Original: F ( x ) ̸
Revised: F ( x ) ̸

Original: = F pl ( x ) } .
Revised: = F pl ( x ) } .

Original: Proof.
Revised: Proof.

Original: We take any element x in N ( V ) ∩ S .
Revised: We take any element x in N ( V ) ∩ S .

Original: Since x ∈ N ( V ) and definition of neighborhoods, there exists x ′ ∈ M ( F pl ) ∩ M ( F ) ∩ S such that B ( x ) ∩ B ( x ′ ) ̸
Revised: Since x ∈ N ( V ) and definition of neighborhoods, there exists x ′ ∈ M ( F pl ) ∩ M ( F ) ∩ S such that B ( x ) ∩ B ( x ′ ) ̸

Original: = ∅ .
Revised: = ∅ .

Original: Since x, x ′ ∈ S B ( F ) , F takes the same values on B ( x ) and B ( x ′ ) .
Revised: Since x, x ′ ∈ S B ( F ) , F takes the same values on B ( x ) and B ( x ′ ) .

Original: By B ( x ) ∩ B ( x ′ ) ̸
Revised: By B ( x ) ∩ B ( x ′ ) ̸

Original: = ∅ , F takes the same value on B ( x ) ∪ B ( x ′ ) .
Revised: = ∅ , F takes the same value on B ( x ) ∪ B ( x ′ ) .

Original: It follows that F ( x ) =
Revised: It follows that F ( x ) =

Original: F ( x ′ ) .
Revised: F ( x ′ ) .

Original: Since we have x, x ′ ∈ S B ( H ) , similarly, we see that H ( x )
Revised: Since we have x, x ′ ∈ S B ( H ) , similarly, we see that H ( x )

Original: =
Revised: =

Original: H ( x ′ ) .
Revised: H ( x ′ ) .

Original: By x ′ ∈ M ( F ) , we have F ( x ) =
Revised: By x ′ ∈ M ( F ) , we have F ( x ) =

Original: F ( x ′ ) ̸
Revised: F ( x ′ ) ̸

Original: = H ( x ′ )
Revised: = H ( x ′ )

Original: =
Revised: =

Original: H ( x ) .
Revised: H ( x ) .

Original: Therefore, we have x ∈ M ( F ) .
Revised: Therefore, we have x ∈ M ( F ) .

Original: Thus, we have proved that x ∈ N ( V ) ∩ S implies x ∈ M ( F ) , i.e., N ( V ) ∩ S ⊆ M ( F ) .
Revised: Thus, we have proved that x ∈ N ( V ) ∩ S implies x ∈ M ( F ) , i.e., N ( V ) ∩ S ⊆ M ( F ) .

Original: Therefore, we have N ( V ) ∩ M c ( F ) ∩ S = ∅ .
Revised: Therefore, we have N ( V ) ∩ M c ( F ) ∩ S = ∅ .

Original: This completes the proof of the first statement.
Revised: This completes the proof of the first statement.

Original: Next, we assume x ∈ N ( V )
Revised: Next, we assume x ∈ N ( V )

Original: ∩M c
Revised: ∩M c

Original: ( F pl ) ∩ S .
Revised: ( F pl ) ∩ S .

Original: Then, we have F ( x ) ̸ = H ( x ) and F pl ( x ) = H ( x ) .
Revised: Then, we have F ( x ) ̸ = H ( x ) and F pl ( x ) = H ( x ) .

Original: Therefore, we obtain
Revised: Therefore, we obtain

Original: F ( x ) ̸ = F pl ( x ) .
Revised: F ( x ) ̸ = F pl ( x ) .

Original: This completes the proof.
Revised: This completes the proof.

Original: L EMMA 13.
Revised: L EMMA 13.

Original: Suppose that assumptions of Lemma 11 hold.
Revised: Suppose that assumptions of Lemma 11 hold.

Original: We define q as follows:
Revised: We define q as follows:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then, we have Q ( S B ( F ) ∩ S B ( H )
Revised: Then, we have Q ( S B ( F ) ∩ S B ( H )

Original: ∩ M ( F pl ) ∩ M ( F )) ≤ q .
Revised: ∩ M ( F pl ) ∩ M ( F )) ≤ q .

Original: In particular, noting that N 1 ∪ N 2 ⊆
Revised: In particular, noting that N 1 ∪ N 2 ⊆

Original: S B ( F ) ∩ S B ( H )
Revised: S B ( F ) ∩ S B ( H )

Original: ∩ M ( F pl ) ∩ M ( F ) , we have Q ( N 1 ∪ N 2 ) ≤ q .
Revised: ∩ M ( F pl ) ∩ M ( F ) , we have Q ( N 1 ∪ N 2 ) ≤ q .

Original: Proof.
Revised: Proof.

Original: We let S = S B ( F ) ∩ S B ( H ) and V = M ( F ) ∩ M ( F pl ) ∩ S as before.
Revised: We let S = S B ( F ) ∩ S B ( H ) and V = M ( F ) ∩ M ( F pl ) ∩ S as before.

Original: Then by Lemma 12, we have
Revised: Then by Lemma 12, we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: Thus, by the c -expansion property, we have
Revised: Thus, by the c -expansion property, we have

Original: [Equation]
Revised: [Equation]

Original: Since V ⊆ M ( F pl ) , c is non-increasing, and γ H > 1 , we have Q ( V ) ≤ Q ( M pl ( F ) ∪ S c ) /
Revised: Since V ⊆ M ( F pl ) , c is non-increasing, and γ H > 1 , we have Q ( V ) ≤ Q ( M pl ( F ) ∪ S c ) /

Original: ( γ H − 1) ≤ q .
Revised: ( γ H − 1) ≤ q .

Original: This completes the proof.
Revised: This completes the proof.

Original: The following lemma provides an upper bound of Q ( N 3 ) .
Revised: The following lemma provides an upper bound of Q ( N 3 ) .

Original: L EMMA 14.
Revised: L EMMA 14.

Original: Suppose that the assumptions of Lemma 11 hold.
Revised: Suppose that the assumptions of Lemma 11 hold.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where q is defined by (16) .
Revised: where q is defined by (16) .

Original: Proof.
Revised: Proof.

Original: We let S = S B ( F ) ∩ S B ( H ) .
Revised: We let S = S B ( F ) ∩ S B ( H ) .

Original: First, we prove
Revised: First, we prove

Original: [Equation]
Revised: [Equation]

Original: Here, for sets A, B , we denote union
Revised: Here, for sets A, B , we denote union

Original: A ∪ B by A ⊔ B if the union is disjoint.
Revised: A ∪ B by A ⊔ B if the union is disjoint.

Original: By definition, we have
Revised: By definition, we have

Original: N 1 = S
Revised: N 1 = S

Original: ∩ M c pl ( F ) ∩ M ( F pl ) and N 3 = S ∩ M pl ( F ) ∩ M c ( F pl ) .
Revised: ∩ M c pl ( F ) ∩ M ( F pl ) and N 3 = S ∩ M pl ( F ) ∩ M c ( F pl ) .

Original: Thus, we have
Revised: Thus, we have

Original: [Equation]
Revised: [Equation]

Original: Similarly, we have the following:
Revised: Similarly, we have the following:

Original: [Equation]
Revised: [Equation]

Original: Since disjointness is obvious by definition
Revised: Since disjointness is obvious by definition

Original: [Equation]
Revised: [Equation]

Original: we obtain (17).
Revised: we obtain (17).

Original: Next
Revised: Next

Original: [Equation]
Revised: [Equation]

Original: we note that the following holds:
Revised: we note that the following holds:

Original: [Equation]
Revised: [Equation]

Original: By (17)
Revised: By (17)

Original: [Equation]
Revised: [Equation]

Original: we obtain the following:
Revised: we obtain the following:

Original: [Equation]
Revised: [Equation]

Original: Here the second inequality follows from (18) and the third inequality follows from Lemma 13.
Revised: Here the second inequality follows from (18) and the third inequality follows from Lemma 13.

Original: This completes the proof.
Revised: This completes the proof.

Original: Now, we can prove Lemma 11 as follows.
Revised: Now, we can prove Lemma 11 as follows.

Original: Proof of Lemma 11.
Revised: Proof of Lemma 11.

Original: [Equation]
Revised: [Equation]

Original: Here, the last inequality follows from Lemmas 13 and 14.
Revised: Here, the last inequality follows from Lemmas 13 and 14.

Original: Since q satisfies q ≤
Revised: Since q satisfies q ≤

Original: [Equation]
Revised: [Equation]

Original: by (16), we have our assertion.
Revised: by (16), we have our assertion.

Original: C.4 Miscellaneous Proofs for Examples
Revised: C.4 Miscellaneous Proofs for Examples

Original: Proof of Example 8.
Revised: Proof of Example 8.

Original: In Example 8, we stated that p (cid:55)→ R h ( p ) /p is non-increasing.
Revised: In Example 8, we stated that p (cid:55)→ R h ( p ) /p is non-increasing.

Original: This follows from the concavity of R h and lim p → +0 R h ( p ) = 0 .
Revised: This follows from the concavity of R h and lim p → +0 R h ( p ) = 0 .

Original: In fact, we can prove the concavity of R h by d 2 R h dp 2 ( p ) = − h exp (cid:16) ξ 2 2 − hξ − 12 h 2 (cid:17) ≤ 0 , where ξ = Φ − 1 ( p ) .
Revised: In fact, we can prove the concavity of R h by d 2 R h dp 2 ( p ) = − h exp (cid:16) ξ 2 2 − hξ − 12 h 2 (cid:17) ≤ 0 , where ξ = Φ − 1 ( p ) .

Original: Proof of Example 10.
Revised: Proof of Example 10.

Original: For each i, j ∈ [ K ] , w i P i ( x ) ≥ w j P j ( x ) is equivalent to ( x − τ i ) · v ji ≤ ∥ τ
Revised: For each i, j ∈ [ K ] , w i P i ( x ) ≥ w j P j ( x ) is equivalent to ( x − τ i ) · v ji ≤ ∥ τ

Original: i
Revised: i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , where v ji = τ j − τ i ∥ τ i − τ j ∥ .
Revised: , where v ji = τ j − τ i ∥ τ i − τ j ∥ .

Original: Thus, for each i ∈ [ K ] , we have (cid:84)
Revised: Thus, for each i ∈ [ K ] , we have (cid:84)

Original: j ∈
Revised: j ∈

Original: [ K ] \{ i } X ij ⊆
Revised: [ K ] \{ i } X ij ⊆

Original: S B ( F opt ) .
Revised: S B ( F opt ) .

Original: Here X ij is defined as { x ∈ X : ( x − τ i ) · v ji ≤ ∥ τ i − τ j ∥+ 2(log w i − log w j ) d ∥ τ i − τ j ∥ − r 2 } .
Revised: Here X ij is defined as { x ∈ X : ( x − τ i ) · v ji ≤ ∥ τ i − τ j ∥+ 2(log w i − log w j ) d ∥ τ i − τ j ∥ − r 2 } .

Original: For any w ∈ R d with ∥ w ∥ 2 = √ d and a > 0 , we have P i ( { x ∈ X : ( x − τ i ) · w > a } )
Revised: For any w ∈ R d with ∥ w ∥ 2 = √ d and a > 0 , we have P i ( { x ∈ X : ( x − τ i ) · w > a } )

Original: = 1 − Φ( a ) ≤ 12 exp( − a 2 / 2) (c.f., [5]).
Revised: = 1 − Φ( a ) ≤ 12 exp( − a 2 / 2) (c.f., [5]).

Original: Thus, P i ( X cij ) ≤ 12 exp( − da 2 ij / 2) , where a ij = ∥
Revised: Thus, P i ( X cij ) ≤ 12 exp( − da 2 ij / 2) , where a ij = ∥

Original: τ i − τ j ∥+ 2(log w i − log w j ) d ∥ τ i − τ j ∥ − r 2 .
Revised: τ i − τ j ∥+ 2(log w i − log w j ) d ∥ τ i − τ j ∥ − r 2 .

Original: By assumptions, we have a ij √ d ≳ √ log d .
Revised: By assumptions, we have a ij √ d ≳ √ log d .

Original: Therefore, P i ( X cij )
Revised: Therefore, P i ( X cij )

Original: =
Revised: =

Original: O ( 1 poly ( d ) ) .
Revised: O ( 1 poly ( d ) ) .

Original: It follows that P i ( S c B ( F opt )) ≤ (cid:80)
Revised: It follows that P i ( S c B ( F opt )) ≤ (cid:80)

Original: j ∈ [ K ] \{ i } P i ( X cij ) =
Revised: j ∈ [ K ] \{ i } P i ( X cij ) =

Original: O ( 1 poly ( d ) ) .
Revised: O ( 1 poly ( d ) ) .

Original: Thus, we have R B ,w ( F opt ) =
Revised: Thus, we have R B ,w ( F opt ) =

Original: O ( 1 poly ( d ) ) .
Revised: O ( 1 poly ( d ) ) .

Original: By the same way, we can prove that Err w ( F opt ) =
Revised: By the same way, we can prove that Err w ( F opt ) =

Original: O ( 1 poly ( d ) ) .
Revised: O ( 1 poly ( d ) ) .

Original: D All-Layer Margin Generalization Bounds
Revised: D All-Layer Margin Generalization Bounds

Original: Following [40, 39], we introduce all layer margin of neural networks and provide generalization bounds of CSST .
Revised: Following [40, 39], we introduce all layer margin of neural networks and provide generalization bounds of CSST .

Original: In this section, we assume that classifier F ( x ) is given as F ( x ) = argmax 1 ≤ i ≤ K Φ i ( x ) , where Φ is a neural network of the form Φ
Revised: In this section, we assume that classifier F ( x ) is given as F ( x ) = argmax 1 ≤ i ≤ K Φ i ( x ) , where Φ is a neural network of the form Φ

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here f i : R d i − 1 → R d i with d 0 = d and d p = K .
Revised: Here f i : R d i − 1 → R d i with d 0 = d and d p = K .

Original: We assume that each f i belongs to a function class
Revised: We assume that each f i belongs to a function class

Original: F i ⊂ Map( R d i − 1 , R d i ) .
Revised: F i ⊂ Map( R d i − 1 , R d i ) .

Original: We define a function class F to which Φ belongs by
Revised: We define a function class F to which Φ belongs by

Original: [Equation]
Revised: [Equation]

Original: For example, for b > 0 , F i is given as { h (cid:55)→
Revised: For example, for b > 0 , F i is given as { h (cid:55)→

Original: Wϕ
Revised: Wϕ

Original: ( h ) :
Revised: ( h ) :

Original: W ∈ R d i − 1 ×
Revised: W ∈ R d i − 1 ×

Original: d
Revised: d

Original: i , ∥ W ∥ fro ≤ b } if i >and { h (cid:55)→
Revised: i , ∥ W ∥ fro ≤ b } if i >and { h (cid:55)→

Original: Wh : W ∈ R d 0 × d 1 , ∥ W ∥ fro ≤ b } if i = 1 , where ϕ is a link function (applied on R d i entry-wise) with bounded operator norm (i.e., ∥ ϕ ∥ op :=
Revised: Wh : W ∈ R d 0 × d 1 , ∥ W ∥ fro ≤ b } if i = 1 , where ϕ is a link function (applied on R d i entry-wise) with bounded operator norm (i.e., ∥ ϕ ∥ op :=

Original: sup x ∈ R di \{ 0 } ∥ ϕ ( x ) ∥ 2 / ∥ x ∥ 2 < ∞ ) and ∥ W ∥ fro denotes the Frobenius norm of the matrix.
Revised: sup x ∈ R di \{ 0 } ∥ ϕ ( x ) ∥ 2 / ∥ x ∥ 2 < ∞ ) and ∥ W ∥ fro denotes the Frobenius norm of the matrix.

Original: However, we do not assume the function class
Revised: However, we do not assume the function class

Original: F i does not have this specific form.
Revised: F i does not have this specific form.

Original: We assume that each function class F i is a normed vector space with norm ∥ · ∥ .
Revised: We assume that each function class F i is a normed vector space with norm ∥ · ∥ .

Original: In the example above, we consider the operator norm, i.e., if f ( h ) =
Revised: In the example above, we consider the operator norm, i.e., if f ( h ) =

Original: ϕ
Revised: ϕ

Original: ( Wh ) ,
Revised: ( Wh ) ,

Original: ∥ f ∥ is defined as ∥ f ∥ op .
Revised: ∥ f ∥ is defined as ∥ f ∥ op .

Original: Let x 1 , . . .
Revised: Let x 1 , . . .

Original: , x n be a finite i.i.d.
Revised: , x n be a finite i.i.d.

Original: sequence of samples drawn from P w .
Revised: sequence of samples drawn from P w .

Original: We denote the corresponding empirical distribution by (cid:98) P w , i.e., for a measurable function f on X ,
Revised: We denote the corresponding empirical distribution by (cid:98) P w , i.e., for a measurable function f on X ,

Original: E x ∼ (cid:98) P w [ f ] = (cid:80) ni
Revised: E x ∼ (cid:98) P w [ f ] = (cid:80) ni

Original: =1 f ( x i ) .
Revised: =1 f ( x i ) .

Original: For ξ = ( ξ 1 , . . . , ξ p ) ∈ (cid:81) pi =1 R d i , we define the perturbed output Φ( x, ξ ) as Φ( x, ξ ) =
Revised: For ξ = ( ξ 1 , . . . , ξ p ) ∈ (cid:81) pi =1 R d i , we define the perturbed output Φ( x, ξ ) as Φ( x, ξ ) =

Original: h p ( x, ξ ) , where
Revised: h p ( x, ξ ) , where

Original: [Equation]
Revised: [Equation]

Original: Let x ∈ X and y ∈
Revised: Let x ∈ X and y ∈

Original: [ K ] .
Revised: [ K ] .

Original: We define Ξ(Φ , x, y ) by { ξ ∈ (cid:81) pi
Revised: We define Ξ(Φ , x, y ) by { ξ ∈ (cid:81) pi

Original: =1 R d i : argmax i Φ i ( x, ξ )
Revised: =1 R d i : argmax i Φ i ( x, ξ )

Original: ̸ = y } .
Revised: ̸ = y } .

Original: Then, the all-layer margin m (Φ , x, y ) is defined as
Revised: Then, the all-layer margin m (Φ , x, y ) is defined as

Original: [Equation]
Revised: [Equation]

Original: where ∥ ξ ∥ 2 is given by (cid:112)(cid:80) pi =1 ∥ ξ i ∥ 22 .
Revised: where ∥ ξ ∥ 2 is given by (cid:112)(cid:80) pi =1 ∥ ξ i ∥ 22 .

Original: Following [40], we define a variant of the all-layer margin that measures robustness of Φ with respect to input transformations defined by B ( x ) as follows:
Revised: Following [40], we define a variant of the all-layer margin that measures robustness of Φ with respect to input transformations defined by B ( x ) as follows:

Original: [Equation]
Revised: [Equation]

Original: A SSUMPTION 15 (c.f. [39], Condition A.1) .
Revised: A SSUMPTION 15 (c.f. [39], Condition A.1) .

Original: Let G be a normed space with norm ∥ · ∥ and ϵ > 0 .
Revised: Let G be a normed space with norm ∥ · ∥ and ϵ > 0 .

Original: We say G satisfies the ϵ − 2 covering condition with complexity C ∥·∥ ( G ) if for all ϵ > 0 , we have
Revised: We say G satisfies the ϵ − 2 covering condition with complexity C ∥·∥ ( G ) if for all ϵ > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Here N ∥·∥ ( ϵ, G ) the ϵ -covering number of G .
Revised: Here N ∥·∥ ( ϵ, G ) the ϵ -covering number of G .

Original: We assume function class F i satisfies the ϵ − 2 covering condition with complexity C ∥·∥ ( F i ) for each 1 ≤ i ≤ p .
Revised: We assume function class F i satisfies the ϵ − 2 covering condition with complexity C ∥·∥ ( F i ) for each 1 ≤ i ≤ p .

Original: Throughout this section, we suppose that Assumption 15 holds.
Revised: Throughout this section, we suppose that Assumption 15 holds.

Original: Essentially, the following two propositions follows were proved by Wei et al.
Revised: Essentially, the following two propositions follows were proved by Wei et al.

Original: P ROPOSITION 16 (c.f., [40], Lemma D.6) .
Revised: P ROPOSITION 16 (c.f., [40], Lemma D.6) .

Original: With probability at least 1 − δ over the draw of the training data, for all t ∈ (0 , ∞ ) , any Φ ∈ F satisfies the following:
Revised: With probability at least 1 − δ over the draw of the training data, for all t ∈ (0 , ∞ ) , any Φ ∈ F satisfies the following:

Original: [Equation]
Revised: [Equation]

Original: where ζ = O (cid:18)(cid:113) log(1 /δ )
Revised: where ζ = O (cid:18)(cid:113) log(1 /δ )

Original: +log n n (cid:19) is a lower order term and F ( x ) =
Revised: +log n n (cid:19) is a lower order term and F ( x ) =

Original: argmax i ∈
Revised: argmax i ∈

Original: [ K ] Φ i ( x ) .
Revised: [ K ] Φ i ( x ) .

Original: P ROPOSITION 17 (c.f., [40], Theorem D.3) .
Revised: P ROPOSITION 17 (c.f., [40], Theorem D.3) .

Original: With probability at least 1 − δ over the draw of the training data, for all t ∈ (0 , ∞ ) , any Φ ∈ F satisfies the following:
Revised: With probability at least 1 − δ over the draw of the training data, for all t ∈ (0 , ∞ ) , any Φ ∈ F satisfies the following:

Original: [Equation]
Revised: [Equation]

Original: where ζ = O (cid:18)(cid:113) log(1 /δ )
Revised: where ζ = O (cid:18)(cid:113) log(1 /δ )

Original: +log n n (cid:19) is a lower order term and F ( x ) =
Revised: +log n n (cid:19) is a lower order term and F ( x ) =

Original: argmax i ∈
Revised: argmax i ∈

Original: [ K ] Φ i ( x ) .
Revised: [ K ] Φ i ( x ) .

Original: R EMARK .
Revised: R EMARK .

Original: Although we have proved Theorem 5 following [40], we had to provide our own proof due to some differences in theoretical assumptions (e.g., in our case there does not necessarily exist the ground-truth classifier, although they assumed the expansion property for each P i , we assume the expansion property for the weighted probability measure).
Revised: Although we have proved Theorem 5 following [40], we had to provide our own proof due to some differences in theoretical assumptions (e.g., in our case there does not necessarily exist the ground-truth classifier, although they assumed the expansion property for each P i , we assume the expansion property for the weighted probability measure).

Original: On the other hand, the proofs of [40,
Revised: On the other hand, the proofs of [40,

Original: Lemma D.6] and [40, Theorem D.3] work for any distribution P on X and its empirical distribution (cid:98) P .
Revised: Lemma D.6] and [40, Theorem D.3] work for any distribution P on X and its empirical distribution (cid:98) P .

Original: Since ∥ w ∥ 1 P w ( S c B ( F ))
Revised: Since ∥ w ∥ 1 P w ( S c B ( F ))

Original: =
Revised: =

Original: R B ,w ( F ) and ∥ w ∥ 1 P w ( { x : F ( x ) ̸
Revised: R B ,w ( F ) and ∥ w ∥ 1 P w ( { x : F ( x ) ̸

Original: = F pl ( x ) } )
Revised: = F pl ( x ) } )

Original: =
Revised: =

Original: L w ( F, F pl ) ,
Revised: L w ( F, F pl ) ,

Original: Proposition 16 and Proposition 17 follow from the corresponding results in [40].
Revised: Proposition 16 and Proposition 17 follow from the corresponding results in [40].

Original: T HEOREM 18.
Revised: T HEOREM 18.

Original: Suppose Assumptions 3, 4, and 15 hold.
Revised: Suppose Assumptions 3, 4, and 15 hold.

Original: Then, with probability at least 1 − δ over the draw of the training data, for all t 1 , t 2 ∈ (0 , ∞ ) , and any neural network Φ in F , we have the following:
Revised: Then, with probability at least 1 − δ over the draw of the training data, for all t 1 , t 2 ∈ (0 , ∞ ) , and any neural network Φ in F , we have the following:

Original: [Equation]
Revised: [Equation]

Original: where ζ = O (cid:18)(cid:113) log(1 /δ )
Revised: where ζ = O (cid:18)(cid:113) log(1 /δ )

Original: +log n n (cid:19) is a lower order term and F ( x ) =
Revised: +log n n (cid:19) is a lower order term and F ( x ) =

Original: argmax i ∈
Revised: argmax i ∈

Original: [ K ] Φ i ( x ) .
Revised: [ K ] Φ i ( x ) .

Original: Proof.
Revised: Proof.

Original: By (15) with H = F ∗ and − 1 ( F pl ( x ) ̸ = F ∗ ( x )) ≤ − 1 ( F pl ( x ) ̸ = j ) + 1 ( F ∗ ( x )
Revised: By (15) with H = F ∗ and − 1 ( F pl ( x ) ̸ = F ∗ ( x )) ≤ − 1 ( F pl ( x ) ̸ = j ) + 1 ( F ∗ ( x )

Original: ̸ = j ) for any x ∈ X and j ∈ [ K ] , we obtain the following inequality:
Revised: ̸ = j ) for any x ∈ X and j ∈ [ K ] , we obtain the following inequality:

Original: [Equation]
Revised: [Equation]

Original: Then, the statement of the theorem follows from Proposition 16 and Proposition 17.
Revised: Then, the statement of the theorem follows from Proposition 16 and Proposition 17.

Original: E Proof of Proposition
Revised: E Proof of Proposition

Original: Proof.
Revised: Proof.

Original: Let the average weighted consistency loss be L wtu = 1 | B
Revised: Let the average weighted consistency loss be L wtu = 1 | B

Original: | (cid:80) x ∈ B
Revised: | (cid:80) x ∈ B

Original: [Equation]
Revised: [Equation]

Original: ( x ) , p m ( A ( x )) , G ) this will be minimized if for each of x ∈ B the ℓ wtu (ˆ p m ( x ) , p m ( A ( x )) , G ) is minimized.
Revised: ( x ) , p m ( A ( x )) , G ) this will be minimized if for each of x ∈ B the ℓ wtu (ˆ p m ( x ) , p m ( A ( x )) , G ) is minimized.

Original: This expression can be expanded as:
Revised: This expression can be expanded as:

Original: [Equation]
Revised: [Equation]

Original: Here we use H to denote the cross entropy between two distributions.
Revised: Here we use H to denote the cross entropy between two distributions.

Original: As we don’t backpropogate gradients from the ˆ p m ( x ) (pseudo-label) branch of prediction network we can consider
Revised: As we don’t backpropogate gradients from the ˆ p m ( x ) (pseudo-label) branch of prediction network we can consider

Original: C = (cid:80) mj
Revised: C = (cid:80) mj

Original: =1 G T ˆ p m ( x ))
Revised: =1 G T ˆ p m ( x ))

Original: j as a constant in our analysis.
Revised: j as a constant in our analysis.

Original: Also adding a constant term of entropy
Revised: Also adding a constant term of entropy

Original: H( norm ( G T ˆ p m ( x ))) to cross entropy term and dropping constant C doesn’t change the outcome of minimization.
Revised: H( norm ( G T ˆ p m ( x ))) to cross entropy term and dropping constant C doesn’t change the outcome of minimization.

Original: Hence we have the following:
Revised: Hence we have the following:

Original: [Equation]
Revised: [Equation]

Original: This final term is the D KL ( G T ˆ p m ( x ) ||
Revised: This final term is the D KL ( G T ˆ p m ( x ) ||

Original: p m ( A ( x )) which is obtained by using the identity
Revised: p m ( A ( x )) which is obtained by using the identity

Original: D KL ( p, q ) =
Revised: D KL ( p, q ) =

Original: H( p, q ) + H( p ) where p, q are the two distributions.
Revised: H( p, q ) + H( p ) where p, q are the two distributions.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Instance
Revised: Instance

Original: [Table]
Revised: [Table]

Original: Number
Revised: Number

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: classifier model
Revised: classifier model

Original: [Table]
Revised: [Table]

Original: distribution
Revised: distribution

Original: [Table]
Revised: [Table]

Original: Lagrange multiplier
Revised: Lagrange multiplier

Original: [Table]
Revised: [Table]

Original: : coefficient of unlabeled loss rec i
Revised: : coefficient of unlabeled loss rec i

Original: [ F ] : recall of i th class for a classifier F acc [ F ] : accuracy for a classifier F prec i
Revised: [ F ] : recall of i th class for a classifier F acc [ F ] : accuracy for a classifier F prec i

Original: [ F ] : precision of i th class for a classifier F
Revised: [ F ] : precision of i th class for a classifier F

Original: [Table]
Revised: [Table]

Original: i [ F ]
Revised: i [ F ]

Original: [Table]
Revised: [Table]

Original: coverage for i th class for a classifier F
Revised: coverage for i th class for a classifier F

Original: [Table]
Revised: [Table]

Original: → X function that is stochastic in nature and applies a strong augmentation to it
Revised: → X function that is stochastic in nature and applies a strong augmentation to it

Original: [Table]
Revised: [Table]

Original: → X function that is stochastic in nature and applies a weak augmentation to it
Revised: → X function that is stochastic in nature and applies a weak augmentation to it

Original: [Table]
Revised: [Table]

Original: an
Revised: an

Original: [Table]
Revised: [Table]

Original: sample
Revised: sample

Original: [Table]
Revised: [Table]

Original: function
Revised: function

Original: [Table]
Revised: [Table]

Original: distribution of confidence for a model’s prediction on a given sample
Revised: distribution of confidence for a model’s prediction on a given sample

Original: [Table]
Revised: [Table]

Original: K × K weight matrix that corresponds to a gain matrix G
Revised: K × K weight matrix that corresponds to a gain matrix G

Original: [Table]
Revised: [Table]

Original: weighted error of F that corresponds to the objective of CSL
Revised: weighted error of F that corresponds to the objective of CSL

Original: [Table]
Revised: [Table]

Original: weighted distribution on X
Revised: weighted distribution on X

Original: [Table]
Revised: [Table]

Original: : class conditional distribution of samples for class
Revised: : class conditional distribution of samples for class

Original: i
Revised: i

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ) : theoretical weighted (cost sensitive) consistency regularizer
Revised: ) : theoretical weighted (cost sensitive) consistency regularizer

Original: F pl : a pseudo labeler
Revised: F pl : a pseudo labeler

Original: L w ( F, F pl ) : weighted error
Revised: L w ( F, F pl ) : weighted error

Original: between
Revised: between

Original: F and F pl
Revised: F and F pl

Original: L w ( F ) : theoretical CSST loss
Revised: L w ( F ) : theoretical CSST loss

Original: c : a non-increasing function used in the definition of the c -expansion property (Definition 2) γ : a value of c defined in Assumptionβ : an upper bound of R B ,w ( F ) in the optimization problem (6) S c : the complement of a set S
Revised: c : a non-increasing function used in the definition of the c -expansion property (Definition 2) γ : a value of c defined in Assumptionβ : an upper bound of R B ,w ( F ) in the optimization problem (6) S c : the complement of a set S

Original: F Notation
Revised: F Notation

Original: We provide the list of notations commonly used in the paper in Table
Revised: We provide the list of notations commonly used in the paper in Table

Original: [Equation]
Revised: [Equation]

Original: G Objective G.1 Logit Adjusted Weighted Consistency Regularizer
Revised: G Objective G.1 Logit Adjusted Weighted Consistency Regularizer

Original: As we have introduced weighted consistency regularizer in Eq.
Revised: As we have introduced weighted consistency regularizer in Eq.

Original: (8) for utilizing unlabeled data, we now provide logit adjusted variant of it for training deep networks in this section.
Revised: (8) for utilizing unlabeled data, we now provide logit adjusted variant of it for training deep networks in this section.

Original: We provide logit adjusted term for ℓ wt u (ˆ p m ( x ) , p m ( A ( x ) , G ) below:
Revised: We provide logit adjusted term for ℓ wt u (ˆ p m ( x ) , p m ( A ( x ) , G ) below:

Original: [Equation]
Revised: [Equation]

Original: The above expression comes from the decomposition G = MD .
Revised: The above expression comes from the decomposition G = MD .

Original: The above loss function can be converted into it’s logit adjusted equivalent variant by following transformation as suggested by
Revised: The above loss function can be converted into it’s logit adjusted equivalent variant by following transformation as suggested by

Original: Narasimhan and Menon [24] which is equivalent in terms of optimization of deep neural networks:
Revised: Narasimhan and Menon [24] which is equivalent in terms of optimization of deep neural networks:

Original: [Equation]
Revised: [Equation]

Original: The above loss is the consistency loss ℓ wt u that we practically implement for CSST .
Revised: The above loss is the consistency loss ℓ wt u that we practically implement for CSST .

Original: Further in case ˆ p m ( x ) is a hard pseudo label y as in FixMatch, the above weighted consistency loss reduces to ℓ hyb ( y, p m ( A ( x ))) .
Revised: Further in case ˆ p m ( x ) is a hard pseudo label y as in FixMatch, the above weighted consistency loss reduces to ℓ hyb ( y, p m ( A ( x ))) .

Original: Further in case the gain matrix G is diagonal the above loss will converge to ℓ LA ( y, p m ( A ( x ))) .
Revised: Further in case the gain matrix G is diagonal the above loss will converge to ℓ LA ( y, p m ( A ( x ))) .

Original: Thus the weighted consistency regularizer can be converted to logit adjusted variants ℓ LA and ℓ hyb based on G matrix.
Revised: Thus the weighted consistency regularizer can be converted to logit adjusted variants ℓ LA and ℓ hyb based on G matrix.

Original: G.2 CSST (FixMatch)
Revised: G.2 CSST (FixMatch)

Original: In FixMatch, we use the prediction made by the model on a sample x after applying a weak augmentation α and is used to get a hard pseudo label for the models prediction on a strongly augmented sample i.e. A ( x ) .
Revised: In FixMatch, we use the prediction made by the model on a sample x after applying a weak augmentation α and is used to get a hard pseudo label for the models prediction on a strongly augmented sample i.e. A ( x ) .

Original: The set of weak augmentations include horizontal flip, We shall refer to this pseudo label as ˆ p m ( x ) .
Revised: The set of weak augmentations include horizontal flip, We shall refer to this pseudo label as ˆ p m ( x ) .

Original: The list of strong augmentations are given in Table 12 of Sohn et al.
Revised: The list of strong augmentations are given in Table 12 of Sohn et al.

Original: Weak augmentations include padding, random horizontal flip and cropping to the desired dimensions (32X32 for CIFAR and 224X224 for ImageNet).
Revised: Weak augmentations include padding, random horizontal flip and cropping to the desired dimensions (32X32 for CIFAR and 224X224 for ImageNet).

Original: Given a batch of labeled and unlabeled samples B s and B u , CSST modifies the supervised and un-supervised component of the loss function depending upon the non-decomposable objective and its corresponding gain matrix G at a given time during training.
Revised: Given a batch of labeled and unlabeled samples B s and B u , CSST modifies the supervised and un-supervised component of the loss function depending upon the non-decomposable objective and its corresponding gain matrix G at a given time during training.

Original: We assume that in the dataset, a sample x , be it labeled or unlabeled is already weakly augmented.
Revised: We assume that in the dataset, a sample x , be it labeled or unlabeled is already weakly augmented.

Original: vanilla FixMatch’s supervised componenet of the loss function is a simple cross entropy loss whereas in our CSST (FixMatch) it is replaced by ℓ hyb s .
Revised: vanilla FixMatch’s supervised componenet of the loss function is a simple cross entropy loss whereas in our CSST (FixMatch) it is replaced by ℓ hyb s .

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The component of the loss function for unlabeled data (i.e. consistency regularization) is where one of our contributions w.r.t the novel thresholding mechanism comes into light.
Revised: The component of the loss function for unlabeled data (i.e. consistency regularization) is where one of our contributions w.r.t the novel thresholding mechanism comes into light.

Original: vanilla FixMatch selects unlabeled samples for which consistency loss is non-zero, such that the model’s confidence on the most likely predicted class is above a certain threshold.
Revised: vanilla FixMatch selects unlabeled samples for which consistency loss is non-zero, such that the model’s confidence on the most likely predicted class is above a certain threshold.

Original: We rather go for a threshold mechanism that select based on the basis of degree of distribution match to a target distribution based on G .
Revised: We rather go for a threshold mechanism that select based on the basis of degree of distribution match to a target distribution based on G .

Original: The final loss function L =
Revised: The final loss function L =

Original: L hybs + λ u L wt u , i.e. a linear combination of L hyb s and L wtu .
Revised: L hybs + λ u L wt u , i.e. a linear combination of L hyb s and L wtu .

Original: Since for
Revised: Since for

Original: FixMatch we are dealing with Wide-ResNets and ResNets which are deep networks, as mentioned in Section G.1, we shall use the alternate logit adjusted formulation as mentioned in Eq.
Revised: FixMatch we are dealing with Wide-ResNets and ResNets which are deep networks, as mentioned in Section G.1, we shall use the alternate logit adjusted formulation as mentioned in Eq.

Original: (19) as substitute for ℓ wt u in Eq.
Revised: (19) as substitute for ℓ wt u in Eq.

Original: G.3 CSST (UDA)
Revised: G.3 CSST (UDA)

Original: The loss function of UDA is a linear combination of supervised loss and consistency loss on unlabeled samples.
Revised: The loss function of UDA is a linear combination of supervised loss and consistency loss on unlabeled samples.

Original: The former is the cross entropy (CE) loss, while the latter for the unlabeled samples minimizes the KL-divergence between the model’s predicted label distribution on an input sample and its augmented sample.
Revised: The former is the cross entropy (CE) loss, while the latter for the unlabeled samples minimizes the KL-divergence between the model’s predicted label distribution on an input sample and its augmented sample.

Original: Often the predicted label distribution on the unaugmented sample is sharpened.
Revised: Often the predicted label distribution on the unaugmented sample is sharpened.

Original: The augmentation we used was a English-French-English backtranslation based on the
Revised: The augmentation we used was a English-French-English backtranslation based on the

Original: MarianMT [11] fast neural machine translation model.
Revised: MarianMT [11] fast neural machine translation model.

Original: In UDA supervised component of the loss is annealed using a method described as Training Signal Annealing (TSA), where the CE loss is considered only for those labeled samples whose max i p m ( x ) i < τ t , where t is a training time step.
Revised: In UDA supervised component of the loss is annealed using a method described as Training Signal Annealing (TSA), where the CE loss is considered only for those labeled samples whose max i p m ( x ) i < τ t , where t is a training time step.

Original: We observed that using TSA in a long tailed setting leads to overfitting on the head classes and hence chose to not include the same in our final implementation.
Revised: We observed that using TSA in a long tailed setting leads to overfitting on the head classes and hence chose to not include the same in our final implementation.

Original: CSST modifies the supervised and unsupervised component of the loss function in UDA depending upon a given objective and its corresponding gain matrix G at a given time during training.
Revised: CSST modifies the supervised and unsupervised component of the loss function in UDA depending upon a given objective and its corresponding gain matrix G at a given time during training.

Original: The supervised component of the loss function for a given constrained optimization problem and a gain matrix G , is the hybrid loss ℓ hyb s .
Revised: The supervised component of the loss function for a given constrained optimization problem and a gain matrix G , is the hybrid loss ℓ hyb s .

Original: For the consistency regularizer part of the loss function, we minimize the KL-divergence between a target distribution and the model’s prediction label distribution on its augmented version.
Revised: For the consistency regularizer part of the loss function, we minimize the KL-divergence between a target distribution and the model’s prediction label distribution on its augmented version.

Original: The target distribution is norm ( G T ˆ p m ( x )) , where ˆ p m ( x ) is the sharpened prediction of the label distribution by the model.
Revised: The target distribution is norm ( G T ˆ p m ( x )) , where ˆ p m ( x ) is the sharpened prediction of the label distribution by the model.

Original: Given a batch of labeled and unlabeled samples
Revised: Given a batch of labeled and unlabeled samples

Original: B s and B u , the final loss function in CSST (UDA) is a linear combination of L hybs and L wtu , i.e
Revised: B s and B u , the final loss function in CSST (UDA) is a linear combination of L hybs and L wtu , i.e

Original: L = L hybs + λ u L wtu .
Revised: L = L hybs + λ u L wtu .

Original: [Equation]
Revised: [Equation]

Original: Since for UDA, we are dealing with DistilBERT, as mentioned in Section G.1, we shall use the alternate formulation as mentioned in Eq.
Revised: Since for UDA, we are dealing with DistilBERT, as mentioned in Section G.1, we shall use the alternate formulation as mentioned in Eq.

Original: (19) as substitute for ℓ wt u in Eq.
Revised: (19) as substitute for ℓ wt u in Eq.

Original: above.
Revised: above.

Original: H Threshold mechanism for diagonal Gain Matrix
Revised: H Threshold mechanism for diagonal Gain Matrix

Original: Consider the case when the gain matrix is a diagonal matrix.
Revised: Consider the case when the gain matrix is a diagonal matrix.

Original: The loss function L wtu ( B u ) as defined in (9) makes uses of a threshold function that selects samples based on the KL divergence based threshold between the target distribution as defined by the gain matrix G and the models predicted distribution of confidence over the classes.
Revised: The loss function L wtu ( B u ) as defined in (9) makes uses of a threshold function that selects samples based on the KL divergence based threshold between the target distribution as defined by the gain matrix G and the models predicted distribution of confidence over the classes.

Original: [Equation]
Revised: [Equation]

Original: Since G is a diagonal matrix and the pseudo-label ˆ p m ( x ) is one hot, the norm ( G T ˆ p m ( x )) is a one-hot vector.
Revised: Since G is a diagonal matrix and the pseudo-label ˆ p m ( x ) is one hot, the norm ( G T ˆ p m ( x )) is a one-hot vector.

Original: The threshold function’s KL divergence based criterion can be expanded as follows where ˆ y is the pseudo-label’s maximum class’s index:
Revised: The threshold function’s KL divergence based criterion can be expanded as follows where ˆ y is the pseudo-label’s maximum class’s index:

Original: [Equation]
Revised: [Equation]

Original: The above equations represents a threshold on the negative log-confidence of the model’s prediction for a given unlabeled sample, for the pseudo-label class ( ˆ y ).
Revised: The above equations represents a threshold on the negative log-confidence of the model’s prediction for a given unlabeled sample, for the pseudo-label class ( ˆ y ).

Original: This can be further simplified to p m ( x ) ˆ y ≥ exp ( − τ ) which is simply a threshold based on the model’s confidence.
Revised: This can be further simplified to p m ( x ) ˆ y ≥ exp ( − τ ) which is simply a threshold based on the model’s confidence.

Original: Since pseudolabel is generated from the model’s prediction, this threshold is nothing but a selection criterion to select only those samples whose maximum confidence for a predicted hard pseudo-label is above a fixed threshold.
Revised: Since pseudolabel is generated from the model’s prediction, this threshold is nothing but a selection criterion to select only those samples whose maximum confidence for a predicted hard pseudo-label is above a fixed threshold.

Original: This is identical to the threshold function which is used in Fixmatch [35] i.e. max( p m ( x )) ≥ exp( − τ ) .
Revised: This is identical to the threshold function which is used in Fixmatch [35] i.e. max( p m ( x )) ≥ exp( − τ ) .

Original: In FixMatch this exp( − τ ) is set to 0.95.
Revised: In FixMatch this exp( − τ ) is set to 0.95.

Original: I Dataset
Revised: I Dataset

Original: CIFAR-10 and CIFAR-100 [15].
Revised: CIFAR-10 and CIFAR-100 [15].

Original: are image classification datasets of images of size 32 X 32.
Revised: are image classification datasets of images of size 32 X 32.

Original: Both the datasets have a size of 50k samples and by default, they have a uniform sample distribution among its classes.
Revised: Both the datasets have a size of 50k samples and by default, they have a uniform sample distribution among its classes.

Original: CIFAR-10 has 10 classes while CIFAR-100 has 100 classes.
Revised: CIFAR-10 has 10 classes while CIFAR-100 has 100 classes.

Original: The test set is a balanced set of 10k images.
Revised: The test set is a balanced set of 10k images.

Original: ImageNet-100 [31]. is an image classification dataset carved out of ImageNet-1k by selecting the first 100 classes.
Revised: ImageNet-100 [31]. is an image classification dataset carved out of ImageNet-1k by selecting the first 100 classes.

Original: The distribution of samples is uniform with 1.3k samples per class.
Revised: The distribution of samples is uniform with 1.3k samples per class.

Original: The test set contains 50 images per class.
Revised: The test set contains 50 images per class.

Original: All have a resolution of 224X224, the same as the original ImageNet-1k dataset.
Revised: All have a resolution of 224X224, the same as the original ImageNet-1k dataset.

Original: IMDb[20].
Revised: IMDb[20].

Original: dataset is a binary text sentiment classification dataset.
Revised: dataset is a binary text sentiment classification dataset.

Original: The data distribution is uniform by default and has a total 25k samples in both trainset and testset.
Revised: The data distribution is uniform by default and has a total 25k samples in both trainset and testset.

Original: In this work, we converted the dataset into a longtailed version of ρ = 10 , 100 and selected 1k labeled samples while truncating the labels of the rest and using them as unlabeled samples.
Revised: In this work, we converted the dataset into a longtailed version of ρ = 10 , 100 and selected 1k labeled samples while truncating the labels of the rest and using them as unlabeled samples.

Original: DBpedia-14[18]. is a topic classification dataset with a uniform distribution of labeled samples.
Revised: DBpedia-14[18]. is a topic classification dataset with a uniform distribution of labeled samples.

Original: The dataset has 14 classes and has a total of 560k samples in the trainset and 70k samples in the test set.
Revised: The dataset has 14 classes and has a total of 560k samples in the trainset and 70k samples in the test set.

Original: Each sample, apart from the content, also has title of the article that could be used for the task of topic classification.
Revised: Each sample, apart from the content, also has title of the article that could be used for the task of topic classification.

Original: In our experiments, we only make use of the content.
Revised: In our experiments, we only make use of the content.

Original: J Algorithms
Revised: J Algorithms

Original: We provide a detailed description of algorithms used for optimizing non decomposable objectives through CSST (FixMatch)
Revised: We provide a detailed description of algorithms used for optimizing non decomposable objectives through CSST (FixMatch)

Original: ans CSST (UDA).
Revised: ans CSST (UDA).

Original: Algorithm 1 is used for experiments in Section 5 for maximizing worst-case recall (i.e. min recall using CSST (FixMatch) and CSST (UDA)).
Revised: Algorithm 1 is used for experiments in Section 5 for maximizing worst-case recall (i.e. min recall using CSST (FixMatch) and CSST (UDA)).

Original: Algorithm 2 is used for experiments in Section 5 for maximizing recall under coverage constraints (i.e. min coverage experiments on CIFAR10-LT, CIFAR100-LT and ImageNet100-LT).
Revised: Algorithm 2 is used for experiments in Section 5 for maximizing recall under coverage constraints (i.e. min coverage experiments on CIFAR10-LT, CIFAR100-LT and ImageNet100-LT).

Original: Inputs: Training set S s (labeled) and S u (unlabeled) , Validation set S val , Step-size ω ∈ R + , Class priors π
Revised: Inputs: Training set S s (labeled) and S u (unlabeled) , Validation set S val , Step-size ω ∈ R + , Class priors π

Original: Initialize: Classifier F 0 , Multipliers λ 0 ∈ ∆ K −for t = 0 to T − 1 do
Revised: Initialize: Classifier F 0 , Multipliers λ 0 ∈ ∆ K −for t = 0 to T − 1 do

Original: Update λ : λ t +1 i
Revised: Update λ : λ t +1 i

Original: = λ
Revised: = λ

Original: ti exp ( − ω · recall i [ F t ]) , ∀ i, λ = norm ( λ )
Revised: ti exp ( − ω · recall i [ F t ]) , ∀ i, λ = norm ( λ )

Original: G = diag( λ t +11 /π 1 , . .
Revised: G = diag( λ t +11 /π 1 , . .

Original: λ t +1 K /π K )
Revised: λ t +1 K /π K )

Original: Compute ℓ wt u , ℓ hyb s using G
Revised: Compute ℓ wt u , ℓ hyb s using G

Original: Cost-sensitive Learning (CSL) for FixMatch:
Revised: Cost-sensitive Learning (CSL) for FixMatch:

Original: B u ∼ S u , B s ∼ S s //
Revised: B u ∼ S u , B s ∼ S s //

Original: Sample batches of data
Revised: Sample batches of data

Original: F t +1 ∈ arg min F (cid:80) B u ,B s λ u L wt u + L hyb s //
Revised: F t +1 ∈ arg min F (cid:80) B u ,B s λ u L wt u + L hyb s //

Original: Replaced by few steps of SGD end for return F T
Revised: Replaced by few steps of SGD end for return F T

Original: [Table]
Revised: [Table]

Original: Algorithm 2 CSST -based Algorithm for Maximizing Mean Recall s.t.
Revised: Algorithm 2 CSST -based Algorithm for Maximizing Mean Recall s.t.

Original: per class coverage > 0.95/K
Revised: per class coverage > 0.95/K

Original: Inputs: Training set S s (labeled) and S u (unlabeled) , Validation set S val , Step-size ω ∈ R + , Class priors π
Revised: Inputs: Training set S s (labeled) and S u (unlabeled) , Validation set S val , Step-size ω ∈ R + , Class priors π

Original: Initialize: Classifier F 0 , Multipliers λ 0 ∈ R K + for t = 0 to T − 1 do
Revised: Initialize: Classifier F 0 , Multipliers λ 0 ∈ R K + for t = 0 to T − 1 do

Original: Update λ :
Revised: Update λ :

Original: [Equation]
Revised: [Equation]

Original: t +1 i
Revised: t +1 i

Original: [Equation]
Revised: [Equation]

Original: Projection
Revised: Projection

Original: [Equation]
Revised: [Equation]

Original: Compute ℓ wt u , ℓ hyb s using G
Revised: Compute ℓ wt u , ℓ hyb s using G

Original: Cost-sensitive Learning (CSL) for FixMatch:
Revised: Cost-sensitive Learning (CSL) for FixMatch:

Original: B u ∼ S u , B s ∼ S s //
Revised: B u ∼ S u , B s ∼ S s //

Original: Sample batches of data
Revised: Sample batches of data

Original: [Equation]
Revised: [Equation]

Original: // Replaced by few steps of SGD end for return F T
Revised: // Replaced by few steps of SGD end for return F T

Original: K Details of Experiments and Hyper-parameters
Revised: K Details of Experiments and Hyper-parameters

Original: The experiment of max F min i recall i [ f ] and max F recall [ F ] s.t. cov
Revised: The experiment of max F min i recall i [ f ] and max F recall [ F ] s.t. cov

Original: i [ F ] > 0 .
Revised: i [ F ] > 0 .

Original: 95 K , ∀ i ∈
Revised: 95 K , ∀ i ∈

Original: [ K ] was performed on the long tailed version of CIFAR-10, IMDb( ρ = 10 , 100 ) and DBpediadatasets.
Revised: [ K ] was performed on the long tailed version of CIFAR-10, IMDb( ρ = 10 , 100 ) and DBpediadatasets.

Original: This was because the optimization of the aforementioned 2 objectives is stable for cases with low number of classes.
Revised: This was because the optimization of the aforementioned 2 objectives is stable for cases with low number of classes.

Original: Hence the objective of max F min( recall H [ F ] , recall T [ F ]) and max F recall [ F ] s.t.
Revised: Hence the objective of max F min( recall H [ F ] , recall T [ F ]) and max F recall [ F ] s.t.

Original: min H , T cov H , T [ F ] > 0 .
Revised: min H , T cov H , T [ F ] > 0 .

Original: 95 K is a relatively easier objective for datasets with large number of classes, hence were the optimization objectives for CIFAR-100 and ImageNet-100 long tailed datasets.
Revised: 95 K is a relatively easier objective for datasets with large number of classes, hence were the optimization objectives for CIFAR-100 and ImageNet-100 long tailed datasets.

Original: For all experiments for a given dataset, we used the same values for a given common hyperparameter.
Revised: For all experiments for a given dataset, we used the same values for a given common hyperparameter.

Original: We ablated the threshold for our novel unlabeled sample selection criterion( τ ) and the ratio of labeled and unlabeled samples, given fixed number of unlabeled samples( µ ) and are available in Fig. 4b.
Revised: We ablated the threshold for our novel unlabeled sample selection criterion( τ ) and the ratio of labeled and unlabeled samples, given fixed number of unlabeled samples( µ ) and are available in Fig. 4b.

Original: L Code, License, Assets and Computation Requirements L.1 Code and Licenses of Assets
Revised: L Code, License, Assets and Computation Requirements L.1 Code and Licenses of Assets

Original: In this work
Revised: In this work

Original: [Table]
Revised: [Table]

Original: we use the open source implementation of FixMatch [35] 3 in PyTorch, which is licensed under MIT License for educational purpose.
Revised: we use the open source implementation of FixMatch [35] 3 in PyTorch, which is licensed under MIT License for educational purpose.

Original: Also for NLP experiments we make use of
Revised: Also for NLP experiments we make use of

Original: DistillBERT [33] pretrained model available in the HuggingFace [41] library.
Revised: DistillBERT [33] pretrained model available in the HuggingFace [41] library.

Original: The code to reproduce the main experiments results can be found at https://github.com/val-iisc/CostSensitiveSelfTraining.
Revised: The code to reproduce the main experiments results can be found at https://github.com/val-iisc/CostSensitiveSelfTraining.

Original: L.2 Computational Requirements
Revised: L.2 Computational Requirements

Original: [Table] [Table]
Revised: [Table] [Table]

Original: done on NLP datasets.
Revised: done on NLP datasets.

Original: The DistilBERT model which we are using is pretrained on a language modeling task, hence it requires much less time for training in comparison to vision models which are trained from scratch.
Revised: The DistilBERT model which we are using is pretrained on a language modeling task, hence it requires much less time for training in comparison to vision models which are trained from scratch.

Original: All experiments were done on a variety of GPUs, with primarily Nvidia A5000 (24GB) with occasional use of Nvidia A100 (80GB) and Nvidia RTX3090 (24GB).
Revised: All experiments were done on a variety of GPUs, with primarily Nvidia A5000 (24GB) with occasional use of Nvidia A100 (80GB) and Nvidia RTX3090 (24GB).

Original: For finetuning DistilBERT and all experiments with ImageNet-100 dataset we used PyTorch data parallel over 4 A5000s.
Revised: For finetuning DistilBERT and all experiments with ImageNet-100 dataset we used PyTorch data parallel over 4 A5000s.

Original: Training was done till no significant change in metrics was observed.
Revised: Training was done till no significant change in metrics was observed.

Original: The detailed list of computation used per experiment type and dataset have been tabulated in Table 7 and Table 8.
Revised: The detailed list of computation used per experiment type and dataset have been tabulated in Table 7 and Table 8.

Original: M Statistical Analysis
Revised: M Statistical Analysis

Original: We establish the statistical soundness and validity of our results we ran our experiments on 3 different seeds.
Revised: We establish the statistical soundness and validity of our results we ran our experiments on 3 different seeds.

Original: Due to the computational requirements for some of the experiments ( ≈ 2days) we chose to run the experiments on multiple seeds for a subset of tasks i.e. for maximising the minimum recall among all classes for CIFAR-10 LT.
Revised: Due to the computational requirements for some of the experiments ( ≈ 2days) we chose to run the experiments on multiple seeds for a subset of tasks i.e. for maximising the minimum recall among all classes for CIFAR-10 LT.

Original: We observe that the std. deviation is significantly smaller than the average values for mean recall and min.
Revised: We observe that the std. deviation is significantly smaller than the average values for mean recall and min.

Original: recall and our performance metrics fall within our std.
Revised: recall and our performance metrics fall within our std.

Original: deviation hence validating the stability and soundness of training.
Revised: deviation hence validating the stability and soundness of training.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: N Additional Details
Revised: N Additional Details

Original: N.1 Formal Statement Omitted in Sec. 2.
Revised: N.1 Formal Statement Omitted in Sec. 2.

Original: In Sec. 2.2, we stated that learning with the hybrid loss ℓ hyb gives the Bayes-optimal classifier for the
Revised: In Sec. 2.2, we stated that learning with the hybrid loss ℓ hyb gives the Bayes-optimal classifier for the

Original: CSL (2).
Revised: CSL (2).

Original: However, due to space constraint, we did not provide a formal statement.
Revised: However, due to space constraint, we did not provide a formal statement.

Original: In this section, we provide a formal statement of it for clarity.
Revised: In this section, we provide a formal statement of it for clarity.

Original: P ROPOSITION ([24] Proposition 4) .
Revised: P ROPOSITION ([24] Proposition 4) .

Original: For any diagonal matrix D ∈ R K × K with D ii > 0 , ∀ i ,
Revised: For any diagonal matrix D ∈ R K × K with D ii > 0 , ∀ i ,

Original: M ∈ R K × K , and G = MD , the hybrid loss ℓ hyb is calibrated for G .
Revised: M ∈ R K × K , and G = MD , the hybrid loss ℓ hyb is calibrated for G .

Original: That is, for any modelprediction (cid:98) p m : X → R K that minimizes E ( x,y ) ∼ D (cid:2) ℓ hyb ( y, (cid:98) p m ( x )) (cid:3) , the associated classifier
Revised: That is, for any modelprediction (cid:98) p m : X → R K that minimizes E ( x,y ) ∼ D (cid:2) ℓ hyb ( y, (cid:98) p m ( x )) (cid:3) , the associated classifier

Original: F ( x ) = argmax y ∈
Revised: F ( x ) = argmax y ∈

Original: [ K ] ( (cid:98) p m ) i ( x ) is the Bayes-optimal classifier for CSL (2).
Revised: [ K ] ( (cid:98) p m ) i ( x ) is the Bayes-optimal classifier for CSL (2).

Original: N.2 Comparison with the ( a, (cid:101) c ) -expansion Property in [40]
Revised: N.2 Comparison with the ( a, (cid:101) c ) -expansion Property in [40]

Original: We compare the c -expansion property with ( a, (cid:101) c ) -expansion property proposed by [40], where a ∈ (0 , 1) and (cid:101) c > 1 .
Revised: We compare the c -expansion property with ( a, (cid:101) c ) -expansion property proposed by [40], where a ∈ (0 , 1) and (cid:101) c > 1 .

Original: Here we say a distribution Q on X satisfies the ( a, (cid:101) c ) -expansion property if
Revised: Here we say a distribution Q on X satisfies the ( a, (cid:101) c ) -expansion property if

Original: Q ( N ( S )) ≥ (cid:101) c for any S ⊂ X with Q ( S ) ≤ a .
Revised: Q ( N ( S )) ≥ (cid:101) c for any S ⊂ X with Q ( S ) ≤ a .

Original: If Q satisfies ( a, (cid:101) c ) -expansion property [40] with (cid:101) c > 1 , then Q satisfies the c -expansion property, where the function c is defined as follows.
Revised: If Q satisfies ( a, (cid:101) c ) -expansion property [40] with (cid:101) c > 1 , then Q satisfies the c -expansion property, where the function c is defined as follows.

Original: c ( p ) =
Revised: c ( p ) =

Original: (cid:101) c if p ≤ a and c ( p ) = 1 otherwise.
Revised: (cid:101) c if p ≤ a and c ( p ) = 1 otherwise.

Original: On the other hand, if Q -satisfies c -expansion property, then for any a ∈ (0 , 1) and S ⊆ X with Q ( S ) ≤ a , we have Q ( N ( S )) ≥ c ( Q ( S ))
Revised: On the other hand, if Q -satisfies c -expansion property, then for any a ∈ (0 , 1) and S ⊆ X with Q ( S ) ≤ a , we have Q ( N ( S )) ≥ c ( Q ( S ))

Original: Q ( S ) ≥ c ( a ) Q ( S ) since c is non-increasing.
Revised: Q ( S ) ≥ c ( a ) Q ( S ) since c is non-increasing.

Original: Therefore, Q satisfies the ( a, c ( a )) -expansion property.
Revised: Therefore, Q satisfies the ( a, c ( a )) -expansion property.

Original: Thus, we could say these two conditions are equivalent.
Revised: Thus, we could say these two conditions are equivalent.

Original: To simplify our analysis, we use our definition of the expansion property.
Revised: To simplify our analysis, we use our definition of the expansion property.

Original: In addition, Wei et al.
Revised: In addition, Wei et al.

Original: [40] showed that the ( a, (cid:101) c ) -expansion property is realistic for vision.
Revised: [40] showed that the ( a, (cid:101) c ) -expansion property is realistic for vision.

Original: Although they assumed the ( a, c ) -expansion property for each P
Revised: Although they assumed the ( a, c ) -expansion property for each P

Original: i (1 ≤ i ≤ K ) and we assume the c -expansion property for P w , it follows that the c -expansion property for P w is also realistic for vision, since P w is a linear combination of P i .
Revised: i (1 ≤ i ≤ K ) and we assume the c -expansion property for P w , it follows that the c -expansion property for P w is also realistic for vision, since P w is a linear combination of P i .

Original: O Additional Experiments
Revised: O Additional Experiments

Original: In this section, we compare CSST (FixMatch) against contemporary semi-supervised learning techniques namely CReST [38], DARP [12], vanilla (FixMatch)
Revised: In this section, we compare CSST (FixMatch) against contemporary semi-supervised learning techniques namely CReST [38], DARP [12], vanilla (FixMatch)

Original: [35] and FlexMatch [46].
Revised: [35] and FlexMatch [46].

Original: We compared these methods on the long-tailed CIFAR-10 ( ρ = 100) and CIFAR-100 ( ρ = 10) datasets.
Revised: We compared these methods on the long-tailed CIFAR-10 ( ρ = 100) and CIFAR-100 ( ρ = 10) datasets.

Original: The objective for long-tailed CIFAR-10 dataset was to maximise the worst-case recall (3) and average recall, subject to a per-class coverage constraint (4).
Revised: The objective for long-tailed CIFAR-10 dataset was to maximise the worst-case recall (3) and average recall, subject to a per-class coverage constraint (4).

Original: For CIFAR-100 LT dataset, we compare these methods for the objectives maximizing HT recall (10) and recall under HT coverage constraints(11).
Revised: For CIFAR-100 LT dataset, we compare these methods for the objectives maximizing HT recall (10) and recall under HT coverage constraints(11).

Original: For the objectives (3) and (10), DARP achieves the best average recall yet it suffers on the worst-case recall.
Revised: For the objectives (3) and (10), DARP achieves the best average recall yet it suffers on the worst-case recall.

Original: CSST (FixMatch) outperforms it on CIFAR-100 and the has superior worst-case recall on CIFAR-10.
Revised: CSST (FixMatch) outperforms it on CIFAR-100 and the has superior worst-case recall on CIFAR-10.

Original: A similar observation can be made for the objectives (4) and (11) where it is only CSST (FixMatch) that has the highest min.
Revised: A similar observation can be made for the objectives (4) and (11) where it is only CSST (FixMatch) that has the highest min.

Original: coverage and either has superior mean recall or has negligible decrease in the average recall.
Revised: coverage and either has superior mean recall or has negligible decrease in the average recall.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]