Original: DBT: A D ETECTION B OOSTER T RAINING METHOD FOR IMPROVING THE ACCURACY OF CLASSIFIERS
Revised: DBT: A D ETECTION B OOSTER T RAINING METHOD FOR IMPROVING THE ACCURACY OF CLASSIFIERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep learning models owe their success at large, to the availability of a largeamount of annotated data.
Revised: Deep learning models owe their success at large, to the availability of a largeamount of annotated data.

Original: They try to extract features from the data that containuseful information needed to improve their performance on target applications.
Revised: They try to extract features from the data that containuseful information needed to improve their performance on target applications.

Original: Most works focus on directly optimizing the target loss functions to improve theaccuracy by allowing the model to implicitly learn representations from the data.
Revised: Most works focus on directly optimizing the target loss functions to improve theaccuracy by allowing the model to implicitly learn representations from the data.

Original: There has not been much work on using background/noise data to estimate thestatistics of in-domain data to improve the feature representation of deep neuralnetworks.
Revised: There has not been much work on using background/noise data to estimate thestatistics of in-domain data to improve the feature representation of deep neuralnetworks.

Original: In this paper, we probe this direction by deriving a relationship betweenthe estimation of unknown parameters of the probability density function (pdf)of input data and classiﬁcation accuracy.
Revised: In this paper, we probe this direction by deriving a relationship betweenthe estimation of unknown parameters of the probability density function (pdf)of input data and classiﬁcation accuracy.

Original: Using this relationship, we show thathaving a better estimate of the unknown parameters using background and indomain data provides better features which leads to better accuracy.
Revised: Using this relationship, we show thathaving a better estimate of the unknown parameters using background and indomain data provides better features which leads to better accuracy.

Original: Based onthis result, we introduce a simple but effective detection booster training (DBT)method that applies a detection loss function on the early layers of a neural networkto discriminate in-domain data points from noise/background data, to improvethe classiﬁer accuracy.
Revised: Based onthis result, we introduce a simple but effective detection booster training (DBT)method that applies a detection loss function on the early layers of a neural networkto discriminate in-domain data points from noise/background data, to improvethe classiﬁer accuracy.

Original: The background/noise data comes from the same familyof pdfs of input data but with different parameter sets (e.g., mean, variance).
Revised: The background/noise data comes from the same familyof pdfs of input data but with different parameter sets (e.g., mean, variance).

Original: Inaddition, we also show that our proposed DBT method improves the accuracy evenwith limited labeled in-domain training samples as compared to normal training.
Revised: Inaddition, we also show that our proposed DBT method improves the accuracy evenwith limited labeled in-domain training samples as compared to normal training.

Original: We conduct experiments on face recognition, image classiﬁcation, and speakerclassiﬁcation problems and show that our method achieves superior performanceover strong baselines across various datasets and model architectures.
Revised: We conduct experiments on face recognition, image classiﬁcation, and speakerclassiﬁcation problems and show that our method achieves superior performanceover strong baselines across various datasets and model architectures.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Modern pattern recognition systems achieve outstanding accuracies on a vast domain of challengingcomputer vision, natural language, and speech recognition benchmarks (Russakovsky et al.
Revised: Modern pattern recognition systems achieve outstanding accuracies on a vast domain of challengingcomputer vision, natural language, and speech recognition benchmarks (Russakovsky et al.

Original: Lin et al.
Revised: Lin et al.

Original: ; Everingham et al. (2015); Panayotov et al.
Revised: ; Everingham et al. (2015); Panayotov et al.

Original: The success of deep learningapproaches relies on the availability of a large amount of annotated data and on extracting usefulfeatures from them for different applications.
Revised: The success of deep learningapproaches relies on the availability of a large amount of annotated data and on extracting usefulfeatures from them for different applications.

Original: Learning rich feature representations from the availabledata is a challenging problem in deep learning.
Revised: Learning rich feature representations from the availabledata is a challenging problem in deep learning.

Original: A related line of work includes learning deep latentspace embedding through deep generative models (Kingma & Welling (2014); Goodfellow et al.(2014); Berthelot et al.
Revised: A related line of work includes learning deep latentspace embedding through deep generative models (Kingma & Welling (2014); Goodfellow et al.(2014); Berthelot et al.

Original: or using self-supervised learning methods (Noroozi & Favaro (2016);
Revised: or using self-supervised learning methods (Noroozi & Favaro (2016);

Original: Gidaris et al.
Revised: Gidaris et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2016b))
Revised: (2016b))

Original: or through transfer learning approaches (Yosinski et al.(2014)
Revised: or through transfer learning approaches (Yosinski et al.(2014)

Original: ; Oquab et al.
Revised: ; Oquab et al.

Original: ; Razavian et al.
Revised: ; Razavian et al.

Original: In this paper, we propose to use a different approach to improve the feature representations of deepneural nets and eventually improve their accuracy by estimating the unknown parameters of theprobability density function (pdf) of input data.
Revised: In this paper, we propose to use a different approach to improve the feature representations of deepneural nets and eventually improve their accuracy by estimating the unknown parameters of theprobability density function (pdf) of input data.

Original: Parameter estimation or Point estimation methodsare well studied in the ﬁeld of statistical inference (Lehmann & Casella (1998)).
Revised: Parameter estimation or Point estimation methodsare well studied in the ﬁeld of statistical inference (Lehmann & Casella (1998)).

Original: The insights fromthe theory of point estimation can help us to develop better deep model architectures for improvingthe model’s performance.
Revised: The insights fromthe theory of point estimation can help us to develop better deep model architectures for improvingthe model’s performance.

Original: We make use of this theory to derive a correlation between the estimationof unknown parameters of pdf and classiﬁer outputs.
Revised: We make use of this theory to derive a correlation between the estimationof unknown parameters of pdf and classiﬁer outputs.

Original: However, directly estimating the unknownpdf parameters for practical problems such as image classiﬁcation is not feasible since it can sumup to millions of parameters.
Revised: However, directly estimating the unknownpdf parameters for practical problems such as image classiﬁcation is not feasible since it can sumup to millions of parameters.

Original: In order to overcome this bottleneck, we assume that the input datapoints are sampled from a family of pdfs instead of a single pdf and propose to use a detectionbased training approach to better estimate the unknowns using in-domain and background/noise data.
Revised: In order to overcome this bottleneck, we assume that the input datapoints are sampled from a family of pdfs instead of a single pdf and propose to use a detectionbased training approach to better estimate the unknowns using in-domain and background/noise data.

Original: One alternative is that we can use generative models for this task, however, they mimic the general distribution of training data conditioned on random latent vectors and hence cannot be directly appliedfor estimating the unknown parameters of a family of pdfs.
Revised: One alternative is that we can use generative models for this task, however, they mimic the general distribution of training data conditioned on random latent vectors and hence cannot be directly appliedfor estimating the unknown parameters of a family of pdfs.

Original: Our proposed detection method involvesa binary class discriminator that separates the target data points from noise or background data.
Revised: Our proposed detection method involvesa binary class discriminator that separates the target data points from noise or background data.

Original: Thenoise or background data is assumed to come from the same family of distribution of in-domaindata but with different moments (Please refer to the appendix for more details about the family ofdistributions and its extension to a general structure).
Revised: Thenoise or background data is assumed to come from the same family of distribution of in-domaindata but with different moments (Please refer to the appendix for more details about the family ofdistributions and its extension to a general structure).

Original: In image classiﬁcation, this typically representsthe background patches from input data that fall under the same distribution family.
Revised: In image classiﬁcation, this typically representsthe background patches from input data that fall under the same distribution family.

Original: In speech domain,it can be random noise or the silence intervals in speech data.
Revised: In speech domain,it can be random noise or the silence intervals in speech data.

Original: Collecting such background data toimprove the feature representations is much simpler as compared to using labeled training data sinceit is time-consuming and expensive to collect labeled data.
Revised: Collecting such background data toimprove the feature representations is much simpler as compared to using labeled training data sinceit is time-consuming and expensive to collect labeled data.

Original: Since the background patches in imagesor noise in speech signals are used for binary classiﬁcation in our method, we refer to such dataas the noise of an auxiliary binary classiﬁcation problem denoted by auxiliary binary classiﬁcation(ABC)-noise dataset.
Revised: Since the background patches in imagesor noise in speech signals are used for binary classiﬁcation in our method, we refer to such dataas the noise of an auxiliary binary classiﬁcation problem denoted by auxiliary binary classiﬁcation(ABC)-noise dataset.

Original: An advantage of using ABC-noise data during training is that it can implicitlyadd robustness to deep neural networks against the background or noisy data.
Revised: An advantage of using ABC-noise data during training is that it can implicitlyadd robustness to deep neural networks against the background or noisy data.

Original: Since ABC-noise data can be collected in large quantities for free and using that data in our approachimproves the classiﬁcation benchmarks, we investigate whether this data can act as a substitute forlabeled data.
Revised: Since ABC-noise data can be collected in large quantities for free and using that data in our approachimproves the classiﬁcation benchmarks, we investigate whether this data can act as a substitute forlabeled data.

Original: We conduct empirical analysis and show that using only a fraction of labeled trainingdata together with ABC-noise data in our DBT method, indeed improves the accuracy as comparedto normal training.
Revised: We conduct empirical analysis and show that using only a fraction of labeled trainingdata together with ABC-noise data in our DBT method, indeed improves the accuracy as comparedto normal training.

Original: To summarize, our contributions are threefold.
Revised: To summarize, our contributions are threefold.

Original: First, we present a detailed theoretical analysis onthe relation between the estimation of unknown parameters of pdf of data and classiﬁcation outputs.
Revised: First, we present a detailed theoretical analysis onthe relation between the estimation of unknown parameters of pdf of data and classiﬁcation outputs.

Original: Second, based on the theoretical analysis, we present a simple booster training method to improveclassiﬁcation accuracy which also doubles up as an augmented training method when only limitedlabeled data is available.
Revised: Second, based on the theoretical analysis, we present a simple booster training method to improveclassiﬁcation accuracy which also doubles up as an augmented training method when only limitedlabeled data is available.

Original: Third, we consistently achieve improved performances over strong baselineson face recognition, image classiﬁcation, and speaker recognition problems using our proposedmethod, showing its generalization across different domains and model architectures.
Revised: Third, we consistently achieve improved performances over strong baselineson face recognition, image classiﬁcation, and speaker recognition problems using our proposedmethod, showing its generalization across different domains and model architectures.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Notations and Preliminary: In this paper, vectors, matrices, functions, and sets are denoted by boldlower case, bold uppercase, lower case, and calligraphic characters, respectively.
Revised: Notations and Preliminary: In this paper, vectors, matrices, functions, and sets are denoted by boldlower case, bold uppercase, lower case, and calligraphic characters, respectively.

Original: Consider a datapointdenoted by x .
Revised: Consider a datapointdenoted by x .

Original: We assume that x belongs to a family of probability density functions (pdf’s) deﬁnedas P
Revised: We assume that x belongs to a family of probability density functions (pdf’s) deﬁnedas P

Original: = { p ( x , θ ) , θ ∈ Θ } , where Θ is the possible set of parameters of the pdf.
Revised: = { p ( x , θ ) , θ ∈ Θ } , where Θ is the possible set of parameters of the pdf.

Original: In general, θ is a realvector in higher dimensions.
Revised: In general, θ is a realvector in higher dimensions.

Original: For example, in a mixture of Gaussians, θ is a vector containing thecomponent weights, the component means, and the component covariance matrices.
Revised: For example, in a mixture of Gaussians, θ is a vector containing thecomponent weights, the component means, and the component covariance matrices.

Original: In this paper, weassume that θ is an unknown deterministic function (There are other approaches such as bayesianthat consider θ as a random vector).
Revised: In this paper, weassume that θ is an unknown deterministic function (There are other approaches such as bayesianthat consider θ as a random vector).

Original: In general, although the structure of the family of pdfs is itselfunknown, deﬁning a family of pdfs such as P can help us to develop theorems and use those resultsto derive a new method.
Revised: In general, although the structure of the family of pdfs is itselfunknown, deﬁning a family of pdfs such as P can help us to develop theorems and use those resultsto derive a new method.

Original: For the family of distribution P , we can deﬁne the following classiﬁcationproblem
Revised: For the family of distribution P , we can deﬁne the following classiﬁcationproblem

Original: [Equation]
Revised: [Equation]

Original: where set of Θ i ’s is a partition of Θ .
Revised: where set of Θ i ’s is a partition of Θ .

Original: The notation of (1) means that, class C i deals with a set ofdata points whose pdf is p ( x , θ i ) where θ
Revised: The notation of (1) means that, class C i deals with a set ofdata points whose pdf is p ( x , θ i ) where θ

Original: i ∈ Θ i .
Revised: i ∈ Θ i .

Original: A wide range of classiﬁcation problems can bedeﬁned using (1) e.g., ((Lehmann & Casella, 2006, Chapter 3)) and ((Duda et al., 2012, Chapter 4)).
Revised: A wide range of classiﬁcation problems can bedeﬁned using (1) e.g., ((Lehmann & Casella, 2006, Chapter 3)) and ((Duda et al., 2012, Chapter 4)).

Original: The problem of estimating θ comes under the category of parametric estimation or point estimation(Lehmann & Casella (1998)).
Revised: The problem of estimating θ comes under the category of parametric estimation or point estimation(Lehmann & Casella (1998)).

Original: Estimating the unknown parameters of a given pdf p ( x , θ ) , have beenextensively studied in the ﬁeld of point estimation methods (Lindgren (2017); Lee et al.
Revised: Estimating the unknown parameters of a given pdf p ( x , θ ) , have beenextensively studied in the ﬁeld of point estimation methods (Lindgren (2017); Lee et al.

Original: Lehmann & Casella (2006)).
Revised: Lehmann & Casella (2006)).

Original: An important estimator in this ﬁeld is the minimum variance unbiasedestimator and it is governed by the Cramer Rao bound.
Revised: An important estimator in this ﬁeld is the minimum variance unbiasedestimator and it is governed by the Cramer Rao bound.

Original: The Cramer Rao bound provides the lowerbound of the variance of an unbiased estimator (Bobrovsky et al.
Revised: The Cramer Rao bound provides the lowerbound of the variance of an unbiased estimator (Bobrovsky et al.

Original: Let the estimation ofθ be denoted by (cid:98) θ , and assume that (cid:98) θ is an unbiased estimator, i.e., E ( (cid:98) θ )
Revised: Let the estimation ofθ be denoted by (cid:98) θ , and assume that (cid:98) θ is an unbiased estimator, i.e., E ( (cid:98) θ )

Original: = θ .
Revised: = θ .

Original: Its covariancematrix denoted by Σ (cid:98) θ satisﬁes Σ (cid:98) θ −
Revised: Its covariancematrix denoted by Σ (cid:98) θ satisﬁes Σ (cid:98) θ −

Original: I − 1 ( θ ) (cid:23) 0 , where A (cid:23) 0 implies that A is a non-negativedeﬁnite matrix ((Lehmann & Casella, 1998, chapter 5))
Revised: I − 1 ( θ ) (cid:23) 0 , where A (cid:23) 0 implies that A is a non-negativedeﬁnite matrix ((Lehmann & Casella, 1998, chapter 5))

Original: and I ( θ ) := − E ( ∂ 2 log( p ( x , θ )) /∂
Revised: and I ( θ ) := − E ( ∂ 2 log( p ( x , θ )) /∂

Original: θ 2 )is called the Fisher information matrix.
Revised: θ 2 )is called the Fisher information matrix.

Original: For an arbitrary differentiable function g ( · ) , an efﬁcientestimator of g ( θ ) is an unbiased estimator when its covariance matrix equals to I − 1 g ( θ ) , where I − 1 g ( θ )is the ﬁsher information matrix of g ( θ ) , i.e., the efﬁcient estimator achieves the lowest possible variance among all unbiased estimators.
Revised: For an arbitrary differentiable function g ( · ) , an efﬁcientestimator of g ( θ ) is an unbiased estimator when its covariance matrix equals to I − 1 g ( θ ) , where I − 1 g ( θ )is the ﬁsher information matrix of g ( θ ) , i.e., the efﬁcient estimator achieves the lowest possible variance among all unbiased estimators.

Original: The efﬁcient estimator can be achieved using factorization
Revised: The efﬁcient estimator can be achieved using factorization

Original: of∂ log( p ( x , θ )) /∂
Revised: of∂ log( p ( x , θ )) /∂

Original: g ( θ ) =
Revised: g ( θ ) =

Original: I g ( θ )( (cid:98) g
Revised: I g ( θ )( (cid:98) g

Original: ( x ) − g ( θ )) , if it exists (Rao (1992); Lehmann & Casella (1998)).
Revised: ( x ) − g ( θ )) , if it exists (Rao (1992); Lehmann & Casella (1998)).

Original: Based on these results, we derive a relationship between the efﬁcient estimation of unknowns andmaximum likelihood classiﬁer of (1) and use auxiliary binary classiﬁers to apply that result in ourproposed DBT method.
Revised: Based on these results, we derive a relationship between the efﬁcient estimation of unknowns andmaximum likelihood classiﬁer of (1) and use auxiliary binary classiﬁers to apply that result in ourproposed DBT method.

Revised: Parameter Estimations: Independent component analysis (Hyvärinen (1999)) decomposes a multivariate signal into independent non-Gaussian signals.

Revised: ICA can extract non-Gaussian features from

Revised: Gaussian noise.

Original: Auxiliary Classiﬁers: Auxiliary classiﬁers were introduced in inception networks (Szegedy et al.(2015)) and used in (Lee et al. 
Revised: Additionally, there is a class of classiﬁers called generalized likelihood ratio functionsthat replaces the estimation of unknown parameters into the likelihood functions. This approachprovides a huge improvement in the ﬁeld of parametric classiﬁers, where the family of pdf of datais given (Zeitouni et al. (1992), Conte et al. , Lehmann & Casella (2006)).

Revised: Noise-contrastiveestimation (NCE) (Gutmann & Hyvärinen (2010)) involves training a generative model that allowsa model to discriminate data from a ﬁxed noise distribution.

Revised: Then, this trained model can be usedfor training a sequence of models of increasing quality.

Revised: This can be seen as an informal competitionmechanism similar in spirit to the formal competition used in the adversarial networks game.

Original: S. et al. 
Revised: In Bachman et al. (2019), a feature selection is proposed by maximizing the mutual information of thedifference between features extracted from multiple views of a shared context. In that work, it isshown that the best results is given by using a mutual information bound based on NCE.

Revised: The keydifference between our method and NCE is that, we do not construct a generative model for noise,instead of estimating the pdf of noise in NCE, we estimate the parameters of pdf of in-domain datasetusing an auxiliary class that has many common parameters in its pdf.

Revised: Moreover, we show that theestimation of that parameters are sufﬁcient statistic for a classiﬁer.

Revised: We assume that the noise dataset isnot pure and it has some similarity with the in-domain dataset, where it can help the feature selectionlayers to select relevant (in-domain) features, e.g., see Fig. 3.

Revised: Further, in our approach, we do notconstruct the pdf of noise or in-domain data, instead we estimate its parameters directly, which ismore efﬁcient in terms of training, computation and also dimensionality reduction.

Revised: Auxiliary classiﬁers were introduced in inception networks (Szegedy et al.

Revised: and used in (Leeet al. (2015); S. et al.

Original: for training very deep networks to preventvanishing gradient problems.
Revised: for training very deep networks to prevent vanishing gradient problems.

Original: Further, auxiliary classiﬁers were also proposed for early exit schemes(Teerapittayanon et al.
Revised: Further, auxiliary classiﬁers were also proposed for early exit schemes (Teerapittayanon et al.

Original: and self-distillation methods (Zhang et al.
Revised: (2016))and self-distillation methods (Zhang et al.

Original: (2019a;b)).
Revised: (2019a;b)).

Original: Such auxiliaryclassiﬁers tackle different problems by predicting the same target as the ﬁnal classiﬁcation layer.
Revised: Such auxiliary classiﬁers tackle differentproblems by predicting the same target as the ﬁnal classiﬁcation layer.

Original: In contrast, our proposed DBT method involves auxiliary binary classiﬁers that detect noise, interference, and/or background data from in-domain data points for improving the target classiﬁcationaccuracy.
Revised: In contrast, our proposed DBTmethod involves auxiliary binary classiﬁers that detect noise, interference, and/or background datafrom in-domain data points for improving the target classiﬁcation accuracy.

Original:  E STIMATION OF P ARAMETERS OF PDF AND C LASSIFICATION
Revised:  E STIMATION OF P ARAMETERS OF PDF AND C LASSIFICATION

Original: For (1), we deﬁne a deterministic discriminative function of Θ i , denoted by t i ( · ) such that thefollowing conditions are satisﬁed:
Revised: For (1), we deﬁne a deterministic discriminative function of Θ i , denoted by t i ( · ) such that thefollowing conditions are satisﬁed:

Original: • t i ( · ) maps Θ to real numbers such that t i ( θ ) > 0 , if θ ∈ Θ i and t i ( θ ) ≤ 0 for θ / ∈ Θ i .
Revised: • t i ( · ) maps Θ to real numbers such that t i ( θ ) > 0 , if θ ∈ Θ i and t i ( θ ) ≤ 0 for θ / ∈ Θ i .

Original: • t i ( · ) is a differentiable function almost everywhere and (cid:82) Θ | t i ( θ ) | d µ l ( θ ) < ∞ , where µ l denotes
Revised: • t i ( · ) is a differentiable function almost everywhere and (cid:82) Θ | t i ( θ ) | d µ l ( θ ) < ∞ , where µ l denotes

Original: the Lebesgue measure.
Revised: the Lebesgue measure.

Original: The following theorem shows the relationship of t i ( · ) and the log-likelihood ratio of class C i versusother classes.
Revised: The following theorem shows the relationship of t i ( · ) and the log-likelihood ratio of class C i versusother classes.

Original: The proofs of Theorems 1, 2 and 3 are provided in the appendix.
Revised: The proofs of Theorems 1, 2 and 3 are provided in the appendix.

Original: Theorem 1 Assume that the pdf p ( x , θ ) is differentiable with respect to θ almost everywhere.
Revised: Theorem 1 Assume that the pdf p ( x , θ ) is differentiable with respect to θ almost everywhere.

Original: If theefﬁcient minimum variance and unbiased estimation of a deterministic discriminative function of Θ iexists, then the log likelihood ratio of class i against the rest of classes is an increasing function ofthe minimum variance and unbiased estimation of Θ i .
Revised: If theefﬁcient minimum variance and unbiased estimation of a deterministic discriminative function of Θ iexists, then the log likelihood ratio of class i against the rest of classes is an increasing function ofthe minimum variance and unbiased estimation of Θ i .

Original: Directly from this theorem, it follows that the optimal classiﬁer using the maximum likelihood for (1)is given as follows d ( x ) = arg max i ∈{ 1 , ··· ,n } k
Revised: Directly from this theorem, it follows that the optimal classiﬁer using the maximum likelihood for (1)is given as follows d ( x ) = arg max i ∈{ 1 , ··· ,n } k

Original: i ( (cid:98) t i ( x )) , where k i ’s are some increasing functions andt i ( · ) ’s are the deterministic discriminative function of Θ
Revised: i ( (cid:98) t i ( x )) , where k i ’s are some increasing functions andt i ( · ) ’s are the deterministic discriminative function of Θ

Original: i ’s such that the efﬁcient minimum varianceand unbiased estimation for them exists.
Revised: i ’s such that the efﬁcient minimum varianceand unbiased estimation for them exists.

Original: Based on this result, a set of minimum variance and unbiasedestimation of deterministic discriminative functions of Θ
Revised: Based on this result, a set of minimum variance and unbiasedestimation of deterministic discriminative functions of Θ

Original: i ’s leads us to the maximum likelihoodclassiﬁer.
Revised: i ’s leads us to the maximum likelihoodclassiﬁer.

Original: One approach is to directly estimate the deterministic discriminative functions, instead ofmaximizing the likelihood function.
Revised: One approach is to directly estimate the deterministic discriminative functions, instead ofmaximizing the likelihood function.

Original: However, ﬁnding deterministic discriminative functions thathave efﬁcient minimum variance and unbiased estimation may not be feasible in practical problems,especially when the dimension of θ increases.
Revised: However, ﬁnding deterministic discriminative functions thathave efﬁcient minimum variance and unbiased estimation may not be feasible in practical problems, especially when the dimension of θ increases.

Original: Theorems 2 and 3 study the same relationship betweenthe estimation of unknown parameters and the accuracy of classiﬁers for sub-optimal estimators andclassiﬁers.
Revised: Theorems 2 and 3 study the same relationship betweenthe estimation of unknown parameters and the accuracy of classiﬁers for sub-optimal estimators andclassiﬁers.

Original: Theorem 2 Consider the output of two classiﬁers for the i th class as follows: r j ( x )
Revised: Theorem 2 Consider the output of two classiﬁers for the i th class as follows: r j ( x )

Original: = i if h j ( x ) > τand r j ( x ) = other classes if h j ( x ) < τ , where j ∈ { 1 , 2 } .
Revised: = i if h j ( x ) > τand r j ( x ) = other classes if h j ( x ) < τ , where j ∈ { 1 , 2 } .

Original: where h j ( x ) is the estimation of adeterministic discriminative function and τ is a classiﬁcation threshold.
Revised: where h j ( x ) is the estimation of adeterministic discriminative function and τ is a classiﬁcation threshold.

Original: Assume that the cumulativedistribution function of h j ( x ) ’s have bounded inﬂection points, and also, the probability of truepositive of r j ( x ) is an increasing function of d ( θ ) , which is the deterministic discriminative functionof class i , for all i .
Revised: Assume that the cumulativedistribution function of h j ( x ) ’s have bounded inﬂection points, and also, the probability of truepositive of r j ( x ) is an increasing function of d ( θ ) , which is the deterministic discriminative functionof class i , for all i .

Original: Further assume that for each τ the probability of false positive of r 1 ( x ) is lessthan the probability of false positive of r 2 ( x ) and the probability of true positive of r 1 ( x ) is greaterthan the probability of true positive of r 2 ( x ) .
Revised: Further assume that for each τ the probability of false positive of r 1 ( x ) is lessthan the probability of false positive of r 2 ( x ) and the probability of true positive of r 1 ( x ) is greaterthan the probability of true positive of r 2 ( x ) .

Original: Then, there exists a h min such that for all d ( θ )
Revised: Then, there exists a h min such that for all d ( θ )

Original: >
Revised: >

Original: h minand all θ we have Pr( | h 1 ( x ) − d ( θ ) | < (cid:15) ) >
Revised: h minand all θ we have Pr( | h 1 ( x ) − d ( θ ) | < (cid:15) ) >

Original: Pr( | h 2 ( x ) − d ( θ ) | < (cid:15) ) .
Revised: Pr( | h 2 ( x ) − d ( θ ) | < (cid:15) ) .

Original: Theorem 2 shows that a better classiﬁer leads to a better estimation of d ( θ ) .
Revised: Theorem 2 shows that a better classiﬁer leads to a better estimation of d ( θ ) .

Original: In the next theorem, weshow the dual property of this result.
Revised: In the next theorem, weshow the dual property of this result.

Original: Theorem 3 Let Θ m be a Borel set with positive Lebesgue measure in (1) for all m ∈ { 1 , · · · , n } .
Revised: Theorem 3 Let Θ m be a Borel set with positive Lebesgue measure in (1) for all m ∈ { 1 , · · · , n } .

Original: Assume that r 1 ( · ) and r 2 ( · ) are given as follows r 1 ( x ) = m , if (cid:98) θ 1 ∈ Θ m and r 2 ( x ) = m , if (cid:98) θ 2 ∈ Θ m .
Revised: Assume that r 1 ( · ) and r 2 ( · ) are given as follows r 1 ( x ) = m , if (cid:98) θ 1 ∈ Θ m and r 2 ( x ) = m , if (cid:98) θ 2 ∈ Θ m .

Original: Also, assume that Pr( (cid:107) (cid:98) θ 1 − θ (cid:107) ≤ (cid:15) ) ≥ Pr( (cid:107) (cid:98) θ 2 − θ (cid:107) ≤ (cid:15) ) , for all θ ∈ Θ = ∪ n m =1 Θ m and (cid:15) > 0 , then the probability of classiﬁcation error r 1 ( · ) is less than r 2 ( · ) where (cid:99) θ
Revised: Also, assume that Pr( (cid:107) (cid:98) θ 1 − θ (cid:107) ≤ (cid:15) ) ≥ Pr( (cid:107) (cid:98) θ 2 − θ (cid:107) ≤ (cid:15) ) , for all θ ∈ Θ = ∪ nm =1 Θ m and (cid:15) > 0 ,then the probability of classiﬁcation error r 1 ( · ) is less than r 2 ( · ) where (cid:99) θ

Original: 1 and (cid:99) θ 2 are two differentestimators of θ ∈ Θ =
Revised: 1 and (cid:99) θ 2 are two differentestimators of θ ∈ Θ =

Original: ∪ M − 1 m =

Revised: ∪ M − 1 m =

Original: Θ m .
Revised: Θ m .

Original: Theorem 3 proves that a more accurate estimator leads to a classiﬁer that has a lower probabilityof classiﬁcation error.
Revised: Theorem 3 proves that a more accurate estimator leads to a classiﬁer that has a lower probabilityof classiﬁcation error.

Original: From Theorem 1, we can infer that a sufﬁcient statistic for developing themaximum likelihood classiﬁcation is (cid:98) t i ( x ) , which is the efﬁcient minimum variance and unbiasedestimation of the deterministic discriminative functions of Θ i ’s denoted by t i ( θ ) .
Revised: From Theorem 1, we can infer that a sufﬁcient statistic for developing themaximum likelihood classiﬁcation is (cid:98) t i ( x ) , which is the efﬁcient minimum variance and unbiasedestimation of the deterministic discriminative functions of Θ i ’s denoted by t i ( θ ) .

Original: In other words, themaximum likelihood classiﬁer is a function of x only via the efﬁcient minimum variance and unbiasedestimation t i ( θ ) .
Revised: In other words, themaximum likelihood classiﬁer is a function of x only via the efﬁcient minimum variance and unbiasedestimation t i ( θ ) .

Original: We can estimate t i ( θ ) by replacing the estimation θ in t i ( · ) , i.e., (cid:91) t i ( θ )
Revised: We can estimate t i ( θ ) by replacing the estimation θ in t i ( · ) , i.e., (cid:91) t i ( θ )

Original: ≈
Revised: ≈

Original: t i ( (cid:98) θ ) ,where (cid:98) θ is a function of x .
Revised: t i ( (cid:98) θ ) ,where (cid:98) θ is a function of x .

Original: From the above theorems, we conclude that improving the estimationof unknown parameters of pdf of data can improve the accuracy of the classiﬁer.
Revised: From the above theorems, we conclude that improving the estimationof unknown parameters of pdf of data can improve the accuracy of the classiﬁer.

Original: On the other side,having a good classiﬁer means having a good estimator of unknowns of the pdf of input data.
Revised: On the other side,having a good classiﬁer means having a good estimator of unknowns of the pdf of input data.

Original: Inmany practical problems, the optimal maximum likelihood classiﬁer may not be achievable, but thelikelihood function of the classiﬁer provides an optimal bound of the probability of error.
Revised: Inmany practical problems, the optimal maximum likelihood classiﬁer may not be achievable, but thelikelihood function of the classiﬁer provides an optimal bound of the probability of error.

Original: In suchcases, we can improve the accuracy of sub-optimal classiﬁers and that is the main focus of this paper.
Revised: In suchcases, we can improve the accuracy of sub-optimal classiﬁers and that is the main focus of this paper.

Original: Fig.
Revised: Fig.

Original: 1 illustrates the proposed theorems visually.
Revised: 1 illustrates the proposed theorems visually.

Original:  P ROPOSED M ETHOD : D ETECTION B OOSTER T RAINING (DBT)
Revised:  P ROPOSED M ETHOD : D ETECTION B OOSTER T RAINING (DBT)

Original: In this section, we propose the detection booster training (DBT) method based on the achievedtheorems in the previous section to improve the accuracy of deep networks.
Revised: In this section, we propose the detection booster training (DBT) method based on the achievedtheorems in the previous section to improve the accuracy of deep networks.

Original: Speciﬁcally, we dividea deep model into two parts - early and later layers.
Revised: Speciﬁcally, we dividea deep model into two parts - early and later layers.

Original: We apply a detector (detection here meansdetecting a target pattern from noise/background) on the early layers of the neural network in orderto improve the estimation of unknown parameters of the family of pdf (based on Theorem 2).
Revised: We apply a detector (detection here meansdetecting a target pattern from noise/background) on the early layers of the neural network in order [Table] [Table] to improve the estimation of unknown parameters of the family of pdf (based on Theorem 2).

Original: Abetter estimation of unknown parameters corresponds to better feature representations in the earlylayers and these features are input to the rest of the layers to construct the deterministic discriminativefunctions (DDF) useful for the in-domain data classiﬁcation (based on Theorem 3).
Revised: Abetter estimation of unknown parameters corresponds to better feature representations in the earlylayers and these features are input to the rest of the layers to construct the deterministic discriminativefunctions (DDF) useful for the in-domain data classiﬁcation (based on Theorem 3).

Original: A general schema for dividing a deep model into two sub-models namely PEF (parameter estimatorfunctions) and DDF is depicted in Figure 2.
Revised: A general schema for dividing a deep model into two sub-models namely PEF (parameter estimatorfunctions) and DDF is depicted in Figure 2.

Original: The early layers of the model estimate the unknownparameters of pdf of data while the later layers construct the discriminative functions essential forclassiﬁcation.
Revised: The early layers of the model estimate the unknownparameters of pdf of data while the later layers construct the discriminative functions essential forclassiﬁcation.

Original: Based on this scheme, we formally deﬁne the three main components of DBT asfollows:
Revised: Based on this scheme, we formally deﬁne the three main components of DBT asfollows:

Original: • parameter estimator functions (PEF): The sub-network from input layer to the k th layer, where k is a hyperparameter in the DBT approach.
Revised: • parameter estimator functions (PEF): The sub-network from input layer to the k th layer, where k isa hyperparameter in the DBT approach.

Original: • auxiliary binary classiﬁcation (ABC): Some additional layers are attached to the end of PEF, mapping the output of the k th layer to a one-dimensional vector.
Revised: • auxiliary binary classiﬁcation (ABC): Some additional layers are attached to the end of PEF,mapping the output of the k th layer to a one-dimensional vector.

Original: • deterministic discriminative functions (DDF):
Revised: • deterministic discriminative functions (DDF):

Original: The sub-network from k th layer to the output of the
Revised: The sub-network from k th layer to the output of themodel.

Original: model.

Original: The output of model is a vector equal to the length of the number of classes n .
Revised: The output of model is a vector equal to the length of the number of classes n .

Original: From Theorem 2, we showed that unknown parameter estimation can be improved using a detectionapproach.
Revised: From Theorem 2, we showed that unknown parameter estimation can be improved using a detectionapproach.

Original: During training, we apply a binary classiﬁcation on the early layers (PEF) of the model toimprove the estimation of unknown parameters of pdf and subsequently provide rich feature vectorsfor DDF.
Revised: During training, we apply a binary classiﬁcation on the early layers (PEF) of the model toimprove the estimation of unknown parameters of pdf and subsequently provide rich feature vectorsfor DDF.

Original: We deﬁne the auxiliary binary classiﬁcation problem (ABC problem) as follows:
Revised: We deﬁne the auxiliary binary classiﬁcation problem (ABC problem) as follows:

Original: • Class 1 (alternative hypothesis) of ABC problem denoted by H 1 is set of all data points of classes [Table] [Table] of C [-
-]
Revised: • Class 1 (alternative hypothesis) of ABC problem denoted by H 1 is set of all data points of classesof C 1 to C [-
-]n ,

Original: to C n , i.e. θ ∈ ∪ ni =1 Θ i .

Original: • Class 0 (null hypothesis) of ABC problem denoted by H 0 is a dataset of data points from samedistribution p ( x , θ ) but θ / ∈ ∪ ni =1 Θ i .

Original: We also deﬁne the dataset of Class 0 of ABC as ABC- noisedataset,

Original: i
Revised: i

Original: e., the ABC is given by the

Original: following hypothesis testing problem:

Original: H θ ∈ ∪ ni =1 Θ i 
Revised: e. θ ∈ ∪ ni =1 Θ i .•

Original: versus
Revised: Class 0 (null hypothesis) of ABC problem denoted by H 0 is a dataset of data points from samedistribution p ( x , θ ) but θ / ∈ ∪ ni =1 Θ i .

Original:  H  0 : θ / ∈ ∪ ni =1 Θ i .
Revised: We also deﬁne the dataset of Class 0 of ABC as ABC- noisedataset, i.e., the ABC is given by the following hypothesis testing problem: H 1 : θ ∈ ∪ ni =1 Θ i versusH 0 : θ / ∈ ∪ ni =1 Θ i .

Original: In many practical problems, the noise, background or interference data related to
Revised: In many practical problems, the noise, background or interference data related to

Original: the in-domain dataset have same type of probability distribution but different pdf parameters.
Revised: the in-domain dataset have same type of probability distribution but different pdf parameters

Original: Hence, using that dataset is a cheap and adept choice for the null hypothesis of ABC
Revised: Hence,using that dataset is a cheap and adept choice for the null hypothesis of ABC

Original: The Auxiliary Binary Classiﬁcation problem inﬂuences only the PEF and ABC units while the mainclassiﬁcation problem with n classes updates the parameters of both PEF and DDF using in-domaindata.
Revised: The Auxiliary Binary Classiﬁcation problem inﬂuences only the PEF and ABC units while the mainclassiﬁcation problem with n classes updates the parameters of both PEF and DDF using in-domaindata.

Original: Since the auxiliary classiﬁer is only used during training, the inference model (IM) consists ofonly PEF and DDF and hence, there is no additional computation cost during inference.
Revised: Since the auxiliary classiﬁer is only used during training, the inference model (IM) consists ofonly PEF and DDF and hence, there is no additional computation cost during inference.

Original: We formulatethe aforementioned method in the following notations and loss functions.
Revised: We formulatethe aforementioned method in the following notations and loss functions.

Original: Assume that x is a datapoint that belongs to Class C i , i ∈ { 1 , · · · , n } or Class H 0 of ABC.
Revised: Assume that x is a datapoint that belongs to Class C i , i ∈ { 1 , · · · , n } or Class H 0 of ABC.

Original: Here, we deﬁne two type oflabels denoted by l ABC and l MC , where the subscription "MC" stands for multi-classes.
Revised: Here, we deﬁne two type oflabels denoted by l ABC and l MC , where the subscription "MC" stands for multi-classes.

Original: So, if xbelongs to class C i , then l ABC = 1 and l MC = i − 1 , else if x is a ABC-noise data point, l ABC =
Revised: So, if xbelongs to class C i , then l ABC = 1 and l MC = i − 1 , else if x is a ABC-noise data point, l ABC =

Original: 0and l MC is None .
Revised: 0and l MC is None .

Original: Therefore, the loss function is deﬁned as:
Revised: Therefore, the loss function is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where Q PEF , Q ABC and Q DDF are the functions of PEF, ABC and DDF blocks, respectively.
Revised: where Q PEF , Q ABC and Q DDF are the functions of PEF, ABC and DDF blocks, respectively.

Original: It isseen that, the second term of the total loss is zero if l ABC = 0 .
Revised: It isseen that, the second term of the total loss is zero if l ABC = 0 .

Original: L ABC and L MC are selected basedon the problem deﬁnition and datasets.
Revised: L ABC and L MC are selected basedon the problem deﬁnition and datasets.

Original: For classiﬁcation, a simple selection for them can be binarycross-entropy and cross-entropy, respectively.
Revised: For classiﬁcation, a simple selection for them can be binarycross-entropy and cross-entropy, respectively.

Original: For a given task and deep neural network, the choiceof k and L ABC inﬂuences the feature representation of early layers differently and consequently theaccuracy of the model.
Revised: For a given task and deep neural network, the choiceof k and L ABC inﬂuences the feature representation of early layers differently and consequently theaccuracy of the model.

Original: We provide empirical studies in the next section to verify the same.
Revised: We provide empirical studies in the next section to verify the same.

Original:  E XPERIMENTAL STUDY OF DBTF ACE R ECOGNITION
Revised:  E XPERIMENTAL STUDY OF DBTF ACE R ECOGNITION

Original: We conduct experiments on face recognition benchmarks and show that the DBT method learnsrich features essential for face recognition.
Revised: We conduct experiments on face recognition benchmarks and show that the DBT method learnsrich features essential for face recognition.

Original: We also discover an important observation that currentstate-of-the-art (SOTA) face recognition models are very sensitive to non-face data, in particular,animal faces.
Revised: We also discover an important observation that currentstate-of-the-art (SOTA) face recognition models are very sensitive to non-face data, in particular,animal faces.

Original: Fig.
Revised: Fig.

Original: 4 shows a few examples of misidentiﬁed faces and their corresponding animaldistractors from the IJB-B dataset using the ArcFace (Deng et al.
Revised: 4 shows a few examples of misidentiﬁed faces and their corresponding animaldistractors from the IJB-B dataset using the ArcFace (Deng et al.

Original: (2019)) model.
Revised: (2019)) model.

Original: We show that
Revised: We show that

Original: our
Revised: our

Original: DBT method not only improves the veriﬁcation accuracy but also implicitly tackles this robustnessissue of 
Revised: DBT method not only improves the veriﬁcation accuracy but also implicitly tackles this robustnessissue of current SOTA models against non-face data. The implementation details are provided in the

Original: [Table]
Revised: Appendix.

Original: We consider the PEF discussed in Section 4 to be the ﬁrst three layers of the model and DDF to bethe rest of layers.
Revised: We consider the PEF discussed in Section 4 to be the ﬁrst three layers of the model and DDF to bethe rest of layers.

Original: Ablation studies on the choice of PEF and DDF are provided in the supplementarymaterial.
Revised: Ablation studies on the choice of PEF and DDF are provided in the supplementarymaterial.

Original: We deﬁne L MC in (2) as the SOTA ArcFace loss function proposed in (Deng et al.
Revised: We deﬁne L MC in (2) as the SOTA ArcFace loss function proposed in (Deng et al.

Original: The ABC-noise is a non-face dataset containing 500K images that we collected from backgroundpatches of MS1MV2 (Guo et al.
Revised: The ABC-noise is a non-face dataset containing 500K images that we collected from backgroundpatches of MS1MV2 (Guo et al.

Original: (More details in Appendix).
Revised: (More details in Appendix).

Original: We experimented with twodifferent loss functions for L ABC .
Revised: We experimented with twodifferent loss functions for L ABC .

Original: For the ﬁrst one, since popular face recognition models (Deng et al.(2019)
Revised: For the ﬁrst one, since popular face recognition models (Deng et al.(2019)

Original: ; Wang et al.
Revised: ; Wang et al.

Original: use normalized output features and compute the losses on a hypersphere,we select L ABC as follows.
Revised: use normalized output features and compute the losses on a hypersphere,we select L ABC as follows.

Original: Let p f ∈ R d and p nf ∈ R
Revised: Let p f ∈ R d and p nf ∈ R

Original: d denote the prototypes for faces and nonfaces, respectively.
Revised: d denote the prototypes for faces and nonfaces, respectively.

Original: Following (Mettes et al.
Revised: Following (Mettes et al.

Original: , we constrain the face/non-face prototypes ondiametrically opposite directions i.e cos( θ p f p nf ) = − 1 and normalize the output feature vectors forfaces and non-faces such that (cid:107) p
Revised: , we constrain the face/non-face prototypes ondiametrically opposite directions i.e cos( θ p f p nf ) = − 1 and normalize the output feature vectors forfaces and non-faces such that (cid:107) p

Original: f i (cid:107) =
Revised: f i (cid:107) =

Original: (cid:107) p nf i
Revised: (cid:107) p nf i

Original: (cid:107) = 1 .
Revised: (cid:107) = 1 .

Original: We then deﬁne the L ABC as,
Revised: We then deﬁne the L ABC as,

Original: [Equation]
Revised: [Equation]

Revised: i

Original:  where θ y i and θ 2 correspond to the angles between the weights and the features for face and non-facelabels, respectively; m 1 , m 2 , m 3 are the angular margins; s denotes the radius of the hypersphere.
Revised: [Equation] log [Equation] where θ y i and θ 2 correspond to the angles between the weights and the features for face and non-facelabels, respectively; m 1 , m 2 , m 3 are the angular margins; s denotes the radius of the hypersphere.

Original: Forthe second choice, we use simple binary cross entropy for L ABC .
Revised: Forthe second choice, we use simple binary cross entropy for L ABC .

Original: Table 1 shows that the veriﬁcationaccuracy on LFW (Huang et al.
Revised: Table 1 shows that the veriﬁcationaccuracy on LFW (Huang et al.

Original: using (3) is 0.16% higher than simple cross entropy loss.
Revised: using (3) is 0.16% higher than simple cross entropy loss.

Original: Thisalso shows that choosing a task-speciﬁc L ABC is essential in obtaining more accurate results.
Revised: Thisalso shows that choosing a task-speciﬁc L ABC is essential in obtaining more accurate results.

Original: We use
Revised: We use

Original: Eqn.1 as the default for L ABC in all our face recognition experiments, unless otherwise stated.
Revised: Eqn.1 as the default for L ABC in all our face recognition experiments, unless otherwise stated.

Original: Table 3 compares the veriﬁcation accuracy of our method versus the current SOTA method ArcFaceon ﬁve different test sets, LFW, CPLFW (Zheng & Deng (2018)), CALFW (Zheng et al.
Revised: Table 3 compares the veriﬁcation accuracy of our method versus the current SOTA method ArcFaceon ﬁve different test sets, LFW, CPLFW (Zheng & Deng (2018)), CALFW (Zheng et al.

Original: CFP-FP (Sengupta et al.
Revised: CFP-FP (Sengupta et al.

Original: and AgeDb-30 (Moschoglou et al.
Revised: and AgeDb-30 (Moschoglou et al.

Original: For the LFW test set,we follow the unrestricted with labeled outside data protocol to report the performance.
Revised: For the LFW test set,we follow the unrestricted with labeled outside data protocol to report the performance.

Original: We trained
Revised: We trained

Original: ResNet-50 and ResNet-100 using ArcFace and DBT approaches on CASIA (small) and MS1MV2(large) datasets, respectively.
Revised: ResNet-50 and ResNet-100 using ArcFace and DBT approaches on CASIA (small) and MS1MV2(large) datasets, respectively.

Original: The results show that DBT method outperforms ArcFace on all datasets.
Revised: The results show that DBT method outperforms ArcFace on all datasets.

Original: Table 7 shows the angle statistics of the trained ArcFace and DBT models on the LFW dataset.
Revised: Table 7 shows the angle statistics of the trained ArcFace and DBT models on the LFW dataset.

Original: Min.
Revised: Min.

Original: Inter and Inter refer to the mean of minimum angles and mean of all angles between the templateembedding features of different classes (mean of the embedding features of all images for each class),respectively.
Revised: Inter and Inter refer to the mean of minimum angles and mean of all angles between the templateembedding features of different classes (mean of the embedding features of all images for each class),respectively.

Original: Intra refers to the mean of angles between x i and template embedding feature for eachclass.
Revised: Intra refers to the mean of angles between x i and template embedding feature for eachclass.

Original: From Table 7, we infer that DBT extracts better face features and hence reduces the intra-classvariations.
Revised: From Table 7, we infer that DBT extracts better face features and hence reduces the intra-class [Table]

Original:  Directly from Tables 3 and 7, we infer that ﬁrst, DBT consistently improves the accuracyon all test sets.
Revised: variations. Directly from Tables 3 and 7, we infer that ﬁrst, DBT consistently improves the accuracyon all test sets.

Original: Second, learning better features in the early layers is crucial to obtain rich face featureembeddings.
Revised: Second, learning better features in the early layers is crucial to obtain rich face featureembeddings.

Original: Third, the achieved gain using DBT is more pronounced on models trained using asmaller (CASIA) dataset (it has fewer identities and images).
Revised: Third, the achieved gain using DBT is more pronounced on models trained using asmaller (CASIA) dataset (it has fewer identities and images).

Original: This shows that DBT can address theissue of the lack of in-domain data using cheap ABC-noise data.
Revised: This shows that DBT can address theissue of the lack of in-domain data using cheap ABC-noise data.

Original: We also provide the results of training Inception-ResNet-V1 and ResNet-64 models using DBT on
Revised: We also provide the results of training Inception-ResNet-V1 and ResNet-64 models using DBT on

Original: MS1MV2 to show the generalization capacity of the DBT method.
Revised: MS1MV2 to show the generalization capacity of the DBT method.

Original: For the Inception-ResNet-V1and ResNet-64, the PEF is set to be the ﬁrst six layers and the DDF is the rest of the model.
Revised: For the Inception-ResNet-V1and ResNet-64, the PEF is set to be the ﬁrst six layers and the DDF is the rest of the model.

Original: We uselarge margin cosine loss (LMCL)
Revised: We uselarge margin cosine loss (LMCL)

Original: Wang et al.
Revised: Wang et al.

Original: (2018) for L MC and Cross entropy (CE) for L ABC .
Revised: (2018) for L MC and Cross entropy (CE) for L ABC .

Original: Table 4 shows the veriﬁcation accuracy on LFW for Inception-ResNet-V1 and ResNet-64 modelstrained on MS1MV2 with and without DBT.
Revised: Table 4 shows the veriﬁcation accuracy on LFW for Inception-ResNet-V1 and ResNet-64 modelstrained on MS1MV2 with and without DBT.

Original: The results show that DBT method is independentof model depth or architectures or loss functions and thereby consistently improves the accuracycompared to baseline results.
Revised: The results show that DBT method is independentof model depth or architectures or loss functions and thereby consistently improves the accuracycompared to baseline results.

Original: Table 4 also compares the DBT method with state-of-the-art methodson LFW and YTF datasets.
Revised: Table 4 also compares the DBT method with state-of-the-art methodson LFW and YTF datasets.

Original: DBT method notably improves the baselines that are comparable to
Revised: DBT method notably improves the baselines that are comparable to

Original: ArcFace and superior to all the other methods.
Revised: ArcFace and superior to all the other methods.

Original: We were not able to reproduce the results of the
Revised: We were not able to reproduce the results of the

Original: ArcFace paper using our Tensorﬂow implementation and dataset.
Revised: ArcFace paper using our Tensorﬂow implementation and dataset.

Original: We believe that using the original [Table] [Table] implementation and dataset from ArcFace will achieve superior results over the baselines on the benchmark datasets as evident from the results of our implementation.
Revised: We believe that using the originalimplementation and dataset from ArcFace will achieve superior results over the baselines on thebenchmark datasets as evident from the results of our implementation.

Original: Finally, we compare the result
Revised: Finally, we compare the result

Original: ArcFace and DBT on IJB-B and IJB-C, in Table 5.
Revised: ArcFace and DBT on IJB-B and IJB-C, in Table 5.

Original: It is seen that DBT provides a notable boost onboth IJB-B and IJB-C by a considerable margin.
Revised: It is seen that DBT provides a notable boost onboth IJB-B and IJB-C by a considerable margin.

Original: DBT improves the veriﬁcation accuracy as high as1.94 % on IJB-B and 2.57 % on IJB-C dataset at 10 − 4 false alarm rate (FAR).
Revised: DBT improves the veriﬁcation accuracy as high as1.94 % on IJB-B and 2.57 % on IJB-C dataset at 10 − 4 false alarm rate (FAR).

Original: We plot the receptiveﬁelds of the top ten maximally activated neurons of an intermediate layer of the face recognitionmodel to visualize the features learned using the DBT method.
Revised: We plot the receptiveﬁelds of the top ten maximally activated neurons of an intermediate layer of the face recognitionmodel to visualize the features learned using the DBT method.

Original: Fig.
Revised: Fig.

Original: 3 shows that the receptive ﬁeldsof layer 15 of the inception-resnet-v1 model trained using DBT attends to the regions of eyes, noseand mouth as compared to insigniﬁcant regions in the normal training method.
Revised: 3 shows that the receptive ﬁeldsof layer 15 of the inception-resnet-v1 model trained using DBT attends to the regions of eyes, noseand mouth as compared to insigniﬁcant regions in the normal training method.

Original: This shows that DBTmethod learns more discriminative features essential to face recognition, corroborating our theoreticalclaims.
Revised: This shows that DBTmethod learns more discriminative features essential to face recognition, corroborating our theoreticalclaims.

Original: To show that current SOTA models are not robust to animal faces, we performed a 1:N identiﬁcationexperiment with approximately 3000 animal distractors on the IJB-B (Whitelam et al.
Revised: To show that current SOTA models are not robust to animal faces, we performed a 1:N identiﬁcationexperiment with approximately 3000 animal distractors on the IJB-B (Whitelam et al.

Original: (2017)) dataset.
Revised: (2017)) dataset.

Original: We trained the face recognition model with about 500K non-face data which contains 200 animalfaces.
Revised: We trained the face recognition model with about 500K non-face data which contains 200 animalfaces.

Original: This is disjoint from the 3000 distractors used in the identiﬁcation experiment.
Revised: This is disjoint from the 3000 distractors used in the identiﬁcation experiment.

Original: We collected theanimal faces from web images using MTCNN (Zhang et al.
Revised: We collected theanimal faces from web images using MTCNN (Zhang et al.

Original: (2016a))
Revised: (2016a))

Original: face detector which are the falsepositives from the face detector.
Revised: face detector which are the falsepositives from the face detector.

Original: Table 2 shows the Rank-1 identiﬁcation accuracy of ResNet-100on IJB-B dataset, trained on MS1MV2 using the ArcFace loss (ResNet-100-AF) versus our DBTapproach (ResNet-100-DBT).
Revised: Table 2 shows the Rank-1 identiﬁcation accuracy of ResNet-100on IJB-B dataset, trained on MS1MV2 using the ArcFace loss (ResNet-100-AF) versus our DBTapproach (ResNet-100-DBT).

Original: The third column of Table 2 denotes the accuracy on a hard subsetof images (false positives from ArcFace model) on the IJB-B dataset denoted by H-set.
Revised: The third column of Table 2 denotes the accuracy on a hard subsetof images (false positives from ArcFace model) on the IJB-B dataset denoted by H-set.

Original: Resultsof Table 2 show that current face recognition models are unable to discriminate out-of-distribution(non-face) images from face images.
Revised: Resultsof Table 2 show that current face recognition models are unable to discriminate out-of-distribution(non-face) images from face images.

Original: Our ResNet-100-DBT signiﬁcantly (as high as 21 %) reduces themisidentiﬁcation rate as compared to the ArcFace model which shows that DBT method inherentlyovercomes this issue while also improving face recognition accuracy.
Revised: Our ResNet-100-DBT signiﬁcantly (as high as 21 %) reduces themisidentiﬁcation rate as compared to the ArcFace model which shows that DBT method inherentlyovercomes this issue while also improving face recognition accuracy.

Original: I MAGE C LASSIFICATION
Revised: I MAGE C LASSIFICATION

Original: In this section

Original: [Table] we evaluate ResNet-110 and ResNext-101 models trained with and without DBT onimage classiﬁcation problem using CIFAR-10, CIFAR- [Table] and ImageNet.
Revised: In this section, we evaluate ResNet-110 and ResNext-101 models trained with and without DBT onimage classiﬁcation problem using CIFAR-10, CIFAR-100, and ImageNet.

Original: We also show the power

Original: Method

Original: [Table]

Original: [Table]

Original: [Table] [Table] of DBT to compensate for the smaller in-domain training set.
Revised: We also show the powerof DBT to compensate for the smaller in-domain training set.

Original: For all implementations, PEF is deﬁnedto be the ﬁrst three layers and DDF is the rest of the model.
Revised: For all implementations, PEF is deﬁnedto be the ﬁrst three layers and DDF is the rest of the model.

Original: L ABC and L MC are set to cross-entropyloss.
Revised: L ABC and L MC are set to cross-entropyloss.

Original: ABC-noise is the same data used in face recognition experiments.
Revised: ABC-noise is the same data used in face recognition experiments.

Original: We follow the same trainingconﬁgurations from (He et al.
Revised: We follow the same trainingconﬁgurations from (He et al.

Original: ; Xie et al. (2017)). 
Revised: ; Xie et al. (2017)). [Table] [Table] [Table]

Original: To study the efﬁcacy of the DBT method in augmenting smaller in-domain training datasets, wealso trained ResNet-100 and ResNext-101 using partial training data on CIFAR-10 and CIFAR-100.
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: To study the efﬁcacy of the DBT method in augmenting smaller in-domain training datasets, wealso trained ResNet-100 and ResNext-101 using partial training data on CIFAR-10 and CIFAR-100.

Original: We randomly selected a fraction of the training data to be our training set, e.g., k/ 5 of datasetmeans that we only used k ﬁfth of total samples for training.
Revised: We randomly selected a fraction of the training data to be our training set, e.g., k/ 5 of datasetmeans that we only used k ﬁfth of total samples for training.

Original: From ﬁrst row of Table 8, we ﬁnd thatmodels trained with DBT show 0.59 % and 0.35 % improvement on CIFAR-10, 0.62 % and 1.45 %improvement on CIFAR-100 over baseline models for ResNet-110 and ResNext-101 architectures,respectively.
Revised: From ﬁrst row of Table 8, we ﬁnd thatmodels trained with DBT show 0.59 % and 0.35 % improvement on CIFAR-10, 0.62 % and 1.45 %improvement on CIFAR-100 over baseline models for ResNet-110 and ResNext-101 architectures,respectively.

Original: Furthermore, using partial training data with our DBT method achieves superior resultsas compared to normal training.
Revised: Furthermore, using partial training data with our DBT method achieves superior results(as high as 5.49 % on ResNext (1/5) CIFAR-100) as compared to normal training.

Original: Table 6 shows the results on Imagenet.
Revised: Table 6 showsthe results on Imagenet.

Original: We see that DBT improvesthe accuracy by 0.28 % on Top-1 accuracy.
Revised: We see that DBT improves the accuracy by 0.28 % on Top-1 accuracy.

Original: This shows that the DBT method consistently improvesthe results on both small and large datasets.
Revised: Thisshows that the DBT method consistently improves the results on both small and large datasets.

Original: S PEAKER I DENTIFICATION
Revised: S PEAKER I DENTIFICATION

Original: We consider the problem of speaker identiﬁcation using the VGG-M (Chatﬁeld et al.
Revised: We consider the problem of speaker identiﬁcation using the VGG-M (Chatﬁeld et al.

Original: (2014)) model.
Revised: (2014)) model.

Original: We set PEF as the ﬁrst two CNN layers and DDF as the remaining CNN layers.
Revised: We set PEF as the ﬁrst two CNN layers and DDF as the remaining CNN layers.

Original: L ABC and L MCare deﬁned to be the cross-entropy loss.
Revised: L ABC and L MCare deﬁned to be the cross-entropy loss.

Original: The ABC-noise is generated from the silence intervals of
Revised: The ABC-noise is generated from the silence intervals of

Original: VoxCeleb (Nagrani et al.
Revised: VoxCeleb (Nagrani et al.

Original: augmented with Gaussian noise with variance one.
Revised: augmented with Gaussian noise with variance one.

Original: The input to themodel is the short-time Fourier transformation of speech signals with a hamming sliding windowof width 25 ms and step 10 ms.
Revised: The input to themodel is the short-time Fourier transformation of speech signals with a hamming sliding windowof width 25 ms and step 10 ms.

Original: Table 9 provides the accuracies of VGG-M model trained with andwithout DBT on VoxCeleb, Librispeech (Panayotov et al.
Revised: Table 9 provides the accuracies of VGG-M model trained with andwithout DBT on VoxCeleb, Librispeech (Panayotov et al.

Original: , VCTK (Veaux et al.
Revised: , VCTK (Veaux et al.

Original: and
Revised: and

Original: ELSDR (L. (2004)) datasets.
Revised: ELSDR (L. (2004)) datasets.

Original: Table 9 shows that the trained models using DBT signiﬁcantly improvesthe accuracy (as high as 5.62% ) for all datasets.
Revised: Table 9 shows that the trained models using DBT signiﬁcantly improvesthe accuracy (as high as 5.62% ) for all datasets.

Original: The implementation details are provided in the
Revised: The implementation details are provided in the

Original: Appendix.
Revised: Appendix.

Revised: M ISCELLANEOUS E XPERIMENTS

Revised: In this section, we experiment with a different possibility of incorporating the background/noise data.

Revised: For face recognition, the naive way of using non-face data is to consider non-faces as a separate classin the ﬁnal classiﬁcation layer.

Revised: Table 11 shows the results of training with an additional backgroundclass on MS1MV2 dataset with and without using DBT.

Revised: ResNet+mod refers to a model trained with

Original: C ONCLUSION
Revised: ArcFace loss and n + 1 classes where the additional class corresponds to the non-faces. ResNetDBT+mod refers to a model trained with both DBT and the additional non-face class. We ﬁnd thatadding the additional non-face class hurts the performance of the model whereas ResNet-DBT+modimproves the results signiﬁcantly relative to ResNet+mod model. This is due to the fact that the classof non-faces are much larger than faces and it is difﬁcult to restrict its features on a hyperspherealong with other faces.

Revised: We also conduct this experiment on CIFAR-10/CIFAR-100 and report it on

Revised: Table 10.

Revised: We see that naively incorporating the background class is inferior to DBT showing theeffectiveness of our approach.

Revised: This analysis also shows that DBT is an effective technique to utilizenoise (non-face) data to boost the performance of classiﬁcation models.

Revised:  C ONCLUSION

Original: In this paper, we presented a detailed theoretical analysis of the dual relationship between estimatingthe unknown pdf parameters and classiﬁcation accuracy.
Revised: In this paper, we presented a detailed theoretical analysis of the dual relationship between estimatingthe unknown pdf parameters and classiﬁcation accuracy.

Original: Based on the theoretical study, we presenteda new method called DBT using ABC-noise data for improving in-distribution classiﬁcation accuracy.
Revised: Based on the theoretical study, we presenteda new method called DBT using ABC-noise data for improving in-distribution classiﬁcation accuracy.

Original: We showed that using ABC-noise data helps in better estimation of unknown parameters of pdf ofinput data and thereby improves the feature representations and consequently the accuracy in imageclassiﬁcation, speaker classiﬁcation, and face recognition benchmarks.
Revised: We showed that using ABC-noise data helps in better estimation of unknown parameters of pdf ofinput data and thereby improves the feature representations and consequently the accuracy in imageclassiﬁcation, speaker classiﬁcation, and face recognition benchmarks.

Original: It also augments the trainingdata when only limited labeled data is available by improving accuracy.
Revised: It also augments the trainingdata when only limited labeled data is available by improving accuracy.

Original: We showed that the conceptof DBT is generic and generalizes well across domains through extensive experiments using differentmodel architectures and datasets.
Revised: We showed that the conceptof DBT is generic and generalizes well across domains through extensive experiments using differentmodel architectures and datasets.

Original: Our framework is complementary to existing training methods andhence, it can be easily integrated with current and possibly future classiﬁcation methods to enhanceaccuracy.
Revised: Our framework is complementary to existing training methods andhence, it can be easily integrated with current and possibly future classiﬁcation methods to enhanceaccuracy.

Original: In summary, the proposed DBT method is a powerful technique that can augment limitedtraining data and improve classiﬁcation accuracy in deep neural networks.R
Revised: In summary, the proposed DBT method is a powerful technique that can augment limitedtraining data and improve classiﬁcation accuracy in deep neural networks.

Original: PPENDIX
Revised: A PPENDIXI N - DOMAIN FAMILY OF PDFS AND THE EXTENDED FAMILY OF DISTRIBUTIONS

Original: I N - DOMAIN FAMILY OF PDFS AND THE EXTENDED FAMILY OF DISTRIBUTIONS

Original: In this section, we discuss about background/noise and in-domain data points and their correspondingdistributions to clarify the deﬁnition of those concepts in this paper.
Revised: In this section we discuss about background/noise and in-domain data points and their correspondingdistributions to clarify the deﬁnition of those concepts in this paper.

Original: Consider a random vector denotedby s .
Revised: Consider a random vector denotedby s .

Original: Assume that the corresponding distribution is Gaussian with mean and variance given by α (cid:54) =
Revised: Assume that the corresponding distribution is Gaussian with mean and variance given by α (cid:54) =

Original: 0and σ = 1 , respectively.
Revised: 0and σ = 1 , respectively.

Original: Now, assume that we observed x = s + n , where the pdf of n is assumed tobe Guassian with zero mean and variance σ 2 n , hence the pdf of x is Gaussian with mean α and variance1 + σ 2 n .
Revised: Now, assume that we observed x = s + n , where the pdf of n is assumed tobe Guassian with zero mean and variance σ 2 n , hence the pdf of x is Gaussian with mean α and variance1 + σ 2 n .

Original: Here, n is the background or noise data and the vector of unknowns is given by, θ =
Revised: Here, n is the background or noise data and the vector of unknowns is given by, θ =

Original: [ α, σ 2 n ] .
Revised: [ α, σ 2 n ] .

Original: The in-domain family of pdfs for x is then given by P x = {N ( α, 1 + σ 2 n ) | α (cid:54) = 0 , σ 2 n > 0 } .
Revised: The in-domain family of pdfs for x is then given by P x = {N ( α, 1 + σ 2 n ) | α (cid:54) = 0 , σ 2 n > 0 } .

Original: If weinclude the family of pdf of n to P x , then we can extend P x as P =
Revised: If weinclude the family of pdf of n to P x , then we can extend P x as P =

Original: {N ( α, 1 + σ 2 n ) | α ∈ R , σ 2 n > 0 } .
Revised: {N ( α, 1 + σ 2 n ) | α ∈ R , σ 2 n > 0 } .

Original: So P is the union of family of pdfs of in-domain data points and noise/background data.
Revised: So P is the union of family of pdfs of in-domain data points and noise/background data.

Original: Fromestimation theory, we know that the sufﬁcient statistics and the unknown parameters of P can alsorepresent the sufﬁcient statistics and the unknown parameters of P x .
Revised: Fromestimation theory, we know that the sufﬁcient statistics and the unknown parameters of P can alsorepresent the sufﬁcient statistics and the unknown parameters of P x .

Original: In other words, an estimation ofα can help us detect if the observed data point is from s + n or n by comparing it with a threshold.
Revised: In other words, an estimation ofα can help us detect if the observed data point is from s + n or n by comparing it with a threshold.

Original: Thus, estimating the unknown parameters of the family of pdfs using P can provide more informationabout the observed data useful for tasks such as classiﬁcation.
Revised: Thus, estimating the unknown parameters of the family of pdfs using P can provide more informationabout the observed data useful for tasks such as classiﬁcation.

Original: In general, we can assume that a generalized family of pdfs is given by the family of pdf of noise or background along with the family of pdfs of in-domain data.
Revised: In general, we can assume that a generalized family of pdfs is given by the family of pdf of noise or background along with the family of pdfs of in-domain data.

Original: Hence, estimating from the extended family of distribution can provide more information about the in-domain distribution.
Revised: Hence, estimating from the extended family of distribution can provide more information about the in-domain distribution.

Original: Let us consider that the pdf of in-domain data points is given by p x ( x , [ θ s , θ n ]) and the pdf of noise/background is given by p n ( x , θ n ) , so the extended pdf can be represented by
Revised: Let us consider that the pdf of in-domain data points is given by p x ( x , [ θ s , θ n ]) and the pdf of noise/background is given by p n ( x , θ n ) , so the extended pdf can be represented by

Original: [Equation]
Revised: [Equation]

Original: where h is a function that combines two pdfs in a general structure.
Revised: where h is a function that combines two pdfs in a general structure.

Original: So a general family of distribution can be denoted as follows:
Revised: So a general family of distribution can be denoted as follows:

Original: [Equation]
Revised: [Equation]

Original: where θ is deﬁned as a new set of parameters in a higher dimension and Θ s,n are set of all possible[ θ s , θ n ] that belongs to p n and p x .
Revised: where θ is deﬁned as a new set of parameters in a higher dimension and Θ s,n are set of all possible[ θ s , θ n ] that belongs to p n and p x .

Original: The extended family of pdf provides more information aboutthe nuisance parameters of pdf of in-domain datapoints.
Revised: The extended family of pdf provides more information aboutthe nuisance parameters of pdf of in-domain datapoints.

Original: Inspired by this observation, we developour detection booster training method using background/noise data.
Revised: Inspired by this observation, we developour detection booster training method using background/noise data.

Original: Figure 5 shows an example ofbackground and in-domain data point.
Revised: Figure 5 shows an example ofbackground and in-domain data point.

Original: P ROOF OF T HEOREM 1
Revised: P ROOF OF T HEOREM 1

Original: Let t i ( · ) denote deterministic discriminative function of Θ i .
Revised: Let t i ( · ) denote deterministic discriminative function of Θ i .

Original: Since the efﬁcient minimum varianceand unbiased estimation of t i ( θ ) exists, we have 
Revised: Since the efﬁcient minimum varianceand unbiased estimation of t i ( θ ) exists, we have [Equation]

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) t i ( x ) is the minimum variance and unbiased estimation of t i ( θ ) using the data point x and
Revised: where (cid:98) t i ( x ) is the minimum variance and unbiased estimation of t i ( θ ) using the data point x and

Original: I t i ( x ) is the Fisher information function of t i ( θ ) , which is given by
Revised: I t i ( x ) is the Fisher information function of t i ( θ ) , which is given by

Original: [Equation]
Revised: [Equation]

Original: where T denotes the transpose and I ( θ ) is the Fisher information matrix of θ .
Revised: where T denotes the transpose and I ( θ ) is the Fisher information matrix of θ .

Original: Now we show thatthe log-likelihood ratio is an increasing function in (cid:98) t i ( x ) .
Revised: Now we show thatthe log-likelihood ratio is an increasing function in (cid:98) t i ( x ) .

Original: Note that I t i ( θ ) ≥ 0 (Lehmann & Casella(2006)).
Revised: Note that I t i ( θ ) ≥ 0 (Lehmann & Casella(2006)).

Original: On the other hand, we have d ln( p ( x , θ )) 
Revised: On the other hand, we have d ln( p ( x , θ )) =

Original: = (cid:80)
Revised: [Equation]

Original: j ∂ ln( p ( x , θ ))
Revised: j ∂ ln( p ( x , θ ))

Original: ∂θ j d θ j , therefore,
Revised: ∂θ j d θ j , therefore,

Original: [Equation]
Revised: [Equation]

Original: ∂t

Original: [Equation] (cid:88) [Equation] where the third equality is archived based on the third property of t i ( · ) in its deﬁnition and the forth equality is given by replacing (4; k
Revised: where the third equality is archived based on the third property of t i ( · ) in its deﬁnition and the forth equality is given by replacing (4; k

Original: ( x ) is the constant of integration.
Revised: ( x ) is the constant of integration.

Original: Finally, the last equality is given by deﬁning the following terms
Revised: Finally, the last equality is given by deﬁning the following terms

Original: [Equation]
Revised: [Equation]

Original: thus d α ( θ ) d t i ( θ )
Revised: thus d α ( θ ) d t i ( θ )

Original: =
Revised: =

Original: I t i ( θ ) ≥ 0 , i.e., α ( θ ) is increasing in t
Revised: I t i ( θ ) ≥ 0 , i.e., α ( θ ) is increasing in t

Original: i ( θ ) .
Revised: i ( θ ) .

Original: Since, t i is a deterministic discriminativefunction of Θ i , so for each j (cid:54) =
Revised: Since, t i is a deterministic discriminativefunction of Θ i , so for each j (cid:54) =

Original: i and θ
Revised: i and θ

Original: i ∈
Revised: i ∈

Original: Θ i and θ j ∈ Θ
Revised: Θ i and θ j ∈ Θ

Original: j , we have t i ( θ i ) >
Revised: j , we have t i ( θ i ) >

Original: t i ( θ j ) , thereforeα ( θ i ) ≥ α ( θ j ) .
Revised: t i ( θ j ) , therefore α ( θ i ) ≥ α ( θ j ) .

Original: The later inequality is achieved based on the increasing property of α ( θ ) withrespect to t i ( θ ) .
Revised: The later inequality is achieved based on the increasing property of α ( θ ) withrespect to t i ( θ ) .

Original: Using (5), the log likelihood ratio of class i against the rest of classes is given by LLR :
Revised: Using (5), the log likelihood ratio of class i against the rest of classes is given by LLR :

Original: =ln( p ( x , θ i )) − ln( p ( x , θ j )) , so we have LLR = (cid:0) α
Revised: =ln( p ( x , θ i )) − ln( p ( x , θ j )) , so we have LLR = (cid:0) α

Original: ( θ i ) − α
Revised: ( θ i ) − α

Original: ( θ i ) (cid:1)(cid:98) t i ( x ) − (cid:0) β
Revised: ( θ i ) (cid:1)(cid:98) t i ( x ) − (cid:0) β

Original: ( θ i ) − β
Revised: ( θ i ) − β

Original: ( θ j ) (cid:1) .
Revised: ( θ j ) (cid:1) .

Original: LLRdepends on x only via (cid:98) t i ( x ) and since for each j (cid:54)
Revised: LLRdepends on x only via (cid:98) t i ( x ) and since for each j (cid:54)

Original: =
Revised: =

Original: i and θ
Revised: i and θ

Original: i ∈
Revised: i ∈

Original: Θ i and θ j / ∈ Θ
Revised: Θ i and θ j / ∈ Θ

Original: i , α ( θ i ) − α
Revised: i , α ( θ i ) − α

Original: ( θ i ) > 0 ,then LLR is increasing in (cid:98) t i ( x ) .
Revised: ( θ i ) > 0 ,then LLR is increasing in (cid:98) t i ( x ) .

Original: (cid:3)
Revised: (cid:3)

Original: P ROOF OF T HEOREM 2
Revised: P ROOF OF T HEOREM 2

Original: The probability of true positive of class i of r j is given by
Revised: The probability of true positive of class i of r j is given by

Original: [Equation]
Revised: [Equation]

Original: where F i θ ( · ) denotes the Cumulative distribution function (CDF) of h j .
Revised: where F i θ ( · ) denotes the Cumulative distribution function (CDF) of h j .

Original: Since the probability of true positive of class i of r 1 is greater than r 2 for all τ , F 1 θ ( τ ) <
Revised: Since the probability of true positive of class i of r 1 is greater than r 2 for all τ , F 1 θ ( τ ) <

Original: F 2 θ ( τ ) , for all τ .
Revised: F 2 θ ( τ ) , for all τ .

Original: Now we deﬁne a function as follows
Revised: Now we deﬁne a function as follows

Original: [Equation]
Revised: [Equation]

Original: Since the CDFs are increasing in τ and tend to 1 and the number of inﬂection points of these CDFs are bounded, there is an h min such that, for τ > h min , such that u ( τ, θ ) is a monotonically decreasing function in τ .
Revised: Since the CDFs are increasing in τ and tend to 1 and the number of inﬂection points of these CDFs are bounded, there is an h min such that, for τ > h min , such that u ( τ, θ ) is a monotonically decreasing function in τ .

Original: Thus for any θ that satisﬁes d ( θ ) >
Revised: Thus for any θ that satisﬁes d ( θ ) >

Original: h min we have
Revised: h min we have

Original: [Equation]
Revised: [Equation]

Original:  Replacing u ( h, θ )
Revised: θ ) . Replacing u ( h, θ )

Original: =
Revised: =

Original: F 2 θ ( h ) − F 1 θ ( h ) in the last inequality, we have
Revised: F 2 θ ( h ) − F 1 θ ( h ) in the last inequality, we have

Original: [Equation]
Revised: [Equation]

Original: Based on the deﬁnition of CDF, we have
Revised: Based on the deﬁnition of CDF, we have

Original: [Equation]
Revised: [Equation]

Original:  P ROOF OF T HEOREM 
Revised:  P ROOF OF T HEOREM 3

Original: First, we prove the following claim,
Revised: First, we prove the following claim,

Original: Claim: For any open set, there exists a set of disjoint countable open balls such that their union equalsthe origin open set.
Revised: Claim: For any open set, there exists a set of disjoint countable open balls such that their union equalsthe origin open set.

Original: Proof of claim: Consider an open set O , and also consider x 0 ∈ O , such that B ( x 0 , r 0 ) ⊆
Revised: Proof of claim: Consider an open set O , and also consider x 0 ∈ O , such that B ( x 0 , r 0 ) ⊆

Original: Oand r 0 is the greatest possible radius between all possible open balls in O , where B ( x 0 , r 0 ) is theopen ball with radius r 0 at point x 0 .
Revised: Oand r 0 is the greatest possible radius between all possible open balls in O , where B ( x 0 , r 0 ) is theopen ball with radius r 0 at point x 0 .

Original: Now, we deﬁne x 1 ∈ O − B ( x 0 , r 0 ) ,
Revised: Now, we deﬁne x 1 ∈ O − B ( x 0 , r 0 ) ,

Original: where B ( x 0 , r 0 ) isthe closure of B ( x 0 , r 0 ) , as the point with greatest radius in O − B ( x 0 , r 0 ) and similarly x i ∈
Revised: where B ( x 0 , r 0 ) isthe closure of B ( x 0 , r 0 ) , as the point with greatest radius in O − B ( x 0 , r 0 ) and similarly x i ∈

Original: O − ∪ i − 1 k =0 B ( x k , r k ) such that B ( x i , r i ) provides the greatest radius in O − ∪ i − 1 k =0 B ( x k , r k ) .
Revised: O − ∪ i − 1 k =0 B ( x k , r k ) such that B ( x i , r i ) provides the greatest radius in O − ∪ i − 1 k =0 B ( x k , r k ) .

Original: Sowe have O = ∪ ∞ k =0 B ( x k , r k ) .
Revised: Sowe have O = ∪ ∞ k =0 B ( x k , r k ) .

Original: This is because, if the latest equality is not valid, then there existsan open ball in O − ∪ ∞ k
Revised: This is because, if the latest equality is not valid, then there existsan open ball in O − ∪ ∞ k

Original: =0 B ( x k , r k ) hence another open ball with greatest radius will be added to∪ ∞ k =0 B ( x k , r k ) , which has a contradiction with the deﬁnition of ∪ ∞ k =0 B ( x k , r k ) .
Revised: =0 B ( x k , r k ) hence another open ball with greatest radius will be added to∪ ∞ k =0 B ( x k , r k ) , which has a contradiction with the deﬁnition of ∪ ∞ k =0 B ( x k , r k ) .

Original: The claim is provenat this point.
Revised: The claim is provenat this point.

Original: Now, we show the true positive probability of r 1 is greater than r 2 .
Revised: Now, we show the true positive probability of r 1 is greater than r 2 .

Original: Let Θ (cid:48) m be the set of interior points of Θ m , then, there exists a union of disjoint open balls such that Θ (cid:48) m
Revised: Let Θ (cid:48) m be the set of interior points of Θ m , then, there exists a union of disjoint open balls such that Θ (cid:48) m

Original: = ∪ ∞ k =0 B ( x k , r k ) .
Revised: = ∪ ∞ k =0 B ( x k , r k ) .

Original: From assumptions in the theorem, we have Pr( (cid:107) (cid:98) θ 1 − θ (cid:107) ≤ (cid:15) ) ≥ Pr( (cid:107) (cid:98) θ 2 − θ (cid:107) ≤ (cid:15) ) , then
Revised: From assumptions in the theorem, we have Pr( (cid:107) (cid:98) θ 1 − [Equation]

Original: Pr θ ( (cid:98) θ 1 ∈ B ( x k , r k )) ≥ Pr θ ( (cid:98) θ 2 ∈ B ( x k , r k )) , where θ ∈ Θ m .
Revised: (cid:107) ≤ (cid:15) ) ≥ Pr( (cid:107) (cid:98) θ 2 − θ (cid:107) ≤ (cid:15) ) , then [Equation] , where θ ∈ Θ m .

Original: Based on the claim we have
Revised: Based on the claim we have

Original: [Equation]
Revised: [Equation]

Original: Moreover, based on deﬁnition of r i , the true positive probability of class m is given byp
Revised: Moreover, based on deﬁnition of r i , the true positive probability of class m is given by

Original: [Equation]
Revised: [Equation]

Original:  for i = 1 , 2 .
Revised: Θ (cid:48) m ) + Pr θ ( (cid:98) θ i ∈ Θ m − Θ (cid:48) m ) , for i = 1 , 2 .

Original: Additionally, from the Cauchy–Schwarz inequality, we have
Revised: Additionally, from the Cauchy–Schwarz inequality, we have

Original: [Equation]
Revised: [Equation]

Original: Θ
Revised: So, p

Original: [Equation]
Revised: [Equation]

Original: Θ (cid:48)
Revised: ,i = Pr θ

Original: [Equation]
Revised: [Equation]

Original: So, p tp,i = Pr θ ( (cid:98) θ i ∈ Θ (cid:48) m ) and from (10) the true positive probability of class i of r 1 is greater thanr 1 .
Revised: θ i ∈ Θ (cid:48) m ) and from (10) the true positive probability of class i of r 1 is greater thanr 1 .

Original: The error probability of r j is given by p er,j = 1 − (cid:80) ni
Revised: The error probability of r j is given by p er,j = 1 − (cid:80) ni

Original: =1 P i P tp,i,j , where P i is the prior probabilityof class i .
Revised: =1 P i P tp,i,j , where P i is the prior probabilityof class i .

Original: Therefor, p er, 1 ≤ p er, 2 . (cid:3)
Revised: Therefor, p er, 1 ≤ p er, 2 . (cid:3)

Original: I MPLEMENTATION D ETAILSF ACE R ECOGNITION
Revised: C ONNECTING THE THEOREMS WITH THE PROPOSED METHOD

Revised: Fig.

Revised: 6 shows the connection between the proposed theorems and the approach.

Revised: In part 1, Theorem2 connects the estimation of unknown parameters to the auxiliary classiﬁer.

Revised: In part 2, the learnedfeatures are passed to a decision making network (result of Theorem 2).

Revised: In part 3, Theorem 3guarantees that the multi-class classiﬁer outperforms other classiﬁers, because it is using the featuresfrom a better estimation of unknown parameters of pdf.

Revised: T OY EXAMPLE :

Revised: We demonstrate the effect of adding background class to the original classiﬁer with a toy exampleand visualize it in Fig.

Revised: In this example, the input is a sequence of binary bits ( +1 and − 1 ) withlength 3 in white Gaussian noise.

Revised: the classiﬁer is constructed using two fully connected layers withsigmoid and the last layer is normalized on unit circle.

Revised: As seen from Fig.

Revised: 7, adding an additionalnoise class visibly reduces the feature separation between all the other classes.

Revised:  I MPLEMENTATION D ETAILSF ACE R ECOGNITION

Original: We use Tensorﬂow (Abadi et al.
Revised: We use Tensorﬂow (Abadi et al.

Original: to conduct all our experiments.
Revised: to conduct all our experiments.

Original: We train with a batchsize of 256 on two NVIDIA TeslaV100 (32G) GPUs.
Revised: We train with a batchsize of 256 on two NVIDIA TeslaV100 (32G) GPUs.

Original: We train our models following small (lessthan 1M training images) and large (more than 1M training images) protocol conventions.
Revised: We train our models following small (lessthan 1M training images) and large (more than 1M training images) protocol conventions.

Original: We use
Revised: We use

Original: CASIA-Webface (Yi et al.
Revised: CASIA-Webface (Yi et al.

Original: (2014)) dataset for small protocol and MS1MV2 dataset for the large protocol.
Revised: (2014)) dataset for small protocol and MS1MV2 dataset for the largeprotocol.

Original: We use ResNet-50 (He et al.
Revised: We use ResNet-50 (He et al.

Original: and ResNet-100 models for small and large protocols,respectively.
Revised: and ResNet-100 models for small and large protocols,respectively.

Original: The PEF is selected as the ﬁrst three layers.
Revised: The PEF is selected as the ﬁrst three layers.

Original: Following (Deng et al.
Revised: Following (Deng et al.

Original: , we apply
Revised: , we apply

Original: BN (Ioffe & Szegedy (2015)), dropout (Srivastava et al.
Revised: BN (Ioffe & Szegedy (2015)), dropout (Srivastava et al.

Original: to the last feature map layer followedby a fully connected layer and batch normalization to obtain the 512-D embedding vector.
Revised: to the last feature map layer followedby a fully connected layer and batch normalization to obtain the 512-D embedding vector.

Original: We setthe feature scale s parameter to 64 following (Wang et al.
Revised: We setthe feature scale s parameter to 64 following (Wang et al.

Original: ; Deng et al.
Revised: ; Deng et al.

Original: and set themargin parameters ( m 1 , m 2 , m 3 ) to (1, 0.5, 0), respectively.
Revised: and set themargin parameters ( m 1 , m 2 , m 3 ) to (1, 0.5, 0), respectively.

Original: For small scale protocol, we start thelearning rate at 0.01 and divide the learning rate by 10 at 40K, 80K, and 100K iterations.
Revised: For small scale protocol, we start thelearning rate at 0.01 and divide the learning rate by 10 at 40K, 80K, and 100K iterations.

Original: We train for120K iterations.
Revised: We train for120K iterations.

Original: For large scale protocol, we start the learning rate at 0.01 and divide the learningrate by 10 at 80K, 100K, and 200K iterations.
Revised: For large scale protocol, we start the learning rate at 0.01 and divide the learningrate by 10 at 80K, 100K, and 200K iterations.

Original: We train for 240K iterations.
Revised: We train for 240K iterations.

Original: We use Momentumoptimizer and set the momentum to 0.9 and weight decay to 5e-4.
Revised: We use Momentumoptimizer and set the momentum to 0.9 and weight decay to 5e-4.

Original: We use the feature centre of allimages from a template or all frames from a video in order to report the results on IJB-B, IJB-C and
Revised: We use the feature centre of allimages from a template or all frames from a video in order to report the results on IJB-B, IJB-C and

Original: YTF datasets.
Revised: YTF datasets.

Original: For ABC-noise data, we cropped background images patches from MS1MV2 (Guoet al.
Revised: For ABC-noise data, we cropped background images patches from MS1MV2 (Guoet al.

Original: (2016)) dataset and cropped hard examples from the Caltech-101 (F. F. Li et al.
Revised: (2016)) dataset and cropped hard examples from the Caltech-101 (F. F. Li et al.

Original: datasetplus a few open sourced images (animal faces) using MTCNN (Zhang et al.
Revised: datasetplus a few open sourced images (animal faces) using MTCNN (Zhang et al.

Original: (2016a))
Revised: (2016a))

Original: face detector.
Revised: face detector.

Original: We generated roughly 500K non-face images for training the ABCs.
Revised: We generated roughly 500K non-face images for training the ABCs.

Original: S PEAKER I DENTIFICATION
Revised: S PEAKER I DENTIFICATION

Original: L2 loss and dropout with a rate of 0 .
Revised: L2 loss and dropout with a rate of 0 .

Original: 2 are applied during training for generalization.
Revised: 2 are applied during training for generalization.

Original: The ABC-noiseis collected form silence intervals of the VoxCeleb dataset, where an energy-based voice activitydetection (VAD) is applied to detect the silence intervals.
Revised: The ABC-noiseis collected form silence intervals of the VoxCeleb dataset, where an energy-based voice activitydetection (VAD) is applied to detect the silence intervals.

Original: To augment the ABC-noise, Gaussiannoise is added to the silence intervals.
Revised: To augment the ABC-noise, Gaussiannoise is added to the silence intervals.

Original: Each batch size is set to 64 and the optimizer is ADAM witha learning rate of 0 . 001 .
Revised: Each batch size is set to 64 and the optimizer is ADAM witha learning rate of 0 . 001 .

Original: The VoxCeleb dataset is trained for 11 epochs and the other datasets aretrained for 6 epochs.
Revised: The VoxCeleb dataset is trained for 11 epochs and the other datasets aretrained for 6 epochs.

Original: LFW AND YTF DATASETS
Revised: LFW AND YTF DATASETS

Original: LFW database contains the annotations for 5171 faces in a set of 2845 images taken from the Facesin the Wild data set (Berg et al.
Revised: LFW database contains the annotations for 5171 faces in a set of 2845 images taken from the Facesin the Wild data set (Berg et al.

Original: YouTubeFaces (Wolf et al.
Revised: YouTubeFaces (Wolf et al.

Original: (2011)) contains 3,425 videos of1,595 people.
Revised: (2011)) contains 3,425 videos of1,595 people.

Original: Following the standard convention, we report the results on 5000 video pairs usingunrestricted with labeled outside data protocol.
Revised: Following the standard convention, we report the results on 5000 video pairs usingunrestricted with labeled outside data protocol.

Original: IJB-B AND IJB-C DATASETS
Revised: IJB-B AND IJB-C DATASETS

Original: The IJB-B contains 1,845 subjects with 21.8K still images and 55K frames from 7,011 videos.
Revised: The IJB-B contains 1,845 subjects with 21.8K still images and 55K frames from 7,011 videos.

Original: Intotal, there are 12,115 templates with 10,270 genuine matches and 8M impostor matches.
Revised: Intotal, there are 12,115 templates with 10,270 genuine matches and 8M impostor matches.

Original: The IJB-Cdataset (Maze et al.
Revised: The IJB-Cdataset (Maze et al.

Original: (2018)) is a further extension of IJB-B, having 3,531 subjects with 31.3K stillimages and 117.5K frames from 11,779 videos.
Revised: (2018)) is a further extension of IJB-B, having 3,531 subjects with 31.3K stillimages and 117.5K frames from 11,779 videos.

Original: In total, there are 23, 124 templates with 19,557genuine matches and 15, 639K impostor matches.
Revised: In total, there are 23, 124 templates with 19,557genuine matches and 15, 639K impostor matches.