Original: L EARNING NON - LINEAR TRANSFORM WITH DISCRIMINATIVE AND MINIMUM INFORMATION LOSS PRIORS
Revised: L EARNING NON - LINEAR TRANSFORM WITH DISCRIMINATIVE AND MINIMUM INFORMATION LOSS PRIORS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper proposes learning a non-linear transform with two priors.
Revised: This paper proposes learning a non-linear transform with two priors.

Original: The ﬁrst is a discriminative prior deﬁned using a measures on a support intersection and the second is a minimum information loss prior expressed as a constraint on the conditioning and the coherence.
Revised: The ﬁrst is a discriminative prior deﬁned using a measures on a support intersection and the second is a minimum information loss prior expressed as a constraint on the conditioning and the coherence.

Original: An approximation of the measures for the discriminative prior is addressed, connecting it to a similarity concentrations.
Revised: An approximation of the measures for the discriminative prior is addressed, connecting it to a similarity concentrations.

Original: Along quantifying the discriminative properties of the transform representation a sensitivity analysis of the similarity concentration w.r.t.
Revised: Along quantifying the discriminative properties of the transform representation a sensitivity analysis of the similarity concentration w.r.t.

Original: the parameters of the nonlinear transform is given.
Revised: the parameters of the nonlinear transform is given.

Original: Furthermore, a measure, related to the similarity concentration, reﬂecting the discriminative properties, named as discrimination power is introduced and its bounds are presented.
Revised: Furthermore, a measure, related to the similarity concentration, reﬂecting the discriminative properties, named as discrimination power is introduced and its bounds are presented.

Original: To support and validate the theoretical analysis a learning algorithm with the proposed prior is presented.
Revised: To support and validate the theoretical analysis a learning algorithm with the proposed prior is presented.

Original: The advantages and the potential of the proposed algorithm are evaluated by a computer simulation
Revised: The advantages and the potential of the proposed algorithm are evaluated by a computer simulation

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Learning a transform that provides a sparse and discriminative representation is an active domain of research in many areas, including data processing, pattern recognition, image processing, language modeling, text analysis, gene separation, etc.
Revised: Learning a transform that provides a sparse and discriminative representation is an active domain of research in many areas, including data processing, pattern recognition, image processing, language modeling, text analysis, gene separation, etc.

Original: A class of algorithms proposed by Kreutz-Delgado et al.
Revised: A class of algorithms proposed by Kreutz-Delgado et al.

Original: , Mairal et al.
Revised: , Mairal et al.

Original: (2009), Bengio et al.
Revised: (2009), Bengio et al.

Original: (2012), Gangeh et al.
Revised: (2012), Gangeh et al.

Original: , Mairal et al.
Revised: , Mairal et al.

Original: Jiang et al.
Revised: Jiang et al.

Original: (2011), Guo et al.
Revised: (2011), Guo et al.

Original: (2012), Cai et al.
Revised: (2012), Cai et al.

Original: (2014) and Liu et al.
Revised: (2014) and Liu et al.

Original: (2016) for learning discriminative sparse representation have been shown to perform well across various learning tasks.
Revised: (2016) for learning discriminative sparse representation have been shown to perform well across various learning tasks.

Original: A subclass of them known as discriminative dictionary learning (DDL) Guo et al.
Revised: A subclass of them known as discriminative dictionary learning (DDL) Guo et al.

Original: , Jiang et al.
Revised: , Jiang et al.

Original: (2013), Cai et al.
Revised: (2013), Cai et al.

Original: (2014), Shekhar et al.
Revised: (2014), Shekhar et al.

Original: (2014), Xu et al.
Revised: (2014), Xu et al.

Original: Liu et al.
Revised: Liu et al.

Original: (2016), Bengio et al.
Revised: (2016), Bengio et al.

Original: (2012), Gangeh et al.
Revised: (2012), Gangeh et al.

Original: (2015), con, Jiang et al.
Revised: (2015), con, Jiang et al.

Original: (2016), Vu & Monga (2016b) address the estimate of the dictionary in a supervised meaner such that the representation w.r.t.
Revised: (2016), Vu & Monga (2016b) address the estimate of the dictionary in a supervised meaner such that the representation w.r.t.

Original: words (vectors) from the resulting dictionary (vector set) is discriminative.
Revised: words (vectors) from the resulting dictionary (vector set) is discriminative.

Original: Most of the DDL methods synthesize the data sample x c,k ∈ (cid:60) N as an approximation by a linear combination y c,k ∈ (cid:60) M (refereed to as a sparse data representation) of a few words (vectors) (cid:107) y c,k (cid:107) 0 << M , from a dictionary (vector set)
Revised: Most of the DDL methods synthesize the data sample x c,k ∈ (cid:60) N as an approximation by a linear combination y c,k ∈ (cid:60) M (refereed to as a sparse data representation) of a few words (vectors) (cid:107) y c,k (cid:107) 0 << M , from a dictionary (vector set)

Original: D ∈ (cid:60) N × M , i.e. , x c,k = Dy c,k
Revised: D ∈ (cid:60) N × M , i.e. , x c,k = Dy c,k

Original: + v c,k , v c,k ∈ (cid:60) N , with v c,k denoting the approximation error.
Revised: + v c,k , v c,k ∈ (cid:60) N , with v c,k denoting the approximation error.

Original: It is important to highlight that with the synthesis model approach the data reconstruction is addressed.
Revised: It is important to highlight that with the synthesis model approach the data reconstruction is addressed.

Original: The differences between the DDL methods Guo et al.
Revised: The differences between the DDL methods Guo et al.

Original: , Jiang et al.
Revised: , Jiang et al.

Original: (2013), Cai et al.
Revised: (2013), Cai et al.

Original: (2014), Shekhar et al.
Revised: (2014), Shekhar et al.

Original: (2014), Gangeh et al.
Revised: (2014), Gangeh et al.

Original: (2015), Xu et al.
Revised: (2015), Xu et al.

Original: Liu et al.
Revised: Liu et al.

Original: (2016), Bengio et al.
Revised: (2016), Bengio et al.

Original: , Jiang et al.
Revised: , Jiang et al.

Original: Vu & Monga (2016b) are determined by the prior deﬁned on the sparse representation and the prior deﬁned for the relations between the sparse representations for the data samples from same/different classes.
Revised: Vu & Monga (2016b) are determined by the prior deﬁned on the sparse representation and the prior deﬁned for the relations between the sparse representations for the data samples from same/different classes.

Original: The discrimination is enforced by approximating the prior with a structural constraints on the dictionary or imposing a discriminative term on the sparse representation.
Revised: The discrimination is enforced by approximating the prior with a structural constraints on the dictionary or imposing a discriminative term on the sparse representation.

Original: Additionally, in a number of works such as Mairal et al.
Revised: Additionally, in a number of works such as Mairal et al.

Original: (2008), Guo et al.
Revised: (2008), Guo et al.

Original: Taalimi et al.
Revised: Taalimi et al.

Original: (2015) is considered even a joint estimation/learning of dictionary, sparse representation and classiﬁer by using iterative alternating minimization strategy.
Revised: (2015) is considered even a joint estimation/learning of dictionary, sparse representation and classiﬁer by using iterative alternating minimization strategy.

Original: A comprehensive overview covering different approaches is given in Bengio et al.
Revised: A comprehensive overview covering different approaches is given in Bengio et al.

Original: (2012), Cai et al.
Revised: (2012), Cai et al.

Original: and Gangeh et al.
Revised: and Gangeh et al.

Original: 1. O PEN ISSUES
Revised: 1. O PEN ISSUES

Original: The general open issue for the DDL methods is the computational complexity w.r.t.
Revised: The general open issue for the DDL methods is the computational complexity w.r.t.

Original: the optimal dictionary/transform learning and the discriminative encoding, since the sparse representation in the synthesis model is a solution to an inverse problem .
Revised: the optimal dictionary/transform learning and the discriminative encoding, since the sparse representation in the synthesis model is a solution to an inverse problem .

Original: An additional open issue with most of the proposed approaches Guo et al.
Revised: An additional open issue with most of the proposed approaches Guo et al.

Original: , Jiang et al.
Revised: , Jiang et al.

Original: (2013), Cai et al.
Revised: (2013), Cai et al.

Original: (2014), Gangeh et al.
Revised: (2014), Gangeh et al.

Original: Liu et al.
Revised: Liu et al.

Original: (2016), Bengio et al.
Revised: (2016), Bengio et al.

Original: , Jiang et al.
Revised: , Jiang et al.

Original: and Vu & Monga (2016b) is that there is no formal notion to measure the discriminative properties.
Revised: and Vu & Monga (2016b) is that there is no formal notion to measure the discriminative properties.

Original: Therefore, there are no means that provide a quantitative evaluation about the quality of the representation, other then the performance of a classiﬁer used on top of the representation.
Revised: Therefore, there are no means that provide a quantitative evaluation about the quality of the representation, other then the performance of a classiﬁer used on top of the representation.

Original: In therms of the speciﬁcs about the discriminative constraints, Yang et al. (2011b) proposed a synthesis model with a discriminative ﬁdelity term and Fisher discriminant constraints, where the withinclass scatter and the between-class scatter of the representation is minimized and maximized, respectively.
Revised: In therms of the speciﬁcs about the discriminative constraints, Yang et al. (2011b) proposed a synthesis model with a discriminative ﬁdelity term and Fisher discriminant constraints, where the withinclass scatter and the between-class scatter of the representation is minimized and maximized, respectively.

Original: Vu & Monga (2016c) proposed extension considering a low rank constraints on the dictionary.
Revised: Vu & Monga (2016c) proposed extension considering a low rank constraints on the dictionary.

Original: The authors in Guo et al.
Revised: The authors in Guo et al.

Original: (2013) used a synthesis model with a pairwise constraints on the sparse representation.
Revised: (2013) used a synthesis model with a pairwise constraints on the sparse representation.

Original: They have modeled the pair-wise constraints using a (cid:96) 2 distance metric.
Revised: They have modeled the pair-wise constraints using a (cid:96) 2 distance metric.

Original: The methods of
Revised: The methods of

Original: Yang et al.
Revised: Yang et al.

Original: (2011b), Vu & Monga (2016c) and Guo et al.
Revised: (2011b), Vu & Monga (2016c) and Guo et al.

Original: (2013) take into account assumption on the metric by deﬁning the scatter and the pair-wise relation.
Revised: (2013) take into account assumption on the metric by deﬁning the scatter and the pair-wise relation.

Original: Therefore, they constrain the space of the representation, that essentially is determined by the dictionary.
Revised: Therefore, they constrain the space of the representation, that essentially is determined by the dictionary.

Original: However, these works do not consider whether that the used metric is optimal for the sparse representation.
Revised: However, these works do not consider whether that the used metric is optimal for the sparse representation.

Original: The authors in Liu et al.
Revised: The authors in Liu et al.

Original: (2016) have proposed a method that ﬁnds a dictionary under which the coefﬁcients of a data sample from the same class
Revised: (2016) have proposed a method that ﬁnds a dictionary under which the coefﬁcients of a data sample from the same class

Original: c have a common sparse structure while the size of the overlapped signal support of different classes, denoted as c 1 and c 2 is minimized.
Revised: c have a common sparse structure while the size of the overlapped signal support of different classes, denoted as c 1 and c 2 is minimized.

Original: Furthermore, assuming y c 1 ,k 1 ∈ (cid:60) M and y c 2 ,k 2 ∈ (cid:60) M are two sparse representations for data samples x c 1 ,k 1 ∈ (cid:60) N and x c 2 ,k 2 ∈ (cid:60) N , then the proposed similarity measure in Liu et al.
Revised: Furthermore, assuming y c 1 ,k 1 ∈ (cid:60) M and y c 2 ,k 2 ∈ (cid:60) M are two sparse representations for data samples x c 1 ,k 1 ∈ (cid:60) N and x c 2 ,k 2 ∈ (cid:60) N , then the proposed similarity measure in Liu et al.

Original: (2016) is deﬁned as expectation of (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 0 , where (cid:12) represents the Hadamard product.
Revised: (2016) is deﬁned as expectation of (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 0 , where (cid:12) represents the Hadamard product.

Original: Note that two transform data samples y c 1 ,k 2 and y c 2 ,kthat have small support overlap (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 

Revised: Note that two transform data samples y c 1 ,k 2 and y c 2 ,kthat have small support overlap (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107)

Original: = s, s << M , might not necessarily be similar or dissimilar, i.e. , y c 1 ,k 1 = y c 2 ,k 2 and y c 1 ,k 1 = − y c 2 ,k 2 with (cid:107) y c 1 ,k 1 (cid:107) 

Revised: = s, s << M , might not necessarily be similar or dissimilar, i.e. , y c 1 ,k 1 = y c 2 ,k 2 and y c 1 ,k 1 = − y c 2 ,k 2 with (cid:107) y c 1 ,k 1 (cid:107)

Original: = (cid:107) y c
Revised: = (cid:107) y c

Original: ,k 2 (cid:107) 

Revised: ,k 2 (cid:107)

Original: = s and s small.
Revised: = s and s small.

Original: M OTIVATION AND APPROACH
Revised: M OTIVATION AND APPROACH

Original: This paper addresses a non-linear transform model , where the data reconstruction is not modeled, and at least it is not targeted explicitly.
Revised: This paper addresses a non-linear transform model , where the data reconstruction is not modeled, and at least it is not targeted explicitly.

Original: Moreover, the model does not restrict the transform representation to be in the column space of the dictionary, allowing a rich class of representations to be modeled.
Revised: Moreover, the model does not restrict the transform representation to be in the column space of the dictionary, allowing a rich class of representations to be modeled.

Original: In this line, we have to mention that a special case of the non-linear transform model is the sparsifying transform model , ﬁrst introduced in Ravishankar & Bresler (2012).
Revised: In this line, we have to mention that a special case of the non-linear transform model is the sparsifying transform model , ﬁrst introduced in Ravishankar & Bresler (2012).

Original: This model assumes that the data sample x is approximately sparsiﬁable under a linear transform A ∈ (cid:60) M × N , i.e. , Ax c,k
Revised: This model assumes that the data sample x is approximately sparsiﬁable under a linear transform A ∈ (cid:60) M × N , i.e. , Ax c,k

Original: = y c,k
Revised: = y c,k

Original: + z c,k , z c,k ∈ (cid:60) M , where y c,k is sparse (cid:107) y c,k (cid:107) 0 << M .
Revised: + z c,k , z c,k ∈ (cid:60) M , where y c,k is sparse (cid:107) y c,k (cid:107) 0 << M .

Original: Moreover, we note that
Revised: Moreover, we note that

Original: [Figure]
Revised: [Figure]

Original: the sparsiﬁying transform model represents a generalization of the analysis model Rubinstein et al.
Revised: the sparsiﬁying transform model represents a generalization of the analysis model Rubinstein et al.

Original: (2010), Rubinstein et al.
Revised: (2010), Rubinstein et al.

Original: (2013), Ravishankar & Bresler (2014) and Rubinstein & Elad (2014).
Revised: (2013), Ravishankar & Bresler (2014) and Rubinstein & Elad (2014).

Original: Together with the non-linear transform model, a novel prior is proposed, which is deﬁned by a parametric measures on the support intersection .
Revised: Together with the non-linear transform model, a novel prior is proposed, which is deﬁned by a parametric measures on the support intersection .

Original: The ﬁrst motivation behind using a measure deﬁned on the support intersection is that it allows more freedom in imposing a regularization on the discriminative properties for the transform representation without taking into account any additional assumptions, contrary to Yang et al.
Revised: The ﬁrst motivation behind using a measure deﬁned on the support intersection is that it allows more freedom in imposing a regularization on the discriminative properties for the transform representation without taking into account any additional assumptions, contrary to Yang et al.

Original: (2011b), Vu & Monga (2016c), Guo et al.
Revised: (2011b), Vu & Monga (2016c), Guo et al.

Original: and Liu et al.
Revised: and Liu et al.

Original: Second, by a simple approximation on the parametric measure the focus of the regularization can be directly put on the contributing components for the similarity, i.e. , consider the measure ( y + c 1 ,k 1 ) T y + c 2 ,k 2 + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 1 between two transform representations y c 1 ,k 1 and y c 2 ,k 2 .
Revised: Second, by a simple approximation on the parametric measure the focus of the regularization can be directly put on the contributing components for the similarity, i.e. , consider the measure ( y + c 1 ,k 1 ) T y + c 2 ,k 2 + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 1 between two transform representations y c 1 ,k 1 and y c 2 ,k 2 .

Original: This measure captures the main contributing components for similarity on the support intersection, whereas ( y + c 1 ,k 1 ) T y − c 2 ,k 

Revised: This measure captures the main contributing components for similarity on the support intersection, whereas ( y + c 1 ,k 1 ) T y − c 2 ,k

Original: +( y − c 1 ,k 1 ) T y + c 2 ,k 2 captures the main contribution for dissimilarity between y c 1 ,k 1 and y c 2 ,k 2 .
Revised: +( y − c 1 ,k 1 ) T y + c 2 ,k 2 captures the main contribution for dissimilarity between y c 1 ,k 1 and y c 2 ,k 2 .

Original: Third, the expectation of this similarity measure across a the data set captures the concentration of similarity for that set.
Revised: Third, the expectation of this similarity measure across a the data set captures the concentration of similarity for that set.

Original: Therefore, it provides the possibility to deﬁne a formal notion to quantify the discriminative properties.
Revised: Therefore, it provides the possibility to deﬁne a formal notion to quantify the discriminative properties.

Original: Forth, the measure ( y + c 1 ,k 1 ) T y + c 2 ,k 2 + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 is not ambiguous w.r.t.
Revised: Forth, the measure ( y + c 1 ,k 1 ) T y + c 2 ,k 2 + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 is not ambiguous w.r.t.

Original: a notion for similarity/dissimilarity between two sparse representations y c 1 ,kand y c 2 ,k 2 .
Revised: a notion for similarity/dissimilarity between two sparse representations y c 1 ,kand y c 2 ,k 2 .

Original: This is because the support intersections for the positive and the negative components (cid:107) y
Revised: This is because the support intersections for the positive and the negative components (cid:107) y

Original: + c 1 ,k 1 (cid:12) y + c 2 ,k 2 (cid:107) 1 and (cid:107) y − c 1 ,k 1 (cid:12) y − c 2 ,k 2 (cid:107) 1 , respectively, are considered separately.
Revised: + c 1 ,k 1 (cid:12) y + c 2 ,k 2 (cid:107) 1 and (cid:107) y − c 1 ,k 1 (cid:12) y − c 2 ,k 2 (cid:107) 1 , respectively, are considered separately.

Original: In addition a measure for the strength on the support intersection, deﬁned as (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 22 is also taken into account.
Revised: In addition a measure for the strength on the support intersection, deﬁned as (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 22 is also taken into account.

Original: Note that the expectation of the measure (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 22 across a the data set captures the expected strength on the support intersection for that set.
Revised: Note that the expectation of the measure (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 22 across a the data set captures the expected strength on the support intersection for that set.

Original: A schematic diagram of the transform and the main idea behind the proposed concept are shown in
Revised: A schematic diagram of the transform and the main idea behind the proposed concept are shown in

Original: Figure 1.
Revised: Figure 1.

Original: In Figure 2 a), b), c) is given an illustrative example for the support intersection between the positive and negative component of two vectors y c 1 ,k 1 and y c 2 ,k 2 in the transform domain.
Revised: In Figure 2 a), b), c) is given an illustrative example for the support intersection between the positive and negative component of two vectors y c 1 ,k 1 and y c 2 ,k 2 in the transform domain.

Original: We propose a learning algorithm for the non-linear transform model with the discriminative prior.
Revised: We propose a learning algorithm for the non-linear transform model with the discriminative prior.

Original: In a direction of quantifying the discriminative properties we give a sensitivity analysis w.r.t.
Revised: In a direction of quantifying the discriminative properties we give a sensitivity analysis w.r.t.

Original: the parameters of the transform.
Revised: the parameters of the transform.

Original: Along the same line, we present an information preservation rela1 y c 1 ,k 

Revised: Along the same line, we present an information preservation rela1 y c 1 ,k

Original: =
Revised: =

Original: y
Revised: y

Original: + c 1 ,k 1 − y − c 1 ,k 

Revised: + c 1 ,k 1 − y − c 1 ,k

Original: where y + c 1 ,k 1 = max( y c 1 ,k 1 , 0 ) and y − c 1 ,k 1 = max( − y c 1 ,k 1 , 0 ) .
Revised: where y + c 1 ,k 1 = max( y c 1 ,k 1 , 0 ) and y − c 1 ,k 1 = max( − y c 1 ,k 1 , 0 ) .

Original: Note that ( y + c 1 ,k 1 ) T y + c 2 ,k 2 +( y − c 1 ,k 1 ) T y − c 2 ,k 2 = (cid:107) y + c 1 ,k 1 (cid:12) y + c 2 ,k 2 (cid:107) 1 + (cid:107) y − c 1 ,k 1 (cid:12) y − c 2 ,k 2 (cid:107) 1 captures the only contribution for the similarity (whereas (cid:107) y + c 1 ,k 1 (cid:12) y − c 2 ,k 2 (cid:107) 1 + (cid:107) y − c 1 ,k 1 (cid:12) y + c 2 ,k 2 (cid:107) 1 captures the only contribution for the dissimilarity) between the vectors y c 1 ,k 1 and y c 2 ,k 2 .
Revised: Note that ( y + c 1 ,k 1 ) T y + c 2 ,k 2 +( y − c 1 ,k 1 ) T y − c 2 ,k 2 = (cid:107) y + c 1 ,k 1 (cid:12) y + c 2 ,k 2 (cid:107) 1 + (cid:107) y − c 1 ,k 1 (cid:12) y − c 2 ,k 2 (cid:107) 1 captures the only contribution for the similarity (whereas (cid:107) y + c 1 ,k 1 (cid:12) y − c 2 ,k 2 (cid:107) 1 + (cid:107) y − c 1 ,k 1 (cid:12) y + c 2 ,k 2 (cid:107) 1 captures the only contribution for the dissimilarity) between the vectors y c 1 ,k 1 and y c 2 ,k 2 .

Original: Moreover, it can be shown that for certain models under some conditions y
Revised: Moreover, it can be shown that for certain models under some conditions y

Original: Tc 1 ,k 1 y c 2 ,k 

Revised: Tc 1 ,k 1 y c 2 ,k

Original: = (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 1 , if and only if the dissimilarity contribution on the support intersection between the vectors y c 1 ,k 1 and y c 2 ,kis 0 .
Revised: = (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 1 , if and only if the dissimilarity contribution on the support intersection between the vectors y c 1 ,k 1 and y c 2 ,kis 0 .

Original: That is y
Revised: That is y

Original: Tc
Revised: Tc

Original: 1 ,k 1 y c 2 ,k 2 =
Revised: 1 ,k 1 y c 2 ,k 2 =

Original: ( y + c 1 ,k 1 ) T y + c 2 ,k 

Revised: ( y + c 1 ,k 1 ) T y + c 2 ,k

Original: + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 − ( y − c 1 ,k 1 ) T y + c 2 ,k 2 − ( y + c 1 ,k 1 ) T y − c 2 ,k 2 ≤ ( y + c 1 ,k 1 )
Revised: + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 − ( y − c 1 ,k 1 ) T y + c 2 ,k 2 − ( y + c 1 ,k 1 ) T y − c 2 ,k 2 ≤ ( y + c 1 ,k 1 )

Original: T y + c 2 ,k 

Revised: T y + c 2 ,k

Original: +( y − c 1 ,k 1 ) T y − c 2 ,k 2 , now if the dissimilarity contribution − ( y − c 1 ,k 1 ) T y + c 2 ,k 2 − ( y + c 1 ,k 1 )
Revised: +( y − c 1 ,k 1 ) T y − c 2 ,k 2 , now if the dissimilarity contribution − ( y − c 1 ,k 1 ) T y + c 2 ,k 2 − ( y + c 1 ,k 1 )

Original: T y − c 2 ,kis zero then (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k
Revised: T y − c 2 ,kis zero then (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: non-linear parametric function with parameters P
Revised: non-linear parametric function with parameters P

Original: [Table]
Revised: [Table]

Original: similarity measure between T P ( x c 1 ,k 1 ) and T P ( x c 1 ,k 2 )
Revised: similarity measure between T P ( x c 1 ,k 1 ) and T P ( x c 1 ,k 2 )

Original: [Table]
Revised: [Table]

Original: x
Revised: x

Original: [Table]
Revised: [Table]

Original: discrimination power in original and
Revised: discrimination power in original and

Original: [Table]
Revised: [Table]

Original: domain
Revised: domain

Original: change of the parameters of the transform connecting all together the proposed prior, the model and the used non-linear transform.
Revised: change of the parameters of the transform connecting all together the proposed prior, the model and the used non-linear transform.

Original: Further more, w.r.t.
Revised: Further more, w.r.t.

Original: the similarity concentrations a notion deﬁned as discrimination power , reﬂecting the discriminate properties of the representation for a data set is introduced.
Revised: the similarity concentrations a notion deﬁned as discrimination power , reﬂecting the discriminate properties of the representation for a data set is introduced.

Original: Based on the sensitivity analysis we present lower and upper bound on the discrimination power in the transform domain.
Revised: Based on the sensitivity analysis we present lower and upper bound on the discrimination power in the transform domain.

Original: On the practical side, the advantages and the potential of the proposed algorithm are demonstrated by a numerical experiments using the Extended YALE B Georghiades et al.
Revised: On the practical side, the advantages and the potential of the proposed algorithm are demonstrated by a numerical experiments using the Extended YALE B Georghiades et al.

Original: (2001), AR Mart ´ ınez &
Revised: (2001), AR Mart ´ ınez &

Original: Benavente (1998), Norb LeCun et al.
Revised: Benavente (1998), Norb LeCun et al.

Original: (2004), Coil- 20 Nene et al.
Revised: (2004), Coil- 20 Nene et al.

Original: , Clatech 101 LeCun et al.
Revised: , Clatech 101 LeCun et al.

Original: (2008), UKB Nist ´ er & Stew ´ enius (2006) and MNIST Lecun & Cortes data sets.
Revised: (2008), UKB Nist ´ er & Stew ´ enius (2006) and MNIST Lecun & Cortes data sets.

Original: 1. N OTATIONS
Revised: 1. N OTATIONS

Original: A scalar variable is denoted using the usual symbols, i.e. , x , a vector is denoted by a bold, low caps symbols, i.e. , x , a matrix by bold, upper cap symbol, i.e. , A .
Revised: A scalar variable is denoted using the usual symbols, i.e. , x , a vector is denoted by a bold, low caps symbols, i.e. , x , a matrix by bold, upper cap symbol, i.e. , A .

Original: A single element from a vector (or matrix) is denoted as x ( n ) (or A ( m, n ) ).
Revised: A single element from a vector (or matrix) is denoted as x ( n ) (or A ( m, n ) ).

Original: A set is denoted by a calligraphic symbol, i.e , S .
Revised: A set is denoted by a calligraphic symbol, i.e , S .

Original: The (cid:96) p − norm is denoted as (cid:107) .
Revised: The (cid:96) p − norm is denoted as (cid:107) .

Original: (cid:107) p and the nuclear norm as (cid:107) .
Revised: (cid:107) p and the nuclear norm as (cid:107) .

Original: (cid:107) ∗ .
Revised: (cid:107) ∗ .

Original: The (cid:12) symbol represents the Hadamard product.
Revised: The (cid:12) symbol represents the Hadamard product.

Original: Throughout the paper it is assumed that a set of data samples X =
Revised: Throughout the paper it is assumed that a set of data samples X =

Original: [ X 1 , X 2 , ..., X C ] ∈ (cid:60) N × L , L =
Revised: [ X 1 , X 2 , ..., X C ] ∈ (cid:60) N × L , L =

Original: CK from C classes is given and that every class c ∈ C
Revised: CK from C classes is given and that every class c ∈ C

Original: = { 1 , 2 , 3 , ..., C } has K samples,
Revised: = { 1 , 2 , 3 , ..., C } has K samples,

Original: X c
Revised: X c

Original: =
Revised: =

Original: [ x c, 1 , x c, 2 , ..., x c,K ] ∈ (cid:60) N × K , x c,k ∈ (cid:60) N , ∀ c ∈ C , ∀ k ∈ K
Revised: [ x c, 1 , x c, 2 , ..., x c,K ] ∈ (cid:60) N × K , x c,k ∈ (cid:60) N , ∀ c ∈ C , ∀ k ∈ K

Original: = { 1 , 2 , ..., K } .
Revised: = { 1 , 2 , ..., K } .

Original: We denote a the transform data as Y =
Revised: We denote a the transform data as Y =

Original: [ Y 1 , Y 2 , ..., Y C ] ∈ (cid:60) M × L , where Y c =
Revised: [ Y 1 , Y 2 , ..., Y C ] ∈ (cid:60) M × L , where Y c =

Original: [ y c, 1 , y c, 2 , ..., y c,K ] ∈ (cid:60) M × K and y c,k ∈ (cid:60) M .
Revised: [ y c, 1 , y c, 2 , ..., y c,K ] ∈ (cid:60) M × K and y c,k ∈ (cid:60) M .

Original: We denote X \{ k ∈ c } = (cid:2)
Revised: We denote X \{ k ∈ c } = (cid:2)

Original: X 1 , X 2 , ..
Revised: X 1 , X 2 , ..

Original: X c, \ k , ...
Revised: X c, \ k , ...

Original: X C (cid:3) ∈ (cid:60) N × ( L − 1) as the matrix that has all the columns of X , except the column x c,k ∈ (cid:60) N , where X c, \ k =
Revised: X C (cid:3) ∈ (cid:60) N × ( L − 1) as the matrix that has all the columns of X , except the column x c,k ∈ (cid:60) N , where X c, \ k =

Original: [ x c, 1 , x c, 2 , ..., x c,k − 1 , x c,k +1 , ..., x c,K ] ∈ (cid:60) N × ( K − 1) is a matrix that has all the columns of block X c , except the column x c,k , ∀ c ∈ C and ∀ k ∈ K .
Revised: [ x c, 1 , x c, 2 , ..., x c,k − 1 , x c,k +1 , ..., x c,K ] ∈ (cid:60) N × ( K − 1) is a matrix that has all the columns of block X c , except the column x c,k , ∀ c ∈ C and ∀ k ∈ K .

Original: We let N = { 1 , 2 , ..., N } and M = { 1 , 2 , ..., M } .
Revised: We let N = { 1 , 2 , ..., N } and M = { 1 , 2 , ..., M } .

Original: We denote D M as the set of all M × M diagonal matrices with non-negative diagonal elements.
Revised: We denote D M as the set of all M × M diagonal matrices with non-negative diagonal elements.

Original:  L EARNING NON - LINEAR TRANSFORM WITH DISCRIMINATIVE AND MINIMUM INFORMATION LOSS PRIORS
Revised:  L EARNING NON - LINEAR TRANSFORM WITH DISCRIMINATIVE AND MINIMUM INFORMATION LOSS PRIORS

Original: Assume that training data X =
Revised: Assume that training data X =

Original: [ X 1 , X 2 , ..., X C ] ∈ (cid:60) N × CK are given, consisting of C classes.
Revised: [ X 1 , X 2 , ..., X C ] ∈ (cid:60) N × CK are given, consisting of C classes.

Original: Per every class c ∈ C there are K training samples, i.e. , X c
Revised: Per every class c ∈ C there are K training samples, i.e. , X c

Original: =
Revised: =

Original: [ x c, 1 , x c, 2 , ..., x c,K ] .
Revised: [ x c, 1 , x c, 2 , ..., x c,K ] .

Original: Furthermore, we assume that per class c ∈ C there exist an unknown non-linear functions deﬁned by set of parameters:
Revised: Furthermore, we assume that per class c ∈ C there exist an unknown non-linear functions deﬁned by set of parameters:

Original: [Equation]
Revised: [Equation]

Original: that separates the data samples from different classes in the transform domain.
Revised: that separates the data samples from different classes in the transform domain.

Original: The linear map A is shared for all of them, but, the parameters τ c per different class c from the corresponding non-linear transforms P c are different.
Revised: The linear map A is shared for all of them, but, the parameters τ c per different class c from the corresponding non-linear transforms P c are different.

Original: Additionally, given any x c,k from class c where k ∈ K it is assumed that the non-linear transform is expressible in two steps, consisting of a linear mapping (step 1 ) followed by an element-wise non-linearity (step 2 ), as follows:
Revised: Additionally, given any x c,k from class c where k ∈ K it is assumed that the non-linear transform is expressible in two steps, consisting of a linear mapping (step 1 ) followed by an element-wise non-linearity (step 2 ), as follows:

Original: [Equation]
Revised: [Equation]

Original: where H τ c , ( . ) : (cid:60) M
Revised: where H τ c , ( . ) : (cid:60) M

Original: → (cid:60) M is a non-linear thresholding function with parameters τ c .
Revised: → (cid:60) M is a non-linear thresholding function with parameters τ c .

Original: It is important to mention that the thresholding is done with a different thresholding parameter per different transform dimension, i.e. , τ c ( m ) , ∀ m ∈ M .
Revised: It is important to mention that the thresholding is done with a different thresholding parameter per different transform dimension, i.e. , τ c ( m ) , ∀ m ∈ M .

Original: The goal of learning a non-linear transform (2) is to estimate only one model deﬁned by a set of parameters
Revised: The goal of learning a non-linear transform (2) is to estimate only one model deﬁned by a set of parameters

Original: [Equation]
Revised: [Equation]

Original: that is common for all data samples that come form all classes by taking into account a set of priors.
Revised: that is common for all data samples that come form all classes by taking into account a set of priors.

Original: That is, we try to ﬁnd as accurate as possible approximation (3) to the models represented by a set of parameters P c
Revised: That is, we try to ﬁnd as accurate as possible approximation (3) to the models represented by a set of parameters P c

Original: =
Revised: =

Original: { A ∈ (cid:60) M × N , τ c ∈ (cid:60) M } with as small as possible loss in the discriminative properties of the representation in the transform domain.
Revised: { A ∈ (cid:60) M × N , τ c ∈ (cid:60) M } with as small as possible loss in the discriminative properties of the representation in the transform domain.

Original: 2. T HE NON - LINEAR TRANSFORM MODEL WITH A DISCRIMINATIVE PRIOR
Revised: 2. T HE NON - LINEAR TRANSFORM MODEL WITH A DISCRIMINATIVE PRIOR

Original: The compact description of the non-linear transform (2) by a non-linear model is expressed as follows:
Revised: The compact description of the non-linear transform (2) by a non-linear model is expressed as follows:

Original: [Equation]
Revised: [Equation]

Original: where the function T P c ( . ) : (cid:60) N → (cid:60) M is a parametric non-linear function that gives y c,k , by using the set of parameters P c .
Revised: where the function T P c ( . ) : (cid:60) N → (cid:60) M is a parametric non-linear function that gives y c,k , by using the set of parameters P c .

Original: The term z c,k
Revised: The term z c,k

Original: = Ax c,k − y c,k is the non-linear transform error vector that represents the deviation of the transform data
Revised: = Ax c,k − y c,k is the non-linear transform error vector that represents the deviation of the transform data

Original: Ax c,k from the targeted transform representation y c,k = T P c ( x c,k ) in the transform domain.
Revised: Ax c,k from the targeted transform representation y c,k = T P c ( x c,k ) in the transform domain.

Original: Assuming Gaussian distributed error vector, the prior on z c,k is modeled as p ( x c,k | y c,k ) ∝ exp( − (cid:107)
Revised: Assuming Gaussian distributed error vector, the prior on z c,k is modeled as p ( x c,k | y c,k ) ∝ exp( − (cid:107)

Original: Ax c,k − y c,k (cid:107) 2 β) , additionally, assuming that the non-linear function T P c ( x c,k ) gives sparse y c,k , then we have the improper prior on y c,k , deﬁned as p ( y c,k )
Revised: Ax c,k − y c,k (cid:107) 2 β) , additionally, assuming that the non-linear function T P c ( x c,k ) gives sparse y c,k , then we have the improper prior on y c,k , deﬁned as p ( y c,k )

Original: ∝ exp( − (cid:107) y c,k (cid:107) 1 β) .
Revised: ∝ exp( − (cid:107) y c,k (cid:107) 1 β) .

Original: This paper models the discriminative prior as p ( τ c , | y c,k ) ∝ exp( − D ( y c,k ; τ c ) β) by using an parametric measure D ( y c,k ; τ c ) with parameters τ c .
Revised: This paper models the discriminative prior as p ( τ c , | y c,k ) ∝ exp( − D ( y c,k ; τ c ) β) by using an parametric measure D ( y c,k ; τ c ) with parameters τ c .

Original: Assuming that the measure D ( y c,k ; τ c ) is deﬁned on the support intersection between y c,k and τ c we propose the following formulation:
Revised: Assuming that the measure D ( y c,k ; τ c ) is deﬁned on the support intersection between y c,k and τ c we propose the following formulation:

Original: [Equation]
Revised: [Equation]

Original: (5) where (cid:107) y + c,k (cid:12) τ
Revised: (5) where (cid:107) y + c,k (cid:12) τ

Original: + c (cid:107) 1 + (cid:107) y − c,k (cid:12) τ − c (cid:107) 1 measures the similarity contribution on the support intersection using the positive and the negative components y + c,k , τ + c and y − c,k , τ − c of y c,k
Revised: + c (cid:107) 1 + (cid:107) y − c,k (cid:12) τ − c (cid:107) 1 measures the similarity contribution on the support intersection using the positive and the negative components y + c,k , τ + c and y − c,k , τ − c of y c,k

Original: and τ c , respectively and (cid:107) y c,k (cid:12)
Revised: and τ c , respectively and (cid:107) y c,k (cid:12)

Original: τ c (cid:107) 22 measures the strength of the support intersection between y c,k and τ c .
Revised: τ c (cid:107) 22 measures the strength of the support intersection between y c,k and τ c .

Original: We highlight that the true p ( τ c ) , τ c , are not known and the only prior about D ( y c,k ; τ c ) and p ( τ c ) is that they should be informative enough w.r.t.
Revised: We highlight that the true p ( τ c ) , τ c , are not known and the only prior about D ( y c,k ; τ c ) and p ( τ c ) is that they should be informative enough w.r.t.

Original: the discriminative properties of the transform representation y c,k .
Revised: the discriminative properties of the transform representation y c,k .

Original: Furthermore, instead of estimating them explicitly, an approximation to D ( y c,k ; τ c ) is considered, based only on the concentrations of the similarity on the support intersection and the expected strength of the support intersection for the transform data in the transform domain.
Revised: Furthermore, instead of estimating them explicitly, an approximation to D ( y c,k ; τ c ) is considered, based only on the concentrations of the similarity on the support intersection and the expected strength of the support intersection for the transform data in the transform domain.

Original: The approximation We propose an approximation that captures two expectations.
Revised: The approximation We propose an approximation that captures two expectations.

Original: The ﬁrst one is the expected similarity on the support intersection for the positive and negative component between y c,k and the set of transform representations Y \ c that come from all classes c 1 different from c , i.e. , c (cid:54) =
Revised: The ﬁrst one is the expected similarity on the support intersection for the positive and negative component between y c,k and the set of transform representations Y \ c that come from all classes c 1 different from c , i.e. , c (cid:54) =

Original: c 1 .
Revised: c 1 .

Original: The second is the expected strength on the support intersection between y c,k and the set of transform representations Y \ c that come from all classes c 1 different from c , i.e. , c (cid:54) =
Revised: The second is the expected strength on the support intersection between y c,k and the set of transform representations Y \ c that come from all classes c 1 different from c , i.e. , c (cid:54) =

Original: c 1 .
Revised: c 1 .

Original: The approximation is deﬁned as:
Revised: The approximation is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where the m -th element y + c,k ( m ) of y + c,k is deﬁned as y + c,k ( m ) =
Revised: where the m -th element y + c,k ( m ) of y + c,k is deﬁned as y + c,k ( m ) =

Original: max( y c,k ( m ) , 0) and similarly, y − c,k ( m ) = max( − y c,k ( m ) , 0) , ∀ m ∈ M .
Revised: max( y c,k ( m ) , 0) and similarly, y − c,k ( m ) = max( − y c,k ( m ) , 0) , ∀ m ∈ M .

Original: We also deﬁne the expected similarity using the positive and negative components of y c,k for the set of transform representations Y c that come from the same classes c as D P (cid:96) 1 ,c ( X )
Revised: We also deﬁne the expected similarity using the positive and negative components of y c,k for the set of transform representations Y c that come from the same classes c as D P (cid:96) 1 ,c ( X )

Original: = (cid:80) k 1 (cid:107)
Revised: = (cid:80) k 1 (cid:107)

Original: y + c,k (cid:12) y + c,k 1 (cid:107) 

Revised: y + c,k (cid:12) y + c,k 1 (cid:107)

Original: + (cid:80) k 1 (cid:107) y − c,k (cid:12) y − c,k 1 (cid:107) 1 .
Revised: + (cid:80) k 1 (cid:107) y − c,k (cid:12) y − c,k 1 (cid:107) 1 .

Original: The terms(( C − 1) K )( CK ) (cid:80) c
Revised: The terms(( C − 1) K )( CK ) (cid:80) c

Original: 1 c 1 (cid:54) =
Revised: 1 c 1 (cid:54) =

Original: c (cid:80) k 1 y − c 1 ,k 1 and 2 (( C − 1) K )( CK ) (cid:80) c
Revised: c (cid:80) k 1 y − c 1 ,k 1 and 2 (( C − 1) K )( CK ) (cid:80) c

Original: 1 c 1 (cid:54) =
Revised: 1 c 1 (cid:54) =

Original: c (cid:80) k 1 y + c 1 ,k 1 might also be seen ﬁnite sample estimates of the positive and negative component τ
Revised: c (cid:80) k 1 y + c 1 ,k 1 might also be seen ﬁnite sample estimates of the positive and negative component τ

Original: + c and τ − c , respectively, for the unknown variable τ c 2 .
Revised: + c and τ − c , respectively, for the unknown variable τ c 2 .

Original: The term 2 (( C − 1) K )( CK ) (cid:80) c 2 c 2 (cid:54) =
Revised: The term 2 (( C − 1) K )( CK ) (cid:80) c 2 c 2 (cid:54) =

Original: c 1 (cid:80) k 2 y c 2 ,k 2 (cid:12) y c 2 ,k 2 might also be seen as ﬁnite sample estimates of the Hadamard square τ c (cid:12) τ c for the unknown variable τ c 3 .
Revised: c 1 (cid:80) k 2 y c 2 ,k 2 (cid:12) y c 2 ,k 2 might also be seen as ﬁnite sample estimates of the Hadamard square τ c (cid:12) τ c for the unknown variable τ c 3 .

Original: If the measure D P (cid:96) 1 ( X ) is not used then the approximation (6) is most similar to the one proposed in Liu et al.
Revised: If the measure D P (cid:96) 1 ( X ) is not used then the approximation (6) is most similar to the one proposed in Liu et al.

Original: Note that the Fisher discriminate constraint Yang et al. (2011b), the pairwise constraint Guo et al.
Revised: Note that the Fisher discriminate constraint Yang et al. (2011b), the pairwise constraint Guo et al.

Original: (2013) and the support intersection constraint Liu et al.
Revised: (2013) and the support intersection constraint Liu et al.

Original: (2016) are all approximations of a discriminative prior.
Revised: (2016) are all approximations of a discriminative prior.

Original: However, all of them are with speciﬁc assumptions on the distribution of the data representation in the transform domain.
Revised: However, all of them are with speciﬁc assumptions on the distribution of the data representation in the transform domain.

Original: The advantage of using (6) is that the aproximation is without any prior on the probability distributions p ( τ c ) and without any explicit assumption about the distance metric/measure, or the space/manifold in the transform domain.
Revised: The advantage of using (6) is that the aproximation is without any prior on the probability distributions p ( τ c ) and without any explicit assumption about the distance metric/measure, or the space/manifold in the transform domain.

Original: 2. T HE LEARNING ALGORITHM
Revised: 2. T HE LEARNING ALGORITHM

Original: The summary of the used priors is as follows:
Revised: The summary of the used priors is as follows:

Original: [Equation]
Revised: [Equation]

Original: Additionally, we have a prior on A to regulirize the information loss in order to avoid trivially unwanted matrices A , i.e. , matrices that have repeated or zero rows.
Revised: Additionally, we have a prior on A to regulirize the information loss in order to avoid trivially unwanted matrices A , i.e. , matrices that have repeated or zero rows.

Original: The corresponding prior is deﬁned as:
Revised: The corresponding prior is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where the (cid:107) A (cid:107) F penalty helps regularize the scale ambiguity, the log | det ( A T A ) | and (cid:107) A (cid:107) 2 F are functions of the singular values of A and together help regularize the conditioning of A .
Revised: where the (cid:107) A (cid:107) F penalty helps regularize the scale ambiguity, the log | det ( A T A ) | and (cid:107) A (cid:107) 2 F are functions of the singular values of A and together help regularize the conditioning of A .

Original: Assuming that the expected coherence µ 2 ( A ) between the rows a m of A ( i.e. , A T =
Revised: Assuming that the expected coherence µ 2 ( A ) between the rows a m of A ( i.e. , A T =

Original: [ a 1 , a 2 , ..., a M ] ) is deﬁned as µ
Revised: [ a 1 , a 2 , ..., a M ] ) is deﬁned as µ

Original: [Equation]
Revised: [Equation]

Original: 2 |
Revised: 2 |

Original: [Equation]
Revised: [Equation]

Original: 2 | 2 , ∀ m 1 , m 2 ∈ { 1 , 2 , .., M } .
Revised: 2 | 2 , ∀ m 1 , m 2 ∈ { 1 , 2 , .., M } .

Original: Then (cid:107) AA T − I (cid:107) 2 F measures the expected coherence µ 2 ( A ) and the (cid:96) 2 norm for the rows of A .
Revised: Then (cid:107) AA T − I (cid:107) 2 F measures the expected coherence µ 2 ( A ) and the (cid:96) 2 norm for the rows of A .

Original: Note that the joint probability can be expressed as:
Revised: Note that the joint probability can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: since p ( x c,k | y c,k , τ c , )
Revised: since p ( x c,k | y c,k , τ c , )

Original: =
Revised: =

Original: p ( x c,k | y c,k ) .
Revised: p ( x c,k | y c,k ) .

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: ∼
Revised: ∼

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: since
Revised: since

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Input X , λ 0 , λ 1 , λ 2 , λ 3 , λ
Revised: Input X , λ 0 , λ 1 , λ 2 , λ 3 , λ

Original: A ← inicialize repeat
Revised: A ← inicialize repeat

Original: DISCRIMINATIVE ENCODING closed form solution per data sample
Revised: DISCRIMINATIVE ENCODING closed form solution per data sample

Original: Y ← AX repeatfor ∀ c ∈ C do d − c ← (cid:80) c 1 c 1 (cid:54) =
Revised: Y ← AX repeatfor ∀ c ∈ C do d − c ← (cid:80) c 1 c 1 (cid:54) =

Original: c (cid:80) k 1 y − c 1 ,k 1 , d + c ← (cid:80) c 1 c 1 (cid:54) =
Revised: c (cid:80) k 1 y − c 1 ,k 1 , d + c ← (cid:80) c 1 c 1 (cid:54) =

Original: c (cid:80) k 1 y + c 1 ,k 1 and s c ← (cid:80) c 1 c 1 (cid:54) =
Revised: c (cid:80) k 1 y + c 1 ,k 1 and s c ← (cid:80) c 1 c 1 (cid:54) =

Original: c (cid:80) k 1 y c 1 ,k 1 (cid:12) y c 1 ,kend for for ∀ c ∈ C and ∀ k ∈ K do g ← sign(max( Ax c,k , 0 ))
Revised: c (cid:80) k 1 y c 1 ,k 1 (cid:12) y c 1 ,kend for for ∀ c ∈ C and ∀ k ∈ K do g ← sign(max( Ax c,k , 0 ))

Original: (cid:12) d
Revised: (cid:12) d

Original: + c + sign(max( − Ax c,k , 0 ))
Revised: + c + sign(max( − Ax c,k , 0 ))

Original: (cid:12) d − c y c,k ← sign( AX ) (cid:12) max ( | Ax c,k | − λ 0 g − λ 1 1 , 0 ) (cid:11) ( 1 + 2 λ 0 s c ) y c,k ← y c,k / (cid:107) y c,k (cid:107) 2 end for until convergence
Revised: (cid:12) d − c y c,k ← sign( AX ) (cid:12) max ( | Ax c,k | − λ 0 g − λ 1 1 , 0 ) (cid:11) ( 1 + 2 λ 0 s c ) y c,k ← y c,k / (cid:107) y c,k (cid:107) 2 end for until convergence

Original: TRANSFORM UPDATE (cid:15) -close closed form solution
Revised: TRANSFORM UPDATE (cid:15) -close closed form solution

Original: U
Revised: U

Original: [Equation]
Revised: [Equation]

Original: ←
Revised: ←

Original: [Equation]
Revised: [Equation]

Original: I and
Revised: I and

Original: [Equation]
Revised: [Equation]

Original: where σ
Revised: where σ

Original: [Equation]
Revised: [Equation]

Original: ( n
Revised: ( n

Original: [Equation]
Revised: [Equation]

Original: until convergence
Revised: until convergence

Original: Output A , Y
Revised: Output [Equation] Y

Original: Given the avlable tranining data set X ∈ (cid:60) N × CK maximizing p ( x c,k , y c,k , τ c , A ) over Y and A is same as minimizing the following problem:
Revised: Given the avlable tranining data set X ∈ (cid:60) N × CK maximizing p ( x c,k , y c,k , τ c , A ) over Y and A is same as minimizing the following problem:

Original: [Equation]
Revised: [Equation]

Original: where { λ 0 , λ 1 , λ 3 , λ 4 } are inversely proportional to the scaling parameters { β 1 , β 2 , β 3 , β 4 , β 5 } .
Revised: where { λ 0 , λ 1 , λ 3 , λ 4 } are inversely proportional to the scaling parameters { β 1 , β 2 , β 3 , β 4 , β 5 } .

Original: Note that the solution to (10) is not equivalent to the maximum a priory (MAP) solution, which is difﬁcult to compute, as it involves integrating over the vectors y c,k .
Revised: Note that the solution to (10) is not equivalent to the maximum a priory (MAP) solution, which is difﬁcult to compute, as it involves integrating over the vectors y c,k .

Original: In therms of optimization the problem is not convex in the variables ( Y , A ) .
Revised: In therms of optimization the problem is not convex in the variables ( Y , A ) .

Original: The solution is obtained by iteratively, marginally maximizing the probability p ( x c,k , y c,k , τ c , A ) over Y and A which is equivalent to maximizing the conditional densities p ( y c,k | x c,k , τ c , A ) and p ( A | y c,k , x c,k ) , respectively.
Revised: The solution is obtained by iteratively, marginally maximizing the probability p ( x c,k , y c,k , τ c , A ) over Y and A which is equivalent to maximizing the conditional densities p ( y c,k | x c,k , τ c , A ) and p ( A | y c,k , x c,k ) , respectively.

Original: Meaning that at one iterating step one of the variables Y or A is ﬁxed and w.r.t.
Revised: Meaning that at one iterating step one of the variables Y or A is ﬁxed and w.r.t.

Original: the other the problem (10) is minimized.
Revised: the other the problem (10) is minimized.

Original: The following text describes the iterating steps that consist of linear map estimation (maximizing p ( A | y c,k , x c,k ) ) and discriminative encoding (maximizing p ( y c,k | x c,k , τ c , A ) ).
Revised: The following text describes the iterating steps that consist of linear map estimation (maximizing p ( A | y c,k , x c,k ) ) and discriminative encoding (maximizing p ( y c,k | x c,k , τ c , A ) ).

Original: Linear map estimation :
Revised: Linear map estimation :

Original: Given the available data samples X and the corresponding transform representations Y the linear map A estimation problem reduces to:
Revised: Given the available data samples X and the corresponding transform representations Y the linear map A estimation problem reduces to:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: and we use the (cid:15) -close closed form solution estimated as follows: Proposition 1 ( (cid:15) -close solution ): Given Y ∈ (cid:60) M × L , ∀ X ∈ (cid:60) N × L and M ≥ N , ∀ λ 2 ≥ 0 ,λ 3 ≥ 0 , λ 4 ≥ 0 , let the eigen value decomposition U X Σ 2 X U TX of XX T +

Original:  andwe use the (cid:15) -close closed form solution as proposed in Kostadinov et al.
Revised: λ 2 I and the singular value decomposition U

Original:  ( we give the proof in Appendix A ).
Revised: U X XY Σ U X XY V TU X XY of U TX XY T exist, then if and only if σ X ( n ) > 0 , ∀ n ∈ N = { 1 , 2 , 3 , ..., N } , (11) has (cid:15) -close approximative solution as : [Equation] where Σ A is diagonal matrix, Σ A ( n, n )

Revised: =

Revised: σ A ( n ) ≥ 0 , and σ A ( n ) are solutions to

Revised: [Equation]

Revised: with σ Γ ( n ) =

Revised: T ( n, n ) , T = U

Revised: U X

Revised: XY Σ

Revised: U X XY U TU X XY , ∀ n ∈ N ( we give the proof in Appendix A ).

Original: Discriminative encoding :
Revised: Discriminative encoding :

Original: Given the available data samples X and the current estimate of the transform A , deﬁne for simplicity Q = AX ∈ (cid:60) M × L .
Revised: Given the available data samples X and the current estimate of the transform A , deﬁne for simplicity Q = AX ∈ (cid:60) M × L .

Original: The discriminative representation estimation problem is formulated as:
Revised: The discriminative representation estimation problem is formulated as:

Original: [Equation]
Revised: [Equation]

Original: Note that (10) and consequently (12) is a well deﬁned problem only if the model variables τ c , or the probability distributions p ( τ c ) are know in advance.
Revised: Note that (10) and consequently (14) is a well deﬁned problem only if the model variables τ c , or the probability distributions p ( τ c ) are know in advance.

Original: Nevertheless, we show that the approximation (6) leads to an efﬁcient solution of (12).
Revised: Nevertheless, we show that the approximation (6) leads to an efﬁcient solution of (14).

Original: Assuming that Y \ c is given, then for any sample k ∈ K from any class c ∈ C by using the approximation (6), problem (12) reduces to a constrained projection :
Revised: Assuming that Y \ c is given, then for any sample k ∈ K from any class c ∈ C by using the approximation (6), problem (14) reduces to a constrained projection :

Original: [Equation]
Revised: [Equation]

Original: ) and has a closed form solution as:
Revised: and has a closed form solution as:

Original: [Equation]
Revised: [Equation]

Original:  where (cid:11) denotes Hadamard (element-wise) division, g = sign(max( Ax c,k , 0 ))
Revised: ,k − λ 0 g + λ 1 1 , 0 )) (cid:11) ( 1 + 2 λ 0 s c ) , (16) where (cid:11) denotes Hadamard (element-wise) division, g = sign(max( Ax c,k , 0 ))

Original: (cid:12) d c + + sign(max
Revised: (cid:12) d c + + sign(max

Original: [Equation]
Revised: − Ax c,k , 0 ))

Original: [Equation]
Revised: (cid:12) d c − , d − c = (cid:80) c 1 c 1 (cid:54) = c (cid:80) k 1 y − c 1 ,k 1 and d + c = (cid:80) c

Original: [Equation]
Revised: 1 c 1 (cid:54) =

Original: and
Revised: c (cid:80) k 1 y + c 1 ,k 1 , s c

Original: [Equation] (cid:12) [Equation]
Revised: = (cid:80) c 1 c 1 (cid:54) =

Original:  ( the proof is given in Appendix B ).
Revised: c (cid:80) k 1 y c 1 ,k 1 (cid:12) y c 1 ,k 1 ( the proof is given in Appendix B ).

Original: We note that at convergence (which we do not prove here) we can only claim that a joint local maximum in ( Y , A ) of p ( x c,k , y c,k , τ c , A ) has been reached, even if, as in this case, each optimization step achieves the (marginal) (cid:15) -close and global optimal solution, respectively.
Revised: We note that at convergence (which we do not prove here) we can only claim that a joint local maximum in ( Y , A ) of p ( x c,k , y c,k , τ c , A ) has been reached, even if, as in this case, each optimization step achieves the (marginal) (cid:15) -close and global optimal solution, respectively.

Original: The exact steps of the proposed non-linear transform learning are described by Algorithm 1.
Revised: The exact steps of the proposed non-linear transform learning are described by Algorithm 1.

Original:  S ENSITIVITY ANALYSIS AND INTERPRETATIONS
Revised:  S ENSITIVITY ANALYSIS AND INTERPRETATIONS

Original: The similarity concentration measure provides possibility to measure the discriminative properties, their deviation, increase (or decrease) and the corresponding relations between different non-linear transform models across one domain or different domains, thereby quantifying their quality w.r.t.
Revised: The similarity concentration measure provides possibility to measure the discriminative properties, their deviation, increase (or decrease) and the corresponding relations between different non-linear transform models across one domain or different domains, thereby quantifying their quality w.r.t.

Original: the discriminative properties.
Revised: the discriminative properties.

Original: 3. S ENSITIVITY ANALYSIS W . R . T . THE SIMILARITY CONCENTRATION MEASURES
Revised: 3. S ENSITIVITY ANALYSIS W . R . T . THE SIMILARITY CONCENTRATION MEASURES

Original: To measure the ability for an increase in discriminative properties by a non-linear transform 4 we ﬁrst have to deﬁne a notion for the discriminative properties on a data set under different non-linear transform models.
Revised: To measure the ability for an increase in discriminative properties by a non-linear transform 4 we ﬁrst have to deﬁne a notion for the discriminative properties on a data set under different non-linear transform models.

Original: Therefore, ﬁrst we introduce the ”special” base models and then analyze the properties of the similarity concentration measures under the change in model parameter and the relation between the base model and the proposed non-linear transform model (3).
Revised: Therefore, ﬁrst we introduce the ”special” base models and then analyze the properties of the similarity concentration measures under the change in model parameter and the relation between the base model and the proposed non-linear transform model (3).

Original: Any data set X in the original domain might have a transform model with parameters B N = { A o ∈ (cid:60) N × N , τ = 0 ∈ (cid:60) N + } , if A o = I ∈
Revised: Any data set X in the original domain might have a transform model with parameters B N = { A o ∈ (cid:60) N × N , τ = 0 ∈ (cid:60) N + } , if A o = I ∈

Original: D N + we refer to it as a base original model .
Revised: D N + we refer to it as a base original model .

Original: Similarly as in the original domain, any data set Y in the transform domain might have a transform model with parameters B M =
Revised: Similarly as in the original domain, any data set Y in the transform domain might have a transform model with parameters B M =

Original: { A t ∈ (cid:60) M × M , τ = 0 ∈ (cid:60) M + } , if A t = I ∈
Revised: { A t ∈ (cid:60) M × M , τ = 0 ∈ (cid:60) M + } , if A t = I ∈

Original: D M
Revised: D M

Original: + we refer to it as a base transform model .
Revised: + we refer to it as a base transform model .

Original: Any base model, deﬁned ether in the original domain B N or in the transform domain B M , has domain equal to the co-domain, since x c,k = T B N ( x c,k ) and y c,k = T B M ( y c,k ) holds trivially, for the respective sets of parameters B N =
Revised: Any base model, deﬁned ether in the original domain B N or in the transform domain B M , has domain equal to the co-domain, since x c,k = T B N ( x c,k ) and y c,k = T B M ( y c,k ) holds trivially, for the respective sets of parameters B N =

Original: { A o = I ∈
Revised: { A o = I ∈

Original: D N + , τ = 0 ∈ (cid:60) N + } and
Revised: D N + , τ = 0 ∈ (cid:60) N + } and

Original: B M =
Revised: B M =

Original: { A t = I ∈
Revised: { A t = I ∈

Original: D M + , τ = 0 ∈ (cid:60) M + } .
Revised: D M + , τ = 0 ∈ (cid:60) M + } .

Original: This is illustrated with a diagram shown in Figure 3. 
Revised: This is illustrated with a diagram shown in Figure 3. [Figure]

Original: A base original model provides a possibility to compare it with any other non-linear transform model with parameters P =
Revised: A base original model provides a possibility to compare it with any other non-linear transform model with parameters P =

Original: { A ∈ (cid:60) M × N , τ ∈ (cid:60) M + } .
Revised: { A ∈ (cid:60) M × N , τ ∈ (cid:60) M + } .

Original: Additionally, note that for B M = { A t = I ∈
Revised: Additionally, note that for B M = { A t = I ∈

Original: D M , τ = 0 ∈ (cid:60) M } we have that D B M (cid:96) 1 ,c ( Y ) =
Revised: D M , τ = 0 ∈ (cid:60) M } we have that D B M (cid:96) 1 ,c ( Y ) =

Original: D P (cid:96) 1 ,c ( X ) and that D B M (cid:96) 1 ( Y ) =
Revised: D P (cid:96) 1 ,c ( X ) and that D B M (cid:96) 1 ( Y ) =

Original: D P (cid:96) 1 ( X ) .
Revised: D P (cid:96) 1 ( X ) .

Original: It implies that the similarity concentrations can be analyzed as a function in the original domain under model
Revised: It implies that the similarity concentrations can be analyzed as a function in the original domain under model

Original: P or in the transform domain under model B M .
Revised: P or in the transform domain under model B M .

Original: The main relations considering the preservation of change in the similarity concentration between two models, deﬁned not necessary in the same domain are stated by Proposition 1.
Revised: The main relations considering the preservation of change in the similarity concentration between two models, deﬁned not necessary in the same domain are stated by Proposition 1.

Original: Lemma 1 : The non-linear transform model (4) totally preserves the information in the change for the similarity concentration for a data set X w.r.t.
Revised: Lemma 1 : The non-linear transform model (4) totally preserves the information in the change for the similarity concentration for a data set X w.r.t.

Original: a small change in the parameters of the models
Revised: a small change in the parameters of the models

Original: [Figure] [Equation] [Figure] B N , B M and P if (cid:107) δ o (cid:107) ∗ = 0 and (cid:107) δ t (cid:107) ∗ = 0 as( R
Revised: B N , B M and P if (cid:107) δ o (cid:107) ∗ = 0 and (cid:107) δ t (cid:107) ∗ = 0 as

Original: [Equation]
Revised: [Equation]

Original: A

Original: [Equation]

Original: ( R

Original: [Equation]

Original: [Equation]

Original: (15) where δ o

Original: =

Original: (cid:80) c,c 1 (cid:80) k,k 1 k (cid:54) =

Original: [-k 
-]
Revised: [-k 
-]R

Original: z c 1 ,k 1 x Tc

Original: [Equation]
Revised: [Equation]

Original: k + z c

Original: [Equation]

Original: k x Tc

Original: [Equation]
Revised: [Equation]

Original: k
Revised: A

Original: [Equation]
Revised: [Equation]

Original: ∂D

Original: P

Original: [Equation]

Original: [Equation]

Original: c ( X ) [Equation] A + ∂D P (cid:96) 1 ( X ) ∂ A = (cid:80) c,c 1 (cid:80) k,k 1 k 
Revised: = I = (cid:80) c,c 1 (cid:80) k,k 1 k (cid:54) =

Original: (cid:54) =

Original: ky c 1 ,k 1 x Tc,k + y c,k x Tc 1 ,k 1 , δ t = (cid:80) c,c 1 (cid:80) k,k 1 k (cid:54) =

Original: k 

Revised: k

Original: z c 1 ,k 1y Tc,k + z  c,k[-y Tc 1 ,k 1 , ∂D B M (cid:96) 1 ,c ( Y ) ∂ A t | A t = I + ∂D B M (cid:96) 
-]
Revised: z c 1 ,k 1 z T c,k[-y Tc 1 ,k 1 , ∂D B M (cid:96) 1 ,c ( Y ) ∂ A t | A t = I + ∂D B M (cid:96) 
-]

Original: ( Y ) ∂

Original: A t | A t =

Original: I = (cid:80) c,c 1 (cid:80) k,k 1 k (cid:54) =

Original: k 1 y c 1 ,k 1 y Tc,k + y c,k y Tc 1 ,k 1 , ∂D

Original: B M (cid:96) 1 ,c ( Z ) ∂

Original: A t | A t

Original: [-= I + ∂D B M (cid:96) 
-]
Revised: [-= I + ∂D B M (cid:96) 
-]

Original: ( Z ) ∂

Original: A t | A t =

Original: I = (cid:80) c,c 1 (cid:80) k,k 1 k (cid:54) =

Original: [-k 
 z c 1 ,k 1 z Tc,k-] + z c,k z Tc 1 ,k 1 , ∂D
Revised: [-k 
 z c 1 ,k 1 z Tc,k-] + z c,k z T c 1 ,k 1 , ∂D

Original: B N (cid:96) 1 ,c ( X ) ∂
Revised: B N (cid:96) 1 ,c ( X ) ∂

Original: A o |
Revised: A o |

Original: A o = I =
Revised: A o = I =

Original: (cid:80) c (cid:80) k,k 1 x c,k x Tc,k 1 + x c,k 1 x Tc,k and ∂D B N (cid:96) 1 ( X ) ∂
Revised: (cid:80) c (cid:80) k,k 1 x c,k x T c,k 1 + x c,k 1 x T c,k and ∂D B N (cid:96) 1 ( X ) ∂

Original: A o |
Revised: A o |

Original: A o = I =
Revised: A o = I =

Original: (cid:80) c,c 1 ,c (cid:54) =
Revised: (cid:80) c,c 1 ,c (cid:54) =

Original: c 1 (cid:80) k,k 1 x c,k x Tc 1 ,k 1 + x c 1 ,k 1 x Tc,k .
Revised: c 1 (cid:80) k,k 1 x c,k x Tc 1 ,k 1 + x c 1 ,k 1 x Tc,k .

Original: The proof is given in Appendix C .
Revised: The proof is given in Appendix C .

Original: The terms z c,k represent the non-linear transform error vectors that appear in the model
Revised: The terms z c,k represent the non-linear transform error vectors that appear in the model

Original: Ax c,k = y c,k + z c,k as a result of applying an element-wise non-liner operation H τ to Ax c,k , i.e. , y c,k =
Revised: Ax c,k = y c,k + z c,k as a result of applying an element-wise non-liner operation H τ to Ax c,k , i.e. , y c,k =

Original: H τ ( Ax c,k ) .
Revised: H τ ( Ax c,k ) .

Original: As an example in the sparsiﬁng transform model z c,k is the ”loss of information”, that is the information about the values of the elements in Ax c,k that are discarded.
Revised: As an example in the sparsiﬁng transform model z c,k is the ”loss of information”, that is the information about the values of the elements in Ax c,k that are discarded.

Original: The terms δ o and δ t correlate the errors z c,k with the original data x c 1 ,k 1 and transform data y c 1 ,k 1 , respectively.
Revised: The terms δ o and δ t correlate the errors z c,k with the original data x c 1 ,k 1 and transform data y c 1 ,k 1 , respectively.

Original: Note that if there is no loss of information (in the earlier example it means that there is no thresholding and just a simple linear transform model is used) then δ o = 

Revised: Note that if there is no loss of information (in the earlier example it means that there is no thresholding and just a simple linear transform model is used) then δ o =

Original: and δ t = 0 .
Revised: and δ t = 0 .

Original: Moreover, δ o and δ t bear important information about the discriminative properties in the transform domain.
Revised: Moreover, δ o and δ t bear important information about the discriminative properties in the transform domain.

Original: The terms ∂D P (cid:96) 1 ,c ( X )
Revised: The terms ∂D [Equation]

Original: ∂ A and ∂D P (cid:96) 1 ( X ) ∂ A represent the change of the similarity concentrations under inﬁnitesimally small change of the parameter A from the model P .
Revised: and ∂D P (cid:96) 1 ( X ) ∂ A represent the change of the similarity concentrations under inﬁnitesimally small change of the parameter A from the model P .

Original: The terms ∂D B N (cid:96) 1 ,c ( X ) ∂
Revised: The terms ∂D B N (cid:96) [Equation] ,c ( X ) ∂

Original: A o |
Revised: A o |

Original: A
Revised: A

Original: o =
Revised: o =

Original: I and ∂D B N (cid:96) 1 ( X ) ∂A o |
Revised: I and [Table] ∂

Original: A
Revised: A

Revised: [Table]

Original:  o =
Revised: | A o =

Original: I have dual interpretation.
Revised: I have dual interpretation.

Original: Assuming metric
Revised: Assuming metric

Original: A o
Revised: A o

Original: = I , then the ﬁrst one is considered as a change of the similarity concentrations under inﬁnitesimally small change of the space metric, or equivalently under small metric perturbation.
Revised: = I , then the ﬁrst one is considered as a change of the similarity concentrations under inﬁnitesimally small change of the space metric, or equivalently under small metric perturbation.

Original: Conversely, assuming the data samples are distributed under a Gaussian distribution with parameters identity covariance matrix and zero mean, i.e. x c,k ∼ N ( µ = 0 , Σ = I ) , then ∂D B N (cid:96) 1 ,c ( X )
Revised: Conversely, assuming the data samples are distributed under a Gaussian distribution with parameters identity covariance matrix and zero mean, i.e. x c,k ∼ N ( µ = 0 , Σ = I ) , then ∂D B N (cid:96) 1 ,c ( X )

Original: ∂
Revised: ∂

Original: A o |
Revised: A o |

Original: A
Revised: A

Original: o =
Revised: o =

Original: I and ∂D B N (cid:96) 1 ( X ) ∂
Revised: I and ∂D B N (cid:96) 1 ( X ) ∂

Original: A o |
Revised: A o |

Original: A
Revised: A

Original: o =
Revised: o =

Original: I represent the change of the similarity concentrations under small change in the assumption away from a Gaussian distribution.
Revised: I represent the change of the similarity concentrations under small change in the assumption away from a Gaussian distribution.

Original: Equation ( R 1) relates the base transform for the original domain B N with any arbitrary transform deﬁned in the original domain P .
Revised: Equation ( R 1) relates the base transform for the original domain B N with any arbitrary transform deﬁned in the original domain P .

Original: The relation ( R 2) is a result about the preservation of change in the similarity concentration between two models B M and P deﬁned on two different domains.
Revised: The relation ( R 2) is a result about the preservation of change in the similarity concentration between two models B M and P deﬁned on two different domains.

Original: Whereas ( R 3) gives the preservation of change in the similarity concentration between the error in the transform domain.
Revised: Whereas ( R 3) gives the preservation of change in the similarity concentration between the error in the transform domain.

Original: [Figure]

Original: The next result highlights the relation between: the linear projection ( by the linear map A that appears in the model P ) of the change in the similarity concentration under the model B N in the original domain and the change of the similarity concentration under the model B M in the transform domain .
Revised: The next result highlights the relation between: the linear projection ( by the linear map A that appears in the model P ) of the change in the similarity concentration under the model B N in the original domain and the change of the similarity concentration under the model B M in the transform domain .

Original: This relation exists independently for
Revised: This relation exists independently for

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]
Revised: [Equation]

Original: I and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: | A o
Revised: | A o

Original: =
Revised: =

Original: I , nevertheless, we will deﬁne the summarized and the independent versions.
Revised: I , nevertheless, we will deﬁne the summarized and the independent versions.

Original: Therefore, ﬁrst we deﬁne ∂ J (cid:96) 1 ( X) ∂
Revised: Therefore, ﬁrst we deﬁne ∂ J [Equation] ( X

Original:  Ao | A o = I =
Revised: [Equation] ∂ A

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: |

Original: [Equation]
Revised: [Equation]

Original: |
Revised: = I = [Equation] [Equation] A

Original: [Equation]
Revised: [Equation]

Revised: |

Revised: [Equation]

Revised: I and

Revised: [Equation]

Revised: (cid:96)

Revised: [Equation]

Revised: | A

Revised: [Equation]

Original: and rewrite ( R 1) as A
Revised: and rewrite ( R 1) as A

Original: [Equation]
Revised: [Equation]

Original: J ( X 
Revised: X [Equation]

Original: [Equation]
Revised: [Equation]

Original:  , then replace ∂D 
Revised: A , then replace ∂D P

Original: [Equation]
Revised: [Equation]

Original:  in ( R 2) by the same term in ( R 1) , use ( R 3) , reorder and we have the following result.
Revised: + ∂D [Equation] in ( R 2) by the same term in ( R 1) , use ( R 3) , reorder and we have the following result.

Original: Lemma 2 : For ﬁxed τ any non-linear transform model (4) preservers the information in the change of similarity concentrations w.r.t. a small change in A by
Revised: Lemma 2 : For ﬁxed τ any non-linear transform model (4) preservers the information in the change of similarity concentrations w

Original: [Equation]
Revised: [Equation]

Original: o
Revised: r

Original: [Equation]
Revised: [Equation]

Revised: t. a small change in A by

Original:  where V = AX, ξ c + ξ =
Revised: [Equation] where V = AX

Original: [Equation]
Revised: [Equation]

Original: and
Revised: ξ

Original: [Equation]
Revised: [Equation]

Original: J
Revised: + [Equation] =

Original: [Equation]
Revised: [Equation]

Original:  | 
Revised: [Table] B N (cid:96) 1 ( Z ) ∂ A t | A t =

Original: {+I and ∂ J (cid:96) 
+}
Revised: {+I and ∂ J (cid:96) 
+}

Original: [Equation]
Revised: ( Y ; Z ) ∂ A t | A t = I = δ t + δ Tt .

Original: Note that ∂ n J (cid:96) 1 ( X ) ∂ n A o | A o = I = 4 ∂ n − 1 J (cid:96) 1 ( X ) ∂ n − 1 A o |
Revised: Note that ∂ n J (cid:96) 1 ( X ) ∂ n A o | A o = I = 4 ∂ n − 1 J (cid:96) 1 ( X ) ∂ n − 1 A o |

Original: A
Revised: A

Original: o = I and ∂ n J (cid:96) 

Revised: o = I and ∂ n J (cid:96)

Original: ( Y ) ∂
Revised: ( Y ) ∂

Original: n A t | A t
Revised: n A t | A t

Original: = I = 4 ∂ n − 1 J (cid:96) 1 ( Y ) ∂ n − 1 A t | A t
Revised: = I = 4 ∂ n − 1 J (cid:96) 1 ( Y ) ∂ n − 1 A t | A t

Original: = I therefore, ∂ J (cid:96) 1 ( X ) ∂
Revised: = I therefore, ∂ J (cid:96) 1 ( X ) ∂

Original: A o |
Revised: A o |

Original: A o = I = 

Revised: A o = I =

Original: ∂ 2 J (cid:96) 1 ( X ) ∂ 2 A o | A
Revised: ∂ 2 J (cid:96) 1 ( X ) ∂ 2 A o | A

Original: o = I and ∂ J (cid:96) 1 ( Y )
Revised: o = I and ∂ J (cid:96) 1 ( Y )

Original: ∂
Revised: ∂

Original: A t | A t = I = 14 ∂ 2 J (cid:96) 1 ( Y ) ∂
Revised: A t | A t = I = 14 ∂ 2 J (cid:96) 1 ( Y ) ∂

Original: 2 A t | A t
Revised: 2 A t | A t

Original: = I might be interpreted as
Revised: = I might be interpreted as

Original: Fisher information matrices evaluated at A o = I ∈
Revised: Fisher information matrices evaluated at A o = I ∈

Original: D N and A t
Revised: D N and A t

Original: = I ∈ D M .
Revised: = I ∈ D M .

Original: The expressions by ( R 4) actually relate the metric in the original domain under the model B N to the induced metric in the transform domain for the model B M , with induction done by the model P with parameter set { A ∈ (cid:60) M × N , τ ∈ (cid:60) M + } .
Revised: The expressions by ( R 4) actually relate the metric in the original domain under the model B N to the induced metric in the transform domain for the model B M , with induction done by the model P with parameter set { A ∈ (cid:60) M × N , τ ∈ (cid:60) M + } .

Original: Moreover, the model P might describe a transform domain with a nonsmooth manifold.
Revised: Moreover, the model P might describe a transform domain with a nonsmooth manifold.

Original: Since the manifolds of the original and the transform domain under the models
Revised: Since the manifolds of the original and the transform domain under the models

Original: B N and B M are smooth the analysis of their relations reveals insights about the relation between the manifolds under the models B N and P .
Revised: B N and B M are smooth the analysis of their relations reveals insights about the relation between the manifolds under the models B N and P .

Original: The terms ∂ J (cid:96) 

Revised: The terms ∂ J (cid:96)

Original: ( Z ) ∂
Revised: ( Z ) ∂

Original: A t | A t
Revised: A t | A t

Original: =
Revised: =

Original: I and ∂ J (cid:96) 

Revised: I and ∂ J (cid:96)

Original: ( Y ; Z )
Revised: ( Y ; Z )

Original: ∂
Revised: ∂

Original: A t | A t =
Revised: A t | A t =

Original: I carry out the information about the breaks and the discontinuities of the regularity and smoothness in the manifold induced by the model P .
Revised: I carry out the information about the breaks and the discontinuities of the regularity and smoothness in the manifold induced by the model P .

Original: Furthermore, if (cid:107) ∂
Revised: Furthermore, if (cid:107) ∂

Original: J (cid:96) 1 ( Z ) ∂
Revised: J (cid:96) 1 ( Z ) ∂

Original: A t | A t = I (cid:107) ∗ = 0 and (cid:107) ∂
Revised: A t | A t = I (cid:107) ∗ = 0 and (cid:107) ∂

Original: J (cid:96) 

Revised: J (cid:96)

Original: ( Y ; Z )
Revised: ( Y ; Z )

Original: ∂
Revised: ∂

Original: A t | A t = I (cid:107) ∗ = 0 , then ( R 4) in information geometry Amari (2013) is seen as change of coordinates on a manifold, where the intrinsic properties of curvature remain unchanged under different parametrization.
Revised: A t | A t = I (cid:107) ∗ = 0 , then ( R 4) in information geometry Amari (2013) is seen as change of coordinates on a manifold, where the intrinsic properties of curvature remain unchanged under different parametrization.

Original: 3. A MEASURE FOR THE DISCRIMINATIVE PROPERTIES AND ITS BOUNDS
Revised: 3. A MEASURE FOR THE DISCRIMINATIVE PROPERTIES AND ITS BOUNDS

Original: This paper proposes a notion for the discriminative properties of a data set under a non-linear transform named as discrimination power, based on a measure for the relations between the concentrations D P (cid:96) 1 ,c ( X ) and D P (cid:96) 1 ( X ) . 
Revised: This paper proposes a notion for the discriminative properties of a data set under a non-linear transform named as discrimination power, based on a measure for the relations between the concentrations D P (cid:96) 1 ,c ( X ) and D P (cid:96) 1 ( X ) . [Figure] log [Figure]

Revised: (cid:96)

Original: [Figure]
Revised: [Figure]

Revised: X

Revised: [Figure]

Revised: (cid:96)

Revised: [Figure]

Revised: ( X )

Revised: [Figure]

Revised: (cid:15)

Revised: [Figure]

Revised: log

Revised: D

Revised: [Figure]

Revised: (cid:96)

Revised: [Figure]

Revised: [Figure]

Revised: ( Y )

Revised: [Figure]

Revised: ( Y

Revised: [Figure]

Revised: (cid:15)

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Revised: min

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: I

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: I

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: I

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: I NT 1 .1 .

Revised: [Table]

Original:  Proposition 1 : The discrimination power for any dataset X ∈ (cid:60) M × CK under any transform with parameter set P is deﬁned as:
Revised: [Table] [Table] [Table] Proposition 1 : The discrimination power for any dataset X ∈ (cid:60) M × CK under any transform with parameter set P is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Remark 1 : The advantage of this measure is that it logarithmically signiﬁes the difference between
Revised: Remark 1 : The advantage of this measure is that it logarithmically signiﬁes the difference between

Original: D P (cid:96) 1 ,c ( X ) and D P (cid:96) 1 ( X ) 5 .
Revised: D P (cid:96) 1 ,c ( X ) and D P (cid:96) 1 ( X ) 5 .

Original: The deﬁnition about the discrimination power of the data set X , but, now under a model with a parameter set B N is equivalent to the one deﬁned for I t , we denote it as I o .
Revised: The deﬁnition about the discrimination power of the data set X , but, now under a model with a parameter set B N is equivalent to the one deﬁned for I t , we denote it as I o .

Original: An illustration is given by a diagram shown in Figure 4.
Revised: An illustration is given by a diagram shown in Figure 4.

Original: The bound on the discrimination power is given by the following result.
Revised: The bound on the discrimination power is given by the following result.

Original: Theorem 1 : The discrimination power for any data set X ∈ (cid:60) N × CK under any transform with parameter set P is bounded as:
Revised: Theorem 1 : The discrimination power for any data set X ∈ (cid:60) N × CK under any transform with parameter set P is bounded as:

Original: [Equation]
Revised: [Equation]

Original: The proof is given in Appendix D.
Revised: The proof is given in Appendix D.

Original: At ﬁrst the resulting bounds might look counterintuitive since the loss of information seems to increase the discrimination power.
Revised: At ﬁrst the resulting bounds might look counterintuitive since the loss of information seems to increase the discrimination power.

Original: This fact is true, however, up to a certain limit.
Revised: This fact is true, however, up to a certain limit.

Original: Therefore, it is important to distinguish two main conclusions.
Revised: Therefore, it is important to distinguish two main conclusions.

Original:  a ﬁnite sample approximation to a discrimination density , since it approximates the density log( p ( τ c | y c,k ) p ( υ c | y c,k ) ) , i.e. ,  D P (cid:96) 1,c ( X ) − D P (cid:96) [-
 ( X ) ∼ − log( p ( τ c , y c,k ) p-] ( υ c , y c,k ) ) = (cid:107)
Revised: First, for any model with a set of parameters P for which there is no loss of information (cid:107) ξ c + ξ (cid:107) ∗ = 0 , the only condition for the increase in the discrimination power is D B N (cid:96) 1 ( X ) ≥ D B M (cid:96) [-
 ( X ) ∼ − log( p ( τ c , y c,k ) p-]1 ( AX ) and

Original: [-τ c + (cid:12) y + c,k (cid:107) 
-]
Revised: [-τ c + (cid:12) y + c,k (cid:107) 
-]

Original: + (cid:107) τ c − (cid:12) y − c,k (cid:107) 1 − ( [-(cid:107) υ c + (cid:12) y + c,k (cid:107) 
 + (cid:107) υ c − (cid:12) y − c,k (cid:107) 1-] ) .
Revised: D B N (cid:96) 1 ,c ( [-(cid:107) υ c + (cid:12) y + c,k (cid:107) 
 + (cid:107) υ c − (cid:12) y − c,k (cid:107) 1-]X ) ≤ D B M (cid:96)

Original: [Figure]
Revised: [Figure]

Original: [Table]

Original: [ min ]

Original: D

Original: [Table]

Original: D

Original: [Table]

Original: D

Original: [Table] D [Table] [Table]
Revised: [Figure]

Original: D [Table] [Table] D {+B M (cid:96) 
+}
Revised: D B M (cid:96) 1 ,c ( Y ) and C 2 = D {+B M (cid:96) 
+}

Original: [Table] D [Table] I O 0 .0 .0 .0 .0 .0 .0 .
Revised: ( Y ) and the discrimination power log( C 1 /C 2) =

Original: IRT 0 .0 .0 .0 .0 .0 .0 . I ST ∗ 0 .1 .0 .0 .1 .1 .1 . I NT 1 .1 .1 .1 .2 .3 .1 . ) for the learned transform A , The execution time te [ min ] in minutes of the proposed algorithm for 28 iterations at the  transform domain dimensionality M = 19000  .
Revised: I t for randomly chosen subsets from all of the used databases under a non-linear transform with transform dimension M = 19000 and varying thresholding parameter τ = λ 1 .

Original: First, for any model with a set of parameters P for which there is no loss of information (cid:107) ξ c + ξ (cid:107) ∗ = 0 , the only condition for the increase in the discrimination power is D B N (cid:96) 1 ( X ) ≥ D B M (cid:96) 1 ( AX ) and

Original: D B N (cid:96) 1 ,c ( X ) ≤ D B M (cid:96) 1 ,c ( AX ) . Second, in the rest of the cases for which D B N (cid:96) 1 ( X ) ≥ D B M (cid:96) 1 ( Y ) and D B N (cid:96) 1 ,c ( X ) ≤ D B M (cid:96) 1 ,c ( Y ) holds true it will be possible to increase the discrimination power.
Revised: Second, in the rest of the cases for which D B N (cid:96) 1 ( X ) ≥ D B M (cid:96) 1 ( Y ) and D B N (cid:96) 1 ,c ( X ) ≤ D B M (cid:96) 1 ,c ( Y ) holds true it will be possible to increase the discrimination power.

Original: Moreover, there is a trade-off between the increase in discrimination power as a result of loss of information (cid:107) ξ c + ξ (cid:107) ∗ > 0 .
Revised: Moreover, there is a trade-off between the increase in discrimination power as a result of loss of information (cid:107) ξ c + ξ (cid:107) ∗ > 0 .

Original:  N UMERICAL EXPERIMENTS
Revised:  N UMERICAL EXPERIMENTS

Original: The numerical experiments are summarized in two different parts.
Revised: The numerical experiments are summarized in two different parts.

Original: In the ﬁrst series of the experiments the properties of the learned map A for the proposed algorithm are investigated.
Revised: In the ﬁrst series of the experiments the properties of the learned map A for the proposed algorithm are investigated.

Original: We evaluate the computational efﬁciency, as run time t e [ min ] , the conditioning number C n ( A ) 6 , the expected mutual coherence µ ( A ) and the discrimination power across several databases for a learned nonlinear transforms having different dimensionality.
Revised: We evaluate the computational efﬁciency, as run time t e [ min ] , the conditioning number C n ( A ) 6 , the expected [Figure] [Figure] B M (cid:96) 1 ,c ( Y ) and C=

Revised: D B M (cid:96) 1 ( Y ) , their ratio C 1 /C 2 and the discrimination power log( C 1 /C 2) =

Original:  Additionally, a comparison between the resulting discrimination power in the original domain, after transform by a random matrix (having Gaussian random samples as entries and transform dimension of M = 19000 ) and after a learned non-linear transform having transform dimension M = 19000 without and with support dissimilarity prior, denoted as I 0 , I RT , I ST ∗
Revised: I t on a subset of the transform data using learned non-linear transform at different dimensionality M ∈ { 100 , 1150 , 2200 , 3250 , 4300 , 5350 , 6400 , 7450 , 8500 , 9550 , 10600 , 11650 , 12700 , 13750 , 14800 , 15850 , 16900 , 17950 , 19000 } . mutual coherence µ ( A ) and the discrimination power across several databases for a learned nonlinear transforms having different dimensionality. Additionally, a comparison between the resulting discrimination power in the original domain, after transform by a random matrix (having Gaussian random samples as entries and transform dimension of M = 19000 ) and after a learned non-linear transform having transform dimension M = 19000 without and with support dissimilarity prior, denoted as I 0 , I RT , I ST ∗

Original: and I NT , respectively, is estimated and presented .
Revised: and I NT , respectively, is estimated and presented .

Original: The second part evaluates a comparison of the discrimination power between the proposed algorithm and different supervised dictionary learning methods (SDL) Ramirez et al.
Revised: The second part evaluates a comparison of the discrimination power between the proposed algorithm and different supervised dictionary learning methods (SDL) Ramirez et al.

Original: (2010), Yang et al.
Revised: (2010), Yang et al.

Original: (2011a), Vu et al.
Revised: (2011a), Vu et al.

Original: and Vu & Monga (2016a).
Revised: and Vu & Monga (2016a).

Original: This comparison considers a setup where the used data sets are spited into a training and test set, moreover, the learning is performed on the
Revised: This comparison considers a setup where the used data sets are spited into a training and test set, moreover, the learning is performed on the

Original:  [Figure]
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: c t e d l o ss pe r d

Revised: i m en s i on

Original:  [Figure]
Revised: [Table] [Figure] [Table] [Figure]

Original: B M (cid:96) 1 ,c ( Y ) and C=
Revised: c t ed l [Figure] ss [Table]

Revised: r

Original: D B M (cid:96) 1 ( Y ) , their ratio C 1 /C 2 and the discrimination power log( C 1 /C 2) =
Revised: [Table] i [Figure] [Table]

Original: I t on a subset of the transform data using learned non-linear transform at different dimensionality
Revised: i [Figure] M  1 . 5 K 4 K

Revised: E [ (cid:107)

Revised: [Equation]

Revised: [Equation]

Original: M ∈ { 100 , 1150 , 2200 , 3250 , 4300 , 5350 , 6400 , 7450 , 8500 , 9550 , 10600 , 11650 , 12700 , 13750 , 14800 , 15850 , 16900 , 17950 , 19000 } .
Revised: M ]10 − 0 .0 .0 . I t 0 .0 .0 .1 .a) M 1 K 4 K 6 K 12 K E [ (cid:107) z c,k (cid:107) 2 M ]10 −3 .0 .0 .0 .

Revised: I t 1 .1 .1 .1 .b)

Revised: [Equation]

Revised: and the discrimination power on the Extended Yale B and MNIST databases, respectively, on the transform representation Y , obtained by using a non-linear transform T P at different dimensionality

Revised: M .

Original: training set and the evaluation is performed on the test set.
Revised: training set and the evaluation is performed on the test set.

Original: In the same series of experiments the recognition accuracy for two data sets is also computed and compared.
Revised: In the same series of experiments the recognition accuracy for two data sets is also computed and compared.

Revised: [Figure]

Original: Data sets and algorithms set up The used data sets are Extended YALE B ( D 1 )
Revised: Data sets and algorithms set up The used data sets are Extended YALE B ( D 1 )

Original: Georghiades et al.
Revised: Georghiades et al.

Original: (2001), AR ( D 2 ) Mart´ınez & Benavente (1998), Norb ( D 3 ) LeCun et al.
Revised: (2001), AR ( D 2 ) Mart ´ ınez & Benavente (1998), Norb ( D 3 ) LeCun et al.

Original: (2004), Coil- 20 ( D 4 )
Revised: (2004), Coil- 20 ( D 4 )

Original: Nene et al.
Revised: Nene et al.

Original: (1996), Clatech 101 ( D 5 )
Revised: (1996), Clatech 101 ( D 5 )

Original: LeCun et al.
Revised: LeCun et al.

Original: (2008), UKB ( D 6 ) Nist´er & Stew´enius (2006) and MNIST ( D 7 ) Lecun & Cortes.
Revised: (2008), UKB ( D 6 ) Nist ´ er & Stew ´ enius (2006) and MNIST ( D 7 ) Lecun & Cortes.

Original: All the images from the respective datasets were downscaled to resolutions 21 × 21 , 32 × 28 , 24 × 24 , 20 × 25 , 21 × 21 , 20 × 25 , 28 ×28 , respectively, and are normalized to unit variance.
Revised: All the images from the respective datasets were downscaled to resolutions 21 × 21 , 32 × 28 , 24 × 24 , 20 × 25 , 21 × 21 , 20 × 25 , 28 ×

Original: Considering the used implementation of the algorithm note that the singular value decomposition for large matrix has high computational complexity.

Original: However, A − ˆ A , where ˆ A is estimated as a solution in the transform update step, can be considered as an [Table] [Figure]
Revised: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table] [Figure] [Table] b) and MNIST databases, respectively, using a non-linear transform with different dimensionality M and linear SVM classiﬁer on top of the transform representation.
Revised: respectively

Original: 1.2 Discriminative power

Original: E x pe c t ed l o ss pe r d i m en s i on × 10E

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] [Figure] 5 Discriminative power
Revised: and are normalized to unit variance.

Original: E x p e c t ed l o ss pe r d i m en s i on × 10MNIST M  1 . 5 K 4 K E [ (cid:107) z c,k (cid:107) 2 M ]10 − 0 .0 .0 .
Revised: Considering the used implementation of the algorithm note that the singular value decomposition for large matrix has high computational complexity.

Original: I t 0 .0 .0 .1 .a) M 1 K 4 K 6 K 12 K E [ (cid:107) z c,k (cid:107) 2 M ]10 −3 .0 .0 .0 . I t 1 .1 .1 .1 .b)
Revised: However,

Original: Figure 10: a) and b) The expected loss per transform dimension E [ (cid:107)

Original: (cid:107)

Original: [Equation] ] and the discrimination power on the Extended Yale B and MNIST databases, respectively, on the transform representation Y , obtained by using a non-linear transform T P at different dimensionality M . proximal operator Parikh & Boyd (2014) for the gradient of the objective (11).
Revised: A − ˆ A , where ˆ A is estimated as a solution in the transform update step, can be considered as an proximal operator Parikh & Boyd (2014) for the gradient of the objective (11).

Original: Additionally, instead of using all of the available data samples X a subset of them might be used.
Revised: Additionally, instead of using all of the available data samples X a subset of them might be used.

Original: Therefore, one simple on-line variant for the update of A w.r.t.
Revised: Therefore, one simple on-line variant for the update of A w.r.t.

Original: a subset of the available training set has the form A t +1 =
Revised: a subset of the available training set has the form A t +1 =

Original: A t − ρ ( A t − ˆ A t ) with ρ predeﬁned step size.
Revised: A t − ρ ( A t − ˆ A t ) with ρ predeﬁned step size.

Original: In the numerical experiments we use the on-line variant[Figure] of the algorithm (the convergence analysis for this variant of the algorithm is left for future work).
Revised: In the numerical experiments we use the on-line variant of the algorithm (the convergence analysis for this variant of the algorithm is left for future work).

Original: the parameters λ 

Revised: the parameters λ

Original: and λ 1 are set such that the resulting non-linear transform representation has very small number of non-zeros w.r
Revised: and λ 1 are set such that the resulting non-linear transform representation has very small number of non-zeros w.r.t.

Original: [Figure]

Original: t. the transform dimension, in the experiments here this number is set to be 15 .
Revised: the transform dimension, in the experiments here this number is set to be 15 .

Original: The rest of the parameters are set as { λ 2 , λ 3 , λ 4 } = { 1000000 , 1000000 , 1000000 } .
Revised: The rest of the parameters are set as { λ 2 , λ 3 , λ 4 } = { 1000000 , 1000000 , 1000000 } .

Original: The algorithm is initialized with a random matrix having i.i
Revised: The algorithm is initialized with a random matrix having i.i.d.

Original: [Figure]

Original: d. Gaussian (zero mean, variance one) entries and is terminated after the 28 th iteration.
Revised: Gaussian (zero mean, variance one) entries and is terminated after the 28 th iteration.

Original: The results are obtained as average of 3 runs.
Revised: The results are obtained as average of 3 runs.

Original: An implementation presented in Vu & Monga (2016a) was used to learn the dictionaries and estimate the sparse codes for the respective supervised dictionary learning methods (SDL) Ramirez et al.
Revised: An implementation presented in Vu & Monga (2016a) was used to learn the dictionaries and estimate the sparse codes for the respective supervised dictionary learning methods (SDL) Ramirez et al.

Original: (2010), Yang et al.
Revised: (2010), Yang et al.

Original: (2011a), Vu et al.
Revised: (2011a), Vu et al.

Original: and Vu & Monga (2016a).
Revised: and Vu & Monga (2016a).

Original: Linear map properties, the change in the similarity concentrations and the discrimination power
Revised: Linear map properties, the change in the similarity concentrations and the discrimination power

Original: The conditioning number and the expected coherence for the learned transforms are shown on Table 2.
Revised: The conditioning number and the expected coherence for the learned transforms are shown on Table 2.

Original: The learned transforms for all the databases have good conditioning numbers and low expected coherence.
Revised: The learned transforms for all the databases have good conditioning numbers and low expected coherence.

Original: The running time t e , measured in minutes and the number of used dimensions, denoted as M are also shown in Table 2.
Revised: The running time t e , measured in minutes and the number of used dimensions, denoted as M are also shown in Table 2.

Original: The learned transforms for all the data sets have relatively low execution time, regardless of a transform dimensions 19000 .
Revised: The learned transforms for all the data sets have relatively low execution time, regardless of a transform dimensions 19000 .

Original: The discrimination power is signiﬁcantly increased in the transform domain I NT compared to the one in the original domain
Revised: The discrimination power is sig- niﬁcantly increased in the transform domain I NT compared to the one in the original domain

Original: I O and is higher than I ST ∗ and I RT .
Revised: I O and is higher than I ST ∗ and I RT .

Original: The evolution of the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96) 

Revised: The evolution of the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96)

Original: ( Y ) , their ratio
Revised: ( Y ) , their ratio

Original: C 1 /C 2 and the discrimination power log( C 1 /C 2) =
Revised: C 1 /C 2 and the discrimination power log( C 1 /C 2) =

Original: I t for a subsets of the used databases after applying a non-linear transform with transform dimension M = 19000 is shown on Figure 5.
Revised: I t for a subsets of the used databases after applying a non-linear transform with transform dimension M = 19000 is shown on Figure 5.

Original: It is important to note that the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96) 1 ( Y ) are decreasing, meaning that there is a loss in information.
Revised: It is important to note that the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96) 1 ( Y ) are decreasing, meaning that there is a loss in information.

Original: However, how this loss effects the resulting similarity concentration is crucial for the discriminative properties.
Revised: However, how this loss effects the resulting similarity concentration is crucial for the discriminative properties.

Original: As shown on Figure 5 the slope of decrease for C 2 is stronger then the slope of decrease for C 1 , therefore, the discrimination power increases per iteration.
Revised: As shown on Figure 5 the slope of decrease for C 2 is stronger then the slope of decrease for C 1 , therefore, the discrimination power increases per iteration.

Original: For the Coil- 20 ( D 4 ) database there is a ﬂuctuation.
Revised: For the Coil- 20 ( D 4 ) database there is a ﬂuctuation.

Original: This is explained by the fact that during learning we used a small number of data samples from the same database and that in the data there is high variability.
Revised: This is explained by the fact that during learning we used a small number of data samples from the same database and that in the data there is high variability.

Original: The conditioning number and the expected coherence for the learned transforms for all the databases at different transform dimensions are shown on Figure 7.
Revised: The conditioning number and the expected coherence for the learned transforms for all the databases at different transform dimensions are shown on Figure 7.

Original: We see that the value of both the conditioning number and the coherence is reducing and converging to a common values, respectively, implying that the conditioning and the coherence constraints are effective.
Revised: We see that the value of both the conditioning number and the coherence is reducing and converging to a common values, respectively, implying that the conditioning and the coherence constraints are effective.

Original: The ratio C 1 /C 2 between the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96) 

Revised: The ratio C 1 /C 2 between the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96)

Original: ( Y ) and the discrimination power log( C 1 /C 2)
Revised: ( Y ) and the discrimination power log( C 1 /C 2)

Original: =
Revised: =

Original: I t on a subsets of the used databases after applying a non-linear transform with transform dimension M = 19000 and varying the thresholding parameter τ = λ 1 is shown on Figure 6.
Revised: I t on a subsets of the used databases after applying a non-linear transform with transform dimension M = 19000 and varying the thresholding parameter τ = λ 1 is shown on Figure 6.

Original: We used 70 different values for the parameter λ , sampled uniformly from the interval (cid:0) 0 , (max c,k max m | a Tm x c,k | ) (cid:1) .
Revised: We used 70 different values for the parameter λ , sampled uniformly from the interval (cid:0) 0 , (max c,k max m | a Tm x c,k | ) (cid:1) .

Original: The result were obtained using a non-linear transforms learned with one value for the parameter λ for all the databases.
Revised: The result were obtained using a non-linear transforms learned with one value for the parameter λ for all the databases.

Original: Since all the databases have different variabilities and the amount of available data is different, this result suggest that per different database there should be different optimal value of the parameter λ .
Revised: Since all the databases have different variabilities and the amount of available data is different, this result suggest that per different database there should be different optimal value of the parameter λ .

Original: The similarity concentrations C
Revised: The similarity concentrations C=

Original: [Equation] and 
Revised: D B M (cid:96) 1 ,c ( Y ) and C=

Original: [Equation] = [Equation] , their ratio C 1 /C 2 and the discrimination power log( C 1 /C 2) =
Revised: D B M (cid:96) 1 ( Y ) , their ratio C 1 /C 2 and the discrimination power log( C 1 /C 2) =

Original: I t for a subsets of the used databases after applying a non-linear transform having transform dimensions
Revised: I t for a subsets of the used databases after applying a non-linear transform having transform dimensions

Original: M ∈ { 100 , 1150 , 2200 , 3250 , 4300 , 5350 , 6400 , 7450 , 8500 , 9550 , 10600 , 11650 , 12700 , 13750 , 14800 , 15850 , 16900 , 17950 , 19000 } is shown on Figure 8.
Revised: M ∈ { 100 , 1150 , 2200 , 3250 , 4300 , 5350 , 6400 , 7450 , 8500 , 9550 , 10600 , 11650 , 12700 , 13750 , 14800 , 15850 , 16900 , 17950 , 19000 } is shown on Figure 8.

Original: We can see a similar behavior as previous, that is, the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96) 

Revised: We can see a similar behavior as previous, that is, the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96)

Original: ( Y ) are decreasing, but, the slope of decrease for C 2 = D B M (cid:96) 

Revised: ( Y ) are decreasing, but, the slope of decrease for C 2 = D B M (cid:96)

Original: ( Y ) is stronger.
Revised: ( Y ) is stronger.

Original: Therefore, the discrimination power increases as the transform dimension increases.
Revised: Therefore, the discrimination power increases as the transform dimension increases.

Original: SNTL vs SDL discrimination power and recognition performance The proposed method is compared with DLSI Ramirez et al.
Revised: SNTL vs SDL discrimination power and recognition performance The proposed method is compared with DLSI Ramirez et al.

Original: , FDDL Yang et al.
Revised: , FDDL Yang et al.

Original: (2011a), COPAR Vu et al.
Revised: (2011a), COPAR Vu et al.

Original: and LRSDL Vu & Monga (2016a).
Revised: and LRSDL Vu & Monga (2016a).

Original: Half of the data samples from the data set Extended-YALEB, sampled at random are used for learning and the remaining other half are used for evaluation.
Revised: Half of the data samples from the data set Extended-YALEB, sampled at random are used for learning and the remaining other half are used for evaluation.

Original: Considering the MNIST database the training set is used for learning and the test set is used for evaluation.
Revised: Considering the MNIST database the training set is used for learning and the test set is used for evaluation.

Original: We compute both the discrimination power and the recognition accuracy on the test sets.
Revised: We compute both the discrimination power and the recognition accuracy on the test sets.

Original: The dictionary size (transform dimension M ) is set to be equal to 150 , 75 , 1515 , 3825 , 570 , 150 ,for the corresponding databases, respectively, in all of the comparing algorithms.
Revised: The dictionary size (transform dimension M ) is set to be equal to 150 , 75 , 1515 , 3825 , 570 , 150 ,for the corresponding databases, respectively, in all of the comparing algorithms.

Original: The discrimination power of the comparing methods is denoted as I DLSI , I FDDL , I COPAR
Revised: The discrimination power of the comparing methods is denoted as I DLSI , I FDDL , I COPAR

Original: and I LRSDL , respectively.
Revised: and I LRSDL , respectively.

Original: The recognition results for the methods DLSI , FDDL , COPAR and LRSDL on the data sets Extended YALE B and MNIST were not computed here, rather than that we use the best reported result form the respective papers Ramirez et al.
Revised: The recognition results for the methods DLSI , FDDL , COPAR and LRSDL on the data sets Extended YALE B and MNIST were not computed here, rather than that we use the best reported result form the respective papers Ramirez et al.

Original: (2010), Yang et al.
Revised: (2010), Yang et al.

Original: (2011a), Vu et al.
Revised: (2011a), Vu et al.

Original: (2015) and Vu & Monga (2016a).
Revised: (2015) and Vu & Monga (2016a).

Original: Considering the proposed algorithm the transform was learned for the transform dimensions M equal to [100 , 500 , 1500 , 4000] and [1000 , 4000 , 6000 , 12000] , respectively, for the used data sets.
Revised: Considering the proposed algorithm the transform was learned for the transform dimensions M equal to [100 , 500 , 1500 , 4000] and [1000 , 4000 , 6000 , 12000] , respectively, for the used data sets.

Original: After the transform was learned the transform data samples were computed for the respective training and test sets.
Revised: After the transform was learned the transform data samples were computed for the respective training and test sets.

Original: Then, the transform training data samples were used as features to learn a linear SVM classiﬁer in one-against-all regime.
Revised: Then, the transform training data samples were used as features to learn a linear SVM classiﬁer in one-against-all regime.

Original: Finally, the evaluation was performed on the respective test sets.
Revised: Finally, the evaluation was performed on the respective test sets.

Original: The results are shown on Table 3 a), b) and c).
Revised: The results are shown on Table 3 a), b) and c).

Original: The discrimination power of the proposed nonlinear transform is higher that the discrimination power of the comparing methods.
Revised: The discrimination power of the proposed nonlinear transform is higher that the discrimination power of the comparing methods.

Original: Whereas the recognition accuracy is higher for high dimensionality of the proposed method and outperforms the
Revised: Whereas the recognition accuracy is higher for high dimensionality of the proposed method and outperforms the

Original: SDL methods at dimensionality 4000 and 12000 .
Revised: SDL methods at dimensionality 4000 and 12000 .

Original: The results about the accuracy of recognition and the expected loss measured as E [ (cid:107) z c,k (cid:107) 2 M ] =
Revised: The results about the accuracy of recognition and the expected loss measured as E [ (cid:107) z c,k (cid:107) 2 M ] =

Original: E [ (cid:107)
Revised: E [ (cid:107)

Original: Ax c,k − y c,k (cid:107) 2 /M ] per the discrimination power at different transform dimension are shown on
Revised: Ax c,k − y c,k (cid:107) 2 /M ] per the discrimination power at different transform dimension are shown on

Original: Figure 9 and Figure 10, respectively.
Revised: Figure 9 and Figure 10, respectively.

Original: It is interesting to highlight that as the discrimination power at different transform dimension increases it also increases the accuracy of recognition, moreover, the results on these two data bases show that this increase is approximately linear.
Revised: It is interesting to highlight that as the discrimination power at different transform dimension increases it also increases the accuracy of recognition, moreover, the results on these two data bases show that this increase is approximately linear.

Original: On the other hand the expected loss per transform dimension decreases as the discrimination power at different transform dimension increases.
Revised: On the other hand the expected loss per transform dimension decreases as the discrimination power at different transform dimension increases.

Original: The results about the ratio C 1 /C 2 between the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96) 

Revised: The results about the ratio C 1 /C 2 between the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96)

Original: ( Y ) and the discrimination power log( C 1 /C 2)
Revised: ( Y ) and the discrimination power log( C 1 /C 2)

Original: =
Revised: =

Original: I t on Extended-Yale-B and MNIST databases after applying a non-linear transform with transform dimensions M = { 100 , 500 , 1500 , 4000 } and M = { 1000 , 4000 , 6000 , 12000 } , respectively, and varying the thresh- olding parameter τ = λ 1 are shown on Figure 11.
Revised: I t on Extended-Yale-B and MNIST databases after applying a non-linear transform with transform dimensions M = { 100 , 500 , 1500 , 4000 } and M = { 1000 , 4000 , 6000 , 12000 } , respectively, and varying the thresholding parameter τ = λ 1 are shown on Figure 11.

Original: We again used 70 different values for the parameter λ , sampled uniformly from the interval (cid:0) 0 , (max c,k max m | a Tm x c,k | ) (cid:1) , that were different for the used databases.
Revised: We again used 70 different values for the parameter λ , sampled uniformly from the interval (cid:0) 0 , (max c,k max m | a Tm x c,k | ) (cid:1) , that were different for the used databases.

Original: The result were obtained using a non-linear transforms learned with optimally choose values (by using cross-validation) of the parameter λ for the two different databases.
Revised: The result were obtained using a non-linear transforms learned with optimally choose values (by using cross-validation) of the parameter λ for the two different databases.

Original: As expected we can see the extreme points of the ratio between the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96) 

Revised: As expected we can see the extreme points of the ratio between the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96)

Original: ( Y ) and the discrimination power log( C 1 /C 2)
Revised: ( Y ) and the discrimination power log( C 1 /C 2)

Original: =
Revised: =

Original: I t is around the optimal values of the parameter τ = λ 1 .
Revised: I t is around the optimal values of the parameter τ = λ 1 .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper presented an analysis on the discriminative properties for non-linear transform models expressible as two step transform, linear mapping ( step 1 ) followed by an element wise non-linearity ( step 2 ).
Revised: This paper presented an analysis on the discriminative properties for non-linear transform models expressible as two step transform, linear mapping ( step 1 ) followed by an element wise non-linearity ( step 2 ).

Original: A novel discriminative prior was proposed and the properties around the model and the proposed prior were investigated.
Revised: A novel discriminative prior was proposed and the properties around the model and the proposed prior were investigated.

Original: A low complexity learning algorithm was presented with the proposed priors.
Revised: A low complexity learning algorithm was presented with the proposed priors.

Original: The preliminary results w.r.t.
Revised: The preliminary results w.r.t.

Original: the introduced measures and the recognition accuracy on the used databases showed promising performance.
Revised: the introduced measures and the recognition accuracy on the used databases showed promising performance.

Original: We showed that it is possible to increase the discrimination power with information loss.
Revised: We showed that it is possible to increase the discrimination power with information loss.

Original: Moreover, we highlight that when expanding to high dimensional space with non-linear transform how the loss of information reﬂects the similarity concentrations is crucial for the discriminative properties.
Revised: Moreover, we highlight that when expanding to high dimensional space with non-linear transform how the loss of information reﬂects the similarity concentrations is crucial for the discriminative properties.

Original: A study on the recognition capabilities for other databases are our next future extensions.
Revised: A study on the recognition capabilities for other databases are our next future extensions.

Original: An analysis for the synthesis model and transform based auto-encoder towards generalization and fair comparison between different transforms and encoding methods is other future direction.
Revised: An analysis for the synthesis model and transform based auto-encoder towards generalization and fair comparison between different transforms and encoding methods is other future direction.

Original: In the line of the encoding one might consider to minimize directly the difference or the ratio between the introduced similarity concentrations or the discrimination power.
Revised: In the line of the encoding one might consider to minimize directly the difference or the ratio between the introduced similarity concentrations or the discrimination power.

Original: In a different direction the extensions covering the sufﬁcient conditions for increase in discrimination power in the transform domain, together with an analysis for a deep architecture where per single layer we have one non-linear transform are left for our future work.
Revised: In a different direction the extensions covering the sufﬁcient conditions for increase in discrimination power in the transform domain, together with an analysis for a deep architecture where per single layer we have one non-linear transform are left for our future work.

Original: A. T HE GLOBAL OPTIMAL SOLUTION
Revised: A. T HE GLOBAL OPTIMAL SOLUTION

Original: Given the current estimate of Y the estimate of the transform A is solution to the following problem:
Revised: Given the current estimate of Y the estimate of the transform A is solution to the following problem:

Original: [Equation]
Revised: [Equation]

Revised: Theorem 1 ( global optimal solution ): Given Y ∈ (cid:60) M × L and X ∈ (cid:60) N × L , if and only if the joint decomposition :

Revised: [Equation]

Revised: exists, where U X ∈ (cid:60) N × N is orthonormal,

Revised: V XY ∈ (cid:60) M × N is per columns orthonormal and Σ X , Σ XY ∈ (cid:60) N × N are diagonal matrices with positive diagonal elements, then (21) has a global minimum as:

Revised: [Equation]

Revised: and Σ A ( n, n ) =

Revised: σ A ( n ) > 0 , σ A ( n ) , ∀ n are solutions to

Revised: [Equation]

Revised: Proof of Theorem 1 :

Original: Consider the equivalent trace form of (19) we have:
Revised: Consider the equivalent trace form of (21) we have:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that ∀ λ 2 ≥ 0 , XX T + λ 

Revised: Note that ∀ λ 2 ≥ 0 , XX T + λ

Original: I is symmetric positive deﬁnite matrix with all eigenvalues nonnegative, therefore it decomposes as:
Revised: I is symmetric positive deﬁnite matrix with all eigenvalues nonnegative, therefore it decomposes as:

Original: [Equation]
Revised: [Equation]

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: 23) where P m,M ∈ D M , P m,M ( m 1 , m 1)
Revised: where P m,M ∈ D M , P m,M ( m 1 , m 1)

Original: = 

Revised: =

Original: if m 1 = m and P m,M ( m 1 , m 1)
Revised: if m 1 = m and P m,M ( m 1 , m 1)

Original: = 

Revised: =

Original: if m 1 (cid:54) = m .
Revised: if m 1 (cid:54) = m .

Original: Then (19) equivalently is:
Revised: Then (21) equivalently is:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Assume that B decomposes as:
Revised: Assume that B decomposes as:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where Σ B is diagonal with positive diagonal elements, U B is column orthogonal and V B is orthogonal square matrix.
Revised: where Σ B is diagonal with positive diagonal elements, U B is column orthogonal and V B is orthogonal square matrix.

Original: Moreover, let the following decomposition of XY T exists
Revised: Moreover, let the following decomposition of XY T exists

Original: [Equation]
Revised: [Equation]

Original: 26) substitute as
Revised: 31) substitute as

Original: [Equation]
Revised: [Equation]

Original: 27) then
Revised: then

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The term
Revised: The term

Original: [Equation]
Revised: [Equation]

Original: Deﬁne Σ = Σ B Σ − 2 X Σ B then
Revised: Deﬁne Σ = Σ B Σ − 2 X Σ B then

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: Tr

Original: [Equation]
Revised: [Equation]

Original: Therefore, λ 3 Tr { g 4 } = λ 

Revised: Therefore, λ 3 Tr { g 4 } = λ

Original: Tr { g 5 } .
Revised: Tr { g 5 } .

Original: The term Tr { g 6 } =
Revised: The term Tr { g 6 } =

Original: Tr { (cid:0) Σ B Σ − 1 X (cid:1) 2 } .
Revised: Tr { (cid:0) Σ [Equation] Σ − 1 X (cid:1) 2 } .

Original: The term g 7 is
Revised: The term g 7 is

Original: [Equation]
Revised: [Equation]

Original: Finally (19) is reduced to:
Revised: Finally (21) is reduced to:

Original: [Equation]
Revised: [Equation]

Original: Equalling to zero the ﬁrst order derivative w.r.t σ B ( n ) of the objective in (33) and multiplying by σ B ( n ) gives:
Revised: Equalling to zero the ﬁrst order derivative w.r.t σ B ( n ) of the objective in (38) and multiplying by σ B ( n ) gives:

Original: [Equation]
Revised: [Equation]

Original: A closed form solution to (34) exists and depends on the discriminant ∆ of the quartic polynomial.
Revised: A closed form solution to (39) exists and depends on the discriminant ∆ of the quartic polynomial.

Original: Moreover [Equation] since 4 λ 4 σ X 4 is positive a global minimum exists as
Revised: Moreover, since 4 λ 4 σ X 4 is positive a global minimum exists as

Original: [Equation]
Revised: [Equation]

Original: [Equation] T HE (cid:15) - CLOSE SOLUTION
Revised: T HE (cid:15) - CLOSE CLOSED FORM APPROXIMATION

Revised: Proof of Proposition 1 :

Original: Consider the equivalent trace form of (19).
Revised: Consider the equivalent trace form of (21).

Original: Note that ∀ λ 2 ≥ 0 , XX T + λ 

Revised: Note that ∀ λ 2 ≥ 0 , XX T + λ

Original: I is symmetric positive deﬁnite matrix with all eigenvalues non-negative, therefore it decomposes as
Revised: I is symmetric positive deﬁnite matrix with all eigenvalues non-negative, therefore it decomposes as

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Let
Revised: Let

Original: D =
Revised: D =

Original: U X Σ − 1 X U TX , and deﬁne A = BD , U U XXY Σ U X
Revised: U X Σ − 1 X U TX , and deﬁne A = BD , U U X

Original: [Equation] TU X [Equation]
Revised: XY Σ U X XY V TU X XY =

Original:  X [Equation]
Revised: U X XY T .

Original: Assume that B decomposes as U B Σ B V TB , where Σ B is diagonal with positive diagonal elements,
Revised: Assume that B decomposes as U B Σ B V TB , where Σ B is diagonal with positive diagonal elements,

Original: U B is column orthogonal and V B is orthogonal square matrix.
Revised: U B is column orthogonal and V B is orthogonal square matrix.

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: 38) deﬁne Σ Γ , where σ Γ ( n ) = T ( n, n ) , ∀ n ∈ N and T =
Revised: 43) deﬁne Σ Γ , where σ Γ ( n ) = T ( n, n ) , ∀ n ∈ N and T =

Original: U
Revised: U

Original: U X
Revised: U X

Original: XY Σ
Revised: XY Σ

Original: U X XY U TU X XY , then
Revised: U X XY U TU X XY , then

Original: [Equation]
Revised: [Equation]

Original: The term
Revised: Moreover, since U B = ( U U X XY V TU X XY ) T , V B = U X note that

Original: [Equation]
Revised: [Equation]

Revised: The term Tr { BB T }

Original: [Equation]
Revised: [Equation]

Original: Deﬁne Σ = Σ B Σ − 2 X Σ B , then Tr { ( AA T ) 2 } 
Revised: } and the term

Original: [Equation]
Revised: [Equation]

Original: The termTr { AA T } =
Revised: The term

Original: Tr { (cid:0) Σ B Σ − 1 X (cid:1) 2 } .
Revised: [Equation]

Original: The termlog | det A T A
Revised: The term

Original: | is log | det A T A | = log | det U X Σ − 1 Y Σ 2 B Σ − 1 Y U X | = | det Σ − 1 Y Σ 2 B Σ − 1 Y | .
Revised: [Equation]

Original: Finally (19) reduces to min Σ B
Revised: Finally, using the bound (45)

Original: [Equation]
Revised: [Equation]

Revised: the approximation of (21) is reduced to min

Original: [Equation]
Revised: [Equation]

Original: Equaling to zero the ﬁrst order derivative w.r.t σ B ( n ) of the objective for (43) and multiplying by σ B ( n ) gives:
Revised: Equaling to zero the ﬁrst order derivative w.r.t σ B ( n ) of the objective for (47) and multiplying by σ B ( n ) gives:

Original: [Equation]
Revised: [Equation]

Original: A closed form solution to (43) exists and it depends on the discriminant ∆ of the quartic polynomial.
Revised: A closed form solution to (47) exists and it depends on the discriminant ∆ of the quartic polynomial.

Original: Moreover, since 4 λ 4 σ X 4 is positive a global minimum to (43) exists.
Revised: Moreover, since 4 λ 4 σ X 4 is positive a global minimum to (47) exists.

Original: Having U B =
Revised: Therefore, having U B =

Original: ( U U X
Revised: ( U U X

Original: XY V TU X XY ) T and V B
Revised: XY V TU X XY ) T and V B

Original: = U X with the solution for Σ B by (43) gives the (cid:15) -close  solution to problem (19) as[Equation] 45) where
Revised: = U X with the solution for Σ B by (47) gives the (cid:15) -close closed form approximative solution to problem (21) as

Original: [Equation]
Revised: [Equation]

Original: 46) implying that the (cid:15) -close closed form solution is a lower bound on the solution to (19).
Revised: 49) where by (45) it implies that the (cid:15) -close closed form solution is a lower bound on the solution to (21).

Original: A PPENDIX B.
Revised: A PPENDIX B.

Original: Let y c,k = y + c,k + y − c,k , y + c,k ∈ (cid:60) M +
Revised: Let y c,k = y + c,k + y − c,k , y + c,k ∈ (cid:60) M +

Original: and y − c,k ∈ (cid:60) M − .
Revised: and y − c,k ∈ (cid:60) M − .

Original: Note that the term D P (cid:96) 1 ( X ) is deﬁned as:
Revised: Note that the term D P (cid:96) 1 ( X ) is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where g + = (cid:80) c 1 ,c
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original:  =  c [Equation]
Revised: (50) where g + = (cid:80) c 1 ,c 2 c 1 (cid:54) =

Original:  k 1 ,k 2 | y + c 2 ,k 2 | , g − = (cid:80) c [Equation] c 2 c [Equation]
Revised: c 2 (cid:80) k 1 ,k 2 | y + c 2 ,k 2 | , g − = (cid:80) c 1 ,c 2 c 1 (cid:54) =

Original: = c 2 (cid:80) k 1 ,k 2 | y − c 2 ,k 2 |
Revised: c 2 (cid:80) k 1 ,k 2 | y − c 2 ,k 2 |

Original: and we abuse notation by denoting | y c 1 ,k 1 | as the vector whose elements are the absolute values of the elements in y c 1 ,k 1 .
Revised: and we abuse notation by denoting | y c 1 ,k 1 | as the vector whose elements are the absolute values of the elements in y c 1 ,k 1 .

Original: Note that the term S P (cid:96) 2 ( X ) is deﬁned as:
Revised: Note that the term S P (cid:96) 2 ( X ) is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where s c =
Revised: where s c =

Original: [Equation]
Revised: [Equation]

Original: Consider the related problem
Revised: Consider the related problem

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  by taking the ﬁrst order derivative w [Equation] r [Equation]
Revised: 52) by taking the ﬁrst order derivative w.r.t.

Original: t. y c 1 ,k 1{+
+} we have that
Revised: y c 1 ,k {+
+} we have that

Original: [Equation]
Revised: [Equation]

Original:  take the sign magnitude decomposition of y c 1 ,k 1 = sign( y c 1 ,k 1 ) (cid:12) | y c 1 ,k 1 | then we have sign 
Revised: 53) take the sign magnitude decomposition of y c 1 ,k 1 = sign( y c 1 ,k 1 ) (cid:12) | y c 1 ,k 1 | then we have sign [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Let the sign of y c 1 ,k 1 , i.e. sign( y c 1 ,k 1 ) be equal to the sign of sign( q c 1 ,k 1 ) , and Hadamard multiply from the left side by sign( q c 1 ,k 1 ) then we have 
Revised: Let the sign of y c 1 ,k 1 , i.e. sign( y c 1 ,k 1 ) be equal to the sign of sign( q c 1 ,k 1 ) , and Hadamard multiply from the left side by sign( q c 1 ,k 1 ) then we have ( [Equation] [Equation] | y c

Original: [Equation]
Revised: [Equation]

Original: sign
Revised: k 1 | − | q c

Original: [Equation]
Revised: [Equation]

Original: sign
Revised: k 1 | + λ 1 1 + λ 0 (sign( q

Original: [Equation]
Revised: [Equation]

Original: 52) note that sign( q c 1 ,k 1 ) (cid:12) sign( q + c 1 ,k 1 )
Revised: 1 ,k 1 ) (cid:12) sign [Equation]

Original: = sign( q + c 1 ,k 1 ) and that sign( q c 1 ,k 1 ) (cid:12)  sign( q− c 1 ,k 1 ) =
Revised: q + [Equation] 1 ,k 1 ) (cid:12) g + + sign( q c 1 ,k 1 ) (cid:12) sign [Equation]

Original: sign( − q − c 1 ,k 1 ) , therefore we have
Revised: g

Original: [Equation]
Revised: [Equation]

Original: 0 (sign( q

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: (55) note that sign( q c 1 ,k 1 ) (cid:12) sign( q + c 1 ,k 1 )

Original: 1 + [Equation] [Equation]
Revised: = sign( q + c 1 ,k 1 ) and that sign( q c 1 ,k 1 ) (cid:12) sign( q − c 1 ,k 1 ) = sign( − q − c 1 ,k 1 ) , therefore we have

Original: [Equation]
Revised: [Equation]

Original: Deﬁne G c 1 ,k 1 as diagonal matrix with positive diagonal elements G c 1 ,k 1 (  m [Equation] m [Equation]
Revised: Deﬁne G c 1 ,k 1 as diagonal matrix with positive diagonal elements G c 1 ,k 1 ( m, m )

Original: =
Revised: =

Original: z ( m [Equation] , ∀ m ∈ M where:
Revised: z ( m ) , ∀ m ∈ M where:

Original: [Equation]
Revised: [Equation]

Original: and note that G c 1 ,k 1 

Revised: and note that G c 1 ,k 1

Original: =
Revised: =

Original: z then
Revised: z then

Original: [Equation]
Revised: [Equation]

Original:  since the magnitude might be only positive | y c [Equation] k
Revised: 58) since the magnitude might be only positive | y c 1 ,k

Original: [Equation]
Revised: [Equation]

Original: max
Revised: max

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Therefore, the closed form solution to (49) is:
Revised: Therefore, the closed form solution to (52) is:

Original: [Equation]
Revised: [Equation]

Original:  which completes the proof (cid:3)
Revised: 59) which completes the proof (cid:3)

Original: A PPENDIX C.
Revised: A PPENDIX C.

Original: Note that for the model P t we have that
Revised: Note that for the model P t we have that

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  sincesign
Revised: 60) since

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst order derivative of the divergence D P t (cid:96) 1 ( x ; g ) w.r.t.
Revised: The ﬁrst order derivative of the divergence D P t (cid:96) 1 ( x ; g ) w.r.t.

Original: the parameter A is:
Revised: the parameter A is:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: A (59) we assume that the threshold parameter τ is chosen such that the vector
Revised: we assume that the threshold parameter τ is chosen such that the vector

Original: | Ax | − τ (or for any other q , the vector | Aq | − τ ) has least one non-zero element, then∂ (max( Ax − [Equation] [Equation] max
Revised: | Ax | − τ (or for any other q , the vector | Aq | − τ ) has least one non-zero element, then

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: combining (60) and (61) we have that
Revised: combining (63) and (64) we have that

Original: [Equation]
Revised: [Equation]

Original: 62) where
Revised: where

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: ∀ m ∈ M .
Revised: ∀ m ∈ M .

Original: Similarity, note that for the model P o 0 = { A o , τ = 0 } we have that
Revised: Similarity, note that for the model P o 0 = { A o , τ = 0 } we have that

Original: [Equation]
Revised: [Equation]

Original: [Equation] The ﬁrst order derivative of the divergence D P o 0 (cid:96)
Revised: The ﬁrst order derivative of the divergence D P o [Equation] [Equation]

Revised: [Equation]

Revised: .t

Original: [Equation]
Revised: [Equation]

Revised: is:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: note that at A o
Revised: note that at A o

Original: =
Revised: =

Original: I , P t 0 =
Revised: I , P t 0 =

Original: B N
Revised: B N

Original: and we have that
Revised: and we have that

Original: [Equation]
Revised: [Equation]

Original: Also for the model P t 

Revised: Also for the model P t

Original: = { A t , τ = 0 } we have that
Revised: = { A t , τ = 0 } we have that

Original: [Equation]
Revised: [Equation]

Original: [Equation] The ﬁrst order derivative of the divergence D P t 0 (cid:96) 1 ( y ; q ) w.r.t A t is:
Revised: The ﬁrst order derivative of the divergence D P t [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: note that at A t =
Revised: note that at A t =

Original: I , P t 0 =
Revised: I , P t 0 =

Original: B M and we have that
Revised: B M and we have that

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Consider the following
Revised: Consider the following

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  where
Revised: 73) where

Original: [Equation]
Revised: [Equation]

Original: A closer look at (70) reveals us that
Revised: A closer look at (73) reveals us that

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: By similar construction applied to the rest of the pairs of data samples we have the result:
Revised: By similar construction applied to the rest of the pairs of data samples we have the result:

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: A closer look at (75) reveals us
Revised: A closer look at (78) reveals us

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: By similar construction applied to the rest of the pairs of data samples we have the result:
Revised: By similar construction applied to the rest of the pairs of data samples we have the result:

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: A closer look at (80) reveals us
Revised: A closer look at (83) reveals us

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: By similar construction applied to the rest of the pairs of data samples we have the result:
Revised: By similar construction applied to the rest of the pairs of data samples we have the result:

Original: [Equation]
Revised: [Equation]

Original: A PPENDIX D.
Revised: A PPENDIX D.

Original: The result in (16) decomposes on the contributing components for similarity and the contributing components for dissimilarity, 
Revised: The result in (18) decomposes on the contributing components for similarity and the contributing components for dissimilarity, i.e. , ∂ J (cid:96) 1 ( AX ) ∂

Original: i
Revised: A t

Original: [Equation]
Revised: [Equation]

Original: e. ,
Revised: A t [Equation] −

Original: [Equation]
Revised: [Equation]

Revised: c

Original: [Equation]
Revised: [Equation]

Revised: d + ξ s − ξ d .

Original: Moreover, w.r.t.
Revised: Moreover, w.r.t.

Original: the similarity concentrations we have the following splitting Tr {
Revised: the similarity concentrations we have the following splitting Tr {

Original: [Equation]
Revised: [Equation]

Original: X ) +Tr { ξ
Revised: +

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  } and
Revised: c,s } and

Original: [Equation]
Revised: [Equation]

Original: |

Original: [Equation] [Equation] } = D P (cid:96) 1 ,c ( X ) + Tr { ξ s } .
Revised: + Tr { ξ s } = D P (cid:96) 1 ,c ( X ) + Tr { ξ s } .

Original: Therefore, we have the following bounds
Revised: Therefore, we have the following bounds

Original: [Equation]
Revised: [Equation]

Original: Note that c :λ min ( A T A )
Revised: Note that c :

Original: Tr { ∂D B N (cid:96)

Original: [Equation]
Revised: [Equation]

Original: X [Equation] o |
Revised: I A T } where λ min ( A T A ) is the minimum singular value to the matrix A T A .

Original: [Equation] o = [Equation] Tr {
Revised: Taking the logarithm of the ratio D

Original: [Equation]
Revised: [Equation]

Original: N (cid:96) 1 ,c (
Revised: X

Original: [Equation]
Revised: [Equation]

Original: A o | A o = I A T } where λ min ( A T A ) is the minimum singular value to the matrix A T A .
Revised: D P

Original: Taking the logarithm of the ratio D P (cid:96) 1 ,c ( X ) D P (cid:96) 1 ( X )+ (cid:15) and using the bounds a, b and c we arrive at the desired result (cid:3)
Revised: [Equation] [Equation] and using the bounds a, b and c we arrive at the desired result (cid:3)