Original: L EARNING NON - LINEAR TRANSFORM WITH DISCRIMINATIVE AND MINIMUM INFORMATION LOSS PRIORS
Revised: L EARNING NON - LINEAR TRANSFORM WITH DISCRIMINATIVE AND MINIMUM INFORMATION LOSS PRIORS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper proposes a novel approach for the learning of discriminative and sparse representations.
Revised: This paper proposes a novel approach for the learning of discriminative and sparse representations.

Original: It consists of utilizing two different models.
Revised: It consists of utilizing two different models.

Original: A predeﬁned number of non-linear transform models are used in the learning stage and one sparsifying transform model is used at test time.
Revised: A predeﬁned number of non-linear transform models are used in the learning stage and one sparsifying transform model is used at test time.

Original: The non-linear transform models have discriminative and minimum information loss priors.
Revised: The non-linear transform models have discriminative and minimum information loss priors.

Original: A novel measure related to the discriminative prior is proposed and deﬁned on the support intersection for the transform representations.
Revised: A novel measure related to the discriminative prior is proposed and deﬁned on the support intersection for the transform representations.

Original: The minimum information loss prior is expressed as a constraint on the conditioning and the expected coherence of the transform matrix.
Revised: The minimum information loss prior is expressed as a constraint on the conditioning and the expected coherence of the transform matrix.

Original: This matrix is also seen as a shared parameter across the non-linear models.

Original: An equivalence between the non-linear models and the sparsiﬁyng model is shown only when the measure that is used to deﬁne the discriminative prior goes to zero.
Revised: An equivalence between the non-linear models and the sparsiﬁyng model is shown only when the measure that is used to deﬁne the discriminative prior goes to zero.

Original: An approximation of the measure used in the discriminative prior is addressed, connecting it to a similarity concentration.
Revised: An approximation of the measure used in the discriminative prior is addressed, connecting it to a similarity concentration.

Original: Towards quantifying the discriminative properties of the transform representation another measure, reﬂecting the discriminative quality of the transform representation, named as discrimination power is introduced and its bounds are presented.
Revised: Towards quantifying the discriminative properties of the transform representation another measure, reﬂecting the discriminative quality of the transform representation, named as discrimination power is introduced and its bounds are presented.

Original: To support and validate the theoretical analysis a learning algorithm is presented
Revised: To support and validate the theoretical analysis a learning algorithm is presented.

Original: The advantages and the potential of the proposed algorithm are evaluated by a computer simulation.
Revised: The advantages and the potential of the proposed algorithm are evaluated by a computer simulation.

Original: A favorable performance is shown considering the execution time, the quality of the representation, measured by the discrimination power and the recognition accuracy compared with the state-of-the-art methods of the same class.
Revised: A favorable performance is shown considering the execution time, the quality of the representation, measured by the discrimination power and the recognition accuracy compared with the state-of-the-art methods of the same class.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Learning a transform that provides a sparse and discriminative representation is an active domain of research in many areas, including data processing, pattern recognition, image processing, language modeling, text analysis, gene separation, etc.
Revised: Learning a transform that provides a sparse and discriminative representation is an active domain of research in many areas, including data processing, pattern recognition, image processing, language modeling, text analysis, gene separation, etc.

Original: A class of algorithms proposed by (Kreutz-Delgado et al., 2003), (Mairal et al., 2009), (Bengio et al., 2012), (Gangeh et al., 2015), (Mairal et al., 2008), (Jiang et al., 2011), (Guo et al., 2012), (Cai et al., 2014) and (Liu et al., 2016) for learning discriminative sparse representations have been shown to perform well across various learning tasks.
Revised: A class of algorithms proposed by (Kreutz-Delgado et al., 2003), (Mairal et al., 2009), (Bengio et al., 2012), (Gangeh et al., 2015), (Mairal et al., 2008), (Jiang et al., 2011), (Guo et al., 2012), (Cai et al., 2014) and (Liu et al., 2016) for learning discriminative sparse representations have been shown to perform well across various learning tasks.

Original: A subclass of them known as discriminative dictionary learning (DDL) (Guo et al., 2012), (Jiang et al., 2013), (Cai et al., 2014), (Shekhar et al., 2014), (Xu et al., 2015), (Liu et al., 2016), (Bengio et al., 2012), (Gangeh et al., 2015), (Jiang et al., 2016), (Vu & Monga, 2016) address the estimate of the dictionary in a supervised manner such that the representation w.r.t.
Revised: A subclass of them known as discriminative dictionary learning (DDL) (Guo et al., 2012), (Jiang et al., 2013), (Cai et al., 2014), (Shekhar et al., 2014), (Xu et al., 2015), (Liu et al., 2016), (Bengio et al., 2012), (Gangeh et al., 2015), (Jiang et al., 2016), (Vu & Monga, 2016) address the estimate of the dictionary in a supervised manner such that the representation w.r.t.

Original: words (vectors) from the resulting dictionary (vector set) is discriminative.
Revised: words (vectors) from the resulting dictionary (vector set) is discriminative.

Original: Most of the DDL methods synthesize the data sample x c,k ∈ (cid:60) N as an approximation by a linear combination y c,k ∈ (cid:60) M (referred to as a sparse data representation) of a few words (vectors) (cid:107) y c,k (cid:107) 0 << M , from a dictionary (vector set)
Revised: Most of the DDL methods synthesize the data sample x c,k ∈ (cid:60) N as an approximation by a linear combination y c,k ∈ (cid:60) M (referred to as a sparse data representation) of a few words (vectors) (cid:107) y c,k (cid:107) 0 << M , from a dictionary (vector set)

Original: D ∈ (cid:60) N × M , i.e. , x c,k = Dy c,k
Revised: D ∈ (cid:60) N × M , i.e. , x c,k = Dy c,k

Original: + v c,k , v c,k ∈ (cid:60) N , with v c,k denoting the approximation error.
Revised: + v c,k , v c,k ∈ (cid:60) N , with v c,k denoting the approximation error.

Original: It is important to highlight that with the synthesis model approach the data reconstruction is addressed.
Revised: It is important to highlight that with the synthesis model approach the data reconstruction is addressed.

Original: The differences between the DDL methods (Guo et al., 2012), (Jiang et al., 2013), (Cai et al., 2014), (Shekhar et al., 2014), (Gangeh et al., 2015), (Xu et al., 2015), (Liu et al., 2016), (Bengio et al., 2012), (Jiang et al., 2016), (Vu & Monga, 2016) are determined by the prior deﬁned on the sparse representation and the prior deﬁned for the relations between the sparse representations for the data samples from the same/different classes.
Revised: The differences between the DDL methods (Guo et al., 2012), (Jiang et al., 2013), (Cai et al., 2014), (Shekhar et al., 2014), (Gangeh et al., 2015), (Xu et al., 2015), (Liu et al., 2016), (Bengio et al., 2012), (Jiang et al., 2016), (Vu & Monga, 2016) are determined by the prior deﬁned on the sparse representation and the prior deﬁned for the relations between the sparse representations for the data samples from the same/different classes.

Original: The discrimination is enforced by replacing the prior with a structural constraint on the dictionary or imposing a discriminative term on the sparse representa- tions.
Revised: The discrimination is enforced by replacing the prior with a structural constraint on the dictionary or imposing a discriminative term on the sparse representations.

Original: Additionally, in a number of works such as (Mairal et al., 2008), (Guo et al., 2012), (Taalimi et al., 2015) is considered even a joint estimation/learning of a dictionary, sparse representation and classiﬁer by using iterative alternating minimization strategy.
Revised: Additionally, in a number of works such as (Mairal et al., 2008), (Guo et al., 2012), (Taalimi et al., 2015) is considered even a joint estimation/learning of a dictionary, sparse representation and classiﬁer by using iterative alternating minimization strategy.

Original: A comprehensive overview covering different approaches is given in (Bengio et al., 2012), (Cai et al., 2014) and (Gangeh et al., 2015).
Revised: A comprehensive overview covering different approaches is given in (Bengio et al., 2012), (Cai et al., 2014) and (Gangeh et al., 2015).

Original: 1. O PEN ISSUES
Revised: 1. O PEN ISSUES

Original: The general open issue for DDL methods is the computational complexity w.r.t.
Revised: The general open issue for DDL methods is the computational complexity w.r.t.

Original: the optimal dictionary/transform learning and the discriminative encoding, since the sparse representation in the synthesis model is a solution to an inverse problem .
Revised: the optimal dictionary/transform learning and the discriminative encoding, since the sparse representation in the synthesis model is a solution to an inverse problem .

Original: An additional open issue with most of the proposed approaches (Guo et al., 2012), (Jiang et al., 2013), (Cai et al., 2014), (Gangeh et al., 2015), (Liu et al., 2016), (Bengio et al., 2012), (Jiang et al., 2016) and (Vu & Monga, 2016) is that there is no formal notion to measure the discriminative properties.
Revised: An additional open issue with most of the proposed approaches (Guo et al., 2012), (Jiang et al., 2013), (Cai et al., 2014), (Gangeh et al., 2015), (Liu et al., 2016), (Bengio et al., 2012), (Jiang et al., 2016) and (Vu & Monga, 2016) is that there is no formal notion to measure the discriminative properties.

Original: Therefore, there are no means that provide a quantitative evaluation about the quality of the representation, other then the performance of a classiﬁer used on top of the representation.
Revised: Therefore, there are no means that provide a quantitative evaluation about the quality of the representation, other then the performance of a classiﬁer used on top of the representation.

Original: In terms of the speciﬁcs about the discriminative constraints, (Yang et al., 2011b) proposed a synthesis model with a discriminative ﬁdelity term and Fisher discriminant constraints, where the withinclass scatter and the between-class scatter of the representation is minimized and maximized, respectively.
Revised: In terms of the speciﬁcs about the discriminative constraints, (Yang et al., 2011b) proposed a synthesis model with a discriminative ﬁdelity term and Fisher discriminant constraints, where the withinclass scatter and the between-class scatter of the representation is minimized and maximized, respectively.

Original: (Vu & Monga, 2016) proposed an extension considering a low rank constraint on the dictionary.
Revised: (Vu & Monga, 2016) proposed an extension considering a low rank constraint on the dictionary.

Original: The authors in (Guo et al., 2013) used a synthesis model with a pairwise constraints on the sparse representation.
Revised: The authors in (Guo et al., 2013) used a synthesis model with a pairwise constraints on the sparse representation.

Original: They have modeled the pair-wise constraint using a (cid:96) 2 distance metric.
Revised: They have modeled the pair-wise constraint using a (cid:96) 2 distance metric.

Original: The methods of (Yang et al., 2011b), (Vu & Monga, 2016) and (Guo et al., 2013) take into account assumption on the metric by deﬁning the scatter and the pair-wise relation.
Revised: The methods of (Yang et al., 2011b), (Vu & Monga, 2016) and (Guo et al., 2013) take into account assumption on the metric by deﬁning the scatter and the pair-wise relation.

Original: Therefore, they constrain the space of the representation, that essentially is determined by the dictionary.
Revised: Therefore, they constrain the space of the representation, that essentially is determined by the dictionary.

Original: However, these works do not consider whether the used metric is optimal w.r.t.
Revised: However, these works do not consider whether the used metric is optimal w.r.t.

Original: the sparse representation.
Revised: the sparse representation.

Original: The authors in (Liu et al., 2016) have proposed a method that ﬁnds a dictionary under which the coefﬁcients of a data sample from the same class
Revised: The authors in (Liu et al., 2016) have proposed a method that ﬁnds a dictionary under which the coefﬁcients of a data sample from the same class

Original: c have a common sparse structure while the size of the overlapping signal support of different classes, denoted as c 1 and c 2 is minimized.
Revised: c have a common sparse structure while the size of the overlapping signal support of different classes, denoted as c 1 and c 2 is minimized.

Original: Furthermore, assuming y c 1 ,k 1 ∈ (cid:60) M and y c 2 ,k 2 ∈ (cid:60) M are two sparse representations for data samples x c 1 ,k 1 ∈ (cid:60) N and x c 2 ,k 2 ∈ (cid:60) N , then the proposed similarity measure in (Liu et al., 2016) is deﬁned as expectation of (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 0 , where (cid:12) represents the Hadamar product.
Revised: Furthermore, assuming y c 1 ,k 1 ∈ (cid:60) M and y c 2 ,k 2 ∈ (cid:60) M are two sparse representations for data samples x c 1 ,k 1 ∈ (cid:60) N and x c 2 ,k 2 ∈ (cid:60) N , then the proposed similarity measure in (Liu et al., 2016) is deﬁned as expectation of (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 0 , where (cid:12) represents the Hadamar product.

Original: Note that two transform data samples y c 1 ,k 2 and y c 2 ,k 2 that have small support overlap (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 

Revised: Note that two transform data samples y c 1 ,k 2 and y c 2 ,k 2 that have small support overlap (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107)

Original: = s, s << M , might not necessarily be similar or dissimilar, i.e. , y c 1 ,k 1 = y c 2 ,k 2 and y c 1 ,k 1 = − y c 2 ,k 2 with (cid:107) y c 1 ,k 1 (cid:107) 

Revised: = s, s << M , might not necessarily be similar or dissimilar, i.e. , y c 1 ,k 1 = y c 2 ,k 2 and y c 1 ,k 1 = − y c 2 ,k 2 with (cid:107) y c 1 ,k 1 (cid:107)

Original: = (cid:107) y c
Revised: = (cid:107) y c

Original: ,k 2 (cid:107) 

Revised: ,k 2 (cid:107)

Original: = s and s small.
Revised: = s and s small.

Original: 1. M OTIVATION AND APPROACH
Revised: M OTIVATION AND APPROACH

Original: Instead of adressing a synthesis modellike in the DDL methods, where the  estimation of the discriminative representation is an inverse problem 1 we propose a novel, alternative aproach.
Revised: Instead of adressing a synthesis model where the data reconstruction is modeled and the estimation of the discriminative representation is an inverse problem 1 we propose a novel, alternative aproach.

Original: Namely, we propose a non-linear transform model as an extention of the sparsifying transform model (Ru- binstein et al., 2010), (Rubinstein et al., 2013), (Rubinstein & Elad, 2014) and (Ravishankar & Bresler, 2014) in the learning stage and a sparsifying transform model for testing.
Revised: Namely, we propose a non-linear transform model as an extention of the sparsifying transform model (Rubinstein et al., 2010), (Rubinstein et al., 2013), (Rubinstein & Elad, 2014) and (Ravishankar & Bresler, 2014) in the learning stage and a sparsifying transform model for testing.

Original: Moreover, in both of the models the data reconstruction is not modeled, and at least it is not targeted explicitly.

Original: By these models we address a direct problem , where the estimation of the discriminative representation represents a low complexity constrained projection problem.

Original: The other advantege of the the non-linear transform model is that it does not restrict the transform representation to be in the column space of the dictionary like in the DDL methods. This allows more freedom in modeling and imposing constraints on the transform representation 2 . In this line, we have to note that the sparsifying transform model was ﬁrst introduced in (Ravishankar & Bresler, 2012). This model assumes that the data sample x c,k is approximately sparsiﬁable under a linear transformA ∈ (cid:60) M × N , i.e. ,
Revised: The latter model assumes that the data sample x c,k is approximately sparsiﬁable under a linear transform

Original:  Ax c,k = y c,k + z c,k , z c,k ∈ (cid:60) M , where y c,k is sparse (cid:107) y c,k (cid:107) 0 << M .
Revised: A ∈ (cid:60) M × N , i.e. , Ax c,k = y c,k + z c,k , z c,k ∈ (cid:60) M , where y c,k is sparse (cid:107) y c,k (cid:107) 0 << M .

Original: Moreover, we alsonote that the sparsiﬁying transform model represents a generalization of the analysis model (Rubinstein et al., 2010), (Rubinstein et al., 2013), (Ravishankar & Bresler, 2014) and (Rubinstein & Elad, 2014). 
Revised: It also represents a generalization of the analysis model (Rubinstein et al., 2010), (Rubinstein et al., 2013), (Ravishankar & Bresler, 2014) and (Rubinstein & Elad, 2014). Whereas the non-linear tranform model cosideres an additional assumption.

Revised: Both of the models used in this paper address a direct problem , where the estimation of the discriminative representation represents a low complexity constrained projection problem.

Revised: Additionaly, since these models have no restrictions on the transform representation to be in the column space of the dictionary (transform matrix A ), they allow more freedom in modeling and imposing constraints on the transform representation 2 .

Original: Central in the non-linear transform models, that are used in the learning stage, is the proposed novel prior, which is deﬁned by a measure on the support intersection for the positive and negative components of a representation.
Revised: Central in the non-linear transform models, that are used in the learning stage, is the proposed novel prior, which is deﬁned by a measure on the support intersection for the positive and negative components of a representation.

Original: The ﬁrst motivation behind this prior is that the support  intersection on the positive and negativecomponents between a transform data allows more freedom in imposing regularization on the discriminative properties for the transform representation without taking into account any additional assumptions, contrary to (Yang et al., 2011b), (Vu & Monga, 2016), (Guo et al., 2013) and (Liu et al., 2016).
Revised: The ﬁrst motivation behind this prior is that the support [Figure] intersection on the positive and negative between a transform data allows more freedom in imposing regularization on the discriminative properties for the transform representation without taking into account any additional assumptions, contrary to (Yang et al., 2011b), (Vu & Monga, 2016), (Guo et al., 2013) and (Liu et al., 2016).

Original: Second, by a simple approximation on the prior the focus of the regularization is directly put on the contributing components for similarity/dissimilarity.
Revised: Second, by a simple approximation on the prior the focus of the regularization is directly put on the contributing components for similarity/dissimilarity.

Original: Consider the measure ( y + c 1 ,k 1 )
Revised: Consider the measure ( y + c 1 ,k 1 )

Original: T y + c 2 ,k 

Revised: T y + c 2 ,k

Original: + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 between two transform representations y c 1 ,k 1 and y c 2 ,k 2 , 3 .
Revised: + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 between two transform representations y c 1 ,k 1 and y c 2 ,k 2 , 3 .

Original: Note that ( y + c 1 ,k 1 ) T y + c 2 ,k 2 + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 = (cid:107) y + c 1 ,k 1 (cid:12) y + c 2 ,k 2 (cid:107) 1 + (cid:107) y − c 1 ,k 1 (cid:12) y − c 2 ,k 2 (cid:107) 1 captures the only contribution for the similarity (whereas (cid:107)
Revised: Note that ( y + c 1 ,k 1 ) T y + c 2 ,k 2 + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 = (cid:107) y + c 1 ,k 1 (cid:12) y + c 2 ,k 2 (cid:107) 1 + (cid:107) y − c 1 ,k 1 (cid:12) y − c 2 ,k 2 (cid:107) 1 captures the only contribution for the similarity (whereas (cid:107)

Original: y + c 1 ,k 1 (cid:12) y − c 2 ,k 2 (cid:107) 1 + (cid:107) y − c 1 ,k 1 (cid:12) y
Revised: y + c 1 ,k 1 (cid:12) y − c 2 ,k 2 (cid:107) 1 + (cid:107) y − c 1 ,k 1 (cid:12) y

Original: + c 2 ,k 2 (cid:107) 1 captures the only contribution for the dissimilarity) between the vectors y c 1 ,k 1 and y c 2 ,k 2 .
Revised: + c 2 ,k 2 (cid:107) 1 captures the only contribution for the dissimilarity) between the vectors y c 1 ,k 1 and y c 2 ,k 2 .

Original: Moreover, it can be shown that for certain models under some conditions y
Revised: Moreover, it can be shown that for certain models under some conditions y

Original: Tc [Equation]{+1 ,k 1 y c 2 ,k 
+}
Revised: Tc {+1 ,k 1 y c 2 ,k 
+}

Original: [Equation] y c 2 ,k 2 (cid:107) 1 , if and only if the dissimilarity contribution on the support intersection between the vectors y c 1 ,k 1 and y c 2 ,k 2 is 0 .
Revised: = (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 1 , if and only if the dissimilarity contribution on the support intersection between the vectors y c 1 ,k 1 and y c 2 ,k 2 is 0 .

Original: That is y
Revised: That is y

Original: Tc
Revised: Tc

Original: 1 ,k 1 y c 2 ,k 2 =
Revised: 1 ,k 1 y c 2 ,k 2 =

Original: ( y + c 1 ,k 1 ) T y + c 2 ,k 

Revised: ( y + c 1 ,k 1 ) T y + c 2 ,k

Original: + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 − ( y − c 1 ,k 1 ) T y + c 2 ,k 2 − ( y + c 1 ,k 1 ) T y − c 2 ,k 2 ≤ ( y + c 1 ,k 1 )
Revised: + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 − ( y − c 1 ,k 1 ) T y + c 2 ,k 2 − ( y + c 1 ,k 1 ) T y − c 2 ,k 2 ≤ ( y + c 1 ,k 1 )

Original: T y + c 2 ,k 

Revised: T y + c 2 ,k

Original: + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 , now if the dissimilarity contribution − ( y − c 1 ,k 1 ) T y + c 2 ,k 2 − ( y + c 1 ,k 1 )
Revised: + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 , now if the dissimilarity contribution − ( y − c 1 ,k 1 ) T y + c 2 ,k 2 − ( y + c 1 ,k 1 )

Original: T y − c 2 ,k 2 is zero then (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 

Revised: T y − c 2 ,k 2 is zero then (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107)

Original: =
Revised: =

Original: ( y + c 1 ,k 1 ) T y + c 2 ,k 

Revised: ( y + c 1 ,k 1 ) T y + c 2 ,k

Original: + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 = y
Revised: + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 = y

Original: Tc 1 ,k 1 y c 2 ,k 2 .
Revised: Tc 1 ,k 1 y c 2 ,k 2 .

Original: Third, the expectation of this similarity measure across the data set captures the concentration of similarity for that set.
Revised: Third, the expectation of this similarity measure across the data set captures the concentration of similarity for that set.

Original: Therefore, it provides the possibility to deﬁne a formal notion to quantify the discriminative properties.
Revised: Therefore, it provides the possibility to deﬁne a formal notion to quantify the discriminative properties.

Original: Forth, the measure ( y + c 1 ,k 1 ) T y + c 2 ,k 2 +( y − c 1 ,k 1 ) T y − c 2 ,k 2 is not ambiguous w.r.t.
Revised: Forth, the measure ( y + c 1 ,k 1 ) T y + c 2 ,k 2 +( y − c 1 ,k 1 ) T y − c 2 ,k 2 is not ambiguous w.r.t.

Original: a notion for similarity/dissimilarity between two sparse representations y c 1 ,k 1 and y c 2 ,k 2 .
Revised: a notion for similarity/dissimilarity between two sparse representations y c 1 ,k 1 and y c 2 ,k 2 .

Original: This is because the support 
Revised: This is because the support intersections for the positive and the negative components ( y + c 1 ,k 1 ) T y + c 2 ,k 2 = (cid:107) ( y + c 1 ,k 1 ) (cid:12) y + c 2 ,k 2 (cid:107)and ( y − c 1 ,k 1 )

Original: [Equation] where y {+− c 2 ,k 
+}
Revised: T y {+− c 2 ,k 
+}

Original: [Equation] max( y intersections for the positive and the negative components ( y + c 1 ,k 1 ) T y + c 2 ,k 2 = (cid:107) ( y + c 1 ,k 1 ) (cid:12) y + c 2 ,k 2 (cid:107)and ( y − c 1 ,k 1 ) T y − c 2 ,k 2 = (cid:107) ( y + c 1 ,k 1 ) T y + c 2 ,k 2 (cid:107) 1 , respectively, are considered separately.
Revised: = (cid:107) ( y + c 1 ,k 1 ) T y + c 2 ,k 2 (cid:107) 1 , respectively, are considered separately.

Original: In addition a measure for the strength on the support intersection, deﬁned as (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 22 is also taken into account.
Revised: In addition a measure for the strength on the support intersection, deﬁned as (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 2 2 is also taken into account.

Original: The expectation of the measure (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 22 across a data set captures the expected strength on the support intersection for that set.
Revised: The expectation of the measure (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 22 across a data set captures the expected strength on the support intersection for that set.

Original: A schematic diagram of the transform and the main idea behind the proposed concept are shown in Figure 1  .
Revised: A schematic diagram of the transform and the main idea behind the proposed concept are shown in Figure 1 , a ) , b ) and c ) .

Original: On Figure 2 d), e) and f) are given illustrative examples for the support intersection between the positive and negative component of two vectors y c 1 ,k 1 and y c 2 ,k 2 in the transform domain.
Revised: On Figure 1 d), e) and f) are given illustrative examples for the support intersection between the positive and negative component of two vectors y c 1 ,k 1 and y c 2 ,k 2 in the transform domain.

Original: We propose a learning algorithm considering the non-linear transform models with discriminative and minimum information priors.
Revised: We propose a learning algorithm considering the non-linear transform models with discriminative and minimum information priors.

Original: In a direction of quantifying the discriminative properties we introduce a measure named as discrimination power, that reﬂects the discriminate properties of the representation for a data set.
Revised: In a direction of quantifying the discriminative properties we introduce a measure named as discrimination power, that reﬂects the discriminate properties of the representation for a data set.

Original: In addition we present lower and upper bound on the discrimination power in the transform domain, that depends on the parameters of the transform.
Revised: In addition we present lower and upper bound on the discrimination power in the transform domain, that depends on the parameters of the transform.

Original: On the practical side, the advantages and the potential of the proposed algorithm are demonstrated by a numerical experiments using the Extended YALE B (Georghiades et al., 2001), AR (Mart ´ ınez & Benavente, 1998), Norb (LeCun et al., 2004), Coil- 20 (Nene et al., 1996), Clatech 101 (LeCun et al., 2008), UKB (Nist ´ er & Stew ´ enius, 2006) and MNIST (Lecun & Cortes) data sets.
Revised: On the practical side, the advantages and the potential of the proposed algorithm are demonstrated by a numerical experiments using the Extended YALE B (Georghiades et al., 2001), AR (Mart´ınez & Benavente, 1998), Norb (LeCun et al., 2004), Coil- 20 (Nene et al., 1996), Clatech 101 (LeCun et al., 2008), UKB (Nist´er & Stew´enius, 2006) and MNIST (Lecun & Cortes) data sets.

Revised: 3 y c 1 ,k 1 = y + c 1 ,k 1 − y − c 1 ,k 1 where y + c 1 ,k 1 = max( y c 1 ,k 1 , 0 ) and y − c 1 ,k 1 = max( − y c 1 ,k 1 , 0 ) .

Original: 1. N OTATIONS
Revised: 1. N OTATIONS

Original: A scalar variable is denoted using x , a vector is denoted by a bold, low caps symbols x , a matrix by bold, upper cap symbol A .
Revised: A scalar variable is denoted using x , a vector is denoted by a bold, low caps symbols x , a matrix by bold, upper cap symbol A .

Original: A single element from a vector (or matrix) is denoted as x ( n ) (or
Revised: A single element from a vector (or matrix) is denoted as x ( n ) (or

Original: A ( m, n ) ).
Revised: A ( m, n ) ).

Original: A set is denoted by a calligraphic symbol, i.e , S .
Revised: A set is denoted by a calligraphic symbol, i.e , S .

Original: The (cid:96) p − norm is denoted as (cid:107) .
Revised: The (cid:96) p − norm is denoted as (cid:107) .

Original: (cid:107) p and the nuclear norm as (cid:107) .
Revised: (cid:107) p and the nuclear norm as (cid:107) .

Original: (cid:107) ∗ .
Revised: (cid:107) ∗ .

Original: The (cid:12) symbol represents the Hadamard product.
Revised: The (cid:12) symbol represents the Hadamard product.

Original: Throughout the paper it is assumed that a set of data samples X =
Revised: Throughout the paper it is assumed that a set of data samples X =

Original: [ X 1 , X 2 , ..., X C ] ∈ (cid:60) N × L , L = CK from C classes is given and that every class c ∈ C
Revised: [ X 1 , X 2 , ..., X C ] ∈ (cid:60) N × L , L = CK from C classes is given and that every class c ∈ C

Original: = { 1 , 2 , 3 , ..., C } has K samples, X c
Revised: = { 1 , 2 , 3 , ..., C } has K samples, X c

Original: =
Revised: =

Original: [ x c, 1 , x c, 2 , ..., x c,K ] ∈ (cid:60) N × K , x c,k ∈ (cid:60) N , ∀ c ∈ C , ∀ k ∈ K
Revised: [ x c, 1 , x c, 2 , ..., x c,K ] ∈ (cid:60) N × K , x c,k ∈ (cid:60) N , ∀ c ∈ C , ∀ k ∈ K

Original: = { 1 , 2 , ..., K } .
Revised: = { 1 , 2 , ..., K } .

Original: We denote the transform data as Y =
Revised: We denote the transform data as Y =

Original: [ Y 1 , Y 2 , ..., Y C ] ∈ (cid:60) M × L , where Y c =
Revised: [ Y 1 , Y 2 , ..., Y C ] ∈ (cid:60) M × L , where Y c =

Original: [ y c, 1 , y c, 2 , ..., y c,K ] ∈ (cid:60) M × K and y c,k ∈ (cid:60) M .
Revised: [ y c, 1 , y c, 2 , ..., y c,K ] ∈ (cid:60) M × K and y c,k ∈ (cid:60) M .

Original: We denote X \{ k ∈ c } = (cid:2)
Revised: We denote X \{ k ∈ c } = (cid:2)

Original: X 1 , X 2 , ..
Revised: X 1 , X 2 , ..

Original: X c, \ k , ...
Revised: X c, \ k , ...

Original: X C (cid:3) ∈ (cid:60) N × ( L − 1) as the matrix that has all the columns of X , except the column x c,k ∈ (cid:60) N , where X c, \ k =
Revised: X C (cid:3) ∈ (cid:60) N × ( L − 1) as the matrix that has all the columns of X , except the column x c,k ∈ (cid:60) N , where X c, \ k =

Original: [ x c, 1 , x c, 2 , ..., x c,k − 1 , x c,k +1 , ..., x c,K ] ∈ (cid:60) N × ( K − 1) is a matrix that has all the columns of block X c , except the column x c,k , ∀ c ∈ C and ∀ k ∈ K .
Revised: [ x c, 1 , x c, 2 , ..., x c,k − 1 , x c,k +1 , ..., x c,K ] ∈ (cid:60) N × ( K − 1) is a matrix that has all the columns of block X c , except the column x c,k , ∀ c ∈ C and ∀ k ∈ K .

Original: We let N = { 1 , 2 , ..., N } and M = { 1 , 2 , ..., M } .
Revised: We let N = { 1 , 2 , ..., N } and M = { 1 , 2 , ..., M } .

Original:  L EARNING NON - LINEAR TRANSFORM WITH DISCRIMINATIVE AND MINIMUM INFORMATION LOSS PRIORS
Revised:  L EARNING NON - LINEAR TRANSFORM WITH DISCRIMINATIVE AND MINIMUM INFORMATION LOSS PRIORS

Original: It is assumed that training data X =
Revised: It is assumed that training data X =

Original: [ X 1 , X 2 , ..., X C ] ∈ (cid:60) N × CK for C classes are given.
Revised: [ X 1 , X 2 , ..., X C ] ∈ (cid:60) N × CK for C classes are given.

Original: Every class c ∈ C has K training samples, i.e. , X c
Revised: Every class c ∈ C has K training samples, i.e. , X c

Original: =
Revised: =

Original: [ x c, 1 , x c, 2 , ..., x c,K ] .
Revised: [ x c, 1 , x c, 2 , ..., x c,K ] .

Original: The proposed approach considers two different models.
Revised: The proposed approach considers two different models.

Original: Moreover, a predeﬁned number of non-linear transforms and one sparsifying transform are used for the learning stage and the testing stage.
Revised: Moreover, a predeﬁned number of non-linear transforms and one sparsifying transform are used for the learning stage and the testing stage.

Original: 2. T HE PARAMETRIC NON - LINEAR TRANSFORM MODELING
Revised: 2. T HE PARAMETRIC NON - LINEAR TRANSFORM MODELING

Original: More speciﬁcally, considering the learning stage we assume that for every class c ∈ C there exist one non-linear transform deﬁned by a set of parameters
Revised: More speciﬁcally, considering the learning stage we assume that for every class c ∈ C there exist one non-linear transform deﬁned by a set of parameters

Original: P c = { A ∈ (cid:60) M × N , τ c ∈
Revised: P c = { A ∈ (cid:60) M × N , τ c ∈

Original: (cid:60) M } , ∀ c ∈ C 4 .
Revised: (cid:60) M } , ∀ c ∈ C 4 .

Original: The linear map A is shared for all of the C different models, but, the parameters τ c for the different classes c from the corresponding non-linear transforms P c are different.
Revised: The linear map A is shared for all of the C different models, but, the parameters τ c for the different classes c from the corresponding non-linear transforms P c are different.

Original: Furthermore, we assume that when all the non-linear transforms are applayed to the corresponding class samples then the transform data samples are separable w.r.t.
Revised: Furthermore, we assume that when all the non-linear transforms are applayed to the corresponding class samples then the transform data samples are separable w.r.t.

Original: the different classes, in the transform domain.
Revised: the different classes, in the transform domain.

Original: As far as the non-linear transform, we focus on transforms expressible by a two steps operation, consisting of a linear mapping (step 1 ) followed by an element-wise non-linearity (step 2 ):
Revised: As far as the non-linear transform, we focus on transforms expressible by a two steps operation, consisting of a linear mapping (step 1 ) followed by an element-wise non-linearity (step 2 ):

Original: [Equation]
Revised: [Equation]

Original:  where ∀ l ∈ (cid:60) M , H τ c ( l ) = sign( l ) (cid:12) max( | l | − τ c , 0 ) :(cid:60) M
Revised: (1) where ∀ w c,k = Ax c,k ∈ (cid:60) M , H τ c ( w c,k ) = sign( w c,k ) (cid:12) max( | w c,k | − τ c , 0 ) :

Original:  → (cid:60) M , represents a non-linear thresholding function with parameters τ c ∈ (cid:60) M .
Revised: (cid:60) M → (cid:60) M , represents a non-linear thresholding function with parameters τ c ∈ (cid:60) M .

Original: We also have to mention that the thresholding is done with different thresholding parameters τ c ( m ) for the corresponding different transform dimensions m ∈ M .
Revised: We also have to mention that the thresholding is done with different thresholding parameters τ c ( m ) for the corresponding different transform dimensions m ∈ M .

Original: Considering the testing stage we use the sparsifying transform deﬁned by a set of parameters
Revised: Considering the testing stage we use the sparsifying transform deﬁned by a set of parameters

Original: P = { A ∈ (cid:60) M × N , τ 1 ∈ (cid:60) M } .
Revised: P = { A ∈ (cid:60) M × N , τ 1 ∈ (cid:60) M } .

Original: In the following section we describe the non-linear transform model, the proposed discriminative prior and its mesure, together with the main reason behind this particular use of the non-linear transform models for learning and a sparsifying transform model for testing.
Revised: In the following section we describe the non-linear transform model, the proposed discriminative prior and its mesure, together with the main reason behind this particular use of the non-linear transform models for learning and a sparsifying transform model for testing.

Original: 2. T HE NON - LINEAR TRANSFORM MODEL WITH A DISCRIMINATIVE PRIOR
Revised: 2. T HE NON - LINEAR TRANSFORM MODEL WITH A DISCRIMINATIVE PRIOR

Original: The learning model This paper deﬁnes a compact description of the non-linear transform (1) by a non-linear transform model as follows:
Revised: The learning model This paper deﬁnes a compact description of the non-linear transform (1) by a non-linear transform model as follows:

Original: [Equation]
Revised: [Equation]

Original: where T P c ( . ) :
Revised: where T P c ( . ) :

Original: (cid:60) N
Revised: (cid:60) N

Original: →
Revised: →

Original: (cid:60) M is a parametric non-linear function that gives y c,k , by using the set of parameters P c .
Revised: (cid:60) M is a parametric non-linear function that gives y c,k , by using the set of parameters P c .

Original: The term z c,k
Revised: The term z c,k

Original: = Ax c,k − y c,k is the non-linear transform error vector that represents the deviation of Ax c,k from the targeted transform representation y c,k = T P c ( x c,k ) in the transform domain.
Revised: = Ax c,k − y c,k is the non-linear transform error vector that represents the deviation of Ax c,k from the targeted transform representation y c,k = T P c ( x c,k ) in the transform domain.

Original: Since the transform representation y c,k = T P c ( x c,k ) takes into acount a non-linearity, Ax c,k is only seen as its linear approximation.
Revised: Since the transform representation y c,k = T P c ( x c,k ) takes into acount a non-linearity, Ax c,k is only seen as its linear approximation.

Original: Knowing something in advance about the difference between y c,k − Ax c,k can be used in our model.
Revised: Knowing something in advance about the difference between y c,k − Ax c,k can be used in our model.

Original: However, since in advance we do not have any prior we assume that it is Gausssian like distributed.
Revised: However, since in advance we do not have any prior we assume that it is Gausssian like distributed.

Original: Therefore, the prior on z c,k is modeled as p ( x c,k | y c,k , A ) ∝ exp( − (cid:107)
Revised: Therefore, the prior on z c,k is modeled as p ( x c,k | y c,k , A ) ∝ exp( − (cid:107)

Original: [Equation]
Revised: Ax c,k − y c,k (cid:107) 2 β) .

Original: Assuming additionally that the non-linear function T P c ( x c,k ) gives sparse y c,k , then we have the improper prior on y c,k , deﬁned as p ( y c,k )
Revised: Assuming additionally that the non-linear function T P c ( x c,k ) gives sparse y c,k , then we have the improper prior on y c,k , deﬁned as p ( y c,k )

Original: ∝ exp( − (cid:107) y c,k (cid:107) 1 β) .
Revised: ∝ exp( − (cid:107) y c,k (cid:107) 1 β) .

Original: This paper assumes that the joint probability p ( τ 1 , τ 2 , ...., τ C , y c,k ) has the form
Revised: This paper assumes that the joint probability p ( τ 1 , τ 2 , ...., τ C , y c,k ) has the form

Original: [Equation]
Revised: [Equation]

Original: and that p ( τ 1 , τ 2 , ...., τ C ) = (cid:81) c p ( τ c ) .
Revised: and that p ( τ 1 , τ 2 , ...., τ C ) = (cid:81) c p ( τ c ) .

Original: Given the class label c we model the discriminative prior as p ( τ c , | y c,k ) ∝ exp( − D ( y c,k ; τ c ) β) by using a parametric measure D ( y c,k ; τ c ) with parameters τ c .
Revised: Given the class label c we model the discriminative prior as p ( τ c , | y c,k ) ∝ exp( − D ( y c,k ; τ c ) β) by using a parametric measure D ( y c,k ; τ c ) with parameters τ c .

Original: Furthermore, assuming that the measure D ( y c,k ; τ c ) is deﬁned on the support intersection for the positive and the negative components y + c,k , τ + c and y − c,k , τ − c , respectively, of y c,k
Revised: Furthermore, assuming that the measure D ( y c,k ; τ c ) is deﬁned on the support intersection for the positive and the negative components y + c,k , τ + c and y − c,k , τ − c , respectively, of y c,k

Original: and τ c we propose the following formulation:
Revised: and τ c we propose the following formulation:

Original: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where (cid:107) y
Revised: 4) where (cid:107) y

Original: + c,k (cid:12)
Revised: + c,k (cid:12)

Original: τ
Revised: τ

Original: + c (cid:107) 1 + (cid:107) y − c,k (cid:12) τ − c (cid:107) 1 measures the similarity contribution on the support intersection using the positive and the negative components, y + c,k , τ + c and y − c,k , τ − c , respectively, of y c,k and τ c and (cid:107) y c,k
Revised: + c (cid:107) 1 + (cid:107) y − c,k (cid:12) τ − c (cid:107) 1 measures the similarity contribution on the support intersection using the positive and the negative components, y + c,k , τ + c and y − c,k , τ − c , respectively, of y c,k and τ c and (cid:107) y c,k

Original: (cid:12)
Revised: (cid:12)

Original: τ c (cid:107) 22 measures the strength of the support intersection between y c,k and τ c .
Revised: τ c (cid:107) 22 measures the strength of the support intersection between y c,k and τ c .

Original: We highlight that the true p ( τ c ) , τ c , are not known and the only prior about D ( y c,k ; τ c ) and p ( τ c ) is that they should be informative enough w.r.t.
Revised: The true p ( τ c ) , τ c , are not known and the only prior about D ( y c,k ; τ c ) and p ( τ c ) is that they should be informative enough w.r.t.

Original: the discriminative properties of the transform representation y c,k .
Revised: the discriminative properties of the transform representation y c,k .

Original: Furthermore, instead of estimating them explicitly, an approximation to D ( y c,k ; τ c ) is considered, based only on the concentrations of the similarity on the support intersection and the expected strength of the support intersection for the transform data in the transform domain.
Revised: Furthermore, instead of estimating them explicitly, an approximation to D ( y c,k ; τ c ) is considered, based only on the concentrations of the similarity on the support intersection and the expected strength of the support intersection for the transform data in the transform domain.

Original: The approximation We propose an approximation that captures two expectations.
Revised: The approximation We propose an approximation that captures two expectations.

Original: The ﬁrst one is the expected similarity on the support intersection for the positive and negative component between all y c,k and the coresponding sets of the transform representations Y \ c that come from all classes c 1 different from c , i.e., c (cid:54) =
Revised: The ﬁrst one is the expected similarity on the support intersection for the positive and negative component between all y c,k and the coresponding sets of the transform representations Y \ c that come from all classes c 1 different from c , i.e., c (cid:54) =

Original: c 1 .
Revised: c 1 .

Original: The second is the expected strength on the support intersection between y c,k and the set of transform representations Y \ c that come from all classes c 1 different from c , c (cid:54) =
Revised: The second is the expected strength on the support intersection between y c,k and the set of transform representations Y \ c that come from all classes c 1 different from c , c (cid:54) =

Original: c 1 .
Revised: c 1 .

Original: The approximation is deﬁned as:
Revised: The approximation is deﬁned as:

Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Revised: where we highlight that the transform represntations y c,k used in the approximation D P (cid:96) 1 ( X ) and S P (cid:96) 2 ( X ) are the result of applying the sparsifying transform with parameter set P to the data samples x c,k .

Original: where the m -th element y + c,k ( m ) of y + c,k is deﬁned as y + c,k ( m ) =
Revised: The m -th element y + c,k ( m ) of y + c,k is deﬁned as y + c,k ( m ) =

Original: max( y c,k ( m ) , 0) and similarly, y − c,k ( m ) = max( − y c,k ( m ) , 0) , ∀ m ∈ M5 .
Revised: max( y c,k ( m ) , 0) and similarly, y − c,k ( m ) = max( − y c,k ( m ) , 0) , ∀ m ∈ M .

Original: We also deﬁne the expected similarity using the positive and negative components of  y c,k for theset of transform representations Y \ c that come from the same classes c as D P 
Revised: We also deﬁne the expected similarity using the positive and negative components of all y c,k across the transform representations Y \ c that come from the same classes c as D P (cid:96) 1 ,c ( X ) =

Original: [Equation]
Revised: [Equation]

Original:  y + c,k 1 (cid:107) 1 +(cid:80) k 1 k 1 (cid:54) = k (cid:107) y − c,k (cid:12) y − c,k 1 (cid:107)(cid:19) . 
Revised: ...,K }\ k } (cid:16) (cid:107) y + c,k (cid:12) y + c,k 1 (cid:107) 1 + (cid:107) y − c,k (cid:12) y − c,k 1 (cid:107) 1 (cid:17) . If the measure D P (cid:96) 1 ( X ) is

Revised: By (5) a link is established between the non-linear transform models and the sparsifying transform model, or more generaly, assuming A is known, between a parametric and a non-parametric modeling view.

Original: The terms 2 (( C 1) K )( CK ) (cid:80) c 1 ,c 1 (cid:54) =
Revised: The terms [Equation] [Equation] [Equation]

Original: c (cid:80) k 1 y − c 1 ,k 1 and 2 (( C 1) K )( CK ) (cid:80) c 1 ,c 1 (cid:54) =
Revised: ,C [Equation] (cid:80) [Equation]

Original: c (cid:80) k 1 y + c1 ,k 1 , might be seen as ﬁnite sample estimates 6 of the positive and negative component τ − c and τ + c , respectively, for the unknown variable τ c , ∀ c ∈ C . The term
Revised: = [Equation] y + c

Original: [Equation]
Revised: [Equation]

Original: ∈ C is also seen asa ﬁnite sample estimate 7 of the  Hadamard square τ c (cid:12) τ c for the unknown variable τ c , ∀ c ∈ C .If the measure D P (cid:96) 1 ( X ) is not used then the approximation (5) is most similar to the one proposed in (Liu et al., 2016).
Revised: ,k 1 and 2 (( C 1) K )( CK ) (cid:80) c 1 ,c 1 (cid:54) = c (cid:80) k 1 y c 1 ,k 1 (cid:12) y c 1 ,k 1 are seen as ﬁnite sample estimates 5 6 of the positive τ − c , negative τ + c components and the the Hadamard square τ c (cid:12) τ c for the unknown variable τ c , ∀ c ∈ C .

Original: Note that the Fisher discriminate constraint (Yang et al., 2011b), the pairwise constraint (Guo et al., 2013) and the support intersection constraint (Liu et al., 2016) are all approximations of a discriminative prior.
Revised: Note that the Fisher discriminate constraint (Yang et al., 2011b), the pairwise constraint (Guo et al., 2013) and the support intersection constraint (Liu et al., 2016) are all approximations of a discriminative prior.

Original: However,  all of them are with speciﬁc assumptions on the distribution of the data representation in the transform domain.
Revised: However, they all have a speciﬁc assumptions on the distribution of the data representation in the transform domain.

Original: The advantage of using (5) is that the aproximation is without any prior on the probability distributions p ( τ c ) and without any explicit assumption about the distance metric/measure, or the space/manifold in the transform domain.
Revised: The advantage of using (5) is that the aproximation is without any prior on the probability distributions p ( τ c ) and without any explicit assumption about the distance metric/measure, or the space/manifold in the transform domain.

Original: Given a training set X , the learning prespective w.r.t.
Revised: Given a training set X , the learning prespective w.r.t.

Original: the discriminative properties is to estimate a non-linear transform models P c =
Revised: the discriminative properties is to estimate a non-linear transform models P c =

Original: { A ∈ (cid:60) M × N , τ c ∈ (cid:60) M } that minimize the empirical expectation E [min 1 ≤ c ≤ C D ( y c,k ; τ c )] ∼ (cid:80) c,k min 1 ≤ c ≤ C D ( y c,k ; τ c ) .
Revised: { A ∈ (cid:60) M × N , τ c ∈ (cid:60) M } that minimize the empirical expectation E [min 1 ≤ c ≤ C D ( y c,k ; τ c )] ∼ (cid:80) c,k min 1 ≤ c ≤ C D ( y c,k ; τ c ) .

Original: Moreover, if the corresponding class labels for the training set X are given 8 then E [min 1 ≤ c ≤ C D ( y c,k ; τ c )] equals to (cid:80) c,k D ( y c,k ; τ c ) and exactly mathces to the similarity concentration D P (cid:96) 

Revised: Moreover, if the corresponding class labels for the training set X are given 7 then E [min 1 ≤ c ≤ C D ( y c,k ; τ c )] equals to (cid:80) c,k D ( y c,k ; τ c ) and exactly mathces to the similarity concentration D P (cid:96)

Original: ( X ) + S P
Revised: ( X ) + S P

Original: (cid:96) 2 ( X ) , i.e. , (cid:80) [Equation]
Revised: (cid:96) 2 ( X ) .

Original: The testing model Assume that the transform matrix A and the parameters τ 1 , τ 2 , ..., τ C are known, then given any data sample x c,k the transform representaton y c,k is estimated as:
Revised: The testing model Assume that the transform matrix A and the parameters τ 1 , τ 2 , ..., τ C are known, then given any data sample x c,k the transform representaton y c,k is estimated as:

Original: [Equation]
Revised: [Equation]

Original: that
Revised: which is euqialent to min 1 ≤ c 1 ≤ C (cid:0) min y c,k (cid:107)

Original: [Equation]{+Ax c,k − y c,k (cid:107) 
+}
Revised: {+Ax c,k − y c,k (cid:107) 
+}

Original: whereas
Revised: + λ

Original: [Equation]
Revised: [Equation]

Original: Since
Revised: D ( y c

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [-[Table] [Table] which is euqialent to min 1 ≤ c 1 ≤ C (cid:0) min y c,k (cid:107) Ax c,k − y c,k (cid:107) 
-]
Revised: [-[Table] [Table] which is euqialent to min 1 ≤ c 1 ≤ C (cid:0) min y c,k (cid:107) Ax c,k − y c,k (cid:107) 
-]k

Original: +
Revised: [Equation]

Original: λ 0 D ( y c,k ; τ c 1 ) + λ 1 (cid:107) y c,k (cid:107) 1 (cid:1) .
Revised: τ [Equation] 1 ) + λ 1 (cid:107) y c,k (cid:107) 1 (cid:1) .

Original: Furtheremore, if we assume that the measures D ( y c,k ; τ c 1 ) are zero, i.e. , D ( y c,k ; τ c 1 ) = 0 , then the discriminative prior is non-informative, in a sence that it has no inﬂuence in the models.
Revised: Furtheremore, if we assume that the measures D ( y c,k ; τ c 1 ) are zero, i.e. , D ( y c,k ; τ c 1 ) = 0 , then the discriminative prior is non-informative, in a sence that it has no inﬂuence in the models.

Original: Only then do the non-linear transforms reduce to the sparsifying transform model, since (6) reduce to:
Revised: Only then do the non-linear transforms reduce to the sparsifying transform model, since (6) reduce to:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Considering the testing stage, we note that this result spreads light on another view.
Revised: Considering the testing stage, we note that the result (5) sheds light on another view.

Original: Naemly, the sparsifying transform model P = { A ∈ (cid:60) M × N , λ 1 ∈ (cid:60) M } is also seen as an approximation to the models represented by a set of parameters P c
Revised: Naemly, the sparsifying transform model P = { A ∈ (cid:60) M × N , λ 1 ∈ (cid:60) M } is also seen as an approximation to the models represented by a set of parameters P c

Original: =
Revised: =

Original: { A ∈ (cid:60) M × N , τ c ∈ (cid:60) M } with expected loss in the discriminative properties of the transform representations expressed by the similarity concentration measure D P (cid:96) 1 ( X ) + S P (cid:96) 2 ( X ) .
Revised: { A ∈ (cid:60) M × N , τ c ∈ (cid:60) M } with expected loss in the discriminative properties of the transform representations expressed by the similarity concentration measure D P (cid:96) 1 ( X ) + S P (cid:96) 2 ( X ) .

Original: In the same time this measure can also be considered as an empirical risk (Vapnik, 1995) w.r.t.
Revised: In the same time this measure can also be considered as an empirical risk (Vapnik, 1995) w.r.t.

Original: the discriminative properties, related to the generalization capabilities (Vapnik, 1995) and (Mark, 2010) of the model.
Revised: the discriminative properties, related to the generalization capabilities (Vapnik, 1995) and (Mark, 2010) of the model.

Original: Note that the sparsifying transform model is also the simplest that approximates the non-linear transform models used in the learning stage.
Revised: Note that the sparsifying transform model is also the simplest that approximates the non-linear transform models used in the learning stage.

Original: 2. T HE LEARNING ALGORITHM
Revised: 2. T HE LEARNING ALGORITHM

Original: In summary, the used priors are:
Revised: In summary, the used priors are:

Original: [Equation]
Revised: [Equation]

Original:  Additionally, we have a prior on A that penalizes the information loss in order to avoid trivially unwanted matrices A , i.e. , matrices that have repeated or zero rows.
Revised: 8) 5 Since [Equation] Additionally, we have a prior on A that penalizes the information loss in order to avoid trivially unwanted matrices A , i.e. , matrices that have repeated or zero rows.

Original: The corresponding prior is deﬁned as
Revised: The corresponding prior is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where the (cid:107) A (cid:107) F penalty helps regularize the scale ambiguity, the log | det ( A T A ) | and (cid:107) A (cid:107) 2 F are functions of the singular values of A and together help regularize the conditioning of A .
Revised: where the (cid:107) A (cid:107) F penalty helps regularize the scale ambiguity, the log | det ( A T A ) | and (cid:107) A (cid:107) 2 F are functions of the singular values of A and together help regularize the conditioning of A .

Original: Assuming that the expected coherence µ 2 ( A ) between the rows a m of A ( i.e. , A T =
Revised: Assuming that the expected coherence µ 2 ( A ) between the rows a m of A ( i.e. , A T =

Original: [ a 1 , a 2 , ..., a M ] ) is deﬁned as µ 2 ( A )=
Revised: [ a 1 , a 2 , ..., a M ] ) is deﬁned as µ 2 ( A )

Original: [Equation]
Revised: [Equation]

Original: ∈ {

Original: [Equation]

Original: [Equation]

Original: , .., M } .
Revised: .., M } .

Original: Then (cid:107) AA T − I (cid:107) 2 F measures the expected coherence µ 2 ( A ) and the (cid:96) 2 norm for the rows of A .
Revised: Then (cid:107) AA T − I (cid:107) 2 F measures the expected coherence µ 2 ( A ) and the (cid:96) 2 norm for the rows of A .

Original: Note that the joint probability can be expressed as:
Revised: Note that the joint probability can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: since p ( x c,k | y c,k , τ c , A ) =
Revised: where p ( x c,k , y c,k , τ c | A ) =

Original: p ( x c,k | y c,k , A )  .
Revised: p ( x c,k | y c,k , A ) p ( τ c , | y c,k ) p ( y c,k ) , since p ( x c,k | y c,k , τ c , A ) = p ( x c,k | y c,k , A ) .

Original: Given the available training data set X , maximizing p ( x c,k , y c,k , τ c , A ) over Y and A is same as minimizing the following problem:
Revised: Given the available training data set X , maximizing p ( x c,k , y c,k , τ c , A ) over Y and A is same as minimizing the following problem:

Original: [Equation]
Revised: [Equation]

Original: where { λ 0 , λ 1, λ 2 , λ 3 , λ 4 } are inversely proportional to the scaling parameters { β 2 , β 1, β 3 , β 4 , β 5 } .
Revised: where { λ 0 , λ 1 } are inversely proportional to the scaling parameters { β 2 , β 1 } .

Original: Note that the solution to (11) is not equivalent to the maximum a priory (MAP) solution, which is difﬁcult to compute, as it involves integrating over the vectors y c,k .
Revised: Note that the solution to (11) is not equivalent to the maximum a priory (MAP) solution, which is difﬁcult to compute, as it involves integrating over the vectors y c,k .

Original: Considering the optimization perspective, the problem is not convex in the variables ( Y , A ) .
Revised: Considering the optimization perspective, the problem is not convex in the variables ( Y , A ) .

Original: The proposed solution here is obtained by iteratively, marginally maximizing the probability p ( x c,k , y c,k , τ c , A ) over Y and A which is Input X , λ 0 , λ 1 , λ 2 , λ 3 , λ A ← inicialize repeat DISCRIMINATIVE ENCODING closed form solution per data sample
Revised: The proposed solution here is obtained by iteratively, marginally maximizing the probability p ( x c,k , y c,k , τ c , A ) over Y and A which is equivalent to maximizing the conditional densities p ( y c,k | x c,k , τ c , A ) and p ( A | x c,k , y c,k , τ c ) , respectively.

Original: Y ← AX repeatfor ∀ c ∈ C do d − c ← (cid:80) c 1 c 1 (cid:54) =

Original: c (cid:80) k 1 y − c 1 ,k 1 , d + c ← (cid:80) c 1 c 1 (cid:54) =

Original: c (cid:80) k 1 y + c 1 ,k 1 and s c ← (cid:80) c 1 c 1 (cid:54) =

Original: c (cid:80) k 1 y c 1 ,k 1 (cid:12) y c 1 ,kend for for ∀ c ∈ C and ∀ k ∈ K do g ← sign(max( Ax c,k , 0 ))

Original: (cid:12) d

Original: + c + sign(max( − Ax c,k , 0 ))

Original: (cid:12) d − c y c,k ← sign( Ax c,k ) (cid:12) max ( Ax c,k − λ 0 g + λ 1 1 , 0 ) (cid:11) (1 + 2 λ 0 s c ) end for until convergence

Original: TRANSFORM UPDATE (cid:15) -close closed form solution

Original: [Equation]

Original: where σ Γ ( n ) ← T ( n, n

Original: [Equation]

Original: , T ← U

Original: U X

Original: XY Σ

Original: U X XY U TU X XY ,

Original: [Equation]

Original: N

Original: A ←

Original: [Equation]

Original: X

Original: [Equation]

Original: U TU X

Original: [Equation]

Original: A Σ

Original: [Equation]

Original: X U TX until convergence

Original: Output A , Y equivalent to maximizing the conditional densities p ( y c,k | x c,k , τ c , A ) and p ( A | x c,k , y c,k , τ c ) , respectively.

Original: Meaning that at one iterating step one of the variables Y or A is ﬁxed and w.r.t.
Revised: Meaning that at one iterating step one of the variables Y or A is ﬁxed and w.r.t.

Original: the other the problem (11) is minimized.
Revised: the other the problem (11) is minimized.

Original: The following describes the iterating steps that consist of linear map estimation (maximizing p ( A | x c,k , y c,k , τ c ) ) and discriminative encoding (maximizing p ( y c,k | x c,k , τ c , A ) ).
Revised: The following describes the iterating steps that consist of linear map estimation (maximizing p ( A | x c,k , y c,k , τ c ) ) and discriminative encoding (maximizing p ( y c,k | x c,k , τ c , A ) ).

Original: Linear map estimation :
Revised: Linear map estimation :

Original: Given the available data samples X and the corresponding transform representations Y the linear map A estimation problem reduces to:
Revised: Given the available data samples X and the corresponding transform representations Y the linear map A estimation problem reduces to:

Original: [Equation]
Revised: [Equation]

Original:  and we use the (cid:15) -close closed form solution estimated as follows:
Revised: [Equation] where { λ 2 , λ 3 , λ 4 } are inversely proportional to the scaling parameters { β 3 , β 4 , β 5 } and we use the (cid:15) -close closed form solution estimated as follows:

Original: Proposition 1 ( (cid:15) -close closed form solution ): Given Y ∈ (cid:60) M × CK , ∀ X ∈ (cid:60) N × CK and M ≥ N , ∀ λ 2 ≥ 0 , λ 3 ≥ 0 and λ 4 ≥ 0 let the eigen value decomposition U X Σ X V TX of XX T +
Revised: Proposition 1 ( (cid:15) -close closed form solution ): Given Y ∈ (cid:60) M × CK , ∀ X ∈ (cid:60) N × CK and M ≥ N , ∀ λ 2 ≥ 0 , λ 3 ≥ 0 and λ 4 ≥ 0 let the eigen value decomposition U X Σ X V TX of XX T +

Original: λ 2 I and the singular value decomposition
Revised: λ 2 I and the singular value decomposition

Original: U
Revised: U

Original: U X
Revised: U X

Original: XY Σ U X XY V TU X XY of U TX XY T exist, then if and only if σ X ( n ) > 0 , ∀ n ∈ N = { 1 , 2 , 3 , ..., N } , (12) has (cid:15) -close approximative solution as :
Revised: XY Σ U X XY V TU X XY of U TX XY T exist, then if and only if σ X ( n ) > 0 , ∀ n ∈ N = { 1 , 2 , 3 , ..., N } , (12) has (cid:15) -close approximative solution as :

Original: [Equation]
Revised: [Equation]

Original: (13) where Σ A is diagonal matrix, Σ A ( n, n )
Revised: where Σ A is diagonal matrix, Σ A ( n, n )

Original: =
Revised: =

Original: σ A ( n ) ≥ 0 , and σ A ( n ) are solutions to 
Revised: σ A ( n ) ≥ 0 , and σ A ( n ) are solutions to min

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  with σ Γ ( n ) =
Revised: 14) with σ Γ ( n ) =

Original: T ( n, n ) , T = U
Revised: T ( n, n ) , T = U

Original: U X
Revised: U X

Original: XY Σ U X XY V TU X XY , ∀ n ∈{ 1 , 2 , ..., N } (we give the proof  in Appendix A.2) .
Revised: XY Σ U X XY V TU X XY , ∀ n ∈ N (the proof is in Appendix A.2) .

Original: Discriminative encoding :
Revised: Discriminative encoding :

Original: Given the available data samples X and the current estimate of the transform A , deﬁnefor simplicity Q = AX ∈ (cid:60) M × L .
Revised: Given the available data samples X and the current estimate of the transform A , deﬁne Q = AX ∈ (cid:60) M × L .

Original: The discriminative representation estimation problem is formulated as P DR : min Y (cid:107) Q − Y (cid:107) 2 F + (cid:80) c,k λ 0 D ( y c,k ; τ c ) + λ 1 (cid:107) y c,k (cid:107) .
Revised: The discriminative representation estimation problem is formulated as P DR : min Y (cid:107) Q − Y (cid:107) 2 F + (cid:80) c,k λ 0 D ( y c,k ; τ c ) + λ 1 (cid:107) y c,k (cid:107) .

Original: Note that[Figure] (11) and consequently ( P DR ) is a well deﬁned problem only if the model variables τ c , or the probability distributions p ( τ c ) are know in advance.
Revised: Note that (11) and consequently ( P DR ) is a well deﬁned problem only if the model variables τ c , or the probability distributions p ( τ c ) are know in advance.

Original: Nevertheless, we show that the approximation (5) leads to an efﬁcient solution of ( P DR ) .
Revised: Nevertheless, we show that the approximation (5) leads to an efﬁcient solution of ( P DR ) .

Original: Assuming that Y \ c is given, then for any sample k ∈ K from any class c ∈ C and the approximation (5), problem ( P DR ) reducess to a constrained projection taking the form of: [Equation] [Figure]
Revised: Assuming that Y \ c is given, then for any sample k ∈ K from any class c ∈ C and the approximation (5), problem ( P DR ) reducess to a constrained projection problem

Original: [Equation]
Revised: [Equation]

Original: (15) and has a closed form solution as:
Revised: 15) and has a closed form solution as:

Original: [Equation]
Revised: [Equation]

Original: (16) where (cid:11) is a Haddamard division, g = sign(max( Ax c,k , 0 ))
Revised: where (cid:11) is a Haddamard division, g = sign(max( Ax c,k , 0 ))

Original: (cid:12) d + +sign(max
Revised: (cid:12) d + +sign(max

Original: ( − Ax c,k , 0 ))
Revised: ( − Ax c,k , 0 ))

Original: (cid:12) d − , 
Revised: (cid:12) d − , d + c = (cid:80) c

Original: [Equation]
Revised: 1 c 1 (cid:54) = c (cid:80) k 1 y + c 1 ,k 1 , d − c

Original: =
Revised: =

Original: [Equation] [Equation] y
Revised: (cid:80) c

Original: [Equation] = (cid:80) c 1 c 1 (cid:54) =
Revised: 1 c 1 (cid:54) =

Original: c (cid:80) k 1 y − c 1 ,k 1 and s c = (cid:80) c
Revised: c (cid:80) k 1 y − c 1 ,k 1 and s c = (cid:80) c

Original: 1 c 1 (cid:54) =
Revised: 1 c 1 (cid:54) =

Original: c (cid:80) k 1 y c 1 ,k 1 (cid:12) y c 1 ,k 1 ( the proof is given in Appendix B ).
Revised: c (cid:80) k 1 y c 1 ,k 1 (cid:12) y c 1 ,k 1 ( the proof is given in Appendix B ).

Original: We note that at convergence (which we do not prove here) we can only claim that a joint local maximum in ( Y , A ) of p ( x c,k , y c,k , τ c , A ) has been reached, even if, as in this case, each optimization step achieves the (marginal) (cid:15) -close and the global optimal solution, respectively.
Revised: We note that at convergence (which we do not prove here) we can only claim that a joint local maximum in ( Y , A ) of p ( x c,k , y c,k , τ c , A ) has been reached, even if, as in this case, each optimization step achieves the (marginal) (cid:15) -close and the global optimal solution, respectively.

Original: The exact steps of the proposed non-linear transform learning are described by Algorithm 1.

Original: 2. A MEASURE FOR THE DISCRIMINATIVE PROPERTIES AND ITS BOUNDS
Revised: 2. A MEASURE FOR THE DISCRIMINATIVE PROPERTIES AND ITS BOUNDS

Original: This paper proposes a notion for the discriminative properties of a data set under a non-linear transform named as discrimination power, based on a measure for the relations between the concentrations D P (cid:96) 1 ,c ( X ) and D P (cid:96) 1 ( X ) .
Revised: This paper proposes a notion for the discriminative properties of a data set under a non-linear transform named as discrimination power, based on a measure for the relations between the concentrations D P (cid:96) 1 ,c ( X ) and D P (cid:96) 1 ( X ) .

Original: Proposition 1 : The discrimination power for any dataset Y ∈ (cid:60) M × CK under a non-linear transform with parameter set P is deﬁned as:
Revised: Proposition 2 : The discrimination power for any dataset Y ∈ (cid:60) M × CK under a non-linear transform with parameter set P is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: (17) where B M = { A t ∈ (cid:60) M × M , 0 ∈ (cid:60) M } and A t = I .
Revised: 17) where B M = { A t ∈ (cid:60) M × M , 0 ∈ (cid:60) M } and A t = I .

Original: Remark 1 : The advantage of this measure is that it logarithmically signiﬁes the difference between
Revised: Remark 1 : The advantage of this measure is that it logarithmically signiﬁes the difference between

Original: D P (cid:96) 1 ,c ( X ) and D P (cid:96) 1 ( X ) 9 .
Revised: D P (cid:96) 1 ,c ( X ) and D P (cid:96) 1 ( X ) 8 .

Original: The deﬁnition about the discrimination power of the original data set X , but, now under a model with a parameter set B N = { A o ∈ (cid:60) N × N , 0 ∈ (cid:60) N } where
Revised: The deﬁnition about the discrimination power of the original data set X , but, now under a model with a parameter set B N = { A o ∈ (cid:60) N × N , 0 ∈ (cid:60) N } where

Original: A o
Revised: A o

Original: =
Revised: =

Original: I is equivalent to the one deﬁned for I t .
Revised: I is equivalent to the one deﬁned for I t .

Original: We denote it as I o .
Revised: We denote it as I o .

Original: An illustration is given by a diagram shown in Figure 3.

Original: The bound on the discrimination power is given by the following result where τ c and υ c are unknown parameters Then the differ- [Table] [Table]
Revised: The bound on the discrimination power is given by the following result.

Original: [Figure]

Original: [Figure]

Original: (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96) 1 ( Y ) and the discrimination power log( C 1 /C 2) =

Original: I t for randomly chosen subsets from all of the used databases under a non-linear transform with transform dimension M = 19000 and varying thresholding parameter τ = λ 1 .

Original: Theorem 1 : The discrimination power for any data set X ∈ (cid:60) N × CK under any non-linear transform with parameter set P is bounded as :
Revised: Theorem 1 : The discrimination power for any data set X ∈ (cid:60) N × CK under any non-linear transform with parameter set P is bounded as :

Original: [Equation]
Revised: [Equation]

Original: The proof is given in Appendix D10 .
Revised: The proof is given in Appendix D

Original: At ﬁrst the resulting bounds might look counterintuitive since the loss of information seems to increase the discrimination power.
Revised: At ﬁrst the resulting bounds might look counterintuitive since the loss of information seems to increase the discrimination power.

Original: This fact is true, however, up to a certain limit.
Revised: This fact is true, however, up to a certain limit.

Original: Therefore, it is important to distinguish two main conclusions.
Revised: Therefore, it is [Equation] [Equation]

Original: C n ( A ) µ ( A ) t e [ min ] D2 .0 .5 . D1 .0 .5 .
Revised: ) where τ c and υ c are unknown parameters. Then the differ-

Original: D2 .0 .6 .
Revised: [Table]

Original: [-D0 .0 .8 . D6 .
 D33 .0 .30 . D1 .0 .5 .-]
Revised: [-D0 .0 .8 . D6 .
 D33 .0 .30 . D1 .0 .5 .-]In addition in Appendix C we also provide a sensitivity analysis that we consider beneﬁcial, since it is related to the notion about the discriminative quality of the representation.

Original: I O 0 .0 .0 .0 .0 .0 .0 . I RT 0 .0 .0 .0 .0 .0 .0 . I ST ∗ 0 .1 .0 .0 .1 .1 .1 . I NT 1 .1 .1 .1 .2 .3 .1 .
Revised: The same result also gives an information-theoretic interpretation and information-geometric perspective about the model and the similarity concentration measure without the need of strict conditions for regularity, i.e., smoothness of the manifolds.

Original: [Figure]
Revised: [Figure]

Original:  First, for any model with a set of parameters P for which there is no loss of information, that is, no thresholding, the only condition for the increase in the discrimination power is D B N (cid:96) 1 ( X ) ≥ 
Revised: important to distinguish two main conclusions. First, for any model with a set of parameters P for which there is no loss of information, that is, no thresholding, the only condition for the increase in the discrimination power is D B N (cid:96) 1 ( X ) ≥ D B M (cid:96) 1 ( AX ) and D B N (cid:96) 1 ,c

Original: D B M (cid:96) 1 ( AX ) and D B N (cid:96) 1 ,c ( X ) ≤ D B M (cid:96) 1 ,c ( AX ) .
Revised: ( X ) ≤ D B M (cid:96) 1 ,c ( AX ) .

Original: Second, in the rest of the cases for which D B N (cid:96) 1 ( X ) ≥ D B M (cid:96) 1 ( Y ) and D B N (cid:96) 1 ,c ( X ) ≤ D B M (cid:96) 1 ,c ( Y ) holds true it will be possible to increase the discrimination power.
Revised: Second, in the rest of the cases for which D B N (cid:96) 1 ( X ) ≥ D B M (cid:96) 1 ( Y ) and D B N (cid:96) 1 ,c ( X ) ≤ D B M (cid:96) 1 ,c ( Y ) holds true it will be possible to increase the discrimination power.

Original: Moreover, there is a trade-off between the increase in discrimination power as a result of the loss of information as consequence of the non-linear thresholding operation.
Revised: Moreover, there is a trade-off between the increase in discrimination power as a result of the loss of information as consequence of the non-linear thresholding operation.

Original:  N UMERICAL EXPERIMENTS
Revised:  N UMERICAL EXPERIMENTS

Original: The numerical experiments are summarized in two different parts.
Revised: The numerical experiments are summarized in two different parts.

Original: In the ﬁrst series of the experiments the properties of the learned map A for the proposed algorithm are investigated.
Revised: In the ﬁrst series of the experiments the properties of the learned map A for the proposed algorithm are investigated.

Original: We evaluate the computational efﬁciency, as run time t e [ min ] , the conditioning number C n ( A ) =
Revised: We evaluate the computational efﬁciency, as run time t e [ min ] , the conditioning number C n ( A ) =

Original: λ max λ min , the expected mutual coherence µ ( A ) and the discrimination power across several databases for a learned non-linear transforms having different dimensionality.
Revised: λ max λ min , the expected mutual coherence µ ( A ) and the discrimination power across several databases for a learned non-linear transforms having different dimensionality.

Original: Additionally, a comparison between the resulting discrimination power in the original domain, after transform by a random matrix (having
Revised: Additionally, a comparison between the resulting discrimination power in the original domain, after transform by a random matrix (having

Original: Gaussian random samples as entries and transform dimension of M = 19000 ) and after a learned non-linear transform having transform dimension M = 19000 without and with support dissimilarity prior, denoted as I 0 , I RT , I ST ∗ and I NT , respectively, is estimated and presented .
Revised: Gaussian random samples as entries and transform dimension of M = 19000 ) and after a learned non-linear transform having transform dimension M = 19000 without and with support dissimilarity prior, denoted as I 0 , I RT , I ST ∗ and I NT , respectively, is estimated and presented.

Original: The second part evaluates a comparison of the discrimination power between the proposed algorithm and different discriminative dictionary learning methods (DDL) (Ramirez et al., 2010), (Yang et al., 2011a), (Vu et al., 2015) and (Vu & Monga, 2016).
Revised: The second part evaluates a comparison of the discrimination power between the proposed algorithm and different discriminative dictionary learning methods (DDL) (Ramirez et al., 2010), (Yang et al., 2011a), (Vu et al., 2015) and (Vu & Monga, 2016).

Original: This comparison considers a setup where the used data sets are divided into a training and test set.
Revised: This comparison considers a setup where the used data sets are divided into a training and test set.

Original: Moreover, the learning is performed on the training set and the evaluation is performed on the test set.
Revised: Moreover, the learning is performed on the training set and the evaluation is performed on the test set.

Original: In the same series of experiments the recognition accuracy for the two data sets is also computed and compared.[Figure] D D
Revised: In the same series of experiments the recognition accuracy for the two data sets is also computed and compared.

Original: I DLSI 0.

Original: I FDDL 0.

Original: I COPAR 0.

Original: I LRSDL 0.

Original: I NT 0 .0 .a)

Original: D 1 Acc.

Original: DLSI 96. FDDL 97. [Table] Data sets and algorithms set up The used data sets are Extended YALE B ( D 1 )(Georghiades et al., 2001), AR ( D 2 ) (Mart´ınez & Benavente, 1998), Norb ( D 3 ) (LeCun et al., 2004), Coil- 

Revised: Data sets and algorithms set up The used data sets are Extended YALE B ( D 1 )(Georghiades et al., 2001), AR ( D 2 ) (Mart´ınez & Benavente, 1998), Norb ( D 3 ) (LeCun et al., 2004), Coil-

Original: ( D 4 ) (Nene et al., 1996), Clatech 101 ( D 5 ) (LeCun et al., 2008), UKB ( D 6 ) (Nist´er & Stew´enius, 2006) and MNIST ( D 7 ) (Lecun & Cortes).
Revised: ( D 4 ) (Nene et al., 1996), Clatech 101 ( D 5 ) (LeCun et al., 2008), UKB ( D 6 ) (Nist´er & Stew´enius, 2006) and MNIST ( D 7 ) (Lecun & Cortes).

Original: All the images from the respective datasets were downscaled to resolutions 21 × 21 , 32 × 28 , 24 × 24 , 20 × 25 , 21 × 21 , 20 × 25 , 28 × 28 , respectively, and are normalized to unit variance.
Revised: All the images from the respective datasets were downscaled to resolutions 21 × 21 , 32 × 28 , 24 × 24 , 20 × 25 , 21 × 21 , 20 × 25 , 28 × 28 , respectively, and are C n ( A ) µ ( A ) t e [ min ] D2 .0 .5 . D1 .0 .5 .

Revised: D2 .0 .6 .

Revised: D0 .0 .8 .

Original: Considering the used implementation of the algorithm we note that the singular value decomposition for a large matrix has high computational complexity.{+D6 .
 D33 .0 .30 . D1 .0 .5 . I O 0 .0 .0 .0 .0 .0 .0 .+}
Revised: {+D6 .
 D33 .0 .30 . D1 .0 .5 . I O 0 .0 .0 .0 .0 .0 .0 .+}

Revised: I RT 0 .0 .0 .0 .0 .0 .0 .

Revised: I ST ∗ 0 .1 .0 .0 .1 .1 .1 .

Revised: I NT 1 .1 .1 .1 .2 .3 .1 .

Revised: [Figure]

Revised: [Figure]

Revised: normalized to unit variance.

Revised: Considering the used implementation of the algorithm we note that the singular value decomposition for a large matrix has high computational complexity.

Original: However, A − ˆ
Revised: However, A − ˆ

Original: A , where ˆ A is estimated as a solution in the transform update step, can be considered as an proximal operator (Parikh & Boyd, 2014) for the gradient of the objective (12).
Revised: A , where ˆ A is estimated as a solution in the transform update step, can be considered as an proximal operator (Parikh & Boyd, 2014) for the gradient of the objective (12).

Original: Additionally, instead of using all of the available data samples X , a subset of them might be used.
Revised: Additionally, instead of using all of the available data samples X , a subset of them might be used.

Original: Therefore, one simple on-line variant for the update of A w.r.t.
Revised: Therefore, one simple on-line variant for the update of A w.r.t.

Original: a subset of the available training set has the form A t +

Revised: a subset of the available training set has the form A t +

Original: =
Revised: =

Original: A t − ρ
Revised: A t − ρ

Original: ( A t − ˆ A t ) with ρ a predeﬁned step size.
Revised: ( A t − ˆ A t ) with ρ a predeﬁned step size.

Original: In the numerical experiments we use the on-line variant of the algorithm (the convergence analysis for this variant of the algorithm is left for future work) were we used a baches of sizes equal to 10% − 12% of the total amount of the available training data.
Revised: In the numerical experiments we use the on-line variant of the algorithm (the convergence analysis for this variant of the algorithm is left for future work) were we used a baches of sizes equal to 10% − 12% of the total amount of the available training data.

Original: The parameters λ 

Revised: The parameters λ

Original: and λ 1 are set such that the resulting non-linear transform representation has a very small number of non-zeros w.r.t.
Revised: and λ 1 are set such that the resulting non-linear transform representation has a very small number of non-zeros w.r.t.

Original: the transform dimension.
Revised: the transform dimension.

Original: In the experiments here this number [Figure] A cc u r a cy MNIST
Revised: In the experiments here this number is set to be 15 .

Original: M  1 .

Original: 5 K 4 K

Original: Acc.

Original: I t 0 .0 .0 .1 .a)

Original: M 1 K 4 K 6 K 12 K

Original: Acc.

Original: I t 1 .1 .1 .1 .

Original: [Figure]

Original: [Figure]

Original: M  1 .

Original: 5 K 4 K E [ z c,k M ][10 − 3 ] 0 .0 .0 .0 .

Original: I t 0 .0 .0 .1 .a)

Original: M 1 K 4 K 6 K 12 K E

Original: [ z c,k M ][10 − 4 ] 0 .0 .

Original: [Table]

Original: [Table]

Original: I

Original: [Table]

Original: [Table]

Original: [Table] [Table] b) is set to be 15 . The rest of the parameters are set as { λ 2 , λ 3 , λ 4 } = { 1000000 , 1000000 , 1000000 } .
Revised: The rest of the parameters are set as { λ 2 , λ 3 , λ 4 } = { 1000000 , 1000000 , 1000000 } .

Original: The algorithm is initialized with a random matrix having i.i.d.
Revised: The algorithm is initialized with a random matrix having i.i.d.

Original: Gaussian (zero mean, variance one) entries and is terminated after the 28 th iteration.
Revised: Gaussian (zero mean, variance one) entries and is terminated after the 28 th iteration.

Original: The results are obtained as the average of 3 runs.
Revised: The results are obtained as the average of 3 runs.

Original: An implementation presented in (Vu & Monga, 2016) was used to learn the dictionaries and estimate the sparse codes for the respective supervised dictionary learning methods DDL (Ramirez et al., 2010), (Yang et al., 2011a), (Vu et al., 2015) and (Vu & Monga, 2016).
Revised: An implementation presented in (Vu & Monga, 2016) was used to learn the dictionaries and estimate the sparse codes for the respective supervised dictionary learning methods DDL (Ramirez et al., 2010), (Yang et al., 2011a), (Vu et al., 2015) and (Vu & Monga, 2016).

Original: Linear map properties, the change in the similarity concentrations and the discrimination power
Revised: Linear map properties, the change in the similarity concentrations and the discrimination power

Original: The conditioning number and the expected coherence for the learned transforms are shown on Table 1.
Revised: The conditioning number and the expected coherence for the learned transforms are shown on Table 1.

Original: The learned transforms for all the databases have good conditioning numbers and low expected coherence.
Revised: The learned transforms for all the databases have good conditioning numbers and low expected coherence.

Original: The running time t e , measured in minutes and the number of used dimensions, denoted as M are also shown in Table 1.
Revised: The running time t e , measured in minutes and the number of used dimensions, denoted as M are also shown in Table 1.

Original: The learned transforms for all the data sets have relatively low execution time, regardless of the very high transform dimension M = 19000 .
Revised: The learned transforms for all the data sets have relatively low execution time, regardless of the very high transform dimension M = 19000 .

Original: The discrimination power is signiﬁcantly increased in the transform domain I NT compared to the one in the original domain
Revised: The discrimination power is signiﬁcantly increased in the transform domain I NT compared to the one in the original domain

Original: I O and is higher than I ST ∗ and I RT .
Revised: I O and is higher than I ST ∗ and I RT .

Original: The evolution of the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96) 

Revised: The evolution of the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96)

Original: ( Y ) , their ratio
Revised: ( Y ) , their ratio

Original: C 1 /C 2 and the discrimination power log( C 1 /C 2) =
Revised: C 1 /C 2 and the discrimination power log( C 1 /C 2) =

Original: I t for subsets of the used databases after applying a non-linear transform with transform dimension M = 19000 is shown in Figure 4.
Revised: I t for subsets of the used databases after applying a non-linear transform with transform dimension M = 19000 is shown in Figure 2.

Original: It
Revised: It

Original: [Figure]
Revised: [Figure]

Revised: D

Revised: D

Revised: I DLSI 0.

Revised: I FDDL 0.

Revised: I COPAR 0.

Original: is important to note that the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = DB M (cid:96) 1 ( Y ) are decreasing, meaning that there is a loss of information.
Revised: I LRSDL 0. I NT 0 .0 .a) D 1 Acc. DLSI 96.

Revised: FDDL 97.

Revised: [Table]

Revised: is important to note that the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96) 1 ( Y ) are decreasing, meaning that there is a loss of information.

Original: However how this loss effects the resulting similarity concentration is crucial for the discrimination properties.
Revised: However how this loss effects the resulting similarity concentration is crucial for the discrimination properties.

Original: As shown in Figure 4, the slope of decrease for C 2 is stronger.
Revised: As shown in Figure 2, the slope of decrease for C 2 is stronger.

Original: Therefore, the discrimination power increases per iteration.
Revised: Therefore, the discrimination power increases per iteration.

Original: For the Coil- 20 ( D 4 ) database there is a ﬂuctuation.
Revised: For the Coil- 20 ( D 4 ) database there is a ﬂuctuation.

Original: This is explained by the fact that during learning we used a small number of data samples from the same database and that in the data there is high variability.
Revised: This is explained by the fact that during learning we used a small number of data samples from the same database and that in the data there is high variability.

Original: The conditioning number and the expected coherence for the learned transforms for all the databases at different transform dimensions are shown in Figure 6.
Revised: The conditioning number and the expected coherence for the learned transforms for all the databases at different transform dimensions are shown in Figure 3.

Original: We see that the value of both the conditioning number and the coherence is reducing and it converges to a common values.
Revised: We see that the value of both the conditioning number and the coherence is reducing and it converges to a common values.

Original: This conﬁrms the effectiveness of the conditioning and the coherence constraints.
Revised: This conﬁrms the effectiveness of the conditioning and the coherence constraints.

Original: [-The ratio C 1 /C 2 between the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96) 
-]
Revised: [-The ratio C 1 /C 2 between the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96) 
-]

Original: ( Y ) and the discrimination power log( C 1 /C 2) =

Original: I t on subsets of the used databases after applying a nonlinear transform with transform dimension M = 19000 and varying the thresholding parameter τ = λ 1 is shown in Figure 5.

Original: We used 70 different values for the parameter λ , sampled uniformly from the interval (cid:0) 0 , (max 1 ≤ c ≤ C, 1 ≤ k ≤ K max 1 ≤ m ≤ M | a Tm x c,k | ) (cid:1) .

Original: The results were obtained using a non-linear transform learned with one value for parameter λ for all the databases.

Original: Since all the databases have different variabilities and the amount of available data is different, this result suggest that per different database there should be different optimal values for the parameter λ .

Original: The similarity concentrations C=
Revised: The similarity concentrations C=

Original: [Equation]

Original:  M [Equation] [Equation] Y ) and C=
Revised: D B M (cid:96) 1 ,c ( Y ) and C=

Original: D B M (cid:96) 1 ( Y ) , their ratio C 1 /C 2 and the discrimination power log( C 1 /C 2) =
Revised: D B M (cid:96) 1 ( Y ) , their ratio C 1 /C 2 and the discrimination power log( C 1 /C 2) =

Original: I t for a subsets of the used databases after applying a non-linear transform having transform dimensions
Revised: I t for a subsets of the used databases after applying a non-linear transform having transform dimensions

Original: M ∈ { 100 , 1150 , 2200 , 3250 , 4300 , 5350 , 6400 , 7450 , 8500 , 9550 , 10600 , 11650 , 12700 , 13750 , 14800 , 15850 , 16900 , 17950 , 19000 } is shown in Figure 7.
Revised: M ∈ { 100 , 1150 , 2200 , 3250 , 4300 , 5350 , 6400 , 7450 , 8500 , 9550 , 10600 , 11650 , 12700 , 13750 , 14800 , 15850 , 16900 , 17950 , 19000 } is shown in Figure 4.

Original: We can see similar behavior as previous, that is, the (cid:96) ∗ -norms C 1 and C 1 on the similarity concentrations are decreasing, but, the slope of decrease for C 2 is stronger.
Revised: We can see similar behavior as previous, [Figure] A cc u r a cy MNIST

Revised: M  1 .

Revised: 5 K 4 K

Revised: Acc.

Revised: I t 0 .0 .0 .1 .a)

Revised: M 1 K 4 K 6 K 12 K

Revised: Acc.

Revised: I t 1 .1 .1 .1 .

Revised: [Figure]

Revised: [Figure]

Revised: M  1 .

Revised: 5 K 4 K E [ z c,k M ][10 − 3 ] 0 .0 .0 .0 .

Revised: I t 0 .0 .0 .1 .a)

Revised: M 1 K 4 K 6 K 12 K E

Revised: [ z c,k M ][10 − 4 ] 0 .0 .0 .0 .

Revised: I t 1 .1 .1 .1 .b)

Revised: MNIST obtained by using a non-linear transform T P at different dimensionality M that is, the (cid:96) ∗ -norms C 1 and C 1 on the similarity concentrations are decreasing, but, the slope of decrease for C 2 is stronger.

Original: Therefore, the discrimination power increases as the transform dimension increases.
Revised: Therefore, the discrimination power increases as the transform dimension increases.

Original: NTL vs DDL discrimination power and recognition performance The proposed method is compared with DLSI (Ramirez et al., 2010), FDDL (Yang et al., 2011a), COPAR (Vu et al., 2015) and LRSDL (Vu & Monga, 2016).
Revised: recognition performance The proposed method is compared with DLSI (Ramirez et al., 2010), FDDL (Yang et al., 2011a), COPAR (Vu et al., 2015) and LRSDL (Vu & Monga, 2016).

Original: Half of the data samples from the data set Extended YALE
Revised: Half of the data samples from the data set Extended YALE

Original: B, sampled at random are used for learning and the remaining other half are used for evaluation.
Revised: B, sampled at random are used for learning and the remaining other half are used for evaluation.

Original: Considering the MNIST database the training set is used for learning and the test set is used for both evaluating the discrimination power and the recognition accuracy.
Revised: Considering the MNIST database the training set is used for learning and the test set is used for both evaluating the discrimination power and the recognition accuracy.

Original: We compute bouth the discrimination power and the recognition accuracy on a subsets from the test sets.
Revised: We compute bouth the discrimination power and the recognition accuracy on a subsets from the test sets.

Original: The dictionary size (transform dimension
Revised: The dictionary size (transform dimension

Original: M ) is set to be equal to { 150 , 75 , 1515 , 3825 , 570 , 150 , 300 } for the used databases, respectivly, in all of the comparing algorithms.
Revised: M ) is set to be equal to { 150 , 75 , 1515 , 3825 , 570 , 150 , 300 } for the used databases, respectivly, in all of the comparing algorithms.

Original: The discrimination power of the comparing methods is denoted as I DLSI , I FDDL ,
Revised: The discrimination power of the comparing methods is denoted as I DLSI , I FDDL ,

Original: I COPAR and I LRSDL .
Revised: I COPAR and I LRSDL .

Original: The recognition results for the methods DLSI , FDDL , COPAR and LRSDL on the data sets Extended YALE B and MNIST were not computed here, rather we use the best reported result form the respective papers (Ramirez et al., 2010), (Yang et al., 2011a), (Vu et al., 2015) and (Vu & Monga, 2016).
Revised: The recognition results for the methods DLSI , FDDL , COPAR and LRSDL on the data sets Extended YALE B and MNIST were not computed here, rather we use the best reported result form the respective papers (Ramirez et al., 2010), (Yang et al., 2011a), (Vu et al., 2015) and (Vu & Monga, 2016).

Original: Considering the proposed algorithm the non-linear transform was learned for the transform dimensions M = { 100 , 500 , 1500 , 4000 } and M = { 1000 , 4000 , 6000 , 12000 } , respectively, for the used data sets.
Revised: Considering the proposed algorithm the non-linear transform was learned for the transform dimensions M = { 100 , 500 , 1500 , 4000 } and M = { 1000 , 4000 , 6000 , 12000 } , respectively, for the used data sets.

Original: After the transform was learned, the transform data samples were computed for the respective training and test sets.
Revised: After the transform was learned, the transform data samples were computed for the respective training and test sets.

Original: Then, the transform training data samples were used as features to learn a linear SVM classiﬁer in one-against-all regime.
Revised: Then, the transform training data samples were used as features to learn a linear SVM classiﬁer in one-against-all regime.

Original: The results are shown in Table 2 a), b) and c).
Revised: The results are shown in Table 2 a), b) and c).

Original: The discrimination power of the proposed non-linear transform is higher that the discrimination power of the comparing methods.
Revised: The discrimination power of the proposed non-linear transform is higher that the discrimination power of the comparing methods.

Original: The recognition accuracy is higher for high dimensionality of the proposed method and outperforms the DDL methods at dimensionality 4000 and 12000 .
Revised: The recognition accuracy is higher for high dimensionality of the proposed method and outperforms the DDL methods at dimensionality 4000 and 12000 .

Original: In Figure 8 and Figure 9 are shown the recognition accuracy and the expected loss measured as
Revised: In Figure 5 and Figure 6 are shown the recognition accuracy and the expected loss measured as

Original: E [ (cid:107) z c,k M (cid:107) 22 ] =
Revised: E [ (cid:107) z c,k M (cid:107) 22 ] =

Original: E [ (cid:107)
Revised: E [ (cid:107)

Original: Ax c,k − y c,k (cid:107) 22 M ] as a linear function of the discrimination power evaluated at transform dimension M = { 100 , 500 , 1500 , 4000 } and M = { 1000 , 4000 , 6000 , 12000 } .
Revised: Ax c,k − y c,k (cid:107) 22 M ] as a linear function of the discrimination power evaluated at transform dimension M = { 100 , 500 , 1500 , 4000 } and M = { 1000 , 4000 , 6000 , 12000 } .

Original: It is interesting to highlight that as the discrimination power at different transform dimension increases it also increases the accuracy of recognition.
Revised: It is interesting to highlight that as the discrimination power at different transform dimension increases it also increases the accuracy of recognition.

Original: Moreover, the results on these two data sets show that this increase is approximately linear.
Revised: Moreover, the results on these two data sets show that this increase is approximately linear.

Original: On the other hand the expected loss decreases as the discrimination power at different transform dimensions increases.
Revised: On the other hand the expected loss decreases as the discrimination power at different transform dimensions increases.

Original: The results about the ratio C 1 /C 2 between the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and

Original: [-C 2 = D B M (cid:96) 
-]
Revised: [-C 2 = D B M (cid:96) 
-]

Original: ( Y ) and the discrimination power log( C 1 /C 2)

Original: =

Original: I t on the Extanded-Yale-B and the MNIST data sets after applying a non-linear transform with transform dimensions M = { 100 , 500 , 1500 , 4000 } and M = { 1000 , 4000 , 6000 , 12000 } , and varying the thresholding parameter τ = λ 1 are shown in Figure 10.

Original: We again used 70 different values for the parameter λ , sampled uniformly from the interval (cid:0) 0 , (max 1 ≤ c ≤ C, 1 ≤ k ≤ K max 1 ≤ m ≤ M | a Tm x c,k | ) (cid:1) .

Original: The results were obtained using a non-linear transforms learned with optimally choose values (by using crossvalidation) of the parameter λ for the two different databases.

Original: As expected, we can see that the extreme points of the ratio between the (cid:96) ∗ -norms C 1 and C 2 of the similarity concentrations and the discrimination power is around the optimal values of the parameter τ = λ 1 .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper presented a novel approach for learning of discriminative and sparse representation by using two different models.
Revised: This paper presented a novel approach for learning of discriminative and sparse representation by using two different models.

Original: A non-linear transform models with discriminative and minimum information loss priors were used during the learning stage and a sparsifying transform model during the testing stage.

Original: A novel discriminative prior was proposed and the properties around the models and the proposed prior were investigated.
Revised: A novel discriminative prior was proposed and the properties around the models and the proposed prior were investigated.

Original: A low complexity learning algorithm was presented with the proposed priors.
Revised: A low complexity learning algorithm was presented with the proposed priors.

Original: The preliminary results w.r.t.
Revised: The preliminary results w.r.t.

Original: the introduced measures and the recognition accuracy on the used databases showed promising performance.
Revised: the introduced measures and the recognition accuracy on the used databases showed promising performance.

Original: We showed that it is possible to increase the discrimination power with information loss.

Original: Moreover, we highlight that when expanding to high dimensional space with non-linear transforms how the loss of information reﬂects the similarity concentrations is crucial for the discriminative properties.
Revised: We showed and we highlight that when expanding to high dimensional space with non-linear transforms how the loss of information reﬂects the similarity concentrations is crucial for the discriminative properties.

Original: A study on the recognition capabilities for other databases are our next future extensions.
Revised: A study on the recognition capabilities for other databases are our next future extensions.

Revised: An analysis for the synthesis model and transform based auto-encoder towards generalization and fair comparison between different transforms and encoding methods is other future direction.

Original: In a different direction the extensions covering the sufﬁcient conditions for increase in discrimination power in the transform domain, together with an analysis for a deep architecture where per single layer we have one non-linear transform are left for our future work.
Revised: The extensions covering the sufﬁcient conditions for increase in discrimination power in the transform domain, together with an analysis for a deep architecture where per single layer we have one non-linear transform are left for our future work.

Original: A PPENDIX A. A.1 T HE GLOBAL OPTIMAL SOLUTION
Revised: A PPENDIX A. A.1 T HE GLOBAL OPTIMAL SOLUTION

Original: Given X and the curent estimate of Y , the estimate of the transform is a solution to the following problem
Revised: Given X and the curent estimate of Y , the estimate of the transform is a solution to the following problem

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 (golobal optimal solution) Given X ∈ (cid:60) N × CK and Y ∈ (cid:60) M × CK , if and only if the joint decomposition
Revised: Theorem 1 (golobal optimal solution) Given X ∈ (cid:60) N × CK and Y ∈ (cid:60) M × CK , if and only if the joint decomposition

Original: [Equation]
Revised: [Equation]

Original: exists, where U X ∈ (cid:60) N × N is orthonormal,
Revised: exists, where U X ∈ (cid:60) N × N is orthonormal,

Original: V XY ∈ (cid:60) M × N is per columns orthonormal and Σ X , Σ XY ∈ (cid:60) N × N are diagonal matrices with positive diagonal elements, then (19) has a global minimum as
Revised: V XY ∈ (cid:60) M × N is per columns orthonormal and Σ X , Σ XY ∈ (cid:60) N × N are diagonal matrices with positive diagonal elements, then (19) has a global minimum as

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Σ A ( n, n ) =
Revised: Σ A ( n, n ) =

Original: σ A ( n ) , ∀ n, σ A ( n ) ≥ 0 and σ A ( n ) are positive solutions to
Revised: σ A ( n ) , ∀ n, σ A ( n ) ≥ 0 and σ A ( n ) are positive solutions to

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 1 Consider the equvalent trace form of (19)
Revised: Proof of Theorem 1 Consider the equvalent trace form of (19)

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that since λ 2 ≥ 0 , XX T + λ 2 I is a symetric positive deﬁnite matrix whit all eigenvalues non-negative, therfore it decomposes as
Revised: Note that since λ 2 ≥ 0 , XX T + λ 2 I is a symetric positive deﬁnite matrix whit all eigenvalues non-negative, therfore it decomposes as

Original: [Equation]
Revised: [Equation]

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Then (19) equvalently is
Revised: Then (19) equvalently is

Original: [Equation]
Revised: [Equation]

Original: Asumme that B decomposes as
Revised: Asumme that B decomposes as

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where Σ B is a diagonal matrix with positive diagonal elements, U B is column orthogonal and V B is orthogonal square matrix.
Revised: where Σ B is a diagonal matrix with positive diagonal elements, U B is column orthogonal and V B is orthogonal square matrix.

Original: Moreover, let the following decomposition on XY T exists
Revised: Moreover, let the following decomposition on XY T exists

Original: [Equation]
Revised: [Equation]

Original: , and substitute as
Revised: , and substitute as

Original: [Equation]
Revised: [Equation]

Original: (29) then
Revised: (29) then

Original: Tr
Revised: Tr

Original: [Equation]
Revised: [Equation]

Original: The term
Revised: The term

Original: [Equation]
Revised: [Equation]

Original: (31) and
Revised: (31) and

Original: Tr {
Revised: Tr {

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Finnaly, (19) is reduced to
Revised: Finnaly, (19) is reduced to

Original: [Equation]
Revised: [Equation]

Original: equalling to zero the ﬁrst order derivative of the objective (34) w.r.t.
Revised: equalling to zero the ﬁrst order derivative of the objective (34) w.r.t.

Original: σ B ( n ) and multiplaing by σ B ( n ) gives
Revised: σ B ( n ) and multiplaing by σ B ( n ) gives

Original: [Equation]
Revised: [Equation]

Original: A closed form solution to (35) exists and depends on the discriminint of the quartic polynomial.
Revised: A closed form solution to (35) exists and depends on the discriminint of the quartic polynomial.

Original: Moreover, since 4 λ 3 σ 4 X ( n ) is positive a global minimum to (19) exists if and only if the decompsition (20) exists (cid:3)
Revised: Moreover, since 4 λ 3 σ 4 X ( n ) is positive a global minimum to (19) exists if and only if the decompsition (20) exists (cid:3)

Original: A.2 T HE (cid:15) - CLOSE CLOSED FORM APPROXIMATION
Revised: A.2 T HE (cid:15) - CLOSE CLOSED FORM APPROXIMATION

Original: Consider the equvalent trace form (27) of (19) and note that since λ 2 ≥ 0 , XX T + λ 2 I is a symetric positive deﬁnite matrix whit all eigenvalues non-negative, therfore it decomposes as
Revised: Consider the equvalent trace form (27) of (19) and note that since λ 2 ≥ 0 , XX T + λ 2 I is a symetric positive deﬁnite matrix whit all eigenvalues non-negative, therfore it decomposes as

Original: [Equation]
Revised: [Equation]

Original: Let the following decomposition exists
Revised: Let the following decomposition exists

Original: [Equation]
Revised: [Equation]

Original: Deﬁne
Revised: Deﬁne

Original: A = BD , where D =
Revised: A = BD , where D =

Original: U X
Revised: U X

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Assume that B decomposes as U B Σ B V TB = B , where Σ B is a diagonal matrix with positive diagonal elements, U B is column orthogonal and V B is orthogonal square matrix and let
Revised: Assume that B decomposes as U B Σ B V TB = B , where Σ B is a diagonal matrix with positive diagonal elements, U B is column orthogonal and V B is orthogonal square matrix and let

Original: [Equation]
Revised: [Equation]

Original: then
Revised: then

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Consider the decomposition U B Σ B V TB of B , use (Mirsky, 1959) and (Neumann, 1937) and note that
Revised: Consider the decomposition U B Σ B V TB of B , use (Mirsky, 1959) and (Neumann, 1937) and note that

Original: [Equation]
Revised: [Equation]

Original: where Σ Γ is a diagonal matrix, having diagonal elements Σ Γ ( n, n )
Revised: where Σ Γ is a diagonal matrix, having diagonal elements Σ Γ ( n, n )

Original: =
Revised: =

Original: σ Γ ( n ) =
Revised: σ Γ ( n ) =

Original: T ( n, n ) , ∀ n ∈ N and T =
Revised: T ( n, n ) , ∀ n ∈ N and T =

Original: U
Revised: U

Original: U X
Revised: U X

Original: XY Σ U X XY V TU X XY .
Revised: XY Σ U X XY V TU X XY .

Original: Note that the term Tr { ( AX )( AX ) T } =
Revised: Note that the term Tr { ( AX )( AX ) T } =

Original: Tr { BB T } =
Revised: Tr { BB T } =

Original: Tr { Σ 2 B } and as in the Apendix subsection A.

Revised: Tr { Σ 2 B } and as in the Apendix subsection A.

Original: Tr { AA T } =
Revised: Tr { AA T } =

Original: Tr { BDD T B T } =
Revised: Tr { BDD T B T } =

Original: Tr { Σ 2 B Σ − 2 X } ,
Revised: Tr { Σ 2 B Σ − 2 X } ,

Original: Tr { ( AA T )( AA T ) T } =
Revised: Tr { ( AA T )( AA T ) T } =

Original: Tr { ( BDD T B T )( BDD T B T ) } =
Revised: Tr { ( BDD T B T )( BDD T B T ) } =

Original: Tr { Σ 4 B Σ − 4 X } and log | det AA T | = log | det D T B T BD | = log | det U X Σ − 1 X Σ 2 B Σ − 1 X U TX | = log | det Σ − 2 X Σ 2 B | .
Revised: Tr { Σ 4 B Σ − 4 X } and log | det AA T | = log | det D T B T BD | = log | det U X Σ − 1 X Σ 2 B Σ − 1 X U TX | = log | det Σ − 2 X Σ 2 B | .

Original: Finally, the aproximation of (19) using the bound (42) is reduced to min σ B ( n )
Revised: Finally, the aproximation of (19) using the bound (42) is reduced to min σ B ( n )

Original: [Equation]
Revised: [Equation]

Original: equalling to zero the ﬁrst order derivative of the objective (43) w.r.t.
Revised: equalling to zero the ﬁrst order derivative of the objective (43) w.r.t.

Original: σ B ( n ) and multiplaing by σ B ( n ) gives
Revised: σ B ( n ) and multiplaing by σ B ( n ) gives

Original: [Equation]
Revised: [Equation]

Original: A closed form solution to (44) exists and depends on the discriminint of the quartic polynomial.
Revised: A closed form solution to (44) exists and depends on the discriminint of the quartic polynomial.

Original: Moreover, since 4 λ 3 σ 4 X ( n ) is positive a global minimum to (43) exists.
Revised: Moreover, since 4 λ 3 σ 4 X ( n ) is positive a global minimum to (43) exists.

Original: Therfore, having the decomposition U B Σ B V TB = B , the substitutions U B =
Revised: Therfore, having the decomposition U B Σ B V TB = B , the substitutions U B =

Original: ( U U X
Revised: ( U U X

Original: XY V TU X XY ) T and V B
Revised: XY V TU X XY ) T and V B

Original: = U X with the solution of (43) gives the (cid:15) -close closed form approximative solution to problem (19) as
Revised: = U X with the solution of (43) gives the (cid:15) -close closed form approximative solution to problem (19) as

Original: [Equation]
Revised: [Equation]

Original: where the bound (42) implies that the (cid:15) -close closed form approximative solution is a lower bound to the solution of (19) (cid:3)
Revised: where the bound (42) implies that the (cid:15) -close closed form approximative solution is a lower bound to the solution of (19) (cid:3)

Original: A PPENDIX B.
Revised: A PPENDIX B.

Original: Let y c,k = y + c,k + y − c,k , y + c,k ∈ (cid:60) M +
Revised: Let y c,k = y + c,k + y − c,k , y + c,k ∈ (cid:60) M +

Original: and y − c,k ∈ (cid:60) M − .
Revised: and y − c,k ∈ (cid:60) M − .

Original: Consider the similarity concentration measure
Revised: Consider the similarity concentration measure

Original: [Equation]
Revised: [Equation]

Original: c
Revised: c

Original: [Equation]
Revised: [Equation]

Original: c
Revised: c

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: c
Revised: c

Original: [Equation]
Revised: [Equation]

Original: c
Revised: c

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: 46) where g + = (cid:80) c 1 ,c
Revised: 46) where g + = (cid:80) c 1 ,c

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: c
Revised: c

Original: [Equation]
Revised: [Equation]

Original: k 1 ,k 2 | y + c 2 ,k 2 | , g − = (cid:80) c
Revised: k 1 ,k 2 | y + c 2 ,k 2 | , g − = (cid:80) c

Original: [Equation]
Revised: [Equation]

Original: c 2 c
Revised: c 2 c

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: c 2 (cid:80) k 1 ,k 2 | y − c 2 ,k 2 |
Revised: c 2 (cid:80) k 1 ,k 2 | y − c 2 ,k 2 |

Original: and we abuse notation by denoting | y c 1 ,k 1 | as the vector whose elements are the absolute values of the elements in y c 1 ,k 1 .
Revised: and we abuse notation by denoting | y c 1 ,k 1 | as the vector whose elements are the absolute values of the elements in y c 1 ,k 1 .

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: 47) where s c = (cid:80) c 1 ,c 2 c 1 (cid:54)
Revised: 47) where s c = (cid:80) c 1 ,c 2 c 1 (cid:54)

Original: =
Revised: =

Original: c 2 (cid:80) k 1 ,k 2 y + c 2 ,k 2 (cid:12) y + c 2 ,k 2 .
Revised: c 2 (cid:80) k 1 ,k 2 y + c 2 ,k 2 (cid:12) y + c 2 ,k 2 .

Original: Denote q c 1 ,k 

Revised: Denote q c 1 ,k

Original: =
Revised: =

Original: Ax c 1 ,k 1 and consider the problem min y
Revised: Ax c 1 ,k 1 and consider the problem min y

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 48) by taking the ﬁrst order derivative w.r.t.
Revised: 48) by taking the ﬁrst order derivative w.r.t.

Original: y c 1 ,k 

Revised: y c 1 ,k

Original: we have that
Revised: we have that

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: (49) take sign magnitude decomposition of y c 1 ,k 1 = sign( y c 1 ,k 1 ) (cid:12) | y c 1 ,k 1 | then we have sign( y c 1 ,k 1 ) (cid:12) | y c 1 ,k 1 | (cid:12) ( 1 + 2 λ 0 s c ) − sign( q c 1 ,k 1 ) (cid:12) | q c 1 ,k 1 | + λ 0 (sign( y + c 1 ,k 1 ) (cid:12) g + + sign( y − c 1 ,k 1 ) (cid:12) g − ) + λ 1 sign( y c 1 ,k 1 ) = 0 .
Revised: (49) take sign magnitude decomposition of y c 1 ,k 1 = sign( y c 1 ,k 1 ) (cid:12) | y c 1 ,k 1 | then we have sign( y c 1 ,k 1 ) (cid:12) | y c 1 ,k 1 | (cid:12) ( 1 + 2 λ 0 s c ) − sign( q c 1 ,k 1 ) (cid:12) | q c 1 ,k 1 | + λ 0 (sign( y + c 1 ,k 1 ) (cid:12) g + + sign( y − c 1 ,k 1 ) (cid:12) g − ) + λ 1 sign( y c 1 ,k 1 ) = 0 .

Original: Let the sign of y c 1 ,k 1 , i.e. sign( y c 1 ,k 1 ) be equal to the sign of sign( q c 1 ,k 1 ) , and Hadamar multiply from the left side by sign( q c 1 ,k 1 ) then we have[Equation] [Figure] [Equation]
Revised: Let the sign of y c 1 ,k 1 , i.e. sign( y c 1 ,k 1 ) be equal to the sign of sign( q c 1 ,k 1 ) , and Hadamar multiply from the left side by sign( q c 1 ,k 1 ) then we have

Original: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: note that sign( q c 1 ,k 1 ) (cid:12) sign( q + c 1 ,k 1 )
Revised: note that sign( q c 1 ,k 1 ) (cid:12) sign( q + c 1 ,k 1 )

Original: = sign( q + c 1 ,k 1 ) and that sign( q c 1 ,k 1 ) (cid:12) sign( q − c 1 ,k 1 ) =
Revised: = sign( q + c 1 ,k 1 ) and that sign( q c 1 ,k 1 ) (cid:12) sign( q − c 1 ,k 1 ) =

Original: sign( − q − c 1 ,k 1 ) , theretofore we have 
Revised: sign( − q − c 1 ,k 1 ) , theretofore we have | y [Equation] [Equation] |

Original: [Equation]
Revised: [Equation]

Revised: (sign( q +

Revised: [Equation]

Revised: [Equation]

Original:  (52) since the magnitude might be only positive we have that | y c 1 ,k 1 | (cid:12) ( 1 + 2 λ 0 s c ) =
Revised: k 1 ) [Equation] g + + sign( − q − c 1 ,k 1 ) (cid:12) g − ) − λ 1 1 . (52) since the magnitude might be only positive we have that | y c 1 ,k 1 | (cid:12) ( 1 + 2 λ 0 s c ) =

Original: max( | q c 1 ,k 1 | − λ 0 (sign( q + c 1 ,k 1 ) (cid:12) g + + sign( − q − c 1 ,k 1 ) (cid:12) g − ) − λ 1 1 , 0 ) .
Revised: max( | q c 1 ,k 1 | − λ 0 (sign( q + c 1 ,k 1 ) (cid:12) g + + sign( − q − c 1 ,k 1 ) (cid:12) g − ) − λ 1 1 , 0 ) .

Original: Denote g =
Revised: Denote g =

Original: (sign( q + c 1 ,k 1 ) (cid:12) g + + sign( − q − c 1 ,k 1 ) (cid:12) g − ) then the closed form solution to (48) is:y [Equation] [Equation] 1 =sign
Revised: (sign( q + c 1 ,k 1 ) (cid:12) g + + sign( − q − c 1 ,k 1 ) (cid:12) g − ) then the closed form solution to (48) is:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: max

Original: [Equation] [Equation] ( 1 + λ 0 s [Equation]
Revised: | q c 1 ,k 1 | − λ 0 g − λ 1 1 , 0 ) (cid:11) ( 1 + λ 0 s c ) .

Original: (53) which completes the proof (cid:3)
Revised: (53) which completes the proof (cid:3)

Original: A PPENDIX C. S ENSITIVITY ANALYSIS AND INTERPRETATIONS
Revised: A PPENDIX C. S ENSITIVITY ANALYSIS AND INTERPRETATIONS

Original: The similarity concentration measure provides possibility to measure the discriminative properties, their deviation, increase (or decrease) and the corresponding relations between different non-linear transform models across one domain or different domains, thereby quantifying their quality w.r.t.
Revised: The similarity concentration measure provides possibility to measure the discriminative properties, their deviation, increase (or decrease) and the corresponding relations between different non-linear transform models across one domain or different domains, thereby quantifying their quality w.r.t.

Original: the discriminative properties.
Revised: the discriminative properties.

Original: C.1 S ENSITIVITY ANALYSIS W . R . T . THE SIMILARITY CONCENTRATION MEASURES
Revised: C.1 S ENSITIVITY ANALYSIS W . R . T . THE SIMILARITY CONCENTRATION MEASURES

Revised: An illustration about the deﬁnition of discriminative power given by a diagram is shown in Figure 7.

Original: To measure the ability for an increase in discriminative properties by a non-linear transform 11 we ﬁrst have to deﬁne a notion for the discriminative properties on a data set under different nonlinear transform models.
Revised: To measure the ability for an increase in discriminative properties by a non-linear transform 10 we ﬁrst have to deﬁne a notion for the discriminative properties on a data set under different nonlinear transform models.

Original: Therefore, ﬁrst we introduce the ”special” base models and then analyze the properties of the similarity concentration measures under the change in model parameter and the relation between the base model and the proposed non-linear transform model deﬁned by a parameter set P = { A ∈ (cid:60) M × N , τ ∈ (cid:60) M } .
Revised: Therefore, ﬁrst we introduce the ”special” base models and then analyze the properties of the similarity concentration measures under the change in model parameter and the relation between the base model and the proposed non-linear transform model deﬁned by a parameter set P = { A ∈ (cid:60) M × N , τ ∈ (cid:60) M } .

Original: Any data set X in the original domain might have a transform model with parameters B N = { A o ∈ (cid:60) N × N , τ = 0 ∈ (cid:60) N + } , if A o = I ∈
Revised: Any data set X in the original domain might have a transform model with parameters B N = { A o ∈ (cid:60) N × N , τ = 0 ∈ (cid:60) N + } , if A o = I ∈

Original: D N + we refer to it as a base original model .
Revised: D N + we refer to it as a base original model .

Original: Similarly as in the original domain, any data set Y in the transform domain might have a transform model with parameters B M =
Revised: Similarly as in the original domain, any data set Y in the transform domain might have a transform model with parameters B M =

Original: { A t ∈ (cid:60) M × M , τ = 0 ∈ (cid:60) M + } , if A t = I ∈
Revised: { A t ∈ (cid:60) M × M , τ = 0 ∈ (cid:60) M + } , if A t = I ∈

Original: D M
Revised: D M

Original: + we refer to it as a base transform model .
Revised: + we refer to it as a base transform model .

Original: Any base model, deﬁned ether in the original domain B N or in the transform domain B M , has domain equal to the co-domain, since x c,k = T B N ( x c,k ) and y c,k = T B M ( y c,k ) holds trivially, for the respective sets of parameters B N =
Revised: Any base model, deﬁned ether in the original domain B N or in the transform domain B M , has domain equal to the co-domain, since x c,k = T B N ( x c,k ) and y c,k = T B M ( y c,k ) holds trivially, for the respective sets of parameters B N =

Original: { A o = I ∈
Revised: { A o = I ∈

Original: D N + , τ = 0 ∈ (cid:60) N + } and
Revised: D N + , τ = 0 ∈ (cid:60) N + } and

Original: B M =
Revised: B M =

Original: { A t = I ∈
Revised: { A t = I ∈

Original: D M + , τ = 0 ∈ (cid:60) M + } .
Revised: D M + , τ = 0 ∈ (cid:60) M + } .

Original: This is illustrated with a diagram shown in Figure 11.
Revised: This is illustrated with a diagram shown in Figure 8.

Original: A base original model provides a possibility to compare it with any other non-linear transform model with parametersP =
Revised: A base original model provides a possibility to compare it with any other non-linear transform model with parameters

Original: { A ∈ (cid:60) M × N , τ ∈ (cid:60) M+ } . Additionally, note that for B M = { A t = I ∈ D M , τ = 0 ∈ (cid:60) M } we have that D B M (cid:96) 1 ,c ( Y ) = D P (cid:96) 1 ,c ( X ) and that D B M (cid:96) 1 ( Y ) =
Revised: M } c

Original: D P (cid:96) 1 ( X ) .
Revised: D Y ) [Figure]

Original: It implies that the similarity concentrations can be analyzed as a function in the original domain under model
Revised: that the similarity concentrations can be analyzed as a function in the original domain under model

Original: P or in the transform domain under model B M .
Revised: P or in the transform domain under model B M .

Original: The main relations considering the preservation of change in the similarity concentration between two models, deﬁned not necessary in the same domain are stated by Proposition 1.
Revised: The main relations considering the preservation of change in the similarity concentration between two models, deﬁned not necessary in the same domain are stated by Proposition 1.

Original: Lemma 1 : The non-linear transform model (2) totally preserves the information in the change for the similarity concentration for a data set X w.r.t.
Revised: Lemma 1 : The non-linear transform model (2) totally preserves the information in the change for the similarity concentration for a data set X w.r.t.

Original: a small change in the parameters of the models
Revised: a small change in the parameters of the models

Original: B N , B M and P if (cid:107) δ o (cid:107) ∗ = 0 and (cid:107) δ t (cid:107) ∗ = 0 as 
Revised: B N , B M and P if (cid:107) δ o (cid:107) ∗ = 0 and (cid:107) δ t (cid:107) ∗ = 0 as [Equation] [Equation] A

Original: [Equation]
Revised: [Equation]

Original:  :A
Revised: R 3) :

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Revised: [Equation]

Revised: k

Original: [Equation]
Revised: [Equation]

Original: The proof is given in Appendix C.2 .
Revised: The proof is given in Appendix C.2 .

Original: The terms z c,k represent the non-linear transform error vectors that appear in the model
Revised: The terms z c,k represent the non-linear transform error vectors that appear in the model

Original: Ax c,k = y c,k + z c,k as a result of applying an element-wise non-liner operation H τ to Ax c,k , i.e. , y c,k =
Revised: Ax c,k = y c,k + z c,k as a result of applying an element-wise non-liner operation H τ to Ax c,k , i.e. , y c,k =

Original: H τ ( Ax c,k ) .
Revised: H τ ( Ax c,k ) .

Original: As an example in the sparsifying transform model z c,k is the ”loss of information”, that is the information about the values of the elements in Ax c,k that are discarded.
Revised: As an example in the sparsifying transform model z c,k is the ”loss of information”, that is the information about the values of the elements in Ax c,k that are discarded.

Original: The terms δ o and δ t correlate the errors z c,k with the original data x c 1 ,k 1 and transform data y c 1 ,k 1 , respectively.
Revised: The terms δ o and δ t correlate the errors z c,k with the original data x c 1 ,k 1 and transform data y c 1 ,k 1 , respectively.

Original: Note that if there is no loss of information (in the earlier example it means that there is no thresholding and just a simple linear transform model is used) then δ o = 

Revised: Note that if there is no loss of information (in the earlier example it means that there is no thresholding and just a simple linear transform model is used) then δ o =

Original: and δ t = 0 .
Revised: and δ t = 0 .

Original: Moreover, δ o and δ t bear important information about the discriminative properties in the transform domain.
Revised: Moreover, δ o and δ t bear important information about the discriminative properties in the transform domain.

Original: The terms∂D
Revised: The terms

Original: [Equation]
Revised: [Equation]

Original: A and ∂D P (cid:96) 1 ( X ) ∂
Revised: and ∂D [Equation]

Original: A represent the change of the similarity concentrations under inﬁnitesimally small change of the parameter A from the model P .
Revised: represent the change of the similarity concentrations under inﬁnitesimally small change of the parameter A from the model P .

Original: The terms ∂D B N (cid:96) 1 ,c ( X ) ∂
Revised: The terms ∂D B N (cid:96) 1 ,c ( X ) ∂

Original: A o |
Revised: A o |

Original: A
Revised: A

Original: o =
Revised: o =

Original: I and ∂D B N
Revised: I and ∂D B N

Original: [Equation]
Revised: [Equation]

Original: X
Revised: ∂ A

Original: [Equation]
Revised: [Equation]

Original: ∂ A o | A 
Revised: | A o

Original: o =
Revised: =

Original: I have dual interpretation.
Revised: I have dual interpretation.

Original: Assuming metric
Revised: Assuming metric

Original: A o
Revised: A o

Original: = I , then the ﬁrst one is considered as a change of the similarity concentrations under inﬁnitesimally small change of the space metric, or equivalently under small metric perturbation.
Revised: = I , then the ﬁrst one is considered as a change of the similarity concentrations under inﬁnitesimally small change of the space metric, or equivalently under small metric perturbation.

Original: Conversely, assuming the data samples are distributed under a Gaussian distribution with parameters identity covariance matrix and zero mean, i.e. x c,k ∼ N ( µ = 0 , Σ = I ) , then ∂D B N (cid:96) 1 ,c ( X )
Revised: Conversely, assuming the data samples are distributed under a Gaussian distribution with parameters identity covariance matrix and zero mean, i.e. x c,k ∼ N ( µ = 0 , Σ = I ) , then ∂D B N (cid:96) 1 ,c ( X [Equation]

Original: ∂
Revised: ∂

Original: A o |
Revised: A o |

Original: A o = I and ∂D B N (cid:96) 1 ( X ) ∂
Revised: [Equation]

Original: A o |
Revised: and

Revised: [Equation]

Original: A
Revised: A

Revised: [Equation]

Original:  o =
Revised: | A o =

Original: I represent the change of the similarity concentrations under small change in the assumption away from a Gaussian distribution.
Revised: I represent the change of the similarity concentrations under small change in the assumption away from a Gaussian distribution.

Original: Equation ( R 1) relates the base transform for the original domain B N with any arbitrary transform deﬁned in the original domain P .
Revised: Equation ( R 1) relates the base transform for the original domain B N with any arbitrary transform deﬁned in the original domain P .

Original: The relation ( R 2) is a result about the preservation of change in the similarity concentration between two models B M and P deﬁned on two different domains.
Revised: The relation ( R 2) is a result about the preservation of change in the similarity concentration between two models B M and P deﬁned on two different domains.

Original: Whereas ( R 3) gives the preservation of change in the similarity concentration between the error in the transform domain.
Revised: Whereas ( R 3) gives the preservation of change in the similarity concentration between the error in the transform domain.

Original: The next result highlights the relation between: the linear projection ( by the linear map A that appears in the model P ) of the change in the similarity concentration under the model B N in the original domain and the change of the similarity concentration under the model B M in the transform domain .
Revised: The next result highlights the relation between: the linear projection ( by the linear map A that appears in the model P ) of the change in the similarity concentration under the model B N in the original domain and the change of the similarity concentration under the model B M in the transform domain .

Original: This relation exists independently for ∂D
Revised: This relation exists independently for ∂D

Original: [Equation]
Revised: [Equation]

Revised: |

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: | A o
Revised: | A o

Original: =
Revised: =

Original: I , nevertheless, we will deﬁne the summarized and the independent versions.
Revised: I , nevertheless, we will deﬁne the summarized and the independent versions.

Original: Therefore, ﬁrst we deﬁne ∂ J (cid:96) 
Revised: Therefore, ﬁrst we deﬁne ∂ J (cid:96) 1 ( X ) ∂ A o | A o = I = ∂D B

Original: [Equation]
Revised: [Equation]

Original: =
Revised: [Equation] |

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: 1 ( Y ) ∂ A t | A t
Revised: | A [Equation]

Original: =
Revised: =

Original: I and rewrite ( R 1) as A ∂ J ( X )
Revised: I and rewrite ( R 1) as A ∂ J ( X )

Original: [Equation] |
Revised: ∂ A o |

Original: [Equation]
Revised: A o = I − δ o

Original: =
Revised: =

Revised: ∂D

Original: I
Revised: P

Original: [Equation]
Revised: [Equation]

Original: δ o = ∂D P (cid:96) 1 ,c ( X ) ∂ 
Revised: 1 ,c [Equation] X ) ∂ A + ∂D

Revised: [Equation]

Original: A + ∂D P (cid:96) 1 ( X ) ∂ A , then replace ∂D P (cid:96) 1 ,c ( X)
Revised: [Equation] A , then replace ∂D P [Equation] c ( X

Original:  ∂ A + ∂D P (cid:96) 1 ( X ) ∂ A in ( R 2) by the same term in ( R 1) , use ( R 3) , reorder and we have the following result.
Revised: [Equation] ∂ A + [Equation] X ) ∂ A in ( R 2) by the same term in ( R 1) , use ( R 3) , reorder and we have the following result.

Original: Lemma 2 : For ﬁxed τ any non-linear transform model (2) preservers the information in the change of similarity concentrations w.r.t.
Revised: Lemma 2 : For ﬁxed τ any non-linear transform model (2) preservers the information in the change of similarity concentrations w.r.t.

Original: a small change in A by
Revised: a small change in A by

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: The expressions ( R 4) actually relate the metric in the original domain under the model B N to the induced metric in the transform domain for the model B M , with induction done by the model P with parameter set { A ∈ (cid:60) M × N , τ ∈ (cid:60) M + } .
Revised: The expressions ( R 4) actually relate the metric in the original domain under the model B N to the induced metric in the transform domain for the model B M , with induction done by the model P with parameter set { A ∈ (cid:60) M × N , τ ∈ (cid:60) M + } .

Original: Moreover, the model P might describe a transform domain with a non-smooth manifold.
Revised: Moreover, the model P might describe a transform domain with a non-smooth manifold.

Original: Since the manifolds of the original and the transform domain under the models B N and B M are smooth the analysis of their relations reveals insights about the relation between the manifolds under the models B N and P .
Revised: Since the manifolds of the original and the transform domain under the models B N and B M are smooth the analysis of their relations reveals insights about the relation between the manifolds under the models B N and P .

Original: The terms ∂ J (cid:96) 

Revised: The terms ∂ J (cid:96)

Original: ( Z ) ∂
Revised: ( Z ) ∂

Original: A t | A t
Revised: A t | A t

Original: =
Revised: =

Original: I and ∂ J (cid:96) 

Revised: I and ∂ J (cid:96)

Original: ( Y ; Z )
Revised: ( Y ; Z )

Original: ∂
Revised: ∂

Original: A t | A t =
Revised: A t | A t =

Original: I carry out the information about the breaks and the discontinuities of the regularity and smoothness in the manifold induced by the model P .
Revised: I carry out the information about the breaks and the discontinuities of the regularity and smoothness in the manifold induced by the model P .

Original: Also we note that ∂ n J (cid:96) 1 ( X ) ∂ n A o |
Revised: Also we note that [Equation] n J [Equation]

Revised: X

Revised: [Equation]

Revised: n

Revised: [Equation]

Original: A o = I = 4 ∂ n − 1 J (cid:96) 1 ( X ) ∂ n − 1 A o |
Revised: [Equation] 1 J (cid:96) 1 ( X ) ∂ n [Equation] 1 A o |

Original: A
Revised: A

Original: o =
Revised: o =

Original: I and ∂ n J (cid:96) [-
-]
Revised: I and [Equation] n J (cid:96) [-
-]1 ( Y )

Original: ( Y ) ∂
Revised: [Equation]

Original: n A t | A t = I = 4 ∂ n − 1 J (cid:96) 1 ( Y ) ∂ n − 1 A t | A t 
Revised: n A t | A t = I = 4 ∂ n − 1 J (cid:96) 1 ( Y ) ∂ n − 1 A t | A t =

Original: = I therefore, ∂ J (cid:96) 1 ( X ) ∂
Revised: I therefore, [Equation] J (cid:96) 1 ( X ) [Equation]

Original: A o |
Revised: A o |

Original: A o = I =[-
-]
Revised: A o = I =[-
-]

Revised: [Equation]

Original: ∂ 2 J (cid:96) 1 ( X ) ∂ 2 A o | A
Revised: 2 J (cid:96) 1 ( X ) ∂ 2 A o | A

Original: o = I and ∂ J (cid:96) 1 ( Y )
Revised: o = I and [Equation] J (cid:96) 1 ( Y )

Original: ∂
Revised: [Equation]

Original: A t | A t = I = 14 ∂ 2 J (cid:96) 1 ( Y ) ∂
Revised: A t | A t = I = 14 ∂ 2 J (cid:96) 1 ( Y ) ∂

Original: 2 A t | A t
Revised: 2 A t | A t

Original: = I might be interpreted as
Revised: = I might be interpreted as

Original: Fisher information matrices evaluated at A o = I ∈
Revised: Fisher information matrices evaluated at A o = I ∈

Original: D N and A t
Revised: D N and A t

Original: = I ∈ D M .
Revised: = I ∈ D M .

Original: Furthermore, if (cid:107) ∂
Revised: Furthermore, if (cid:107) ∂

Original: J (cid:96) 1 ( Z ) 
Revised: J (cid:96) 1 ( Z ) ∂

Original: A t | A t = I (cid:107) ∗ = 0 and (cid:107) ∂
Revised: A t | A t = I (cid:107) ∗ = 0 and (cid:107) ∂

Original: J [-(cid:96) 
-]
Revised: J [-(cid:96) 
-][Equation] [Equation] [Equation]

Original: ( Y ; Z )
Revised: [Equation]

Original: ∂
Revised: A t

Original: A t | A t = I (cid:107) ∗ = 0 , then ( R 4) in information geometry is seen as change of coordinates on a manifold, where the intrinsic properties of curvature remain unchanged under different parametrization.
Revised: [Equation] , then ( R 4) in information geometry is seen as change of coordinates on a manifold, where the intrinsic properties of curvature remain unchanged under different parametrization.

Original: A PPENDIX C. 
Revised: A PPENDIX C.

Original: Note that for the model P t we have that
Revised: Note that for the model P t we have that

Original: [Equation]
Revised: [Equation]

Original: since 
Revised: since sign

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst order derivative of the divergence D P t (cid:96) 1 ( x ; g ) w.r.t.
Revised: The ﬁrst order derivative of the divergence D P t (cid:96) 1 ( x ; g ) w.r.t.

Original: the parameter A is:
Revised: the parameter A is:

Original: [Equation]
Revised: [Equation]

Original: we assume that the threshold parameter τ is chosen such that the vector
Revised: we assume that the threshold parameter τ is chosen such that the vector

Original: | Ax | − τ (or for any other q , the vector | Aq | − τ ) has least one non-zero element, then
Revised: | Ax | − τ (or for any other q , the vector | Aq | − τ ) has least one non-zero element, then

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: combining (61) and (62) we have that
Revised: combining (61) and (62) we have that

Revised: [Equation]

Revised: T

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: ∀ m ∈ M .
Revised: ∀ m ∈ M .

Original: Similarity, note that for the model P o 0 = { A o , τ = 0 } we have that
Revised: Similarity, note that for the model P o 0 = { A o , τ = 0 } we have that

Original: [Equation]
Revised: [Equation]

Original: [Equation] The ﬁrst order derivative of the divergence D P o 0 (cid:96) 1 ( x ; g ) w.r.t
Revised: The ﬁrst order derivative of the divergence D P o 0 (cid:96) 1 ( x ; g ) [Equation]

Original: A o is:
Revised: .r.

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  note that at A o
Revised: 66) note that at A o

Original: =
Revised: =

Original: I , P t 0 =
Revised: I , P t 0 =

Original: B N
Revised: B N

Original: and we have that
Revised: and we have that

Original: [Equation]
Revised: [Equation]

Original: Also for the model P t 

Revised: Also for the model P t

Original: = { A t , τ = 0 } we have that
Revised: = { A t , τ = 0 } we have that

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst order derivative of the divergence D P t [Equation] [Equation]
Revised: The ﬁrst order derivative of the divergence D P t 0 (cid:96) 1 ( y ; q ) w.r.t A t is:

Original: [Equation]
Revised: [Equation]

Original:  note that at A t=
Revised: 69) note that at A t

Original:  I , P t 0 =
Revised: = I , P t 0 =

Original: B M and we have that
Revised: B M and we have that

Original: [Equation]
Revised: [Equation]

Original: Consider the following
Revised: Consider the following

Original: [Equation]
Revised: [Equation]

Original: 71) where
Revised: where

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: A closer look at (71) reveals us that
Revised: A closer look at (71) reveals us that

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: By similar construction applied to the rest of the pairs of data samples we have the result:
Revised: By similar construction applied to the rest of the pairs of data samples we have the result:

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: A closer look at (76) reveals us
Revised: A closer look at (76) reveals us

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: By similar construction applied to the rest of the pairs of data samples we have the result:
Revised: By similar construction applied to the rest of the pairs of data samples we have the result:

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: A closer look at (81) reveals us
Revised: A closer look at (81) reveals us

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: By similar construction applied to the rest of the pairs of data samples we have the result:
Revised: By similar construction applied to the rest of the pairs of data samples we have the result:

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Original: A PPENDIX D
Revised: A PPENDIX D

Original: The result in by Lemma 2 (57) decomposes on the contribution components for simmilarity and on the contributiong components for dissimilarity
Revised: The result in by Lemma 2 (57) decomposes on the contribution components for simmilarity and on the contributiong components for dissimilarity

Original: [Equation]
Revised: [Equation]

Original: Moreover
Revised: Moreover

Original: [Equation]
Revised: [Equation]

Original: w.r
Revised: w.r.t.

Original: [Equation]

Original: t. the simmilarity concentrations we have the following decompositions
Revised: the simmilarity concentrations we have the following decompositions

Original: [Equation]
Revised: [Equation]

Original: therfore we have the following bounds
Revised: therfore we have the following bounds

Original: [Equation]
Revised: [Equation]

Original: Additionaly, we have that c : λ min( A T A )
Revised: Additionaly, we have that c : λ min

Original: Tr { ∂D B N (cid:96) 1 ,c ( X ) ∂ A o | A o = I } ≤ Tr { A ∂D B N (cid:96) 1 ,c ( X ) ∂
Revised: [Equation]

Original: A o |
Revised: T

Original: A o = I A T } , where λ min ( A T A ) is the minimum singlular value of the matrix A T A .
Revised: [Equation] where λ min ( A T A ) is the minimum singlular value of the matrix A T A .

Original: Taking the logarithm of the ratio D P (cid:96) 1 ,c ( X ) D P (cid:96) 1 ( X )+ (cid:15) using the bounds a , b and c we arrive at the desired result (cid:3) 
Revised: Taking the logarithm of the ratio D P (cid:96) 1 ,c ( X ) D P (cid:96) 1 ( X )+ (cid:15) using the bounds a , b and c we arrive at the desired result (cid:3) A PPENDIX E

Revised: The exact steps of the proposed non-linear transform learning are described by Algorithm 1.

Revised: A PPENDIX F

Original: {+The ratio C 1 /C 2 between the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96) 
+}
Revised: {+The ratio C 1 /C 2 between the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96) 
+}

Revised: ( Y ) and the discrimination power log( C 1 /C 2) =

Revised: I t on subsets of the used databases after applying a nonlinear transform with transform dimension M = 19000 and varying the thresholding parameter τ = λ 1 is shown in Figure 9.

Revised: We used 70 different values for the parameter λ , sampled uniformly from the interval (cid:0) 0 , (max 1 ≤ c ≤ C, 1 ≤ k ≤ K max 1 ≤ m ≤ M | a Tm x c,k | ) (cid:1) .

Revised: The results were obtained using

Revised: Input X , λ 0 , λ 1 , λ 2 , λ 3 , λ

Revised: A ← inicialize repeat

Revised: DISCRIMINATIVE ENCODING closed form solution per data sample

Revised: Y ← AX repeatfor ∀ c ∈ C do d − c ← (cid:80) c 1 c 1 (cid:54) =

Revised: c (cid:80) k 1 y − c 1 ,k 1 , d + c ← (cid:80) c 1 c 1 (cid:54) =

Revised: c (cid:80) k 1 y + c 1 ,k 1 and s c ← (cid:80) c 1 c 1 (cid:54) =

Revised: c (cid:80) k 1 y c 1 ,k 1 (cid:12) y c 1 ,kend for for ∀ c ∈ C and ∀ k ∈ K do g ← sign(max( Ax c,k , 0 ))

Revised: (cid:12) d

Revised: + c + sign(max( − Ax c,k , 0 ))

Revised: (cid:12) d − c y c,k ← sign( Ax c,k ) (cid:12) max ( Ax c,k − λ 0 g + λ 1 1 , 0 ) (cid:11) (1 + 2 λ 0 s c ) end for until convergence

Revised: TRANSFORM UPDATE (cid:15) -close closed form solution

Revised: [Equation]

Revised: where σ

Revised: [Equation]

Revised: until convergence

Revised: Output A , Y

Revised: [Figure]

Revised: a non-linear transform learned with one value for parameter λ for all the databases.

Revised: Since all the databases have different variabilities and the amount of available data is different, this result suggest that per different database there should be different optimal values for the parameter λ .

Revised: The results about the ratio C 1 /C 2 between the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and

Original: {+C 2 = D B M (cid:96) 
+}
Revised: {+C 2 = D B M (cid:96) 
+}

Revised: ( Y ) and the discrimination power log( C 1 /C 2)

Revised: =

Revised: I t on the Extanded-Yale-B and the MNIST data sets after applying a non-linear transform with transform dimensions M = { 100 , 500 , 1500 , 4000 } and M = { 1000 , 4000 , 6000 , 12000 } , and varying the thresholding parameter τ = λ 1 are shown in Figure 10.

Revised: We again used 70 different values for the parameter λ , sampled uniformly from the interval (cid:0) 0 , (max 1 ≤ c ≤ C, 1 ≤ k ≤ K max 1 ≤ m ≤ M | a Tm x c,k | ) (cid:1) .

Revised: The results were obtained using a non-linear transforms learned with optimally choose values (by using crossvalidation) of the parameter λ for the two different databases.

Revised: As expected, we can see that the extreme points of the ratio between the (cid:96) ∗ -norms C 1 and C 2 of the similarity concentrations and the discrimination power is around the optimal values of the parameter τ = λ 1 .