Original: L EARNING NON - LINEAR TRANSFORM WITH DISCRIMINATIVE AND MINIMUM INFORMATION LOSS PRIORS
Revised: L EARNING NON - LINEAR TRANSFORM WITH DISCRIMINATIVE AND MINIMUM INFORMATION LOSS PRIORS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper proposes a novel approach for learning discriminative and sparse representations.
Revised: This paper proposes a novel approach for learning discriminative and sparse representations.

Original: It consists of utilizing two different models.
Revised: It consists of utilizing two different models.

Original: A predeﬁned number of non-linear transform models are used in the learning stage, and one sparsifying transform model is used at test time.
Revised: A predeﬁned number of non-linear transform models are used in the learning stage, and one sparsifying transform model is used at test time.

Original: The non-linear transform models have discriminative and minimum information loss priors.
Revised: The non-linear transform models have discriminative and minimum information loss priors.

Original: A novel measure related to the discriminative prior is proposed and deﬁned on the support intersection for the transform representations.
Revised: A novel measure related to the discriminative prior is proposed and deﬁned on the support intersection for the transform representations.

Original: The minimum information loss prior is expressed as a constraint on the conditioning and the expected coherence of the transform matrix.
Revised: The minimum information loss prior is expressed as a constraint on the conditioning and the expected coherence of the transform matrix.

Original: An equivalence between the non-linear models and the sparsifying model is shown only when the measure that is used to deﬁne the discriminative prior goes to zero.
Revised: An equivalence between the non-linear models and the sparsifying model is shown only when the measure that is used to deﬁne the discriminative prior goes to zero.

Original: An approximation of the measure used in the discriminative prior is addressed, connecting it to a similarity concentration.
Revised: An approximation of the measure used in the discriminative prior is addressed, connecting it to a similarity concentration.

Original: To quantify the discriminative properties of the transform representation, we introduce another measure and present its bounds.
Revised: To quantify the discriminative properties of the transform representation, we introduce another measure and present its bounds.

Original: Reﬂecting the discriminative quality of the transform representation we name it as discrimination power.
Revised: Reﬂecting the discriminative quality of the transform representation we name it as discrimination power.

Original: To support and validate the theoretical analysis a practical learning algorithm is presented
Revised: To support and validate the theoretical analysis a practical learning algorithm is presented

Original: We evaluate the advantages and the potential of the proposed algo
Revised: We evaluate the advantages and the potential of the proposed algo

Original: - rithm by a computer simulation.
Revised: - rithm by a computer simulation.

Original: A favorable performance is shown considering the execution time, the quality of the representation, measured by the discrimination power and the recognition accuracy in comparison with the state-of-the-art methods of the same category.
Revised: A favorable performance is shown considering the execution time, the quality of the representation, measured by the discrimination power and the recognition accuracy in comparison with the state-of-the-art methods of the same category.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Learning a transform that provides sparse and discriminative representation is an active domain of research in various areas, some of which are data processing, pattern recognition, image processing, language modeling, text analysis and gene separation.
Revised: Learning a transform that provides sparse and discriminative representation is an active domain of research in various areas, some of which are data processing, pattern recognition, image processing, language modeling, text analysis and gene separation.

Original: A class of algorithms proposed by KreutzDelgado et al. (2003); Mairal et al.
Revised: A class of algorithms proposed by KreutzDelgado et al. (2003); Mairal et al.

Original: ; Bengio et al.
Revised: ; Bengio et al.

Original: ; Gangeh et al. (2015); Mairal et al.
Revised: ; Gangeh et al. (2015); Mairal et al.

Original: ; Jiang et al.
Revised: ; Jiang et al.

Original: (2011); Guo et al.
Revised: (2011); Guo et al.

Original: ; Cai et al.
Revised: ; Cai et al.

Original: (2014) and Liu et al.
Revised: (2014) and Liu et al.

Original: (2016) for learning discriminative sparse representations have been shown to perform well across various learning tasks.
Revised: (2016) for learning discriminative sparse representations have been shown to perform well across various learning tasks.

Original: A subclass of them known as discriminative dictionary learning (DDL) Guo et al.
Revised: A subclass of them known as discriminative dictionary learning (DDL) Guo et al.

Original: ; Jiang et al.
Revised: ; Jiang et al.

Original: ; Cai et al.
Revised: ; Cai et al.

Original: ; Shekhar et al.
Revised: ; Shekhar et al.

Original: Xu et al. (2015); Liu et al.
Revised: Xu et al. (2015); Liu et al.

Original: ; Bengio et al.
Revised: ; Bengio et al.

Original: (2012); Gangeh et al.
Revised: (2012); Gangeh et al.

Original: ; Jiang et al.
Revised: ; Jiang et al.

Original: and Vu & Monga (2016) addreses the estimate of the dictionary in a supervised manner such that the representation w.r.t.
Revised: and Vu & Monga (2016) addreses the estimate of the dictionary in a supervised manner such that the representation w.r.t.

Original: words (vectors) from the resulting dictionary (vector set) is discriminative.
Revised: words (vectors) from the resulting dictionary (vector set) is discriminative.

Original: Most of the DDL methods synthesize the data sample k from class c , i.e , x c,k ∈ (cid:60) N as an approximation by a linear combination y c,k ∈ (cid:60) M (referred to as a sparse data representation (cid:107) y c,k (cid:107) 0 << M ) of a few words (vectors d m ), from a dictionary (vector set)
Revised: Most of the DDL methods synthesize the data sample k from class c , i.e , x c,k ∈ (cid:60) N as an approximation by a linear combination y c,k ∈ (cid:60) M (referred to as a sparse data representation (cid:107) y c,k (cid:107) 0 << M ) of a few words (vectors d m ), from a dictionary (vector set)

Original: D ∈ (cid:60) N × M , i.e. , x c,k = Dy c,k
Revised: D ∈ (cid:60) N × M , i.e. , x c,k = Dy c,k

Original: + v c,k , v c,k ∈ (cid:60) N , with v c,k denoting the approximation error.
Revised: + v c,k , v c,k ∈ (cid:60) N , with v c,k denoting the approximation error.

Original: It is important to highlight that with the synthesis model approach the data reconstruction is addressed.
Revised: It is important to highlight that with the synthesis model approach the data reconstruction is addressed.

Original: The differences between the DDL methods Guo et al.
Revised: The differences between the DDL methods Guo et al.

Original: ; Jiang et al.
Revised: ; Jiang et al.

Original: ; Cai et al.
Revised: ; Cai et al.

Original: Shekhar et al.
Revised: Shekhar et al.

Original: (2014); Gangeh et al.
Revised: (2014); Gangeh et al.

Original: ; Xu et al. (2015); Liu et al.
Revised: ; Xu et al. (2015); Liu et al.

Original: ; Bengio et al.
Revised: ; Bengio et al.

Original: Jiang et al.
Revised: Jiang et al.

Original: and Vu & Monga (2016) are determined by the prior deﬁned on the sparse representation and the prior deﬁned for the relations between the sparse representations for the data samples from the same/different classes.
Revised: and Vu & Monga (2016) are determined by the prior deﬁned on the sparse representation and the prior deﬁned for the relations between the sparse representations for the data samples from the same/different classes.

Original: The discrimination is enforced by replacing the prior with a structural constraint on the dictionary or imposing a discriminative term on the sparse representations.
Revised: The discrimination is enforced by replacing the prior with a structural constraint on the dictionary or imposing a discriminative term on the sparse representations.

Original: Additionally, some works by Mairal et al.
Revised: Additionally, some works by Mairal et al.

Original: ; Guo et al.
Revised: ; Guo et al.

Original: (2012) and Taalimi et al.
Revised: (2012) and Taalimi et al.

Original: (2015) consider even a joint estimation/learning of a dictionary, sparse representation, and classiﬁer by using iterative alternating minimization strategy.
Revised: (2015) consider even a joint estimation/learning of a dictionary, sparse representation, and classiﬁer by using iterative alternating minimization strategy.

Original: The manuscripts by Bengio et al.
Revised: The manuscripts by Bengio et al.

Original: ; Cai et al.
Revised: ; Cai et al.

Original: (2014) and Gangeh et al.
Revised: (2014) and Gangeh et al.

Original: (2015) give comprehensive overview covering different approaches.
Revised: (2015) give comprehensive overview covering different approaches.

Original: 1. O PEN ISSUES
Revised: 1. O PEN ISSUES

Original: The general open issue for DDL methods is the computational complexity w.r.t.
Revised: The general open issue for DDL methods is the computational complexity w.r.t.

Original: the optimal dictionary/transform learning and the discriminative encoding, since the sparse representation in the synthesis model is a solution to an inverse problem .
Revised: the optimal dictionary/transform learning and the discriminative encoding, since the sparse representation in the synthesis model is a solution to an inverse problem .

Original: An additional open issue with most of the proposed approaches Guo et al.
Revised: An additional open issue with most of the proposed approaches Guo et al.

Original: ; Jiang et al.
Revised: ; Jiang et al.

Original: Cai et al.
Revised: Cai et al.

Original: (2014); Gangeh et al.
Revised: (2014); Gangeh et al.

Original: ; Liu et al.
Revised: ; Liu et al.

Original: (2016), Bengio et al.
Revised: (2016), Bengio et al.

Original: ; Jiang et al.
Revised: ; Jiang et al.

Original: ; Vu & Monga (2016) is that there is no formal notion to measure the discriminative properties.
Revised: ; Vu & Monga (2016) is that there is no formal notion to measure the discriminative properties.

Original: Therefore, there are no means that provide a quantitative evaluation of the quality of the representation, other than the performance of a classiﬁer used on top of the representation.
Revised: Therefore, there are no means that provide a quantitative evaluation of the quality of the representation, other than the performance of a classiﬁer used on top of the representation.

Original: Concerning the speciﬁcs in the discriminative constraints, Yang et al. (2011b) proposed a synthesis model with a discriminative ﬁdelity term and Fisher discriminant constraints, where the within-class scatter and the between-class scatter of the representation is minimized and maximized, respectively.
Revised: Concerning the speciﬁcs in the discriminative constraints, Yang et al. (2011b) proposed a synthesis model with a discriminative ﬁdelity term and Fisher discriminant constraints, where the within-class scatter and the between-class scatter of the representation is minimized and maximized, respectively.

Original: The authors Vu & Monga (2016) proposed an extension considering a low-rank constraint on the dictionary.
Revised: The authors Vu & Monga (2016) proposed an extension considering a low-rank constraint on the dictionary.

Original: An approach by Guo et al.
Revised: An approach by Guo et al.

Original: (2013) used a synthesis model with a constraint on the pair-wise relation between the sparse representation expressed by (cid:96) 2 distance metric.
Revised: (2013) used a synthesis model with a constraint on the pair-wise relation between the sparse representation expressed by (cid:96) 2 distance metric.

Original: The methods reported by Yang et al.
Revised: The methods reported by Yang et al.

Original: (2011b)
Revised: (2011b)

Original: Vu & Monga (2016) and Guo et al.
Revised: Vu & Monga (2016) and Guo et al.

Original: (2013) take into account assumption on the metric by deﬁning the scatter and the pair-wise relations.
Revised: (2013) take into account assumption on the metric by deﬁning the scatter and the pair-wise relations.

Original: Therefore, they constrain the space of the representation, which essentially is determined by the dictionary.
Revised: Therefore, they constrain the space of the representation, which essentially is determined by the dictionary.

Original: However, these works do not consider whether the used metric is optimal w.r.t.
Revised: However, these works do not consider whether the used metric is optimal w.r.t.

Original: the sparse representation.
Revised: the sparse representation.

Original: The method proposed by Liu et al.
Revised: The method proposed by Liu et al.

Original: (2016) ﬁnds a dictionary under which the representation of a data sample from the same class
Revised: (2016) ﬁnds a dictionary under which the representation of a data sample from the same class

Original: c have a common sparse structure by minimizing the size of the support overlap for the representation from different classes.
Revised: c have a common sparse structure by minimizing the size of the support overlap for the representation from different classes.

Original: Assuming y c 1 ,k 1 ∈ (cid:60) M and y c 2 ,k 2 ∈ (cid:60) M are two sparse representations for two data samples x c 1 ,k 1 ∈ (cid:60) N and x c 2 ,k 2 ∈ (cid:60) N , from two classes cand c
Revised: Assuming y c 1 ,k 1 ∈ (cid:60) M and y c 2 ,k 2 ∈ (cid:60) M are two sparse representations for two data samples x c 1 ,k 1 ∈ (cid:60) N and x c 2 ,k 2 ∈ (cid:60) N , from two classes cand c

Original: 2 , they proposed a similarity measure deﬁned by empirical expectation on (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 0 , where (cid:12) represents the Hadamar product.
Revised: 2 , they proposed a similarity measure deﬁned by empirical expectation on (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 0 , where (cid:12) represents the Hadamar product.

Original: Note that two transform data samples y c 1 ,k 2 and y c 2 ,kthat have small support overlap (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 

Revised: Note that two transform data samples y c 1 ,k 2 and y c 2 ,kthat have small support overlap (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107)

Original: = s, s << M , might not necessarily be similar or dissimilar, i.e. , y c 1 ,k 1 = y c 2 ,k 2 and y c 1 ,k 1 = − y c 2 ,k 2 with (cid:107) y c 1 ,k 1 (cid:107) 

Revised: = s, s << M , might not necessarily be similar or dissimilar, i.e. , y c 1 ,k 1 = y c 2 ,k 2 and y c 1 ,k 1 = − y c 2 ,k 2 with (cid:107) y c 1 ,k 1 (cid:107)

Original: = (cid:107) y c
Revised: = (cid:107) y c

Original: ,k 2 (cid:107) 

Revised: ,k 2 (cid:107)

Original: = s and s small.
Revised: = s and s small.

Original: A PPROACH AND MOTIVATIONS
Revised: A PPROACH AND MOTIVATIONS

Original: Model Instead of addressing a synthesis model where the data reconstruction is targeted and the estimation of the discriminative representation is an inverse problem 1 we present a novel, alternative approach.
Revised: Model Instead of addressing a synthesis model where the data reconstruction is targeted and the estimation of the discriminative representation is an inverse problem 1 we present a novel, alternative approach.

Original: That is we propose non-linear transform models in the learning stage and a sparsifying transform model Rubinstein et al.
Revised: That is we propose non-linear transform models in the learning stage and a sparsifying transform model Rubinstein et al.

Original: (2010), Rubinstein et al.
Revised: (2010), Rubinstein et al.

Original: (2013), Rubinstein & Elad (2014)
Revised: (2013), Rubinstein & Elad (2014)

Original: and
Revised: and

Original: Ravishankar & Bresler (2014) for testing.
Revised: Ravishankar & Bresler (2014) for testing.

Original: The sparsifying transform model assumes that the data sample x c,k is approximately sparsiﬁable under a linear transform A ∈ (cid:60) M × N , i.e. , Ax c,k
Revised: The sparsifying transform model assumes that the data sample x c,k is approximately sparsiﬁable under a linear transform A ∈ (cid:60) M × N , i.e. , Ax c,k

Original: = y c,k
Revised: = y c,k

Original: + z c,k , z c,k ∈ (cid:60) M , where y c,k is sparse (cid:107) y c,k (cid:107) 0 << M .
Revised: + z c,k , z c,k ∈ (cid:60) M , where y c,k is sparse (cid:107) y c,k (cid:107) 0 << M .

Original: It also represents a generalization of the analysis model Rubinstein et al.
Revised: It also represents a generalization of the analysis model Rubinstein et al.

Original: ; Ravishankar & Bresler (2014); Rubinstein &
Revised: ; Ravishankar & Bresler (2014); Rubinstein &

Original: Elad (2014).
Revised: Elad (2014).

Original: The proposed non-linear transform model is an extension to the sparsifying transform model that considers additional assumptions.
Revised: The proposed non-linear transform model is an extension to the sparsifying transform model that considers additional assumptions.

Original: Both of the models used in this paper address a direct problem , where the estimation of the discriminative representation represents a low complexity constrained projection problem.
Revised: Both of the models used in this paper address a direct problem , where the estimation of the discriminative representation represents a low complexity constrained projection problem.

Original: Additionally, since these models have no restrictions on the transform representation to be in the column space of the dictionary (transform matrix A ), they allow more freedom in modeling and imposing constraints on the transform representation 2 .
Revised: Additionally, since these models have no restrictions on the transform representation to be in the column space of the dictionary (transform matrix A ), they allow more freedom in modeling and imposing constraints on the transform representation 2 .

Original: [Figure]
Revised: [Figure]

Original: Prior and its measure In the learning stage, central in the non-linear transform models is the novel parametric measure for a discriminative prior.
Revised: Prior and its measure In the learning stage, central in the non-linear transform models is the novel parametric measure for a discriminative prior.

Original: It is deﬁned on the support intersection for the transform representations.
Revised: It is deﬁned on the support intersection for the transform representations.

Original: The ﬁrst motivation behind the used measure is that the support intersection of the transform data allows more freedom in imposing regularization on the discriminative properties without taking into account any additional assumptions.
Revised: The ﬁrst motivation behind the used measure is that the support intersection of the transform data allows more freedom in imposing regularization on the discriminative properties without taking into account any additional assumptions.

Original: Second, by approximating the parametric mesure with an non-parametric one the focus of the regularization is directly put on the contributing components for similarity/dissimilarity.
Revised: Second, by approximating the parametric mesure with an non-parametric one the focus of the regularization is directly put on the contributing components for similarity/dissimilarity.

Original: Consider the measure ( y + c 1 ,k 1 )
Revised: Consider the measure ( y + c 1 ,k 1 )

Original: T y + c 2 ,k 

Revised: T y + c 2 ,k

Original: + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 between two transform representations y c 1 ,k 1 and y c 2 ,k 2 where y c 1 ,k 1 = y + c 1 ,k 1 − y − c 1 ,k 1 where y + c 1 ,k 1 = max( y c 1 ,k 1 , 0 ) and y − c 1 ,k 1 = max( − y c 1 ,k 1 , 0 ) .
Revised: + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 between two transform representations y c 1 ,k 1 and y c 2 ,k 2 where y c 1 ,k 1 = y + c 1 ,k 1 − y − c 1 ,k 1 where y + c 1 ,k 1 = max( y c 1 ,k 1 , 0 ) and y − c 1 ,k 1 = max( − y c 1 ,k 1 , 0 ) .

Original: Note that ( y + c 1 ,k 1 ) T y + c 2 ,k 2 + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 = (cid:107) y + c 1 ,k 1 (cid:12) y + c 2 ,k 2 (cid:107) 1 + (cid:107) y − c 1 ,k 1 (cid:12) y − c 2 ,k 2 (cid:107) 1 captures the only contribution for the similarity (whereas (cid:107)
Revised: Note that ( y + c 1 ,k 1 ) T y + c 2 ,k 2 + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 = (cid:107) y + c 1 ,k 1 (cid:12) y + c 2 ,k 2 (cid:107) 1 + (cid:107) y − c 1 ,k 1 (cid:12) y − c 2 ,k 2 (cid:107) 1 captures the only contribution for the similarity (whereas (cid:107)

Original: y + c 1 ,k 1 (cid:12) y − c 2 ,k 2 (cid:107) 1 + (cid:107) y − c 1 ,k 1 (cid:12) y
Revised: y + c 1 ,k 1 (cid:12) y − c 2 ,k 2 (cid:107) 1 + (cid:107) y − c 1 ,k 1 (cid:12) y

Original: + c 2 ,k 2 (cid:107) 1 captures the only contribution for the dissimilarity) between the vectors y c 1 ,k 1 and y c 2 ,k 2 .
Revised: + c 2 ,k 2 (cid:107) 1 captures the only contribution for the dissimilarity) between the vectors y c 1 ,k 1 and y c 2 ,k 2 .

Original: Moreover, y Tc 1 ,k 1 y c 2 ,k 2 = (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 1 , only if the dissimilarity contribution − ( y − c 1 ,k 1 ) T y + c 2 ,k 2 − ( y + c 1 ,k 1 ) T y − c 2 ,k 2 for the vectors y c 1 ,kand y c 2 ,k 2 is 0 .
Revised: Moreover, y Tc 1 ,k 1 y c 2 ,k 2 = (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 1 , only if the dissimilarity contribution − ( y − c 1 ,k 1 ) T y + c 2 ,k 2 − ( y + c 1 ,k 1 ) T y − c 2 ,k 2 for the vectors y c 1 ,kand y c 2 ,k 2 is 0 .

Original: That is y
Revised: That is y

Original: Tc
Revised: Tc

Original: 1 ,k 1 y c 2 ,k 2 =
Revised: 1 ,k 1 y c 2 ,k 2 =

Original: ( y + c 1 ,k 1 ) T y + c 2 ,k 

Revised: ( y + c 1 ,k 1 ) T y + c 2 ,k

Original: + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 − ( y − c 1 ,k 1 ) T y + c 2 ,k 2 − ( y + c 1 ,k 1 ) T y − c 2 ,k 2 ≤ ( y + c 1 ,k 1 )
Revised: + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 − ( y − c 1 ,k 1 ) T y + c 2 ,k 2 − ( y + c 1 ,k 1 ) T y − c 2 ,k 2 ≤ ( y + c 1 ,k 1 )

Original: T y + c 2 ,k 

Revised: T y + c 2 ,k

Original: + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 , now if − ( y − c 1 ,k 1 ) T y + c 2 ,k 2 − ( y + c 1 ,k 1 )
Revised: + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 , now if − ( y − c 1 ,k 1 ) T y + c 2 ,k 2 − ( y + c 1 ,k 1 )

Original: T y − c 2 ,kis zero then (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 

Revised: T y − c 2 ,kis zero then (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107)

Original: =
Revised: =

Original: ( y + c 1 ,k 1 ) T y + c 2 ,k 

Revised: ( y + c 1 ,k 1 ) T y + c 2 ,k

Original: + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 = y
Revised: + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 = y

Original: Tc 1 ,k 1 y c 2 ,k 2 .
Revised: Tc 1 ,k 1 y c 2 ,k 2 .

Original: Third, the expectation E [ (cid:107) y − c 1 ,k 1 (cid:12) y − c 2 ,k 2 (cid:107) 

Revised: Third, the expectation E [ (cid:107) y − c 1 ,k 1 (cid:12) y − c 2 ,k 2 (cid:107)

Original: + (cid:107) y
Revised: + (cid:107) y

Original: + c 1 ,k 1 (cid:12) y + c 2 ,k 2 (cid:107) 1 ] captures the concentration of similarity .
Revised: + c 1 ,k 1 (cid:12) y + c 2 ,k 2 (cid:107) 1 ] captures the concentration of similarity .

Original: Therefore, it provides the possibility to deﬁne a formal notion that quantiﬁes the discriminative properties.
Revised: Therefore, it provides the possibility to deﬁne a formal notion that quantiﬁes the discriminative properties.

Original: Fourth, ( y + c 1 ,k 1 ) T y + c 2 ,k 2 + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 is not ambiguous w.r.t.
Revised: Fourth, ( y + c 1 ,k 1 ) T y + c 2 ,k 2 + ( y − c 1 ,k 1 ) T y − c 2 ,k 2 is not ambiguous w.r.t.

Original: a notion for similarity/dissimilarity between two sparse representations y c 1 ,k 1 and y c 2 ,k 2 .
Revised: a notion for similarity/dissimilarity between two sparse representations y c 1 ,k 1 and y c 2 ,k 2 .

Original: This is because the support intersections for the positive and the negative components ( y + c 1 ,k 1 ) T y + c 2 ,k 2 = (cid:107) ( y + c 1 ,k 1 ) (cid:12) y + c 2 ,k 2 (cid:107)and ( y − c 1 ,k 1 )
Revised: This is because the support intersections for the positive and the negative components ( y + c 1 ,k 1 ) T y + c 2 ,k 2 = (cid:107) ( y + c 1 ,k 1 ) (cid:12) y + c 2 ,k 2 (cid:107)and ( y − c 1 ,k 1 )

Original: T y − c 2 ,k 

Revised: T y − c 2 ,k

Original: = (cid:107) ( y + c 1 ,k 1 ) T y + c 2 ,k 2 (cid:107) 1 , respectively, are considered separately.
Revised: = (cid:107) ( y + c 1 ,k 1 ) T y + c 2 ,k 2 (cid:107) 1 , respectively, are considered separately.

Original: In addition, taken into account is the strength on the support intersection, deﬁned as (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 22 .
Revised: In addition, taken into account is the strength on the support intersection, deﬁned as (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 22 .

Original: Its empirical expectation (cid:80) c,k (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 22 ∼ E [ (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 22 ] captures the expected strength on the support intersection for that set.
Revised: Its empirical expectation (cid:80) c,k (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 22 ∼ E [ (cid:107) y c 1 ,k 1 (cid:12) y c 2 ,k 2 (cid:107) 22 ] captures the expected strength on the support intersection for that set.

Original: A schematic diagram of the transform and the main idea behind the proposed concept are shown in Figure 1 , a ) , b ) and c ) .
Revised: A schematic diagram of the transform and the main idea behind the proposed concept are shown in Figure 1 , a ) , b ) and c ) .

Original: On Figure 1 d), e) and f) are given illustrative examples for the support intersections between the positive and negative component of two vectors y c 1 ,k 1 and y c 2 ,k 2 in the transform domain.
Revised: On Figure 1 d), e) and f) are given illustrative examples for the support intersections between the positive and negative component of two vectors y c 1 ,k 1 and y c 2 ,k 2 in the transform domain.

Original: A learning algorithm is presented using the proposed model with discriminative and minimum information loss priors.
Revised: A learning algorithm is presented using the proposed model with discriminative and minimum information loss priors.

Original: To quantify the discriminative properties, we introduce a measure named as discrimination power , which reﬂects the discriminate properties of the representation for a dataset.
Revised: To quantify the discriminative properties, we introduce a measure named as discrimination power , which reﬂects the discriminate properties of the representation for a dataset.

Original: In addition, we present its lower and upper bounds, which depend on the parameters of the transform.
Revised: In addition, we present its lower and upper bounds, which depend on the parameters of the transform.

Original: On the practical side, the advantages and the potential of the proposed algorithm are demonstrated by a numerical experiments using the Extended YALE B Georghiades et al.
Revised: On the practical side, the advantages and the potential of the proposed algorithm are demonstrated by a numerical experiments using the Extended YALE B Georghiades et al.

Original: (2001), AR Mart´ınez &
Revised: (2001), AR Mart´ınez &

Original: Benavente (1998), Norb LeCun et al.
Revised: Benavente (1998), Norb LeCun et al.

Original: (2004), Coil- 20 Nene et al.
Revised: (2004), Coil- 20 Nene et al.

Original: , Clatech 101 LeCun et al.
Revised: , Clatech 101 LeCun et al.

Original: (2008), UKB Nist ´ er & Stew ´ enius (2006) and MNIST Lecun & Cortes datasets.
Revised: (2008), UKB Nist ´ er & Stew ´ enius (2006) and MNIST Lecun & Cortes datasets.

Original: 1. N OTATIONS
Revised: 1. N OTATIONS

Original: A scalar variable is denoted using x , a vector is denoted by a bold, low caps symbols x , a matrix by bold, upper cap symbol A .
Revised: A scalar variable is denoted using x , a vector is denoted by a bold, low caps symbols x , a matrix by bold, upper cap symbol A .

Original: A single element from a vector (or matrix) is denoted as x ( n ) (or
Revised: A single element from a vector (or matrix) is denoted as x ( n ) (or

Original: A ( m, n ) ).
Revised: A ( m, n ) ).

Original: A set is denoted by a calligraphic symbol, i.e , S .
Revised: A set is denoted by a calligraphic symbol, i.e , S .

Original: The (cid:96) p − norm is denoted as (cid:107) .
Revised: The (cid:96) p − norm is denoted as (cid:107) .

Original: (cid:107) p and the nuclear norm as (cid:107) .
Revised: (cid:107) p and the nuclear norm as (cid:107) .

Original: (cid:107) ∗ .
Revised: (cid:107) ∗ .

Original: The (cid:12) symbol represents the Hadamard product.
Revised: The (cid:12) symbol represents the Hadamard product.

Original: Throughout the paper it is assumed that a set of data samples X =
Revised: Throughout the paper it is assumed that a set of data samples X =

Original: [ X 1 , X 2 , ..., X C ] ∈ (cid:60) N × L , L = CK from C classes is given and that every class c ∈ C
Revised: [ X 1 , X 2 , ..., X C ] ∈ (cid:60) N × L , L = CK from C classes is given and that every class c ∈ C

Original: = { 1 , 2 , 3 , ..., C } has K samples, X c
Revised: = { 1 , 2 , 3 , ..., C } has K samples, X c

Original: =
Revised: =

Original: [ x c, 1 , x c, 2 , ..., x c,K ] ∈ (cid:60) N × K , x c,k ∈ (cid:60) N , ∀ c ∈ C , ∀ k ∈ K
Revised: [ x c, 1 , x c, 2 , ..., x c,K ] ∈ (cid:60) N × K , x c,k ∈ (cid:60) N , ∀ c ∈ C , ∀ k ∈ K

Original: = { 1 , 2 , ..., K } .
Revised: = { 1 , 2 , ..., K } .

Original: We denote the transform data as Y =
Revised: We denote the transform data as Y =

Original: [ Y 1 , Y 2 , ..., Y C ] ∈ (cid:60) M × L , where Y c =
Revised: [ Y 1 , Y 2 , ..., Y C ] ∈ (cid:60) M × L , where Y c =

Original: [ y c, 1 , y c, 2 , ..., y c,K ] ∈ (cid:60) M × K and y c,k ∈ (cid:60) M .
Revised: [ y c, 1 , y c, 2 , ..., y c,K ] ∈ (cid:60) M × K and y c,k ∈ (cid:60) M .

Original: We denote X \{ k ∈ c } = (cid:2)
Revised: We denote X \{ k ∈ c } = (cid:2)

Original: X 1 , X 2 , ..
Revised: X 1 , X 2 , ..

Original: X c, \ k , ...
Revised: X c, \ k , ...

Original: X C (cid:3) ∈ (cid:60) N × ( L − 1) as the matrix that has all the columns of X , except the column x c,k ∈ (cid:60) N , where X c, \ k =
Revised: X C (cid:3) ∈ (cid:60) N × ( L − 1) as the matrix that has all the columns of X , except the column x c,k ∈ (cid:60) N , where X c, \ k =

Original: [ x c, 1 , x c, 2 , ..., x c,k − 1 , x c,k +1 , ..., x c,K ] ∈ (cid:60) N × ( K − 1) is a matrix that has all the columns of block X c , except the column x c,k , ∀ c ∈ C and ∀ k ∈ K .
Revised: [ x c, 1 , x c, 2 , ..., x c,k − 1 , x c,k +1 , ..., x c,K ] ∈ (cid:60) N × ( K − 1) is a matrix that has all the columns of block X c , except the column x c,k , ∀ c ∈ C and ∀ k ∈ K .

Original: We let N = { 1 , 2 , ..., N } and M = { 1 , 2 , ..., M } .
Revised: We let N = { 1 , 2 , ..., N } and M = { 1 , 2 , ..., M } .

Original:  L EARNING NON - LINEAR TRANSFORM WITH DISCRIMINATIVE AND MINIMUM INFORMATION LOSS PRIORS
Revised:  L EARNING NON - LINEAR TRANSFORM WITH DISCRIMINATIVE AND MINIMUM INFORMATION LOSS PRIORS

Original: The proposed approach has two operational modes: learning and test.
Revised: The proposed approach has two operational modes: learning and test.

Original: It considers two different models.
Revised: It considers two different models.

Original: A predeﬁned number of non-linear transforms are used in the learning mode and one sparsifying transform is used for test time.
Revised: A predeﬁned number of non-linear transforms are used in the learning mode and one sparsifying transform is used for test time.

Original: 2. T HE PARAMETRIC NON - LINEAR TRANSFORM MODELING
Revised: 2. T HE PARAMETRIC NON - LINEAR TRANSFORM MODELING

Original: Considering the learning mode, we assume that for every class c ∈ C there exist one non-linear transform deﬁned by a set of parameters
Revised: Considering the learning mode, we assume that for every class c ∈ C there exist one non-linear transform deﬁned by a set of parameters

Original: P c
Revised: P c

Original: =
Revised: =

Original: { A ∈ (cid:60) M × N , τ c ∈ (cid:60) M } , ∀ c ∈ C 3 .
Revised: { A ∈ (cid:60) M × N , τ c ∈ (cid:60) M } , ∀ c ∈ C 3 .

Original: All nonlinear transforms described by {P 1 , ..., P C } share the linear map A and have distinct parameters τ c .
Revised: All nonlinear transforms described by {P 1 , ..., P C } share the linear map A and have distinct parameters τ c .

Original: One
Revised: One

Original: P c with τ c is related to only one class c .
Revised: P c with τ c is related to only one class c .

Original: It is assumed that the parameters τ c are spread far apart in the transform domain.
Revised: It is assumed that the parameters τ c are spread far apart in the transform domain.

Original: In addition, when all the non-linear transforms are applied to the corresponding class samples then the transform data samples are separable w.r.t.
Revised: In addition, when all the non-linear transforms are applied to the corresponding class samples then the transform data samples are separable w.r.t.

Original: the different classes, in the transform domain.
Revised: the different classes, in the transform domain.

Original: As far as the non-linear transform we focus on transforms expressible by a two-steps operation, consisting of a linear mapping (step 1 ) followed by an element-wise non-linearity (step 2 ):
Revised: As far as the non-linear transform we focus on transforms expressible by a two-steps operation, consisting of a linear mapping (step 1 ) followed by an element-wise non-linearity (step 2 ):

Original: [Equation]
Revised: [Equation]

Original: c,k
Revised: c,k

Original: [Equation]
Revised: [Equation]

Original: (1) where ∀ w c,k
Revised: (1) where ∀ w c,k

Original: = Ax c,k ∈ (cid:60) M , H τ c ( w c,k )
Revised: = Ax c,k ∈ (cid:60) M , H τ c ( w c,k )

Original: =
Revised: =

Original: sign( w c,k ) (cid:12) max( | w c,k | − τ c , 0 ) :
Revised: sign( w c,k ) (cid:12) max( | w c,k | − τ c , 0 ) :

Original: (cid:60) M → (cid:60) M , represents a non-linear thresholding function with parameters τ c ∈ (cid:60) M .
Revised: (cid:60) M → (cid:60) M , represents a non-linear thresholding function with parameters τ c ∈ (cid:60) M .

Original: We also have to mention that the thresholding is done with different thresholding parameters τ c ( m ) for the corresponding different transform dimensions m ∈ M .
Revised: We also have to mention that the thresholding is done with different thresholding parameters τ c ( m ) for the corresponding different transform dimensions m ∈ M .

Original: At testing time we use one, common sparsifying transform deﬁned by a set of parameters
Revised: At testing time we use one, common sparsifying transform deﬁned by a set of parameters

Original: P = { A ∈ (cid:60) M × N , τ
Revised: P = { A ∈ (cid:60) M × N , τ

Original: [Equation]
Revised: [Equation]

Original: ∈
Revised: ∈

Original: [Equation]
Revised: [Equation]

Original: M } for all data samples, with a constant tresholding parameter τ .
Revised: M } for all data samples, with a constant tresholding parameter τ .

Original: The transform matrix A is one and same for the C training and the testing models.
Revised: The transform matrix A is one and same for the C training and the testing models.

Original: In the following section we describe the non-linear transform models, the proposed discriminative prior and its mesure, together with the main reason behind this particular use of non-linear transform models for learning and a sparsifying transform model for testing.
Revised: In the following section we describe the non-linear transform models, the proposed discriminative prior and its mesure, together with the main reason behind this particular use of non-linear transform models for learning and a sparsifying transform model for testing.

Original: 2. T HE NON - LINEAR TRANSFORM MODEL WITH A DISCRIMINATIVE PRIOR
Revised: 2. T HE NON - LINEAR TRANSFORM MODEL WITH A DISCRIMINATIVE PRIOR

Original: The learning model This paper deﬁnes a compact description of the non-linear transform (1) by a non-linear transform model as follows:
Revised: The learning model This paper deﬁnes a compact description of the non-linear transform (1) by a non-linear transform model as follows:

Original: [Equation]
Revised: [Equation]

Original: where T P c ( . ) :
Revised: where T P c ( . ) :

Original: (cid:60) N
Revised: (cid:60) N

Original: →
Revised: →

Original: (cid:60) M is a parametric non-linear function that gives y c,k , by using the set of parameters P c .
Revised: (cid:60) M is a parametric non-linear function that gives y c,k , by using the set of parameters P c .

Original: The term z c,k
Revised: The term z c,k

Original: = Ax c,k − y c,k is the non-linear transform error vector that represents the deviation of Ax c,k from the targeted transform representation y c,k = T P c ( x c,k ) in the transform domain.
Revised: = Ax c,k − y c,k is the non-linear transform error vector that represents the deviation of Ax c,k from the targeted transform representation y c,k = T P c ( x c,k ) in the transform domain.

Original: Since the transform representation y c,k = T P c ( x c,k ) takes into acount a non-linearity, Ax c,k is only seen as its linear approximation.
Revised: Since the transform representation y c,k = T P c ( x c,k ) takes into acount a non-linearity, Ax c,k is only seen as its linear approximation.

Original: Knowing something in advance about the difference between y c,k − Ax c,k can be used in our model.
Revised: Knowing something in advance about the difference between y c,k − Ax c,k can be used in our model.

Original: However, since in advance we do not have any prior we assume that it is Gausssian like distributed.
Revised: However, since in advance we do not have any prior we assume that it is Gausssian like distributed.

Original: Therefore, the prior on z c,k is modeled as p ( x c,
Revised: Therefore, the prior on z c,k is modeled as p ( x c,

Original: [Equation]
Revised: [Equation]

Original: | y
Revised: | y

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , A
Revised: , A

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Ax
Revised: Ax

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: ) , where β 0 is a scaling parameter.
Revised: ) , where β 0 is a scaling parameter.

Original: Assuming additionally that the non-linear function T P c ( x c,k ) gives sparse y c,k , then we have the improper prior on y c,k , deﬁned as p ( y c,k )
Revised: Assuming additionally that the non-linear function T P c ( x c,k ) gives sparse y c,k , then we have the improper prior on y c,k , deﬁned as p ( y c,k )

Original: ∝ exp( − (cid:107) y c,k (cid:107) 1 β) where β 1 is a scaling parameter.
Revised: ∝ exp( − (cid:107) y c,k (cid:107) 1 β) where β 1 is a scaling parameter.

Original: This paper models the joint probability p ( τ 1 , τ 2 , ...., τ C , y c,k ) as
Revised: This paper models the joint probability p ( τ 1 , τ 2 , ...., τ C , y c,k ) as

Original: [Equation]
Revised: [Equation]

Original: and assumes that ( A iid ) : p ( τ 1 , τ 2 , ...., τ C ) =
Revised: and assumes that ( A iid ) : p ( τ 1 , τ 2 , ...., τ C ) =

Original: (cid:81) c
Revised: (cid:81) c

Original: p ( τ c ) , where β 2 is a scaling parameter.
Revised: p ( τ c ) , where β 2 is a scaling parameter.

Original: By the assumption in subsection 2.1, τ c 1 are spread far apart in the transform domain.
Revised: By the assumption in subsection 2.1, τ c 1 are spread far apart in the transform domain.

Original: Therefore, a minimum over D ( y c,k ; τ c 1 ) ensures that y c,k in the transform domain will be located to the closest τ c 1 w.r.t.
Revised: Therefore, a minimum over D ( y c,k ; τ c 1 ) ensures that y c,k in the transform domain will be located to the closest τ c 1 w.r.t.

Original: to the mesure D ( y c,k ; τ c 1 ) .
Revised: to the mesure D ( y c,k ; τ c 1 ) .

Original: Moreover, given the class label c using (3) and ( A iid ) the model for the discriminative prior reduces to p ( τ c | y c,k ) ∝ exp( − D ( y c,k ; τ c ) β) where D ( y c,k ; τ c ) is a parametric measure with parameter τ c .
Revised: Moreover, given the class label c using (3) and ( A iid ) the model for the discriminative prior reduces to p ( τ c | y c,k ) ∝ exp( − D ( y c,k ; τ c ) β) where D ( y c,k ; τ c ) is a parametric measure with parameter τ c .

Original: Assuming that D ( y c,k ; τ c ) is determined by a relation on the support intersection between y c,k and τ c we propose the following deﬁnition of measure:
Revised: Assuming that D ( y c,k ; τ c ) is determined by a relation on the support intersection between y c,k and τ c we propose the following deﬁnition of measure:

Original: [Equation]
Revised: [Equation]

Original: where (cid:107) y
Revised: where (cid:107) y

Original: + c,k (cid:12)
Revised: + c,k (cid:12)

Original: τ
Revised: τ

Original: + c (cid:107) 1 + (cid:107) y − c,k (cid:12) τ − c (cid:107) 1 measures the similarity contribution on the support intersection using the positive and the negative components, y + c,k , τ + c and y − c,k , τ − c , respectively, of y c,k and τ c and (cid:107) y c,k
Revised: + c (cid:107) 1 + (cid:107) y − c,k (cid:12) τ − c (cid:107) 1 measures the similarity contribution on the support intersection using the positive and the negative components, y + c,k , τ + c and y − c,k , τ − c , respectively, of y c,k and τ c and (cid:107) y c,k

Original: (cid:12)
Revised: (cid:12)

Original: τ c (cid:107) 22 measures the strength of the support intersection between y c,k and τ c .
Revised: τ c (cid:107) 22 measures the strength of the support intersection between y c,k and τ c .

Original: The true p ( τ c ) and τ c are not known and instead of estimating them explicitly, an approximation to
Revised: The true p ( τ c ) and τ c are not known and instead of estimating them explicitly, an approximation to

Original: D ( y c,k ; τ c ) is considered based only on the concentrations of the similarity on the support intersection and the expected strength of the support intersection for the transform data.
Revised: D ( y c,k ; τ c ) is considered based only on the concentrations of the similarity on the support intersection and the expected strength of the support intersection for the transform data.

Original: Non-parametric approximation We propose an approximation by sum of two expectations.
Revised: Non-parametric approximation We propose an approximation by sum of two expectations.

Original: The ﬁrst one is the expected similarity on the support intersection for the positive and negative component between all y c,k and the coresponding sets of the transform representations Y \ c that come from all classes c 1 different from c , i.e., c (cid:54) =
Revised: The ﬁrst one is the expected similarity on the support intersection for the positive and negative component between all y c,k and the coresponding sets of the transform representations Y \ c that come from all classes c 1 different from c , i.e., c (cid:54) =

Original: c 1 .
Revised: c 1 .

Original: The second is the expected strength on the support intersection between y c,k and the set of transform representations Y \ c that come from all classes cdifferent from c , c (cid:54) =
Revised: The second is the expected strength on the support intersection between y c,k and the set of transform representations Y \ c that come from all classes cdifferent from c , c (cid:54) =

Original: c 1 .
Revised: c 1 .

Original: We deﬁne the approximation as:
Revised: We deﬁne the approximation as:

Original: [Equation]
Revised: [Equation]

Original: we highlight that the transform represntations y c,k used in the approximation D P (cid:96) 1 ( X ) and S P (cid:96) 2 ( X ) are the result of applying the sparsifying transform with parameter set P to the data samples x c,k .
Revised: we highlight that the transform represntations y c,k used in the approximation D P (cid:96) 1 ( X ) and S P (cid:96) 2 ( X ) are the result of applying the sparsifying transform with parameter set P to the data samples x c,k .

Original: The m -th element y + c,k
Revised: The m -th element y + c,k

Original: ( m ) of y + c,k is deﬁned as y + c,k ( m ) =
Revised: ( m ) of y + c,k is deﬁned as y + c,k ( m ) =

Original: max( y c,k ( m ) , 0) and similarly,
Revised: max( y c,k ( m ) , 0) and similarly,

Original: [Equation]
Revised: [Equation]

Original: We also deﬁne the expected similarity using the positive and negative components of all y c,k across the transform representations Y \ c that come from the same classes c as D P (cid:96) 1 ,c ( X )
Revised: We also deﬁne the expected similarity using the positive and negative components of all y c,k across the transform representations Y \ c that come from the same classes c as D P (cid:96) 1 ,c ( X )

Original: =
Revised: =

Original: (cid:80) c (cid:80)
Revised: (cid:80) c (cid:80)

Original: Kk =1 (cid:80) k 1 ∈{{ 1 , 2 ,...,K }\ k } (cid:16) (cid:107) y
Revised: Kk =1 (cid:80) k 1 ∈{{ 1 , 2 ,...,K }\ k } (cid:16) (cid:107) y

Original: + c,k (cid:12)
Revised: + c,k (cid:12)

Original: y + c,k 1 (cid:107) 

Revised: y + c,k 1 (cid:107)

Original: + (cid:107) y − c,k (cid:12) y − c,k 1 (cid:107) 1 (cid:17) .
Revised: + (cid:107) y − c,k (cid:12) y − c,k 1 (cid:107) 1 (cid:17) .

Original: If the measure D P (cid:96) 1 ( X ) is not used then the approximation (5) is most similar to the one proposed in Liu et al.
Revised: If the measure D P (cid:96) 1 ( X ) is not used then the approximation (5) is most similar to the one proposed in Liu et al.

Original: By (5) a link is established between the non-linear transform models and the sparsifying transform model, or more generaly, assuming A is known, between a parametric and a non-parametric modeling view.
Revised: By (5) a link is established between the non-linear transform models and the sparsifying transform model, or more generaly, assuming A is known, between a parametric and a non-parametric modeling view.

Original: The terms
Revised: The terms

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:80)
Revised: (cid:80)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: c
Revised: c

Original: [Equation]
Revised: [Equation]

Original: ,k
Revised: ,k

Original: [Equation]
Revised: [Equation]

Original: and 2 (( C 1) K )( CK ) (cid:80) c 1 ,c 1 (cid:54) =
Revised: and 2 (( C 1) K )( CK ) (cid:80) c 1 ,c 1 (cid:54) =

Original: c (cid:80) k 1 y c 1 ,k 1 (cid:12) y c 1 ,k 1 are seen as ﬁnite sample estimates 4 5 of the positive, negative components and the Hadamard square, τ − c , τ + c and τ c (cid:12) τ c , respectivly, for the unknown variable τ c , ∀ c ∈ C .
Revised: c (cid:80) k 1 y c 1 ,k 1 (cid:12) y c 1 ,k 1 are seen as ﬁnite sample estimates 4 5 of the positive, negative components and the Hadamard square, τ − c , τ + c and τ c (cid:12) τ c , respectivly, for the unknown variable τ c , ∀ c ∈ C .

Original: Note that the Fisher discriminate constraint Yang et al. (2011b), the pairwise constraint Guo et al.
Revised: Note that the Fisher discriminate constraint Yang et al. (2011b), the pairwise constraint Guo et al.

Original: (2013) and the support intersection constraint Liu et al.
Revised: (2013) and the support intersection constraint Liu et al.

Original: (2016) are all approximations of a discriminative prior.
Revised: (2016) are all approximations of a discriminative prior.

Original: However, they all have speciﬁc assumptions on the distribution of the data representation in the transform domain.
Revised: However, they all have speciﬁc assumptions on the distribution of the data representation in the transform domain.

Original: The advantage of using (5) is that the approximation is without any prior to the probability distributions p ( τ c ) and without any explicit assumption about the metric/measure, or space/manifold in the transform domain.
Revised: The advantage of using (5) is that the approximation is without any prior to the probability distributions p ( τ c ) and without any explicit assumption about the metric/measure, or space/manifold in the transform domain.

Original: Given a training set X , the learning perspective w.r.t.
Revised: Given a training set X , the learning perspective w.r.t.

Original: a discriminative property is to estimate a non-linear transform models P c =
Revised: a discriminative property is to estimate a non-linear transform models P c =

Original: { A ∈ (cid:60) M × N , τ c ∈ (cid:60) M } that minimize the empirical expectation 1 CK (cid:80)
Revised: { A ∈ (cid:60) M × N , τ c ∈ (cid:60) M } that minimize the empirical expectation 1 CK (cid:80)

Original: [Equation]
Revised: [Equation]

Original: ,k min 1 ≤
Revised: ,k min 1 ≤

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: D
Revised: D

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: τ c ) ∼ E [min 1 ≤ c ≤ C D ( y c,k ; τ c )] .
Revised: τ c ) ∼ E [min 1 ≤ c ≤ C D ( y c,k ; τ c )] .

Original: Moreover, if the corresponding class labels for the training set X are given 6 then (cid:80) c,k min 1 ≤ c ≤ C D
Revised: Moreover, if the corresponding class labels for the training set X are given 6 then (cid:80) c,k min 1 ≤ c ≤ C D

Original: ( y c,k ; τ c ) equals to (cid:80) c,k D ( y c,k ; τ c ) and exactly matches the empirical expectation D P (cid:96) 1 ( X ) + S P (cid:96) 2 ( X ) .
Revised: ( y c,k ; τ c ) equals to (cid:80) c,k D ( y c,k ; τ c ) and exactly matches the empirical expectation D P (cid:96) 1 ( X ) + S P (cid:96) 2 ( X ) .

Original: The testing model Assume that the transform matrix A and the parameters τ 1 , τ 2 , ..., τ C are known, then given any data sample x c,k the transform representaton y c,k is estimated as:
Revised: The testing model Assume that the transform matrix A and the parameters τ 1 , τ 2 , ..., τ C are known, then given any data sample x c,k the transform representaton y c,k is estimated as:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: which is euqialent to min 1 ≤ c 1 ≤ C (cid:0) min y c,k (cid:107)
Revised: which is euqialent to min 1 ≤ c 1 ≤ C (cid:0) min y c,k (cid:107)

Original: Ax c,k − y c
Revised: Ax c,k − y c

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: 22 + λ
Revised: 22 + λ

Original: [Equation]
Revised: [Equation]

Original: D
Revised: D

Original: [Equation]
Revised: [Equation]

Original: y c
Revised: y c

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: 1 ) + λ 1 (cid:107) y c,k (cid:107) 1 (cid:1) .
Revised: 1 ) + λ 1 (cid:107) y c,k (cid:107) 1 (cid:1) .

Original: Furtheremore, if we assume that the measures D ( y c,k ; τ c 1 ) are zero, i.e. , D ( y c,k ; τ c 1 ) = 0 , then the discriminative prior is non-informative, in a sence that it has no inﬂuence in the models.
Revised: Furtheremore, if we assume that the measures D ( y c,k ; τ c 1 ) are zero, i.e. , D ( y c,k ; τ c 1 ) = 0 , then the discriminative prior is non-informative, in a sence that it has no inﬂuence in the models.

Original: Only then do the non-linear transforms reduce to the sparsifying transform model, since (6) reduce to:
Revised: Only then do the non-linear transforms reduce to the sparsifying transform model, since (6) reduce to:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Considering the testing stage, we note that the result (5) sheds light on another view.
Revised: Considering the testing stage, we note that the result (5) sheds light on another view.

Original: Namely, the sparsifying transform model P = { A ∈ (cid:60) M × N , λ 1 ∈ (cid:60) M } is also seen as an approximation to the models represented by a set of parameters P c
Revised: Namely, the sparsifying transform model P = { A ∈ (cid:60) M × N , λ 1 ∈ (cid:60) M } is also seen as an approximation to the models represented by a set of parameters P c

Original: =
Revised: =

Original: { A ∈ (cid:60) M × N , τ c ∈ (cid:60) M } with expected loss in the discriminative properties of the transform representations expressed by the similarity concentration measure D P (cid:96) 1 ( X ) + S P (cid:96) 2 ( X ) .
Revised: { A ∈ (cid:60) M × N , τ c ∈ (cid:60) M } with expected loss in the discriminative properties of the transform representations expressed by the similarity concentration measure D P (cid:96) 1 ( X ) + S P (cid:96) 2 ( X ) .

Original: At the same time this measure can also be considered as an empirical risk Vapnik (1995) w.r.t.
Revised: At the same time this measure can also be considered as an empirical risk Vapnik (1995) w.r.t.

Original: the discriminative properties, related to the generalization capabilities
Revised: the discriminative properties, related to the generalization capabilities

Original: Vapnik (1995) and Mark (2010) of the sparsifying transform model.
Revised: Vapnik (1995) and Mark (2010) of the sparsifying transform model.

Original: Note that the same model is also the simplest that approximates the non-linear transform models used in the learning stage.
Revised: Note that the same model is also the simplest that approximates the non-linear transform models used in the learning stage.

Original: 2. T HE LEARNING ALGORITHM
Revised: 2. T HE LEARNING ALGORITHM

Original: In summary, the used priors are:
Revised: In summary, the used priors are:

Original: [Equation]
Revised: [Equation]

Original: 4 Since
Revised: 4 Since

Original: [Equation]
Revised: [Equation]

Original: Additionally, we have a prior on A that penalizes the information loss in order to avoid trivially unwanted matrices A , i.e. , matrices that have repeated or zero rows.
Revised: Additionally, we have a prior on A that penalizes the information loss in order to avoid trivially unwanted matrices A , i.e. , matrices that have repeated or zero rows.

Original: The prior is deﬁned as:
Revised: The prior is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where the (cid:107) A (cid:107) F penalty helps regularize the scale ambiguity, the log | det ( A T A ) | and (cid:107) A (cid:107) 2 F are functions of the singular values of A and together help regularize the conditioning of A .
Revised: where the (cid:107) A (cid:107) F penalty helps regularize the scale ambiguity, the log | det ( A T A ) | and (cid:107) A (cid:107) 2 F are functions of the singular values of A and together help regularize the conditioning of A .

Original: Assuming that the expected coherence µ 2 ( A ) between the rows a m of A ( i.e. , A T =
Revised: Assuming that the expected coherence µ 2 ( A ) between the rows a m of A ( i.e. , A T =

Original: [ a 1 , a 2 , ..., a M ] ) is deﬁned as µ 2 ( A ) =
Revised: [ a 1 , a 2 , ..., a M ] ) is deﬁned as µ 2 ( A ) =

Original: [Equation]
Revised: [Equation]

Original: , .., M } .
Revised: , .., M } .

Original: Then (cid:107) AA T − I (cid:107) 2 F measures the expected coherence µ 2 ( A ) and the (cid:96) 2 norm for the rows of A .
Revised: Then (cid:107) AA T − I (cid:107) 2 F measures the expected coherence µ 2 ( A ) and the (cid:96) 2 norm for the rows of A .

Original: Note that the joint probability can be expressed as:
Revised: Note that the joint probability can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: where p ( x c,k , y c,k , τ c | A ) =
Revised: where p ( x c,k , y c,k , τ c | A ) =

Original: p ( x c,k | y c,k , A ) p ( τ c , | y c,k ) p ( y c,k ) , since p ( x c,k | y c,k , τ c , A ) = p ( x c,k | y c,k , A ) .
Revised: p ( x c,k | y c,k , A ) p ( τ c , | y c,k ) p ( y c,k ) , since p ( x c,k | y c,k , τ c , A ) = p ( x c,k | y c,k , A ) .

Original: Given the available training data set X , maximizing p ( x c,k , y c,k , τ c , A ) over Y and A is same as minimizing the following problem:
Revised: Given the available training data set X , maximizing p ( x c,k , y c,k , τ c , A ) over Y and A is same as minimizing the following problem:

Original: [Equation]
Revised: [Equation]

Original: where { λ 0 , λ 1 } are inversely proportional to the scaling parameters { β 2 , β 1 } .
Revised: where { λ 0 , λ 1 } are inversely proportional to the scaling parameters { β 2 , β 1 } .

Original: Note that the solution to (11) is not equivalent to the maximum a priory (MAP) solution, which is difﬁcult to compute, as it involves integrating over the vectors y c,k .
Revised: Note that the solution to (11) is not equivalent to the maximum a priory (MAP) solution, which is difﬁcult to compute, as it involves integrating over the vectors y c,k .

Original: Considering the optimization perspective, the problem is not convex in the variables ( Y , A ) .
Revised: Considering the optimization perspective, the problem is not convex in the variables ( Y , A ) .

Original: The proposed solution here is obtained by iteratively, marginally maximizing the probability p ( x c,k , y c,k , τ c , A ) over Y and A which is equivalent to maximizing the conditional densities p ( y c,k | x c,k , τ c , A ) and p ( A | x c,k , y c,k , τ c ) , respectively.
Revised: The proposed solution here is obtained by iteratively, marginally maximizing the probability p ( x c,k , y c,k , τ c , A ) over Y and A which is equivalent to maximizing the conditional densities p ( y c,k | x c,k , τ c , A ) and p ( A | x c,k , y c,k , τ c ) , respectively.

Original: Meaning that at one iterating step one of the variables Y or A is ﬁxed and w.r.t.
Revised: Meaning that at one iterating step one of the variables Y or A is ﬁxed and w.r.t.

Original: the other the problem (11) is minimized.
Revised: the other the problem (11) is minimized.

Original: The following describes the iterating steps that consist of linear map estimation (maximizing p ( A | x c,k , y c,k , τ c ) ) and discriminative encoding (maximizing p ( y c,k | x c,k , τ c , A ) ).
Revised: The following describes the iterating steps that consist of linear map estimation (maximizing p ( A | x c,k , y c,k , τ c ) ) and discriminative encoding (maximizing p ( y c,k | x c,k , τ c , A ) ).

Original: Linear map estimation :
Revised: Linear map estimation :

Original: Given the available data samples X and the corresponding transform representations Y the linear map A estimation problem reduces to:
Revised: Given the available data samples X and the corresponding transform representations Y the linear map A estimation problem reduces to:

Original: [Equation]
Revised: [Equation]

Original: where { λ 2 , λ 3 , λ 4 } are inversely proportional to the scaling parameters { β 3 , β 4 , β 5 } and we use the (cid:15) -close closed form solution estimated as follows:
Revised: where { λ 2 , λ 3 , λ 4 } are inversely proportional to the scaling parameters { β 3 , β 4 , β 5 } and we use the (cid:15) -close closed form solution estimated as follows:

Original: Proposition 1 ( (cid:15) -close closed form solution ): Given Y ∈ (cid:60) M × CK , ∀ X ∈ (cid:60) N × CK and M ≥ N , ∀ λ 2 ≥ 0 , λ 3 ≥ 0 and λ 4 ≥ 0 let the eigen value decomposition U X Σ X V TX of XX T +
Revised: Proposition 1 ( (cid:15) -close closed form solution ): Given Y ∈ (cid:60) M × CK , ∀ X ∈ (cid:60) N × CK and M ≥ N , ∀ λ 2 ≥ 0 , λ 3 ≥ 0 and λ 4 ≥ 0 let the eigen value decomposition U X Σ X V TX of XX T +

Original: λ 2 I and the singular value decomposition
Revised: λ 2 I and the singular value decomposition

Original: U
Revised: U

Original: U X
Revised: U X

Original: XY Σ U X XY V TU X XY of U TX XY T exist, then if and only if σ X ( n ) > 0 , ∀ n ∈ N = { 1 , 2 , 3 , ..., N } , (12) has (cid:15) -close approximative solution as :
Revised: XY Σ U X XY V TU X XY of U TX XY T exist, then if and only if σ X ( n ) > 0 , ∀ n ∈ N = { 1 , 2 , 3 , ..., N } , (12) has (cid:15) -close approximative solution as :

Original: [Equation]
Revised: [Equation]

Original: 13) where Σ A is diagonal matrix, Σ A ( n, n ) =
Revised: 13) where Σ A is diagonal matrix, Σ A ( n, n ) =

Original: σ A ( n ) ≥ 0 , and σ A ( n ) are solutions to quartic polynomials with global minimums (the proof is given in Appendix A.2).
Revised: σ A ( n ) ≥ 0 , and σ A ( n ) are solutions to quartic polynomials with global minimums (the proof is given in Appendix A.2).

Original: Discriminative encoding :
Revised: Discriminative encoding :

Original: Given the available data samples X and the current estimate of the transform A the discriminative representation estimation problem is formulated as ( P DR ) : min Y
Revised: Given the available data samples X and the current estimate of the transform A the discriminative representation estimation problem is formulated as ( P DR ) : min Y

Original: [Equation]
Revised: [Equation]

Original: AX
Revised: AX

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: D
Revised: D

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: τ
Revised: τ

Original: [Equation]
Revised: [Equation]

Original: ) + λ
Revised: ) + λ

Original: [Equation]
Revised: [Equation]

Original: y c,k (cid:107) .
Revised: y c,k (cid:107) .

Original: ( P DR ) .
Revised: ( P DR ) .

Original: Even not knowing p ( τ c ) or the model variables τ
Revised: Even not knowing p ( τ c ) or the model variables τ

Original: c we show that by the approximation (5), ( P DR ) has an efﬁcient solution.
Revised: c we show that by the approximation (5), ( P DR ) has an efﬁcient solution.

Original: Assuming that Y \ c is given, using the approximation (5), then for any sample k ∈ K from any class c ∈ C , problem ( P DR ) reduces to a constrained projection problem:
Revised: Assuming that Y \ c is given, using the approximation (5), then for any sample k ∈ K from any class c ∈ C , problem ( P DR ) reduces to a constrained projection problem:

Original: [Equation]
Revised: [Equation]

Original: and has a closed form solution as:
Revised: and has a closed form solution as:

Original: [Equation]
Revised: [Equation]

Original: where we abuse notation to denote | y c,k | as the vector having as elements the absolute values of the corresponding elements in y c,k , (cid:11) is a Haddamard division, g c = sign(max( Ax c,k , 0 ))
Revised: where we abuse notation to denote | y c,k | as the vector having as elements the absolute values of the corresponding elements in y c,k , (cid:11) is a Haddamard division, g c = sign(max( Ax c,k , 0 ))

Original: (cid:12) d
Revised: (cid:12) d

Original: + c + sign(max( − Ax c,k , 0 ))
Revised: + c + sign(max( − Ax c,k , 0 ))

Original: (cid:12) d − c , d + c = (cid:80) c
Revised: (cid:12) d − c , d + c = (cid:80) c

Original: 1 c 1 (cid:54) =
Revised: 1 c 1 (cid:54) =

Original: c (cid:80) k 1 y + c 1 ,k 1 , d − c
Revised: c (cid:80) k 1 y + c 1 ,k 1 , d − c

Original: =
Revised: =

Original: (cid:80) c
Revised: (cid:80) c

Original: 1 c 1 (cid:54) =
Revised: 1 c 1 (cid:54) =

Original: c (cid:80) k 1 y − c 1 ,k 1 and s c = (cid:80) c
Revised: c (cid:80) k 1 y − c 1 ,k 1 and s c = (cid:80) c

Original: 1 c 1 (cid:54) =
Revised: 1 c 1 (cid:54) =

Original: c (cid:80) k 1 y c 1 ,k 1 (cid:12) y c 1 ,k 1 ( the proof is given in Appendix B ).
Revised: c (cid:80) k 1 y c 1 ,k 1 (cid:12) y c 1 ,k 1 ( the proof is given in Appendix B ).

Original: We note that at convergence (which we do not prove here) we can only claim that a joint local maximum in ( Y , A ) of p ( x c,k , y c,k , τ c , A ) has been reached, even if, as in this case, each optimization step achieves the (marginal) (cid:15) -close and the global optimal solution, respectively.
Revised: We note that at convergence (which we do not prove here) we can only claim that a joint local maximum in ( Y , A ) of p ( x c,k , y c,k , τ c , A ) has been reached, even if, as in this case, each optimization step achieves the (marginal) (cid:15) -close and the global optimal solution, respectively.

Original: 2. A MEASURE FOR THE DISCRIMINATIVE PROPERTIES AND ITS BOUNDS
Revised: 2. A MEASURE FOR THE DISCRIMINATIVE PROPERTIES AND ITS BOUNDS

Original: This paper proposes a notion for the discriminative properties of a data set under a non-linear transform named as discrimination power, based on a measure for the relations between the concentrations D P (cid:96) 1 ,c ( X ) and D P (cid:96) 1 ( X ) .
Revised: This paper proposes a notion for the discriminative properties of a data set under a non-linear transform named as discrimination power, based on a measure for the relations between the concentrations D P (cid:96) 1 ,c ( X ) and D P (cid:96) 1 ( X ) .

Original: Proposition 2 : The discrimination power for any dataset Y ∈ (cid:60) M × CK under a non-linear transform with parameter set P is deﬁned as:
Revised: Proposition 2 : The discrimination power for any dataset Y ∈ (cid:60) M × CK under a non-linear transform with parameter set P is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where B M = { A t ∈ (cid:60) M × M , 0 ∈ (cid:60) M } , A t =
Revised: where B M = { A t ∈ (cid:60) M × M , 0 ∈ (cid:60) M } , A t =

Original: I and (cid:15) > 0 is a small positive constant .
Revised: I and (cid:15) > 0 is a small positive constant .

Original: Remark 1 : The advantage of this measure is that it logarithmically signiﬁes the difference between
Revised: Remark 1 : The advantage of this measure is that it logarithmically signiﬁes the difference between

Original: D P (cid:96) 1 ,c ( X ) and D P (cid:96) 1 ( X ) 7 .
Revised: D P (cid:96) 1 ,c ( X ) and D P (cid:96) 1 ( X ) 7 .

Original: The deﬁnition about the discrimination power of the original data set X , but, now under a model with a parameter set B N = { A o ∈ (cid:60) N × N , 0 ∈ (cid:60) N } where
Revised: The deﬁnition about the discrimination power of the original data set X , but, now under a model with a parameter set B N = { A o ∈ (cid:60) N × N , 0 ∈ (cid:60) N } where

Original: A o
Revised: A o

Original: =
Revised: =

Original: I is equivalent to the one deﬁned for I t .
Revised: I is equivalent to the one deﬁned for I t .

Original: We denote it as I o .
Revised: We denote it as I o .

Original: The bound on the discrimination power is given by the following result.
Revised: The bound on the discrimination power is given by the following result.

Original: Theorem 1 : The discrimination power for any data set X ∈ (cid:60) N × CK under any non-linear transform with parameter set P is bounded as :
Revised: Theorem 1 : The discrimination power for any data set X ∈ (cid:60) N × CK under any non-linear transform with parameter set P is bounded as :

Original: [Equation]
Revised: [Equation]

Original: The proof is given in Appendix D 8 .
Revised: The proof is given in Appendix D 8 .

Original: At ﬁrst the resulting bounds might look counterintuitive since the loss of information seems to increase the discrimination power.
Revised: At ﬁrst the resulting bounds might look counterintuitive since the loss of information seems to increase the discrimination power.

Original: This fact is true, however, up to a certain limit.
Revised: This fact is true, however, up to a certain limit.

Original: Therefore, it is important to distinguish two main conclusions.
Revised: Therefore, it is important to distinguish two main conclusions.

Original: First, for any model with a set of parameters P for which there is no loss of information, that is, no thresholding, the only condition for the increase in the discrimination power is D B N (cid:96) 1 ( X ) ≥ D B M (cid:96) 1 ( AX ) and D B N (cid:96) 1 ,c
Revised: First, for any model with a set of parameters P for which there is no loss of information, that is, no thresholding, the only condition for the increase in the discrimination power is D B N (cid:96) 1 ( X ) ≥ D B M (cid:96) 1 ( AX ) and D B N (cid:96) 1 ,c

Original: ( X ) ≤ D B M (cid:96) 1 ,c ( AX ) .
Revised: ( X ) ≤ D B M (cid:96) 1 ,c ( AX ) .

Original: Second, in the rest of the cases for which D B N (cid:96) 1 ( X ) ≥ D B M (cid:96) 1 ( Y ) and D B N (cid:96) 1 ,c ( X ) ≤ D B M (cid:96) 1 ,c ( Y ) holds true it will be possible to increase the discrimination power.
Revised: Second, in the rest of the cases for which D B N (cid:96) 1 ( X ) ≥ D B M (cid:96) 1 ( Y ) and D B N (cid:96) 1 ,c ( X ) ≤ D B M (cid:96) 1 ,c ( Y ) holds true it will be possible to increase the discrimination power.

Original: Moreover, there is a trade-off between the increase in discrimination power as a result of the loss of information as consequence of the non-linear thresholding operation.
Revised: Moreover, there is a trade-off between the increase in discrimination power as a result of the loss of information as consequence of the non-linear thresholding operation.

Original: ) , where τ c and υ c are unknown parameters.
Revised: ) , where τ c and υ c are unknown parameters.

Original: Then the differ-
Revised: Then the differ-

Original: [Table]
Revised: [Table]

Original: In Appendix C we also provide a sensitivity analysis that complements our result since it is related to the no- tion about the discriminative quality of the representation.
Revised: In Appendix C we also provide a sensitivity analysis that complements our result since it is related to the no- tion about the discriminative quality of the representation.

Original: The result in Appendix C also gives an informationtheoretic interpretation and information-geometric perspective about the model and the similarity concentration measure without the need of strict conditions for regularity, i.e., smoothness of the manifolds.
Revised: The result in Appendix C also gives an informationtheoretic interpretation and information-geometric perspective about the model and the similarity concentration measure without the need of strict conditions for regularity, i.e., smoothness of the manifolds.

Original: [Figure]
Revised: [Figure]

Original: C n ( A ) µ ( A ) t e [ min ]
Revised: C n ( A ) µ ( A ) t e [ min ]

Original: D2 .0 .5 .
Revised: D2 .0 .5 .

Original: D1 .0 .5 .
Revised: D1 .0 .5 .

Original: D2 .0 .6 .
Revised: D2 .0 .6 .

Original: D0 .0 .8 .
Revised: D0 .0 .8 .

Original: D6 .

Revised: D6 .

Original: D33 .0 .30 .
Revised: D33 .0 .30 .

Original: D1 .0 .5 .
Revised: D1 .0 .5 .

Original: I O 0 .0 .0 .0 .0 .0 .
Revised: I O 0 .0 .0 .0 .0 .0 .

Original: [Table]
Revised: [Table]

Original:  N UMERICAL EXPERIMENTS
Revised:  N UMERICAL EXPERIMENTS

Original: The numerical experiments are summarized in two different parts.
Revised: The numerical experiments are summarized in two different parts.

Original: In the ﬁrst series of the experiments the properties of the learned map A for the proposed algorithm are investigated.
Revised: In the ﬁrst series of the experiments the properties of the learned map A for the proposed algorithm are investigated.

Original: We evaluate the computational efﬁciency, as run time t e [ min ] , the conditioning number C n ( A ) =
Revised: We evaluate the computational efﬁciency, as run time t e [ min ] , the conditioning number C n ( A ) =

Original: λ max λ min , the expected mutual coherence µ ( A ) and the discrimination power across several databases for a learned non-linear transforms having different dimensionality.
Revised: λ max λ min , the expected mutual coherence µ ( A ) and the discrimination power across several databases for a learned non-linear transforms having different dimensionality.

Original: A comparison between the discrimination power uder different transforms is presented.
Revised: A comparison between the discrimination power uder different transforms is presented.

Original: The discriminatation power is estimated in the original domain, after transform by a random matrix (having Gaussian random samples as entries and transform dimension of M = 19000 ) and after a learned non-linear transform having transform dimension M = 19000 without and with discriminative prior, denoted as I 0 , I RT , I ST ∗ and I NT , respectively.
Revised: The discriminatation power is estimated in the original domain, after transform by a random matrix (having Gaussian random samples as entries and transform dimension of M = 19000 ) and after a learned non-linear transform having transform dimension M = 19000 without and with discriminative prior, denoted as I 0 , I RT , I ST ∗ and I NT , respectively.

Original: The second part evaluates a comparison of the discrimination power between the proposed algorithm and different DDL methods Ramirez et al.
Revised: The second part evaluates a comparison of the discrimination power between the proposed algorithm and different DDL methods Ramirez et al.

Original: (2010), Yang et al.
Revised: (2010), Yang et al.

Original: (2011a), Vu et al.
Revised: (2011a), Vu et al.

Original: (2015) and Vu & Monga (2016).
Revised: (2015) and Vu & Monga (2016).

Original: This comparison considers a setup where the used data sets are divided into a training and test set.
Revised: This comparison considers a setup where the used data sets are divided into a training and test set.

Original: Moreover, the learning is performed on the training set and the evaluation is performed on the test set.
Revised: Moreover, the learning is performed on the training set and the evaluation is performed on the test set.

Original: In the same series of experiments the recognition accuracy for the two data sets is also computed and compared.
Revised: In the same series of experiments the recognition accuracy for the two data sets is also computed and compared.

Original: Data sets and algorithms set up The used data sets are Extended YALE al
Revised: Data sets and algorithms set up The used data sets are Extended YALE al

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Nene et al.
Revised: Nene et al.

Original: (1996), Clatech 101 ( D 5 )
Revised: (1996), Clatech 101 ( D 5 )

Original: LeCun et al.
Revised: LeCun et al.

Original: (2008), UKB ( D 6 )
Revised: (2008), UKB ( D 6 )

Original: Nist ´ er & Stew ´ enius (2006) and MNIST ( D 7 ) Lecun & Cortes.
Revised: Nist ´ er & Stew ´ enius (2006) and MNIST ( D 7 ) Lecun & Cortes.

Original: All the images from the respective datasets were downscaled to resolutions 21 × 21 , 32 × 28 , 24 × 24 , 20 × 25 , 21 × 21 , 20 × 25 , 28 × 28 , respectively, and are normalized to unit variance.
Revised: All the images from the respective datasets were downscaled to resolutions 21 × 21 , 32 × 28 , 24 × 24 , 20 × 25 , 21 × 21 , 20 × 25 , 28 × 28 , respectively, and are normalized to unit variance.

Original: Considering the used implementation of the algorithm we note that the singular value decomposition for a large matrix has high computational complexity.
Revised: Considering the used implementation of the algorithm we note that the singular value decomposition for a large matrix has high computational complexity.

Original: However,
Revised: However,

Original: A − ˆ A , where ˆ A is estimated as a solution in the transform update step, can be considered as an proximal operator Parikh & Boyd (2014) for the gradient of the objective (12).
Revised: A − ˆ A , where ˆ A is estimated as a solution in the transform update step, can be considered as an proximal operator Parikh & Boyd (2014) for the gradient of the objective (12).

Original: Additionally, instead of using all of the available data samples X , a subset of them might be used.
Revised: Additionally, instead of using all of the available data samples X , a subset of them might be used.

Original: Therefore, one simple on-line variant for the update of A w.r.t.
Revised: Therefore, one simple on-line variant for the update of A w.r.t.

Original: a subset of the available training set has the form A t +1 =
Revised: a subset of the available training set has the form A t +1 =

Original: A t − ρ ( A t − ˆ A t ) with ρ a predeﬁned step size.
Revised: A t − ρ ( A t − ˆ A t ) with ρ a predeﬁned step size.

Original: In the numerical experiments we use the on-line variant of the algorithm (the convergence analysis for this variant of the algorithm is left for future work) were we used a baches of sizes equal to 10% − 12% of the total amount of the available training data.
Revised: In the numerical experiments we use the on-line variant of the algorithm (the convergence analysis for this variant of the algorithm is left for future work) were we used a baches of sizes equal to 10% − 12% of the total amount of the available training data.

Original: The parameters λ 

Revised: The parameters λ

Original: and λ 1 are set such that the resulting non-linear transform representation has a very small number of non-zeros w.r.t.
Revised: and λ 1 are set such that the resulting non-linear transform representation has a very small number of non-zeros w.r.t.

Original: the transform dimension.
Revised: the transform dimension.

Original: In the experiments this number is set to be 15 .
Revised: In the experiments this number is set to be 15 .

Original: The rest of the parameters are set as { λ 2 , λ 3 , λ 4 } = { 1000000 , 1000000 , 1000000 } .
Revised: The rest of the parameters are set as { λ 2 , λ 3 , λ 4 } = { 1000000 , 1000000 , 1000000 } .

Original: The algorithm is initialized with a random matrix having i.i.d.
Revised: The algorithm is initialized with a random matrix having i.i.d.

Original: Gaussian (zero mean, variance one) entries and is terminated after the 28 th iteration.
Revised: Gaussian (zero mean, variance one) entries and is terminated after the 28 th iteration.

Original: The results are obtained as the average of 3 runs.
Revised: The results are obtained as the average of 3 runs.

Original: An implementation presented in Vu & Monga (2016) was used to learn the dictionaries and estimate the
Revised: An implementation presented in Vu & Monga (2016) was used to learn the dictionaries and estimate the

Original: D
Revised: D

Original: [Table]
Revised: [Table]

Original: I DLSI
Revised: I DLSI

Original: [Table]
Revised: [Table]

Original: I FDDL
Revised: I FDDL

Original: [Table] [Table]
Revised: [Table] [Table]

Original: I COPAR
Revised: I COPAR

Original: [Table]
Revised: [Table]

Original: I LRSDL
Revised: I LRSDL

Original: [Table] [Table]
Revised: [Table] [Table]

Original: NT
Revised: NT

Original: [Table] [Table]
Revised: [Table] [Table]

Original: DLSI 96.
Revised: DLSI 96.

Original: [Table]
Revised: [Table]

Original: COPAR
Revised: COPAR

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: DLSI 98.
Revised: DLSI 98.

Original: FDDL 96.
Revised: FDDL 96.

Original: COPAR 96.
Revised: COPAR 96.

Original: LRSDL −
Revised: LRSDL −

Original: NT
Revised: NT

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: M
Revised: M

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: M
Revised: M

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: I
Revised: I

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: M
Revised: M

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: supervised
Revised: supervised

Original: Linear map properties, the similarity concentrations and the discrimination power The conditioning number and the expected coherence for the learned transforms are shown on Table 1.
Revised: Linear map properties, the similarity concentrations and the discrimination power The conditioning number and the expected coherence for the learned transforms are shown on Table 1.

Original: The learned transforms for all the databases have good conditioning numbers and low expected coherence.
Revised: The learned transforms for all the databases have good conditioning numbers and low expected coherence.

Original: The running time t e , measured in minutes, and the number of used dimensions, denoted as M are also shown in Table 1.
Revised: The running time t e , measured in minutes, and the number of used dimensions, denoted as M are also shown in Table 1.

Original: The learned transforms for all the data sets have relatively low execution time, regardless of the very high transform dimension M = 19000 .
Revised: The learned transforms for all the data sets have relatively low execution time, regardless of the very high transform dimension M = 19000 .

Original: The discrimination power is signiﬁcantly increased in the transform domain I NT compared to the one in the original domain
Revised: The discrimination power is signiﬁcantly increased in the transform domain I NT compared to the one in the original domain

Original: I O and is higher than I ST ∗ and I RT .The
Revised: I O and is higher than I ST ∗ and I RT .The

Original: evolution of the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96) 1 ( Y ) , their ratio C 1 /C 2 and the discrimination power log( C 1 /C 2) =
Revised: evolution of the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96) 1 ( Y ) , their ratio C 1 /C 2 and the discrimination power log( C 1 /C 2) =

Original: I t for subsets of the used databases after applying a non-linear transform with transform dimension M =is shown in Figure 2.
Revised: I t for subsets of the used databases after applying a non-linear transform with transform dimension M =is shown in Figure 2.

Original: It is important to note that the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and
Revised: It is important to note that the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and

Original: C 2 = D B M (cid:96) 

Revised: C 2 = D B M (cid:96)

Original: ( Y ) are decreasing, meaning that there is a loss of information.
Revised: ( Y ) are decreasing, meaning that there is a loss of information.

Original: However how this loss effects the resulting similarity concentration is crucial for the discrimination properties.
Revised: However how this loss effects the resulting similarity concentration is crucial for the discrimination properties.

Original: As shown in Figure 2, the slope of decrease for C 2 is stronger.
Revised: As shown in Figure 2, the slope of decrease for C 2 is stronger.

Original: Therefore, the discrimination power increases per iteration.
Revised: Therefore, the discrimination power increases per iteration.

Original: For the Coil- 20 ( D 4 ) database there is a ﬂuctuation.
Revised: For the Coil- 20 ( D 4 ) database there is a ﬂuctuation.

Original: This is explained by the fact that during learning we used a small number of data samples from the same database and that in the data there is high variability.
Revised: This is explained by the fact that during learning we used a small number of data samples from the same database and that in the data there is high variability.

Original: The conditioning number and the expected coherence for the learned transforms for all the databases at different transform dimensions M ∈ Q = { 100 , 1150 , 2200 , 3250 , 4300 , 5350 , 6400 , 7450 , 8500 , 9550 , 10600 , 11650 , 12700 , 13750 , 14800 , 15850 , 16900 , 17950 , 19000 } are shown in Figure 3.
Revised: The conditioning number and the expected coherence for the learned transforms for all the databases at different transform dimensions M ∈ Q = { 100 , 1150 , 2200 , 3250 , 4300 , 5350 , 6400 , 7450 , 8500 , 9550 , 10600 , 11650 , 12700 , 13750 , 14800 , 15850 , 16900 , 17950 , 19000 } are shown in Figure 3.

Original: We see that the value of both the conditioning number and the coherence is reducing and it converges to common values.
Revised: We see that the value of both the conditioning number and the coherence is reducing and it converges to common values.

Original: This conﬁrms the effectiveness of the conditioning and the coherence constraints.
Revised: This conﬁrms the effectiveness of the conditioning and the coherence constraints.

Original: The similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96) 

Revised: The similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96)

Original: ( Y ) , their ratio C 1 /C 2 and the discrimination power log( C 1 /C 2) =
Revised: ( Y ) , their ratio C 1 /C 2 and the discrimination power log( C 1 /C 2) =

Original: I t for a subsets of the used databases after applying a non-linear transform having transform dimensions M ∈ Q is shown in Figure 4.
Revised: I t for a subsets of the used databases after applying a non-linear transform having transform dimensions M ∈ Q is shown in Figure 4.

Original: We can see similar behavior as previous, that is, C 1 and C 1 are decreasing, but, the slope of decrease for C 2 is stronger.
Revised: We can see similar behavior as previous, that is, C 1 and C 1 are decreasing, but, the slope of decrease for C 2 is stronger.

Original: Therefore, the discrimination power increases as the transform dimension increases.
Revised: Therefore, the discrimination power increases as the transform dimension increases.

Original: NT vs DDL discrimination power and recognition performance
Revised: NT vs DDL discrimination power and recognition performance

Original: The proposed method is compared with DLSI Ramirez et al.
Revised: The proposed method is compared with DLSI Ramirez et al.

Original: , FDDL Yang et al.
Revised: , FDDL Yang et al.

Original: (2011a), COPAR Vu et al.
Revised: (2011a), COPAR Vu et al.

Original: and LRSDL Vu & Monga (2016).
Revised: and LRSDL Vu & Monga (2016).

Original: Half of the data samples from the data set Extended YALE B, sampled at random are used for learning and the remaining other half are used for evaluation.
Revised: Half of the data samples from the data set Extended YALE B, sampled at random are used for learning and the remaining other half are used for evaluation.

Original: Considering the MNIST database the training set is used for learning and the test set is used for both evaluating the discrimination power and the recognition accuracy.
Revised: Considering the MNIST database the training set is used for learning and the test set is used for both evaluating the discrimination power and the recognition accuracy.

Original: We compute both the discrimination power and the recognition accuracy on a subsets from the test sets.
Revised: We compute both the discrimination power and the recognition accuracy on a subsets from the test sets.

Original: The dictionary size (transform dimension M ) is set to be equal to { 150 , 75 , 1515 , 3825 , 570 , 150 , 300 } for the used databases, respectivly, in all of the comparing algorithms.
Revised: The dictionary size (transform dimension M ) is set to be equal to { 150 , 75 , 1515 , 3825 , 570 , 150 , 300 } for the used databases, respectivly, in all of the comparing algorithms.

Original: The discrimination power of the comparing methods is denoted as I DLSI , I FDDL , I COPAR
Revised: The discrimination power of the comparing methods is denoted as I DLSI , I FDDL , I COPAR

Original: and I LRSDL .
Revised: and I LRSDL .

Original: The recognition results for the methods DLSI ,
Revised: The recognition results for the methods DLSI ,

Original: FDDL , COPAR and LRSDL on the data sets Extended YALE B and MNIST were not computed here, rather we use the best reported result form the respective papers Ramirez et al.
Revised: FDDL , COPAR and LRSDL on the data sets Extended YALE B and MNIST were not computed here, rather we use the best reported result form the respective papers Ramirez et al.

Original: (2010), Yang et al.
Revised: (2010), Yang et al.

Original: (2011a), Vu et al.
Revised: (2011a), Vu et al.

Original: and Vu & Monga (2016).
Revised: and Vu & Monga (2016).

Original: Considering the proposed algorithm the non-linear transform was learned for the transform dimensions M = { 100 , 500 , 1500 , 4000 } and
Revised: Considering the proposed algorithm the non-linear transform was learned for the transform dimensions M = { 100 , 500 , 1500 , 4000 } and

Original: M = { 1000 , 4000 , 6000 , 12000 } , respectively, for the used data sets.
Revised: M = { 1000 , 4000 , 6000 , 12000 } , respectively, for the used data sets.

Original: After the transform was learned, the transform data samples were computed for the respective training and test sets.
Revised: After the transform was learned, the transform data samples were computed for the respective training and test sets.

Original: Then, the transform training data samples were used as features to learn a linear SVM classiﬁer in oneagainst-all regime.
Revised: Then, the transform training data samples were used as features to learn a linear SVM classiﬁer in oneagainst-all regime.

Original: The results are shown in Table 2 a), b) and c).
Revised: The results are shown in Table 2 a), b) and c).

Original: The discrimination power of the proposed non-linear transform is higher that the discrimination power of the comparing methods.
Revised: The discrimination power of the proposed non-linear transform is higher that the discrimination power of the comparing methods.

Original: The recognition accuracy is higher for high dimensionality of the proposed method and outperforms the DDL methods at dimensionality 4000 and 12000 .
Revised: The recognition accuracy is higher for high dimensionality of the proposed method and outperforms the DDL methods at dimensionality 4000 and 12000 .

Original: In Figure 5 and Figure 6 are shown the recognition accuracy and the expected loss measured as E [ (cid:107) z c,k M (cid:107) 22 ] =
Revised: In Figure 5 and Figure 6 are shown the recognition accuracy and the expected loss measured as E [ (cid:107) z c,k M (cid:107) 22 ] =

Original: E [ (cid:107)
Revised: E [ (cid:107)

Original: Ax c,k − y c,k (cid:107) 22 M ] as a linear function of the discrimination power evaluated at transform dimension M = { 100 , 500 , 1500 , 4000 } and M = { 1000 , 4000 , 6000 , 12000 } .
Revised: Ax c,k − y c,k (cid:107) 22 M ] as a linear function of the discrimination power evaluated at transform dimension M = { 100 , 500 , 1500 , 4000 } and M = { 1000 , 4000 , 6000 , 12000 } .

Original: It is interesting to highlight that as the discrimination power at different transform dimension increases it also increases the accuracy of recognition.
Revised: It is interesting to highlight that as the discrimination power at different transform dimension increases it also increases the accuracy of recognition.

Original: Moreover, the results on these two data sets show that this increase is approximately linear.
Revised: Moreover, the results on these two data sets show that this increase is approximately linear.

Original: On the other hand the expected loss decreases as the discrimination power at different transform dimensions increases.
Revised: On the other hand the expected loss decreases as the discrimination power at different transform dimensions increases.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper presented a novel approach for learning discriminative and sparse representations.
Revised: This paper presented a novel approach for learning discriminative and sparse representations.

Original: A novel discriminative prior was proposed and the properties of the models with the prior were investigated.
Revised: A novel discriminative prior was proposed and the properties of the models with the prior were investigated.

Original: A low complexity learning algorithm was presented.
Revised: A low complexity learning algorithm was presented.

Original: The preliminary results w.r.t.
Revised: The preliminary results w.r.t.

Original: the introduced measures and the recognition accuracy on the used databases showed promising performance.
Revised: the introduced measures and the recognition accuracy on the used databases showed promising performance.

Original: We showed that it is possible to increase the discrimination power with information loss.
Revised: We showed that it is possible to increase the discrimination power with information loss.

Original: Moreover, we highlight that when expanding to high dimensional space with non-linear transforms how the loss of information reﬂects the similarity concentrations is crucial for the discriminative properties.
Revised: Moreover, we highlight that when expanding to high dimensional space with non-linear transforms how the loss of information reﬂects the similarity concentrations is crucial for the discriminative properties.

Original: A study on the recognition capabilities for other databases are our next steps.
Revised: A study on the recognition capabilities for other databases are our next steps.

Original: An extention considering the sufﬁcient conditions for increase in discrimination power in the transform domain, under supervised and unsupervised case, together with an analysis for a deep architecture where per single layer we have a non-linear transforms are left for our future work.
Revised: An extention considering the sufﬁcient conditions for increase in discrimination power in the transform domain, under supervised and unsupervised case, together with an analysis for a deep architecture where per single layer we have a non-linear transforms are left for our future work.

Original: A PPENDIX
Revised: A PPENDIX

Original: [Equation]
Revised: [Equation]

Original: . A.1 T HE GLOBAL OPTIMAL SOLUTION
Revised: . A.1 T HE GLOBAL OPTIMAL SOLUTION

Original: Given X and the curent estimate of Y , the estimate of the transform is a solution to the following problem
Revised: Given X and the curent estimate of Y , the estimate of the transform is a solution to the following problem

Original: [Equation]
Revised: [Equation]

Original: Theorem 2 (golobal optimal solution) Given X ∈ (cid:60) N × CK and Y ∈ (cid:60) M × CK , if and only if the joint decomposition
Revised: Theorem 2 (golobal optimal solution) Given X ∈ (cid:60) N × CK and Y ∈ (cid:60) M × CK , if and only if the joint decomposition

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: exists, where U X ∈ (cid:60) N × N is orthonormal,
Revised: exists, where U X ∈ (cid:60) N × N is orthonormal,

Original: V XY ∈ (cid:60) M × N is per columns orthonormal and Σ X , Σ XY ∈ (cid:60) N × N are diagonal matrices with positive diagonal elements, then (18) has a global minimum as
Revised: V XY ∈ (cid:60) M × N is per columns orthonormal and Σ X , Σ XY ∈ (cid:60) N × N are diagonal matrices with positive diagonal elements, then (18) has a global minimum as

Original: [Equation]
Revised: [Equation]

Original: Σ A ( n, n ) =
Revised: Σ A ( n, n ) =

Original: σ A ( n ) , ∀ n, σ A ( n ) ≥ 0 and σ A ( n ) are positive solutions to
Revised: σ A ( n ) , ∀ n, σ A ( n ) ≥ 0 and σ A ( n ) are positive solutions to

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 2 Consider the equvalent trace form of (18)
Revised: Proof of Theorem 2 Consider the equvalent trace form of (18)

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that since λ 2 ≥ 0 , XX T + λ 2 I is a symetric positive deﬁnite matrix whit all eigenvalues non-negative, therfore it decomposes as
Revised: Note that since λ 2 ≥ 0 , XX T + λ 2 I is a symetric positive deﬁnite matrix whit all eigenvalues non-negative, therfore it decomposes as

Original: [Equation]
Revised: [Equation]

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: Then (18) equvalently is
Revised: Then (18) equvalently is

Original: [Equation]
Revised: [Equation]

Original: Asumme that B decomposes as
Revised: Asumme that B decomposes as

Original: [Equation]
Revised: [Equation]

Original: where Σ B is a diagonal matrix with positive diagonal elements, U B is column orthogonal and V B is orthogonal square matrix.
Revised: where Σ B is a diagonal matrix with positive diagonal elements, U B is column orthogonal and V B is orthogonal square matrix.

Original: Moreover, let the following decomposition on XY T exists
Revised: Moreover, let the following decomposition on XY T exists

Original: XY
Revised: XY

Original: [Equation]
Revised: [Equation]

Original: TXY , and substitute as U B =
Revised: TXY , and substitute as U B =

Original: [Equation]
Revised: [Equation]

Original: (28) then
Revised: (28) then

Original: Tr
Revised: Tr

Original: [Equation]
Revised: [Equation]

Original: The term
Revised: The term

Original: [Equation]
Revised: [Equation]

Original: (30) and
Revised: (30) and

Original: Tr {
Revised: Tr {

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Finnaly, (18) is reduced to
Revised: Finnaly, (18) is reduced to

Original: [Equation]
Revised: [Equation]

Original: equalling to zero the ﬁrst order derivative of the objective (33) w.r.t.
Revised: equalling to zero the ﬁrst order derivative of the objective (33) w.r.t.

Original: σ B ( n ) and multiplaing by σ B ( n ) gives
Revised: σ B ( n ) and multiplaing by σ B ( n ) gives

Original: [Equation]
Revised: [Equation]

Original: A closed form solution to (34) exists and depends on the discriminint of the quartic polynomial.
Revised: A closed form solution to (34) exists and depends on the discriminint of the quartic polynomial.

Original: Moreover, since 4 λ 3 σ 4 X ( n ) is positive a global minimum to (18) exists if and only if the decompsition (19) exists (cid:3)
Revised: Moreover, since 4 λ 3 σ 4 X ( n ) is positive a global minimum to (18) exists if and only if the decompsition (19) exists (cid:3)

Original: A.2 T HE (cid:15) - CLOSE CLOSED FORM APPROXIMATION
Revised: A.2 T HE (cid:15) - CLOSE CLOSED FORM APPROXIMATION

Original: Consider the equvalent trace form (26) of (18) and note that since λ 2 ≥ 0 , XX T + λ 2 I is a symetric positive deﬁnite matrix whit all eigenvalues non-negative, therfore it decomposes as
Revised: Consider the equvalent trace form (26) of (18) and note that since λ 2 ≥ 0 , XX T + λ 2 I is a symetric positive deﬁnite matrix whit all eigenvalues non-negative, therfore it decomposes as

Original: [Equation]
Revised: [Equation]

Original: Let the following decomposition exists
Revised: Let the following decomposition exists

Original: [Equation]
Revised: [Equation]

Original: Deﬁne
Revised: Deﬁne

Original: A = BD , where D =
Revised: A = BD , where D =

Original: U X
Revised: U X

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Assume that B decomposes as U B Σ B V TB = B , where Σ B is a diagonal matrix with positive diagonal elements, U B is column orthogonal and V B is orthogonal square matrix and let
Revised: Assume that B decomposes as U B Σ B V TB = B , where Σ B is a diagonal matrix with positive diagonal elements, U B is column orthogonal and V B is orthogonal square matrix and let

Original: [Equation]
Revised: [Equation]

Original: then
Revised: then

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Consider the decomposition U B Σ B V TB of B , use Mirsky (1959) and Neumann (1937) and note that
Revised: Consider the decomposition U B Σ B V TB of B , use Mirsky (1959) and Neumann (1937) and note that

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where Σ Γ is a diagonal matrix, having diagonal elements Σ Γ ( n, n )
Revised: where Σ Γ is a diagonal matrix, having diagonal elements Σ Γ ( n, n )

Original: =
Revised: =

Original: σ Γ ( n ) =
Revised: σ Γ ( n ) =

Original: T ( n, n ) , ∀ n ∈ N and T =
Revised: T ( n, n ) , ∀ n ∈ N and T =

Original: U
Revised: U

Original: U X
Revised: U X

Original: XY Σ U X XY V TU X XY .
Revised: XY Σ U X XY V TU X XY .

Original: Note that the term Tr { ( AX )( AX ) T } =
Revised: Note that the term Tr { ( AX )( AX ) T } =

Original: Tr { BB T } =
Revised: Tr { BB T } =

Original: Tr { Σ 2 B } and as in the Apendix subsection A.

Revised: Tr { Σ 2 B } and as in the Apendix subsection A.

Original: Tr { AA T } =
Revised: Tr { AA T } =

Original: Tr { BDD T B T } =
Revised: Tr { BDD T B T } =

Original: Tr { Σ 2 B Σ − 2 X } ,
Revised: Tr { Σ 2 B Σ − 2 X } ,

Original: Tr { ( AA T )( AA T ) T } =
Revised: Tr { ( AA T )( AA T ) T } =

Original: Tr { ( BDD T B T )( BDD T B T ) } =
Revised: Tr { ( BDD T B T )( BDD T B T ) } =

Original: Tr { Σ 4 B Σ − 4 X } and log | det AA T | = log | det D T B T BD | = log | det U X Σ − 1 X Σ 2 B Σ − 1 X U TX | = log | det Σ − 2 X Σ 2 B | .
Revised: Tr { Σ 4 B Σ − 4 X } and log | det AA T | = log | det D T B T BD | = log | det U X Σ − 1 X Σ 2 B Σ − 1 X U TX | = log | det Σ − 2 X Σ 2 B | .

Original: Finally, the aproximation of (18) using the bound (41) is reduced to min σ B ( n )
Revised: Finally, the aproximation of (18) using the bound (41) is reduced to min σ B ( n )

Original: [Equation]
Revised: [Equation]

Original: equalling to zero the ﬁrst order derivative of the objective (42) w.r.t.
Revised: equalling to zero the ﬁrst order derivative of the objective (42) w.r.t.

Original: σ B ( n ) and multiplaing by σ B ( n ) gives
Revised: σ B ( n ) and multiplaing by σ B ( n ) gives

Original: [Equation]
Revised: [Equation]

Original: A closed form solution to (43) exists and depends on the discriminint of the quartic polynomial.
Revised: A closed form solution to (43) exists and depends on the discriminint of the quartic polynomial.

Original: Moreover, since 4 λ 3 σ 4 X ( n ) is positive a global minimum to (42) exists.
Revised: Moreover, since 4 λ 3 σ 4 X ( n ) is positive a global minimum to (42) exists.

Original: Therfore, having the decomposition U B Σ B V TB = B , the substitutions U B =
Revised: Therfore, having the decomposition U B Σ B V TB = B , the substitutions U B =

Original: ( U U X
Revised: ( U U X

Original: XY V TU X XY ) T and V B
Revised: XY V TU X XY ) T and V B

Original: = U X with the solution of (42) gives the (cid:15) -close closed form approximative solution to problem (18) as
Revised: = U X with the solution of (42) gives the (cid:15) -close closed form approximative solution to problem (18) as

Original: [Equation]
Revised: [Equation]

Original: where the bound (41) implies that the (cid:15) -close closed form approximative solution is a lower bound to the solution of (18) (cid:3)
Revised: where the bound (41) implies that the (cid:15) -close closed form approximative solution is a lower bound to the solution of (18) (cid:3)

Original: A PPENDIX B.
Revised: A PPENDIX B.

Original: Let y c,k{+c 1 ,k 
+} = y + c,k + y − c,k , y + c,k ∈ (cid:60) M +
Revised: Let y {+c 1 ,k 
+} = y + c 1 ,k 1 + y − c 1 ,k 1 , y + c 1 ,k 1 ∈ (cid:60) M +

Original: and y − c,k ∈ (cid:60) M − .
Revised: and y − c 1 ,k 1 ∈ (cid:60) M − .

Original: Consider thesimilarity concentration measure 
Revised: Consider the measure D P(cid:96) 1 ( X ) D P(cid:96) 1 ( X ) =

Revised: (cid:88) c 1 ,c 2 c 1 (cid:54) =

Revised: c(cid:88)

Revised: k 1 ,k(cid:107)

Original: [Equation]
Revised: y + c 1 ,k 1 (cid:12) y + c 2 ,k 2 (cid:107) 1 + (cid:88) c 1 ,c 2 c 1 (cid:54) = c(cid:88) k 1 ,k(cid:107)

Revised: y − c 1 ,k 1 (cid:12) y − c 2 ,k 2 (cid:107) 1 = (cid:88) c 1 ,c 2 c 1 (cid:54) =

Original:  c 
Revised: c(cid:88) k 1 ,k| y + c 1 ,k 1 | T | y + c 2 ,k 2 | + (cid:88) c 1 ,c 2 c 1 (cid:54) = c(cid:88)

Revised: k 1 ,k|

Original: [Equation]
Revised: [Equation]

Revised: Let A and Y \{ c 1 ,k 1 } be given then problem P DP has only one varible y c 1 ,k 1 .

Revised: Conseqently in (45), y c 1 ,k 1 is releted with only a part of the transform representations in D P(cid:96)

Revised: 1 ( X ) , the rest are constants for the reduced problem P DP , in particulary we have

Original: [Equation]
Revised: [Equation]

Original: ) where d + c = (cid:80) c 
Revised: where d + c = (cid:80) c 2 , c

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Revised: c 1 (cid:80) k

Revised: | y +

Original: [Equation]
Revised: [Equation]

Revised: k

Original: [Equation]
Revised: [Equation]

Original: k 1 ,k 2 | y + c 2 ,k 2 | , d − c 
Revised: | , d − c = [Equation]

Original: =
Revised: =

Original: (cid:80) c

Original: [Equation]
Revised: [Equation]

Original: c 2 c
Revised: k

Original: [Equation]
Revised: [Equation]

Original: = c 2 (cid:80) k 1 ,k 2 | y − c 2 ,k 2 | and we abuse notation by denoting | y c 1 ,k 1 | as the vector whose elements are the absolute values of the elements in y c 1 ,k 1 .
Revised: | y − c 2 ,k 2 | and we abuse notation by denoting | y c 1 ,k 1 | as the vector whose elements are the absolute values of the elements in y c 1 ,k 1 .

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: 46) where s c = (cid:80) c 1 ,
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: c
Revised: simmilary as in (46) we have

Original: [Equation]
Revised: [Equation]

Original:  =
Revised: 48) where s c = (cid:80) c 2 c 2 (cid:54) =

Original: c 2 (cid:80) k1 ,k 2 y c 2 ,k 2 (cid:12) y c 2 ,k 2 .
Revised: c 1 (cid:80) k 2 y c 2 ,k 2 (cid:12) y c 2 ,k 2 .

Original: Denote q c 1 ,k 

Revised: Denote q c 1 ,k

Original: =
Revised: =

Original: Ax c 1 ,k 1 and consider the problemmin
Revised: Ax c 1 ,k 1 and consider the problem

Original: [Equation]
Revised: [Equation]

Original: 47) by taking the ﬁrst order derivative w.r.t.
Revised: by taking the ﬁrst order derivative w.r.t.

Original: y c 1 ,k 

Revised: y c 1 ,k

Original: we have that
Revised: we have that

Original: [Equation]
Revised: [Equation]

Original: λ
Revised: 50) take sign magnitude decomposition of y c 1 ,k 1 = sign( y c 1 ,k 1 ) (cid:12) | y c 1 ,k 1 | then we have

Original: [Equation]
Revised: [Equation]

Original:  ( y
Revised: [Figure] (cid:96) [Figure] ( Y )

Original: [Equation]
Revised: [Figure]

Original: = 0 , (48) take sign magnitude decomposition of y c 1 ,k 1 = sign( y c 1 ,k 1 ) (cid:12) | y c 1 ,k 1 | then we have sign( y c 1 ,k 1 ) (cid:12) | y c 1 ,k 1 | (cid:12) ( 1 + 2 λ 0 s c ) − sign( q c 1 ,k 1 ) (cid:12) | q c 1 ,k 1 | + λ 0 (sign( y + c 1 ,k 1 ) (cid:12) d + c + sign( y − c 1 ,k 1 ) (cid:12) d − c ) + λ 1 sign( y c 1 ,k 1 ) = 0 .
Revised: (cid:15) [Figure]

Original: Let the sign of y c 1 ,k 1 , i.e. sign( y c 1 ,k 1 ) be equal to the sign of sign( q c 1 ,k 1 ) , and Hadamar multiply from the left side by sign( q c 1 ,k 1 ) then we have
Revised: Let the sign of y c 1 ,k 1 , i.e. sign( y c 1 ,k 1 ) be equal to the sign of sign( q c 1 ,k 1 ) , and Hadamar multiply from the left side by sign( q c 1 ,k 1 ) then we have

Original: [Equation]
Revised: [Equation]

Original: [Figure] note that sign( q c 1 ,k 1 ) (cid:12) sign( q + c 1 ,k 1 )
Revised: 52) note that sign( q c 1 ,k 1 ) (cid:12) sign( q + c 1 ,k 1 )

Original: = sign( q + c 1 ,k 1 ) and that sign( q c 1 ,k 1 ) (cid:12) sign( q − c 1 ,k 1 ) =
Revised: = sign( q + c 1 ,k 1 ) and that sign( q c 1 ,k 1 ) (cid:12) sign( q − c 1 ,k 1 ) =

Original: sign( − q − c 1 ,k 1 ) , theretofore we have 
Revised: sign( − q − c 1 ,k 1 ) , theretofore we have [Equation] − λ

Original: [Equation]
Revised: [Equation]

Original: (51) since the magnitude might be only positive we have that | y c 1 ,k 1 | (cid:12) ( 1 + 2 λ 0 s c ) = max( | q c 1 ,k 1 | − λ 0 (sign( q + c 1 ,k 1 ) (cid:12) d + c + sign( − q − c 1 ,k 1 ) (cid:12) d − c ) − λ 1 1 , 0 ) .
Revised: 1 , (53) since the magnitude might be only positive we have that | y c 1 ,k 1 | (cid:12) ( 1 + 2 λ 0 s c ) = max( | q c 1 ,k 1 | − λ 0 (sign( q + c 1 ,k 1 ) (cid:12) d + c + sign( − q − c 1 ,k 1 ) (cid:12) d − c ) − λ 1 1 , 0 ) .

Original: Denote g c
Revised: Denote g c

Original: = (sign( q + c 1 ,k 1 ) (cid:12) d
Revised: = (sign( q + c 1 ,k 1 ) (cid:12) d

Original: + c + sign( − q − c 1 ,k 1 ) (cid:12) d − c )
Revised: + c + sign( − q − c 1 ,k 1 ) (cid:12) d − c )

Original: then the closed form solution to (47) is:
Revised: then the closed form solution to (49) is:

Original: [Equation]
Revised: [Equation]

Original: which completes the proof (cid:3)
Revised: which completes the proof (cid:3)

Original: A PPENDIX C. S ENSITIVITY ANALYSIS AND INTERPRETATIONS
Revised: A PPENDIX C. S ENSITIVITY ANALYSIS AND INTERPRETATIONS

Original: The similarity concentration measure provides possibility to measure the discriminative properties, their deviation, increase (or decrease) and the corresponding relations between different non-linear transform models across one domain or different domains, thereby quantifying their quality w.r.t.
Revised: The similarity concentration measure provides possibility to measure the discriminative properties, their deviation, increase (or decrease) and the corresponding relations between different non-linear transform models across one domain or different domains, thereby quantifying their quality w.r.t.

Original: the discriminative properties.
Revised: the discriminative properties.

Original: C.1 S ENSITIVITY ANALYSIS W . R . T . THE SIMILARITY CONCENTRATION MEASURES
Revised: C.1 S ENSITIVITY ANALYSIS W . R . T . THE SIMILARITY CONCENTRATION MEASURES

Original: An illustration about the deﬁnition of discriminative power given by a diagram is shown in Figure 7.
Revised: An illustration about the deﬁnition of discriminative power given by a diagram is shown in Figure 7.

Original: To measure the ability for an increase in discriminative properties by a non-linear transform 9 we ﬁrst have to deﬁne a notion for the discriminative properties on a data set under different nonlinear transform models.
Revised: To measure the ability for an increase in discriminative properties by a non-linear transform 9 we ﬁrst have to deﬁne a notion for the discriminative properties on a data set under different nonlinear transform models.

Original: Therefore, ﬁrst we introduce the ”special” base models and then analyze the properties of the similarity concentration measures under the change in model parameter and the relation between the base model and the proposed non-linear transform model deﬁned by a parameter set P = { A ∈ (cid:60) M × N , τ ∈ (cid:60) M } .
Revised: Therefore, ﬁrst we introduce the ”special” base models and then analyze the properties of the similarity concentration measures under the change in model parameter and the relation between the base model and the proposed non-linear transform model deﬁned by a parameter set P = { A ∈ (cid:60) M × N , τ ∈ (cid:60) M } .

Original: Any data set X in the original domain might have a transform model with parameters B N = { A o ∈ (cid:60) N × N , τ = 0 ∈ (cid:60) N + } , if A o = I ∈
Revised: Any data set X in the original domain might have a transform model with parameters B N = { A o ∈ (cid:60) N × N , τ = 0 ∈ (cid:60) N + } , if A o = I ∈

Original: D N + we refer to it as a base original model .
Revised: D N + we refer to it as a base original model .

Original: Similarly as in the original domain, any data set Y in the transform domain might have a transform model with parameters B M =
Revised: Similarly as in the original domain, any data set Y in the transform domain might have a transform model with parameters B M =

Original: { A t ∈ (cid:60) M × M , τ = 0 ∈ (cid:60) M + } , if A t = I ∈
Revised: { A t ∈ (cid:60) M × M , τ = 0 ∈ (cid:60) M + } , if A t = I ∈

Original: D M
Revised: D M

Original: + we refer to it as a base transform model .
Revised: + we refer to it as a base transform model .

Original: Any base model, deﬁned ether in the original domain B N or in the transform domain B M , has domain equal to the co-domain, since x c,k = T B N ( x c,k ) and y c,k = T B M ( y c,k ) holds trivially, for the respective sets of parameters B N =
Revised: Any base model, deﬁned ether in the original domain B N or in the transform domain B M , has domain equal to the co-domain, since x c,k = T B N ( x c,k ) and y c,k = T B M ( y c,k ) holds trivially, for the respective sets of parameters B N =

Original: { A o = I ∈
Revised: { A o = I ∈

Original: D N + , τ = 0 ∈ (cid:60) N + } and
Revised: D N + , τ = 0 ∈ (cid:60) N + } and

Original: B M =
Revised: B M =

Original: { A t = I ∈
Revised: { A t = I ∈

Original: D M + , τ = 0 ∈ (cid:60) M + } .
Revised: D M + , τ = 0 ∈ (cid:60) M + } .

Original: This is illustrated with a diagram shown in Figure 8. 
Revised: This is illustrated with a diagram shown in Figure 8. [Figure]

Original: A base original model provides a possibility to compare it with any other non-linear transform model with parameters P =
Revised: A base original model provides a possibility to compare it with any other non-linear transform model with parameters P =

Original: { A ∈ (cid:60) M × N , τ ∈ (cid:60) M + } .
Revised: { A ∈ (cid:60) M × N , τ ∈ (cid:60) M + } .

Original: Additionally, note that for B M = { A t = I ∈
Revised: Additionally, note that for B M = { A t = I ∈

Original: D M , τ = 0 ∈ (cid:60) M } we have that D B M (cid:96) 1 ,c ( Y ) =
Revised: D M , τ = 0 ∈ (cid:60) M } we have that D B M (cid:96) 1 ,c ( Y ) =

Original: D P (cid:96) 1 ,c ( X ) and that D B M (cid:96) 1 ( Y ) =
Revised: D P (cid:96) 1 ,c ( X ) and that D B M (cid:96) 1 ( Y ) =

Original: D P (cid:96) 1 ( X ) .
Revised: D P (cid:96) 1 ( X ) .

Original: It implies[Figure] that the similarity concentrations can be analyzed as a function in the original domain under model
Revised: It implies that the similarity concentrations can be analyzed as a function in the original domain under model

Original: P or in the transform domain under model B M .
Revised: P or in the transform domain under model B M .

Original: The main relations considering the preservation of change in the similarity concentration between two models, deﬁned not necessary in the same domain are stated by Lemma 1 .
Revised: The main relations considering the preservation of change in the similarity concentration between two models, deﬁned not necessary in the same domain are stated by Lemma 1 .

Original: Lemma 1 : The non-linear transform model (2) totally preserves the information in the change for the similarity concentration for a data set X w.r.t.
Revised: Lemma 1 : The non-linear transform model (2) totally preserves the information in the change for the similarity concentration for a data set X w.r.t.

Original: a small change in the parameters of the models
Revised: a small change in the parameters of the models

Original: B N , B M and P if (cid:107) δ o (cid:107) ∗ = 0 and (cid:107) δ t (cid:107) ∗ = 0 as 
Revised: B N , B M and P if (cid:107) δ o (cid:107) ∗ = 0 and (cid:107) δ t (cid:107) ∗ = 0 as ( R 1) : A [Equation] R 2) :

Revised: A (cid:32) ∂D

Revised: P (cid:96)

Original: [Equation]
Revised: [Equation]

Revised: c ( X ) ∂

Original: A
Revised: A

Revised: +

Revised: [Equation]

Revised: ( X ) ∂

Revised: A (cid:33) T = ∂D

Original: [Equation]
Revised: [Equation]

Revised: M

Revised: [Equation]

Revised: [Equation]

Revised: c

Revised: [Equation]

Original: R 3) :
Revised: Y ) ∂ A t | A t = I

Revised: [Equation]

Revised: ∂D

Revised: B M (cid:96) 1 ( Y )

Original: [Equation]
Revised: [Equation]

Revised: A t | A t =

Revised: I

Revised: [Equation]

Revised: t ( R

Revised: [Equation]

Revised: ) : A δ To = ∂D B

Revised: M

Revised: [Equation]

Revised: [Equation]

Revised: | A

Revised: [Equation]

Revised: ∂D

Revised: B

Revised: [Equation]

Revised: Z

Revised: [Equation]

Revised: ∂

Revised: [Equation]

Original: 53) where 
Revised: t | A [Equation] , (55) where ∂D P

Revised: [Equation]

Revised: ∂D

Revised: [Equation]

Revised: ( X ) ∂

Revised: A

Original: [Equation]
Revised: [Equation]

Revised: (cid:88) c,c(cid:88) k,

Revised: k 1 k (cid:54) =

Revised: ky c 1 ,k 1 x Tc

Revised: [Equation]

Revised: k + y c

Revised: [Equation]

Revised: k x Tc

Original: [Equation]
Revised: [Equation]

Revised: k∂D B M (cid:96)

Revised: [Equation]

Revised: c ( Y ) ∂

Revised: A t | A t

Revised: = I + ∂D

Revised: B M (cid:96)

Revised: [Equation]

Revised: ( Y )

Revised: [Equation]

Revised: t | A t =

Revised: I

Revised: [Equation]

Revised: (cid:88) c

Revised: [Equation]

Revised: c(cid:88) k,k 1 k (cid:54) =

Revised: ky c

Revised: [Equation]

Revised: k 1 y Tc

Revised: [Equation]

Revised: k

Revised: [Equation]

Original: c
Revised: c

Original: [Equation]
Revised: [Equation]

Revised: y

Revised: [Equation]

Revised: [Equation]

Revised: k∂D B M

Revised: [Equation]

Revised: [Equation]

Revised: ( Z ) ∂

Revised: A t | A

Revised: [Equation]

Revised: =

Revised: I + ∂D

Revised: B M (cid:96)

Revised: [Equation]

Revised: ( Z ) ∂

Revised: A t | A t =

Revised: I = (cid:88) c,c(cid:88) k,k 1 k (cid:54) =

Revised: kz c 1 ,k 1 z

Revised: Tc,k

Revised: [Equation]

Revised: z c

Revised: [Equation]

Original: k
Revised: k

Revised: [Equation]

Revised: [Equation]

Revised: k∂D

Original: [Equation]
Revised: [Equation]

Revised: N (cid:96)

Original: k 1 k
Revised: [Equation] c ( X ) ∂ A o | A o = I = (cid:88) c (cid:88)

Original: [Equation]
Revised: [Equation]

Original: The proof is given in Appendix C.2 .
Revised: The proof is given in Appendix C.2 .

Original: The terms z c,k represent the non-linear transform error vectors that appear in the model
Revised: The terms z c,k represent the non-linear transform error vectors that appear in the model

Original: Ax c,k = y c,k + z c,k as a result of applying an element-wise non-liner operation H τ to Ax c,k , i.e. , y c,k =
Revised: Ax c,k = y c,k + z c,k as a result of applying an element-wise non-liner operation H τ to Ax c,k , i.e. , y c,k =

Original: H τ ( Ax c,k ) .
Revised: H τ ( Ax c,k ) .

Original: As an example in the sparsifying transform model z c,k is the ”loss of information”, that is the information about the values of the elements in Ax c,k that are discarded.
Revised: As an example in the sparsifying transform model z c,k is the ”loss of information”, that is the information about the values of the elements in Ax c,k that are discarded.

Original: The terms δ o and δ t correlate the errors z c,k with the original data x c 1 ,k 1 and transform data y c 1 ,k 1 , respectively.
Revised: The terms δ o and δ t correlate the errors z c,k with the original data x c 1 ,k 1 and transform data y c 1 ,k 1 , respectively.

Original: Note that if there is no loss of information (in the earlier example it means that there is no thresholding and just a simple linear transform model is used) then δ o = 

Revised: Note that if there is no loss of information (in the earlier example it means that there is no thresholding and just a simple linear transform model is used) then δ o =

Original: and δ t = 0 .
Revised: and δ t = 0 .

Original: Moreover, δ o and δ t bear important information about the discriminative properties in the transform domain.
Revised: Moreover, δ o and δ t bear important information about the discriminative properties in the transform domain.

Original: The terms
Revised: The terms

Original: [Equation]
Revised: [Equation]

Original: and ∂D 
Revised: and ∂D P (cid:96) 1 ( X )

Original: [Equation]
Revised: [Equation]

Original:  represent the change of the similarity concentrations under inﬁnitesimally small change of the parameter A from the model P .
Revised: A represent the change of the similarity concentrations under inﬁnitesimally small change of the parameter A from the model P .

Original: The terms ∂D B N (cid:96) 1 ,c ( X ) ∂
Revised: The terms ∂D B N (cid:96) 1 ,c ( X ) ∂

Original: A o |
Revised: A o |

Original: A
Revised: A

Original: o =
Revised: o =

Original: I and ∂DB N
Revised: I and ∂D

Original: [Equation]
Revised: [Equation]

Original: ∂

Original: A
Revised: N

Original: [Equation]
Revised: [Equation]

Original: | A o
Revised: | A o

Original: =
Revised: =

Original: I have dual interpretation.
Revised: I have dual interpretation.

Original: Assuming metric
Revised: Assuming metric

Original: A o
Revised: A o

Original: = I , then the ﬁrst one is considered as a change of the similarity concentrations under inﬁnitesimally small change of the space metric, or equivalently under small metric perturbation.
Revised: = I , then the ﬁrst one is considered as a change of the similarity concentrations under inﬁnitesimally small change of the space metric, or equivalently under small metric perturbation.

Original: Conversely, assuming the data samples are distributed under a Gaussian distribution with parameters identity covariance matrix and zero mean, i.e. x c,k ∼ N ( µ = 0 , Σ = I ) , then ∂D B N (cid:96) 1 ,c ( X [Equation]
Revised: Conversely, assuming the data samples are distributed under a Gaussian distribution with parameters identity covariance matrix and zero mean, i.e. x c,k ∼ N ( µ = 0 , Σ = I ) , then ∂D B N (cid:96) 1 ,c ( X )

Original: ∂
Revised: ∂

Original: A o |
Revised: A o |

Original: [Equation]

Original: and

Original: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: o = I and ∂D B N (cid:96) 1 ( X ) ∂

Original: | A o 
Revised: A o |

Original:  =
Revised: A o =

Original: I represent the change of the similarity concentrations under small change in the assumption away from a Gaussian distribution.
Revised: I represent the change of the similarity concentrations under small change in the assumption away from a Gaussian distribution.

Original: Equation ( R 1) relates the base transform for the original domain B N with any arbitrary transform deﬁned in the original domain P .
Revised: Equation ( R 1) relates the base transform for the original domain B N with any arbitrary transform deﬁned in the original domain P .

Original: The relation ( R 2) is a result about the preservation of change in the similarity concentration between two models B M and P deﬁned on two different domains.
Revised: The relation ( R 2) is a result about the preservation of change in the similarity concentration between two models B M and P deﬁned on two different domains.

Original: Whereas ( R 3) gives the preservation of change in the similarity concentration between the error in the transform domain.
Revised: Whereas ( R 3) gives the preservation of change in the similarity concentration between the error in the transform domain.

Original: The next result highlights the relation between: the linear projection ( by the linear map A that appears in the model P ) of the change in the similarity concentration under the model B N in the original domain and the change of the similarity concentration under the model B M in the transform domain .
Revised: The next result highlights the relation between: the linear projection ( by the linear map A that appears in the model P ) of the change in the similarity concentration under the model B N in the original domain and the change of the similarity concentration under the model B M in the transform domain .

Original: This relation exists independently for∂D
Revised: This relation exists independently for

Original: [Equation]
Revised: [Equation]

Original: |

Original: [Equation]

Original: and

Original: [Equation]

Original: | A o
Revised: | A o

Original: =
Revised: =

Original: I , nevertheless, we will deﬁne the summarized and the independent versions.
Revised: I , nevertheless, we will deﬁne the summarized and the independent versions.

Original: Therefore, ﬁrst we deﬁne ∂ J (cid:96) 1 ( X ) ∂
Revised: Therefore, ﬁrst we deﬁne ∂ J (cid:96) 1 ( X ) ∂

Original: A o |
Revised: A o |

Original: A o = I = ∂D 
Revised: A o = I = ∂D B N

Original: [Equation]
Revised: [Equation]

Revised: 1 ,c

Original: [Equation]
Revised: [Equation]

Original:  |
Revised: X [Equation] ∂ A o |

Revised: [Equation]

Revised: +

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: |A
Revised: |

Original: [Equation]
Revised: [Equation]

Original: = I and rewrite ( R 1) as A ∂ J ( X ) ∂A o |
Revised: and rewrite ( R 1) as A ∂ J ( X ) ∂

Original: A
Revised: A

Original: o =
Revised: [Equation]

Original: I − δ
Revised: I [Equation]

Original: o =
Revised: o =

Original: ∂D

Original: P

Original: [Equation]
Revised: [Equation]

Original: 1 ,c
Revised: X

Original: [Equation]
Revised: [Equation]

Original: X ) ∂

Original:  A +∂D
Revised: ∂ A +

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: A , then replace ∂D P 
Revised: , then replace ∂D P (cid:96)

Original: [Equation]
Revised: [Equation]

Original: c ( X [Equation]
Revised: c ( X ) ∂

Original: ∂ A+
Revised: A

Original: [Equation]
Revised: [Equation]

Original: X ) ∂ A in ( R 2) by the same term in ( R 1) , use ( R 3) , reorder and we have the following result.
Revised: A in ( R 2) by the same term in ( R 1) , use ( R 3) , reorder and we have the following result.

Original: Lemma 2 : For ﬁxed τ any non-linear transform model (2) preservers the information in the change of similarity concentrations w.r.t.
Revised: Lemma 2 : For ﬁxed τ any non-linear transform model (2) preservers the information in the change of similarity concentrations w.r.t.

Original: a small change in A by
Revised: a small change in A by

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: The expressions ( R 4) actually relate the metric in the original domain under the model B N to the induced metric in the transform domain for the model B M , with induction done by the model P with parameter set { A ∈ (cid:60) M × N , τ ∈ (cid:60) M + } .
Revised: The expressions ( R 4) actually relate the metric in the original domain under the model B N to the induced metric in the transform domain for the model B M , with induction done by the model P with parameter set { A ∈ (cid:60) M × N , τ ∈ (cid:60) M + } .

Original: Moreover, the model P might describe a transform domain with a non-smooth manifold.
Revised: Moreover, the model P might describe a transform domain with a non-smooth manifold.

Original: Since the manifolds of the original and the transform domain under the models B N and B M are smooth the analysis of their relations reveals insights about the relation between the manifolds under the models B N and P .
Revised: Since the manifolds of the original and the transform domain under the models B N and B M are smooth the analysis of their relations reveals insights about the relation between the manifolds under the models B N and P .

Original: The terms ∂ J (cid:96) 

Revised: The terms ∂ J (cid:96)

Original: ( Z ) ∂
Revised: ( Z ) ∂

Original: A t | A t
Revised: A t | A t

Original: =
Revised: =

Original: I and ∂ J (cid:96) 

Revised: I and ∂ J (cid:96)

Original: ( Y ; Z )
Revised: ( Y ; Z )

Original: ∂
Revised: ∂

Original: A t | A t =
Revised: A t | A t =

Original: I carry out the information about the breaks and the discontinuities of the regularity and smoothness in the manifold induced by the model P .
Revised: I carry out the information about the breaks and the discontinuities of the regularity and smoothness in the manifold induced by the model P .

Original: Also we note that
Revised: Also we note that

Original: [Equation]
Revised: [Equation]

Original: n J
Revised: n J

Original: [Equation]
Revised: [Equation]

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 1 J (cid:96) 1 ( X ) ∂ n
Revised: 1 J (cid:96) 1 ( X ) ∂ n

Original: [Equation]
Revised: [Equation]

Original: 1 A o |
Revised: 1 A o |

Original: A
Revised: A

Original: o =
Revised: o =

Original: I and
Revised: I and

Original: [Equation]
Revised: [Equation]

Original: n J (cid:96) 1 ( Y )
Revised: n J (cid:96) 1 ( Y )

Original: [Equation]
Revised: [Equation]

Original: n A t | A t = I = 4 ∂ n − 1 J (cid:96) 1 ( Y ) ∂ n − 1 A t | A t
Revised: n A t | A t = I = 4 ∂ n − 1 J (cid:96) 1 ( Y ) ∂ n − 1 A t | A t

Original: =
Revised: =

Original: I therefore,
Revised: I therefore,

Original: [Equation]
Revised: [Equation]

Original: J (cid:96) 1 ( X )
Revised: J (cid:96) 1 ( X )

Original: [Equation]
Revised: [Equation]

Original: A o |
Revised: A o |

Original: A o = I =
Revised: A o = I =

Original: [Equation]
Revised: [Equation]

Original: 2 J (cid:96) 1 ( X ) ∂ 2 A o |
Revised: 2 J (cid:96) 1 ( X ) ∂ 2 A o |

Original: A
Revised: A

Original: o =
Revised: o =

Original: I and
Revised: I and

Original: [Equation]
Revised: [Equation]

Original: J (cid:96) 1 ( Y )
Revised: J (cid:96) 1 ( Y )

Original: [Equation]
Revised: [Equation]

Original: A t | A t = I = 14 ∂ 2 J (cid:96) 1 ( Y ) ∂
Revised: A t | A t = I = 14 ∂ 2 J (cid:96) 1 ( Y ) ∂

Original: 2 A t | A t
Revised: 2 A t | A t

Original: = I might be interpreted as
Revised: = I might be interpreted as

Original: Fisher information matrices evaluated at A o = I ∈
Revised: Fisher information matrices evaluated at A o = I ∈

Original: D N and A t
Revised: D N and A t

Original: = I ∈ D M .
Revised: = I ∈ D M .

Original: Furthermore, if (cid:107) ∂
Revised: Furthermore, if (cid:107) ∂

Original: J (cid:96) 1 ( Z ) ∂
Revised: J (cid:96) 1 ( Z ) ∂

Original: A t | A t = I (cid:107) ∗ = 0 and (cid:107) ∂
Revised: A t | A t = I (cid:107) ∗ = 0 and (cid:107) ∂

Original: J
Revised: J

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: A t
Revised: A t

Original: [Equation]
Revised: [Equation]

Original: , then ( R 4) in information geometry is seen as change of coordinates on a manifold, where the intrinsic properties of curvature remain unchanged under different parametrization.
Revised: , then ( R 4) in information geometry is seen as change of coordinates on a manifold, where the intrinsic properties of curvature remain unchanged under different parametrization.

Original: A PPENDIX C. 
Revised: A PPENDIX C.

Original: Note that for the model P t we have that
Revised: Note that for the model P t we have that

Original: [Equation]
Revised: [Equation]

Original: since sign
Revised: since sign

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst order derivative of the divergence D P t (cid:96) 1 ( x ; g ) w.r.t.
Revised: The ﬁrst order derivative of the divergence D P t (cid:96) 1 ( x ; g ) w.r.t.

Original: the parameter A is:
Revised: the parameter A is:

Original: [Equation]
Revised: [Equation]

Original: we assume that the threshold parameter τ is chosen such that the vector
Revised: we assume that the threshold parameter τ is chosen such that the vector

Original: | Ax | − τ (or for any other q , the vector | Aq | − τ ) has least one non-zero element, then
Revised: | Ax | − τ (or for any other q , the vector | Aq | − τ ) has least one non-zero element, then

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: combining (60) and (61) we have that
Revised: combining (62) and (63) we have that

Original: [Equation]
Revised: [Equation]

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: 62) where
Revised: 64) where

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: ∀ m ∈ M .
Revised: ∀ m ∈ M .

Original: Similarity, note that for the model P o 0 = { A o , τ = 0 } we have that
Revised: Similarity, note that for the model P o 0 = { A o , τ = 0 } we have that

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst order derivative of the divergence D P o 0 (cid:96) 1 ( x ; g )
Revised: The ﬁrst order derivative of the divergence D P o 0 (cid:96) 1 ( x ; g )

Original: [Equation]
Revised: [Equation]

Original: .r.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  note that at A o
Revised: 67) note that at A o

Original: =
Revised: =

Original: I , P t 0 =
Revised: I , P t 0 =

Original: B N
Revised: B N

Original: and we have that
Revised: and we have that

Original: [Equation]
Revised: [Equation]

Original: Also for the model P t 

Revised: Also for the model P t

Original: = { A t , τ = 0 } we have that
Revised: = { A t , τ = 0 } we have that

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst order derivative of the divergence D P t 0 (cid:96) 1 ( y ; q ) w.r.t A t is:
Revised: The ﬁrst order derivative of the divergence D P t 0 (cid:96) 1 ( y ; q ) w.r.t A t is:

Original: [Equation]
Revised: [Equation]

Original: 68) note that at A t
Revised: 70) note that at A t

Original: = I , P t 0 =
Revised: = I , P t 0 =

Original: B M and we have that
Revised: B M and we have that

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Consider the following
Revised: Consider the following

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: A closer look at (70) reveals us that
Revised: A closer look at (72) reveals us that

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: By similar construction applied to the rest of the pairs of data samples we have the result:
Revised: By similar construction applied to the rest of the pairs of data samples we have the result:

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: A closer look at (75) reveals us
Revised: A closer look at (77) reveals us

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: By similar construction applied to the rest of the pairs of data samples we have the result:
Revised: By similar construction applied to the rest of the pairs of data samples we have the result:

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: A closer look at (80) reveals us
Revised: A closer look at (82) reveals us

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: By similar construction applied to the rest of the pairs of data samples we have the result:
Revised: By similar construction applied to the rest of the pairs of data samples we have the result:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: A PPENDIX D
Revised: A PPENDIX D

Original: The result in by Lemma 2 (56) decomposes on the contribution components for simmilarity and on the contributiong components for dissimilarity
Revised: The result in by Lemma 2 (58) decomposes on the contribution components for simmilarity and on the contributiong components for dissimilarity

Original: [Equation]
Revised: [Equation]

Original: Moreover
Revised: Moreover

Original: [Equation]
Revised: [Equation]

Original: w.r.t.
Revised: w.r.t.

Original: the simmilarity concentrations we have the following decompositions
Revised: the simmilarity concentrations we have the following decompositions

Original: [Equation]
Revised: [Equation]

Original: therfore we have the following bounds
Revised: therfore we have the following bounds

Original: [Equation]
Revised: [Equation]

Original: Additionaly, we have that c : λ min
Revised: Additionaly, we have that c : λ min

Original: [Equation]
Revised: [Equation]

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: where λ min ( A T A ) is the minimum singlular value of the matrix A T A .
Revised: where λ min ( A T A ) is the minimum singlular value of the matrix A T A .

Original: Taking the logarithm of the ratio D P (cid:96) 1 ,c ( X ) D P (cid:96) 1 ( X )+ (cid:15) using the bounds a , b and c we arrive at the desired result (cid:3)
Revised: Taking the logarithm of the ratio D P (cid:96) 1 ,c ( X ) D P (cid:96) 1 ( X )+ (cid:15) using the bounds a , b and c we arrive at the desired result (cid:3)

Original: A PPENDIX E
Revised: A PPENDIX E

Original: The exact steps of the proposed non-linear transform learning are described by Algorithm 1.
Revised: The exact steps of the proposed non-linear transform learning are described by Algorithm 1.

Original: A PPENDIX F
Revised: A PPENDIX F

Original: The ratio C 1 /C 2 between the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96) 

Revised: The ratio C 1 /C 2 between the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and C 2 = D B M (cid:96)

Original: ( Y ) and the discrimination power log( C 1 /C 2) =
Revised: ( Y ) and the discrimination power log( C 1 /C 2) =

Original: I t on subsets of the used databases after applying a nonlinear transform with transform dimension M = 19000 and varying the thresholding parameter τ = λ 1 is shown in Figure 9.
Revised: I t on subsets of the used databases after applying a nonlinear transform with transform dimension M = 19000 and varying the thresholding parameter τ = λ 1 is shown in Figure 9.

Original: We used 70 different values for the parameter λ , sampled uniformly from the interval (cid:0) 0 , (max 1 ≤ c ≤ C, 1 ≤ k ≤ K max 1 ≤ m ≤ M | a Tm x c,k | ) (cid:1) .
Revised: We used 70 different values for the parameter λ , sampled uniformly from the interval (cid:0) 0 , (max 1 ≤ c ≤ C, 1 ≤ k ≤ K max 1 ≤ m ≤ M | a Tm x c,k | ) (cid:1) .

Original: The results were obtained using
Revised: The results were obtained using

Original: Input X , λ 0 , λ 1 , λ 2 , λ 3 , λ
Revised: Input X , λ 0 , λ 1 , λ 2 , λ 3 , λ

Original: A ← inicialize repeat
Revised: A ← inicialize repeat

Original: DISCRIMINATIVE ENCODING closed form solution per data sample
Revised: DISCRIMINATIVE ENCODING closed form solution per data sample

Original: Y ← AX repeatfor ∀ c ∈ C do d − c ← (cid:80) c 1 c 1 (cid:54) =
Revised: Y ← AX repeatfor ∀ c ∈ C do d − c ← (cid:80) c 1 c 1 (cid:54) =

Original: c (cid:80) k 1 y − c 1 ,k 1 , d + c ← (cid:80) c 1 c 1 (cid:54) =
Revised: c (cid:80) k 1 y − c 1 ,k 1 , d + c ← (cid:80) c 1 c 1 (cid:54) =

Original: c (cid:80) k 1 y + c 1 ,k 1 and s c ← (cid:80) c 1 c 1 (cid:54) =
Revised: c (cid:80) k 1 y + c 1 ,k 1 and s c ← (cid:80) c 1 c 1 (cid:54) =

Original: c (cid:80) k 1 y c 1 ,k 1 (cid:12) y c 1 ,kend for for ∀ c ∈ C and ∀ k ∈ K do g ← sign(max( Ax c,k , 0 ))
Revised: c (cid:80) k 1 y c 1 ,k 1 (cid:12) y c 1 ,kend for for ∀ c ∈ C and ∀ k ∈ K do g ← sign(max( Ax c,k , 0 ))

Original: (cid:12) d
Revised: (cid:12) d

Original: + c + sign(max( − Ax c,k , 0 ))
Revised: + c + sign(max( − Ax c,k , 0 ))

Original: (cid:12) d − c y c,k ← sign( Ax c,k ) (cid:12) max ( Ax c,k − λ 0 g + λ 1 1 , 0 ) (cid:11) (1 + 2 λ 0 s c ) end for until convergence
Revised: (cid:12) d − c y c,k ← sign( Ax c,k ) (cid:12) max ( Ax c,k − λ 0 g + λ 1 1 , 0 ) (cid:11) (1 + 2 λ 0 s c ) end for until convergence

Original: TRANSFORM UPDATE (cid:15) -close closed form solution
Revised: TRANSFORM UPDATE (cid:15) -close closed form solution

Original: [Equation]
Revised: [Equation]

Original: where σ
Revised: where σ

Original: [Equation]
Revised: [Equation]

Original: until convergence
Revised: until convergence

Original: Output A , Y
Revised: Output A , Y

Original: [Figure]
Revised: [Figure]

Original: a non-linear transform learned with one value for parameter λ for all the databases.
Revised: a non-linear transform learned with one value for parameter λ for all the databases.

Original: Since all the databases have different variabilities and the amount of available data is different, this result suggest that per different database there should be different optimal values for the parameter λ .
Revised: Since all the databases have different variabilities and the amount of available data is different, this result suggest that per different database there should be different optimal values for the parameter λ .

Original: The results about the ratio C 1 /C 2 between the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and
Revised: The results about the ratio C 1 /C 2 between the similarity concentrations C 1 = D B M (cid:96) 1 ,c ( Y ) and

Original: C 2 = D B M (cid:96) 

Revised: C 2 = D B M (cid:96)

Original: ( Y ) and the discrimination power log( C 1 /C 2)
Revised: ( Y ) and the discrimination power log( C 1 /C 2)

Original: =
Revised: =

Original: I t on the Extanded-Yale-B and the MNIST data sets after applying a non-linear transform with transform dimensions M = { 100 , 500 , 1500 , 4000 } and M = { 1000 , 4000 , 6000 , 12000 } , and varying the thresholding parameter τ = λ 1 are shown in Figure 10.
Revised: I t on the Extanded-Yale-B and the MNIST data sets after applying a non-linear transform with transform dimensions M = { 100 , 500 , 1500 , 4000 } and M = { 1000 , 4000 , 6000 , 12000 } , and varying the thresholding parameter τ = λ 1 are shown in Figure 10.

Original: We again used 70 different values for the parameter λ , sampled uniformly from the interval (cid:0) 0 , (max 1 ≤ c ≤ C, 1 ≤ k ≤ K max 1 ≤ m ≤ M | a Tm x c,k | ) (cid:1) .
Revised: We again used 70 different values for the parameter λ , sampled uniformly from the interval (cid:0) 0 , (max 1 ≤ c ≤ C, 1 ≤ k ≤ K max 1 ≤ m ≤ M | a Tm x c,k | ) (cid:1) .

Original: The results were obtained using a non-linear transforms learned with optimally choose values (by using crossvalidation) of the parameter λ for the two different databases.
Revised: The results were obtained using a non-linear transforms learned with optimally choose values (by using crossvalidation) of the parameter λ for the two different databases.

Original: As expected, we can see that the extreme points of the ratio between the (cid:96) ∗ -norms C 1 and C 2 of the similarity concentrations and the discrimination power is around the optimal values of the parameter τ = λ 1 .
Revised: As expected, we can see that the extreme points of the ratio between the (cid:96) ∗ -norms C 1 and C 2 of the similarity concentrations and the discrimination power is around the optimal values of the parameter τ = λ 1 .