Original: D EEP L EARNING M EETS P ROJECTIVE C LUSTERING
Revised: D EEP L EARNING M EETS P ROJECTIVE C LUSTERING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A common approach for compressing Natural Language Processing (NLP) networks is to encode the embedding layer as a matrix A ∈ R n × d , compute its rank- j approximation A j via SVD (Singular Value Decomposition), and then factor A j into a pair of matrices that correspond to smaller fully-connected layers to replace the original embedding layer.
Revised: A common approach for compressing Natural Language Processing (NLP) networks is to encode the embedding layer as a matrix A ∈ R n × d , compute its rank- j approximation A j via SVD (Singular Value Decomposition), and then factor A j into a pair of matrices that correspond to smaller fully-connected layers to replace the original embedding layer.

Original: Geometrically, the rows of A represent points in R d , and the rows of A j represent their projections onto the j -dimensional subspace that minimizes the sum of squared distances (“errors”) to the points.
Revised: Geometrically, the rows of A represent points in R d , and the rows of A j represent their projections onto the j -dimensional subspace that minimizes the sum of squared distances (“errors”) to the points.

Original: In practice, these rows of A may be spread around k > 1 subspaces, so factoring A based on a single subspace may lead to large errors that turn into large drops in accuracy.
Revised: In practice, these rows of A may be spread around k > 1 subspaces, so factoring A based on a single subspace may lead to large errors that turn into large drops in accuracy.

Original: Inspired by projective clustering from computational geometry, we suggest replacing this subspace by a set of k subspaces, each of dimension j , that minimizes the sum of squared distances over every point (row in A ) to its closest subspace.
Revised: Inspired by projective clustering from computational geometry, we suggest replacing this subspace by a set of k subspaces, each of dimension j , that minimizes the sum of squared distances over every point (row in A ) to its closest subspace.

Original: Based on this approach, we provide a novel architecture that replaces the original embedding layer by a set of k small layers that operate in parallel and are then recombined with a single fully-connected layer.
Revised: Based on this approach, we provide a novel architecture that replaces the original embedding layer by a set of k small layers that operate in parallel and are then recombined with a single fully-connected layer.

Original: Extensive experimental results on the GLUE benchmark yield networks that are both more accurate and smaller compared to the standard matrix factorization (SVD).
Revised: Extensive experimental results on the GLUE benchmark yield networks that are both more accurate and smaller compared to the standard matrix factorization (SVD).

Original: For example, we further compress DistilBERT by reducing the size of the embedding layer by 40% while incurring only a 0 .
Revised: For example, we further compress DistilBERT by reducing the size of the embedding layer by 40% while incurring only a 0 .

Original: 5% average drop in accuracy over all nine GLUE tasks, compared to a 2 . 8% drop using the existing SVD approach.
Revised: 5% average drop in accuracy over all nine GLUE tasks, compared to a 2 . 8% drop using the existing SVD approach.

Original: On RoBERTa we achieve 43% compression of the embedding layer with less than a 0 .
Revised: On RoBERTa we achieve 43% compression of the embedding layer with less than a 0 .

Original: 8% average drop in accuracy as compared to a 3% drop previously.
Revised: 8% average drop in accuracy as compared to a 3% drop previously.

Original:  I NTRODUCTION AND M OTIVATION
Revised:  I NTRODUCTION AND M OTIVATION

Original: Deep Learning revolutionized Machine Learning by improving the accuracy by dozens of percents for fundamental tasks in Natural Language Processing (NLP) through learning representations of a natural language via a deep neural network (Mikolov et al., 2013;
Revised: Deep Learning revolutionized Machine Learning by improving the accuracy by dozens of percents for fundamental tasks in Natural Language Processing (NLP) through learning representations of a natural language via a deep neural network (Mikolov et al., 2013;

Original: Radford et al., 2018;
Revised: Radford et al., 2018;

Original: Le and
Revised: Le and

Original: Mikolov, 2014; Peters et al., 2018;
Revised: Mikolov, 2014; Peters et al., 2018;

Original: Radford et al., 2019).
Revised: Radford et al., 2019).

Original: Lately, it was shown that there is no need to train those networks from scratch each time we receive a new task/data, but to ﬁne-tune a full pre-trained model on the speciﬁc task (Dai and Le, 2015;
Revised: Lately, it was shown that there is no need to train those networks from scratch each time we receive a new task/data, but to ﬁne-tune a full pre-trained model on the speciﬁc task (Dai and Le, 2015;

Original: Radford et al., 2018;
Revised: Radford et al., 2018;

Original: Devlin et al., 2019).
Revised: Devlin et al., 2019).

Original: However, in many cases, those networks are extremely large compared to classical machine learning models.
Revised: However, in many cases, those networks are extremely large compared to classical machine learning models.

Original: For example, both BERT (Devlin et al., 2019) and XLNet (Yang et al., 2019) have more than 110 million parameters, and RoBERTa (Liu et al., 2019b) consists of more than 125 million parameters.
Revised: For example, both BERT (Devlin et al., 2019) and XLNet (Yang et al., 2019) have more than 110 million parameters, and RoBERTa (Liu et al., 2019b) consists of more than 125 million parameters.

Original: Such large networks have two main drawbacks: (i) they use too much storage, e.g. memory or disk space, which may be infeasible for small IoT devices, smartphones, or when a personalized network is needed for each user/object/task, and (ii) classiﬁcation may take too much time, especially for real-time applications such as NLP tasks: speech recognition, translation or speech-to-text.
Revised: Such large networks have two main drawbacks: (i) they use too much storage, e.g. memory or disk space, which may be infeasible for small IoT devices, smartphones, or when a personalized network is needed for each user/object/task, and (ii) classiﬁcation may take too much time, especially for real-time applications such as NLP tasks: speech recognition, translation or speech-to-text.

Original: Compressed Networks.
Revised: Compressed Networks.

Original: To this end, many papers suggested different techniques to compress large NLP networks, e.g., by low-rank factorization (Wang et al., 2019; Lan et al., 2019), pruning (McCarley, 2019;
Revised: To this end, many papers suggested different techniques to compress large NLP networks, e.g., by low-rank factorization (Wang et al., 2019; Lan et al., 2019), pruning (McCarley, 2019;

Original: Michel et al., 2019; Fan et al., 2019; Guo et al., 2019; Gordon et al., 2020), quantization (Zafrir et al., 2019;
Revised: Michel et al., 2019; Fan et al., 2019; Guo et al., 2019; Gordon et al., 2020), quantization (Zafrir et al., 2019;

Original: Shen et al., 2020), weight sharing (Lan et al., 2019), and knowledge distillation (Sanh et al., 2019; Tang et al., 2019; Mukherjee and Awadallah, 2019; Liu et al., 2019a; Sun et al., 2019;
Revised: Shen et al., 2020), weight sharing (Lan et al., 2019), and knowledge distillation (Sanh et al., 2019; Tang et al., 2019; Mukherjee and Awadallah, 2019; Liu et al., 2019a; Sun et al., 2019;

Original: Jiao et al., 2019); see more example papers and a comparison table in Gordon (2019) for compressing the BERT model.
Revised: Jiao et al., 2019); see more example papers and a comparison table in Gordon (2019) for compressing the BERT model.

Original: There is no consensus on which approach should be used in what contexts.
Revised: There is no consensus on which approach should be used in what contexts.

Original: However, in the context of compressing the embedding layer, the most common approach is low-rank factorization as in Lan et al.
Revised: However, in the context of compressing the embedding layer, the most common approach is low-rank factorization as in Lan et al.

Original: and it may be combined with other techniques such as quantization and pruning.
Revised: and it may be combined with other techniques such as quantization and pruning.

Original: In this work, we suggest a novel low-rank factorization technique for compressing the embedding layer of a given model.
Revised: In this work, we suggest a novel low-rank factorization technique for compressing the embedding layer of a given model.

Original: This is motivated by the fact that in many networks, the embedding layer accounts for 20% − 40% of the network size.
Revised: This is motivated by the fact that in many networks, the embedding layer accounts for 20% − 40% of the network size.

Original: Our approach - MESSI:
Revised: Our approach - MESSI:

Original: Multiple (parallel)
Revised: Multiple (parallel)

Original: Estimated
Revised: Estimated

Original: SVDs for Smaller Intralayers - achieves a better accuracy for the same compression rate compared to the known standard matrix factorization.
Revised: SVDs for Smaller Intralayers - achieves a better accuracy for the same compression rate compared to the known standard matrix factorization.

Original: To present it, we ﬁrst describe an embedding layer, the known technique for compressing it, and the geometric assumptions underlying this technique.
Revised: To present it, we ﬁrst describe an embedding layer, the known technique for compressing it, and the geometric assumptions underlying this technique.

Original: Then, we give our approach followed by geometric intuition, and detailed explanation about the motivation and the architecture changes.
Revised: Then, we give our approach followed by geometric intuition, and detailed explanation about the motivation and the architecture changes.

Original: Finally, we report our experimental results that demonstrate the strong performance of our technique.
Revised: Finally, we report our experimental results that demonstrate the strong performance of our technique.

Original: Embedding Layer.
Revised: Embedding Layer.

Original: The embedding layer aims to represent each word from a vocabulary by a real-valued vector that reﬂects the word’s semantic and syntactic information that can be extracted from the language.
Revised: The embedding layer aims to represent each word from a vocabulary by a real-valued vector that reﬂects the word’s semantic and syntactic information that can be extracted from the language.

Original: One can think of the embedding layer as a simple matrix multiplication as follows.
Revised: One can think of the embedding layer as a simple matrix multiplication as follows.

Original: The layer receives a standard vector x ∈ R n (a row of the identity matrix, exactly one nonzero entry, usually called one-hot vector ) that represents a word in the vocabulary, it multiplies x by a matrix A T ∈ R d × n to obtain the corresponding d -dimensional word embedding vector y =
Revised: The layer receives a standard vector x ∈ R n (a row of the identity matrix, exactly one nonzero entry, usually called one-hot vector ) that represents a word in the vocabulary, it multiplies x by a matrix A T ∈ R d × n to obtain the corresponding d -dimensional word embedding vector y =

Original: A T x , which is the row in A that corresponds to the non-zero entry of x .
Revised: A T x , which is the row in A that corresponds to the non-zero entry of x .

Original: The embedding layer has n input neurons, and the output has d neurons.
Revised: The embedding layer has n input neurons, and the output has d neurons.

Original: The nd edges between the input and output neurons deﬁne the matrix A ∈ R n × d .
Revised: The nd edges between the input and output neurons deﬁne the matrix A ∈ R n × d .

Original: Here, the entry in the i th row and j th column of A is the weight of the edge between the i th input neuron to the j th output neuron; see Figure.
Revised: Here, the entry in the i th row and j th column of A is the weight of the edge between the i th input neuron to the j th output neuron; see Figure.

Original: Compressing by Matrix Factorization.
Revised: Compressing by Matrix Factorization.

Original: A common approach for compressing an embedding layer is to compute the j -rank approximation A j ∈ R n × d of the corresponding matrix A via SVD (Singular Value Decomposition; see e.g., Lan et al.
Revised: A common approach for compressing an embedding layer is to compute the j -rank approximation A j ∈ R n × d of the corresponding matrix A via SVD (Singular Value Decomposition; see e.g., Lan et al.

Original: Yu et al.
Revised: Yu et al.

Original: (2017) and Acharya et al.
Revised: (2017) and Acharya et al.

Original: (2019)), factor A j into two smaller matrices U ∈ R n × j and V ∈ R j × d (i.e. A j = UV ), and replace the original embedding layer that corresponds to A by a pair of layers that correspond to U and
Revised: (2019)), factor A j into two smaller matrices U ∈ R n × j and V ∈ R j × d (i.e. A j = UV ), and replace the original embedding layer that corresponds to A by a pair of layers that correspond to U and

Original: V .
Revised: V .

Original: The number of parameters is then reduced to j ( n + d ) .
Revised: The number of parameters is then reduced to j ( n + d ) .

Original: Moreover, computing the output takes
Revised: Moreover, computing the output takes

Original: O ( j ( n + d )) time, compared to the O ( nd ) time for computing A T x .
Revised: O ( j ( n + d )) time, compared to the O ( nd ) time for computing A T x .

Original: As above, we continue to use
Revised: As above, we continue to use

Original: A j to refer to a rank- j approximation of a matrix A .
Revised: A j to refer to a rank- j approximation of a matrix A .

Original: Fine tuning.
Revised: Fine tuning.

Original: The layers that correspond to the matrices U and V above are sometimes used only as initial seeds for a training process that is called ﬁne tuning .
Revised: The layers that correspond to the matrices U and V above are sometimes used only as initial seeds for a training process that is called ﬁne tuning .

Original: Here, the training data is fed into the network, and the error is measured with respect to the ﬁnal classiﬁcation.
Revised: Here, the training data is fed into the network, and the error is measured with respect to the ﬁnal classiﬁcation.

Original: Hence, the structure of the data remains the same but the edges are updated in each iteration to give a better accuracy.
Revised: Hence, the structure of the data remains the same but the edges are updated in each iteration to give a better accuracy.

Original: Observe that typically, the SVD takes the form A j = UD ˜ V , where the columns of U ∈ R n × j are orthogonal, the rows of ˜ V ∈ R j × d are orthogonal, and
Revised: Observe that typically, the SVD takes the form A j = UD ˜ V , where the columns of U ∈ R n × j are orthogonal, the rows of ˜ V ∈ R j × d are orthogonal, and

Original: D ∈ R j × j is a diagonal matrix.
Revised: D ∈ R j × j is a diagonal matrix.

Original: In this paper
Revised: In this paper

Original: In this example, the factorization was done based on a 4 -dimensional subspace.
Revised: In this example, the factorization was done based on a 4 -dimensional subspace.

Original: The result is a compressed layer that consists of 120 parameters.
Revised: The result is a compressed layer that consists of 120 parameters.

Original: The original matrix had 200 parameters.
Revised: The original matrix had 200 parameters.

Original: See more details in the and in others, we say that A j = UV where V = D ˜ V .
Revised: See more details in the and in others, we say that A j = UV where V = D ˜ V .

Original: Furthermore, the orthogonalization is used only to obtain a low rank approximation A j = UV using SVD.
Revised: Furthermore, the orthogonalization is used only to obtain a low rank approximation A j = UV using SVD.

Original: After that, this property is not kept in the network during the training process (when applying the ﬁne-tuning).
Revised: After that, this property is not kept in the network during the training process (when applying the ﬁne-tuning).

Original: Geometric intuition.
Revised: Geometric intuition.

Original: The embedding layer can be encoded into a matrix A ∈ R n × d as explained above.
Revised: The embedding layer can be encoded into a matrix A ∈ R n × d as explained above.

Original: Hence, each of the n rows of A corresponds to a point (vector) in R d , and the j -rank approximation A j ∈ R n × d represents the projection on the j -dimensional subspace that minimizes the sum of squared distances (“errors”) to the points.
Revised: Hence, each of the n rows of A corresponds to a point (vector) in R d , and the j -rank approximation A j ∈ R n × d represents the projection on the j -dimensional subspace that minimizes the sum of squared distances (“errors”) to the points.

Original: Projecting these points onto any j -dimensional subspace of R d would allow us to encode every point only via its j -coordinates on this subspace, and store only nj entries instead of the original nd entries of A .
Revised: Projecting these points onto any j -dimensional subspace of R d would allow us to encode every point only via its j -coordinates on this subspace, and store only nj entries instead of the original nd entries of A .

Original: This is the matrix U ∈ R n × j , where each row encodes the corresponding row in A by its j -coordinates on this subspace.
Revised: This is the matrix U ∈ R n × j , where each row encodes the corresponding row in A by its j -coordinates on this subspace.

Original: The subspace itself can be represented by its basis of j d -dimensional vectors ( jd entries), which is the column space of a matrix V T ∈ R d × j .
Revised: The subspace itself can be represented by its basis of j d -dimensional vectors ( jd entries), which is the column space of a matrix V T ∈ R d × j .

Original: Figure 2 illustrates the small pair of layers that corresponds to U and
Revised: Figure 2 illustrates the small pair of layers that corresponds to U and

Original: V , those layers are a compression for the original big layer that corresponds to A .
Revised: V , those layers are a compression for the original big layer that corresponds to A .

Original: However, our goal is not only to compress the network or matrix, but also to approximate the original matrix operator A .
Revised: However, our goal is not only to compress the network or matrix, but also to approximate the original matrix operator A .

Original: To this end, among all the possible j -subspaces of R d , we may be interested in the j -subspace that minimizes the sum of squared distances to the points, i.e., the sum of squared projected errors.
Revised: To this end, among all the possible j -subspaces of R d , we may be interested in the j -subspace that minimizes the sum of squared distances to the points, i.e., the sum of squared projected errors.

Original: This subspace can be computed easily via SVD.
Revised: This subspace can be computed easily via SVD.

Original: The corresponding projections of the rows of A on this subspace are the rows of the j -rank matrix A j .
Revised: The corresponding projections of the rows of A on this subspace are the rows of the j -rank matrix A j .

Original: The hidden or statistical assumption in this model is that the rows of the matrix A (that represents the embedding layer) were actually generated by adding i.i.d.
Revised: The hidden or statistical assumption in this model is that the rows of the matrix A (that represents the embedding layer) were actually generated by adding i.i.d.

Original: Gaussian noise to each point in a set of n points on a j -dimensional subspace, that is spanned by what are called latent variables or factors.
Revised: Gaussian noise to each point in a set of n points on a j -dimensional subspace, that is spanned by what are called latent variables or factors.

Original: Given only the resulting matrix A , the j -subspace that maximizes the likelihood (probability) of generating the original points is spanned by the j largest singular vectors of A .
Revised: Given only the resulting matrix A , the j -subspace that maximizes the likelihood (probability) of generating the original points is spanned by the j largest singular vectors of A .

Original: Why a single distribution?
Revised: Why a single distribution?

Original: Even if we accept the assumption of Gaussian noise, e.g. due to simplicity of computations or the law of large numbers, it is not intuitively clear why we should assume that the rows of A were sampled from a single distribution.
Revised: Even if we accept the assumption of Gaussian noise, e.g. due to simplicity of computations or the law of large numbers, it is not intuitively clear why we should assume that the rows of A were sampled from a single distribution.

Original: Natural questions that arise are:
Revised: Natural questions that arise are:

Original: (i) Can we get smaller and/or more accurate models in real-world networks by assuming multiple instead of a single generating distribution (i
Revised: (i) Can we get smaller and/or more accurate models in real-world networks by assuming multiple instead of a single generating distribution (i

Original: e. multiple subspaces)?
Revised: e. multiple subspaces)?

Original: (ii) Can we efﬁciently compute the corresponding factorizations and represent them as part of a network ?
Revised: (ii) Can we efﬁciently compute the corresponding factorizations and represent them as part of a network ?

Original:  O UR CONTRIBUTION
Revised:  O UR CONTRIBUTION

Original: We answer the above open questions by suggesting the following contributions.
Revised: We answer the above open questions by suggesting the following contributions.

Original: In short, the answers are:
Revised: In short, the answers are:

Original: (i)
Revised: (i)

Original: In all the real-world networks that we tested, it is almost always better to assume k ≥distributions rather than a single one that generated the data.
Revised: In all the real-world networks that we tested, it is almost always better to assume k ≥distributions rather than a single one that generated the data.

Original: It is better in the sense that the resulting accuracy of the network is better compared to k = 1 (SVD) for the same compression rate.
Revised: It is better in the sense that the resulting accuracy of the network is better compared to k = 1 (SVD) for the same compression rate.

Original: (ii) While approximating the global minimum is Max-SNP-Hard, our experiments show that we can efﬁciently compute many local minima and take the smallest one.
Revised: (ii) While approximating the global minimum is Max-SNP-Hard, our experiments show that we can efﬁciently compute many local minima and take the smallest one.

Original: We then explain how to encode the result back into the network.
Revised: We then explain how to encode the result back into the network.

Original: This is by suggesting a new embedding layer architecture that we call MESSI (Multiple (parallel) Estimated SVDs for Smaller Intralayers); see Figure 3.
Revised: This is by suggesting a new embedding layer architecture that we call MESSI (Multiple (parallel) Estimated SVDs for Smaller Intralayers); see Figure 3.

Original: Extensive experimental results show signiﬁcant improvement.
Revised: Extensive experimental results show signiﬁcant improvement.

Original: Computational Geometry meets Deep Learning.
Revised: Computational Geometry meets Deep Learning.

Original: Our technique also constructs the matrix
Revised: Our technique also constructs the matrix

Original: A ∈ R n × d from a given embedding layer.
Revised: A ∈ R n × d from a given embedding layer.

Original: However, inspired by the geometric intuition from the previous section, we suggest to approximate the n rows of A by clustering them to k ≥subspaces instead of one.
Revised: However, inspired by the geometric intuition from the previous section, we suggest to approximate the n rows of A by clustering them to k ≥subspaces instead of one.

Original: More precisely, given an integer k ≥ 1 we aim to compute a set of k subspaces in R d , each of dimension j , that will minimize the sum over every squared distance of every point (row in A ) to its nearest subspace.
Revised: More precisely, given an integer k ≥ 1 we aim to compute a set of k subspaces in R d , each of dimension j , that will minimize the sum over every squared distance of every point (row in A ) to its nearest subspace.

Original: This can be considered as a combination of j -rank or j -subspace approximation, as deﬁned above, and k -means clustering.
Revised: This can be considered as a combination of j -rank or j -subspace approximation, as deﬁned above, and k -means clustering.

Original: In the k -means clustering problem we wish to approximate n points by k center points that minimizes the sum over squared distance between every point to its nearest center.
Revised: In the k -means clustering problem we wish to approximate n points by k center points that minimizes the sum over squared distance between every point to its nearest center.

Original: In our case, the k centers points are replaced by k subspaces, each of dimension j .
Revised: In our case, the k centers points are replaced by k subspaces, each of dimension j .

Original: In computational geometry, this type of problem is called projective clustering (see Figure 4), and its used in many tasks in the ﬁelds of Machine Learning and Computer
Revised: In computational geometry, this type of problem is called projective clustering (see Figure 4), and its used in many tasks in the ﬁelds of Machine Learning and Computer

Original: Vision (Feng et al., 2011; Xu et al., 2005; Liu et al., 2012; Trittenbach and B¨ohm, 2019),
Revised: Vision (Feng et al., 2011; Xu et al., 2005; Liu et al., 2012; Trittenbach and B¨ohm, 2019),

Original: From Embedding layer to Embedding layers.
Revised: From Embedding layer to Embedding layers.

Original: The result of the above technique is a set of k matrices A 1 j , · · · , A kj , each of rank j and dimension n i × d where the i th matrix corresponds to the cluster of n i points that were projected on the i th j -dimensional subspace.
Revised: The result of the above technique is a set of k matrices A 1 j , · · · , A kj , each of rank j and dimension n i × d where the i th matrix corresponds to the cluster of n i points that were projected on the i th j -dimensional subspace.

Original: Each of those matrices can be factored into two smaller matrices (due to its low rank), i.e., for every i ∈ { 1 , · · · , k } , we have A ij
Revised: Each of those matrices can be factored into two smaller matrices (due to its low rank), i.e., for every i ∈ { 1 , · · · , k } , we have A ij

Original: =
Revised: =

Original: U
Revised: U

Original: i V i , where U i ∈ R n i × j , and V i ∈
Revised: i V i , where U i ∈ R n i × j , and V i ∈

Original: R
Revised: R

Original: j × d .
Revised: j × d .

Original: To plug these matrices as part of the ﬁnal network instead of the embedded layer, we suggest to encode these matrices via k parallel sub-layers as described in what follows and illustrated in Figure 3.
Revised: To plug these matrices as part of the ﬁnal network instead of the embedded layer, we suggest to encode these matrices via k parallel sub-layers as described in what follows and illustrated in Figure 3.

Original: Our pipeline: MESSI.
Revised: Our pipeline: MESSI.

Original: We construct our new architecture as follows.
Revised: We construct our new architecture as follows.

Original: We use A to refer to the n × d matrix from the embedding layer we seek to compress.
Revised: We use A to refer to the n × d matrix from the embedding layer we seek to compress.

Original: The input to our pipeline is the matrix
Revised: The input to our pipeline is the matrix

Original: A , positive integers j and k , and (for the ﬁnal step) parameters for the ﬁne-tuning.
Revised: A , positive integers j and k , and (for the ﬁnal step) parameters for the ﬁne-tuning.

Original: Treating the n rows of A as n points in R d , compute an approximate ( k, j ) -projective clustering.
Revised: Treating the n rows of A as n points in R d , compute an approximate ( k, j ) -projective clustering.

Original: The result is k subspaces in R d , each of dimension j , that minimize the sum of squared distances from each point (row in A ) to its closest subspace.
Revised: The result is k subspaces in R d , each of dimension j , that minimize the sum of squared distances from each point (row in A ) to its closest subspace.

Original: For the approximation, we compute a local minimum for this problem using the Expectation-Maximization (EM) method (Dempster et al., 1977).
Revised: For the approximation, we compute a local minimum for this problem using the Expectation-Maximization (EM) method (Dempster et al., 1977).

Original: Partition the rows of A into k different subsets according to their nearest subspace from the previous step.
Revised: Partition the rows of A into k different subsets according to their nearest subspace from the previous step.

Original: The result is submatrices A 1 , . . .
Revised: The result is submatrices A 1 , . . .

Original: , A k where A i is a n i × d matrix and n 1 + . . .
Revised: , A k where A i is a n i × d matrix and n 1 + . . .

Original: + n k = n .
Revised: + n k = n .

Original: For each matrix A i where 1 ≤ i ≤ k , factor it to two smaller matrices
Revised: For each matrix A i where 1 ≤ i ≤ k , factor it to two smaller matrices

Original: U i (of dimensions n i × j ) and V i (of dimensions j × d ) such that U
Revised: U i (of dimensions n i × j ) and V i (of dimensions j × d ) such that U

Original: i V i is the rank- j approximation of A i . 4.
Revised: i V i is the rank- j approximation of A i . 4.

Original: In the full network, replace the original fully-connected embedding layer by 2 layers.
Revised: In the full network, replace the original fully-connected embedding layer by 2 layers.

Original: The ﬁrst layer is a parallelization of k separate fully-connected layers,
Revised: The ﬁrst layer is a parallelization of k separate fully-connected layers,

Original: where for every i ∈ { 1 , · · · , k } the i th parallel layer consists of the matrix U i , i.e., it has n i input neurons and j output neurons.
Revised: where for every i ∈ { 1 , · · · , k } the i th parallel layer consists of the matrix U i , i.e., it has n i input neurons and j output neurons.

Original: Here, each row of A is mapped appropriately.
Revised: Here, each row of A is mapped appropriately.

Original: The second layer is by combining the matrices V 1 , · · · V k .
Revised: The second layer is by combining the matrices V 1 , · · · V k .

Original: Each of the k output vectors from the previous layer u 1 , . .
Revised: Each of the k output vectors from the previous layer u 1 , . .

Original: , u k are combined as V 1 u 1 + . . .
Revised: , u k are combined as V 1 u 1 + . . .

Original: + V k u k ; see Figure 3 for an illustration.
Revised: + V k u k ; see Figure 3 for an illustration.

Original: Fine-tune the network.
Revised: Fine-tune the network.

Original: The result is a compressed embedding layer.
Revised: The result is a compressed embedding layer.

Original: Every matrix U i has n i j parameters and the matrix
Revised: Every matrix U i has n i j parameters and the matrix

Original: V i has jd parameters.
Revised: V i has jd parameters.

Original: Therefore the compressed embedding layer consists of nj + kjd parameters, in comparison to the uncompressed layer of nd parameters.
Revised: Therefore the compressed embedding layer consists of nj + kjd parameters, in comparison to the uncompressed layer of nd parameters.

Original: Practical Solution.
Revised: Practical Solution.

Original: The projective clustering problem is known to be Max-SNP-hard even for d = 2 and j = 2 , for any approximation factor that is independent of n .
Revised: The projective clustering problem is known to be Max-SNP-hard even for d = 2 and j = 2 , for any approximation factor that is independent of n .

Original: Instead, we suggest to use an algorithm that provably converges to a local minimum via the Expectation-Maximization (EM) method (Dempster et al., 1977), which is a generalization of the well known Lloyd algorithm (Lloyd, 1982).
Revised: Instead, we suggest to use an algorithm that provably converges to a local minimum via the Expectation-Maximization (EM) method (Dempster et al., 1977), which is a generalization of the well known Lloyd algorithm (Lloyd, 1982).

Original: The resulting clusters and factorizations are used to determine the new architecture and its initial weights; see Figure 3 for more details.
Revised: The resulting clusters and factorizations are used to determine the new architecture and its initial weights; see Figure 3 for more details.

Original: We run on instances of AWS Amazon EC2 cloud, and detail our results in the next section.
Revised: We run on instances of AWS Amazon EC2 cloud, and detail our results in the next section.

Original: Open code and networks.
Revised: Open code and networks.

Original: Complete open code to reproduce the resulting networks is provided in Code (2020).
Revised: Complete open code to reproduce the resulting networks is provided in Code (2020).

Original: We expect it to be useful for future research, and give the following few examples.
Revised: We expect it to be useful for future research, and give the following few examples.

Original: 2. G ENERALIZATIONS AND E XTENSIONS .
Revised: 2. G ENERALIZATIONS AND E XTENSIONS .

Original: Our suggested architecture can be generalized and extended to support many other optimization functions that may be relevant for different types of datasets, tasks or applications besides NLP.
Revised: Our suggested architecture can be generalized and extended to support many other optimization functions that may be relevant for different types of datasets, tasks or applications besides NLP.

Original: (cid:96) q -error.
Revised: (cid:96) q -error.

Original: For simplicity, our suggested approach aims to minimize sum of squared distances to k subspaces.
Revised: For simplicity, our suggested approach aims to minimize sum of squared distances to k subspaces.

Original: However, it can be easily applied also to sum of distances from the points to the subspace, which is a more robust approach toward outliers (“far away points”).
Revised: However, it can be easily applied also to sum of distances from the points to the subspace, which is a more robust approach toward outliers (“far away points”).

Original: Even for k = 1 recent results of Tukan et al.
Revised: Even for k = 1 recent results of Tukan et al.

Original: (2020b) show improvement over SVD.
Revised: (2020b) show improvement over SVD.

Original: Distance functions.
Revised: Distance functions.

Original: Similarly, we can replace the Euclidean (cid:96) 2 -distance by e.g. the Manhattan distance which is the (cid:96) 1 -norm between a point x and its projection, i.e., (cid:107) x − x (cid:48) (cid:107) 1 or sum of differences between the corresponding entries, instead of sum of squared entries, as in the Euclidean distance (cid:107) x − x (cid:48) (cid:107) 2 in this paper.
Revised: Similarly, we can replace the Euclidean (cid:96) 2 -distance by e.g. the Manhattan distance which is the (cid:96) 1 -norm between a point x and its projection, i.e., (cid:107) x − x (cid:48) (cid:107) 1 or sum of differences between the corresponding entries, instead of sum of squared entries, as in the Euclidean distance (cid:107) x − x (cid:48) (cid:107) 2 in this paper.

Original: Non-uniform dimensions.
Revised: Non-uniform dimensions.

Original: In this paper we assume that k subspaces approximate the input points, and each subspace has dimension exactly j , where j, k ≥ 1 are given integers.
Revised: In this paper we assume that k subspaces approximate the input points, and each subspace has dimension exactly j , where j, k ≥ 1 are given integers.

Original: A better strategy is to allow each subspace to have a different dimension, j i for every i ∈ { 1 , · · · , k } , or add a constraint only on the sum j 1 + · · · + j k of dimensions.
Revised: A better strategy is to allow each subspace to have a different dimension, j i for every i ∈ { 1 , · · · , k } , or add a constraint only on the sum j 1 + · · · + j k of dimensions.

Original: Similarly, the number k may be tuned as in our experimental results.
Revised: Similarly, the number k may be tuned as in our experimental results.

Original: Using this approach we can improve the accuracy and enjoy the same compression rate.
Revised: Using this approach we can improve the accuracy and enjoy the same compression rate.

Original: For more details about those generalizations and others, we refer the interested reader to section E. at the appendix.
Revised: For more details about those generalizations and others, we refer the interested reader to section E. at the appendix.

Original: 120 + 3)
Revised: 120 + 3)

Original: = 246 parameters, compared to 120 × 1 + 3 × 3 = 129 parameters based on (cid:96) 1 , (cid:96) 2 , and (cid:96) 3 .
Revised: = 246 parameters, compared to 120 × 1 + 3 × 3 = 129 parameters based on (cid:96) 1 , (cid:96) 2 , and (cid:96) 3 .

Original: I.e., less memory and a better result.
Revised: I.e., less memory and a better result.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: GLUE benchmark.
Revised: GLUE benchmark.

Original: We run all of our experiments on the General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2018).
Revised: We run all of our experiments on the General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2018).

Original: It is widely-used collection of 9 datasets for evaluating natural language understanding systems.
Revised: It is widely-used collection of 9 datasets for evaluating natural language understanding systems.

Original: Networks.
Revised: Networks.

Original: We compress the followingtwo networks: (i) RoBERTa
Revised: We use the following networks: (i) RoBERTa

Original: (Liu et al., 2019b), it consists of 120 millions parameters, and its embedding layer has 38 .
Revised: (Liu et al., 2019b), it consists of 120 millions parameters, and its embedding layer has 38 .

Original: 9 million parameters ( 32 . 5% of the entire network size),and (ii) DistilBERT (Sanh et al., 2019) consists of 66 million parameters, and its embedding layer has 23 .
Revised: 9 million parameters ( 32 . 5% of the entire network size), (ii) DistilBERT (Sanh et al., 2019) consists of 66 million parameters, and its embedding layer has 23 .

Original: 5 million parameters ( 35 .
Revised: 5 million parameters ( 35 .

Original: 5% of the entire network size).
Revised: 5% of the entire network size), and (iii) ALBERT (base-v2) (Lan et al., 2019), which consists of 11 . 7 million parameters, and its embedding layer has 3 . 8 million parameters ( 33% of the entire network).

Original: Software, and Hardware.
Revised: Software, and Hardware.

Original: All the experiments were conducted on a AWS c 5 a.16xlarge machine with 64 CPUs and 128 RAM [GiB].
Revised: All the experiments were conducted on a AWS c 5 a.16xlarge machine with 64 CPUs and 128 RAM [GiB].

Original: To build and train networks, we used the suggested implementation at the Transformers 1 library from HuggingFace (Wolf et al., 2019)
Revised: To build and train networks, we used the suggested implementation at the Transformers 1 library from HuggingFace (Wolf et al., 2019)

Original: (Transformers version  3 . 1 . 0 , and PyTorch version 1 . 6 . 

Revised: (Transformers version transformers [Table] 3 . 1 . 0 , and PyTorch version 1 . 6 .

Original: (Paszke et al., 2017)).
Revised: (Paszke et al., 2017)).

Original: For more detailes about the implementation, we refer the reader to section A at the appendix.
Revised: For more detailes about the implementation, we refer the reader to section A at the appendix.

Original: The setup.
Revised: The setup.

Original: All our experiments are benchmarked against their publicly available implementations of the DistilBERT and RoBERTa models, ﬁne-tuned for each task, which was in some cases higher and in other cases lower than the values printed in the publications introducing these models.
Revised: All our experiments are benchmarked against their publicly available implementations of the DistilBERT, RoBERTa, and ALBERT models, ﬁne-tuned for each task, which was in some cases higher and in other cases lower than the values printed in the publications introducing these models.

Original: Given an embedding layer from a network that is trained on a task from GLUE, an integer k ≥ 1 , and an integer j ≥ 1 .
Revised: Given an embedding layer from a network that is trained on a task from GLUE, an integer k ≥ 1 , and an integer j ≥ 1 .

Original: We build and initialize a new architecture that replaces the original embedding layer by two smaller layers as explained in Figure 3.
Revised: We build and initialize a new architecture that replaces the original embedding layer by two smaller layers as explained in Figure 3.

Original: We then ﬁne tune the resulted network for 2 epochs.
Revised: We then ﬁne tune the resulted network forepochs.

Original: We ran the same experiments for several values of k and j that deﬁnes different compression rates.
Revised: We ran the same experiments for several values of k and j that deﬁnes different compression rates.

Original: We compete with the standard matrix factorization approach in all experiments.
Revised: We compete with the standard matrix factorization approach in all experiments.

Original: Reported results.
Revised: 3. R EPORTED RESULTS

Revised: Compressing RoBERTA and DistilBERT.

Original: (i)
Revised: (i)

Original: In Figures 5 and 6 the x -axis is the compression rate of the embedding layer, i.e. a compression of 40% means the layer is 60% its original size.
Revised: In Figures 5 and 6 the x -axis is the compression rate of the embedding layer, i.e. a compression of 40% means the layer is 60% its original size.

Original: The y -axis is the accuracy drop (relative error) with respect to the original accuracy of the network (with ﬁne tuning forepochs).
Revised: The y -axis is the accuracy drop (relative error) with respect to the original accuracy of the network (with ﬁne tuning for 2 epochs).

Original: In Figure 5, each graph reports the results for a speciﬁc task from theGLUE benchmark on
Revised: In Figure 5, each graph reports the results for a speciﬁc task from the

Original:  RoBERTa, while Figure 6 reports the results of DistilBERT.
Revised: GLUE benchmark on RoBERTa, while Figure 6 reports the results of DistilBERT.

Original: (ii)
Revised: (ii)

Original: On the task WNLI we achievederror on both networks using the two approaches of SVD and our approach until 60% compression rate, so we did not add a ﬁgure on it.
Revised: On the task WNLI we achieved 0 error on both networks using the two approaches of SVD and our approach until 60% compression rate, so we did not add a ﬁgure on it.

Original: (iii)
Revised: (iii)

Original: In RoBERTa, we ran one compression rate on MNLI due to time constraints, and we achieved similar results in both techniques. We compressed 45% of the embedding layer, based on our technique with k = 5 and j = 384 to obtain only 0 .
Revised: In RoBERTa, we checked only 2 compression rates on MNLI due to time constraints, and we achieved similar results in both techniques, e.g., we compressed 45% of the embedding layer, based on our technique with k = 5 and j = 384 to obtain only 0 .

Original: 61% drop in accuracy with ﬁne tuning and 4 . 2% without, this is compared to 0 .
Revised: 61% drop in accuracy with ﬁne tuning and 4 . 2% without, this is compared to 0 .

Original: 61% and 13 .
Revised: 61% and 13 .

Original: 9% respectively for the same compression rate via SVD factorization.
Revised: 9% respectively for the same compression rate via SVD factorization.

Original: In DistilBERT, we compressed 40% of the embedding layer with k = 4 and achieved a 0 .
Revised: In DistilBERT, we compressed 40% of the embedding layer with k =and achieved a 0 .

Original: 1% increase in accuracy after ﬁne-tuning, as compared to a 0 .
Revised: 1% increase in accuracy after ﬁne-tuning, as compared to a 0 .

Original: 05% drop via SVD factorization (on MNLI).
Revised: 05% drop via SVD factorization (on MNLI).

Original: More results that are placed in the appendix: (i) Figure 8 in section B shows the accuracydrop as a function of  the compression rate on the RoBERTA model before ﬁne-tuning.
Revised: (iv) Table 1 suggests the best compressed networks in terms of accuracy VS size. Improving the accuracy of pre-trained models using MESSI. In Table 2, we test if the MESSI architecture can improve the accuracy of a pre-trained model, while maintaining the same number of parameters. The only change done on the given model is factoring its embedding layer to the suggested architecture using the detailed pipeline at section 2.

Revised: Here, we make sure to choose the right values of k and j such that the original embedding layer size is maintained (up to a very small

Revised: [Table]

Revised: change).

Revised: We conducted this experiment on the model ALBERT (base v2).

Revised: The results are actually promising.

Revised: More results that are placed in the appendix: (i) Figure 8 in section B shows the accuracy drop as a function of the compression rate on the RoBERTA model before ﬁne-tuning.

Original: (ii)
Revised: (ii)

Original: In section C we compress a fully connected layer in different settings, speciﬁcally speaking we compress the two popular models: LeNet- 300 - 100 on MNIST (LeCun et al., 1998), and VGG- 19 (Simonyan and
Revised: In section C we compress a fully connected layer in different settings, speciﬁcally speaking we compress the two popular models: LeNet- 300 - 100 on MNIST (LeCun et al., 1998), and VGG- 19 (Simonyan and

Original: Zisserman, 2014) on CIFAR10 (Krizhevsky et al., 2009), see results at Figures 9 and 10.
Revised: Zisserman, 2014) on CIFAR10 (Krizhevsky et al., 2009), see results at Figures 9 and 10.

Original: (iii)
Revised: (iii)

Original: In section D, we suggest a way to determine the values of k and j in practice for a given compression rate, and we report the results on compressing DistilBERT based on this suggestion; see Figure 11.
Revised: In section D, we suggest a way to determine the values of k and j in practice for a given compression rate, and we report the results on compressing DistilBERT based on this suggestion; see Figure 11.

Original: (iv) Finally, in section E we check how other clustering method can ﬁt in our pipeline, i.e., instead of clustering the input neurons of the fully connected layer (rows of A ) via projective clustering (stepsand 2 in the pipeline at Section 2), we try the known k -means clustering, and then we continue the same by applying SVD on each cluster and building the corresponding new layers.
Revised: (iv) Finally, in section E we check how another clustering method can ﬁt in our pipeline, i.e., instead of clustering the input neurons of the fully connected layer (rows of A ) via projective clustering (steps 1 and 2 in the pipeline at Section 2), we try the known k -means clustering, and then we continue the same by applying SVD on each cluster and building the corresponding new layers.

Original: See results in Figures 12, 13, 14 and 15.
Revised: See results in Figures 12, 13, 14 and 15.

Original: 3. DISCUSSION, CONCLUSION AND FUTURE WORK
Revised: 3. DISCUSSION

Original: As shown by Figures 5 and 6, our approach outperforms the traditional SVD factorization.
Revised: As shown by Figures 5 and 6, our approach outperforms the traditional SVD factorization.

Original: In all experiments, our suggested compression achieves better accuracy for the same compression rate compared to the traditional SVD.
Revised: In all experiments, our suggested compression achieves better accuracy for the same compression rate compared to the traditional SVD.

Original: For example, in RobERTa, we compress 43% of the embedding layer with less that 0 .
Revised: For example, in RobERTa, we compress 43% of the embedding layer with less that 0 .

Original: 8% average drop in accuracy, this is compared to the 3% drop in the standard technique for a smaller compression rate of 40% .
Revised: 8% average drop in accuracy, this is compared to the 3% drop in the standard technique for a smaller compression rate of 40% .

Original: In DistilBERT, we achieved 40% compression of the embedding layer while incurring only a 0 .
Revised: In DistilBERT, we achieved 40% compression of the embedding layer while incurring only a 0 .

Original: 5% average drop in accuracy over all nine GLUE tasks, compared to a 2 . 8% drop using the existing SVD approach.
Revised: 5% average drop in accuracy over all nine GLUE tasks, compared to a 2 . 8% drop using the existing SVD approach.

Revised: As the reader will notice, the same dataset (and network) may be ideally handled by different values of k depending on the desired compression.

Original: We observed that our technique shines mainly when the network is efﬁcient, and any small change will lead to large error, e.g., as in the CoLA/RTE/MRPC graph of Figure 5.
Revised: We observed that our technique shines mainly when the network is efﬁcient, and any small change will lead to large error, e.g., as in the CoLA/RTE/MRPC graph of Figure 5.

Original: Although we achieve better results in all of the cases, but here the difference is more signiﬁcant (up to 10% ), since our compressed layer approximates the original layer better than SVD, the errors are smaller, and the accuracy is better.
Revised: Although we achieve better results in all of the cases, but here the difference is more signiﬁcant (up to 10% ), since our compressed layer approximates the original layer better than SVD, the errors are smaller, and the accuracy is better.

Original: Finally, Figure 8 shows clearly that even without ﬁne tuning, the new approach yields more accurate networks.
Revised: Furthermore, Figure 8 shows clearly that even without ﬁne tuning, the new approach yields more accurate networks.

Original: Hence, we can ﬁne tune for smaller number of epochs and achieve higher accuracy and smaller networks.
Revised: Hence, we can ﬁne tune for smaller number of epochs and achieve higher accuracy and smaller networks.

Revised: Finally, by Table 2 we can see that the MESSI architecture can be used also to improve the accuracy of pre-trained models while maintaining the original size.

Original: Conclusion: We suggested a novel approach for compressing a fully connected layer.
Revised: We suggested a novel approach for compressing a fully connected layer.

Original: This is by clustering the  neurons of the layer into k -subsets (via projective clustering) and then factoring the corresponding weights matrix of each subset.
Revised: This is by clustering the input neurons of the layer into k -subsets (via projective clustering) and then factoring the corresponding weights matrix of each subset.

Original: We the provided a novel architecture that replaces the original fully connected layer by a set of k small layers that operate in parallel and are then recombined with a single fully-connected layer.
Revised: We then provided a novel architecture that replaces the original fully connected layer by a set of k small layers that operate in parallel and are then recombined with a single fully-connected layer.

Original: The experimental results showed that our suggested algorithm overcomes the traditional factorization technique and achieves higher accuracy for the same compression rate before and after ﬁne-tuning.
Revised: The experimental results showed that our suggested algorithm overcomes the traditional factorization technique and achieves higher accuracy for the same compression rate before and after ﬁne-tuning.

Original: Future work includes: (i) Experiments on other networks and data sets both from the ﬁeld of NLP and outside it, (ii) an inserting experiment is to modify the ALBERT network (Lan et al., 2019), by changing its embedding layer architecture (that consists of two layers based on the standard matrix factorization) to the suggested architecture in this paper, while maintaining the same number of parameters, and to check if this modiﬁcation improved its accuracy, and (iii) try the suggested generalizations and extensions from section 2.1, where we strongly believe they will allow us to achieve even better results. 
Revised: The future work includes experiments on other networks and data sets both from the ﬁeld of NLP and outside it, e.g., an inserting experiment is to modify the ALBERT network (Lan et al., 2019), by changing its embedding layer architecture (that consists of two layers based on the standard matrix factorization) to the suggested architecture in this paper, while maintaining the same number of parameters, and to check if this modiﬁcation improved its accuracy, also the suggested generalizations and extensions from section 2.1 should be tried, where we strongly believe they will allow us to achieve even better results. Finally, generalizing the approach to other type of layers.

Original: A I MPLEMENTATION IN P YTORCH
Revised: A I MPLEMENTATION IN P YTORCH

Original: Since we did not ﬁnd straight support for the new suggested architecture, we implemented it as follows.
Revised: Since we did not ﬁnd straight support for the new suggested architecture, we implemented it as follows.

Original: To represent the matrices V 1 , · · · , V k that are described is Section 2, we concatenate them all to a one large matrix V =
Revised: To represent the matrices V 1 , · · · , V k that are described is Section 2, we concatenate them all to a one large matrix V =

Original: [( V 1 ) T , · · · , ( V k ) T ] T of kj rows and d columns, and we build a fullyconnected layer the corresponds to V .
Revised: [( V 1 ) T , · · · , ( V k ) T ] T of kj rows and d columns, and we build a fullyconnected layer the corresponds to V .

Original: For the k parallel layers (matrices) U 1 , · · · , U k , we build one large sparse matrix U of n rows and kj columns.
Revised: For the k parallel layers (matrices) U 1 , · · · , U k , we build one large sparse matrix U of n rows and kj columns.

Original: Every row of this matrix has at least ( k − 1)
Revised: Every row of this matrix has at least ( k − 1)

Original: j zero entries, and at most j non zero entries, where the non-zero entries of the i th row corresponds to the rows in matrix V which encode the closest subspace to that row’s point.
Revised: j zero entries, and at most j non zero entries, where the non-zero entries of the i th row corresponds to the rows in matrix V which encode the closest subspace to that row’s point.

Original: Finally, during the ﬁne tuning or training, we set those zero entries in U as non-trainable parameters, and we make sure that after every batch of back-propagation they remain zero.
Revised: Finally, during the ﬁne tuning or training, we set those zero entries in U as non-trainable parameters, and we make sure that after every batch of back-propagation they remain zero.

Original: Hence we have at most nj non-zero entries (trainable parameters) in U and nj + ndk in total.
Revised: Hence we have at most nj non-zero entries (trainable parameters) in U and nj + ndk in total.

Original: We hope that in the future, the suggested architecture will be implemented in the known DeepLearning libraries so it can be easily used while taking advantage of the substantial time and space beneﬁts presented in this paper.
Revised: We hope that in the future, the suggested architecture will be implemented in the known DeepLearning libraries so it can be easily used while taking advantage of the substantial time and space beneﬁts presented in this paper.

Original: B R ESULTS B EFORE F INE T UNING
Revised: B R ESULTS B EFORE F INE T UNING

Original: In this section we report the result of compressing RoBERTa without ﬁne-tuning.
Revised: In this section we report the result of compressing RoBERTa without ﬁne-tuning.

Original: By Figurewe can clearly see that even without ﬁne tuning, the new approach yields more accurate networks compared to the standard SVD factorization.
Revised: By Figurewe can clearly see that even without ﬁne tuning, the new approach yields more accurate networks compared to the standard SVD factorization.

Original: Hence, our approach gives a better start for the learning (ﬁne-tuning) process, which implies that we can ﬁne tune for smaller number of epochs and achieve higher accuracy and smaller networks.
Revised: Hence, our approach gives a better start for the learning (ﬁne-tuning) process, which implies that we can ﬁne tune for smaller number of epochs and achieve higher accuracy and smaller networks.

Original: C C OMPRESSING FULLY CONNECTED LAYERS USING MESSI.
Revised: C C OMPRESSING FULLY CONNECTED LAYERS USING MESSI.

Original: In this section we test our approach on two popular models: LeNet- 300 - 100 on MNIST (LeCun et al., 1998), and VGG- 19 (Simonyan and Zisserman, 2014) on CIFAR10 (Krizhevsky et al., 2009).
Revised: In this section we test our approach on two popular models: LeNet- 300 - 100 on MNIST (LeCun et al., 1998), and VGG- 19 (Simonyan and Zisserman, 2014) on CIFAR10 (Krizhevsky et al., 2009).

Original: Also here, we conducted our experiments on the same hardware described at section 3.
Revised: Also here, we conducted our experiments on the same hardware described at section 3.

Original: In both experiments, we test our approach on multiple values of k and compare it to k = 1 (standard
Revised: In both experiments, we test our approach on multiple values of k and compare it to k = 1 (standard

Original: SVD factorization).
Revised: SVD factorization).

Original: For every value of k , we compress each layer from the hidden fully connected layers of the given model by the same percentage and using the same value of k .
Revised: For every value of k , we compress each layer from the hidden fully connected layers of the given model by the same percentage and using the same value of k .

Original: LeNet- 300 - 100 .
Revised: LeNet- 300 - 100 .

Original: The network consists of 266610 parameters, it comprises two fully connected hidden layers with 300 and 100 neurons correspondingly, trained on MNIST data set.
Revised: The network consists of 266610 parameters, it comprises two fully connected hidden layers with 300 and 100 neurons correspondingly, trained on MNIST data set.

Original: We test our approach on k ∈ { 2 , 3 , 4 , 5 } .
Revised: We test our approach on k ∈ { 2 , 3 , 4 , 5 } .

Original: In Figure 9, we report the accuracy drop as a function of the compression rate for the whole network.
Revised: In Figure 9, we report the accuracy drop as a function of the compression rate for the whole network.

Original: We can see the advantage of our approach when compressing more than 90% of the network.
Revised: We can see the advantage of our approach when compressing more than 90% of the network.

Original: We used implementation network consists of 16 convolutions layers folby
Revised: We used implementation network consists of 16 convolutions layers fol

Original: Finallyclassiﬁcation
Revised: Finally

Original: Here, we tested our approach for k ∈ { 2 , 5 } .
Revised: Here, we tested our approach for k ∈ { 2 , 5 } .

Original: In Figure 10, we report the accuracy drop as a function of the compression rate of the fully connected layers.
Revised: In Figure 10, we report the accuracy drop as a function of the compression rate of the fully connected layers.

Original: The suggested approach has a clear advantage for high compression rates.
Revised: The suggested approach has a clear advantage for high compression rates.

Original: D MESSI- ENSEMBLE
Revised: D MESSI- ENSEMBLE

Original: In this section we show only the best computed results of DistilBERT: that is obtained by training models at several k values and then evaluating the model that achieves the best accuracy on the training set.
Revised: In this section we show only the best computed results of DistilBERT: that is obtained by training models at several k values and then evaluating the model that achieves the best accuracy on the training set.

Original: I.e., given a fully connected layer of n input neurons and d output neurons, for a given compression rate x (e.g., x = 0 .
Revised: I.e., given a fully connected layer of n input neurons and d output neurons, for a given compression rate x (e.g., x = 0 .

Original: 4 means that we want to remove 40% of the parameters), we try multiple values of k via binary search on k .
Revised: 4 means that we want to remove 40% of the parameters), we try multiple values of k via binary search on k .

Original: For every such k value we compute the implied value j = (1 − x ) dn/ ( n + kd ) , and we compress the network based on those k and j via the MESSI pipeline.
Revised: For every such k value we compute the implied value j = (1 − x ) dn/ ( n + kd ) , and we compress the network based on those k and j via the MESSI pipeline.

Original: Finally, we save the model that achieves the best accuracy on the training set, and evaluate its results on the test set.
Revised: Finally, we save the model that achieves the best accuracy on the training set, and evaluate its results on the test set.

Original: Figure 11 reports the results for this approach.
Revised: Figure 11 reports the results for this approach.

Original: E P ROTECTIVE CLUSTERING VS k - MEANS
Revised: E P ROTECTIVE CLUSTERING VS k - MEANS

Original: Recall the suggested pipeline from section 2:
Revised: Recall the suggested pipeline from section 2:

Original: The ﬁrst step of it is to compute a set of k subspaces in R d , each of dimension j that approximates the ( k, j ) -projective clustering of the input matrix A .
Revised: The ﬁrst step of it is to compute a set of k subspaces in R d , each of dimension j that approximates the ( k, j ) -projective clustering of the input matrix A .

Original: Then, the second step partitions the input neurons (rows of A ) according to there closest subspace from the set of k subspace that is computed in the ﬁrst step.
Revised: Then, the second step partitions the input neurons (rows of A ) according to there closest subspace from the set of k subspace that is computed in the ﬁrst step.

Original: After that in step 3 , we compute the SVD for each partition, and in step 4 and 5 we build (and maybe ﬁne-tune) the corresponding architecture as described e.g., in Figure 3.
Revised: After that in step 3 , we compute the SVD for each cluster, and in step 4 and 5 we build (and maybe ﬁne-tune) the corresponding architecture as described e.g., in Figure 3.

Original: In this section we compare the used clustering approach (projective clustering) to the known clustering method: k -means.
Revised: In this section we compare the used clustering approach (projective clustering) to the known clustering method: k -means.

Original: I.e, we do not apply steps 1 and 2 , and instead, we partition the input neurons (rows of A ) into k groups via applying k -means clustering on them (instead of projective clustering), we then apply steps 3 , 4 and 5 exactly the same.
Revised: I.e, we do not apply steps 1 and 2 , and instead, we partition the input neurons (rows of A ) into k groups via applying k -means clustering on them (instead of projective clustering), we then apply steps 3 , 4 and 5 exactly the same.

Original: Here, we evaluated our results on the networks: RoBERTa (Liu et al., 2019b) and DistilBERT (Sanh et al., 2019) on the ”RTE” and ”MRPC” tasks from the GLUE benchmark (Wang et al., 2018).
Revised: Here, we evaluated our results on the networks: RoBERTa (Liu et al., 2019b) and DistilBERT (Sanh et al., 2019) on the ”RTE” and ”MRPC” tasks from the GLUE benchmark (Wang et al., 2018).

Original: Figures 12 and 13 reports the results on RoBERTA with ﬁne tuning for two epochs (in both clustering methods) and without respectively, while Figures 14 and 15 do the same for the results on
Revised: Figures 12 and 13 reports the results on RoBERTA with ﬁne tuning for two epochs (in both clustering methods) and without respectively, while Figures 14 and 15 do the same for the results on

Original: DistilBERT.
Revised: DistilBERT.

Original: We also used the LeNet- 300 - 100 model on MNIST LeCun et al.
Revised: We also used the LeNet- 300 - 100 model on MNIST LeCun et al.

Original: (1998) to check this (same) experiment in a different setting.
Revised: (1998) to check this (same) experiment in a different setting.

Original: See Figure 16.
Revised: See Figure 16.

Original: discussion.
Revised: discussion.

Original: In Figure 13, where we test the accuracy drop before applying the ﬁne-tuning, we can see that using projective clustering for partitioning the neurons is better than running k -means on them, i.e., the projective clustering approach yielded a better start (accuracy before ﬁne-tuning) than the k -means approach for the learning process.
Revised: In Figure 13, where we test the accuracy drop before applying the ﬁne-tuning, we can see that using projective clustering for partitioning the neurons is better than running k -means on them, i.e., the projective clustering approach yielded a better start (accuracy before ﬁne-tuning) than the k -means approach for the learning process.

Original: This could be explained by the fact that our original approach (projective clustering) aims to compute a set of k subspaces (each of dimension j ) that minimzes the sum of squared distances from each row in the input matrix A (neuron) to its closest subsace from the set.
Revised: This could be explained by the fact that our original approach (projective clustering) aims to compute a set of k subspaces (each of dimension j ) that minimzes the sum of squared distances from each row in the input matrix A (neuron) to its closest subsace from the set.

Original: Hence, factoring the matrix
Revised: Hence, factoring the matrix

Original: A based on those subspaces gives a good approximation for it, which is not the case in the k -means clustering.
Revised: A based on those subspaces gives a good approximation for it, which is not the case in the k -means clustering.

Original: This advantage may explain the difference between the two approaches after ﬁne-tuning for the same number of epochs as can be seen in Figure 12.
Revised: This advantage may explain the difference between the two approaches after ﬁne-tuning for the same number of epochs as can be seen in Figure 12.

Original: On the other hand, in Figure 15, the two methods gave similar results in terms of accuracy before ﬁn tuning, and we can see that this effects the results after the ﬁne-tuning, where the two approaches also succeeded to get similar results as can be seen in Figure 13.
Revised: On the other hand, in Figure 15, the two methods gave similar results in terms of accuracy before ﬁn tuning, and we can see that this effects the results after the ﬁne-tuning, where the two approaches also succeeded to get similar results as can be seen in Figure 13.

Original: Hence, the better way to determine the partition (which determines the compressed architecture) and to initialize the new layer in the MESSI pipeline is the projective clustering approach.
Revised: Hence, the better way to determine the partition (which determines the compressed architecture) and to initialize the new layer in the MESSI pipeline is the projective clustering approach.

Original: E. G ENERALIZATIONS AND E XTENSIONS .
Revised: E. G ENERALIZATIONS AND E XTENSIONS .

Original: Here, we give more details about the suggested generalizations and extensions from section 2.1, we also add few more:
Revised: Here, we give more details about the suggested generalizations and extensions from section 2.1, we also add few more:

Original: (cid:96) q -error.
Revised: (cid:96) q -error.

Original: For simplicity, our suggested approach aims to minimize sum of squared distances to k subspaces.
Revised: For simplicity, our suggested approach aims to minimize sum of squared distances to k subspaces.

Original: However, it can be easily applied also to sum of distances from the points to the subspace.
Revised: However, it can be easily applied also to sum of distances from the points to the subspace.

Original: In this case, we aim to compute the maximum-likelihood of the generating subspaces assuming a
Revised: In this case, we aim to compute the maximum-likelihood of the generating subspaces assuming a

Original: Laplacian instead of Gaussian distribution.
Revised: Laplacian instead of Gaussian distribution.

Original: More generally, we may want to minimize the sum over every distance to the power of q > 0 .
Revised: More generally, we may want to minimize the sum over every distance to the power of q > 0 .

Original: , i.e., we take the q -norm (cid:107) err (cid:107) q where err is the distance between a point to its projection on its closest subspace.
Revised: , i.e., we take the q -norm (cid:107) err (cid:107) q where err is the distance between a point to its projection on its closest subspace.

Original: Even for k = 1 recent results of Tukan et al.
Revised: Even for k = 1 recent results of Tukan et al.

Original: (2020b) show improvement over SVD.
Revised: (2020b) show improvement over SVD.

Original: Observe that given the optimal subpaces, the system architecture in these cases remains the same as ours in Figure 3, and a local minimum can still be obtained by the suggested algorithm.
Revised: Observe that given the optimal subpaces, the system architecture in these cases remains the same as ours in Figure 3.

Original: The only difference is that the SVD computation of the optimal subspace for a cluster of points ( k = 1 ) should be replaced by more involved approximation algorithm for computing the subspace that minimizes sum over distances to the power of q ; see e.g. Tukan et al.

Original: (2020b)

Original: ; Clarkson and Woodruff (2015).

Original: Distance functions.
Revised: Distance functions.

Original: Similarly, we can replace the Euclidean (cid:96) 2 -distance by e.g. the Manhattan distance which is the (cid:96) 1 -norm between a point x and its projection, i.e., (cid:107) x − x (cid:48) (cid:107) 1 or sum of differences between the corresponding entries, instead of sum of squared entries, as in the Euclidean distance (cid:107) x − x (cid:48) (cid:107) 2 in this paper.
Revised: Similarly, we can replace the Euclidean (cid:96) 2 -distance by e.g. the Manhattan distance which is the (cid:96) 1 -norm between a point x and its projection, i.e., (cid:107) x − x (cid:48) (cid:107) 1 or sum of differences between the corresponding entries, instead of sum of squared entries, as in the Euclidean distance (cid:107) x − x (cid:48) (cid:107) 2 in this paper.

Original: More generally, we may use the (cid:96) p distance (cid:107) x − x (cid:48) (cid:107) p , or even non-distance functions such as M-Estimators that can handle outliers (as in Tukan et al.
Revised: More generally, we may use the (cid:96) p distance (cid:107) x − x (cid:48) (cid:107) p , or even non-distance functions such as M-Estimators that can handle outliers (as in Tukan et al.

Original: (2020a))
Revised: (2020a))

Original: by replacing dist( p, x ) with min { dist( p, x ) , t } where t > 0 is constant (threshold) that makes sure that far away points will not affect the overall sum too much.
Revised: by replacing dist( p, x ) with min { dist( p, x ) , t } where t > 0 is constant (threshold) that makes sure that far away points will not affect the overall sum too much.

Original: From an implementation perspective, the EM-algorithm for k -subspaces uses a k = 1 solver routine as a blackbox.
Revised: From an implementation perspective, the EM-algorithm for k -subspaces uses a k = 1 solver routine as a blackbox.

Original: Therefore extending to other distance functions is as simple as replacing the SVD solver (the k = 1 for Euclidean distance) by the corresponding solver for k = 1 .
Revised: Therefore extending to other distance functions is as simple as replacing the SVD solver (the k = 1 for Euclidean distance) by the corresponding solver for k = 1 .

Original: Non-uniform dimensions.
Revised: Non-uniform dimensions.

Original: In this paper we assume that k subspaces approximate the input points, and each subspace has dimension exactly j , where j, k ≥ 1 are given integers.
Revised: In this paper we assume that k subspaces approximate the input points, and each subspace has dimension exactly j , where j, k ≥ 1 are given integers.

Original: A better strategy is to allow each subspace to have a different dimension, j i for every i ∈ { 1 , · · · , k } , or add a constraint only on the sum j 1 + · · · + j k of dimensions.
Revised: A better strategy is to allow each subspace to have a different dimension, j i for every i ∈ { 1 , · · · , k } , or add a constraint only on the sum j 1 + · · · + j k of dimensions.

Original: Similarly, the number k may be tuned as in our experimental results.
Revised: Similarly, the number k may be tuned as in our experimental results.

Original: Using this approach we can improve the accuracy and enjoy the same compression rate.
Revised: Using this approach we can improve the accuracy and enjoy the same compression rate.

Original: This search or parameter tuning, however, might increase the computation time of the compressed network.
Revised: This search or parameter tuning, however, might increase the computation time of the compressed network.

Original: It also implies layers of different sizes (for each subspace) in Figure 3.
Revised: It also implies layers of different sizes (for each subspace) in Figure 3.

Original: Dictionary Learning.
Revised: Dictionary Learning.

Original: Our approach of projective clustering is strongly related to Dictionary Learning (Tosic and Frossard, 2011; Mairal et al., 2009).
Revised: Our approach of projective clustering is strongly related to Dictionary Learning (Tosic and Frossard, 2011; Mairal et al., 2009).

Original: Here, the input is a matrix A ∈ R n × d and the output is a “dictionary” V T ∈ R d × j and projections or atoms which are the rows of U ∈ R n × j that minimize (cid:107) A − UV (cid:107) under some norm.
Revised: Here, the input is a matrix A ∈ R n × d and the output is a “dictionary” V T ∈ R d × j and projections or atoms which are the rows of U ∈ R n × j that minimize (cid:107) A − UV (cid:107) under some norm.

Original: It is easy to prove that UV is simply the j -rank approximation of A , as explained in Section 1.
Revised: It is easy to prove that UV is simply the j -rank approximation of A , as explained in Section 1.

Original: However, if we have additional constraints, such as that every row of U should have, say, only k = 1 non-zero entries, then geometrically the columns of V T are the j lines that intersects the origin and minimize the sum of distances to the points.
Revised: However, if we have additional constraints, such as that every row of U should have, say, only k = 1 non-zero entries, then geometrically the columns of V T are the j lines that intersects the origin and minimize the sum of distances to the points.

Original: For k > 1 every point is projected onto the subspace that minimizes its distance and is spanned by k columns of V T .
Revised: For k > 1 every point is projected onto the subspace that minimizes its distance and is spanned by k columns of V T .

Original: Coresets.
Revised: Coresets.

Original: Coresets are a useful tool, especially in projective clustering, to reduce the size of the input (compress it in some sense) while preserving the optimal solution or even the sum of distances to any set of k subspaces.
Revised: Coresets are a useful tool, especially in projective clustering, to reduce the size of the input (compress it in some sense) while preserving the optimal solution or even the sum of distances to any set of k subspaces.

Original: However, we are not aware of any efﬁcient implementations and the dependency on d and k is usually exponential as in Edwards and Varadarajan (2005).
Revised: However, we are not aware of any efﬁcient implementations and the dependency on d and k is usually exponential as in Edwards and Varadarajan (2005).

Original: A natural open problem is to compute more efﬁcient and practical coresets for projective clustering.
Revised: A natural open problem is to compute more efﬁcient and practical coresets for projective clustering.

Revised: E. E XPERIMENTING ON (cid:96) q - ERROR

Revised: To get a taste of the suggested extensions, we tried the ﬁrst suggestion of (cid:96) q -error, with q = 1 .

Revised: I.e., we cluster the rows of the input matrix A based on the set of k -subspaces that minimizes the sum of (non-squared) distances from each row in A to its closest subspace from the set.

Revised: The local minimum of the new clustering problem can still be obtained by the suggested EM algorithm.

Revised: The only difference is that the SVD computation of the optimal subspace for a cluster of points ( k = 1 ) should be replaced by more involved approximation algorithm for computing the

Revised: [Table]

Revised: subspace that minimizes sum over distances to the power of q = 1 ; see e.g. Tukan et al.

Revised: (2020b);

Revised: Clarkson and Woodruff (2015).

Revised: However, this change increased the running time of the algorithm from minutes to days, this is due to the fact the deterministic approximation algorithms for the new problem ( (cid:96) 1 error) with k

Revised: =take a time of O ( nd 4 ) , where d = 768 in our case, and we need to run this approximation algorithm many times in the EM procedure.

Revised: For that, we conducted our experiments only on one network (RoBERTA) on 2 tasks from the GLUE benchmark (MRPC and RTE).

Revised: Table 3, shows the accuracy drop for both techniques for two values of j with k = 5 on the MRPC task, and the same on RTE with k = 10 .

Revised: It can be seen from the table, that mostly, using the (cid:96) 1 error as an initialization is better that the (cid:96) 2 .

Revised: However, after ﬁne-tuning for 2 epochs both approaches reached almost the same accuracy, even more, the (cid:96) 2 approach achieved a better accuracy sometime.

Revised: We leave this for future research.