Original: H OW INFORMATIVE IS THE A PPROXIMATION E RROR FROM T ENSOR D ECOMPOSITION FOR N EURAL N ETWORK C OMPRESSION ?
Revised: H OW INFORMATIVE IS THE A PPROXIMATION E RROR FROM T ENSOR D ECOMPOSITION FOR N EURAL N ETWORK C OMPRESSION ?

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Tensor decompositions have been successfully applied to compress neural networks.
Revised: Tensor decompositions have been successfully applied to compress neural networks.

Original: The compression algorithms using tensor decompositions commonly minimize the approximation error on the weights.
Revised: The compression algorithms using tensor decompositions commonly minimize the approximation error on the weights.

Original: Recent work assumes the approximation error on the weights is a proxy for the performance of the model to compress multiple layers and fine-tune the compressed model.
Revised: Recent work assumes the approximation error on the weights is a proxy for the performance of the model to compress multiple layers and fine-tune the compressed model.

Original: Surprisingly, little research has systematically evaluated which approximation errors can be used to make choices regarding the layer, tensor decomposition method, and level of compression.
Revised: Surprisingly, little research has systematically evaluated which approximation errors can be used to make choices regarding the layer, tensor decomposition method, and level of compression.

Original: To close this gap, we perform an experimental study to test if this assumption holds across different layers and types of decompositions, and what the effect of fine-tuning is.
Revised: To close this gap, we perform an experimental study to test if this assumption holds across different layers and types of decompositions, and what the effect of fine-tuning is.

Original: We include the approximation error on the features resulting from a compressed layer in our analysis to test if this provides a better proxy, as it explicitly takes the data into account.
Revised: We include the approximation error on the features resulting from a compressed layer in our analysis to test if this provides a better proxy, as it explicitly takes the data into account.

Original: We find the approximation error on the weights has a positive correlation with the performance error, before as well as after fine-tuning.
Revised: We find the approximation error on the weights has a positive correlation with the performance error, before as well as after fine-tuning.

Original: Basing the approximation error on the features does not improve the correlation significantly.
Revised: Basing the approximation error on the features does not improve the correlation significantly.

Original: While scaling the approximation error commonly is used to account for the different sizes of layers, the average correlation across layers is smaller than across all choices (i.e. layers, decompositions, and level of compression) before fine-tuning.
Revised: While scaling the approximation error commonly is used to account for the different sizes of layers, the average correlation across layers is smaller than across all choices (i.e. layers, decompositions, and level of compression) before fine-tuning.

Original: When calculating the correlation across the different decompositions, the average rank correlation is larger than across all choices.
Revised: When calculating the correlation across the different decompositions, the average rank correlation is larger than across all choices.

Original: This means multiple decompositions can be considered for compression and the approximation error can be used to choose between them.
Revised: This means multiple decompositions can be considered for compression and the approximation error can be used to choose between them.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Tensor Decompositions (TD) have shown potential for compressing pre-trained models, such as convolutional neural networks, by replacing the optimized weight tensor with a low-rank multilinear approximation with fewer parameters (Jaderberg et al., 2014; Lebedev et al., 2015;
Revised: Tensor Decompositions (TD) have shown potential for compressing pre-trained models, such as convolutional neural networks, by replacing the optimized weight tensor with a low-rank multi-linear approximation with fewer parameters (Jaderberg et al., 2014; Lebedev et al., 2015;

Original: Kim et al., 2016; Garipov et al., 2016; Kossaifi et al., 2019a).
Revised: Kim et al., 2016; Garipov et al., 2016; Kossaifi et al., 2019a).

Original: Common compression procedures (Lebedev et al., 2015; Garipov et al., 2016; Hawkins et al., 2021) work by iteratively applying TD on a selected weight tensor, where each time several decomposition choices have to be made regarding (i) the layer to compress, (ii) the type of decomposition, and (iii) the compression level.
Revised: Common compression procedures (Lebedev et al., 2015; Garipov et al., 2016; Hawkins et al., 2021) work by iteratively applying TD on a selected weight tensor, where each time several decomposition choices have to be made regarding (i) the layer to compress, (ii) the type of decomposition, and (iii) the compression level.

Original: Selecting the best hyperparameters for these choices at a given iteration requires costly re-evaluating the full model for each option.
Revised: Selecting the best hyperparameters for these choices at a given iteration requires costly re-evaluating the full model for each option.

Original: Recently, Liebenwein et al.
Revised: Recently, Liebenwein et al.

Original: (2021) suggested comparing the approximation errors on the decomposed weights as a more efficient alternative, though they only considered matrix decompositions for which analytical bounds on the resulting performance exist.
Revised: (2021) suggested comparing the approximation errors on the decomposed weights as a more efficient alternative, though they only considered matrix decompositions for which analytical bounds on the resulting performance exist.

Original: These bounds rely on the Eckhart-Young theorem.
Revised: These bounds rely on the Eckhart-Young-Mirsky theorem.

Original: For TD, no equivalent theorem is possible (Vannieuwenhoven et al., 2014).
Revised: For TD, no equivalent theorem is possible (Vannieuwenhoven et al., 2014).

Original: While theoretical bounds are not available for more general TD methods, the same concept could still be practical when considering TDs too.
Revised: While theoretical bounds are not available for more general TD methods, the same concept could still be practical when considering TDs too.

Original: We summarize this as the following general assumption:
Revised: We summarize this as the following general assumption:

Original: Assumption 1.
Revised: Assumption 1.

Original: A lower TD approximation error on a model’s weight tensor indicates better overall model performance after compression.
Revised: A lower TD approximation error on a model’s weight tensor indicates better overall model performance after compression.

Original: While this assumption appears intuitive and reasonable, we observe several gaps in the existing literature: First, most existing TD compression literature only focuses on a few decomposition choices, e.g. fixing the TD method (Lebedev et al., 2015;
Revised: While this assumption appears intuitive and reasonable, we observe several gaps in the existing literature: First, most existing TD compression literature only focuses on a few decomposition choices, e.g. fixing the TD method (Lebedev et al., 2015;

Original: Kim et al., 2016).
Revised: Kim et al., 2016).

Original: Although various error measures and decomposition choices have been studied in separation, no prior work systematically compares different decomposition errors across multiple decomposition choices.
Revised: Although various error measures and decomposition choices have been studied in separation, no prior work systematically compares different decomposition errors across multiple decomposition choices.

Original: Second, different decomposition errors with different properties have been used throughout the literature (Jaderberg et al., 2014), and it is unclear if some error measure should be preferred.
Revised: Second, different decomposition errors with different properties have been used throughout the literature (Jaderberg et al., 2014), and it is unclear if some error measure should be preferred.

Original: Third, a benefit of TD is that no training data is needed for compression, though if labeled data is available, more recent methods combine TD with a subsequent fine-tuning step.
Revised: Third, a benefit of TD is that no training data is needed for compression, though if labeled data is available, more recent methods combine TD with a subsequent fine-tuning step.

Original: Is the approximation error equally valid for the model performance with and without fine-tuning?
Revised: Is the approximation error equally valid for the model performance with and without fine-tuning?

Original: Overall, to the best of the authors’ knowledge, no prior work investigates if and which decomposition choices for TD network compression can be made using specific approximation errors.
Revised: Overall, to the best of the authors’ knowledge, no prior work investigates if and which decomposition choices for TD network compression can be made using specific approximation errors.

Original: This paper studies empirically to what extent a single decomposition error can predict a compressed model’s performance across varied decomposition choices, identifying how existing procedures could be improved, and providing support for specific practices.
Revised: This paper studies empirically to what extent a single decomposition error can predict a compressed model’s performance across varied decomposition choices, identifying how existing procedures could be improved, and providing support for specific practices.

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Original: • A first empirical study is proposed on the correlation between the approximation error on the model weights that result from compression with TD, and the performance of the compressed model 1 .
Revised: • A first empirical study is proposed on the correlation between the approximation error on the model weights that result from compression with TD, and the performance of the compressed model 1 .

Original: Studied decomposition choices include the layer, multiple decomposition methods (CP, Tucker, and Tensor Train), and level of compression.
Revised: Studied decomposition choices include the layer, multiple decomposition methods (CP, Tucker, and Tensor Train), and level of compression.

Original: Measurements are made using several models and datasets.
Revised: Measurements are made using several models and datasets.

Original: We show that the error is indicative of model performance, even when comparing multiple TD methods, though useful correlation only occurs at the higher compression levels.
Revised: We show that the error is indicative of model performance, even when comparing multiple TD methods, though useful correlation only occurs at the higher compression levels.

Original: • Different formulations for the approximation error measure are compared, including measuring the error on the features as motivated by the work Jaderberg et al.
Revised: • Different formulations for the approximation error measure are compared, including measuring the error on the features as motivated by the work Jaderberg et al.

Original: (2014); Denil et al.
Revised: (2014); Denil et al.

Original: (2014) which considers the data distribution.
Revised: (2014) which considers the data distribution.

Original: We further study how using training labeled data for additional fine-tuning affects the correlation.
Revised: We further study how using training labeled data for additional fine-tuning affects the correlation.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: There is currently no systematic study on how well the approximation error relates to a compressed neural network’s performance across multiple choices of network layers, TD methods, and compression levels.
Revised: There is currently no systematic study on how well the approximation error relates to a compressed neural network’s performance across multiple choices of network layers, TD methods, and compression levels.

Original: We here review the most similar and related studies where we distinguish works with theoretical versus empirical validation, different approximation error measures, and the role of fine-tuning after compression.
Revised: We here review the most similar and related studies where we distinguish works with theoretical versus empirical validation, different approximation error measures, and the role of fine-tuning after compression.

Original: The relationship between the approximation error on the weights and the performance of the model can be studied by theoretical analysis for matrix decompositions.
Revised: The relationship between the approximation error on the weights and the performance of the model was studied by theoretical analysis for matrix decompositions.

Original: Liebenwein et al.
Revised: Liebenwein et al.

Original: (2021) derive bounds on the model performance for SVD-based compression on the convolutional layers, and thus motivate that the SVD approximation error is a good proxy for the compressed model performance.
Revised: (2021) derive bounds on the model performance for SVD-based compression on the convolutional layers, and thus motivate that the SVD approximation error is a good proxy for the compressed model performance.

Original: Arora et al.
Revised: Arora et al.

Original: (2018) derive bounds on the generalization error for convolutional layers based on a compression error from their matrix projection algorithm.
Revised: (2018) derive bounds on the generalization error for convolutional layers based on a compression error from their matrix projection algorithm.

Original: Baykal et al.
Revised: Baykal et al.

Original: (2019) show how the amount of sparsity introduced in a model’s layers relates to its generalization performance.
Revised: (2019) show how the amount of sparsity introduced in a model’s layers relates to its generalization performance.

Original: While these works show that some theoretical bounds can be found for specific compression methods, such bounds are not available for TDs in general.
Revised: While these works show that some theoretical bounds can be found for specific compression methods, such bounds are not available for TDs in general.

Original: Other works, therefore, study the relationship for TD empirically.
Revised: Other works, therefore, study the relationship for TD empirically.

Original: For instance, Lebedev et al.
Revised: For instance, Lebedev et al.

Original: (2015) show how CP decomposition rank affects the approximation error, and the resulting accuracy drop as the rank is decreased.
Revised: (2015) show how CP decomposition rank affects the approximation error, and the resulting accuracy drop as the rank is decreased.

Original: Hawkins et al.
Revised: Hawkins et al.

Original: (2020) observe that, for networks with repeated layer blocks, the approximation error depends on the convolutional layers within the block.
Revised: (2020) observe that, for networks with repeated layer blocks, the approximation error depends on the convolutional layers within the block.

Original: When considering the model’s final task performance, the approximation error on the weights might not be the most relevant measure.
Revised: When considering the model’s final task performance, the approximation error on the weights might not be the most relevant measure.

Original: To consider the effect on the actual data distribution, Jaderberg et al.
Revised: To consider the effect on the actual data distribution, Jaderberg et al.

Original: (2014) instead propose to compute an error on the approximated output features of a layer after its weights have been compressed.
Revised: (2014) instead propose to compute an error on the approximated output features of a layer after its weights have been compressed.

Original: They found that compressing weights by minimizing the error on features, rather than the error on the weights, results in a smaller loss in classification accuracy.
Revised: They found that compressing weights by minimizing the error on features, rather than the error on the weights, results in a smaller loss in classification accuracy.

Original: However, Jaderberg et al.
Revised: However, Jaderberg et al.

Original: (2014) do not fine-tune the decomposed model, and only use a toy model with few layers.
Revised: (2014) do not fine-tune the decomposed model, and only use a toy model with few layers.

Original: Denil et al.
Revised: Denil et al.

Original: (2014) try to capture the information from the data via the empirical covariance matrix.
Revised: (2014) try to capture the information from the data via the empirical covariance matrix.

Original: Although this method eliminates the need for multiple passes over the data during the compression step, it is limited to a two-dimensional case.
Revised: Although this method eliminates the need for multiple passes over the data during the compression step, it is limited to a two-dimensional case.

Original: Eo et al.
Revised: Eo et al.

Original: (2021) forego looking at a compression error altogether, and selects the rank based on the accuracy on the validation set.
Revised: (2021) forego looking at a compression error altogether, and selects the rank based on the accuracy on the validation set.

Original: This requires the labels to be present and a forward pass through the whole network, even if the compressed layer is near the input.
Revised: This requires the labels to be present and a forward pass through the whole network, even if the compressed layer is near the input.

Original: Several norms have been used in the literature to quantify the approximation error, which is the difference between the pretrained weights and the compressed weights.
Revised: Several norms have been used in the literature to quantify the approximation error, which is the difference between the pretrained weights and the compressed weights.

Original: The works that decompose pretrained layers (Denton et al., 2014; Jaderberg et al., 2014; Lebedev et al., 2015; Novikov et al., 2015; Kim et al., 2016) explicitly minimize the Frobenius norm.
Revised: The works that decompose pretrained layers (Denton et al., 2014; Jaderberg et al., 2014; Lebedev et al., 2015; Novikov et al., 2015; Kim et al., 2016) explicitly minimize the Frobenius norm.

Original: Hawkins et al.
Revised: Hawkins et al.

Original: ; Liebenwein et al.
Revised: ; Liebenwein et al.

Original: (2021) calculate the relative Frobenius norm, i.e., the norm of the error proportional to the norm of the pretrained weights, to compare the error for layers of different sizes.
Revised: (2021) calculate the relative Frobenius norm, i.e., the norm of the error proportional to the norm of the pretrained weights, to compare the error for layers of different sizes.

Original: Still, it remains unclear which error measure is most informative for the compressed model’s final performance.
Revised: Still, it remains unclear which error measure is most informative for the compressed model’s final performance.

Original: In practice, when training data is still available for compression, fine-tuning for the target task after compressing weights could recover some of the lost performance (Denton et al., 2014; Lebedev et al., 2015; Kim et al., 2016).
Revised: In practice, when training data is still available for compression, fine-tuning for the target task after compressing weights could recover some of the lost performance (Denton et al., 2014; Lebedev et al., 2015; Kim et al., 2016).

Original: Adding fine-tuning results in a three-step process: pretrain, compress and fine-tune.
Revised: Adding fine-tuning results in a three-step process: pretrain, compress and fine-tune.

Original: Optimization thus alternates between minimizing the error of respectively the features, the weights, and finally the features again.
Revised: Optimization thus alternates between minimizing the error of respectively the features, the weights, and finally the features again.

Original: While Denton et al.
Revised: While Denton et al.

Original: Kim et al.
Revised: Kim et al.

Original: (2016) compare compressed model performance before and after fine-tuning, they do not investigate how the finetuned network performance relates to the weight compression error.
Revised: (2016) compare compressed model performance before and after fine-tuning, they do not investigate how the finetuned network performance relates to the weight compression error.

Original: Lebedev et al.
Revised: Lebedev et al.

Original: (2015) does study the compression error for CP decomposition, but only reports performance with fine-tuning.
Revised: (2015) does study the compression error for CP decomposition, but only reports performance with fine-tuning.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: We consider the task of compressing a pretrained neural network with TD.
Revised: We consider the task of compressing a pretrained neural network with TD.

Original: While TD is a general technique that could be applied to many types of layers, the focus will be on convolutional layers.
Revised: While TD is a general technique that could be applied to many types of layers, the focus will be on convolutional layers.

Original: Due to their ubiquity and suitability to compare different types of higher-order decompositions, as the layer weights are four-dimensional tensors.
Revised: Due to their ubiquity and suitability to compare different types of higher-order decompositions, as the layer weights are four-dimensional tensors.

Original: Generally, a compression procedure will iteratively apply TD to the weights of selected layers, making several choices on how and what weight tensor to decompose, while ideally maintaining as much of the original network’s performance.
Revised: Generally, a compression procedure will iteratively apply TD to the weights of selected layers, making several choices on how and what weight tensor to decompose, while ideally maintaining as much of the original network’s performance.

Original: In its original uncompressed form, the full-rank weight tensors W ∈ R C × H × W × T of a layer represent a local optimum in the network’s parameter space with respect to the training data and loss, where C is the number of input channels, H and W are the height and width of the convolutional kernel, and T is the number of output channels.
Revised: In its original uncompressed form, the full-rank weight tensors W ∈ R C × H × W × T of a layer represent a local optimum in the network’s parameter space with respect to the training data and loss, where C is the number of input channels, H and W are the height and width of the convolutional kernel, and T is the number of output channels.

Original: When a
Revised: When a

Original: TD is applied to the weights of a specific layer, this results in a factorized structure (cid:101) W composed of multiple smaller tensor multiplications, which replaces the original weights in the network.
Revised: TD is applied to the weights of a specific layer, this results in a factorized structure (cid:101) W composed of multiple smaller tensor multiplications, which replaces the original weights in the network.

Original: Each time TD is applied, several decomposition choices need to be evaluated:
Revised: Each time TD is applied, several decomposition choices need to be evaluated:

Original: Layer:
Revised: Layer:

Original: The layer l from the set of network layers L = { 1 , 2 , · · · , L } to decompose.
Revised: The layer l from the set of network layers L = { 1 , 2 , · · · , L } to decompose.

Original: Method: The type of TD method m ∈ M = { CP , Tucker , Tensor Train } .
Revised: Method: The type of TD method m ∈ M = { CP , Tucker , Tensor Train } .

Original: The decomposition determines the factorized structure of (cid:101) W . 3.
Revised: The decomposition determines the factorized structure of (cid:101) W . 3.

Original: Compression: The compression level c ∈ C for the selected layer.
Revised: Compression: The compression level c ∈ C for the selected layer.

Original: Here C ⊂ [0 , 1] is some finite set of testable levels, and c = 0 .
Revised: Here C ⊂ (0 , 1) is some finite set of testable levels, and c = 0 .

Original: 75 means the number of parameters is reduced by 75% and the factorized layer contains only 25% of the parameters.
Revised: 75 means the number of parameters is reduced by 75% and the factorized layer contains only 25% of the parameters.

Original: A given compression level is achieved by decomposing the tensor to some rank R .
Revised: A given compression level is achieved by decomposing the tensor to some rank R .

Original: The rank for some compression level c depends on the selected TD method.
Revised: The rank for some compression level c depends on the selected TD method.

Original: We will refer to H =
Revised: We will refer to H =

Original: L × M × C as the set with possible hyperparameter values for ( l, m, c ) to consider.
Revised: L × M × C as the set with possible hyperparameter values for ( l, m, c ) to consider.

Original: Note that compression procedures in the literature might only consider a subset of these choices.
Revised: Note that compression procedures in the literature might only consider a subset of these choices.

Original: For example, a procedure might fix the layer for a given iteration or only consider a single TD method.
Revised: For example, a procedure might fix the layer for a given iteration or only consider a single TD method.

Original: In practice, it is computationally infeasible to evaluate the compressed network’s performance for every possible hyperparameter choice at every compression iteration, especially when optimizing for performance after fine-tuning.
Revised: In practice, it is computationally infeasible to evaluate the compressed network’s performance for every possible hyperparameter choice at every compression iteration, especially when optimizing for performance after fine-tuning.

Original: Instead, automated compression procedures will efficiently compare an approximation error a i =
Revised: Instead, automated compression procedures will efficiently compare an approximation error a i =

Original: e ( (cid:101) W i , W ) between the original and decomposed weights using a particular choice of hyperparameters
Revised: e ( (cid:101) W i , W ) between the original and decomposed weights using a particular choice of hyperparameters

Original: h i ∈ H .
Revised: h i ∈ H .

Original: In doing so one relies on Assumptionthat a lower approximation error is indicative of better compressed performance p i .
Revised: In doing so one relies on Assumptionthat a lower approximation error is indicative of better compressed performance p i .

Original: If annotated training data is available, additional network fine-tuning on the decomposed structure could result in improved performance p ⋆ i > p i .
Revised: If annotated training data is available, additional network fine-tuning on the decomposed structure could result in improved performance p ⋆ i > p i .

Original: In this work, we propose to focus on a single iteration and investigate Assumption 1 in isolation of any specific compression procedure.
Revised: In this work, we propose to focus on a single iteration and investigate Assumption 1 in isolation of any specific compression procedure.

Original: Our aim is thus to assess how well computing an approximation error e can predict the optimal compression choice from some hypothesis set H , i.e. the choice that results in the lowest compressed network performance error p , or even performance after fine-tuning p ⋆ .
Revised: Our aim is thus to assess how well computing an approximation error e can predict the optimal compression choice from some hypothesis set H , i.e. the choice that results in the lowest compressed network performance error p , or even performance after fine-tuning p ⋆ .

Original: We will study the correlation between approximation error and model performance empirically in our experiments, using the procedure and correlation metric explained in Section 3.1.
Revised: We will study the correlation between approximation error and model performance empirically in our experiments, using the procedure and correlation metric explained in Section 3.1.

Original: Details on the different approximation errors that we will explore are covered in Section 3.2.
Revised: Details on the different approximation errors that we will explore are covered in Section 3.2.

Original: Finally, the considered TD methods are explained in Section 3.3.
Revised: Finally, the considered TD methods are explained in Section 3.3.

Original: 3. E MPIRICAL EVALUATION OF ERROR - PERFORMANCE CORRELATION
Revised: 3. E MPIRICAL EVALUATION OF ERROR - PERFORMANCE CORRELATION

Original: Our proposed empirical evaluation procedure will evaluate a large set of hyperparameters H
Revised: Our proposed empirical evaluation procedure will evaluate a large set of hyperparameters H

Original: = { h 1 , h 2 , · · · } on multiple convolutional neural networks and datasets (see Section 4 for details) for different options of approximation error metric
Revised: = { h 1 , h 2 , · · · } on multiple convolutional neural networks and datasets (see Section 4) for different options of approximation error metric

Original: (see Section 3.2).
Revised: (see Section 3.2).

Original: For a given model, dataset, and approximation error metric e , the procedure evaluates for each set of hyperparameter choices h
Revised: For a given model, dataset, and approximation error metric e , the procedure evaluates for each set of hyperparameter choices h

Original: i ∈ H the approximation error
Revised: i ∈ H the approximation error

Original: a i =
Revised: a i =

Original: e ( ˜ W i , W ) , the model performance error p
Revised: e ( ˜ W i , W ) , the model performance error p

Original: i on the validation split, and the model performance error p
Revised: i on the validation split, and the model performance error p

Original: ⋆i after additional fine-tuning on the training data.
Revised: ⋆i after additional fine-tuning on the training data.

Original: We thus obtain sets of measurements
Revised: We thus obtain sets of measurements

Original: A = { a 1 , a 2 , · · · } , P = { p 1 , p 2 , · · · } and P ⋆ = { p ⋆ 1 , p ⋆ 2 , · · · } for H .
Revised: A = { a 1 , a 2 , · · · } , P = { p 1 , p 2 , · · · } and P ⋆ = { p ⋆ 1 , p ⋆ 2 , · · · } for H .

Original: When comparing two sets of hyperparameters h i ∈ H and h j ∈ H , we want to establish if the set with the smaller approximation error results in a smaller classification error of the model.
Revised: When comparing two sets of hyperparameters h i ∈ H and h j ∈ H , we want to establish if the set with the smaller approximation error results in a smaller performance error of the model.

Original: In other words, the concordance of pairs of measurements needs to be established.
Revised: In other words, the concordance of pairs of measurements needs to be established.

Original: Concordant pairs have a larger (smaller) performance error when the approximation error is larger (smaller) between two sets of hyperparameter choices, i.e. i and j are concordant if a i > a j and p i > p j or if a i < a j and p i < p j , and discordant otherwise.
Revised: Concordant pairs have a larger (smaller) performance error when the approximation error is larger (smaller) between two sets of hyperparameter choices, i.e. i and j are concordant if a i > a j and p i > p j or if a i < a j and p i < p j , and discordant otherwise.

Original: Kendall’s τ is a measure for the rank correlation (Kendall, 1938) or ordinal association between two order sets, in our case between approximation errors e and a model performances P (or P ⋆ ).
Revised: Kendall’s τ is a measure for the rank correlation (Kendall, 1938) or ordinal association between two order sets, in our case between approximation errors e and a model performances P (or P ⋆ ).

Original: To avoid confusion with the concept of tensor rank, we will refer to Kendall’s τ simply as correlation .
Revised: To avoid confusion with the concept of tensor rank, we will refer to Kendall’s τ simply as correlation .

Original: For this correlation measure, the difference between the number of concordant pairs ( k ) and discordant pairs ( d ) is scaled with the binomial coefficient m ( m − 1) / 2 to account for the different ways two measurements can be sampled from a total of m measurements: 
Revised: For this correlation measure, the difference between the number of concordant pairs ( k ) and discordant pairs ( d ) is scaled with the binomial coefficient m ( m − 1) /to account for the different ways two measurements can be sampled from a total of m measurements: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Kendall’s τ can be interpreted as follows:
Revised: Kendall’s τ can be interpreted as follows:

Original: τ = 1 indicates a completely positive rank correlation, τ [-= 
 no-] correlation, and τ = − 1 a strong negative correlation.
Revised: τ = 1 indicates a perfect positive rank correlation, τ [-= 
 no-]=no correlation, and τ = − 1 a strong negative correlation.

Original: For a set of hyperparameters H , a useful approximation error e would thus result in a τ close to ±1, indicating it is predictive of the model’s performance.
Revised: For a set of hyperparameters H , a useful approximation error e would thus result in a τ close to ±1, indicating it is predictive of the model’s performance.

Original: Note that Kendall’s τ does not depend on assumptions about the underlying distribution, whereas Pearson correlation assumes a linear relationship between the two measurements.
Revised: Note that Kendall’s τ does not depend on assumptions about the underlying distribution, whereas Pearson correlation assumes a linear relationship between the two measurements.

Original: Kendall’s τ is used over Spearman’s ρ because the interpretation of con- and discordance pairs for Kendall’s τ is closely related to our use case of choosing between hyperparameter sets.
Revised: Kendall’s τ is used over Spearman’s ρ because the interpretation of con- and discordance pairs for Kendall’s τ is closely related to our use case of choosing between hyperparameter sets.

Original: 3. A PPROXIMATION ERRORS
Revised: 3. A PPROXIMATION ERRORS

Original: We now discuss various measures e ( (cid:101) W , W ) to quantify the approximation error.
Revised: We now discuss various measures e ( (cid:101) W , W ) to quantify the approximation error.

Original: The basis is to compute some norm on the difference between these tensors, in this work we use the Frobenius norm as is common in the literature (Lebedev et al., 2015; Hawkins et al., 2020).
Revised: The basis is to compute some norm on the difference between these tensors, in this work we use the Frobenius norm as is common in the literature (Lebedev et al., 2015; Hawkins et al., 2020).

Original: We shall consider three options to scale the norm, which could help make the error more robust when comparing hypotheses with different layers.
Revised: We shall consider three options to scale the norm, which could help make the error more robust when comparing hypotheses with different layers.

Original: Additionally, we can consider two options to compute the error on, namely either directly the weights or on the features.
Revised: Additionally, we can consider two options to compute the error on, namely either directly the weights or on the features.

Original: In total, we shall thus explore six different approximation errors in this work.
Revised: In total, we shall thus explore six different approximation errors in this work.

Original: An overview is presented in Table 1.
Revised: An overview is presented in Table 1.

Original: Normalization
Revised: Normalization

Original: The norm between the difference of the weights is referred to as absolute norm and is used in the objective function when decomposing the pretrained weights.
Revised: The norm between the difference of the weights is referred to as absolute norm and is used in the objective function when decomposing the pretrained weights.

Original: The relative norm is used in TD literature to compare errors between different layers (Lebedev et al., 2015; Hawkins et al., 2020), as it is invariant to the size of the weights.
Revised: The relative norm is used in TD literature to compare errors between different layers (Lebedev et al., 2015; Hawkins et al., 2020), as it is invariant to the size of the weights.

Original: Alternatively, the norm of the difference can be scaled to account for the number of parameters, while keeping the distance from the weights.
Revised: Alternatively, the norm of the difference can be scaled to account for the number of parameters, while keeping the distance from the weights.

Original: Target tensor
Revised: Target tensor

Original: The most common option is to compute approximation error on the decomposed layer’s weights , W .
Revised: The most common option is to compute approximation error on the decomposed layer’s weights , W .

Original: However, Jaderberg et al.
Revised: However, Jaderberg et al.

Original: (2014) achieved promising results basing the decomposition on the approximation error of the features.
Revised: (2014) achieved promising results basing the decomposition on the approximation error of the features.

Original: Errors in some elements of the weight tensor might be more permissible if they do not affect the resulting feature space.
Revised: Errors in some elements of the weight tensor might be more permissible if they do not affect the resulting feature space.

Original: We therefore also consider the expected error on the features F = X · W , which is the output tensor of the layer after convolving its input with weights W on input data X .
Revised: We therefore also consider the expected error on the features F = X · W , which is the output tensor of the layer after convolving its input with weights W on input data X .

Original: Likewise, approximated weights (cid:101) W result in an approximated (cid:101) F .
Revised: Likewise, approximated weights (cid:101) W result in an approximated (cid:101) F .

Original: In practice, computing the feature space requires input data and is computationally more demanding than computing the weight approximation error.
Revised: In practice, computing the feature space requires input data and is computationally more demanding than computing the weight approximation error.

Original: However, it is potentially more representative of an approximation’s effect on the output, plus unlike a later fine-tuning step, it could even be used if only data without labels is available.
Revised: However, it is potentially more representative of an approximation’s effect on the output, plus unlike a later fine-tuning step, it could even be used if only data without labels is available.

Original: [Table]
Revised: [Table]

Original: 3. T ENSOR D ECOMPOSITION METHODS
Revised: 3. T ENSOR D ECOMPOSITION METHODS

Original: Our experiments shall consider three popular decomposition methods for convolutional layers, namely CP (Denton et al., 2014; Jaderberg et al., 2014;
Revised: Our experiments shall consider three popular decomposition methods for convolutional layers, namely CP (Denton et al., 2014; Jaderberg et al., 2014;

Original: Lebedev et al., 2015), Tucker (Kim et al., 2016), and Tensor Train (TT) (Garipov et al., 2016).
Revised: Lebedev et al., 2015), Tucker (Kim et al., 2016), and Tensor Train (TT) (Garipov et al., 2016).

Original: During the decomposition step the decomposed weights are found by minimizing the approximation error between the pretrained weights and the estimated decomposition: arg min (cid:101) W || W − (cid:101) W
Revised: During the decomposition step the decomposed weights are found by minimizing the approximation error between the pretrained weights and the estimated decomposition: arg min (cid:101) W || W − (cid:101) W

Original: || .
Revised: || .

Original: For CP this is done with ALS (Carroll & Chang, 1970; Harshman, 1972), for Tucker with HOSVD (De Lathauwer et al., 2000), and TT-SVD (Oseledets, 2011) for Tensor Train.
Revised: For CP this is done with ALS (Carroll & Chang, 1970; Harshman, 1972), for Tucker with HOSVD (De Lathauwer et al., 2000), and TT-SVD (Oseledets, 2011) for Tensor Train.

Original: The ALS algorithm requires a random initialization.
Revised: The ALS algorithm requires a random initialization.

Original: We sample from a uniform distribution [0,1) using Tensorly (Kossaifi et al., 2019b).
Revised: We sample from a uniform distribution [0,1) using Tensorly (Kossaifi et al., 2019b).

Original: The desired compression level is achieved by finding the corresponding rank, using the package Tensorly-Torch (Kossaifi et al., 2019b).
Revised: The desired compression level is achieved by finding the corresponding rank, using the package Tensorly-Torch (Kossaifi et al., 2019b).

Original: The ranks used for CP, Tucker, and TT are given in Appendix A.1.
Revised: The ranks used for CP, Tucker, and TT are given in Appendix A.1.

Original: For completeness, we list all considered decompositions with a 4-way tensor W .
Revised: For completeness, we list all considered decompositions with a 4-way tensor W .

Original: CP decomposition A rank- R CP decomposition (Hitchcock, 1927) sums R rank-one tensors:
Revised: CP decomposition A rank- R CP decomposition (Hitchcock, 1927) sums R rank-one tensors:

Original: [Equation]
Revised: [Equation]

Original: Tucker decomposition A Tucker decomposition (Tucker, 1966) is distinct from a CP by the
Revised: Tucker decomposition A Tucker decomposition (Tucker, 1966) is distinct from a CP by the

Original: Tucker core G ∈ R R 1 × R 2 × R 3 × R 4 .
Revised: Tucker core G ∈ R R 1 × R 2 × R 3 × R 4 .

Original: The Tucker rank is defined as the four-tuple ( R 1 , R 2 , R 3 , R 4 ) .
Revised: The Tucker rank is defined as the four-tuple ( R 1 , R 2 , R 3 , R 4 ) .

Original: Since the dimensions of the convolutional weights are small with respect to the width and height dimensions, it is computationally more efficient to contract these with the Tucker core G and form a new core H =
Revised: Since the dimensions of the convolutional weights are small with respect to the width and height dimensions, it is computationally more efficient to contract these with the Tucker core G and form a new core H =

Original: G × 2 Y × 3 X , where × n is the n-mode product (Appendix A.2):
Revised: G × 2 Y × 3 X , where × n is the n-mode product (Appendix A.2):

Original: [Equation]
Revised: [Equation]

Original: Tensor Train decomposition Another alternative is the Tensor Train decomposition (Oseledets, 2011), which decomposes a given tensor as a linear chain of 2-way and 3-way tensors, where the first and last tensors are 2-way.
Revised: Tensor Train decomposition Another alternative is the Tensor Train decomposition (Oseledets, 2011), which decomposes a given tensor as a linear chain of 2-way and 3-way tensors, where the first and last tensors are 2-way.

Original: The TT-rank in our four-dimensional case is the 3-tuple ( R 1 , R 2 , R 3 ) :
Revised: The TT-rank in our four-dimensional case is the 3-tuple ( R 1 , R 2 , R 3 ) :

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: This section discusses the implementation details and  the results of our empirical approach.
Revised: This section provides the implementation details and discusses the results of our empirical approach.

Original: 4. E XPERIMENTAL SETUP
Revised: 4. E XPERIMENTAL SETUP

Original: We discuss details of the datasets, models, and decomposition choices used for our experiments.

Original: Datasets The experiments are run on the datasets CIFAR-10 (Krizhevsky, 2009) and FashionMNIST (Xiao et al., 2017).
Revised: Datasets The experiments are run on the datasets CIFAR-10 (Krizhevsky, 2009) and FashionMNIST (Xiao et al., 2017).

Original: These datasets are common classification benchmarks for testing TD in CNNs (Cheng et al., 2021;
Revised: These datasets are common classification benchmarks for testing TD in CNNs (Cheng et al., 2021;

Original: Denil et al., 2014; Wu et al., 2020; Garipov et al., 2016; Hawkins et al., 2021).
Revised: Denil et al., 2014; Wu et al., 2020; Garipov et al., 2016; Hawkins et al., 2021).

Original: For both datasets, the original training sets are split into the set used for training and validation.
Revised: For both datasets, the original training sets are split into the set used for training and validation.

Original: The split is made such that equal class distributions are maintained.
Revised: The split is made such that equal class distributions are maintained.

Original: The details specific to the datasets are as follows: CIFAR-10 has 10 classes distributed equally across 60,000 images of 32 × 32 pixels with 3 color channels.
Revised: The details specific to the datasets are as follows: CIFAR-10 has 10 classes distributed equally across 60,000 images of 32 × 32 pixels with 3 color channels.

Original: After our validation split, there are 45,000 images in the training set and 5,000 in the validation set.
Revised: After our validation split, there are 45,000 images in the training set and 5,000 in the validation set.

Original: The test set of 10,000 images remains unchanged.
Revised: The test set of 10,000 images remains unchanged.

Original: Fashion-MNIST has 10 classes distributed equally across 70,000 grayscale images of 28 × 28 pixels.
Revised: Fashion-MNIST has 10 classes distributed equally across 70,000 grayscale images of 28 × 28 pixels.

Original: After our validation split, there are 55,000 images in the training set and 5,000 in the validation set.
Revised: After our validation split, there are 55,000 images in the training set and 5,000 in the validation set.

Original: Model architecture and training The models used are ResNet-18 (He et al., 2016) and
Revised: Model architecture and training The models used are ResNet-18 (He et al., 2016) and

Original: GaripovNet (Garipov et al., 2016).
Revised: GaripovNet (Garipov et al., 2016).

Original: ResNet is a well-performing state-of-the-art convolutional neural network.
Revised: ResNet is a well-performing state-of-the-art convolutional neural network.

Original: GaripovNet is a 7-layer convolutional neural network proposed in Garipov et al.
Revised: GaripovNet is a 7-layer convolutional neural network proposed in Garipov et al.

Original: and used by Hawkins et al.
Revised: and used by Hawkins et al.

Original: (2020) for image classification.
Revised: (2020) for image classification.

Original: These models enable comparison with other works within and beyond TD for deep learning (Gusak et al., 2019;
Revised: These models enable comparison with other works within and beyond TD for deep learning (Gusak et al., 2019;

Original: Garipov et al., 2016; Hawkins et al., 2020; Chu & Lee, 2021; Kossaifi et al., 2020).
Revised: Garipov et al., 2016; Hawkins et al., 2020; Chu & Lee, 2021; Kossaifi et al., 2020).

Original: The following hyperparameters are used:
Revised: The following hyperparameters are used:

Original: ResNet-18 is trained with batch size 128, for 300 epochs, with Adam optimizer and a learning rate of 10 − 3 .
Revised: ResNet-18 is trained with batch size 128, for 300 epochs, with Adam optimizer and a learning rate of 10 − 3 .

Original: At epochs 100 and 150 the learning rate is multiplied by 
Revised: At epochs 100 and 150 the learning rate is multiplied by 0.1.

Original: GaripovNet is trained with the same settings as the original paper Garipov et al.
Revised: GaripovNet is trained with the same settings as the original paper Garipov et al.

Original: The model is trained with Stochastic
Revised: The model is trained with Stochastic

Original: Gradient Descent (SGD) with a momentum of 0.9 and a learning rate of 0.1, multiplied by 0.1 at epochs 30, 60, and 90.
Revised: Gradient Descent (SGD) with a momentum of 0.9 and a learning rate of 0.1, multiplied by 0.1 at epochs 30, 60, and 90.

Original: The validation set is used for early stopping and the selection of the training hyperparameters, i.e. learning rate, schedule, level of annealing, batch size, and optimizer.
Revised: The validation set is used for early stopping and the selection of the training hyperparameters, i.e. learning rate, schedule, level of annealing, batch size, and optimizer.

Original: Training data is augmented with a random crop (padding with 4 pixels and cropping to the original size) and a random horizontal flip.
Revised: Training data is augmented with a random crop (padding with 4 pixels and cropping to the original size) and a random horizontal flip.

Original: All images are standardized based on the training mean and standard deviation per channel overall training samples.
Revised: All images are standardized based on the training mean and standard deviation per channel overall training samples.

Original: Early stopping is applied for both training the baseline and fine-tuning the decomposed model.
Revised: Early stopping is applied for both training the baseline and fine-tuning the decomposed model.

Original: The classification error on the test set is used for performance errors P .
Revised: The classification error on the test set is used for performance errors P .

Original: To
Revised: To

Original: Fine-tune after decomposition and obtain performance errors P ⋆ , the ResNet-18 is optimized for another 25 epochs, and GaripovNet for 10 epochs, using the last learning rate from the training.
Revised: Fine-tune after decomposition and obtain performance errors P ⋆ , the ResNet-18 is optimized for another 25 epochs, and GaripovNet for 10 epochs, using the last learning rate from the training.

Original: Decomposition choices We now explain the considered values for the decomposition choices H explained in Section 3.
Revised: Decomposition choices We now explain the considered values for the decomposition choices H explained in Section 3.

Original: For both neural network models, neither the first nor the last layer will be decomposed, as these layers already contain a relatively small amount of parameters.
Revised: For both neural network models, neither the first nor the last layer will be decomposed, as these layers already contain a relatively small amount of parameters.

Original: For GaripovNet the other five layers are part of L .
Revised: For GaripovNet the other five layers are part of L .

Original: For ResNet-18, L contains a selection of eight convolutional layers, details of which can be found in Appendix A.3.
Revised: For ResNet-18, L contains a selection of eight convolutional layers, details of which can be found in Appendix A.3.

Original: The set of TD methods that will be considered is M = { CP , Tucker , Tensor Train } , which were discussed in Section 3.3.
Revised: The set of TD methods that will be considered is M = { CP , Tucker , Tensor Train } , which were discussed in Section 3.3.

Original: The set of compression levels is C = { 10% , 25% , 50% , 75% , 90% } .
Revised: The set of compression levels is C = { 10% , 25% , 50% , 75% , 90% } .

Original: Multiple levels of compression are considered as each neural network layer can have different efficiency-performance trade-offs (Lebedev et al., 2015).
Revised: Multiple levels of compression are considered as each neural network layer can have different efficiency-performance trade-offs (Lebedev et al., 2015).

Original: In the experiments, we evaluate ResNet-18 on CIFAR-10, GaripovNet on CIFAR-10, and GaripovNet on F-MNIST.
Revised: In the experiments, we evaluate ResNet-18 on CIFAR-10, GaripovNet on CIFAR-10, and GaripovNet on F-MNIST.

Original: We exclude ResNet-18 on F-MNIST as the dataset is not sufficiently challenging for this model, and compressing one layer does not lead to a viable impact on the performance due to the model’s size and skip-connections.
Revised: We exclude ResNet-18 on F-MNIST as the dataset is not sufficiently challenging for this model, and compressing one layer does not lead to a viable impact on the performance due to the model’s size and skip-connections.

Original: Variance The process of decomposing and fine-tuning is repeated for five independent runs for each choice of layer, decomposition method, and compression level to assess and report variance in the results.
Revised: Variance The process of decomposing and fine-tuning is repeated for five independent runs for each choice of layer, decomposition method, and compression level to assess and report variance in the results.

Original: Note that due to the stochasticity of the ALS algorithms, the random initializations can result in different CP decomposition estimates.
Revised: Note that due to the stochasticity of the ALS algorithms, the random initializations can result in different CP decomposition estimates.

Original: The variance in correlation shown in the plots without fine-tuning results from the randomness in the CP initialization.
Revised: The variance in correlation shown in the plots without fine-tuning results from the randomness in the CP initialization.

Original: Fine-tuning adds additional variance through its use of batched SGD.
Revised: Fine-tuning adds additional variance through its use of batched SGD.

Original: The observed variance with fine-tuning accounts for both the randomness from CP initialization as well as from fine-tuning, thereby representing all sources of randomness in our methodology.
Revised: The observed variance with fine-tuning accounts for both the randomness from CP initialization as well as from fine-tuning, thereby representing all sources of randomness in our methodology.

Original: All runs for a given model and dataset are based on the same pretrained weights, so this is not a source of reported variance.
Revised: All runs for a given model and dataset are based on the same pretrained weights, so this is not a source of reported variance.

Original: In total, this results inmeasurements for ResNet-18 and 375 measurements for GaripovNet per dataset.
Revised: In total, this results inmeasurements for ResNet-18 and 375 measurements for GaripovNet per dataset.

Original: [Figure]
Revised: [Figure]

Original: 4. E XPERIMENTAL RESULTS
Revised: E XPERIMENTAL RESULTS

Original: Impact of compression levels on correlation We start by calculating the correlation across the layers and decomposition methods for multiple runs and calculate the averages grouped by compression levels.
Revised: Impact of compression levels on correlation We start by calculating the correlation across the layers and decomposition methods for multiple runs and calculate the averages grouped by compression levels.

Original: This is presented in Figure 1, where the bars are the average correlation τ between the
Revised: This is presented in Figure 1, where the bars are the average correlation τ between the

Original: Relative Weights and the classification error.
Revised: Relative Weights and the classification error.

Original: The correlation is only based on the Relative Weights, as this is the most common metric in recent literature (Lebedev et al., 2015; Hawkins et al., 2020;
Revised: The correlation is only based on the Relative Weights, as this is the most common metric in recent literature (Lebedev et al., 2015; Hawkins et al., 2020;

Original: Liebenwein et al., 2021).
Revised: Liebenwein et al., 2021).

Original: The correlations are grouped by the different levels of compression and represented by different colors.
Revised: The correlations are grouped by the different levels of compression and represented by different colors.

Original: The error bars are ±1 standard deviation, representing the variance from multiple runs.
Revised: The error bars are ±1 standard deviation, representing the variance from multiple runs.

Original: In Figure 1, it can be seen that the larger the compression, the higher the correlation is.
Revised: In Figure 1, it can be seen that the larger the compression, the higher the correlation is.

Original: This is a positive result for our use case.
Revised: This is a positive result for our use case.

Original: In the end, we are interested in making decomposition choices when compressing.
Revised: In the end, we are interested in making decomposition choices when compressing.

Original: The more we compress, the higher the correlation and therefore the more certain we are that basing our choice on the approximation error results in the optimal choice.
Revised: The more we compress, the higher the correlation and therefore the more certain we are that basing our choice on the approximation error results in the optimal choice.

Original: It can also be noted that a certain level of compression is needed to be able to make choices based on the approximation error.
Revised: It can also be noted that a certain level of compression is needed to be able to make choices based on the approximation error.

Original: For both models and datasets, the correlation is small when the compression is only 10% and 25%.
Revised: For both models and datasets, the correlation is small when the compression is only 10% and 25%.

Original: The variance in the correlation at smaller compression levels is larger than at higher compression levels.
Revised: The variance in the correlation at smaller compression levels is larger than at higher compression levels.

Original: When the compression is too small the effect on the performance of the model is too small compared to the observed variance, especially after fine-tuning.
Revised: When the compression is too small the effect on the performance of the model is too small compared to the observed variance, especially after fine-tuning.

Original: In the remainder of the experiments, we therefore focus on compression levels of C =
Revised: In the remainder of the experiments, we therefore focus on compression levels of C =

Original: Comparison of approximation error measures Works such as Liebenwein et al.
Revised: Comparison of approximation error measures Works such as Liebenwein et al.

Original: (2021) have used a single approximation error, e.g. Relative Weights, to identify which layer to compress next, implicitly assuming that relative errors between layers are indicative of the relative model performance differences.
Revised: (2021) have used a single approximation error, e.g. Relative Weights, to identify which layer to compress next, implicitly assuming that relative errors between layers are indicative of the relative model performance differences.

Original: We here compare the various approximation error measures, testing the correlation with performance over all decomposition choices.
Revised: We here compare the various approximation error measures, testing the correlation with performance over all decomposition choices.

Original: In Figure 2, the correlation is calculated based on measurements of all combinations of layer, decomposition method, and compression level once.
Revised: In Figure 2, the correlation is calculated based on measurements of all combinations of layer, decomposition method, and compression level once.

Original: The correlations are averaged over runs, as well as the ±1 standard deviation is calculated over the runs.
Revised: The correlations are averaged over runs, as well as the ±1 standard deviation is calculated over the runs.

Original: Figure 2 shows that the correlations are generally positive and significantly different from zero.
Revised: Figure 2 shows that the correlations are generally positive and significantly different from zero.

Original: This means that the decomposition choices can (to some extent) be based on the approximation error.
Revised: This means that the decomposition choices can (to some extent) be based on the approximation error.

Original: There is one exception where the correlation is close to zero, namely for the Absolute Weights measure on ResNet-18.
Revised: There is one exception where the correlation is close to zero, namely for the Absolute Weights measure on ResNet-18.

Original: The difference in correlations can be explained by the difference in approximation error between layers, a detailed explanation is provided in Appendix A.4.
Revised: The difference in correlations can be explained by the difference in approximation error between layers, a detailed explanation is provided in Appendix A.4.

Original: These results suggest that using Absolute-based approximation errors, while they may show high correlations in some cases, are not generally a reliable indicator for future model performance, and that normalized measures should be used instead. 
Revised: These results suggest that using Absolute-based approximation errors, while they may show high correlations in some cases, are not generally a reliable indicator for future model performance, and that normalized measures should be used instead. [Figure] l

Original: [Figure]
Revised: [Figure]

Revised: t

Revised: i

Original:  Comparing the different approximation error metrics, we observe the highest correlation with the performance for Relative Weights in all tested cases.
Revised: [Figure] n ( [Figure] Comparing the different approximation error metrics, we observe the highest correlation with the performance for Relative Weights in all tested cases.

Original: Interestingly, the magnitude of the correlation for Feature-based measures is similar to or smaller than the correlation for Weight-based measures.
Revised: Interestingly, the magnitude of the correlation for Feature-based measures is similar to or smaller than the correlation for Weight-based measures.

Original: Although the findings of Jaderberg et al.
Revised: Although the findings of Jaderberg et al.

Original: (2014) would suggest a stronger correlation for the features, at least before fine-tuning, we do not observe benefits for basing decomposition choices on the approximation error of the features rather than the weights.
Revised: (2014) would suggest a stronger correlation for the features, at least before fine-tuning, we do not observe benefits for basing decomposition choices on the approximation error of the features rather than the weights.

Original: Possibly pretraining already ensures all the elements in the weight tensor are equally important for the target data distribution, thus a
Revised: Possibly pretraining already ensures all the elements in the weight tensor are equally important for the target data distribution, thus a

Original: Weight-based error already reliably reflects resulting errors on the output features.
Revised: Weight-based error already reliably reflects resulting errors on the output features.

Original: Comparing the error bars with and without fine-tuning, the randomness from fine-tuning has a larger impact on the variance in correlation than the randomness from CP initialization.
Revised: Comparing the error bars with and without fine-tuning, the randomness from fine-tuning has a larger impact on the variance in correlation than the randomness from CP initialization.

Original: In summary, our results support the use of the Relative Weights approximation error to make decomposition choices.
Revised: In summary, our results support the use of the Relative Weights approximation error to make decomposition choices.

Original: Impact of fine-tuning Most works use fine-tuning to recover some of the lost performance (Denton et al., 2014; Lebedev et al., 2015; Kim et al., 2016).
Revised: Impact of fine-tuning Most works use fine-tuning to recover some of the lost performance (Denton et al., 2014; Lebedev et al., 2015; Kim et al., 2016).

Original: The right subfigure of Figure 2 shows the mean and standard deviation offive correlations per model anddataset after fine-tuning.
Revised: The right subfigure of Figure 2 shows the mean and standard deviation of and

Original: After fine-tuning, the correlation between the approximation error and the performance error is smaller than before fine-tuning for GaripovNet, as additional training adapts the model and reduces the performance gap between the different choices, but this effect is not observed for ResNet-18 where the correlation was already lower.
Revised: After fine-tuning, the correlation between the approximation error and the performance error is smaller than before fine-tuning for GaripovNet, as additional training adapts the model and reduces the performance gap between the different choices, but this effect is not observed for ResNet-18 where the correlation was already lower.

Original: However, for both models, there is still a clear positive correlation between the approximation error and the performance after fine-tuning.
Revised: However, for both models, there is still a clear positive correlation between the approximation error and the performance after fine-tuning.

Original: This means that decomposition choices can still be based on the approximation error when intending to perform fine-tuning later, even though different hyperparameters might be optimal without and with fine-tuning.
Revised: This means that decomposition choices can still be based on the approximation error when intending to perform fine-tuning later, even though different hyperparameters might be optimal without and with fine-tuning.

Original: While the correlation is positive and significantly different from zero, the correlation is only around +0.5 for ResNet-18.
Revised: While the correlation is positive and significantly different from zero, the correlation is only around +0.5 for ResNet-18.

Original: We therefore investigate if the correlation is higher when only considering specific decomposition choices next.
Revised: We therefore investigate if the correlation is higher when only considering specific decomposition choices next.

Original: Correlation across Layers vs. Methods In the previous experiments, we compared how decomposition choices on both different layers and methods correlated with performance.
Revised: Correlation across Layers vs. Methods In the previous experiments, we compared how decomposition choices on both different layers and methods correlated with performance.

Original: Here we investigate if the correlation is stronger if only one of these choices would be considered.
Revised: Here we investigate if the correlation is stronger if only one of these choices would be considered.

Original: For instance, previous works often only include layers as decomposition choice, and have not compared across decomposition methods.
Revised: For instance, previous works often only include layers as decomposition choice, and have not compared across decomposition methods.

Original: We compare correlation on all choices for both sets ( L × M × C ) as before, to layers only ( L × { m } × { c } with reported results averaged for all m ∈ M and c ∈ C ), and to methods only ( { l } × M × { c } with reported results averaged for all l ∈ L and c ∈ C ).
Revised: We compare correlation on all choices for both sets ( L × M × C ) as before, to layers only ( L × { m } × { c } with reported results averaged for all m ∈ M and c ∈ C ), and to methods only ( { l } × M × { c } with reported results averaged for all l ∈ L and c ∈ C ).

Original: Figure 3 shows that before fine-tuning the approximation error has a lower correlation with the performance of the model when considering layers only compared to all decomposition choices.
Revised: Figure 3 shows that before fine-tuning the approximation error has a lower correlation with the performance of the model when considering layers only compared to all decomposition choices.

Original: Not all layers of a neural network have the same efficiency-performance trade-off (Lebedev et al., 2015;
Revised: Not all layers of a neural network have the same efficiency-performance trade-off (Lebedev et al., 2015;

Original: Hawkins et al., 2021).
Revised: Hawkins et al., 2021).

Original: Therefore, the correlation is lower when we fix the decomposition method and compression level.
Revised: Therefore, the correlation is lower when we fix the decomposition method and compression level.

Original: It is better to combine layers with compression levels (and decomposition
Revised: It is better to combine layers with compression levels (and decomposition

Original: [Figure]
Revised: [Figure]

Original: methods).
Revised: methods).

Original: However, fine-tuning recovers some of the correlation for layers.
Revised: However, fine-tuning recovers some of the correlation for layers.

Original: Across decomposition methods, the correlation before fine-tuning is comparable to the correlation calculated across all decomposition choices.
Revised: Across decomposition methods, the correlation before fine-tuning is comparable to the correlation calculated across all decomposition choices.

Original: These results suggest decomposition methods can be compared better than just the layers before fine-tuning, although the former is not an optimization choice considered in previous works.
Revised: These results suggest decomposition methods can be compared better than just the layers before fine-tuning, although the former is not an optimization choice considered in previous works.

Original: Interestingly, for GaripovNet the correlation across decomposition methods drops significantly after fine-tuning.
Revised: Interestingly, for GaripovNet the correlation across decomposition methods drops significantly after fine-tuning.

Original: We find that this is due to difficulties in optimizing the CP decomposed layers, since the gradient flow through CP convolutions is a known problem (Silva & Lim, 2008; Lebedev et al., 2015), whereas ResNet does not suffer from this due to its skip connections.
Revised: We find that this is due to difficulties in optimizing the CP decomposed layers, since the gradient flow through CP convolutions is a known problem (Silva & Lim, 2008; Lebedev et al., 2015), whereas ResNet does not suffer from this due to its skip connections.

Original: We conclude that (unlike current practice) network compression could consider multiple decomposition methods as their approximation errors can be compared, though most reliably when aiming for compression without fine-tuning.
Revised: We conclude that (unlike current practice) network compression could consider multiple decomposition methods as their approximation errors can be compared, though most reliably when aiming for compression without fine-tuning.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have tested Assumption 1, and find that there is a positive correlation between the relative approximation error on the weights and the resulting performance error of the model for a wide range of TD choices, including layers, methods, and compression levels.
Revised: We have tested Assumption 1, and find that there is a positive correlation between the relative approximation error on the weights and the resulting performance error of the model for a wide range of TD choices, including layers, methods, and compression levels.

Original: We further find that using data to compute the approximation error on the features, rather than simply on the model weights directly, does not improve the correlation.
Revised: We further find that using data to compute the approximation error on the features, rather than simply on the model weights directly, does not improve the correlation.

Original: Scaling the approximation error with the norm of the original tensor provides the highest and most stable correlation across all compared models and datasets.
Revised: Scaling the approximation error with the norm of the original tensor provides the highest and most stable correlation across all compared models and datasets.

Original: Our findings suggest that the Relative Weights approximation error is best suited to select among TD decomposition choices.
Revised: Our findings suggest that the Relative Weights approximation error is best suited to select among TD decomposition choices.

Original: While these choices can be made across layers, TD methods, and compression levels, we observe that the correlation before fine-tuning is smaller when comparing between layers for a fixed method, than when comparing across methods (here: CP, Tucker, and Tensor Train) for a fixed layer.
Revised: While these choices can be made across layers, TD methods, and compression levels, we observe that the correlation before fine-tuning is smaller when comparing between layers for a fixed method, than when comparing across methods (here: CP, Tucker, and Tensor Train) for a fixed layer.

Original: Integrating multiple types of decompositions within a network compression technique is therefore a potential direction for future work, although care has to be taken when the use case includes later fine-tuning, as the correlation for selecting across decomposition methods can degrade since backpropagation through certain factorized structures remains challenging.
Revised: Integrating multiple types of decompositions within a network compression technique is therefore a potential direction for future work, although care has to be taken when the use case includes later fine-tuning, as the correlation for selecting across decomposition methods can degrade since backpropagation through certain factorized structures remains challenging.

Original: Our experiments are limited to a set of decomposition choices and network layers commonly found in the TD literature.
Revised: Our experiments are limited to a set of decomposition choices and network layers commonly found in the TD literature.

Original: Future work can extend to other decompositions and other types of neural network layers, e.g. fully connected layers.
Revised: Future work can extend to other decompositions and other types of neural network layers, e.g. fully connected layers.

Original: While the weights are matrices, tensor decomposition has been applied to fully connected layers by reshaping the weight matrix into a higher-order tensor.
Revised: While the weights are matrices, tensor decomposition has been applied to fully connected layers by reshaping the weight matrix into a higher-order tensor.

Original: The choice of reshaping then becomes an additional decomposition choice.
Revised: The choice of reshaping then becomes an additional decomposition choice.

Original: R EPRODUCIBILITY
Revised: R EPRODUCIBILITY

Original: The authors find it important that this work is reproducible.
Revised: The authors find it important that this work is reproducible.

Original: To this extent the following efforts have been made: The datasets and test-validation splits are described in Section 4.1.
Revised: To this extent the following efforts have been made: The datasets and test-validation splits are described in Section 4.1.

Original: The datasets are collected from PyTorch Vision .
Revised: The datasets are collected from PyTorch Vision .

Original: The models and hyperparameters used for training are covered in Section 4.1.
Revised: The models and hyperparameters used for training are covered in Section 4.1.

Original: The implementations of the baseline models are from the PyTorch Model Zoo .
Revised: The implementations of the baseline models are from the PyTorch Model Zoo .

Original: The models are factorized with Tensorly-Torch  , using CP initialization described in Section 4.1 and ranks provided in Appendix A.1.
Revised: The models are factorized with Tensorly-Torch (Kossaifi et al., 2019b) , using CP initialization described in Section 4.1 and ranks provided in Appendix A.1.

Original: The experimental setup is explained in Section 4.1.
Revised: The experimental setup is explained in Section 4.1.

Original: The calculation of metrics is formulated in Section 3.
Revised: The calculation of metrics is formulated in Section 3.

Original: Finally, the code that has been used for these experiments will be made available online upon paper acceptance.
Revised: Finally, the code to reproduce these experiments is available at: https://github.com/JSchuurmans/tddl .

Original: The camera-ready version will contain the respective URL.

Original: It is left out of this version due to the blind review process.

Original: A A PPENDIX A. R ANK AND C OMPRESSION LEVEL
Revised: A A PPENDIX A. R ANK AND C OMPRESSION LEVEL

Original: Tables 2 and 3 present the ranks that are used for GaripovNet (Garipov et al., 2016) and ResNet- (
Revised: Tables 2 and 3 present the ranks that are used for GaripovNet (Garipov et al., 2016) and ResNet- (

Original: He et al., 2016) respectively.
Revised: He et al., 2016) respectively.

Original: Note that in these tables, the Tucker rank includes the kernel ranks corresponding to the width and height, and the TT rank includes R 

Revised: Note that in these tables, the Tucker rank includes the kernel ranks corresponding to the width and height, and the TT rank includes R

Original: = R 4 = 1 , which are left out of Equation 4 for conciseness.
Revised: = R 4 = 1 , which are left out of Equation 4 for conciseness.

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: A. N -M ODE P RODUCT
Revised: A. N -M ODE P RODUCT

Original: The definition of n-Mode Product × n given by Kolda & Bader (2009) is used in this paper.
Revised: The definition of n-Mode Product × n given by Kolda & Bader (2009) is used in this paper.

Original: The contraction of a tensor G ∈ R R 1 ,R 2 , ··· ,R N with matrix Y ∈ R Y,R n along the n th mode of the tensor is defined elementwise as:
Revised: The contraction of a tensor G ∈ R R 1 ,R 2 , ··· ,R N with matrix Y ∈ R Y,R n along the n th mode of the tensor is defined elementwise as:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: A. S ELECTION OF R ES N ET -18 LAYERS
Revised: A. S ELECTION OF R ES N ET -18 LAYERS

Original: The following layers are considered for decomposition in ResNet-18: The last layer of the first two blocks.
Revised: The following layers are considered for decomposition in ResNet-18: The last layer of the first two blocks.

Original: The first layer with stride two.
Revised: The first layer with stride two.

Original: The first layer after a 1x1 convolution.
Revised: The first layer after a 1x1 convolution.

Original: A 1x1 convolution with a similar number of parameters as the first choice of layer.
Revised: A 1x1 convolution with a similar number of parameters as the first choice of layer.

Original: The layer before and after the 1x convolution.
Revised: The layer before and after the 1x convolution.

Original: The final two convolutional layers, as they are the largest convolutional layers.
Revised: The final two convolutional layers, as they are the largest convolutional layers.

Original: [Table]
Revised: [Table]

Original: A. D IFFERENCE BETWEEN LAYERS IN A BSOLUTE W EIGHTS FOR R ES N ET - 
Revised: A. D IFFERENCE BETWEEN LAYERS IN A BSOLUTE W EIGHTS FOR R ES N ET -

Original: Let us recall from Figure 2 that when basing the approximation error on Absolute Weights, the correlation with the performance error is close to zero for ResNet-18.
Revised: Let us recall from Figure 2 that when basing the approximation error on Absolute Weights, the correlation with the performance error is close to zero for ResNet-18.

Original: The correlation is zero due to the difference between layers.
Revised: The correlation is zero due to the difference between layers.

Original: Figure 4 plots the approximation errors of Relative and Absolute Weights versus the performance error before and after fine-tuning.
Revised: Figure 4 plots the approximation errors of Relative and Absolute Weights versus the performance error before and after fine-tuning.

Original: The points resulting from CP, Tucker, and Tensor Train with compression of 50%, 75%, and 90% are grouped per layer. 
Revised: The points resulting from CP, Tucker, and Tensor Train with compression of 50%, 75%, and 90% are grouped per layer. [Figure] i f i [Figure]

Original: Layers 15, 19, and 28 have a smaller absolute weight error and large performance error relative to layers 38, 60, and 63.
Revised: i [Figure] n [Figure]

Original:  Compared to the Relative Weights where layers 60 and 63 have a small relative weight error and small performance error and layer 28 has comparable errors to layers 15, 19, and 28.
Revised: rr [Figure] Layers 15, 19, and 28 have a smaller absolute weight error and large performance error relative to layers 38, 60, and 63. Compared to the Relative Weights where layers 60 and 63 have a small relative weight error and small performance error and layer 28 has comparable errors to layers 15, 19, and 28.

Original: Layers 15, 19, and 28 have a smaller absolute weight error and large performance error relative to layers 38, 60, and 63.
Revised: Layers 15, 19, and 28 have a smaller absolute weight error and large performance error relative to layers 38, 60, and 63.

Original: Compare this to the data for Relative Weights where layers 60 and 63 have a small relative weight error and small performance error and layer 28 has comparable errors to 15, 19, and 28.
Revised: Compare this to the data for Relative Weights where layers 60 and 63 have a small relative weight error and small performance error and layer 28 has comparable errors to 15, 19, and 28.

Original: This leads to the correlation between Absolute Weights and performance errors being close to zero, while it is positive for Relative Weights.
Revised: This leads to the correlation between Absolute Weights and performance errors being close to zero, while it is positive for Relative Weights.

Original: Therefore, the difference in correlations can be explained by the difference in approximation error between layers.
Revised: Therefore, the difference in correlations can be explained by the difference in approximation error between layers.