Original: F IX YOUR CLASSIFIER : THE MARGINAL VALUE OF TRAINING THE LAST WEIGHT LAYER
Revised: F IX YOUR CLASSIFIER : THE MARGINAL VALUE OF TRAINING THE LAST WEIGHT LAYER

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Neural networks are commonly used as models for classiﬁcation for a wide variety of tasks.
Revised: Neural networks are commonly used as models for classiﬁcation for a wide variety of tasks.

Original: Typically, a learned afﬁne transformation is placed at the end of such models, yielding a per-class value used for classiﬁcation.
Revised: Typically, a learned afﬁne transformation is placed at the end of such models, yielding a per-class value used for classiﬁcation.

Original: This classiﬁer can have a vast number of parameters, which grows linearly with the number of possible classes, thus requiring increasingly more resources.
Revised: This classiﬁer can have a vast number of parameters, which grows linearly with the number of possible classes, thus requiring increasingly more resources.

Original: In this work we argue that this classiﬁer can be ﬁxed, up to a global scale constant, with little or no loss of accuracy for most tasks, allowing memory and computational beneﬁts.
Revised: In this work we argue that this classiﬁer can be ﬁxed, up to a global scale constant, with little or no loss of accuracy for most tasks, allowing memory and computational beneﬁts.

Original: Moreover, we show that by initializing the classiﬁer with a
Revised: Moreover, we show that by initializing the classiﬁer with a

Original: Hadamard matrix we can speed up inference as well.
Revised: Hadamard matrix we can speed up inference as well.

Original: We discuss the implications for current understanding of neural network models.
Revised: We discuss the implications for current understanding of neural network models.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural network have become a widely used model for machine learning, achieving state-ofthe-art results on many tasks.
Revised: Deep neural network have become a widely used model for machine learning, achieving state-ofthe-art results on many tasks.

Original: The most common task these models are used for is to perform classiﬁcation, as in the case of convolutional neural networks (CNNs) used to classify images to a semantic category.
Revised: The most common task these models are used for is to perform classiﬁcation, as in the case of convolutional neural networks (CNNs) used to classify images to a semantic category.

Original: CNN models are currently considered the standard for visual tasks, allowing far better accuracy than preceding approaches (Krizhevsky et al., 2012; He et al., 2016; Szegedy et al., 2015).
Revised: CNN models are currently considered the standard for visual tasks, allowing far better accuracy than preceding approaches (Krizhevsky et al., 2012; He et al., 2016; Szegedy et al., 2015).

Original: Neural networks (NN) are known to be a universal approximation model (Cybenko, 1989; Hornik, 1991), even with a small number of non-linear transformations (layers).

Original: They were recently shown to even allow arbitrarily chosen data points to be memorized (Zhang et al., 2017a).

Original: Training NN models and using them for inference requires large amounts of memory and computational resources, thus, extensive amount of research has been done lately to reduce the size of networks.
Revised: Training NN models and using them for inference requires large amounts of memory and computational resources, thus, extensive amount of research has been done lately to reduce the size of networks.

Original: Han et al.
Revised: Han et al.

Original: (2015) used weight sharing and speciﬁcation, Micikevicius et al.
Revised: (2015) used weight sharing and speciﬁcation, Micikevicius et al.

Original: (2017) used mixed precision to reduce the size of the neural networks by half.
Revised: (2017) used mixed precision to reduce the size of the neural networks by half.

Original: Tai et al. (2015) and Jaderberg et al.
Revised: Tai et al. (2015) and Jaderberg et al.

Original: (2014) used low rank approximations to speed up NNs.
Revised: (2014) used low rank approximations to speed up NNs.

Original: Hubara et al.
Revised: Hubara et al.

Original: (2016b), Li et al.
Revised: (2016b), Li et al.

Original: (2016) and Zhou et al.
Revised: (2016) and Zhou et al.

Original: (2016), used a more aggressive approach, in which weights, activations and gradients were quantized to further reduce computation during training.
Revised: (2016), used a more aggressive approach, in which weights, activations and gradients were quantized to further reduce computation during training.

Original: Although aggressive quantization beneﬁts from smaller model size, the extreme compression rate comes with a loss of accuracy.
Revised: Although aggressive quantization beneﬁts from smaller model size, the extreme compression rate comes with a loss of accuracy.

Original: Past work noted the fact that predeﬁned (Park & Sandberg, 1991) and random (Huang et al., 2006) projections can be used together with a learned afﬁne transformation to achieve competitive results on several tasks.
Revised: Past work noted the fact that predeﬁned (Park & Sandberg, 1991) and random (Huang et al., 2006) projections can be used together with a learned afﬁne transformation to achieve competitive results on several tasks.

Original: In this study suggest the reversed proposal - that common NN models used can learn useful representation even without modifying the ﬁnal output layer, which often holds a large number of parameters that grows linearly with number of classes.
Revised: In this study suggest the reversed proposal - that common NN models used can learn useful representation even without modifying the ﬁnal output layer, which often holds a large number of parameters that grows linearly with number of classes.

Original: 1. C LASSIFIERS IN CONVOLUTIONAL NEURAL NETWORKS
Revised: 1. C LASSIFIERS IN CONVOLUTIONAL NEURAL NETWORKS

Original: Convolutional neural networks (CNNs) are commonly used to solve a variety of spatial and temporal tasks.
Revised: Convolutional neural networks (CNNs) are commonly used to solve a variety of spatial and temporal tasks.

Original: CNNs are usually composed of a stack of convolutional parameterized layers, spatial pooling layers and fully connected layers, separated by non-linear activation functions.
Revised: CNNs are usually composed of a stack of convolutional parameterized layers, spatial pooling layers and fully connected layers, separated by non-linear activation functions.

Original: Earlier architectures of CNNs (LeCun et al., 1998; Krizhevsky et al., 2012) used a set of fully-connected layers at later stage of the network, presumably to allow classiﬁcation based on global features of an image.
Revised: Earlier architectures of CNNs (LeCun et al., 1998; Krizhevsky et al., 2012) used a set of fully-connected layers at later stage of the network, presumably to allow classiﬁcation based on global features of an image.

Revised: The ﬁnal classiﬁer can also be replaced with a convolutional layer with output feature maps matching the number of classes, as demonstrated by Springenberg et al.

Original: Despite the enormous number of trainable parameters these layers added to the model, they are known to have a rather marginal impact on the ﬁnal performance of the network (Zeiler & Fergus, 2014) and are easily compressed and reduced after a model was trained by simple means such as matrix decomposition and sparsiﬁcation (Han et al., 2015).
Revised: Despite the enormous number of trainable parameters these layers added to the model, they are known to have a rather marginal impact on the ﬁnal performance of the network (Zeiler & Fergus, 2014) and are easily compressed and reduced after a model was trained by simple means such as matrix decomposition and sparsiﬁcation (Han et al., 2015).

Original: Further more, modern architecture choices are characterized with the removal of most of the fully connected layers (Lin et al., 2013;
Revised: Further more, modern architecture choices are characterized with the removal of most of the fully connected layers (Lin et al., 2013;

Original: Szegedy et al., 2015; He et al., 2016), which was found to lead to better generalization and overall accuracy, together with a huge decrease in the number of trainable parameters.
Revised: Szegedy et al., 2015; He et al., 2016), which was found to lead to better generalization and overall accuracy, together with a huge decrease in the number of trainable parameters.

Original: Additionally, numerous works showed that CNNs can be trained in a metric learning regime (Bromley et al., 1994; Schroff et al., 2015; Hoffer & Ailon, 2015), where no explicit classiﬁcation layer was introduced and the objective regarded only distance measures between intermediate representations.
Revised: Additionally, numerous works showed that CNNs can be trained in a metric learning regime (Bromley et al., 1994; Schroff et al., 2015; Hoffer & Ailon, 2015), where no explicit classiﬁcation layer was introduced and the objective regarded only distance measures between intermediate representations.

Original: All of these properties provide evidence that fully-connected layers are in fact redundant and play a small role in learning and generalization.
Revised: All of these properties provide evidence that fully-connected layers are in fact redundant and play a small role in learning and generalization.

Original: Despite the apparent minor role they play, fully-connected layers are still commonly used as classiﬁcation layers, transforming from the dimension of network features N to the number of required class categories C .
Revised: Despite the apparent minor role they play, fully-connected layers are still commonly used as classiﬁcation layers, transforming from the dimension of network features N to the number of required class categories C .

Original: Therefore, each classiﬁcation model must hold N · C number of trainable parameters that grows in a linear manner with the number of classes.
Revised: Therefore, each classiﬁcation model must hold N · C number of trainable parameters that grows in a linear manner with the number of classes.

Revised: This property still holds when the fully-connected layer is replaced with a convolutional classiﬁer as shown by Springenberg et al.

Original: In this work we claim that for common use-cases of convolutional network, the parameters used for the ﬁnal classiﬁcation transform are completely redundant, and can be replaced with a predetermined linear transform.
Revised: In this work we claim that for common use-cases of convolutional network, the parameters used for the ﬁnal classiﬁcation transform are completely redundant, and can be replaced with a predetermined linear transform.

Original: The use of a ﬁxed transform can, in many cases, allow a huge decrease in model parameters, and a possible computational beneﬁt.
Revised: The use of a ﬁxed transform can, in many cases, allow a huge decrease in model parameters, and a possible computational beneﬁt.

Original: We suggest that these can help the deployment of those models in devices with low computation ability and smaller memory capacity.
Revised: We suggest that these can help the deployment of those models in devices with low computation ability and smaller memory capacity.

Original: Moreover, as we keep the classiﬁer ﬁxed, less parameters need to be updated, reducing the communication cost for models deployed in distributed systems.
Revised: Moreover, as we keep the classiﬁer ﬁxed, less parameters need to be updated, reducing the communication cost for models deployed in distributed systems.

Original: The use of a ﬁxed transform which does not depend on the number classes can allow models to scale to a large number of possible outputs, without a linear cost in the number of parameters.
Revised: The use of a ﬁxed transform which does not depend on the number classes can allow models to scale to a large number of possible outputs, without a linear cost in the number of parameters.

Original: We also suggest that these ﬁnding might shed light on the importance of the preceding non-linear layers to learning and generalization.
Revised: We also suggest that these ﬁnding might shed light on the importance of the preceding non-linear layers to learning and generalization.

Original:  U SING A FIXED CLASSIFIER 2. F ULLY - CONNECTED CLASSIFIERS
Revised:  U SING A FIXED CLASSIFIER 2. F ULLY - CONNECTED CLASSIFIERS

Original: We focus our attention on the ﬁnal representation obtained by the network (the last hidden layer), before the classiﬁer.
Revised: We focus our attention on the ﬁnal representation obtained by the network (the last hidden layer), before the classiﬁer.

Original: We denote these representation as x = F ( z ; θ ) where F is assumed to be a deep neural network with input z and parameters θ , e.g., a convolutional network, trained by backpropagation.
Revised: We denote these representation as x = F ( z ; θ ) where F is assumed to be a deep neural network with input z and parameters θ , e.g., a convolutional network, trained by backpropagation.

Original: In common NN models, this representation is followed by an additional afﬁne transformation
Revised: In common NN models, this representation is followed by an additional afﬁne transformation

Original: [Equation]
Revised: [Equation]

Original: where W and b are also trained by back-propagation.
Revised: where W and b are also trained by back-propagation.

Original: For input x of N length, and C different possible outputs, W is required to be a matrix of N ×
Revised: For input x of N length, and C different possible outputs, W is required to be a matrix of N ×

Original: C .
Revised: C .

Original: Training is done using cross-entropy loss, by feeding the network outputs through a softmax activation
Revised: Training is done using cross-entropy loss, by feeding the network outputs through a softmax activation

Original: [Equation]
Revised: [Equation]

Original: and reducing the expected negative log likelihood with respect to ground-truth target t ∈ { 1 , . .
Revised: and reducing the expected negative log likelihood with respect to ground-truth target t ∈ { 1 , . .

Original: , C } , by minimizing
Revised: , C } , by minimizing

Original: [Equation]
Revised: [Equation]

Original:   where w i is the i -th column of W .
Revised:  where w i is the i -th column of W .

Original: 2. C HOOSING THE PROJECTION MATRIX
Revised: 2. C HOOSING THE PROJECTION MATRIX

Original: To evaluate our conjecture regarding the importance of the ﬁnal classiﬁcation transformation, we replaced the trainable parameter matrix W with a ﬁxed orthonormal projection Q ∈ R N × C , such that ∀ i (cid:54)
Revised: To evaluate our conjecture regarding the importance of the ﬁnal classiﬁcation transformation, we replaced the trainable parameter matrix W with a ﬁxed orthonormal projection Q ∈ R N × C , such that ∀ i (cid:54)

Original: =
Revised: =

Original: j : q i · q j = 0 and (cid:107) q i (cid:107) 2 = 1 , where q i is the i th column of Q .
Revised: j : q i · q j = 0 and (cid:107) q i (cid:107) 2 = 1 , where q i is the i th column of Q .

Original: This can be ensured by a simple random sampling and singular-value decomposition
Revised: This can be ensured by a simple random sampling and singular-value decomposition

Original: As the rows of classiﬁer weight matrix are ﬁxed with an equally valued L 2 norm, we ﬁnd it beneﬁcial to also restrict the representation of x by normalizing it to reside on the n -dimensional sphere
Revised: As the rows of classiﬁer weight matrix are ﬁxed with an equally valued L 2 norm, we ﬁnd it beneﬁcial to also restrict the representation of x by normalizing it to reside on the n -dimensional sphere

Original: [Equation]
Revised: [Equation]

Original: This allows faster training and convergence, as the network does not need to account for changes in the scale of its weights.
Revised: This allows faster training and convergence, as the network does not need to account for changes in the scale of its weights.

Original: We now face the problem that q i · ˆ x is bounded between − 1 and 1 .
Revised: We now face the problem that q i · ˆ x is bounded between − 1 and 1 .

Original: This causes convergence issues, as the softmax function is scale sensitive, and the network is affected by the inability to re-scale its input.
Revised: This causes convergence issues, as the softmax function is scale sensitive, and the network is affected by the inability to re-scale its input.

Original: This is similar to the phenomenon described by Vaswani et al.
Revised: This is similar to the phenomenon described by Vaswani et al.

Original: (2017) with respect to softmax function used for attention mechanisms.
Revised: (2017) with respect to softmax function used for attention mechanisms.

Original: In the same spirit, we can amend this issue with a ﬁxed scale T applied to softmax inputs f ( y ) =
Revised: In the same spirit, we can amend this issue with a ﬁxed scale T applied to softmax inputs f ( y ) =

Original: softmax( 1 T y ) , also known as a softmax temperature.
Revised: softmax( 1 T y ) , also known as a softmax temperature.

Original: However, this introduces an additional hyper-parameter which may differ between networks and datasets.
Revised: However, this introduces an additional hyper-parameter which may differ between networks and datasets.

Original: Instead, we suggest to introduce a single scalar parameter α to learn the softmax scale, effectively functioning as an inverse of the softmax temperature 1 T .
Revised: Instead, we suggest to introduce a single scalar parameter α to learn the softmax scale, effectively functioning as an inverse of the softmax temperature 1 T .

Original: Using normalized weights and an additional scale coefﬁcient is similar in spirit to weightnormalization (Salimans & Kingma, 2016), with the difference that we use a single scale for all entries in the weight matrix, in contrast to a scale for each row that Salimans & Kingma (2016) uses.
Revised: Using normalized weights and an additional scale coefﬁcient is similar in spirit to weightnormalization (Salimans & Kingma, 2016), with the difference that we use a single scale for all entries in the weight matrix, in contrast to a scale for each row that Salimans & Kingma (2016) uses.

Original: We keep the additional vector of bias parameters b ∈ R C , and train using the same negative-loglikelihood criterion.
Revised: We keep the additional vector of bias parameters b ∈ R C , and train using the same negative-loglikelihood criterion.

Original: More explicitly, our classiﬁer output is now
Revised: More explicitly, our classiﬁer output is now

Original: [Equation]
Revised: [Equation]

Original: and we minimize the loss:
Revised: and we minimize the loss:

Original: [Equation]
Revised: [Equation]

Original: where we recall x is the ﬁnal representation obtained by the network for a speciﬁc sample, and t ∈ { 1 , . . .
Revised: where we recall x is the ﬁnal representation obtained by the network for a speciﬁc sample, and t ∈ { 1 , . . .

Original: , C } is the ground-truth label for that sample.
Revised: , C } is the ground-truth label for that sample.

Original: Observing the behavior of the α parameter over time revealed a logarithmic growth depicted in graph 1.
Revised: Observing the behavior of the α parameter over time revealed a logarithmic growth depicted in graph 1.

Original: Interestingly, this is the same behavior exhibited by the norm of a learned classiﬁer, ﬁrst described by Hoffer et al.
Revised: Interestingly, this is the same behavior exhibited by the norm of a learned classiﬁer, ﬁrst described by Hoffer et al.

Original: and linked to the generalization of the network.
Revised: and linked to the generalization of the network.

Original: This was recently explained by the under-review work of Anonymous (2018) as convergence to a max margin classiﬁer.
Revised: This was recently explained by the under-review work of Anonymous (2018) as convergence to a max margin classiﬁer.

Original: We suggest that using a single parameter will enable a simpler examination and possible further exploration of this phenomenon and its implications.
Revised: We suggest that using a single parameter will enable a simpler examination and possible further exploration of this phenomenon and its implications.

Original: We note that as − 1 ≤ q i · ˆ x ≤ 1 , we also found it possible to train the network with a simple cosine angle loss:
Revised: We note that as − 1 ≤ q i · ˆ x ≤ 1 , we also found it possible to train the network with a simple cosine angle loss:

Original: [Equation]
Revised: [Equation]

Original: allowing to discard the softmax function and its scale altogether, but resulting in a slight decrease in ﬁnal validation accuracy compared to original models.
Revised: allowing to discard the softmax function and its scale altogether, but resulting in a slight decrease in ﬁnal validation accuracy compared to original models.

Original: 2. U SING A FIXED H ADMARD MATRIX
Revised: 2. U SING A FIXED H ADMARD MATRIX

Original: We further suggest the use of a Hadamard matrix (Hedayat et al., 1978) as the ﬁnal classiﬁcation transform.
Revised: We further suggest the use of a Hadamard matrix (Hedayat et al., 1978) as the ﬁnal classiﬁcation transform.

Original: Hadamard matrix H is a n × n matrix, where all of its entries are either +1 or − 1 .
Revised: Hadamard matrix H is an n × n matrix, where all of its entries are either +1 or − 1 .

Original: Further more, H is orthogonal, such that HH T = nI n where I n is the identity matrix.
Revised: Further more, H is orthogonal, such that HH T = nI n where I n is the identity matrix.

Original: We can use a truncated Hadamard matrix ˆ H ∈ {− 1 , 1 } C × N where all C rows are orthogonal as our ﬁnal classiﬁcation layer such that
Revised: We can use a truncated Hadamard matrix ˆ H ∈ {− 1 , 1 } C × N where all C rows are orthogonal as our ﬁnal classiﬁcation layer such that

Original: [Equation]
Revised: [Equation]

Original: This usage allows two main beneﬁts:
Revised: This usage allows two main beneﬁts:

Original: • A deterministic, low-memory and easily generated matrix that can be used to classify.
Revised: • A deterministic, low-memory and easily generated matrix that can be used to classify.

Original: • Removal of the need to perform a full matrix-matrix multiplication - as multiplying by a Hadamard matrix can be done by simple sign manipulation and addition.
Revised: • Removal of the need to perform a full matrix-matrix multiplication - as multiplying by a Hadamard matrix can be done by simple sign manipulation and addition.

Original: We note that n must be a multiple of 4 , but it can be easily truncated to ﬁt normally deﬁned networks.
Revised: We note that n must be a multiple of 4 , but it can be easily truncated to ﬁt normally deﬁned networks.

Original: We also note the similarity of using a Hadamard matrix as a ﬁnal classiﬁer to methods of weight binarization such as the one suggested by Courbariaux et al. (2015).
Revised: We also note the similarity of using a Hadamard matrix as a ﬁnal classiﬁer to methods of weight binarization such as the one suggested by Courbariaux et al. (2015).

Original: As the classiﬁer weights are ﬁxed to need only 1-bit precision, it is now possible to focus our attention on the features preceding it.
Revised: As the classiﬁer weights are ﬁxed to need only 1-bit precision, it is now possible to focus our attention on the features preceding it.

Original:  E XPERIMENTAL RESULTS
Revised:  E XPERIMENTAL RESULTS

Original: [Table]
Revised: [Table]

Original: 3. C IFAR 10/ 
Revised: 3. C IFAR 10/

Original: We used the well known Cifar10 and Cifar100 datasets by Krizhevsky (2009) as an initial test-bed to explore the idea of a ﬁxed classiﬁer.
Revised: We used the well known Cifar10 and Cifar100 datasets by Krizhevsky (2009) as an initial test-bed to explore the idea of a ﬁxed classiﬁer.

Original: Cifar10 is an image classiﬁcation benchmark dataset containing 50 , 000 training images and 10 , 000 test images.
Revised: Cifar10 is an image classiﬁcation benchmark dataset containing 50 , 000 training images and 10 , 000 test images.

Original: The images are in color and contain 32 × 32 pixels.
Revised: The images are in color and contain 32 × 32 pixels.

Original: There are 10 possible classes of various animals and vehicles.
Revised: There are 10 possible classes of various animals and vehicles.

Original: Cifar100 holds the same number of images of same size
Revised: Cifar100 holds the same number of images of same size

Original: [Table]
Revised: [Table]

Original: but contains 100 different classes.
Revised: but contains 100 different classes.

Original: We trained a residual network of He et al.
Revised: We trained a residual network of He et al.

Original: (2016) on the Cifar10 dataset.
Revised: (2016) on the Cifar10 dataset.

Original: We used a network of depth 56 and the same hyper-parameters used in the original work.
Revised: We used a network of depth 56 and the same hyper-parameters used in the original work.

Original: We compared two variants: the original model with a learned classiﬁer
Revised: We compared two variants: the original model with a learned classiﬁer

Original: [Table]
Revised: [Table]

Original: and our version, where a ﬁxed transformation is used.
Revised: and our version, where a ﬁxed transformation is used.

Original: The results shown in ﬁgure 2 demonstrate that although the training error is considerably lower for the network with learned classiﬁer, both models achieve the same classiﬁcation accuracy on the validation set
Revised: The results shown in ﬁgure 2 demonstrate that although the training error is considerably lower for the network with learned classiﬁer, both models achieve the same classiﬁcation accuracy on the validation set

Original: [Table]
Revised: [Table]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Revised: Our conjecture is that with our new ﬁxed parameterization, the network can no longer increase the norm of a given sample’s representation - thus learning its label requires more effort.

Revised: As this may happen for speciﬁc seen samples - it affects only training error.

Revised: We also compared using a ﬁxed scale variable α at different values vs. a learned parameter.

Revised: Results for α = { 0 . 1 , 1 , 10 } are depicted in ﬁgure 3 for both training and validation error.

Original:  We then followed to train a model on the Cifar100 dataset.
Revised: As can be seen, similar validation accuracy can be obtained using a ﬁxed scale value (in this case α = 1 orwill sufﬁce) at the expense of another hyper-parameter to seek. In all our experiments we opted to train this parameter instead. In all experiments the α scale parameter was regularized with the same weight decay coefﬁcient used on original classiﬁer. We then followed to train a model on the Cifar100 dataset.

Original: We used the DenseNet-BC model of
Revised: We used the DenseNet-BC model of

Original: Huang et al.
Revised: Huang et al.

Original: (2017) with depth of 100 layers and k = 12 .
Revised: (2017) with depth of 100 layers and k = 12 .

Original: We continued to train according to the original regime and setting described for this network and dataset.
Revised: We continued to train according to the original regime and setting described for this network and dataset.

Original: Naturally, the higher number of classes caused the number of parameters to grow and encompass about 4% of the whole model.
Revised: Naturally, the higher number of classes caused the number of parameters to grow and encompass about 4% of the whole model.

Original: Validation accuracy for the ﬁxed-classiﬁer model remained equally good as the original model, and we continued to observe the same training curve.
Revised: Validation accuracy for the ﬁxed-classiﬁer model remained equally good as the original model, and we continued to observe the same training curve.

Original: 3. I MAGENET
Revised: 3. I MAGENET

Original: In order to validate our results on a more challenging dataset, we used the Imagenet dataset introduced by Deng et al.
Revised: In order to validate our results on a more challenging dataset, we used the Imagenet dataset introduced by Deng et al.

Original: The Imagenet dataset spans over 1000 visual classes, and over 1. million samples.
Revised: The Imagenet dataset spans over 1000 visual classes, and over 1. million samples.

Original: CNNs used to classify Imagenet such as Krizhevsky et al.
Revised: CNNs used to classify Imagenet such as Krizhevsky et al.

Original: , He et al.
Revised: , He et al.

Original: Szegedy et al.
Revised: Szegedy et al.

Original: (2016) usually have a hidden representation leading to the ﬁnal classiﬁer of at least 1024 dimensions.
Revised: (2016) usually have a hidden representation leading to the ﬁnal classiﬁer of at least 1024 dimensions.

Original: This architectural choice, together with the large number of classes, causes the size of classiﬁer to exceed millions of parameters and taking a sizable share from the entire model size.
Revised: This architectural choice, together with the large number of classes, causes the size of classiﬁer to exceed millions of parameters and taking a sizable share from the entire model size.

Original: We evaluated our ﬁxed classiﬁer method on Imagenet using Resnet50 by He et al.
Revised: We evaluated our ﬁxed classiﬁer method on Imagenet using Resnet50 by He et al.

Original: with the same training regime and hyper-parameters.
Revised: with the same training regime and hyper-parameters.

Original: By using a ﬁxed classiﬁer, approximately 2-million parameters were removed from the model, accounting for about 8% of the model parameters.
Revised: By using a ﬁxed classiﬁer, approximately 2-million parameters were removed from the model, accounting for about 8% of the model parameters.

Original: Following the same procedure, we trained a Densenet169 model (Huang et al., 2017) for which a ﬁxed classiﬁer reduced about 12% of the parameters.
Revised: Following the same procedure, we trained a Densenet169 model (Huang et al., 2017) for which a ﬁxed classiﬁer reduced about 12% of the parameters.

Original: Similarly to results on Cifar10 dataset, we observed the same convergence speed and approximately the same ﬁnal accuracy on both the validation and training sets.
Revised: Similarly to results on Cifar10 dataset, we observed the same convergence speed and approximately the same ﬁnal accuracy on both the validation and training sets.

Original: Furthermore, we were interested in evaluating more challenging models where the classiﬁer parameters constitutes the majority amount.
Revised: Furthermore, we were interested in evaluating more challenging models where the classiﬁer parameters constitutes the majority amount.

Original: For this reason we chose the Shufﬂenet architecture (Zhang et al., 2017b), which was designed to be used in low memory and limited computing platforms.
Revised: For this reason we chose the Shufﬂenet architecture (Zhang et al., 2017b), which was designed to be used in low memory and limited computing platforms.

Original: The
Revised: The

Original: Shufﬂenet network contains about 1.8 million parameters, out of which 0.96 million are part of the ﬁnal classiﬁer.
Revised: Shufﬂenet network contains about 1.8 million parameters, out of which 0.96 million are part of the ﬁnal classiﬁer.

Original: Fixing the classiﬁer resulted with a model with only 0.86 million parameters.
Revised: Fixing the classiﬁer resulted with a model with only 0.86 million parameters.

Original: This model was trained and found, again, to converge to similar validation accuracy as the original.
Revised: This model was trained and found, again, to converge to similar validation accuracy as the original.

Original: Interestingly, this method allowed Imagenet training in an under-speciﬁed regime, where there are more training samples than number of parameters.
Revised: Interestingly, this method allowed Imagenet training in an under-speciﬁed regime, where there are more training samples than number of parameters.

Original: This is an unconventional regime for modern deep networks, which are usually over-speciﬁed to have many more parameters than training samples (Zhang et al., 2017a).
Revised: This is an unconventional regime for modern deep networks, which are usually over-speciﬁed to have many more parameters than training samples (Zhang et al., 2017a).

Original: Moreover, many recent theoretical results related to neural network training (Soudry & Hoffer, 2017; Xie et al., 2016; Safran & Shamir, 2016; Soltanolkotabi et al., 2017; Soudry & Carmon, 2016) and even generalization (Gunasekar et al., 2017; Advani & Saxe, 2017; Wilson et al., 2017) usually assume over-speciﬁcation.
Revised: Moreover, many recent theoretical results related to neural network training (Soudry & Hoffer, 2017; Xie et al., 2016; Safran & Shamir, 2016; Soltanolkotabi et al., 2017; Soudry & Carmon, 2016) and even generalization (Gunasekar et al., 2017; Advani & Saxe, 2017; Wilson et al., 2017) usually assume over-speciﬁcation.

Original: Table 1 summarizes our ﬁxed-classiﬁer results on convolutional networks, comparing to originally reported results.
Revised: Table 1 summarizes our ﬁxed-classiﬁer results on convolutional networks, comparing to originally reported results.

Original: We offer our drop-in replacement for learned classiﬁer that can be used to train models with ﬁxed classiﬁers and replicate our results 1 .
Revised: We offer our drop-in replacement for learned classiﬁer that can be used to train models with ﬁxed classiﬁers and replicate our results 1 .

Original: 3. L ANGUAGE MODELING
Revised: 3. L ANGUAGE MODELING

Original: As language modeling requires classiﬁcation of all possible tokens available in the task vocabulary, we were interested to see if a ﬁxed classiﬁer can be used, possible saving a very large number of trainable parameters (vocabulary size can have tens or even hundreds of thousands of different words).
Revised: As language modeling requires classiﬁcation of all possible tokens available in the task vocabulary, we were interested to see if a ﬁxed classiﬁer can be used, possible saving a very large number of trainable parameters (vocabulary size can have tens or even hundreds of thousands of different words).

Original: Recent works have already found empirically that using the same weights for both word embedding and classiﬁer can yield equal or better results than using a separate pair of weights (Inan et al., 2016; Press & Wolf, 2017; Vaswani et al., 2017).
Revised: Recent works have already found empirically that using the same weights for both word embedding and classiﬁer can yield equal or better results than using a separate pair of weights (Inan et al., 2016; Press & Wolf, 2017; Vaswani et al., 2017).

Original: This is compliant with our ﬁndings that the linear classiﬁer is largely redundant.
Revised: This is compliant with our ﬁndings that the linear classiﬁer is largely redundant.

Original: To examine further reduction in the number of parameters, we removed both classiﬁer and embedding weights and replaced them with a ﬁxed transform.
Revised: To examine further reduction in the number of parameters, we removed both classiﬁer and embedding weights and replaced them with a ﬁxed transform.

Original: We trained a language model on the WikiText2 dataset described in Merity et al.
Revised: We trained a language model on the WikiText2 dataset described in Merity et al.

Original: (2016), using the same setting in Merity et al.
Revised: (2016), using the same setting in Merity et al.

Original: We used a recurrent model with 2-layers of LSTM (Hochreiter & Schmidhuber, 1997) and embedding + hidden size of 512.
Revised: We used a recurrent model with 2-layers of LSTM (Hochreiter & Schmidhuber, 1997) and embedding + hidden size of 512.

Original: As the vocabulary of WikiText2 holds about 33 K different words, the expected number of parameters in embedding and classiﬁer is about 34-million.
Revised: As the vocabulary of WikiText2 holds about 33 K different words, the expected number of parameters in embedding and classiﬁer is about 34-million.

Original: This number makes for about 89% from the 38 M parameters used for the whole model.
Revised: This number makes for about 89% from the 38 M parameters used for the whole model.

Original: We found that using a random orthogonal transform yielded poor results compared to learned embedding.
Revised: We found that using a random orthogonal transform yielded poor results compared to learned embedding.

Original: We suspect that, in oppose to image classiﬁcation benchmarks, the embedding layer in language models holds information of the words similarities and relations, thus requiring a ﬁne initialization.
Revised: We suspect that, in oppose to image classiﬁcation benchmarks, the embedding layer in language models holds information of the words similarities and relations, thus requiring a ﬁne initialization.

Original: To test our intuition, we opted to use pre-trained embeddings using word2vec algorithm by Mikolov et al.
Revised: To test our intuition, we opted to use pre-trained embeddings using word2vec algorithm by Mikolov et al.

Original: (2013) or PMI factorization as suggested by Levy & Goldberg (2014).
Revised: (2013) or PMI factorization as suggested by Levy & Goldberg (2014).

Original: We ﬁnd that using ﬁxed word2vec embeddings, we achieve much better results.
Revised: We ﬁnd that using ﬁxed word2vec embeddings, we achieve much better results.

Original: Speciﬁcally, we use 89% less parameters than the fully learned model, and obtain only somewhat worse perplexity.
Revised: Speciﬁcally, we use 89% less parameters than the fully learned model, and obtain only somewhat worse perplexity.

Original: We argue that this implies a required structure in word embedding that stems from semantic relatedness between words and the natural imbalance between classes.
Revised: We argue that this implies a required structure in word embedding that stems from semantic relatedness between words and the natural imbalance between classes.

Original: However, we suggest that with a much more cost effective ways to train word embeddings (e.g., Mikolov et al.
Revised: However, we suggest that with a much more cost effective ways to train word embeddings (e.g., Mikolov et al.

Original: , we can narrow the gap and avoid their cost when training bigger models.
Revised: , we can narrow the gap and avoid their cost when training bigger models.

Original: [Table]
Revised: [Table]

Original:  D ISCUSSION 4. I MPLICATIONS TO FUTURE DNN MODELS AND USE CASES
Revised:  D ISCUSSION 4. I MPLICATIONS TO FUTURE DNN MODELS AND USE CASES

Original: In the last couple of years a we observe a rapid growth in the number of classes benchmark datasets contain, for example: Cifar100 (Krizhevsky, 2009), ImageNet1K, ImageNet22k (Deng et al., 2009) and language modeling (Merity et al., 2016).
Revised: In the last couple of years a we observe a rapid growth in the number of classes benchmark datasets contain, for example: Cifar100 (Krizhevsky, 2009), ImageNet1K, ImageNet22k (Deng et al., 2009) and language modeling (Merity et al., 2016).

Original: Therefore the computational demands of the ﬁnal classiﬁer will increase as well and should be considered no less than the architecture chosen.
Revised: Therefore the computational demands of the ﬁnal classiﬁer will increase as well and should be considered no less than the architecture chosen.

Original: We use the work by Sun et al.
Revised: We use the work by Sun et al.

Original: (2017) as our use case, which introduced JFT-300M - an internal Google dataset with over 18K different classes.
Revised: (2017) as our use case, which introduced JFT-300M - an internal Google dataset with over 18K different classes.

Original: Using a Resnet50 (He et al., 2016), with a 2048 sized representation, this led to a model with over 36M parameters.
Revised: Using a Resnet50 (He et al., 2016), with a 2048 sized representation, this led to a model with over 36M parameters.

Original: This means that over 60% of the model parameters reside in the ﬁnal classiﬁcation layer.
Revised: This means that over 60% of the model parameters reside in the ﬁnal classiﬁcation layer.

Original: Sun et al.
Revised: Sun et al.

Original: (2017) further describes the difﬁculty in distributing this amount of parameters between the training servers, and the need to split them between 50 sub-layers.
Revised: (2017) further describes the difﬁculty in distributing this amount of parameters between the training servers, and the need to split them between 50 sub-layers.

Original: We also note the fact that the training procedure needs to account for synchronization after each parameter update - which must incur a non-trivial overhead.
Revised: We also note the fact that the training procedure needs to account for synchronization after each parameter update - which must incur a non-trivial overhead.

Original: Our work can help considerably in this kind of scenario - where using a ﬁxed classiﬁer removes the need to do any gradient synchronization for the ﬁnal layer.
Revised: Our work can help considerably in this kind of scenario - where using a ﬁxed classiﬁer removes the need to do any gradient synchronization for the ﬁnal layer.

Original: Furthermore, using a Hadamard matrix, we can remove the need to save the transformation altogether, and make it more efﬁcient, allowing considerable memory and computational savings.
Revised: Furthermore, using a Hadamard matrix, we can remove the need to save the transformation altogether, and make it more efﬁcient, allowing considerable memory and computational savings.

Original: 4. F UTURE WORK
Revised: 4. P OSSIBLE CAVEATS

Revised: We argue that our method works due to the ability of preceding layers in the network to learn separable representations that are easily classiﬁed even when the classiﬁer itself is ﬁxed.

Revised: This property can be affected when the ratio between learned features and number of classes is small – that is, when C > N .

Revised: We’ve been experimenting with such cases, for example Imagenet classiﬁcation ( C = 1000 ) using mobilenet-0.5 (Howard et al., 2017)

Revised: where N = 512 , or reduced version of

Revised: ResNet (He et al., 2016) where N = 256 .

Revised: In both scenarios, our method converged similarly to a fully learned classiﬁer reaching the same ﬁnal validation accuracy.

Revised: This is strengthening our ﬁnding, showing that even in cases in which C > N , ﬁxed classiﬁer can provide equally good results.

Revised: Another possible issue may appear when the possible classes are highly correlated.

Revised: As a ﬁxed orthogonal classiﬁer does not account for this kind of correlation, it may prove hard for the network to learn in this case.

Revised: This may suggest another reason for the difﬁculties we experienced in training a language model using an orthogonal ﬁxed classiﬁer, as word classes tend to have highly correlated instances.

Revised: 4. F UTURE WORK

Original: Understanding that linear classiﬁers used in NN models are largely redundant allows us to consider new approaches in training and understanding these models.
Revised: Understanding that linear classiﬁers used in NN models are largely redundant allows us to consider new approaches in training and understanding these models.

Original: Recent works (Neyshabur et al., 2017; Bartlett et al., 2017) suggested a connection between generalization capabilities of models and various norm-related quantities of their weights.
Revised: Recent works (Neyshabur et al., 2017; Bartlett et al., 2017) suggested a connection between generalization capabilities of models and various norm-related quantities of their weights.

Original: Such results might be potentially simpliﬁed in our model, since we have a single scalar variable, which seems to be the only relevant scale in the model (since we normalize the last hidden layer, and ﬁx the last weight layer).
Revised: Such results might be potentially simpliﬁed in our model, since we have a single scalar variable (i.e., scale), which seems to be the only relevant parameter in the model (since we normalize the last hidden layer, and ﬁx the last weight layer).

Original: The use of ﬁxed classiﬁers might be further simpliﬁed in Binarized Neural Networks (Hubara et al., 2016a), where the activations and weights are restricted to ± 1 during propagations.
Revised: The use of ﬁxed classiﬁers might be further simpliﬁed in Binarized Neural Networks (Hubara et al., 2016a), where the activations and weights are restricted to ± 1 during propagations.

Original: In this case the norm of the last hidden layer is constant for all samples (equal to the square root of the hidden layer width).
Revised: In this case the norm of the last hidden layer is constant for all samples (equal to the square root of the hidden layer width).

Original: This constant can be absorbed into the scale constant α , and there is no need in a per-sample normalization as in eq. 1.
Revised: This constant can be absorbed into the scale constant α , and there is no need in a per-sample normalization as in eq. 1.

Original: We also plan to further explore more efﬁcient ways to learn word embeddings, where similar redundancy in classiﬁer weights may suggest simpler forms of token representations - such as low-rank or sparse versions, allowing similar beneﬁts to the ﬁxed transformations we suggested.
Revised: We also plan to further explore more efﬁcient ways to learn word embedding, where similar redundancy in classiﬁer weights may suggest simpler forms of token representations - such as low-rank or sparse versions, allowing similar beneﬁts to the ﬁxed transformations we suggested.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work we suggested removing the parameters from the classiﬁcation layer used in deep neural networks.
Revised: In this work we suggested removing the parameters from the classiﬁcation layer used in deep neural networks.

Original: We showed empirical results suggesting that keeping the classiﬁer ﬁxed cause little or no decline in classiﬁcation performance for common balanced datasets such as Cifar and Imagenet, while allowing a noticeable reduction in trainable parameters.
Revised: We showed empirical results suggesting that keeping the classiﬁer ﬁxed cause little or no decline in classiﬁcation performance for common balanced datasets such as Cifar and Imagenet, while allowing a noticeable reduction in trainable parameters.

Original: We argue that ﬁxing the last layer can reduce the computational complexity for training as well as the communication cost in distributed learning.
Revised: We argue that ﬁxing the last layer can reduce the computational complexity for training as well as the communication cost in distributed learning.

Original: Further more, using a Hadamard matrix as classiﬁer might lead to some computational beneﬁts when properly implemented, and save memory otherwise spent on large amount of transformation coefﬁcients.
Revised: Furthermore, using a Hadamard matrix as classiﬁer might lead to some computational beneﬁts when properly implemented, and save memory otherwise spent on large amount of transformation coefﬁcients.

Original: As datasets tend to become more complex by time (e.g., Cifar100,
Revised: As datasets tend to become more complex by time (e.g., Cifar100,

Original: ImageNet1K, ImageNet22k, JFT-300M, and language modeling) we believe that resource hungry afﬁne transformation should remain ﬁxed during training, at least partially.
Revised: ImageNet1K, ImageNet22k, JFT-300M, and language modeling) we believe that resource hungry afﬁne transformation should remain ﬁxed during training, at least partially.

Original: We also found that new efﬁcient methods to create pre-deﬁned word embeddings should be explored, as they require huge amount of parameters that can possibly be avoided when learning a new task.
Revised: We also found that new efﬁcient methods to create pre-deﬁned word embeddings should be explored, as they require huge amount of parameters that can possibly be avoided when learning a new task.

Original: Based on these ﬁndings, we recommend future research to focus on representations learned by the non-linear part of neural networks - up to the ﬁnal classiﬁer, as it seems to be highly redundant.
Revised: Based on these ﬁndings, we recommend future research to focus on representations learned by the non-linear part of neural networks - up to the ﬁnal classiﬁer, as it seems to be highly redundant.