Paper:
Published as a conference paper at ICLR 2018 FIX YOUR CLASSIFIER: THE MARGINAL VALUE OF TRAINING THE LAST WEIGHT LAYER
Neural networks are commonly used as models for classification for a wide variety of tasks. Typically, a learned affine transformation is placed at the end of such models, yielding a per-class value used for classification. This classifier can have a vast number of parameters, which grows linearly with the number of possible classes, thus requiring increasingly more resources. In this work we argue that this classifier can be fixed, up to a global scale constant, with little or no loss of accuracy for most tasks, allowing memory and computational benefits. Moreover, we show that by initializing the classifier with a Hadamard matrix we can speed up inference as well. We discuss the implications for current understanding of neural network models.

Review:
Revised Review:
The authors have largely addressed my concerns with the revised manuscript. I still have some doubts about the C > N setting (the new settings of C / N of 4 and 2 aren't C >> N, and the associated results aren't detailed clearly in the paper), but I think the paper warrants acceptance.
Original Review:
The paper proposes fixing the classification layers of neural networks, replacing the traditional learned affine transformation with a fixed (e.g., Hadamard) matrix. This is motivated by the observation that classification layers frequently constitute a non-trivial fraction of a network's overall parameter count, compute requirements, and memory usage, and by the observation that removal of pre-classification fully-connected layers has often been found to have minimal impact on performance. Experiments are performed on a range of datasets and network architectures, in both image classification and NLP settings.
First, I'd like to note that the empirical component of this paper is strong: I was impressed by the breadth of architectures and settings covered, and the experiments left me reasonably convinced that the classification layer can often be fixed, at least for image classification tasks, without significant loss of accuracy.
I have two general concerns. For one, removing the fully connected classification layer is not a novel idea; All Convolutional Networks (https://arxiv.org/abs/1412.6806) reported excellent results without an additional fully connected affine transform (just a global average pooling after the last convolutional layer). I think it would be worth at least referencing/discussing differences with this and other all-convolutional architectures. Including a fixed Hadamard matrix for the classification layer is I believe new (although related to an existing literature on using structured matrices in neural networks).
However, I have doubts about the ability of the approach to scale to problems with a larger number of classes, which arguably is a primary motivation of the paper ("parameters ... grow linearly with the number of classes"). Specifically, the idea of using a fixed N x C matrix with C orthogonal columns (such as Hadamard) is only possible when N > C. This is a critical point: in the N > C regime, a final hidden representation with N dimensions can be chosen to achieve *any* C-dimensional output, regardless of the projection matrix used (so long as it is full rank). This makes it seem fairly reasonable to me that the network can (at least approximately, and complicated by the ReLU nonlinearities) fold the "desired" classification layer into the previous layer, especially with a learned scaling and bias term. In fact it's not clear to me that the fixed classification layer accomplishes anything here, beyond projecting from N -> C (i.e., if N = C, I'd guess it could be removed entirely similar to all convolutional nets, as long as the learned scaling and bias were retained).
On the other hand, when C > N, it is not possible to have mutually orthogonal columns, and in general the output is constrained to lie in an N-dimensional subspace of the overall C-dimensional output space. Picking somewhat randomly a *fixed* N-dimensional subspace seems like a bad idea when N << C, since it is unlikely to select a subspace in which it is possible to adequately capture correlations between the different classes. This makes the proposed technique much less appealing for precisely the family of problems where it would be most effective in reducing compute/memory requirements. It also provides (in my view) a clearer explanation for the failure of the approach in the NLP setting. These issues were not discussed anywhere in the text as far as I can tell, and I think it's necessary to at least acknowledge that mutually orthogonal columns can't be chosen when C > N in section 2.2 (and probably include a longer discussion on the probable implications).
Overall, I think the paper provides a useful observation that clearly isn't common knowledge, since classification layers persist in many popular recent architectures. But the notion of fixing or removing the classification layer isn't particularly novel, and I don't believe the proposed technique would scale well to settings with many classes. As is I think the paper falls slightly short.