Paper:
Dense Associative Memory for Pattern Recognition
A model of associative memory is studied, which stores and reliably retrieves many more patterns than the number of neurons in the network. We propose a simple duality between this dense associative memory and neural networks commonly used in deep learning. On the associative memory side of this duality, a family of models that smoothly interpolates between two limiting cases can be constructed. One limit is referred to as the feature-matching mode of pattern recognition, and the other one as the prototype regime. On the deep learning side of the duality, this family corresponds to feedforward neural networks with one hidden layer and various activation functions, which transmit the activities of the visible neurons to the hidden layer. This family of activation functions includes logistics, rectified linear units, and rectified polynomials of higher degrees. The proposed duality makes it possible to apply energy-based intuition from associative memory to analyze computational properties of neural networks with unusual activation functions -the higher rectified polynomials which until now have not been used in deep learning. The utility of the dense memories is illustrated for two test cases: the logical gate XOR and the recognition of handwritten digits from the MNIST data set.

Review:
The paper studies an associative memory model with polynomial and rectified polynomial Hamiltonians. The results focus on the one-step retrieval case, for which the model is closely related to a standard feedforward neural network. The authors motivate the use of rectified polynomial activation functions experimentally and invoking an increase in capacity argument.
The theoretical contribution presented in 291—310 is a welcome insight on the computational power of ReLUs. The experimental results for rectified polynomial units reported in figures 2 and 3 are interesting and apparently novel, even in the context of standard feedforward multi-layer networks.  Being 291—297 a central point of the paper it should be expanded and better justified. Furthermore, the simple capacity analysis developed in p. 3 for the polynomial energy function is invoked here for the rectified polynomial energy function. This has to be justified.  The paper starts from and mostly focuses on the associative memory (Hamiltonian) formulation, but then the findings are restricted to one-step retrieval. Can the authors provide at least some discussion on how to extrapolate the current findings to the case of recurrent network dynamics? Could a toy example relevant to learning be designed to illustrate the model under iterative retrieval?  The reference list should be extended and the novel aspects of the work more clearly delineated. There is previous published research on variants of the Hopfield model where the energy function is extended to include higher order terms. In particular, energy functions of the form of Eq. 2 with polynomial F have been studied before, for instance by Abbott and Arian (Phys Rev A, 1987; see also Ref. 4 therein). Such works should be cited. For example, the reported nonlinear increase (98–104) in capacity with polynomial degree has been found before.  Regarding the introduction, it should be noted that the reformulation of a classification problem as a partial input completion one is not novel. It is actually a common approach to use similar energy-based models to perform classification.  I couldn’t understand the remark on 105–108 on the structuring of the higher order interactions. In fact, how was the number of ‘memories’ K set for the MNIST experiments, and where is it reported?  Regarding section 5: can a comment be made on how a network with more than one hidden layer could be obtained starting from an associative memory problem? How would the picture change beyond one-step retrieval?  The Appendix would become clearer by including the weight update obtained after differentiating C.