Paper:
Breaking the Activation Function Bottleneck through Adaptive Parameterization
Standard neural network architectures are non-linear only by virtue of a simple element-wise activation function, making them both brittle and excessively large. In this paper, we consider methods for making the feed-forward layer more flexible while preserving its basic structure. We develop simple drop-in replacements that learn to adapt their parameterization conditional on the input, thereby increasing statistical efficiency significantly. We present an adaptive LSTM that advances the state of the art for the Penn Treebank and WikiText-2 word-modeling tasks while using fewer parameters and converging in less than half the number of iterations.

Review:
The work proposes a form of adaptive parameterization for the activation function in neural nets. Several instantiations for feed-forward layers and a delicate adaptive design for LSTM are presented. Empirically, the adaptive version with few parameters outperforms the static version for MNIST classification and word-level language modeling.   While the idea of using adaptive parameterization is not new, the work introduces the concept of activation function bottleneck, which is not rigorously defined. If I understand correctly, the bottleneck conceptually refers to the fact that since the non-linearity is a static function of the pre-activation, in order to well exploit the "non-linear scaling" effect of the non-linearity, the weight matrix has to additionally encode some input specific information. As a result, the capacity of the weight matrix is not used efficiently. To break the bottleneck, one can make the weight matrix input specific (HyperNetworks) or the entire layer input specific in a more complicated way.   This work proposes a particular input-dependent parameterization of a layer by (1) removing the nonlinearity and (2) interleaving adaptive diagonal matrices produced by hyper-networks and static weight matrices. As the parameterization has good empirical performance, there are a few issues: - The proof of Property 1 is not satisfying, as it essentially assumes pi(x) can approximate G x / W.  - Further, while the original nonlinearity in a layer is removed, there are nonlinearities in the HyperNetworks. Hence, I tend to feel that the proposed adaptive layer is more like a more complicated sub-network with "multiplicative interactions" between various components, where some of these components are affine mappings of the input and others are non-linear mappings of the input. Thus, I'm not quite convinced by the perspective that the proposed layer structure is just an adaptive parameterization of the activation function.  In addition, for the Penn Treebank experiment, the performance improvement of the aLSTM with 17M parameters is relatively small compared to the improvement of the aLSTM with comparable parameter size in the wikitext-2 experiment. So, it's better to run an aLSTM with comparable parameter size to AWD-LSTM for Penn Treebank, which is more informative.  In summary, this work hypothesizes a "capacity bottlenck" due to the static behavior of the activation function without a formal definition. Notably, an effective adaptive parameterization of LSTM is proposed. However, no detailed analysis is given to justify the effectiveness of the proposed method, leaving the source of the effectiveness not clear.