Ising Dropout with Node Grouping for Training and Compression of Deep Neural Networks Dropout is a popular regularization method to reduce overfitting while training deep neural networks and compress the inference model. In this paper, we propose Ising dropout with node grouping, which represents a deep multilayer perceptron (MLP) neural network as a graph with fixed grouped nodes and uses the Ising energy to drop group of nodes. This method is an extension to our proposed Ising dropout method, which had the limit of solving the Ising energy model for MLPs with limited graph order. The proposed fixed grouping method enables applying dropout to deep MLPs with any order. Performance of this method is evaluated on handwritten digits (MNIST), FashionMNIST, Free Spoken Digit Dataset (FSDD), and Street View House Numbers (SVHN) datasets and compared with the standard dropout and standout methods. Preliminary results show that the proposed approach can keep the classification performance competitive to the original network while eliminating optimization of unnecessary network parameters in each training cycle. This method can compress the inference model significantly while maintaining the classification performance.

Reconciling FeatureReuse and Overfitting in DenseNet with Specialized Dropout Recently convolutional neural networks (CNNs) achieve great accuracy in visual recognition tasks. DenseNets become one of the most popular CNN models due to its effectiveness in the featurereuse. However, like other CNN models, DenseNets also face the overfitting problem if not more severe. Existing dropout methods can be applied but not effective. In particular, the property of the featurereuse in DenseNets will be impeded, and the dropout effect will be weakened by the spatial correlation inside feature maps. To address these problems, we craft the design of a specialized dropout method from three aspects, the dropout location, the dropout granularity, and the dropout probability. The insights attained here could potentially be applied as a general approach for boosting the accuracy of other CNN models with similar shortcut connections. Experimental results show that DenseNets with our specialized dropout method yield better accuracies compared to vanilla DenseNets and stateoftheart CNN models, and such accuracy boost increases with the model depth.