Compared with cheap addition operation, multiplication operation is of much higher computation complexity. The widely-used convolutions in deep neural networks are exactly cross-correlation to measure the similarity between input feature and convolution filters, which involves massive multiplications between float values. In this paper, we present adder networks (AdderNets) to trade these massive multiplications in deep neural networks, especially convolutional neural networks (CNNs), for much cheaper additions to reduce computation costs. In AdderNets, we take the \ell_1-norm distance between filters and input feature as the output response. The influence of this new similarity measure on the optimization of neural network have been thoroughly analyzed. To achieve a better performance, we develop a special training approach for AdderNetsby investigating the \ell_p-norm. We then propose an adaptive learning rate strategy  to enhance the training procedure of AdderNets according to the magnitude of each neuron's gradient. As a result, the proposed AdderNets can achieve  75.7\% Top-1 accuracy  92.3\% Top-5 accuracy using ResNet-50 on the ImageNet dataset without any multiplication in convolutional layer. Moreover, we develop a theoretical foundation for AdderNets, by showing that both the single hidden layer AdderNet and the width-bounded deep AdderNet with ReLU activation functions are universal function approximators. These results match those of the traditional neural networks using the more complex multiplication units. An approximation bound for AdderNets with a single hidden layer is also presented .
Compared with cheap addition operation, multiplication operation is of much higher computation complexity. The widely-used convolutions in deep neural networks are exactly cross-correlation to measure the similarity between input feature and convolution filters, which involves massive multiplications between float values. In this paper, we present adder networks (AdderNets) to trade these massive multiplications in deep neural networks, especially convolutional neural networks (CNNs), for much cheaper additions to reduce computation costs. In AdderNets, we take the \ell_1-norm distance between filters and input feature as the output response. We first develop a theoretical foundation for AdderNets, by showing that both the single hidden layer AdderNet and the width-bounded deep AdderNet with ReLU activation functions are universal function approximators. An approximation bound for AdderNets with a single hidden layer is also presented. We further analyze the influence of this new similarity measure on the optimization of neural network and develop a special training scheme for AdderNets. Based on the gradient magnitude, an adaptive learning rate strategy is proposed to enhance the training procedure of AdderNets . AdderNets can achieve a 75.7\% Top-1 accuracy and a 92.3\% Top-5 accuracy using ResNet-50 on the ImageNet dataset without any multiplication in the convolutional layer .
R
The
We first develop a theoretical foundation for AdderNets, by showing that both the single hidden layer AdderNet and the width-bounded deep AdderNet with ReLU activation functions are universal function approximators. An approximation bound for AdderNets with a single hidden layer is also presented. We further analyze the
R
have been thoroughly analyzed. To achieve a better performance, we
and
R
approach for AdderNetsby investigating the \ell_p-norm. We then propose
scheme for AdderNets. Based on the gradient magnitude,
A
is proposed
R
according to the magnitude of each neuron's gradient. As a result, the proposed
.
A
a
A
and a
R
convolutional layer. Moreover, we develop a theoretical foundation for AdderNets, by showing that both the single hidden layer AdderNet and the width-bounded deep AdderNet with ReLU activation functions are universal function approximators. These results match those of the traditional neural networks using the more complex multiplication units. An approximation bound for AdderNets with a single hidden layer is also presented
the convolutional layer