BNReLU: Combine Batch Normalization and Rectified Linear Unit to Reduce Hardware Overhead With the pursuit of higher accuracy, the convolutional neural network has become deeper. Thus, the hardware overhead occupied have increased. Batch Normalization (BN) operation is an indispensable part of the network, and its occupied hardware resources cannot be ignored. In this paper, aiming to reduce hardware overhead caused by BN operation, we combine BN operation and ReLU activation function and propose the BNReLU operation without retraining the network. The experiment results show that compared with traditional BNReLU layer, hardware overhead of BNReLU operation is reduced by about 50%. With 32bit float point input for each layer, the hardware overhead of BRAM, DSP, FF, and LUT for convolutional layer in the FPGA is reduced by 17.16%, 23.08%, 17.64%, and 16.05%.

Minimizing OffChip Memory Access for Deep Convolutional Neural Network Training When training convolutional neural networks, a large amount of operations and memory access are in need, which easily lead to the bottleneck of xe2x80x9cmemory wallxe2x80x9d and decrease the computational performance and efficiency. Batch Normalization (BN) can effectively speed up the deep network training convergence, but it has complex data dependence and causes more serious xe2x80x9cmemory wallxe2x80x9d bottleneck. Aiming at the xe2x80x9cmemory wallxe2x80x9d problem occurred in the training for convolutional neural network using BN algorithm, the training method with splitting BN layer and multilayer fusion calculation is proposed to reduce the memory access in model training. Firstly, by reordering xe2x80x9cCONVBNRELUxe2x80x9d (CBR) block, we trade computation for memory access with extra computation to reduce data accessed during training. Secondly, according to the memory access characteristics of the BN layer, the BN layer is divided into two sublayers, which are respectively fused with the adjacent layers and the CBR block is recombined into xe2x80x9cBN_BRELUCONVBN_Axe2x80x9d (BRCB), which further reduces the readwrite of the main memory during training and alleviates the xe2x80x9cmemory wallxe2x80x9d bottleneck to improve accelerator computational efficiency. The experimental results show that when using the NVIDIA TESLA V100 GPU to train ResNet50, Inception V3 and DenseNet models, compared with the original training method, the amount of data accessed using BRCB multilayer fusion optimization method is reduced by 33%, 22% and 31% respectively, and the actual computing efficiency of V100 is improved by 19%, 18% and 21% respectively.