  Low-latency, low-power portable recurrent neural network (RNN) accelerators
offer powerful inference capabilities for real-time applications such as IoT,
robotics, and human-machine interaction. We propose a lightweight Gated
Recurrent Unit (GRU)-based RNN accelerator called EdgeDRNN that is optimized
for low-latency edge RNN inference with batch size of 1. EdgeDRNN adopts the
spiking neural network inspired delta network algorithm to exploit temporal
sparsity in RNNs. Weights are stored in inexpensive DRAM which enables EdgeDRNN
to compute large multi-layer RNNs on the most inexpensive FPGA. The sparse
updates reduce DRAM weight memory access by a factor of up to 10x and the delta
can be varied dynamically to trade-off between latency and accuracy. EdgeDRNN
updates a 5 million parameter 2-layer GRU-RNN in about 0.5ms. It achieves
latency comparable with a 92W Nvidia 1080 GPU. It outperforms NVIDIA Jetson
Nano, Jetson TX2 and Intel Neural Compute Stick 2 in latency by 5X. For a batch
size of 1, EdgeDRNN achieves a mean effective throughput of 20.2GOp/s and a
wall plug power efficiency that is over 4X higher than the commercial edge AI
platforms.
