Scalable LowLatency Persistent Neural Machine Translation on CPU Server with Multiple FPGAs We present a CPU server with multiple FPGAs that is purely softwareprogrammable by a unified framework to enable flexible implementation of modern reallife complex AI that scales to large model size (100M parameters), while delivering realtime inference latency (ms). Using multiple FPGAs, we scale by keeping a large model persistent in onchip memories across FPGAs to avoid costly offchip accesses. We study systems with 1 to 8 FPGAs for different devices: Intelxc2xae Arriaxc2xae 10, Stratixxc2xae 10, and a research Stratix 10 with an AI chiplet. We present the first multiFPGA evaluation of a complex NMT with bidirectional LSTMs, attention, and beam search. Our system scales well. Going from 1 to 8 FPGAs allows hosting 8xc3x97 larger model with only 2xc3x97 latency increase. A batch1 inference for a 100Mparameter NMT on 8 Stratix 10 FPGAs takes only 10 ms. This system offers 110xc3x97 better latency than the only prior NMT work on FPGAs, which uses a highend FPGA and stores the model offchip.

An OpenCLBased Hybrid CNNRNN Inference Accelerator On FPGA Recently, Convolution Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and CNNRNN hybrid networks have demonstrated great success in many deep learning scenarios. Although many dedicated FPGA accelerators for a certain kind of network have been proposed, few of them combine CNN and RNN acceleration together. In this paper we propose a highthroughput and resourceefficient CNNRNN fusion accelerator on FPGA with commercial OpenCL to support generalpurpose DNNs. It utilizes a novel streaming architecture and mapping strategy to implement the most computationintensive and resourcedemanding parts in DNNs on the same computation logic. By such a hardware reuse method, it realizes resource efficiency in accelerating CNNs, RNNs and their hybrid networks. Our accelerator follows a layerbylayer, subgraphbysubgraph or subnetworkbysubnetwork execution mode, which facilities it to deploy most DNNs flexibly during runtime with best performance. YOLOv2, LSTM and CRNN are tested with our work on Intel Arria10 GX1150 FPGA. It achieves 646 GOPS throughput on CRNN, which is the best performance on CNNRNN hybrid networks among highlevelsynthesis (HLS) based FPGA accelerators. Moreover, its throughput for CNNs and RNNs is competitive to the stateoftheart specialized FPGA accelerators.