Vector-Vector-Matrix Architecture: A Novel Hardware-Aware Framework for Low-Latency Inference in NLP Applications (2020)

Abstract: Deep neural networks have become the standard approach to building reliable Natural Language Processing (NLP) applications, ranging from Neural Machine Translation (NMT) to dialogue systems. However, improving accuracy by increasing the model size requires a large number of hardware computations, which can slow down NLP applications significantly at inference time. To address this issue, we propose a novel vector-vector-matrix architecture (VVMA), which greatly reduces the latency at inference time for NMT. This architecture takes advantage of specialized hardware that has low-latency vector-vector operations and higher-latency vector-matrix operations. It also reduces the number of parameters and FLOPs for virtually all models that rely on efficient matrix multipliers without significantly impacting accuracy. We present empirical results suggesting that our framework can reduce the latency of sequence-to-sequence and Transformer models used for NMT by a factor of four. Finally, we show evidence suggesting that our VVMA extends to other domains, and we discuss novel hardware for its efficient use.

## introduction artificial neural networks have become increasingly popular over the last decade as they excel in tasks such as object detection and speech recognition @xcite , which are becoming more commonplace with the use of self-driving cars and virtual assistants. the rapid development of deep neural networks has also made them the dominant approach for natural language processing (nlp) applications, ranging from neural machine translation (nmt) @xcite @xcite ) and text summarization @xcite @xcite to virtual assistants such as apple siri, amazon alexa, and google home. unfortunately, neural networks are slow for training, inference and use due to their vast computational complexity. several approaches have been proposed to address these issues including (a) quantization and pruning, (b) efficient models with less computational demand, and (c) specialized hardware accelerators @xcite . while direction (a) has been well-studied @xcite @xcite , and can be considered complementary to @xcite , optimizing the combination of (b) and (c) has not been considered, to the best of our knowledge. thus, here we propose a novel vector-vectormatrix architecture (vvma) that compresses neural networks, while optimizing for hardware performance at inference time. therefore, we optimize (b) and (c), without conflicting with (a), i.e., using quantization and pruning can potentially further boost the efficiency of our framework. figure 1 illustrates this vvma in contrast to a traditional vector-matrix architecture. moreover, the inherently sequential nature of many nlp tasks can increase the latency at inference time. constrained by their memory bandwidth and footprint, modern accelerators rely on large batch sizes to avoid under-utilization. however, it is not always possible to increase the batch size if conclusions have to be inferred quickly, e.g., for real-time inference. for example, the matrix multiply unit of state-of-the-art accelerators, such as google's tensor processing unit (tpu), will "stall" when translating a single sentence, thus increasing the overall latency @xcite . our architecture can improve the tpu and other ai accelerators for small-batch inference. thus, unlike other methods for compressing neural networks, the vvma is designed to take advantage of the dataflow and the architecture of certain kinds of hardware accelerators such as the tpu. our contributions are as follows: the rest of this paper is organized as follows: in section 2, we elaborate on directions (b) and (c), and we relate them to vvmas. in section 3, we motivate vvmas as a faster improvement of the tpu's architecture and dataflow at inference time, and we then analyze our framework in its universality, including tips for efficient implementation of vvmas. as a proof of concept, in section 4 we demonstrate empirical inference speed-ups for nmt using seq2seq-lstm and transformer models, which are both notorious for their computational complexity. we also show ablation studies and extensions to other tasks. in section 5, we explore novel accelerators that can benefit from vvmas. finally, we offer concluding remarks in section 6, and we point to possible directions for future work. ## background here, we look at efficient models from the software and the hardware side, and we discuss the advantages of merging them in a co-design manner. we further discuss the importance of wall-clock speed versus floating point operations and why from this perspective our weight sharing matrices will decrease inference rather than training time. ## architecture in this section, we present our approach to constructing a vvma, including implementation details that are necessary to use vvmas in practice. ## results in this section, we present empirical results showing that vvmas can substitute different types of weight matrices in neural networks (nns). specifically, we use our vvmas in seq2seq-lstm and transformer nmt. we report some theoretical speedups that vvmas provide when using a tpustyle architecture. we then present a small ablation study where we modify our vvmas by removing the diagonal terms d i,j or by varying the value of k. we also compare vvma to standard low-rank approximations. finally, we show that our technique extends to language modelling with transformer-xl, and beyond nlp tasks. unless otherwise noted, all results in this section use vvmas with @xmath0. that is, the matrix w in the neural network is replaced with a vvma that uses a 32 × 32 matrix m along with 32 × 32 diagonal matrices d i,j as shown in equation 4. ## discussion below, we discuss new ai hardware that could optimize inference for neural networks via vvmas. this hardware would decrease latency at inference time rather than decreasing the training time. tensor processing unit. as mentioned above, google's tensor processing units (tpu) has a dedicated matrix multiply unit @xcite . we believe that a modified version of the tpu could take advantage of vvmas. the necessary modifications would be relatively simple. as illustrated in figure 2 , we would add a dedicated vector-vector unit before the matrix multiply unit, and we would pipeline it and initialize it at the same time as the matrix multiply unit. as seen in section 4.2, this would noticeably decrease the number of inference clock cycles in seq2seq-lstm models. tensor cores. nvidia's newest gpus have dedicated matrix multiply units called tensor cores, which can perform 4 × 4 matrix multiplications in a single clock cycle @xcite . adding vector-vector units before each tensor core would make them more efficient for vvmas. the largest speedup would come from the time spent loading matrices from the memory into the tensor cores. for instance, if multiple tensor cores share the same matrix elements, this would decrease the latency when performing inference. ## conclusion and future work we have proposed a novel vector-vector-matrix architecture for low-latency inference, and we have demonstrated theoretical and empirical speed-ups for seq2seq-lstm and transformer models, with application to neural machine translation, language modeling, and image classification. we hope that this work would bring the novel concept of ai co-design (between software and hardware) to the domain of nlp applications. in future work, we plan to optimize the lowlevel code and to develop new hardware to deploy vvmas in real-world applications. distilling models to their vvma counterparts would be an interesting experiment, and potentially an orthogonal enhancement to pre-existing frameworks @xcite . vvmas could also be an orthogonal contribution to other factorizations of nlp models, such as in @xcite .