Fast LSTM Inference by Dynamic Decomposition on Cloud Systems LSTM (long shortterm memory) is a powerful deep learning technique that has been widely used in many realworld datamining applications such as language modeling and machine translation. In this paper we aim to minimize the latency of LSTM inference on cloud systems without losing accuracy. If an LSTM model does not fit in cache, the latency due to data movement will likely be greater than that due to computation. In this case we reduce model parameters. If, as in most applications we consider, the LSTM models are able to fit the cache of cloud server processors, we focus on reducing the number of floating point operations, which has a corresponding linear impact on the latency of the inference calculation. Thus, in our system, we dynamically reduce model parameters or flops depending on which most impacts latency. Our inference system is based on Singular Value Decomposition (SVD) and Canonical Polyadic Decomposition (CPD). We evaluate our system based on models from a series of realworld applications like language modeling, computer vision, question answering, and sentiment analysis. Users of our system can use either pretrained models or start from scratch. Our system achieves 15X average speedup for six realworld applications without losing accuracy in inference.

Optimal InputDependent EdgeCloud Partitioning for RNN Inference Recurrent Neural Networks (RNNs) such as those based on the LongShort Term Memory (LSTM) architecture are stateoftheart deep learning models for sequence analysis. Given the complexity of RNNbased inference, IoT devices typically offload this task to a cloud server. However, the complexity of RNN inference strongly depends on the length of the processed input sequence. Therefore, when communication time is taken into account, it may be more convenient to process short input sequences locally and only offload long ones to the cloud. In this paper, we propose a lowoverhead runtime tool that performs this decision automatically. Results based on performance profiling of real edge and cloud devices show that our method is able to reduce the total execution time of the system by up to 20% compared to solutions that execute the RNN inference fully locally or fully in the cloud.