End to End Binarized Neural Networks for Text Classification (2020)

Abstract: Deep neural networks have demonstrated their superior performance in almost every Natural Language Processing task, however, their increasing complexity raises concerns. A particular concern is that these networks pose high requirements for computing hardware and training budgets. The state-of-the-art transformer models are a vivid example. Simplifying the computations performed by a network is one way of addressing the issue of the increasing complexity. In this paper, we propose an end to end binarized neural network for the task of intent and text classification. In order to fully utilize the potential of end to end binarization, both the input representations (vector embeddings of tokens statistics) and the classifier are binarized. We demonstrate the efficiency of such a network on the intent classification of short texts over three datasets and text classification with a larger dataset. On the considered datasets, the proposed network achieves comparable to the state-of-the-art results while utilizing 20-40% lesser memory and training time compared to the benchmarks.

## introduction in recent years, deep neural networks have achieved great success in a variety of domains, but the networks are becoming more and more computationally expensive due to their ever-growing size. this tendency has been noticed in @xcite and it has been recommended that academia and industry researchers should draw their attention towards more computationally efficient methods. at the same time, many important application areas such as chatbots, iot devices, mobile devices, and other types of powerconstrained and resource-constrained platforms re-quire solutions that would be highly computationally and memory efficient. such use-cases limit the potential use of the state-of-the-art deep networks. one viable solution is the transformation of these high-performance neural networks to a more computationally efficient architecture. recently, binarized convolutional neural networks (bnn) @xcite have been developed where both weights and activations are restricted to {+1, -1}. bnn is a highly computationally efficient network with a much lower memory footprint. tasks like language modeling @xcite were performed using binarized neural networks, but, to the best of our knowledge, in the area of text classification, no end to end trainable binarized architectures have been demonstrated yet. in this paper, we introduce an architecture for the tasks of intent and text classifications that fully utilizes the power of binary representations. the input representations are tokenized and embedded in binary high-dimensional (hd) vectors forming distributed representations using the paradigm known as hyperdimensional computing @xcite . the binary input representations are used for training an end to end bnn classifier for intent classification. classification performance-wise, the binarized architecture achieves results comparable to the state-of-the-art on several standard intent classification datasets. the efficiency of the proposed architecture is shown in terms of its time and memory complexity relative to non-binarized architectures. ## proposed method figure 1 presents a schematic overview of the architecture. given an input text document d, we first pre-process the document. the pre-processed document is then tokenized into the corresponding @xmath0, t 2 , ..., t n >, which are used as an input to a count-based vectorizer. the representation of vectorizers, which is sparse and localist, is embedded into an hd vector (distributed representation) using hyperdimensional computing. hd vector representing the counter's content can be binarized. it is used as an input to a classifier. the primary classifier studied in this work is bnn, but other classifiers are also considered for benchmarking. ## conclusion in this work, we show that it is possible to achieve comparable to the state-of-the-art results while using the binarized representations of all the components of the text classification architecture. this allows exploring the effectiveness of binary representations both for reducing the memory footprint of the architecture and for increasing the energy-efficiency of the inference phase due to the effectiveness of binary operations. this work takes a step towards enabling nlp functionality on resource-constrained devices.