Lightweight Text Classifier using Sinusoidal Positional Encoding (2020)

Abstract: Large and complex models have recently been developed that require many parameters and much time to solve various problems in natural language processing. This paper explores an efficient way to avoid models being too complicated and ensure nearly equal performance to models showing the state-of-the-art. We propose a single convolutional neural network (CNN) using the sinusoidal positional encoding (SPE) in text classification. The SPE provides useful position information of a word and can construct a more efficient model architecture than before in a CNN-based approach. Our model can significantly reduce the parameter size (at least 67%) and training time (up to 85%) while maintaining similar performance to the CNN-based approach on multiple benchmark datasets.

## introduction in recent years, convolutional neural networks (cnn) have shown remarkable performance and time-efficiency in text classification tasks such as sentiment analysis and document classification. @xcite @xcite . in particular, @xcite shows the importance of pre-trained word vectors and finetuned word vectors for each task, currently being used in many studies. however, in cnn, convolution and pooling operations lose information about the local order of words @xcite . to solve this problem, various studies have been conducted on model architectures capable of effectively extracting features from cnn @xcite @xcite @xcite . on the other hand, when using a word vector based on a distributed representation, fine-tuning of the pre-trained word vector for each task requires more parameters in proportion to the sequence length. along with this, the proposed large and complex model architectures have improved performance, but additional space and time costs are required due to numerous parameters @xcite . recently, studies have been conducted on efficient model architectures with less parameter size and computational cost without significant performance loss compared to existing models @xcite @xcite . @xcite introduced sinusoidal positional encoding (spe) and multi-head selfattention to introduce a simple model architecture with less parameter size and computational cost than before. the spe provides useful word position information without recurrent neural networks (rnn) and, at the same time, requires less parameter size. due to these advantages, studies have been conducted to show the usefulness of location encoding in computer vision and natural language processing @xcite . in this paper, we propose a single cnn with spe and construct a simpler model architecture than before with useful position information. spe's position information is applied to the pre-trained word vector, and it is maintained as a static vector without fine-tuning. a single cnn extracts significant features from word vectors containing position information. the results of this study are confirmed against six benchmark datasets. the proposed model is challenging to achieve state-ofthe-art, but it maintained similar performance as before, despite significantly reducing the parameter size and time cost compared to the previous cnn-based approaches. ## model as shown in figure 1 , the proposed model's architecture is based on @xcite . the proposed structure is slightly modified. in this paper, a model consisting of spe and a single cnn is proposed. a single cnn consists of three layers: convolutional layer, pooling layer (max and average), and fully-connected layer with softmax. each component is described in detail in the rest of this section. ## results and discussion first, we check the parameter size and training time. when measuring the training time, all models set the mini-batch size to 5. the results are shown in each model, except the proposed model, performs fine-tuning in training. the size of the parameters, including this, is shown in the training times are reduced by up to 90% based on the imdb with the large-scale dataset, additionally, we confirmed spe's usefulness for word vectors represented based on simple bagof-words, as shown in ## conclusion in this paper, we propose a single convolutional neural network to which sinusoidal positional encoding is applied. besides, we describe an experiment to confirm the usefulness of sinusoidal positional encoding in the cnn-based approach. this model shows excellent performance despite being lightweight in text classification. although this model reduced the parameter size by at least 67%, it was possible to confirm a similar performance to the previous model. besides, a similar effect could be confirmed without finetuning in the cnn-based approach. in the future, we intend to study a method that can easily learn the position information of words in a manner other than spe. through this, we want to develop a more straightforward and more powerful model that can show both the timeadvantageous cnn and the sequential learning power of rnn.