Analysis of Resource-efficient Predictive Models for Natural Language Processing (2020)

Abstract: In this paper, we presented an analyses of the resource efficient predictive models, namely Bonsai, Binary Neighbor Compression(BNC), ProtoNN, Random Forest, Naive Bayes and Support vector machine(SVM), in the machine learning field for resource constraint devices. These models try to minimize resource requirements like RAM and storage without hurting the accuracy much. We utilized these models on multiple benchmark natural language processing tasks, which were sentimental analysis, spam message detection, emotion analysis and fake news classification. The experiment results shows that the tree-based algorithm, Bonsai, surpassed the rest of the machine learning algorithms by achieve higher accuracy scores while having significantly lower memory usage.

## introduction in last few years, large pretrained language models have gained a lot of popularity. these models were able to achieve state-of-the-art performance on various natural language processing tasks. but due to the higher resource requirement such as time and computation power, researchers have shifted their focus on developing more efficient and sustainable language models. devices like arduino board, atmega328 etc. which are essential in iot infrastructures like health care, smart grids, wearables, etc, have very limited computational resources. therefore, mostly these devices transfer data to cloud to extract some information and are dependent on them. these devices need models that can run without depending on cloud computing since cloud connectivity is not present everywhere, in a network data security can be compromised, it takes time to compute and transfer data which might not be good for real time analysis. so we need models that can run locally and need limited resources as well as do not hamper the accuracy of the task. there have been many advances in this field like reducing the prediction cost of knn with prototype based methods like protonn @xcite and bnc(binary neighbour compression) @xcite where you learn prototypes to reduce model size, snc(stochastic neighbour compression) @xcite where you learn very small synthetic dataset to perform knn, tree based methods like bonsai tree where under constraints model learns non-linear decision rules at each node @xcite , pruning the random forests based on resource constraints @xcite . in this paper, we present an analysis of various resource efficient machine learning algorithm for performing nlp tasks, such as, sentiment and emotion classification, fake news and spam message detection. we used six models, namely, bonsai, binary neighbor compression(bnc), protonn, random forest, naive bayes and support vector machine(svm) and reported their performance accuracy and memory usage for each task. we observed that the bonsai model performed the best by achieving significantly higher accuracy scores than other models at the cost of minimum memory usage. we believe that our generated insights would be very useful in designing and developing iot for nlp-based application purposes. ## methods in this section, we discussed about the various models used in our analysis. ## dataset we have used four textual datasets(see sentiment140: this contains 1.6m tweets text data extracted using twitter api. each tweet has been annotated with labels neutral, negative, and positive to express their sentiment. ## experiment we experimented with six machine learning models on four benchmark natural language dataset for different tasks. we used tf-idf as our text-features conversions, where we randomly selected the number of features to be considered while conversion. for each of the dataset, as seen in the figure 1 , we used a specific number of features. we fixed the number of features so that we can evenly compare and evaluate the machine learning models. for sms-spam dataset, we used a subset consisting of 5120 instances of dataset with 1226 features. for fake-news data, we used 2000 instances of dataset with 756 features. for sentiment-140 dataset, we used 4000 instances of the total dataset with 1362 features. for the emotion in text dataset, we used 3896 instances of dataset with 1542 feature count. each dataset was splitted into train and validation dataset with a ration of 80/20. the hyperparameter setting of all the models was done based on their best performance on the validation dataset. we used adam (kingma and ba, 2014) and gradient descent optimization for the models. ## conclusion in this paper, we presented a comparative analysis of various machine learning model for performing nlp tasks. we investigated the models performance based on accuracy achieved and memory required for performing a nlp task. we found that bonsai model was able to surpass other models with higher accuracy and lesser memory consumption. we also conclude that these models can be trained on a laptop and can be transferred to iot devices satisfying resource requirements of model. through our work, we aim at contributing towards the goal of sustainable nlp by developing more resource efficient nlp methods.