Does local pruning offer task-specific models to learn effectively ? (2021)

Abstract: The need to deploy large-scale pre-trained models on edge devices under limited computational resources has led to substantial research to compress these large models. However, less attention has been given to compress the task-specific models. In this work, we investigate the different methods of unstructured pruning on task-specific models for Aspect-based Sentiment Analysis (ABSA) tasks. Specifically, we analyze differences in the learning dynamics of pruned models by using the standard pruning techniques to achieve high-performing sparse networks. We develop a hypothesis to demonstrate the effectiveness of local pruning over global pruning considering a simple CNN model. Later, we utilize the hypothesis to demonstrate the efficacy of the pruned state-of-the-art model compared to the over-parameterized state-of-the-art model under two settings, the first considering the baselines for the same task used for generating the hypothesis, i.e., aspect extraction and the second considering a different task, i.e., sentiment analysis. We also provide discussion related to the generalization of the pruning hypothesis.

## introduction significant progress in deep neural networks (dnn) over the last decade @xcite and access to unlimited online and offline data has revolutionized the research in natural language processing (nlp). neural-based language models (lms) @xcite can utilize large volumes of data and discover patterns that can be used to facilitate various downstream tasks @xcite @xcite @xcite . however, the development and deployment of such lms require extensive resources that amplify the costs in industry settings and questions the easy deployment of such models onto low resource capable embedded devices such as mobile phones @xcite . for instance, pre-trained transformer-based lms such as bert @xcite have demonstrated state-of-the-art results for various applications such as machine reading comprehension, information retrieval, and question answering by extracting contextualized word embedding or fine-tuning bert for specific functionality @xcite @xcite . however, these models are over-parametrized and thus are memory hungry and time-intensive to deploy on resource-constrained devices. therefore, it is crucial to develop energyefficient and cost-effective models for use in production. applications in the real world are task-oriented, with a demand for resource-efficient models. so, the models are required to use fewer parameters. given the need to build smaller task-specific architectures to save memory footprint and computational burden @xcite , one popular solution is pruning, a wellvetted topic in computer vision. pruning @xcite is a compression technique that systematically removes less significant parameters from an existing network to produce a smaller compressed model with similar performance comparing the larger model. the evolution of dnn has lead to the rise of research in pruning, even though the concept of pruning has existed for a long time. abundant research has been conducted on compressing deep learning-based architectures for computer vision tasks @xcite ; however, few works have been proposed for compressing the task-specific models in nlp @xcite . in this work, we aim to sparsify the models for absa tasks. absa aims to capture the opinion of the reviewer towards specific aspect in a review. in product-based reviews (reviews for restaurants, websites, etc.), the aspect term describes the attribute of a product, and the opinion term captures the sentiment expressed towards the aspect term. an example of the product-based review is depicted in figure 1 . in the review "the food is great however the service is poor." the aspect terms are food and service and the opinion terms are great and poor. the sentiment captured by the opinion terms is positive and negative respectively. aspect extraction is considered as a sequence labeling task to consider the span of aspect terms. each term in the span is assigned a label following bio scheme @xcite where b and i indicate the beginning and inside of the span, respectively, and o indicates outside of the span. a complex model slows down the speed of inference, and thus a simple model is always preferred over these highly sophisticated architectures @xcite . to this end, we propose pruning a simple convolutional neural network (cnn) (le-cun et al.; @xcite , trained on general embedding and show that it produces promising results for absa tasks. in this paper, we perform an in-depth analysis of local and global pruning to sparsify architectures needed for real-world applications and empirically demonstrate that local pruning offers desirable sparsity with less compromising in performance, inferring that it is more practical in addressing real-world tasks. our contributions are as follows: the rest of the paper is organized as follows: in section 2, we provide an overview of related works in the field. in section 3, we discuss the methodology proposed. section 4 details the experimental setup, datasets, and results of the experiments. section 5 concludes the paper and discusses the possible extensions of this work. ## methodology in this section, we introduce the methodology adopted for this work. we employ magnitudebased unstructured pruning where individual connections are detached based on the magnitude (l 1 ) of synaptic weights @xcite and then use it to implement both local and global pruning. in unstructured pruning, @xcite , individual connections between neurons or filters of adjacent filters are detached from the network, whereas in structured pruning @xcite , the entire neurons or filters are detached from the network. in local pruning, @xcite , a substantial percentage of connections are detached by contrasting each connection to the other connections in the layer, while in global pruning @xcite , all parameters are put together across all the different layers, and then a global percentage of them are taken to prune. we have used unstructured pruning over structured pruning considering the in-feasibility of structured pruning due to ample search space for pruning rates per layer @xcite and for causing large accuracy loss also @xcite . algorithm 1 shows the proposed pruning algorithm where f (x; w ) refers to the neural network model, which is a collection of nested functions parameterized by weight w . w is the updated weight, m denotes the mask, e represents the number of training epochs, and x is the training dataset. in the algorithm, we prune the language model using gradual pruning @xcite . the result after training is the pruned model with desired sparsity. during the training phase, we first perform weight updates to obtain w , later l1norm is applied on the updated weights to remove the least essential weights resulting in an updated binary mask m that affixes a definite number of parameters to 0. the generated pruned model is f (x; m w ), and @xmath0, 1} |w | and is the element-wise product operator which helps to generate pruned weights. this enables the masked parameters to reactivate during training based on gradient updates. at last, we apply a gradual sparsification schedule with sorting-based weights to achieve desirable sparsification. to show the effectiveness of the proposed hy-pothesis, we consider two cnn-based baselines with 4 and 6 convolutional layers respectively working on aspect extraction (ae) task. to study the possibility of pruning hypothesis generalization, we further consider two neural network-based architectures proposed by xu et al. @xcite and li et al. @xcite . the architectures proposed in these works use different language models thus enabling us to perform meta-analysis for our hypothesis. xu et al. (xu et al., 2018) uses a cnn-based model for aspect term extraction by employing double embeddings (domain + general) whereas li et al. (li et al., 2019) uses five different versions of bert-based models to perform ae and sentiment analysis of the extracted aspect terms. the code is implemented in pytorch, and the code is available at the url foot_0 . ## experiments the effectiveness of the inferred hypothesis is tested considering two settings. in the first setting, we consider the baselines for the same task of aspect extraction, which is used to generate the hypothesis. in the second setting, we consider the baselines for a different sentiment analysis task to validate the generalization of the hypothesis. we use fasttext @xcite for the general-purpose embedding. ## conclusion and future works this paper considers standard pruning techniques for compressing the model for two sub-tasks under absa tasks. we propose our hypothesis stating that local pruning is more effective than global pruning for aspect extraction. we empirically then demonstrate the validity of our hypothesis on two benchmark datasets for de-cnn and show that pruned models can achieve comparable perfor0.0 0.2 0.4 0.6 0.8 weights pruning 0.0 0.2 0.4 0.6 0.8 avg-f1-score laptop-local laptop-global restaurant-local restaurant-global figure 7: performance of de-cnn on test set of semeval-14 laptop and semeval-16 restaurant on applying local and global pruning. mance comparing the original models. we further performed experiments on bert-based models to 0.0 0.2 0.4 0.6 0.8 weights pruning 0.0 0.1 0.2 0.3 0.4 0.5 0.6 avg-f1-score bert-linear-local bert-linear-global bert-san-local bert-san-global bert-tfm-local bert-tfm-global bert-gru-local bert-gru-global bert-crf-local bert-crf-global figure 9: performance of bert (linear, crf, tfm, gru, san) on test set of semeval-14 laptop on applying local and global pruning. embedding layer's name parameters percentage position ids 511 0.0005% word embedding 23,440,896 21.4012% position embedding 393,216 0.3591% token type embedding 1,536 0.0014% figure 10: details of bert's crucial embedding layers.