Task-specific Compression for Multi-task Language Models using Attribution-based Pruning (2023)

Abstract: Multi-task language models show outstanding performance for various natural language understanding tasks with only a single model. However, these language models inevitably utilize an unnecessarily large number of model parameters, even when used only for a specific task. In this paper, we propose a novel training-free compression method for multi-task language models using pruning method. Specifically, we use an attribution method to determine which neurons are essential for performing a specific task. We task-specifically prune unimportant neurons and leave only task-specific parameters. Furthermore, we extend our method to be applicable in both low-resource and unsupervised settings. Since our compression method is training-free, it uses little computing resources and does not update the pre-trained parameters of language models, reducing storage space usage. Experimental results on the six widely-used datasets show that our proposed pruning method significantly outperforms baseline pruning methods. In addition, we demonstrate that our method preserves performance even in an unseen domain setting.

## introduction various pre-trained language models with largescale data and parameters have emerged @xcite @xcite . specifically, pre-trained language models like t5 @xcite and gpt-3 @xcite have shown outstanding performance on many natural language understanding tasks. these language models can perform various tasks with a single model by treating every text processing problem as a text generation problem. however, these language models may utilize unnecessary large-scale model parameters even when performing only a specific task. previous works have introduced various compression methods for language models such as pruning @xcite @xcite , knowledge distillation @xcite @xcite , quantization @xcite , and low-rank factorization @xcite . however, these studies have (1) not compressed the language models taskspecifically or (2) demanded an additional training process like the case of knowledge distillation. this additional training process requires excessive computing resources and a massive training dataset. furthermore, this training process can destroy inherent pre-trained knowledge in language models since it updates the model's pre-trained parameters @xcite . due to the catastrophic forgetting @xcite caused by pre-trained knowledge destruction, models which are compressed and trained for a specific task, tend to show degraded performance on solving other pre-trained tasks @xcite . also, additional memory space is required to store the trained parameters separately. in this paper, we propose a novel training-free attribution-based task-specific pruning method that enables more efficient compression and inference by extracting only task-specific parameters from multi-task language models. we can determine which neurons are essential to derive a specific output for each neural network layer by using attribution so that we can extract only task-specific parameters from the entire model, as shown in figure 1 . we can efficiently process input data while preserving the model's task performance by selecting only the important neurons determined by the attribution method. furthermore, we extend our method to be applicable in two challenging scenarios: lowresource and unsupervised scenarios. the former alleviates insufficient labeled data situations, and the latter handles settings when labels are unavailable. both methods can relieve the cost of obtaining labeled datasets, which requires excessive human resources and is time-consuming. especially under the low-resource setting, our attribution-based task-"positive" "not equivalent" "entailment" "sst2 sentence: based on a true and historically significant story." "mrpc sentence1: security lights have also been installed and police have swept the grounds for booby traps. sentence2: security lights have also been installed on a barn near the front gate." "rte sentence1: lin piao, after all, was the creator of mao's "little red book" of quotations. sentence2: lin piao wrote the "little red book"." specific pruning requires only a single forward and backward propagation computation for few-shot data samples (e.g., only ten samples) to derive attribution of each neuron. since this pruning process does not update the model's parameters, it does not destroy the pre-trained knowledge of the language models. therefore, it is irrelevant to the various disadvantages that arise during an additional training process. since our method is model-agnostic, it can be applied to any neural network model broadly and generally. even we can use it to extract only task-specific knowledge after other compression methods are applied. experimental results on the six widely-used natural language understanding tasks show that our proposed method significantly outperforms baseline training-free pruning methods. furthermore, we demonstrate that our method shows robust performance in both low-resource and unsupervised settings. also, we reveal that our proposed method shows outstanding knowledge preservation even for an unseen related domain, which suggests that our method can preserve task-specific knowledge effectively. we additionally investigate to offer a guideline for our task-specific compression method by analyzing which types of layers are significant for processing task-specific knowledge. ## methodologies in this section, we describe our attribution-based pruning method for extracting only the taskspecific knowledge from a multi-task language model t5 @xcite , where attribution is obtained using gradient information. furthermore, we extend our method to low-resource and unsupervised settings to alleviate insufficient labeled data situations. we select t5 because it is a multi-task solving model and can be used in any natural language understanding setting by treating every text processing problem as a text generation problem. for our problem setting, suppose we have input text @xmath0, ..., x n } and output text @xmath1, ..., y m } mapped as (x, y) ∈ d, where each text corresponds to a sequence of tokens, and an input text contains a prefix task description. we can represent a standard conditional language modeling objective to maximize the following likelihood: ## conclusion this paper proposes a novel training-free attribution-based task-specific knowledge extraction method for multi-task language models. specifically, we use attribution to determine which neurons are important to derive a specific output for each task. then, we prune task-specifically unimportant neurons to extract only task-specific knowledge from the entire model. we further propose a method for computing attributions in low-resource and unsupervised settings. we demonstrate that our method outperforms the other pruning methods on the widely used text datasets. in addition, we examine that our taskspecific language model pruning method shows outstanding performance in the unseen domain, especially when the unseen domain is related to the dataset used to configure the compressed version. our compression method does not update the pre-trained parameters of the language models, which enables efficient on-demand compression and inference. also, our proposed method is valuable because it can be universally applied to any neural network-based model architecture.