Adapter Pruning using Tropical Characterization (2023)

Abstract: Adapters are widely popular parameter-efficient transfer learning approaches in natural language processing that insert trainable modules in between layers of a pre-trained language model. Apart from several heuristics, however, there has been a lack of studies analyzing the optimal number of adapter parameters needed for downstream applications. Thus, we propose an adapter pruning approach by studying the tropical characteristics of trainable modules. We cast it as an optimization problem that aims to prune parameters from the adapter layers without changing the orientation of underlying tropical hypersurfaces. Our experiments on five NLP datasets show that tropical geometry tends to identify more relevant parameters to prune when compared with the magnitude-based baseline, while a combined approach works best across the tasks.

## introduction with the increase in network sizes, we are observing an ever-increasing space and computational demand for models needed to solve a given task. to tackle this, model compression @xcite techniques are becoming continuously popular which retain the most important learning from the full model while reducing the size of the network either by pruning or distillation. transfer learning approaches, such as adapters @xcite , are a parameter-efficient alternative to full model fine-tuning which obviates the need to maintain a task-specific copy of the base language model (lm). adapters insert simple modules in between layers of an lm to adapt the pre-trained representation for a given downstream nlp task. however, there is a lack of research in pruning adapter modules to further enhance their parameter efficiency. we hypothesize that adapter weights can be pruned significantly by not compromising the performance observed with unpruned states, this motivates the proposed approach. in this work, we propose a novel approach to pruning adapter layers without any iterative finetuning of the model parameters on downstream tasks. using tropical algebra, we study the (duals of) hypersurfaces generated by adapter modules in the high-dimensional space. as a pruning objective, we aim to minimize the magnitude of adapter weights while constraining the change in hypersurface geometry to be small. related works include adapters pruning using lottery ticket hypothesis @xcite ) that performs iterative pruninga few gradient steps, prune, and reset the parameters to initial weights. @xcite drops adapter from lower transformer layers. while these works are interesting, we provide a more concrete angle to prune adapter layers-prune by preserving the hypersurface geometry. we extend an insightful analysis of tropical geometry of neural networks @xcite to adapters. it consists of down-projection w @xmath0, upprojection w @xmath1, a relu activation function f (•), where typically @xmath2. ## tropical adapter pruning given a frozen language model adapted to a specific task using adapter layers, we divide our approach into two steps: 1) finding adapter weights p t that are crucial to preserving the tropical adapter hypersurface by solving a simple optimization problem; 2) pruning of adapter weights with least magnitudes that do not lie in p t . next, we describe step-1 which is core to the pruning method: a bottleneck adapter block can be expressed by f (x) = b max{ax, 0}. since f (x) in itself is not a tropical polynomial and thus does not form a tropical surface, we rewrite it in terms of the difference between two tropical polynomials f (x) = h(x)-q(x), following the analysis of tropical rational function by @xcite . thus we focus on a relatively lenient problem i.e. identifying weights that preserve tropical hypersurfaces defined by h(x) and q(x). let h(x) and q(x) be the respective hypersurfaces, one can choose a sparse set of â, b that belongs to the set of matrices obtained by solving the following optimization problem where d(•) defines the distance between two geometric objects; ĥ and q are hypersurfaces obtained by substituting a and b with â and b in f (x). in place of preserving the orientation of h(x) and q(x), we aim to preserve the orientation of their respective dual objects denoted by δ(h(x)) and δ(q(x)). thus, without the loss of generality, we assume downprojection is bias-free foot_1 , δ(•) can be expressed in terms of generator matrices g of zonotopes obtained from a and b. to find sparse â, b, we introduce sparse regularization terms in the optimization function. thus, finding adapter weights that preserve the hypersurface geometry can be cast as the following optimization problem: we provide a derivation of the above function in appendix b. it is important to note that in the pruning phase, we do not iteratively fine-tune adapter or lm parameters on the downstream task. given an adapter module, algorithm1 finds the minimizers â and b by performing gradient descent-based updates 3 over two loss terms expressed in terms of generators g 1 and g 2 . t , r denote the maximum gradient steps and the number of rows in a and columns in b. η ∈ r + is step size and λ 1 , λ 2 ∈ r + indicate the importance of pruning over the shift in generators. we employ layer-wise pruning of the network without any iterative fine-tuning on downstream tasks. we find p% parameters with the smallest magnitude in {a, b} and { â, b} separately, denoted by p s and p t . we denote tropical adapter pruning by tropical that prunes only those parameters in p t which are also present in the set p s . the final percentage of pruned parameters decreases to p%. we compare the approach with the baseline that prunes p% of the smallest magnitude parameters from the layer. we denote this setting by standard. combined chooses one of tropical or standard whichever gives better results on the development set. we omit the comparison with adapterdrop 3 not to confuse with gradient descent used to learn model parameters. here, it is used to solve the optimization problem in equation (2). method as even at 50% pruning, the method shows a significant drop in the performance. standard inherently tests the validity of magnitude-based pruning via lottery ticket hypothesis @xcite but without iterative retraining of adapter parameters. we do not iteratively fine-tune adapter parameters on the downstream task. the proposed method is agnostic to downstream tasks, models, and the learning algorithm used to train it. thus, the framework is related to but not directly comparable to model l 0 sparsification @xcite and low-rank compression @xcite . ## experiments we set up a roberta-base @xcite with one adapter module inserted in each layer after add and layer norm sub-layer. we follow the adapter configuration from @xcite . for pruning analysis, we consider three tasks-emotion recognition in conversations (erc), natural language inference (nli), and text classification (tc). for erc, we use meld, the task is to classify the emotion of an utterance given past utterances. keeping the current utterance first, we append the past seven utterances in reverse order @xcite . for nli, we use snli dataset @xcite . we append the premise and hypothesis separated by the special @xmath3. for tc task, we use three datasets: imdb @xcite , rotten tomatoes rt @xcite , and trec @xcite . separately, we pretrain adapters on downstream tasks with batch size 32, lr of 0.001, and 1000 steps with evaluation at every 100 steps using a development set. the evaluation metric for erc is macro f1 and accuracy for all the other tasks. we set pruning percentage @xmath4, 96%, . . . , 2%}. next, we study tropical pruning in different scenarios-class-bind, class-uniform, and nodewise @xcite . in class-blind (cb), all the parameters of adapters are considered for pruning p% of the smallest magnitude weights and biases. in class-uniform (cu), we prune p% of the smallest magnitude parameters of each adapter layer separately. we also refer to it as layer-wise pruning. in node-wise pruning, we prune p% of the node-wise as shown in figure 1 shows the objective function in equation (2) quickly converges to the minimum. this observation corroborates the claim of convexity by @xcite . the plot in figure 2 shows the change in zonotope structure before and after optimization on snli. the black polytope is obtained from generators a, b and the red polytope shows the polytope obtained after optimization, i.e., zonotope obtained from â. b. we observe the optimization preserves the geometry of zonotopes while enforcing the rows of the down-projection matrices to be as much sparse as possible, i.e., many points in the zonotope come close to zero, keeping necessary boundary points to preserve the geometry. these zonotopes are dual to adapter hypersurfaces, thus preserving one structure enforces the other's orientation to remain preserved. hence, one can prune adapters yet maintain their characteristic properties. ## conclusion we proposed a novel approach for adapter pruning by studying their tropical characteristics. we formulated it as an optimization problem that aims to identify row-sparse projection matrices while minimizing the distance between tropical hypersurfaces before and after pruning. we demonstrated the advantages of tropical characterization on five nlp datasets reformulated as classification.