S PECIALIZED T RANSFORMERS : F ASTER , S MALLER AND MORE A CCURATE NLP M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Transformers have greatly advanced the state-of-the-art in Natural Language Processing (NLP) in recent years, but are especially demanding in terms of their computation and storage requirements.
Transformers are ﬁrst pre-trained on a large dataset, and subsequently ﬁne-tuned for different downstream tasks.
We observe that this design process leads to models that are not only over-parameterized for downstream tasks, but also contain elements that adversely impact accuracy of the downstream tasks.
We propose a Specialization framework to create optimized transformer models for a given downstream task.
Our framework systematically uses accuracy-driven pruning, i.e., it identiﬁes and prunes parts of the ﬁne-tuned
Transformer that hinder performance on the downstream task.
We also replace the dense soft-attention in selected layers with sparse hard-attention to help the model focus on the relevant parts of the input.
In effect, our framework leads to models that are not only faster and smaller, but also more accurate .
The large number of parameters contained in Transformers presents a challenge in the form of a large pruning design space.
Further, the traditional iterative prune-retrain approach is not applicable to Transformers, since the ﬁne-tuning data is often very small and re-training quickly leads to overﬁtting.
To address these challenges, we propose a hierarchical, re-training-free pruning method with model- and taskspeciﬁc heuristics.
Our experiments on GLUE and SQUAD show that Specialized models are consistently more accurate (by up to 4.5%), while also being up to 2.5 × faster and up to 3.2 × smaller than the conventional ﬁne-tuned models.
In addition, we demonstrate that Specialization can be combined with previous efforts such as distillation or quantization to achieve further beneﬁts.
For example,
Specialized Q8BERT and DistilBERT models exceed the performance of BERTBase, while being up to 3.7 × faster and up to 12.1 × smaller.
I NTRODUCTION
Transformers models such as BERT (Devlin et al., 2019),
GPT-
(Radford et al., 2019) and GPT3 (Brown et al., 2020) have revolutionized the ﬁeld of Natural Language Processing (NLP), greatly advancing the state-of-the-art in many NLP tasks.
Models that achieve good performance on these tasks are of high practical signiﬁcance, ﬁnding their place in commercial applications such as social media monitoring (sentiment analysis), AI chat assistants (question answering), automated summarization tools (analyzing sentence similarity), etc.
Therefore, there is a strong interest in creating more accurate and efﬁcient Transformer models for these tasks.
Transformers are ﬁrst pre-trained on very large datasets, and subsequently ﬁne-tuned for different downstream tasks.
However, the current method of pre-training and ﬁne-tuning Transformers has two major drawbacks.
First, ﬁne-tuning the pre-trained Transformers leads to models that are highly over-parameterized models for the downstream tasks, especially since many of these tasks have very limited training data.
This could lead to unstable models (Dodge et al., 2020) with sub-optimal generalization ability (Michel et al., 2019).
Second, these large ﬁne-tuned models present high computation and storage requirements for inference.
This problem is further exacerbated by the trend towards larger and more accurate models over time.
For instance, increasing the number of parameters from 1.5B to 175B enabled a reduction in perplexity for Language Modelling (on the
Penn Treebank dataset) from 35.8 in GPT-2 (2019) to 20.5 in GPT-3 (2020).
In this work, we address both these challenges, taking advantage of the over-parameterized nature of pre-trained models to create individually Specialized models for the different downstream tasks that are smaller, faster and more accurate than the conventional ﬁne-tuned models.
Prior research efforts have explored approximation techniques, such as quantization, pruning and distillation, for improving the inference efﬁciency of various classes of neural networks.
However, these techniques invariably involve a trade-off between accuracy and efﬁciency.
In addition, the vast majority of these techniques require re-training or additional ﬁne-tuning.
This becomes especially problematic for Transformers, since these large models require signiﬁcant time and energy to train and ﬁne-tune.
Fine-tuning is usually performed on a very small data set, and as a result, conventional iterative prune-and-retrain methods quickly end up overﬁtting when applied to Transformer models.
In contrast, our Specialization framework utilizes the unique characteristics of Transformer training and deployment to enable substantial gain in both accuracy as well as efﬁciency.
Our framework also does not require any additional re-training or ﬁne-tuning, and can be applied in a plug-and-play manner to any Transformer model that is ﬁne-tuned for any downstream task.
During pre-training, Transformers capture rich linguistic knowledge and gain a deep understanding of the structure of the target language.
Fine-tuning reﬁnes
this knowledge for a speciﬁc downstream task by training a task-speciﬁc ﬁnal layer.
We observe that, due to the nature of their design process,
Transformer models are not only over-parameterized but also contain parts that are, in fact, harmful to performance of downstream tasks.
In order to exploit this observation in a principled manner, we introduce a framework to identify and prune the harmful elements of a Transformer (parameters, grouped at different levels of granularity i.e. , self-attention blocks, feed-forward neural network blocks, attention heads and neurons), with the goal of maximizing accuracy on the downstream task.
In contrast with prior pruning methods that prune elements with little-or-no impact on the network output, the proposed method prunes elements that have a considerable impact on the output, leading to the highest positive impact on accuracy.
In order to reduce the large pruning space, we analyze the different elements of the ﬁne-tuned Transformer in a hierarchical manner, starting with entire self-attention or feed-forward neural network blocks, followed by attention heads, and neurons, and prune the harmful elements.
The core of the Transformer is self-attention, where each token in the input builds its representation based on the extent of attention it places on all the other tokens.
However, we observe that in some cases, restricting the attention span of each token to only focus on the relevant tokens (in certain layers) leads to better information ﬂow inside the model.
Hence, our framework is also equipped with the ability to identify the appropriate layers and replace the ”soft” self-attention with ”hard” self-attention in these layers.
We also introduce Transformer-speciﬁc heuristics to minimize the run-time of our framework, thereby enabling it to scale to large Transformer models.
We summarize our main contributions as follows:
• We introduce a Specialization framework that optimizes Transformer models for speciﬁc downstream tasks through the use of accuracy-driven pruning and selective hard attention.
• We incorporate multiple heuristics in the framework, such as hierarchical processing, model-driven insights, and run-time based ordering of elements, in order to minimize the overheads.
• We propose a signiﬁcance analysis technique to identify the importance of each element of the ﬁne-tuned Transformer for a given downstream task.
We use this technique to prune elements that are harmful to performance on the downstream task.
• We propose the selective replacement of the ”soft” self-attention with hard attention in the appropriate layers, helping the model focus only on the relevant parts on the input to build better representations.
• Across a suite of different Transformer networks, we demonstrate that Specialized models are consistently more accurate and stable, while also being signiﬁcantly faster and smaller than their conventional ﬁne-tuned non-Specialized counterparts.
R ELATED W ORK
Task-agnostic optimizations. Given the effectiveness and popularity of Transformer models, several techniques have been proposed to overcome their computational and memory challenges, and to accelerate inference using these models.
A vast majority of these works introduce task-agnostic optimizations, using popular approximation techniques such as knowledge distillation (Sanh et al., 2019;
Sun et al., 2020; Wang et al., 2020), early exit/ depth modulation (Elbayad et al., 2020; Xin et al., 2020; Zhou et al., 2020), quantization (Zafrir et al., 2019), attention head pruning (Zhang et al., 2020) and parameter sharing (Lan et al., 2020).
In addition, Fan et al. (2020) randomly drop layers during pre-training, thereby enabling their dropping during inference; Khetan & Karnin (2020) learn the optimal sizes of the BERT elements during pre-training, and Wu et al. (2020) use Long-Short
Range Attention to speed up the self-attention operation.
Using DistilBERT and Q8BERT as examples, we demonstrate that our techniques are complementary to these works, and can be applied while ﬁne-tuning for a speciﬁc task to create accurate ultra-efﬁcient models for speciﬁc downstream tasks. Task-speciﬁc optimizations.
Hou et al. (2020), Shen et al. (2020), Jiao et al. (2019), Wang et al.
(2020), Lagunas et al. (2021) and Sajjad et al. (2020) introduce task-speciﬁc optimizations, but the gain in efﬁciency comes at the cost of degradation in accuracy on the downstream task.
In contrast, our framework improves both accuracy as well as efﬁciency, and hence appeals to a wider range of users.
We also demonstrate that task-speciﬁc optimizations are more effective when applied to Specialized models compared to conventional ﬁne-tuned models.
Pruning techniques. Finally, Structured Pruning has been applied to various classes of neural networks (Anwar et al., 2017; Molchanov et al., 2017), and greedy pruning strategies have also been explored to identify weights and parameters that the output is least sensitive to (Zhuang et al., 2018;
Ye et al., 2020).
In contrast, our method is designed to identify and prune parameters that have most detrimental effect on the output.
M ETHODOLOGY TO S PECIALIZE T RANSFORMERS
We propose a framework for producing Specialized Transformer models that are optimized for a speciﬁc downstream task, illustrated in Algorithm 1.
Our framework performs two main optimizations: (1) It identiﬁes and prunes elements that hinder performance on the downstream task at hand.
(2) It selectively replaces soft self-attention with hard self-attention to help the model focus only on the relevant parts of the input.
3. A CCURACY - DRIVEN PRUNING
The problem of identifying an optimal set of elements to prune is challenging, and this is especially true for Transformers.
In order to optimize a given model, we would ideally want to characterize the signiﬁcance of each and every parameter in the model, rank them in order of importance, and ﬁnally prune only the least signiﬁcant parameters.
However, Transformers have billions of parameters, making this process computationally infeasible.
In addition, previously proposed techniques that can efﬁciently estimate the importance of each parameter, such as using Taylor expansion, are not applicable.
This is because the { approximate, ﬁne-tune, approximate } cycle does not work for
Transformers during ﬁne-tuning, since they very quickly overﬁt the limited training data for the downstream tasks (usually within 5 epochs).
We address both these issues through the use of a hierarchical greedy algorithm that does not require any additional training or ﬁne-tuning.
To determine the signiﬁcance of each Transformer element, we ﬁrst ﬁne-tune the original Transformer model for the given downstream task to obtain the baseline loss.
Then, for the element under consideration in each iteration of the framework, we compute the loss of the current Transformer model with the element removed.
We prune the element under consideration if the validation loss when it is removed is less than the minimum loss seen thus far during the optimization process, since the goal is to ﬁnd a model with minimum loss.
Also, in order to prevent overﬁtting to the validation set, we introduce a generalization constraint in addition to the aforementioned loss condition.
This constraint ensures that an element is pruned only if it decreases (or retains) the loss of at least a certain number ( N ) of samples in the validation set over the current best solution (computed by num samples helped function in Alg. 1), where N is the number of elements in the validation dataset whose loss is less than the average loss of the misclassiﬁed samples (we consider samples whose loss is greater than the average loss of the misclassiﬁed samples to be outliers).
Therefore, elements are pruned only if a vast majority of the samples in the validation set beneﬁt from their removal, resulting in improved generalization performance. If the loss with the element removed is greater than the minimum loss
Algorithm
Input : Fine-tuned (for the given downstream task)
Transformer T, Validation set D
Output: Specialized Transformer for the given downstream task T
Function analyze element ( element E ) :
T pruned =
T − E
New Loss = Evaluate ( T pruned , D ) if New Loss <
Min Loss and num samples helped > N then
Min Loss =
New Loss
T = T pruned
Function num samples helped : computes the number of samples in the validation set that beneﬁt from the pruning of an element; must be > N for an element to be pruned
Baseline Loss = Evaluate (T,D)
Min Loss = Baseline Loss
Q = Order elements for inspection ( T, D ) for each layer L in T do
Replace soft self-attention in L with hard self-attention
New Loss = Evaluate ( T, D ) if New Loss <
Min Loss and num samples helped > N then
Min Loss =
New Loss else
Restore soft self-attention in L while Q is not empty
do
TrialElement = Q.pop() analyze element (TrialElement) if TrialElement has not been pruned from T and New Loss < Baseline Loss then if TrialElement is an attention block then for each attention head h in TrialElement do analyze element (h) else if TrialElement is a feed-forward block then for each neuron w in TrialElement do analyze element (w) return T seen so far but less than the baseline loss, we inspect the element at a ﬁner granularity, and prune only parts of the element that hinder performance (rather than pruning the entire element).
Hierarchical processing of elements.
It is computationally prohibitive to analyze every single parameter in large Transformers using the method described in Alg 1.
Since the framework iterates through the entries of the queue sequentially, its efﬁcacy is dependent on both the total number of elements under consideration, and the time required to analyze each element.
We take advantage of the inherently hierarchical structure of Transformers and consider the elements in a hierarchical manner, ordered by increasing granularity.
Speciﬁcally, we analyze entire feed-forward and selfattention blocks ﬁrst, and inspect them at ﬁner granularity (attention heads and neurons) only when required.
Through this ordering, we are able to quickly eliminate large numbers of parameters from further consideration.
In addition, due to the over-parameterized nature of Transformers, it is likely that time-consuming blocks are pruned from the Transformer earlier in the process, thereby speeding up future iterations of the framework.
For example, eliminating a single feed-forward block in the BERT-Base model removes 5.6% of all parameters under consideration, and speeds up future iterations by 1.15 × .
To further reduce the number of elements under consideration, we also dynamically remove elements if they are encompassed by a high-importance block.
For example, if a given self-attention block is determined to be of high importance (the validation loss with the block removed is greater than the baseline loss), we remove all heads within that block from further consideration.
Creating an ordered queue of elements for inspection.
Since our framework performs greedy pruning of highly over-parameterized models, it is essential to know where the harmful elements are likely to be.
Some elements may appear to be harmful for the downstream task (especially in early iterations of the framework), but this often ends up being an artifact of over-parameterization.
As a result, when elements are not carefully ordered for inspection, our framework lands in local minima of the validation loss function, leading to inefﬁcient models with sub-optimal generalization ability.
Our solution to this problem utilizes the unique linguistic properties captured by the different
Transformer layers (Jawahar et al., 2019) to guide the ordering of elements for inspection.
For example, it was found that BERT captures phrase-level information in the lower layers, mapping related tokens together.
The lower layers also capture surface features, while the middle layers capture syntactic features and higher layers capture semantic features.
It was also observed that
BERT requires deeper layers only when long-range dependency information is required.
Different tasks require different types of linguistic knowledge.
For example, sentiment analysis requires only local context, and long-range information often ends up confusing the model, since sentiments often change rapidly; it is also unlikely that syntactic and semantic information are needed.
Hence, we place the ﬁnal layer at the front of the queue, and work our way backwards towards the ﬁrst layer, since blocks in the ﬁnal layers are more likely to hinder performance on sentiment analysis.
This ordering of elements ensures that elements that are pruned early in our framework (when the model is most over-parameterized) do not lead the system into bad local minima.
3. S ELECTIVE USE OF HARD SELF - ATTENTION
In traditional Transformer architectures, the self-attention operation computes the attention scores of each token in the input sequence with all the other tokens.
These attention scores, after passing through the softmax operation, are used to build the new representation for the token based on the extent of attention it places on the other tokens.
The use of this ”soft” attention enables end-to-end training of the Transformer.
However, we observe that replacing the softmax operation on all attention scores with a softmax only on the attention scores above a certain threshold T a in selected layers after training and ﬁne-tuning is completed helps the Transformer focus only on the relevant parts of the input sequence (Fig. 1).
This leads to better information ﬂow inside the model, improving performance on many downstream tasks (especially those involving sequence classiﬁcation and question answering).
This also introduces a large amount of activation sparsity that can be exploited to analyze the information faster, and helps alleviate the critical memory bottleneck in Transformers.
T a is set to a fraction of the maximum attention score of each token, and tokens on which the selected token places signiﬁcantly less attention than it places on its most relevant token are considered irrelevant.
The exact value is tuned for the different downstream tasks (based on maximizing the validation loss), since we observe that different tasks require different attention spans for optimal performance.
We note that replacing soft attention with hard attention in all layers (especially the deeper layers) leads to loss of important information in the model and hence, loss of accuracy, necessitating its selective use.
In order to identify the layers that beneﬁt from hard attention, we replace the soft attention with hard attention in each layer (one by one), and inspect which type of attention leads to smaller validation loss.
E XPERIMENTS AND R ESULTS
We implement our techniques within Huggingface’s
Transformers library in PyTorch (Wolf et al., 2019).
We use Intel AI’s NLP Architect for experiments on Q8BERT.
The experiments were performed on a GeForce RTX 2080 Ti GPU with 11GB memory.
All results are reported on the test set, averaged across 10 runs with random seeds after 3 epochs of ﬁne-tuning, unless otherwise speciﬁed.
We randomly sample 15% of the training set with class balance, and use it as the validation set to guide the Specialization process .
We present results on the GLUE dev set in Appendix A.
Specialization leads to more accurate models that are also faster and smaller.
We present results on GLUE (Wang et al., 2019), a set of Language Understanding tasks, and SQUADv1.1 (Rajpurkar et al., 2016), a Question Answering task, in Table 1.
For GLUE, we present results on the test set using the GLUE evaluation server to obtain the scores (Xu et al., 2020), and for SQUAD, we present results on the dev set.
Specialized models are up to 4.5% more accurate, while also being up to 2.5x faster and up to 3.2x smaller than their baseline counterparts .
In addition, Specialization is able to obtain substantial improvements over Q8BERT-base (already 4x smaller than BERT-base due to the use of 8-bit integer quantization) and DistilBERT-base (60% faster and smaller than
BERT-base).
Specialized Q8BERT-base and DistilBERT-base models exceed the accuracy of nonSpecialized BERT-base models, while being up to 3.7x faster and 12.1x smaller than BERT-base.
Therefore, Specialization can be used in conjunction with other approaches that improve the efﬁciency of Transformers to create highly efﬁcient accurate models for different downstream tasks.
Specialization reduces sensitivity to random seed initialization.
Previous research (Dodge et al., 2020) has shown that the random seed (which determines the initialization of the task-speciﬁc layer and the order of training data for the downstream task) has a signiﬁcant impact on the quality of the ﬁne-tuned models.
When the amount of training data is small (which is the case with most downstream NLP tasks), this effect becomes more pronounced, evidenced by the fact that WNLI, which has the least amount of training samples (634 training samples), exhibits highest variance across runs (Table 2).
Therefore, in order to ﬁnd a model with high accuracy on a downstream task, multiple models need to be created (using different random seeds), and evaluated.
Here, we demonstrate that Specialized models exhibit signiﬁcantly less sensitivity to random seeds than their non-Specialized counterparts (Table 2), thereby greatly increasing the odds of ﬁnding “good” models in fewer iterations.
In particular, we ﬁnd that even if two models have vastly different accuracies on the validation set before Specialization, they converge to similar accuracies after Specialization.
This, in turn, reduces variance on the test set.
[Table]
Larger models Specialize better.
Current state-of-the-art Transformer networks, such as T5 (Raffel et al., 2019) and GPT-3 (Brown et al., 2020), have hundreds of billions of parameters.
Model sizes are also expected to grow further in the future as increasing the number of parameters has been shown to improve performance.
This makes it computationally challenging to train Transformers as well as perform inference using them.
Recent research (Li et al., 2020) has shown that larger models converge in a signiﬁcantly smaller number of training iterations than smaller models, and hence they train faster in spite of requiring more time per iteration.
However, larger models are signiﬁcantly slower than smaller models at inference time.
Here, we demonstrate that Specialized larger models achieve much higher accuracy while being comparable in speed to Specialized smaller models (Table 3).
In addition to having a higher accuracy ceiling, we also demonstrate that Specialized BERTLarge (pruned beyond the max accuracy point, by relaxing the constraints to trade-off accuracy for efﬁciency), is 1.5x faster than Specialized BERT-base at iso-accuracy.
This indicates that larger models capture greater linguistic knowledge, but also contain more redundant parameters for the different downstream tasks, further motivating the development of larger and better language models that can be Specialized for fast and accurate inference.
[Table]
Previously proposed techniques to improve inference efﬁciency are more effective when applied to Specialized models.
Specialization identiﬁes and prunes harmful elements of the Transformer for the downstream task at hand.
Techniques that prune elements that have minimal impact on the output are complementary to our techniques, and we demonstrate this using the popular Lottery
Ticket Hypothesis (Frankle & Carbin, 2019), which ﬁnds sparse sub-networks that can be trained to match the performance of the large network.
Lottery Ticket Hypothesis has been successfully applied to Transformers also (Prasanna et al., 2020; Liang et al., 2021; Chen et al., 2020), and using these techniques, we demonstrate that winning tickets of the Specialized model are consistently more accurate than the iso-efﬁcient winning tickets of the conventional ﬁne-tuned models (Figure 2).
Specialization provides insights into the working of Transformers.
We analyze which elements of the Transformer are pruned for different downstream tasks using different Transformer models (Fig. 3).
We ﬁnd that the differences in importance of elements are more pronounced across different tasks than across different models.
For example, for sentiment analysis, long-range dependency information is not required, and often ends up confusing the model.
Hence, for all models ﬁnetuned for sentiment analysis, we observe that components in later layers (closer to the output) are more likely to be pruned.
This is not the case with Language Modelling tasks (predicting masked words in text), where longer attention spans are required.
Across models, we only observe subtle differences.
For example, we ﬁnd that XLNet (auto-regressive) is able to learn important taskspeciﬁc information earlier than BERT (non auto-regressive), similar to the observation made in (Sajjad et al., 2020).
Hence, we are able to drop more components (in earlier layers) in XLNet than in BERT, leading to more efﬁcient models for inference.
In DistilBERT (a distilled model), we ﬁnd that there is a clear demarcation in linguistic knowledge across layers due to the reduced capacity of the model.
This is evidenced by the fact that elements in the top four layers are never pruned across all Language Understanding tasks, while the boundaries are more soft in the original models.
We also observe that Hard Attention is most useful in the lower layers, where phrase-level information is captured, for all the models and tasks.
As future work, our framework can be combined with previously proposed techniques (such as probing classiﬁers) to gain deeper understanding of the working of Transformers, especially at ﬁner levels of granularity.
Specialization using previously proposed importance estimation techniques leads to less accurate models.
L1-norm and magnitude based pruning techniques are designed to prune elements that have minimal impact on the output, and hence, they are not effective at identifying and pruning elements that have most detrimental impact on output.
Gradient-based methods such as Taylor expansion are not reliable for increasing accuracy on the downstream tasks, since Transformers cannot be repeatedly ﬁne-tuned to recover the accuracy losses from approximating the model (they very quickly overﬁt the limited training data for the downstream tasks).
In addition, while knowledge distillation is complementary to our method (as evidenced by our results on DistilBERT), we show that the accuracy gains from Specialization are signiﬁcantly higher than those from regularized knowledge distillation proposed in (Hou et al., 2020).
Super-tickets (Liang et al., 2021) demonstrates that accuracy can be improved by ﬁnding winning lottery tickets at low levels of pruning.
However, we ﬁnd that alleviating the over-parameterization issue by pruning elements that hinder performance (and the selective use of hard attention) leads to more accurate models than by pruning elements that have least impact on output in the original model (Table 4). Specialization incurs minimal overhead.
Our Specialization framework does not require any additional training or ﬁne-tuning iterations. It only requires multiple passes through a small validation set. Our hierarchical processing ensures that the number of elements to inspect is greatly reduced. In addition, each pass is expected to become progressively faster, since the framework potentially prunes an element in each iteration, and our ordering of elements ensures that large, time-consuming blocks are pruned early in the process, making future iterations faster.
Therefore, the overheads of
Specialization are negligible compared to the time required for re-training or additional ﬁne-tuning.
We ﬁnd that the average (for GLUE and SQUAD) wall-clock time for Specialization is only 7.
minutes for BERT-base and 18.3 minutes for BERT-large on a single GPU.
We also evaluate the heuristics used in our framework (Table 4).
When elements are not carefully ordered for inspection, we ﬁnd that elements that are pruned early in the Specialization process are pruned due to over-parameterization, and not because the linguistic knowledge contained in these elements is harmful/not useful for the task at hand.
This leads the system into a bad local minima, where the validation loss cannot be further reduced by pruning other elements.
We observe that starting at the layer granularity leads to worse models than starting at the block granularity.
This is because the attention and feed-forward blocks in each layer have vastly differently functionality, and hence, the effect of removing an attention block of relatively high signiﬁcance is often masked by removing the feed-forward block that greatly hinders performance in the same layer (and viceversa), causing the entire layer to be pruned.
This destabilizes the Specialization process, and also leads the system into bad local minima, thereby creating inferior models.
To drive home the fact that our greedy approach combined with a global error bound does not lead to inferior models, we also experiment with an adaptive loss threshold.
In particular, we use a very tight constraint when analyzing elements at coarse granularities, and relax the constraint as we move towards ﬁner granularities.
We again ﬁnd that there is negligible change in the quality of the ﬁnal model produced (the accuracy is slightly lower, possibly due to over-ﬁtting to the validation set), but the Specialization process is signiﬁcantly slower.
We hypothesize that a single global error bound is sufﬁcient because we order the elements in such a way that for the given task at hand, we intuitively expect that the elements at the head of the queue are likely to be removed using the linguistic knowledge in different layers.
[Table]
C ONCLUSION
We proposed a Specialization framework to optimize ﬁne-tuned Transformers for the different downstream tasks.
The framework identiﬁes elements that are hinder performance on the downstream task at hand, and prunes these elements.
We also demonstrated the advantage of selectively using hard self-attention in selected layers to improve information ﬂow.
Using this framework, we produced models that were guaranteeably more accurate, while also being up to faster and smaller.
A A DDITIONAL EXPERIMENTS AND STUDIES A. R ESULTS ON THE GLUE DEVELOPMENT SET .
We present additional results on the GLUE dev set using the same models used to present results on the test set in Table 5.
We use 15% of the training set as the validation set for Specialization.
Similar to results on the test set, Specialized models are consistently more accurate, faster and smaller than their conventional ﬁne-tuned counterparts.
In addition, Specialization is able to obtain substantial improvements over already optimized models (Q8BERT and DistilBERT).
A. S IGNIFICANCE OF ORDERING ELEMENTS FOR INSPECTION
We ﬁnd that the order in which elements of the ﬁne-tuned model are inspected during accuracydriven pruning has a large impact on the generalization performance of the Specialized model.
Prior research (Jawahar et al., 2019) has shown that the linguistic knowledge contained in different Transformer layers can be demarcated into three main functional regions: phrase-level information in bottom layers (the ﬁrst one-third of transformer layers closest to the input), semantic and syntactic information in the middle layers, and long-range dependency information in the top layers.
When elements are not carefully ordered for inspection, we ﬁnd that elements that are pruned early in the Specialization process are pruned due to over-parameterization, and not because the linguistic knowledge contained in these elements is harmful/not useful for the task at hand.
This leads the system into a bad local minima, where the validation loss cannot be further reduced by pruning other elements (Table 6).
The quality of the Specialized model is worst when the blocks containing the most important linguistic knowledge is inspected ﬁrst (bottom layers for sentiment analysis, see
Appendix A.3), since these blocks may get pruned simply to alleviate the over-parameterization.
In addition, we ﬁnd that the ordering of elements within the same functional block (for example, blocks in layer 10 before layer 11 or vice-versa), has negligible impact on the quality of the Specialized model, further demonstrating the existence of regions with different kinds of linguistic knowledge.
[Table]
A NALYZING THE LINGUISTIC KNOWLEDGE REQUIRED FOR DIFFERENT DOWNSTREAM TASKS
Different downstream tasks require different types of linguistic knowledge.
We ﬁnd that the most important functional block of the Transformer model for each task (corresponding to the most important kind of linguistic knowledge required to effectively solve the task) agrees with our intuition about the task.
For example, we expect sentiment analysis to require only local context, since longrange information often ends up confusing the model (sentiments often change rapidly); it is also unlikely that syntactic and semantic information are needed.
This is experimentally validated by the fact that inspecting the blocks in top layers, followed by the middle and ﬁnally the bottom layers leads to most accurate models on SST-2 (Table 7).
Similarly, we expect knowledge about language syntax and semantics to be most important for a task that tests the linguistic acceptability of a given sentence (CoLA).
We also ﬁnd that this holds across all Transformer architectures that we study in this work (BERT, DistilBERT, Q8BERT and XLNet).
We leverage this intuition about the different
NLP tasks in our framework to reduce the overheads of Specialization without compromising on the quality of the Specialized model.
If there is no prior knowledge of the task, we need to inspect elements in multiple orders to obtain the best model all the studied Transformer architectures
[Table]
A. A NALYZING THE BENEFITS OF S PECIALIZATION
Transformers are pre-trained on a large text corpus on a difﬁcult NLP task, such as predicting the next word in text given all the preceding words.
When pre-trained models are ﬁne-tuned for different downstream tasks, they contain lots of task-irrelevant information that add noise and confuse the model.
In addition, since the pre-trained models are highly overparameterized, they severely overﬁt the small ﬁne-tuning datasets.
Intuitively, Specialization helps the model focus on the task at hand by eliminating the irrelevant information in the ﬁne-tuned models.
A.4. I MPROVED G ENERALIZATION
Accuracy-driven pruning can be seen as a form of training, since the objective of accuracy-driven pruning – minimizing loss on the (validation) dataset – is exactly the same as the objective of training.
When large, over-parameterized models are trained on very limited data, the constraints of accuracy-driven pruning – (1) loss can be reduced only be setting weights to 0 (equivalent to pruning elements), and no other changes to weights are allowed, and (2) a majority of samples in the validation set must beneﬁt from each weight update; reduction in loss is a necessary but insufﬁcient condition for weights to get updated – help it learn more effectively than SGD.
As a result, we ﬁnd that training with SGD on a subset of the training data, followed by accuracy-driven pruning on the remaining unseen data, produces models with better generalization performance than training with
SGD on the entire dataset (Figure 4).
As an added beneﬁt, accuracy-driven pruning also leads to smaller and faster models at inference time.
separability, and hence, are more accurate.
A.4. R EDUCED VARIANCE
Accuracy-driven pruning and selective Hard Attention ﬁlter out noise in the model by removing irrelevant information that ends up confusing the model.
Accuracy-driven pruning reduces noise by removing task-irrelevant information.
Selective Hard Attention further reduces noise by ensuring that the model focuses only on the relevant parts of the input.
We ﬁnd that this leads to reduced variance on the dev/test sets, where Specialized models from different seeds acquire similar word representations, even if the initial ﬁne-tuned models have vastly different word representations (and hence accuracies).
(Figure 5).
However, the reduction in variance is limited by the effects of random initialization of the task-speciﬁc ﬁnal layer.