Dissecting Lottery Ticket Transformers: Structural and Behavioral Study of Sparse Neural Machine Translation (2020)

Abstract: Recent work on the lottery ticket hypothesis has produced highly sparse Transformers for NMT while maintaining BLEU. However, it is unclear how such pruning techniques affect a model’s learned representations. By probing Transformers with more and more low-magnitude weights pruned away, we find that complex semantic information is first to be degraded. Analysis of internal activations reveals that higher layers diverge most over the course of pruning, gradually becoming less complex than their dense counterparts. Meanwhile, early layers of sparse models begin to perform more encoding. Attention mechanisms remain remarkably consistent as sparsity increases.

## introduction in recent years, transformers @xcite have defined state-of-the-art performance on a variety of nlp tasks, including machine translation (mt) and language modeling. while large transformer models can learn uniquely rich representations, they are also highly overparameterized @xcite . several studies have therefore attempted to prune transformers during or after training while retaining as much performance as possible @xcite . some methods have been fairly successful, achieving compression ratios up to 10× depending on the downstream task. looking beyond task performance, however, it remains unclear how widely-used pruning methods affect a model's learned representations. for example, a pruned transformer may translate text at the same bleu, but does pruning affect the model in ways unaccounted for by this metric? motivated by this question, we apply recent analysis techniques to study the representations of increasingly sparse transformers trained on mt. we * equal contribution. using iterative magnitude pruning (imp), we train an en-de transformer that retains 99.4% of bleu at 66.4% sparsity. during imp, we obtain eight transformer models at varying levels of sparsity, along with the original unpruned model. we probe these models' representations for learned linguistic knowledge on eighteen auxiliary syntactic and semantic tasks @xcite . we then perform an unsupervised comparison of the representations and attention distributions between dense and sparse models, adopting metrics posed in @xcite . our key conclusions are as follows: ## related work much work has attempted to reduce the parameter count of dominant transformer-based architectures @xcite . several papers prune bert @xcite , either via structured removal of layers and attention heads @xcite or unstructured pruning of individual weights @xcite . structured head pruning has also been applied to nmt @xcite , in which bleu is used to quantify effective compression. recent work from @xcite uses iterative magnitude pruning to identify lottery tickets for nmt, retaining 99% of bleu at 67% sparsity for transformer-big. to our knowledge, they achieve the highest net pruning ratio on translation with no drop in performance. while most such studies are primarily considered with maximizing sparsity, a subset of them address other questions. @xcite weight prune bert and finetune on glue tasks to identify how much sparsity each task can accommodate. @xcite prune heads while finetuning bert on glue tasks, and identify which heads are masked most often. they use pruning as an analysis technique to identify 'good' or 'bad' bert subnetworks. similarly, @xcite and @xcite prune heads to identify which types of attention are most relevant to performance. however, these studies focus only on task performance, leaving other behavioral differences between dense and sparse models unexplored. relevant methods of analyzing representations in nlp include probing classifiers, which evaluate model representations on supervised tasks for morphology @xcite , syntax @xcite , and/or semantics @xcite . for transformers, some work has directly examined the attention module @xcite . these analyses include inference of functional annotations for particular heads @xcite , or assessment of atten-tion's ability to perform unsupervised syntax tree prediction @xcite . recent work has also applied high-dimensional similarity analysis methods to compare learned representations within or across models @xcite . for instance, @xcite identify recurring neurons across nmt models, interpret their functions, and control their activations. a broader survey of such literature is covered by @xcite . we leverage some of these representation analysis methods to study and compare sparse and dense transformers, which, to our knowledge, previous work has not addressed. ## discussion a consistent theme in our analysis is the behavioral shift of early layers (1-3), which occurs gradu-ally as sparsity increases. our probing results find that lower layers of sparse models more directly encode pos and syntax information compared to dense models, even though performance of the final encoder representations is similar (4.4). moreover, our similarity analyses conclude that early layer encoder hidden representations (5.2) and attention distributions (5.4.1) trend closer towards their respective final representations in sparse models. information-theoretically, sparse layers have less maximum capacity for encoding, so each individual layer must shoulder more load for the final representations to remain predictively salient. conversely, an overparameterized dense model can compensate for weak lower layer representations with its upper layers. indeed, upper fc layers are pruned more than lower fc layers (3.4), reflecting the shift in modeling power away from higher layers. we also observe a gradual loss of information stored in model representations as weights are pruned, especially in later layers. individual neurons diverge from their dense counterparts (5.2), causing a drop in overall representational complexity in the encoder and decoder. correspondingly, sparse models perform worse at higher-order semantic tasks that are less relevant to bleu (4.3). the reduced overall complexity of sparse representations may partially explain why final layers are observed to be closer to early layers (5.2, 5.4.2). finally, we find that sparse models' attention distributions remain largely similar to their values in the dense model. this ability to reduce weights in attention modules while maintaining nearly identical representations affirms other lines of work @xcite . of the three attention types, encoder-decoder is pruned least (3.4), varies most across sparsities, and exhibits most within-model, inter-layer heterogeneity (5.4.3). these results corroborate existing evidence of its unique importance @xcite . meanwhile, decoder self-attention is extremely homogenous across layers and sparsities, perhaps because encoder-decoder attention is more relevant to creating rich representations. limitations. our work focuses on pruned transformers for which bleu remains similar to the original model. however, bleu is an imperfect measure of translation quality @xcite , and it is possible that our pruned models actually perform worse on the task at lower sparsities than suggested by bleu. still, we think our work is relevant given that sparse models are typically only held to the standard of matching unpruned task performance. next, we emphasize that our work focuses solely on magnitude pruning, which may not be representative of how other pruning methods impact transformers. we chose this style of pruning primarily because it allows for higher overall sparsity without drop in performance @xcite . further, while it might be expected (and has been shown, in some cases) that pruning entire neurons or attention heads would substantially change e.g. the distributions of the model's outputs, we found less existing work specifically measuring the effects of magnitude pruning. this dearth of analysis seemed particularly egregious given recent growth in work on unstructured sparsity @xcite . finally, a note on probing classifiers: as has been widely discussed by the community (e.g. pimentel et al. (2020) ), probes measure correlation between model outputs and auxilliary information. differences in probe performance do not necessarily imply anything about what information actually uses during its forward pass. especially since we find some evidence suggesting that sparse models may be encoding information across layers, it is possible that their differing structure may explain worse probe performance, as opposed to fundamentally weaker linguistic feature extraction. we hope future work supplements our results by analyzing a model's encoded knowledge in other ways. ## conclusions we evaluate how unstructured pruning affects the behavior of transformers while task performance is maintained. we use probing classifiers to demonstrate that pruning degrades semantic knowledge before affecting bleu, and that early layers of sparse models better encode low-level linguistic information. unsupervised similarity analysis reveals that pruning induces representational changes in the encoder and decoder, particularly in higher layers, and that early sparse representations are more similar to their final representations. meanwhile, attention distributions remain remarkably similar, even at high sparsities.