Multi-Granularity Structural Knowledge Distillation for Language Model Compression (2022)

Abstract: Transferring the knowledge to a small model through distillation has raised great interest in recent years. Prevailing methods transfer the knowledge derived from mono-granularity language units (e.g., token-level or sample-level), which is not enough to represent the rich semantics of a text and may lose some vital knowledge. Besides, these methods form the knowledge as individual representations or their simple dependencies, neglecting abundant structural relations among intermediate representations. To overcome the problems, we present a novel knowledge distillation framework that gathers intermediate representations from multiple semantic granularities (e.g., tokens, spans and samples) and forms the knowledge as more sophisticated structural relations specified as the pair-wise interactions and the triplet-wise geometric angles based on multi-granularity representations. Moreover, we propose distilling the well-organized multi-granularity structural knowledge to the student hierarchically across layers. Experimental results on GLUE benchmark demonstrate that our method outperforms advanced distillation methods.

## introduction recent years have witnessed a surge of pre-trained language models @xcite @xcite . building upon the transformer architecture @xcite and pre-trained on large-scale corpora using self-supervised objectives, these plms have achieved remarkable success in a wide range of natural language understanding and generation tasks. despite their high performance, these plms usually suffer from high computation and memory costs, which hinders them from being deployed into resource-scarce scenarios, e.g., mobile phones and embedded devices. various attempts have been made to compress the huge plms into small ones with minimum performance degradation. as one of the main approaches, knowledge distillation @xcite utilizes a large and powerful teacher model to transfer the knowledge to a small student model. based on the teacher-student framework, @xcite ; @xcite distilled the token-level representations and attention dependencies to the student, @xcite ; @xcite taught the student to mimic the output logits of the teacher, @xcite enforced the student's representation to be closed to the teacher's while pushing negative samples to be far apart. although proved effective, existing approaches have some flaws. for one thing, these distillation methods only adopted the representations of mono-granularity language units (i.e., token-level or sample-level), while neglecting other granularity. for another, their distillation objectives either matched the corresponding representations between the teacher and the student or aligned the attention dependencies, failing to capture more sophisticated structural relations between the representations. to address these issues, in this paper we propose a novel knowledge distillation framework named multi-granularity structural knowledge distillation (mgskd) through answering the three research questions: (1) which granularity should the knowledge be, (2) what form of knowledge is effective to transfer and (3) how to teach the student using the knowledge. for the "which" question, given that natural languages have multiple semantic granularities, we consider the intermediate representations in three granularities: tokens, spans and samples. specifically, we first take the sub-word tokens as the smallest granularity, then select phrases and whole words as spans for they hold complete meanings, and finally treat the whole input texts as samples. we use mean-pooling to obtain the representations of spans and samples based on token representations. for the "what" question, we propose to leverage the sophisticated structural relations between the representations as the knowledge. concretely, instead of aligning the corresponding representations of the teacher and the student, we propose to form the knowledge as the pair-wise interactions and the triplet-wise geometric angels of a group of representations. for the "how" question, following the recent findings that the bottom layers capture syntactic features while the upper layers encode semantic features @xcite , we conduct hierarchical distillation where the bottom layers of the student are taught token-level and span-level knowledge while the upper layers learn sample-level knowledge. we conduct comprehensive experiments on standard language understanding benchmark glue @xcite . experimental results demonstrate that our knowledge distillation framework outperforms strong baselines methods. surprisingly, mgskd achieves comparable or better performance than bert base on most of the tasks on glue, while keeping much smaller and faster. our contributions in this paper are three folds: knowledge distillation. knowledge distillation @xcite is another major line of research to do model compression, which is the main concentration in this paper. @xcite first proposed to minimize the kl-divergence between the predicted distributions of the teacher and the student. sanh et al. (2019); sun et al. (2019); liang et al. (2020 ) adopted this objective to teach the student on masked language modeling or text classification tasks. @xcite proposed to directly match the feature activations of the teacher and the student. @xcite followed the idea and took the intermediate representations in each transformer layer of the teacher as one of the knowledge to be transferred. @xcite proposed a contrastive distillation framework where the teacher's representations were treated as positives to the corresponding student's representations. @xcite ; @xcite customized this idea to language model compression and proved its effectiveness. researchers also attempted to use the mutual relations of representations as the knowledge to transfer. in the literature of image classification, @xcite al. (2019); tung and mori (2019); park et al. (2019) pointed out that the relations of the image representations of the teacher should be preserved in the student's feature space, and adopted a series of geometric measurements to model the sample relations. for distilling transformer models, @xcite enforced the relations across tokens and layers between the teacher and the student to be consistent. @xcite ; @xcite used the attention dependencies between tokens to teach the student. in this paper, we propose to transfer the multi-granularity knowledge to the student. different from previous works that only considered a single granularity of representations, we jointly transfer the token-level, span-level and sample-level structural knowledge. and compared with @xcite which considered the multi-granularity visual features in an image as the knowledge, our method works in a different modality, presents a different definition of granularity, policeman jackie chan acted as a in a series of action movies . ## method we propose multi-granularity structural knowledge distillation, a novel framework to distill the knowledge from a large transformer language model to a small one. different from previous works that transferred the knowledge derived from either token-level or sample-level outputs, we prepare the knowledge in three semantic granularities: token-level, span-level and sample-level. given some granularity of representations of the teacher model, we form the knowledge as the structural relations, i.e., the pair-wise interactions and the triplet-wise geometric angles, between the representations. we then distill the well-organized structural knowledge to the student hierarchically across layers, where the token-level and the span-level knowledge are transferred to the bottom layers to provide more syntactic guidance while the sample-level knowledge is transferred to the upper layers to offer more help of semantic understanding. the framework of mgskd is illustrated in figure 1 . ## conclusion in this paper, we propose a novel knowledge distillation framework named mgskd. we leverage intermediate representations of multi-granularity language units (i.e., tokens, spans and samples), and form the knowledge as the sophisticated structural relations between the representations rather than the individual representations themselves. the well-organized structural knowledge is then distilled into the student hierarchically across layers. evaluation results on glue benchmark verify the effectiveness of our method. in the future, we plan to explore more forms of structural knowledge.