Multi-Grained Knowledge Distillation for Named Entity Recognition (2021)

Abstract: Although pre-trained big models (e.g., BERT, ERNIE, XLNet, GPT3 etc.) have delivered top performance in Seq2seq modeling, their deployments in real-world applications are often hindered by the excessive computations and memory demand involved. For many applications, including named entity recognition (NER), matching the state-of-the-art result under budget has attracted considerable attention. Drawing power from the recent advance in knowledge distillation (KD), this work presents a novel distillation scheme to efficiently transfer the knowledge learned from big models to their more affordable counterpart. Our solution highlights the construction of surrogate labels through the k-best Viterbi algorithm to distill knowledge from the teacher model. To maximally assimilate knowledge into the student model, we propose a multi-grained distillation scheme, which integrates cross entropy involved in conditional random field (CRF) and fuzzy learning. To validate the effectiveness of our proposal, we conducted a comprehensive evaluation on five NER benchmarks, reporting cross-the-board performance gains relative to competing prior-arts. We further discuss ablation results to dissect our gains.

## introduction the task of named entity recognition (ner) aims to put named entity mentioned in a sentence into some pre-defined categories such as the person names, organizations, locations, etc. ner is a fundamental task in nature language processing (nlp), and often serves as an information extraction tool embedded in solutions for downstream tasks such as event recognition and aspect-level sentiment analysis. and therefore, the computational and memory efficiency of an ner model is often considered crucial in many empirical settings. given its practical significance, considerable research effort has been devoted to ner in recent years. one fruitful direction is to boost ner performance through the use of more sophisticated model architectures, such as transformer and its variants (e.g., bert @xcite , ernie @xcite , xlnet @xcite , roberta @xcite , gpt3 @xcite , etc.). many of these models are based on the attention mechanism, which allows the models to adaptively focus on different parts of the sentence based on its current understanding. this enables more accurate parsing of the context, which is critical for the ner task. while such advanced models have delivered substantial improvements, a major drawback is that they are typically computationally expensive and memory intensive, preventing their applications in many cost-sensitive settings. as such, it is often desirable to reduce those big models into more affordable counterparts, preferably without any significant performance drop. one strategy is to compress or truncate the original model, examples in this category include parameter pruning @xcite , low-rank approximation @xcite and parameter quantization @xcite . the resulting model has a similar architecture to the original model, and consequently may suffer from similar limitations of the original model. alternatively, knowledge distillation (kd) considers transferring knowledge into models with heterogeneous architectures @xcite , which allows more flexibility in the control of resource usage. in kd, a new learner, often dubbed the student model, assimilates knowledge from a pre-trained model, commonly known as the teacher. for classification tasks, this is typically achieved via minimizing their discrepancy in the output, regardless of the internal model architectures. teacher outputs, sometimes referred to as the soft-labels, typically encode more information than what a student might receive from the raw training feature-label pairs, thus resulting the improved learning efficiency. however, for sequence tasks such as ner, standard techniques from kd do not readily apply. while common kd seeks to minimize the kl divergence between the output label distributions, for ner, the number of label combinations grows exponentially wrt the sequence length. extracting teacher knowledge as if each combination is a different label category would be largely inefficient, if possible at all. on the other hand, many sota ner models are built on conditional random fields (crf) to incorporate the label dependencies, and it only offers an un-normalized likelihood that is not directly amenable to the computation of kldivergence. another challenge for training an ner model is the lack of labeled sequences for training, while the unlabeled sequences may be plentiful. the promise to leverage unlabeled data to improve ner accuracy is appealing. to address the above challenges, in this paper, we present an efficient knowledge distillation scheme that trains a light model (e.g., bilstm) which is able to retain the accuracy of its heavier counterparts, such as bert, while significantly reduce cost. our solution exploits the "soft surrogates", i.e., the most probable label sequences under the teacher model, to inform the student learner. to efficiently identify the most likely label sequences and determine their relative likelihood, we explore the use of viterbi algorithm to expedite computation. we also explore the use of unlabeled data to improve performance. in summary, we highlight the following contributions in this study: (i) we present a novel multi-grained knowledge distillation strategy for sequence labeling via efficiently selecting k-best label sequence using viterbi algorithm; (ii) we advocate the use of a complete cross entropy loss and fuzzy distillation loss to respectively account for probability mass of un-selected sequences and the uncertainties in teacher confidence; (iii) we present a comprehensive empirical analysis to dissect the gains from each individual components. we also show our model delivers substantial improvement relative to competing solutions on a wide-range of real-world benchmarks to demonstrate its utility. ## background our work is inspired by three lines of research: sequence-level knowledge distillation, k-best viterbi algorithm and fuzzy conditional random field (crf). in the following we review the technical backgrounds that are needed for the construction of our model. ## multi-grained distillation in this section we detail the construction of our distillation scheme, with the overall architecture outlined in figure 3 . ## experiment to validate the proposed solution and elaborate the gains, we benchmark it against state-of-the-art methods, through a wide range of experiments on real-world datasets. all experiments are implemented with tensorflow and executed on a single nvidia p100 gpu. details of the experimental setup are provided in supplementary material, due to space limits, and our code will be avialable at https://github.com/11zhouxuan/ multi_grained_kd_ner . ## conclusions in this work, we develop novel multi-grained knowledge distillation techniques to train a light ner model with comparable performance with their more sophisticated counterparts. in particular, we show that viterbi algorithm can be exploited to impart the knowledge of k-best predictions from the teacher model to the student. we further advocated the use of crf adjustments, fuzzy objective and data augmentation to improve performance. our empirical experiments carefully analyze the gains from our proposal on a wide range of ner benchmarks, via efficiently transferring knowledge from a powerful bert model to a much more compact bilstm student. in future work, we seek to extend the proposed framework to more general distillation applications where crf is used, such as speech recognition, and distill with more general representation transfer schemes @xcite .