Structural Knowledge Distillation: Tractably Distilling Information for Structured Predictor (2021)

Abstract: Knowledge distillation is a critical technique to transfer knowledge between models, typically from a large model (the teacher) to a more fine-grained one (the student). The objective function of knowledge distillation is typically the cross-entropy between the teacher and the student’s output distributions. However, for structured prediction problems, the output space is exponential in size; therefore, the cross-entropy objective becomes intractable to compute and optimize directly. In this paper, we derive a factorized form of the knowledge distillation objective for structured prediction, which is tractable for many typical choices of the teacher and student models. In particular, we show the tractability and empirical effectiveness of structural knowledge distillation between sequence labeling and dependency parsing models under four different scenarios: 1) the teacher and student share the same factorization form of the output structure scoring function; 2) the student factorization produces more fine-grained substructures than the teacher factorization; 3) the teacher factorization produces more fine-grained substructures than the student factorization; 4) the factorization forms from the teacher and the student are incompatible.

## introduction deeper and larger neural networks have led to significant improvement in accuracy in various tasks, but they are also more computationally expensive and unfit for resource-constrained scenarios such as online serving. an interesting and viable solution to this problem is knowledge distillation (kd) @xcite @xcite , which can be used to transfer the knowledge of a large model (the teacher) to a smaller model (the student). in the field of natural language processing (nlp), for example, kd has been successfully applied to compress massive pretrained language models such as bert @xcite and xlm-r @xcite into much smaller and faster models without significant loss in accuracy @xcite @xcite . a typical approach to kd is letting the student mimic the teacher model's output probability distributions on the training data by using the cross-entropy objective. for structured prediction problems, however, the output space is exponentially large, making the cross-entropy objective intractable to compute and optimize directly. take sequence labeling for example. if the size of the label set is l, then there are l n possible label sequences for a sentence of n words and it is infeasible to compute the cross-entropy by enumerating the label sequences. previous approaches to structural kd either choose to perform kd on local decisions or substructures instead of on the full output structure, or resort to top-k approximation of the objective @xcite @xcite . in this paper, we derive a factorized form of the structural kd objective based on the fact that almost all the structured prediction models factorize the scoring function of the output structure into scores of substructures. if the student's substructure space is polynomial in size and the teacher's marginal distributions over these substructures can be tractably estimated, then we can tractably compute and optimize the factorized form of the structural kd objective. as will be shown in the paper, many widely used structured prediction models satisfy the assumptions and hence are amenable to tractable kd. in particular, we show the feasibility and empirical effectiveness of structural kd with different combinations of teacher and student models, including those with incompatible factorization forms. we apply this technique to structural kd between sequence labeling and dependency parsing models under four different scenarios. in all the cases, we empirically show that our structural kd approaches can improve the student models. in the few cases where previous kd approaches are applicable, we show our approaches outperform these previous approaches. with unlabeled data, our approaches can further improve student models' performance. in a zero-shot crosslingual transfer case, we show that with sufficient unlabeled data, student models trained by our approaches can even outperform the teacher models. ## structural knowledge distillation when performing knowledge distillation on structured prediction, a major challenge is that the structured output space is exponential in size, leading to intractable computation of the kd objective in eq. 2. however, if the scoring function of the student model can be factorized into scores of substructures (eq. 1), then we can derive the following factorized form of the structural kd objective. based on the difference in score factorization between the teacher and student models, we divide our discussion into four scenarios. ## conclusion in this paper, we propose structural knowledge distillation, which transfers knowledge between structured prediction models. we derive a factorized form of the structural kd objective and make it tractable to compute and optimize for many typical choices of teacher and student models. we apply our approach to four kd scenarios with six cases for sequence labeling and dependency parsing. empirical results show that our approach outperforms baselines without kd as well as previous kd approaches. with sufficient unlabeled data, our approach can even boost the students to outperform the teachers in zero-shot cross-lingual transfer.