Combining Curriculum Learning and Knowledge Distillation for Dialogue Generation (2021)

Abstract: Curriculum learning, a machine training strategy that feeds training instances to the model from easy to hard, has been proven to facilitate the dialogue generation task. Meanwhile, knowledge distillation, a knowledge transformation methodology among teachers and students networks can yield significant performance boost for student models. Hence, in this paper, we introduce a combination of curriculum learning and knowledge distillation for efficient dialogue generation models, where curriculum learning can help knowledge distillation from data and model aspects. To start with, from the data aspect, we cluster the training cases according to their complexity, which is calculated by various types of features such as sentence length and coherence between dialog pairs. Furthermore, we employ an adversarial training strategy to identify the complexity of cases from model level. The intuition is that, if a discriminator can tell the generated response is from the teacher or the student, then the case is difficult that the student model has not adapted to yet. Finally, we use self-paced learning, which is an extension to curriculum learning to assign weights for distillation. In conclusion, we arrange a hierarchical curriculum based on the above two aspects for the student model under the guidance from the teacher model. Experimental results demonstrate that our methods achieve improvements compared with competitive baselines.

## introduction along with the enormous prosperity of social media on the internet, there is a resurgent interest in developing open domain dialogue systems. however, the complexity of conversations crawled from the internet may vary significantly. @xcite ; @xcite ; @xcite . to adapt to this phenomenon, some prior works @xcite @xcite employ curriculum learning @xcite , in which a model is taught by using easy samples firstly and gradually adding more difficult ones. for example, @xcite proposes an adaptive multi-curricula learning framework to train the dialogue model with easy-to-complex dataset based on various concepts of difficulty including the specificity and repetitiveness of the response, the relevance between the query and the response, etc. also, @xcite resolves this problem by introducing selfpaced learning @xcite , which is a special kind of curriculum learning @xcite . @xcite measures the level of confidence on each training example, where an easy sample is actually one of high confidence by the current trained model. both curriculum learning and self-paced learning suggest that samples should be selected in a meaningful order for training. the difference is that curriculum learning uses pre-training or human intuitions while the emphasis of self-paced learning can be dynamically determined by model itself. on the other hand, knowledge distillation @xcite is one of the most popular techniques to train efficient models, which aims to transfer the knowledge encoded in a pretrained teacher network into a student model. @xcite points that the teacher's probability predictions can capture the logarithm relationships between labels that are not obvious in the one-hot ground-truth label. moreover, the teacher model can spread uncertainty over multiple outputs when it is not confident of its prediction. as a consequence, student models can yield significant performance boost under the guidance of a teacher. since the knowledge from the teacher to student also has different difficulty degrees, it is intuitive to apply curriculum learning during this distillation process. to our best knowledge, very little is known about how curriculum learning and knowledge distillation work together. hence, in this work, we propose a dialogue generation model that combines curriculum learning and knowledge distillation. firstly, from the data level, we employ different types of features such as sentence length, word and utterance entropy, and coherence between dialog pairs to estimate the complexity of data for each training example. we preliminary cluster the training cases according to their data level complexity. then based on these difficulty scores, we construct a learning curriculum for the student model. secondly, from the model level, we employ an adversarial training strategy to evaluate the model-aware complexity. concretely, the student model is adversarially trained to fool a discriminator, while the discriminator aims to distinguish the outputs from student and teacher networks. we measure the hardness of each sample by taking both the value and history of the discriminator into account, based on the following two intuitions. (1) the discriminator defines an objective of progressive difficulty @xcite , if the discriminator can successfully distinguish the output, then it is a hard case, and vice versa @xcite . (2) the model evolves during training and therefore additional evaluation pass to measure the change in a performance is needed @xcite . in this paper we consider the change in the discriminator. if the change is negative, this must mean that the sample is difficult to train. then based on these model-level difficulty scores, we further transfer the knowledge from teacher to student network gradually by incorporating self-paced learning methodology. our contributions are summarized as follows: (1) we make an empirical study on the combination of curriculum learning methods and knowledge distillation for efficient dialogue generation models. (2) we arrange a hierarchical curriculum based on the above two aspects (data and model) for the distillation model. (3) we apply the proposed framework on two real-life open-domain conversation datasets, automatic and manual evaluation shows that our approach can be used to enhance the performance of dialogue models. ## related work our work is at the intersection of curriculum learning @xcite and knowledge distillation @xcite for training dialogue generation models. ## problem formulation the overall network architecture is shown in figure 1 . the teacher and student model use the same basic architecture that is related to an encoderdecoder @xcite generative dialogue model based on variational autoencoders (vaes) @xcite . in our model, there are three elements: dialogue context @xmath0, x 2 ...x i , response @xmath1, y 2 ...y i and a latent variable z. the dialogue context x is composed of several history utterances. the response y is the responses towards the given context. the latent variable z is used to capture the latent distribution over the replies with a standard gaussian prior, which is defined as follows: our task is to model the true probability of a response y given an input x, which can be estimated as: the hierarchical curriculum strategy for distillation model consists of two parts: one is for the data-level and the other is for the model-level. in the data-level, easier context-response pairs are presented to the student model before harder ones. as for the model level, we design curriculum schedules to gradually transfer knowledge from the the teacher to student, which controls the difficulty of soften labels that are distilled from teacher to students. the samples that discriminator cannot differentiate between the output provided by the student and the teacher are assumed to be easier ones. starting from easier samples, the model progressively strengthens its relation between the teacher and student models. in the rest of this paper, we give detailed descriptions of the proposed approach. ## conclusion in this work, we consider open-domain dialogue systems. to induce model learning from effective teachers, we propose a learnable distillation model to dynamically distill knowledge by hierarchical curriculum learning. experiments conducted on two public conversation datasets show that our proposed framework is able to boost the performance of existing dialogue systems. besides, our framework is not limited to the neural dialogue generation task. in the future, we would extend our method to deal with other text generation tasks (e.g., abstract summarization) and examine this approach's adaptability to these tasks.