Multistage Collaborative Knowledge Distillation from a Large Language Model for Semi-Supervised Sequence Generation (2024)

Abstract: We study semi-supervised sequence generation tasks, where the few labeled examples are too scarce to finetune a model, and meanwhile, few-shot prompted large language models (LLMs) exhibit room for improvement. In this paper, we present the discovery that a student model distilled from a few-shot prompted LLM can commonly generalize better than its teacher to unseen examples on such tasks. We find that the student is able to learn a general pattern from the high-quality pseudolabels produced by the teacher during knowledge distillation (KD), and favorably not a general pattern from the low-quality pseudolabels. Leveraging this discovery, we propose a new method, Multistage Collaborative Knowledge Distillation from an LLM (MCKD), for these tasks. MCKD first few-shot prompts an LLM to produce pseudolabels for unlabeled data. Then at each stage of an iterative KD process, a new pair of students is trained on disjoint partitions of the pseudolabeled data, and produces new and improved pseudolabels for their unseen partitions. We conduct extensive experiments on four syntactic and semantic parsing datasets and show the effectiveness of MCKD for low-resource semi-supervised sequence generation. On CRAFT biomedical parsing, for example, 3-stage MCKD with 50 labeled examples outperforms an LLM teacher and vanilla KD by 7.5% and 3.7% parsing F1, respectively, and matches the performance of supervised finetuning with 500 labeled examples.

## introduction low-resource tasks are common in real life, including within specialized domains, since data annotation often requires expert knowledge and incurs significant costs @xcite . semisupervised learning has been proposed as a solution when abundant unlabeled data are available @xcite @xcite . in a typical application, a model is trained on limited labeled data and produces pseudolabels for unlabeled data @xcite . the pseudolabeled data are then filtered according to confidence thresholds and used to train a new model. in more extreme few-shot cases, labeled data are too scarce to finetune a model to begin with. large language models (llms) offer a useful mechanism for synthesizing pseudolabels, thanks to their remarkable ability to learn in context from only a handful of demonstrations @xcite @xcite . smaller and faster models can then be trained for these tasks using knowledge distillation (kd) from llms. in this paper, we study a challenging semisupervised sequence generation setting where labeled data are too few to finetune a model and few-shot prompted llms exhibit room for improvement. these will happen when the task is both expensive to annotate and under-represented in the pretraining of off-the-shelf llms. for example, it took 80 annotators around 2.5 years to parse 20k sentences of biomedical text in the craft corpus @xcite . meanwhile, pretrained llms do not always excel at tasks in specialized domains @xcite and tasks that involve specialized structured outputs, e.g., semantic or syntactic parsing. the overarching research question of this paper is whether llms can still be leveraged in such scenarios to develop strong prediction models. to this end, we examine knowledge distillation (kd) from a few-shot prompted llm to a much smaller model. we discover that the student can commonly outperform its llm teacher on unseen evaluation data from such tasks. our analysis reveals that the student can learn a general pattern from high-quality pseudolabels from the teacher, while helpfully failing to capture a general pattern for the low-quality ones due to their noisy nature. this discovery is encouraging because it opens up the possibility for leveraging the kd students as teachers for further distillation. leveraging the discovery, we propose a novel method, multistage collaborative kd from an llm (mckd), for semi-supervised sequence generation. mckd first collects pseudolabels for a large amount of unlabeled data from a few-shot prompted llm, bootstraping a multistage kd process. at each kd stage, a new pair of students is trained on distinct partitions of the pseudolabeled data and asked to produce pseudolabels for the data that they have not been trained on. this process improves upon vanilla kd through two core mechanisms: (1) cross-partition labeling at each kd stage, which splits all pseudolabeled data into two mutually exclusive partitions to train a collaborative pair of students and leverages their strong generalization capabilities to relabel the data, and (2) multistage kd, whereby student models continue to generalize better and produce higher quality pseudolabels than their teachers over multiple kd stages. mckd outperforms the llm teacher and the relevant kd baselines in our evaluation, and is competitive with supervised finetuning with many more labeled data. on craft constituency parsing, for example, mckd with 50 labeled examples outperforms the llm and vanilla kd by 7.5% and 3.7% f1, respectively, and matches supervised finetuning with 500 examples. on atis semantic parsing, mckd with 50 labeled examples outperforms the llm and vanilla kd by 7.2% and 2.3% f1, matching supervised finetuning @xmath0. the following is a summary of our contributions: ## main results baseline performance. as shown in supervised finetuning of t5-base on d labeled has the worst performance. this matches our assumption that the amount of available labeled data is too small to directly finetune a model. on both datasets for constituency parsing, 50 labeled examples completely fail to effectively finetune t5-base to generate well-formed parse trees. vanilla kd yields students that outperform the teacher across all datasets. kd + sd with confidence-based filtering is shown to have limited performance improvement over vanilla kd and is subject to the selection of threshold for highconfidence labels for each case. for instance, keeping the 25% most confident pseudolabels gives the best performance for ptb with 50 labeled data, while keeping 75% performs the best for craft with 50 labeled data. selecting a proper threshold thus can be resource consuming and difficult in practice especially in low-data regimes with no labeled validation data. mckd performance. our mckd approach substantially outperforms the few-shot prompted gpt-3.5 turbo and kd baselines in all settings. for example, on craft with 50 gold labeled training examples, 2-stage mckd outperforms gpt-3.5 turbo by 6.9% f1 and vanilla kd by 3.1% f1. 3-stage mckd further improves results, outperforming gpt-3.5 turbo by 7.5% and vanilla kd by 3.7%. mckd needs much less labeled data to match the performance of sft. ## analysis this section presents additional analyses that motivate the design of mckd and explore more deeply into why it performs well. all analyses are conducted on ptb data. for simplicity and without loss of generality, we examine 2-stage kd with a single student per stage; the analyses can be generalized to multiple students and more stages which are advantageous in more applied settings. in each experiment, we sample subsets of d unlabeled a and d unlabeled b (see section 3.1) and denote them as d a and d b . we vary the sizes of d a and d b across experiments to study the effect of the distillation set size on performance. we train a stage-1 student s 1 using d a with pseudolabels from an llm teacher t and evaluate it on d b against ground truth labels (assumed to be available only for analysis purposes and not during the training in mckd). then we train a stage-2 student s 2 using d b with pseudolabels from s 1 and evaluate it on the test set. ## related work knowledge distillation (kd), apart from being used for model compression @xcite @xcite , has shown effectiveness in semi-supervised learning @xcite , where a teacher model annotates unlabeled data to train a student model. we study low-resource semi-supervised sequence generation with kd. we use hard pseudolabels from teachers to train student models, following common practice in sequence-level knowledge distillation @xcite , including scenarios when the teacher is an llm @xcite @xcite @xcite @xcite . self-distillation (sd) through training the model with its self-generated pseudolabels is often valuable in enhancing generalization, but finding reliable confidence thresholds to filter the pseudolabels and ensure their quality is often tricky @xcite . instead, we improve the pseudolabel quality over stages by crosspartition labeling, taking advantage of the better generalization abilities of models over stages. some past works also leverage several student models to improve the distillation performance. @xcite and @xcite introduce several student models that learn from each other's predicted logits. @xcite and @xcite introduce extra branches of the student model (e.g., high-level layers or the classifier heads) to gather their multiple views as additional information for student training, which is shown helpful to improve the generalization. these works are mainly evaluated on image classification and do not analyze distillation from llms. co-training @xcite @xcite is relevant to our work since it involves using separate models to generate improved pseudolabels. in co-training, different learners are typically initialized with different views of the data. then they provide confident predictions to each other for additional training. @xcite provides theoretical explanation on why generated pseudolables can be more credible than original labels for training. in contrast, our approach begins with an llm teacher and trains better student models over stages. ## conclusion semi-supervised sequence generation in specialized low-resource settings can be challenging for nlp systems. we present the discovery that student models distilled from a few-shot prompted llm can often correct the teacher's errors, thus showing superior generalization, on such tasks. exploiting this phenomenon along with strategies such as data partitioning and cross-partition labeling over multiple stages, we then propose mckd, a novel solution that utilizes knowledge distillation (kd) from an llm, whose performance can be only at a mediocre level, to train much smaller yet high-accuracy students. mckd essentially leverages progressively better models to train future generations with even better performance. extensive empirical evaluation verifies the effectiveness and label efficiency of mckd.