D is C o: Distilled Student Models Co-training for Semi-supervised Text Mining (2023)

Abstract: Many text mining models are constructed by fine-tuning a large deep pre-trained language model (PLM) in downstream tasks. However, a significant challenge that arises nowadays is how to maintain performance when we use a lightweight model with limited labeled samples. We present DisCo, a semi-supervised learning (SSL) framework for fine-tuning a cohort of small student models generated from a large PLM using knowledge distillation. Our key insight is to share complementary knowledge among distilled student cohorts to promote their SSL effectiveness. DisCo employs a novel co-training technique to optimize a cohort of multiple small student models by promoting knowledge sharing among students under diversified views: model views produced by different distillation strategies and data views produced by various input augmentations. We evaluate DisCo on both semi-supervised text classification and extractive summarization tasks. Experimental results show that DisCo can produce student models that are 7.6\times smaller and 4.8 \times faster in inference than the baseline PLMs while maintaining comparable performance. We also show that DisCo-generated student models outperform the similar-sized models elaborately tuned in distinct tasks.

## introduction large pre-trained language models (plms), such as bert @xcite , @xcite , play a crucial role in the development of natural language processing applications, where one prominent training regime is to fine-tune the large and expensive plms for the downstream tasks of interest @xcite . minimizing the model size and accelerating the model inference are desired for systems with limited computation resources, such as mobile @xcite and edge @xcite devices. therefore, maintaining the generalization ability of the reduced-sized model is crucial and feasible @xcite @xcite . semi-supervised learning (ssl) emerges as a practical paradigm to improve model generalization by leveraging both limited labelled data and extensive unlabeled data @xcite @xcite @xcite @xcite @xcite @xcite . while promising, combining ssl with a reduced-size model derived from plms still necessitates a well-defined learning strategy to achieve improved downstream performances @xcite . this necessity arises because these shallow networks typically have lower capacity, and the scarcity of labeled data further curtails the model's optimization abilities. besides, a major hurdle is a lack of labelled data samples -a particular problem for text mining tasks because the labelling text is labour-intensive and error-prone @xcite @xcite @xcite . this paper thus targets using ssl to leverage distilled plms in a situation where only limited labelled data is available and fast model inference is needed on resource-constrained devices. to this end, we use the well-established teacher-student knowledge distillation technique to construct small student models from a teacher plm and then finetune them in the downstream ssl tasks. we aim to improve the effectiveness of fine-tuning small student models for text-mining tasks with limited labelled samples. we present disco, a novel co-training approach aimed at enhancing the ssl performances by using distilled small models and few labelled data. the student models in the disco acquire complemen-tary information from multiple views, thereby improving the generalization ability despite the small model size and limited labelled samples. we introduce two types of view diversities for co-training: i) model view diversity, which leverages diversified initializations for student models in the cohort, ii) data view diversity, which incorporates varied noisy samples for student models in the cohort. specifically, the model view diversity is generated by different task-agnostic knowledge distillations from the teacher model. the data view diversity is achieved through various embedding-based data augmentations to the input instances. intuitively, disco with the model view encourages the student models to learn from each other interactively and maintain reciprocal collaboration. the student cohort with the model views increases each participating model's posterior entropy @xcite @xcite , helping them to converge to a flatter minimum with better generalization. at the same time, disco with the data views regularizes student predictions to be invariant when applying noises to input examples. doing so improves the models' robustness on diverse noisy samples generated from the same instance. this, in turn, helps the models to obtain missing inductive biases on learning behaviour, i.e., adding more inductive biases to the models can lessen their variance @xcite . we have implemented a working prototype of disco 1 and applied it to text classification and extractive summarization tasks. we show that by cotraining just two student models, disco can deliver faster inference while maintaining the performance level of the large plm. specifically, disco can produce a student model that is 7.6× smaller (4layer tinybert) with 4.8× faster inference time by achieving superior rouge performance in extractive summarization than the source teacher model (12-layer bert). it also achieves a better or comparable text classification performance compared to the previous state-of-the-art (sota) ssl methods with 12-layer bert while maintaining a lightweight architecture with only 6-layer tiny-bert. we also show that disco substantially outperforms other ssl baselines by delivering higher accuracy when using the same student models in model size. 1 code and data are available at: https://github.com/ litesslhub/disco . ## conclusion in this paper, we present disco, a framework of co-training distilled students with limited labelled data, which is used for targeting the lightweight models for semi-supervised text mining. disco leverages model views and data views to improve the model's effectiveness. we evaluate disco by applying it to text classification and extractive summarization tasks and comparing it with a diverse set of baselines. experimental results show that disco substantially achieves better performance across scenarios using lightweight ssl models.