Self-Regulated Data-Free Knowledge Amalgamation for Text Classification (2024)

Abstract: Recently, there has been a growing availability of pre-trained text models on various model repositories. These models greatly reduce the cost of training new models from scratch as they can be fine-tuned for specific tasks or trained on large datasets. However, these datasets may not be publicly accessible due to the privacy, security, or intellectual property issues. In this paper, we aim to develop a lightweight student network that can learn from multiple teacher models without accessing their original training data. Hence, we investigate Data-Free Knowledge Amalgamation (DFKA), a knowledge-transfer task that combines insights from multiple pre-trained teacher models and transfers them effectively to a compact student network. To accomplish this, we propose STRATANET, a modeling framework comprising: (a) a steerable data generator that produces text data tailored to each teacher and (b) an amalgamation module that implements a self-regulative strategy using confidence estimates from the teachers’ different layers to selectively integrate their knowledge and train a versatile student. We evaluate our method on three benchmark text classification datasets with varying labels or domains. Empirically, we demonstrate that the student model learned using our STRATANET outperforms several baselines significantly under data-driven and data-free constraints.

## introduction recent nlp advancements have yielded numerous pre-trained models, often achieving state-of-the-art performance across various tasks. these models are publicly available to promote reproducibility and further research. to facilitate knowledge transfer from pre-trained teacher models, @xcite pioneered knowledge distillation (kd), utilizing soft target labels to train light-weight student models effectively. subsequently, diverse kd teacher model 2 transport law labeled text data teacher model 1 business ai sociology labeled text data teacher model 2 teacher model 1 student model business ai sociology transport law pre-trained language model synthetic data pre-trained teacher models figure 1 : given a set of pre-trained teacher models (teacher models 1 & 2), each with distinct expertise, the goal is to train a student model capable of amalgamating their knowledge, mastering prediction across all specialized classes of the teachers. unlike traditional kd, where teachers focus on the same task, knowledge amalgamation (ka) techniques @xcite enable learning in a student network by integrating knowledge from multiple teachers with diverse expertise. these methods enhance the student model's classification abilities across a wider range of labels. while ka techniques are well-established in computer vision, their exploration in nlp literature is limited. @xcite utilized monte-carlo dropout to estimate model uncertainty for merging knowledge from different pre-trained teacher models. however, these techniques often require access to unlabeled data from the original training set used by the pre-trained models @xcite @xcite to train a versatile student model. unfortunately, the original training data and annotations are often unavailable due to various issues. moreover, the diverse expertise of teacher models may lead to uncertain states and probabilities when handling input sequences outside their domains. these challenges hinder the application of ka methods in broader domains. to address this, we explore a practical knowledge-transfer task called data-free knowledge amalgamation (dfka). figure 1 provides an overview of this task, aiming to enhance the student model's capabilities by integrating knowledge from multiple pre-trained teachers without access to the original training data. to achieve our goal, we introduce stratanet 1 , a knowledge amalgamation framework with: (i) a flexible generation module creating pseudo text data for each pre-trained teacher network, and (ii) an amalgamation module enabling self-regulated integration of teachers' knowledge during student model training. integration is guided by a teacherspecific out-of-distribution (ood) score, assessing the reliability of intermediate and output states of every pre-trained teacher model. contributions: (1) introduction of stratanet, a pioneering data-free knowledge amalgamation (dfka) method for lightweight student model training without accessing original training data. ## related work in this section, we explore the relevant literature concerning knowledge distillation (kd) and amalgamation. kd is a technique aimed at transferring knowledge from a large teacher network to a student model, offering benefits across various nlp tasks and facilitating model compression. these tasks encompass question answering @xcite , multi-modal summarization @xcite , and neural machine translation @xcite @xcite , among others. no in contrast to the singular teacher model approach in kd, knowledge amalgamation (ka) involves training a versatile student model by amalgamating insights from multiple pre-trained teacher models. @xcite utilized monte carlo dropout to estimate model uncertainty and perform classification on the union of label sets from different teacher models. although these methods do not rely on human-annotated labels, they leverage input text from the original training data. @xcite proposed a parameter space merging method for dataless knowledge fusion, assuming an impractical uniformity in model architectures across input and merged models. differing from the aforementioned approaches, our method, stratanet, introduces a framework for data-free knowledge amalgamation (dfka) in text, representing a pioneering exploration in nlp literature involving multiple heterogeneous teacher networks. ## problem setup given k pre-trained teacher models @xmath0, each with l t i -layers and its own domain of expertise, i.e., performing a c i -class classification task with few overlapping or disjoint set of labels y @xmath1, our goal is to train a lightweight student model s with l s -layers such that it can compute predictions over the union of all the label sets, @xmath2. 4 proposed approach 4.1 overview in this section, we outline our framework, stratanet, designed to train a lightweight student model using multiple teachers under data-free extract layer-wise representations for ood estimates amalgamate relevant block-wise knowledge pre-trained language model attribute controller pre-trained teacher model great top-m tokens really fun boring controls data generation teacher-specific class-conditional pseudo-data samples teacher-specific steerable data generator confidence scores teacher-specific ood estimator selective transformer-based block-wise @xmath3: illustration of our stratanet framework. ## training objectives & details to amalgamate knowledge at intermediate layers, we compute l2-normalized distance between the student's projected block-level representation and the corresponding teachers' amalgamated embedding. formally, for the output prediction layer, we compute the kl divergence loss based on confidence weighted combination of teacher models and the temperature τ as: l @xmath4, ŝ, τ ). ## results and discussion overall performance the evaluation results are presented in ## conclusion in this study, we introduce data-free knowledge amalgamation (dfka), a method to train a lightweight student network from diverse teacher models without their original training data. our framework, stratanet, employs a steerable data generator and an amalgamation module for effective knowledge transfer. experimental results on text datasets demonstrate the superiority of stratanet over various baselines, both in datadriven and data-free scenarios. ablation studies highlight the importance of different model components. this work opens avenues for efficient knowledge transfer in text classification, offering practical solutions for resource-constrained environments.