F use G en: PLM Fusion for Data-generation based Zero-shot Learning (2024)

Abstract: Data-generation based zero-shot learning, although effective in training Small Task-specific Models (STMs) via synthetic datasets generated by Pre-trained Language Models (PLMs), is often limited by the low quality of such synthetic datasets. Previous solutions have primarily focused on single PLM settings, where synthetic datasets are typically restricted to specific sub-spaces and often deviate from real-world distributions, leading to severe distribution bias. To mitigate such bias, we propose FuseGen, a novel data-generation based zero-shot learning framework that introduces a new criteria for subset selection from synthetic datasets via utilizing multiple PLMs and trained STMs. The chosen subset provides in-context feedback to each PLM, enhancing dataset quality through iterative data generation. Trained STMs are then used for sample re-weighting as well, further improving data quality. Extensive experiments across diverse tasks demonstrate that FuseGen substantially outperforms existing methods, highly effective in boosting STM performance in a PLM-agnostic way. The code is available at https://github.com/LindaLydia/FuseGen.

## introduction despite the prevalence of powerful pre-trained language models (plms) @xcite @xcite such as gpt-4, small task-specific models (stms) are indispensable due to their compact size and efficiency, especially for resource-constrained environments @xcite . to compensate for the scarcity of high-quality training data, synthetic data generated by plms has been widely applied for stm training @xcite . in particular, data-generation based zero-shot learning @xcite @xcite trains stm using the dataset synthesized by one plm through task-related labeldescriptive prompts, requiring only the task name (e.g. movie review sentiment analysis) and label categories (e.g. positive/negative). this zero-shot trained stm is significantly smaller than the original plm with comparable performance @xcite , thus is particularly advantageous for domains with limited computational resources (e.g. on mobile devices) or strict data privacy constraints (e.g. in finance applications). however, the long-standing low-quality issue of synthetic data impedes the practical application of stms to a wider range @xcite . previous works on improving synthetic data quality mainly focus on enhancing data diversity @xcite @xcite , reducing redundancy @xcite , and implementing data-importance-guided in-context feedback @xcite or sample re-weighting @xcite . despite notable advancements, they primarily rely on one single plm as source, inevitably overlooking the inherent distribution biases of synthetic datasets. to thoroughly investigate these biases and their impact on stm performance, we conduct two pilot studies. as illustrated in figure 1 , we use the dataset cartography approach @xcite to plot the cartography of synthetic datasets given by different plms. dataset samples are categorized into easy-to-learn (marked in red), ambiguous (marked in black) and hard-to-learn (marked in blue) based on their confidence and variability, defined as the mean and standard deviation of model probabilities for their labels across training epochs. since easy-to-learn samples aid convergence and ambiguous samples are vital for boosting performance @xcite , an ideal dataset should predominantly contain diverse easy-to-learn and ambiguous samples, with fewer hard-to-learn samples which are often mislabeled (swayamdipta @xcite ) using 1, 000 samples generated by llama-2 and flan-t5 for movie review semantic analysis. zerogen @xcite uses zero-shot prompt for generation, while progen @xcite and fusegen (ours) use few-shot prompt with feedback, with progen relying on a single plm and fusegen leveraging multiple plms. k is the number of plms. numbers within parentheses are the results of stm trained with self-boosting weight adjustment (see section 3.4) and evaluated over imdb @xcite dataset. results for more plms are provided in figure 8 in appendix c.1. figure 2 : performance of stm trained using 6, 000 synthetic data samples generated by various plms. "mixed" uses a dataset comprising 6, 000 total samples given by the 6 listed plms (1, 000 samples per plm). "fusegen" (ours) uses the 6 listed plms and 6, 000 samples. @xcite . this composition of diverse samples promises better stm performance. in a second study, we provide the comparison between stms trained with different datasets that vary in sources and generation methods, as illustrated in figure 2 . these visualization analyses reveal three key observations: (1) synthetic datasets from different plms exhibit significant distribution biases. for example, figures 1(a ) and 1(d) show that the zeroshot synthetic dataset produced by llama-2 @xcite primarily includes easy-to-learn samples, whereas that of flan-t5 @xcite ) contains a more balanced mixture of all 3 categories. (2) distribution biases are difficult to overcome by only relying on a single plm. progen @xcite , an advanced single-plm generation method, only slightly improves the ratio of easy-tolearn and ambiguous samples (figure 1 to tackle these challenges, we propose fusegen, a smart data-generation based zero-shot learning framework that mitigates inherent dataset distribution bias by harnessing the diversity of a plm cluster. in fusegen, given a specific task and its label categories, synthetic datasets are initially generated by various plms in a zero-shot manner, which are then used to train their respective stms. to alleviate distribution bias, fusegen selects superior samples generated by multiple plms as shared in-context feedback, and prompts each plm to accumulate higher-quality data iteratively. to select relevant in-context samples, fusegen pivots on an efficient cross-model criteria that considers both dataset composition and individual sample importance. to mitigate the negative impact of poor-quality samples, fusegen further uses a self-boosting method to dynamically adjust sample weights to optimize stm in training. as demonstrated in figures 1(c ), 1(f) and 2, with these novel techniques, fusegen effectively reduces distribution biases and achieves better stm performance than state-of-the-art methods. our contributions can be summarized as follows: ## conclusion we propose a novel data-generation based zeroshot learning framework fusegen that harnesses the collaborative capability of multiple plms to improve synthetic data generation of plms. we first integrate multiple plms to alleviate distribution bias of synthetic datasets through cross-plm in-context samples selection, for constructing better feedback recursively. to further improve stm performance, we employ a self-boosting weight adjustment strategy to down-weight low-quality samples. extensive experiments and ablation studies on various nli and nlu tasks demonstrate that fusegen is highly effective, query-efficient and plm-agnostic without the reliance on specific plms for downstream tasks, making it a more flexible and resource-efficient solution.