U ni G en: Universal Domain Generalization for Sentiment Classification via Zero-shot Dataset Generation (2024)

Abstract: Although pre-trained language models have exhibited great flexibility and versatility with prompt-based few-shot learning, they suffer from the extensive parameter size and limited applicability for inference. Recent studies have suggested that PLMs be used as dataset generators and a tiny task-specific model be trained to achieve efficient inference. However, their applicability to various domains is limited because they tend to generate domain-specific datasets. In this work, we propose a novel approach to universal domain generalization that generates a dataset regardless of the target domain. This allows for generalization of the tiny task model to any domain that shares the label space, thus enhancing the real-world applicability of the dataset generation paradigm. Our experiments indicate that the proposed method accomplishes generalizability across various domains while using a parameter set that is orders of magnitude smaller than PLMs.

## introduction as the size and performance of pre-trained language models (plms) increase, generation of new data by using plms has attracted the attention of many researchers @xcite @xcite . while scholars have applied this method to solve data augmentation problems, in recent studies, they have started to explore zero-shot dataset generation settings @xcite @xcite . this novel approach first generates training data from a plm based on a specific prompt and trains a tiny task model (tam) by using the dataset generated in the first step. this strategy facilitates effective distillation of the knowledge pertaining to the desired task from the plm and helps train the tam without the need for guidance from human-annotated data, thereby enabling zero-shot learning and achieving low-cost inference compared to the case in which plms are used directly for inference. however, the approaches proposed thus far have relied on domain-specific prompts, for example, "the movie review in positive sentiment is:." because the data generated using this prompt are related only to the domain of movie reviews, the tam trained on these data has limited generalization ability across other domains. this is the primary limitation of the tam-based approach compared to prompt-based zero-shot learning that directly uses plms (prompting), which allows for generalizability across diverse domains. this restricts the real-world applicability of the tambased approach because it requires many separately trained tams for various domains. moreover, as the costs of dataset generation and tam training increase, the cost-efficiency of the tam-based approach may decrease. hence, a novel strategy is desired to effectively distill the domain generalizability of large-scale plms into tams while maintaining the cost-efficiency of tams. meanwhile, the existing approaches to domain generalization often require multiple source domains @xcite . this requirement limits the application of these methods because it is difficult to gather the required data from multiple domains. although the concept of single-domain generalization, which achieves domain generalizability by using data from only one source domain, has been proposed in recent computer vision studies, such a concept is yet to be explored for natural language processing @xcite . in this study, we propose a simple but effective method called unigen to solve the problem of domain generalizability between plms and tams. ## conclusion in this study, we proposed unigen in an attempt to achieve universal domain generalization. uni-gen successfully transferred the domain generalizability of plms into orders-of-magnitude smaller tams. moreover, human annotation was not required for unigen, which significantly reduced the burden of acquiring labeled data from multiple source domains. our relabeling method and denoising memory bank offered additional performance gains. furthermore, our extensive experiments demonstrated that unigen outperformed prompting, facilitating light inference while preserving the domain generalizability of plms. although we explored an interesting framework for zero-shot, lightweight domain generalization, the performance of unigen appears weaker than those of baseline models that are trained on each domain in several cases. it is desirable to achieve a higher level of performance than those of the indomain baselines, which we will attempt in future work. to this end, the generation of small taskspecific data for additional training of the tam trained using unigen is a possible approach, especially when a downstream task domain is introduced. by employing tams that are pre-trained using unigen as a warm start, high performance could be achieved in the target domain with a small amount of task-specific data, which would reduce the total amount of data generated compared to that when individually training each tam by using zerogen or sungen from scratch. another possible approach may involve combining unigen with the concept of test-time learning @xcite . similar to the first strategy, it may generate small amounts of test domain-specific data given test-time data as in-context examples. we are committed to exploring these possible strategies, which will enhance the effectiveness of unigen.