Textual Dataset Distillation via Language Model Embedding (2024)

Abstract: Dataset distillation is a process aimed at condensing datasets while preserving essential characteristics. In the text domain, prevailing methods typically generate distilled data as embedding vectors, which are not human-readable. This approach simplifies optimization but limits the transferability of distilled data across different model architectures. To address this limitation, we introduce a model-agnostic, data-efficient method that leverages Language Model (LM) embeddings. Compared to parameter-efficient methods such as LORA, our approach achieves comparable performance with significantly faster processing times. We evaluate our methodology through classification tasks on datasets like IMDB and AG-News, demonstrating performance that is on par with or exceeds previous model-dependent techniques. By utilizing LM embeddings, our method offers enhanced flexibility and improved transferability, expanding the range of potential applications.

## introduction recent advancements in natural language processing (nlp) models have been significantly driven by the availability of large-scale datasets @xcite . it has been widely observed that larger datasets often lead to better performance. however, there is a growing recognition that "small and high quality data" can be more effective than simply "larger but low-quality data" @xcite . additionally, training nlp models on large-scale datasets can be remarkably time-consuming and resource-intensive, presenting a substantial challenge in the field. this has spurred ongoing research efforts to optimize the efficiency of model training while ensuring the data quality and integrity. ## experiments configuration embedding model. mteb @xcite provides a comprehensive benchmarking on a series of embedding models' performance on different nlp tasks. generally, the dimension of embedding model correlates to the model's representative ability. in order to understand the embedding model's performance on the quality of distilled dataset, we picked the following embedding models with different output dimensions: glove foot_0 (@xmath0, e5base-v2 foot_1 (@xmath1, text-embedding-ada-2-002(@xmath2. training of vec2text model. the vec2text model functions primarily as a decoder and requires training across various embedding models. following @xcite , we initialize the model from t5-base and fine-tune with domain-specific training data. we set the maximum sequence length as 128 to avoid model deterioration. to ensure the vec2text model's efficacy, we employ the bleu score to access similarity between the original and the reconstructed text. each vec2text model is fully trained until the bleu score is converged on testing data. downstream tasks and datasets. we evaluate the proposed method with the text classification task, focusing on the imdb foot_3 and ag-news foot_4 datasets. imdb is a dataset for binary sentiment classification, featuring 50,000 movie reviews labeled as positive or negative. ag-news comprises news articles across four categories, aiming for topic classification. evaluation. our goal is to synthesize a dataset that maintains its effectiveness across a diverse downstream models, thereby ensuring robustness and model-agnostic performance. to this end, we have conducted evaluations using different downstream models, including logistic regression (lr), naive bayes (nb), support vector machine (svm), more advanced neural network models including customized textcnn, textrnn and pre-trained language models, including distil-bert @xcite and t5-base @xcite . for input text vectorization, lr/nb/svm utilize tf-idf, whereas textcnn/textrnn employ glove-100d embeddings, and distilbert/t5-base leverage their own tokenizer and embeddings. utilizing the distilled dataset, we trained these downstream models to full capacity and assessed their performance by measuring accuracy on the original test dataset. higher accuracy indicates that the distilled data retains a high level of informational content. for each result, we conducted 10 experiments and reported the average accuracy along with its standard deviation. more detail about the model architectures are in appendix a. baseline. there are limited works in the domain of textual dataset distillation, and previous work @xcite are all model-specific. even though it's not a fair comparison, we still include it to better understand whether the proposed method can compete with the model-specific ones. meanwhile, we also add random sampling as a naive baseline. overall, we have the following baselines: other implementation details are in appendix b.1. ## results and discussion main results. we generated distilled dataset with a size of 0.1% of the full data, and evaluated the accuracy with different down-streamed models. the accuracy attained using full datasets establishes the upper performance limit for each architecture. text-embedding-3-large achieve the highest averaged accuracy. compared to the accuracy on full data, we recover the accuracy by 79.2% on imdb and 89.9% on ag-news. while the random sampling only recovers 64.6% on imdb and 78.9% ag-news. compared to the model-specific method tdd, our method is worse on imdb by 4.2% but better on ag-news by 2.9%. ## limitation if we revisit the text distillation problem, our proposed framework introduces an encoder-decoder architecture to utilize the existing powerful embedding model. similar to the image compression problem with a encoder-decoder model, the meaningful and valuable information are condensed on a manifold in the high dimensional embedding space. the original data are discretely distributed on the manifold, and the sampling process is picking "synthetic" embedding vectors that contains more information. then, in order to generate transferable textual data, the vec2text model reconstructs plain-text data with these embedding vectors. from an informational perspective, the sampling stage represents a process of compressing information, whereas the vec2text phase involves the loss of information to facilitate transferability. while our proposed method demonstrates significant potential in dataset distillation for nlp tasks, it is not without its challenges. extending our approach to tasks beyond classification, such as semantic text similarity (sts), requires a nuanced redesign of the sampling and vec2text inversion stages due to the complex relationships inherent in sts data formats. moreover, crafting a versatile vec2text model that is both high-quality and broadly applicable across different tasks remains a demanding endeavor, constrained by the specificity of task and dataset requirements. despite these limitations, our work opens up promising avenues for future research, notably in exploring the use of encoder-decoder models like t5 as universal vec2text solutions, and in integrating our framework with bi-level optimization methods for enhanced dataset distillation. these directions not only hold the promise of overcoming current constraints but also advancing the efficiency and applicability of dataset distillation in nlp. ## conclusion we introduced a novel dataset distillation framework for textual data that utilizes language model embeddings to create compact, highly informative datasets. our experiments demonstrate that the distilled text maintains high accuracy in classification tasks and is compatible with various downstream architectures. ablation studies highlight critical factors affecting distillation quality, such as the embedding model quality, task-specific prompts, and vec2text model training.