VAEGPT -Sim: Improving Sentence Representation with Limited Corpus Using Gradually-Denoising VAE (2024)

Abstract: Text embedding requires a highly efficient method for training domain-specific models on limited data, as general models trained on large corpora lack universal applicability in highly specific fields. Therefore, we have introduced VAEGPT-Sim, an innovative model for generating synonyms that combines a denoising variational autoencoder with a target-specific discriminator to generate synonymous sentences that closely resemble human language. Even when trained with completely unsupervised settings, it maintains a harmonious balance between semantic similarity and lexical diversity, as shown by a comprehensive evaluation metric system with the highest average scores compared to other generative models. When VAEGPT-Sim is utilized as a module for contrastive learning in text representation, it delivers state-of-the-art results in small-dataset training on STS benchmarks, surpassing ConSERT by 2.8 points. This approach optimizes the effectiveness of text representation despite a limited corpus, signifying an advancement in domain-specific embedding technology.

## introduction text representation is vital for nlp tasks such as clustering, classification, and similarity comparison @xcite . the resource of corpus labeling and the inadequacy of public datasets for certain domains make unsupervised learning with unlabeled data a preferred method. text embeddings are generated either through word vector methods like word2vec, glove, and fasttext @xcite @xcite , or via pre-trained language models such as bert, infersent, and sentence-bert @xcite that capture sentence context. the latter, though, often lack semantic precision due to vector space anisotropy @xcite . techniques like bert-flow, whitening, consert, simcse, dif-fcse, and infocse have been developed to enhance embeddings by clustering similar sentences and separating dissimilar ones @xcite @xcite @xcite . however, those top-performing models in sts benchmarks typically require retraining on datasets larger than 1,000,000 sentences. however, field-specific corpora contain unique specialized words, professional phrases, common expressions, and even distinct text styles. in such real scenarios, it's questioned if a general model trained on a vast corpus can deliver satisfactory results. to address this, we construct a dataset of similar sentences from the biomedical environment domain, using pubmed professional sentences. we then evaluate the text embedding quality of different unsupervised trained bert base ( as a solution, we propose a novel generation model vaegpt-sim for synonymous data augmentation to train a better text representation model with small dataset and limited resource. it combines gradually-denoising vae and selfjudgment loss with the powerful generative model gpt. therefore, it produces synonym sentences that closely resemble human writing during contrastive learning, keeping semantic similarity and lexical diversity in generation. when we insert vaegpt-sim into the framework of comparative learning (generate-cse) as a core module, it is sufficient to generate rich and high-quality synonyms for the same input sentence among multiple training batches. compared with bert2bert, bart, t5, and gpt2 @xcite @xcite , our vaegpt-sim achieves the best result: with training on around 70,000 unlabeled sentences, bert base trained by generate-cse-vaegpt-sim achieves an impressive average wmean spearman's correlation of 77.29 on 6 sts tasks. it outperforms all previous models trained on similar-scale in-domain datasets, as well as even approaches the performance level of simcse, trained on a dataset 14 times larger. the solution we recommend is aimed at providing an effective solution for real-world industrial applications. first, in practical scenarios, the text corpora for each task vary significantly in domain, specialty, and style, making general models unsuitable. thus the practical approach is to construct a specialized text representation model for each task based on the limited available data (only the data to be dealt with or historical data from the same source). as a result, by innovatively creating a universal synonym generation model and integrating it into a contrastive learning framework, we enables the rapid construction of specific text representation models with limited in-domain data and resources. besides, our proposed lightweight synonym generation solution is more suitable for various industrial scenarios, overcoming limitations related to data security and privacy concerns associated with the use of public llms, and the high cost of training and running self-made llms in business-related fields. consequently, vaegpt-sim significantly improves bert's text embedding. this solution is valuable for achieving text representation in resource-limited, data-limited industry settings in numerous specific domains, with a comparable results to models trained on much larger datasets. ## conclusion to address the industry's need for small-scale data training in specific domain text embedding models, we propose an innovative generation model vaegpt-sim to obtain diverse positive synonymous sentence pairs in contrastive learning process for text representation model. bert trained with vaegpt-sim surpasses models trained on similarscale data and achieves remarkable performance in sentence embedding on sts tasks. it can even rival larger-scale models like simcse.