GASE : Generatively Augmented Sentence Encoding (2025)

Abstract: We propose a training-free approach to improve sentence embeddings leveraging test-time compute by applying generative text models for data augmentation at inference time. Unlike conventional data augmentation that utilises synthetic training data, our approach does not require access to model parameters or the computational resources typically required for fine-tuning state-of-the-art models. Generatively Augmented Sentence Encoding variates the input text by paraphrasing, summarising, or extracting keywords, followed by pooling the original and synthetic embeddings.Experimental results on the Massive Text Embedding Benchmark for Semantic Textual Similarity (STS) demonstrate performance improvements across a range of embedding models using different generative models for augmentation. We find that generative augmentation leads to larger performance improvements for embedding models with lower baseline performance. These findings suggest that integrating generative augmentation at inference time adds semantic diversity and can enhance the robustness and generalisability of sentence embeddings for embedding models. Our results show that performance gains depend on the embedding model and the dataset.

## introduction representation learning has emerged as a fundamental technique in natural language processing (nlp). however, the quality and robustness of the embeddings are highly dependent on the richness and diversity of the training data. recent advancements of generative large language models (llms) led to remarkable capabilities in generating human-like text. llms were also used for data augmentation by @xcite , who applied paraphrasing techniques to train a bert model @xcite . @xcite showed that llm-generated paraphrases are harder to detect for humans than paraphrases generated with simple techniques like synonym replacement. the integration of generative models with representation learning has been explored by augmenting generative models, e.g., in retrieval-augmented generation @xcite . also aiming to propose a training-free method, @xcite introduced meta-task prompting, which employs use-case-specific prompts to generate multiple embeddings via a generative model, after which the embeddings are pooled. for retrieval systems, @xcite and @xcite described query expansion approaches using generative models that generate synthetic documents as a response to a retrieval request. extending this work, gase combines generative and embedding models without any assumptions on the type of models (unlike meta-task prompting), nor is it limited to augmentation of retrieval queries such as @xcite and @xcite . more recently, geneol @xcite , a related approach, was introduced (see ## results llama-3 (+6.49pp), bert (+6.11pp), and glove (+2.96pp). paraphrase augmentation worked best for most models; however, glove improved most with keyword extraction, while bert, mpnet, and llama-3 excelled with summarisation. to validate our approach we compared the augmentation with keyword extraction using llms to randomly extracting keywords foot_0 which demonstrated that llms extract semantically more meaningful keywords ( evaluating the different generative models for paraphrase augmentation we find that using a single generative model gpt-3.5 turbo performed best for all embedding models, except glove, voyage-large-2, and llama-3 ( applying more than one generative model for augmentation yielded higher scores than a single model (see ## discussion we believe that embedding models with lower baseline performance benefited more from the semantic diversity induced by augmentation due to their inherent lower capability to model diverse semantics. similarly, they gained more from the ensemble effect using @xmath1. our extension to 11 non-english languages and the pc and ir tasks did confirm the effectiveness of gase and also showed a negative association between base- the effectiveness of keyword extraction for glove, bert, and llama-3 may be explained by the model's limited capability to handle complex semantics. therefore, the reduction to keywords might help the model to reduce noise. longer texts likely caused summary augmentation to outperform paraphrase augmentation on sts22, as summarising significantly reduced the length. on other datasets, however, summaries were only slightly shorter, diminishing its augmen- our results show pooling original and synthetic embeddings outperforms individual use (figure 3 ). across datasets, tasks, and languages, gase particularly benefits embedding models with lower baseline performance and shorter texts, while summary augmentation is more effective for longer texts ( ## conclusion we propose gase, which augments sentence encoders at inference with generative models for paraphrasing, summarising, or keyword extraction. on mteb sts, gase significantly improves lowerperforming embedding models and incrementally enhances sota models. gase applies broadly, needing only embedding and generative model outputs. extensions to 11 non-english languages and pair classification and information retrieval as additional tasks, demonstrated consistent findings.