S tyle D istance: Stronger Content-Independent Style Embeddings with Synthetic Parallel Examples (2025)

Abstract: Style representations aim to embed texts with similar writing styles closely and texts with different styles far apart, regardless of content. However, the contrastive triplets often used for training these representations may vary in both style and content, leading to potential content leakage in the representations. We introduce StyleDistance, a novel approach to training stronger content-independent style embeddings. We use a large language model to create a synthetic dataset of near-exact paraphrases with controlled style variations, and produce positive and negative examples across 40 distinct style features for precise contrastive learning. We assess the quality of our synthetic data and embeddings through human and automatic evaluations. StyleDistance enhances the content-independence of style embeddings, which generalize to real-world benchmarks and outperform leading style representations in downstream applications.

## introduction the most common objective when training text embeddings is to place texts with similar semantics close together in the embedding space @xcite . style representations, by contrast, aim to embed texts with similar writing styles near each other and texts with different styles far apart, regardless of their semantic content @xcite . embeddings are usually trained via contrastive learning, with triplets consisting of an anchor text, a positive text (which should be embedded closely to the anchor), and a negative text (which should be embedded far from the anchor) @xcite @xcite . existing approaches often use social media datasets with the assumption that all writing by the same author shares a similar style and that texts by different authors exhibit dissimilar styles @xcite . these methods also attempt to minimize content representation in the resulting embeddings. they select a text from the same author on a different topic as the positive example, and a text from a different author on the same topic as the negative example, approximating topic similarity using subreddit or conversation metadata. however, these methods are limited by the imperfect nature of data acquired under such assumptions. for example, the same author may write about the same topic even in different subreddits. as a result, these imperfect contrastive triplets do not explicitly control for content, leading to style embeddings with weak content-independence. the "content leakage" caused by such proxy objectives (illustrated in figure 2 ) can undermine the effectiveness of style representations in tasks that require strict separation between style and content, such as stylistic analysis, authorship tasks, style transfer steering, and automatic style transfer evaluation. to overcome this, a more controlled approach to style contrastive learning is necessary. in this paper, we introduce styledistance, a novel method for training stronger, contentindependent style embeddings which leverages synthetic parallel text examples generated by a large language model (llm) @xcite . by creating near-exact paraphrases with controlled stylistic variations, we produce positive and negative examples across 40 distinct style features. this synthetic dataset, which we call synthstel, enables more precise contrastive learning (visualized in figure 1 ) and is more robust to the content leakage inherent in existing datasets. we evaluate our method on both human and automated benchmarks, measuring the content-independence, quality, and utility of styledistance embeddings. in summary, our primary contributions are: style representations in previous work, style representations were learned from unlabeled texts @xcite @xcite . due to the lack of parallel datasets covering diverse style features, @xcite and @xcite have both recently used authorship as a proxy for style as we discussed previously. @xcite trained embeddings to uniquely represent different authors. although these embeddings capture features representative of authors' style, they also capture content features due to the lack of control for content-related aspects. @xcite trained lisa, a style vector created by annotating texts with their stylistic features using llms. lisa, however, trades off performance to create a vector with interpretable dimensions. the highest quality style representations to date come from approaches that employ a contrastive learning objective @xcite . in this paper, we propose leveraging the strengths of generative llms to build a dataset that will serve to train strong content-independent style embeddings using a contrastive learning objective. from a practical perspective, style representations are useful in downstream applications such as arbitrary text style transfer @xcite @xcite where they help steer and guide transfer, and content-independence is important for reducing hallucinations in generations. llms and text style llms are commonly used for style transfer @xcite @xcite and style analysis @xcite . their strength with style-related tasks has been demonstrated, yet leveraging this knowledge to build strong content-independent style representations has not yet been explored. ## data generation a core component of our proposed styledis-tance approach is a llm that generates a synthetic dataset with controlled content and style. the dataset is composed of pairs of sentences which are paraphrases of each other (i.e. their content is similar) but differ in style. each pair includes a positive example that showcases a specific stylistic feature (e.g., usage of active voice or emojis) and a corresponding negative example that lacks this feature. in this section, we outline the selected contrastive example used for wegmann et al. (2022) contrastive example used for styledistance ("usage of active voice" style feature) anchor: awesome game. took off from work to play it. halfway through the week playing nonstop. such a quality product. the spider-man game we've needed for years. i adored the dolce & gabbana mauve leather creation with a signature lock. positive: such a great spider-man game. it really is the best one i've played. easy to play crazy fun to master. there is so much you can do with peter's moves... same style same content (leads to content leakage in style representations) i observed the impact rising temperatures had on tropical diseases. same style different content negative: no android version? i guess i'm not getting it different style different content (leads to content leakage in style representations) the dolce & gabbana mauve leather creation with a signature lock was adored by me. ## styledistance we next describe how we trained styledistance embeddings using our synthetic dataset. ## evaluation we propose a direct evaluation of the quality of styledistance embeddings, and an evaluation of their utility in downstream applications. we use other leading style representations like lisa @xcite and the embeddings from @xcite as baselines, and compare them with our style embeddings on the stel and stel-or-content tasks @xcite . we also compare to the luar model @xcite , an authorship representation model which does not explicitly train content-independent representations but captures model formal complex numb3r c'tion emoji avg ## conclusion we introduced styledistance, a novel method for training content-independent style embeddings using synthetic parallel examples. by employing a large language model to generate a dataset of nearexact paraphrases with controlled style variations, we overcome limitations associated with content leakage and imperfect parallel examples in existing style embedding methods. evaluations using the stel and stel-or-content tasks demonstrate that embeddings trained solely on synthetic examples can capture style extremely well. additionally, the technique's ability to generalize to unseen style features indicates its potential to represent a broader range of style attributes beyond those addressed in synthetic data generation. notably, our results highlight the efficiency of large language models in creating task-specific representations. this approach circumvents the need for manual dataset collection and the reliance on weak implicit assumptions over data, offering a more direct and accurate method for training on the precise task-specific features of interest. model, data, and code we release the styledistance models, the synthstel dataset, and our code for other researchers to use at: https://huggingface.co/styledistance/ .