Challenging Assumptions in Learning Generic Text Style Embeddings (2025)

Abstract: Recent advancements in language representation learning primarily emphasize language modeling for deriving meaningful representations, often neglecting style-specific considerations. This study addresses this gap by creating generic, sentence-level style embeddings crucial for style-centric tasks. Our approach is grounded on the premise that low-level text style changes can compose any high-level style. We hypothesize that applying this concept to representation learning enables the development of versatile text style embeddings. By fine-tuning a general-purpose text encoder using contrastive learning and standard cross-entropy loss, we aim to capture these low-level style shifts, anticipating that they offer insights applicable to high-level text styles. The outcomes prompt us to reconsider the underlying assumptions as the results do not always show that the learned style representations capture high-level text styles.

## introduction learning meaningful representations of text has received much attention recently as models pretrained on large corpora have become the standard for extracting representations that capture prior knowledge (i.a. @xcite . however, most pre-trained models focus on general aspects, as they are trained with causal or masked language modeling objectives. therefore, they are suboptimal for nlp tasks focusing on a specific aspect, such as style. there are many notable nlp tasks, such as identifying chatbot-written text based on its style (e.g., soto et al. ( 2024 )) or style transfer where models rely on dedicated style representations (e.g., @xcite ). existing work on style transfer focuses on highlevel stylistic aspects such as sentiment @xcite or formality @xcite , taking style definitions based on the dataset's structure. @xcite view those high-level text style changes as compositions of more fine-grained, lowlevel style changes. generic style representations incorporating low-level aspects such as lexical, syntactic, semantic, and thematic stylistic traits @xcite but also high-level, composed stylistic traits might significantly improve style-focused tasks. based on the hypothesis that low-level style changes compose high-level style changes, this work explores learning generic, sentence-level style representations. we take a pre-trained encoder model producing general-purpose representations and fine-tune it to distinguish between low-level stylistic changes using contrastive learning and cross-entropy loss. we hypothesize that the resulting text encoder generalizes to high-level styles, applying the view by @xcite on how lowlevel stylistic changes compose high-level changes to representation learning for styles. we train our method using contrastive learning and cross-entropy loss on the styleptb dataset @xcite comprising low-level, finegrained style changes to obtain high-level, generic style embeddings. we evaluate our method by training a simple classifier on the representations of the learned style. the results show an ambiguous picture of the resulting style embeddings, challenging the underlying assumptions. style representations as a byproduct text style transfer is conducted by prominent models by 1 disentangling content and style to learn separate task-specific representations to control them independently @xcite @xcite @xcite . other text style transfer models learn content representations and multiple decoders (one for each style) @xcite . another group of models uses a structured style code to enforce a particular style in the decoder, either given as a structured code @xcite or learned @xcite . for text style classification, textcnn @xcite is the most widely used method @xcite . bert @xcite and roberta @xcite , fine-tuned for style classification, are strong baselines. in contrast, our proposed method learns style representations at the sentence level from low-level, linguistically motivated style changes. this is much more fine-grained and allows applications to unseen styles. ## method in the beginning god created… and kenan lived seventy years… won't go back best sandwiches ever! →← →← ←→ ←→ figure 1: our training objective pushes sentence representations of the same style close together. in this example, reviews (in orange) are pushed close together, and sentences of one bible version (in blue) are pushed close together, while the representations of different styles (bible vs reviews) are pushed to be far apart. assumption 1 low-level stylistic changes can be composed to form high-level styles in line with previous work @xcite . assumption 2 learning style embeddings can be achieved by contrasting different styles @xcite . we combine the aforementioned assumptions to come up with the following method. method the core idea-illustrated in figure 1 is to embed the data into a space where texts of the same style resemble each other, while texts of different styles are easily distinguishable. formally, we want to learn a neural encoder f mapping any two input sentences x and x onto vector representations f (x) and f (x) such that f (x) ≈ f (x) if and only if x and x are of the same style: style(x) = style(x) = s. to achieve this, we first form pairs of sentences, some equal and others of a different style. then, we use a contrastive objective to push the vector representations of sentences of the same style closer together and ones of different styles far apart. for a given sentence x with style(x) = s, our set of candidate sentences x cand contains sentences of the same style and different style(s): x @xmath0. we employ a simple architecture to compute the similarity between the outputs f (x) and f (x), namely the inner product f (x) ⊤ f (x). other functions to consider are, e.g., the cosine similarity or kernel functions. we compute the similarity using a simple architecture to avoid learning a rich similarity measure compensating for the encoder producing poor style representations. therefore, we have the following formulation for the probability of each candidate sentence @xmath1: our training objective is to maximize the probability of identifying all sentences @xmath2: we also experiment with a cross-entropy loss and its combination with a contrastive loss. ## experiments in this section, we describe how we evaluated the effectiveness in learning style representations for multiple styles. ## conclusion learning generic, high-level text style representations from low-level, linguistically motivated changes and generalizing to high-level styles according to assumption 1 by @xcite using contrastive learning according to assumption 2 by @xcite presents challenges. although the approach does not yield the expected results using contrastive learning, cross-entropy loss shows improvements for some settings. however, our approach does not yield the expected results compared to previous work @xcite using contrastive learning to learn generic style representations.