Balancing the Effect of Training Dataset Distribution of Multiple Styles for Multi-Style Text Transfer (2023)

Abstract: Text style transfer is an exciting task within the field of natural language generation that is often plagued by the need for high-quality paired datasets. Furthermore, training a model for multi-attribute text style transfer requires datasets with sufficient support across all combinations of the considered stylistic attributes, adding to the challenges of training a style transfer model. This paper explores the impact of training data input diversity on the quality of the generated text from the multi-style transfer model. We construct a pseudo-parallel dataset by devising heuristics to adjust the style distribution in the training samples. We balance our training dataset using marginal and joint distributions to train our style transfer models. We observe that a balanced dataset produces more effective control effects over multiple styles than an imbalanced or skewed one. Through quantitative analysis, we explore the impact of multiple style distributions in training data on style-transferred output. These findings will better inform the design of style-transfer datasets.

## introduction multi-style text transfer is a challenging task today with applications such as automatic domainappropriate, style-conformant writing @xcite and ai-assisted stylistic language editing. text style transfer is an intricate task as all language has a specific context, and those contexts influence the attributes of the language @xcite . text style transfer is challenging because it involves dealing with the aspects of style coupled with the textual content @xcite @xcite . this domain's other obstacles include the need for parallel corpus @xcite and quality training data. as the number of style dimensions increases with multi-style text transfer, not only is the requirement of a jointly annotated corpus across all the figure 1 : when an input sentence is passed to the multistyle transfer model, to increase formality and decrease arousal, we hypothesize that when the model is trained on a balanced joint distribution of formality and arousal (all four style combinations have a 25% representation)the style transfer is more successful as opposed to when the model is trained on a skewed joint distribution (there is no representation of the "informal unaroused" style combination) of styles in the training data. much research exists on models implementing multi-style text transfer and interdependency of micro-styles @xcite , 2020; @xcite . however, there needs to be more exploration of the joint distribution of inherent micro-styles in the style transfer training dataset and how these micro-style distributions are related. therefore, we pose a question -can a dataset with minimal variance across multiple micro-style combinations, such that it experiences a "balancing effect", lead to a better style transferred output ? figure 1 illustrates our intuition that a dataset that experiences a "balancing effect" will have more control over the multi-style transferred output than a "skewed" dataset. suppose the style transfer model sees examples of every style combination that can exist -this could aid in the style generation of even unlikely combinations of styles compared to a skewed distribution of these joint micro-styles. in this research, we consider a multi-style text style transfer pipeline assuming that the user has no access to parallel data or the style of the original text that he wishes to transfer, as would seem natural for a style language editing application. we introduce the changing of the training dataset microstyle joint distributions in such a pipeline and quantitatively explore the impact of this modification on the style transferred output. we perform a set of empirical analyses to demonstrate the influence of joint distributions on style-transferred output and show how this trend varies as the number of micro-styles considered changes. the 'balancing effect' on a training dataset leads to style transferred sentences from even the joint style combinations that are typically rare ("informal unbiased and unaroused"). our study is the first of its kind on the distribution of micro styles in training datasets for multi-style text style transfer and is likely to have implications for designing datasets for multi-style transfer model training and fall within the context of and align with recent work on characterizing datasets and factors impacting style transfer @xcite @xcite @xcite . ## multi style transfer pipeline datasets: we chose four micro-styles from the style hierarchy defined in troiano et al.: formality, arousal, sentiment, and bias, for our study and used publicly available nlp datasets built by other researchers @xcite @xcite @xcite to develop and test our models. appendix a mentions the details of the datasets and their usage. pipeline overview: our experimental setup for multi-style transfer is inspired by the work of @xcite . like them, we first generate a "diverse" paraphrase of the input sentence, and then the paraphrased sentence is rewritten in the style of choice. towards this end, we train a paraphrasing model (separately on a parallel paraphrase dataset). then, the trained paraphrase model is used to create "pseudo-gold" parallel data for training style models. first, we adopted a pre-trained t5 model @xcite to generate paraphrases. this model was trained for the task of paraphrase generation on the paranmt-filtered dataset provided by @xcite . once we had this trained paraphrase model, we used diverse beam search @xcite to generate diverse fluent paraphrased outputs. an important assumption is that the paraphrase is stripped of its original style and does not leak into the training. we address this potential issue by training classifiers @xcite to predict style on the original and paraphrased datasets and find that all our micro-style classifiers have a classification accuracy of higher than 80% f1, which is acceptable for pseudo-label creation. after we generate diverse paraphrases, we choose the most diverse paraphrase and then derive micro-style classifications for the paraphrased sentence using our trained micro-style classifiers. therefore each sentence is assigned a classification score for each micro-style label and can form a "pseudo parallel" dataset for training the t5-based joint transfer model. thus, our approach does not need a parallel dataset. we then converted the classifier predictions into buckets of style (ranging from "very low" to "very high") based on the chosen style of the original and then paraphrased sentences. the bucketing process is described in appendix b. after this step, we introduce our contribution of "constructing style distributions" into the pipeline, as illustrated in figure 2 . following that, we perform multi-style text style transfer. we appended the "bucket" information to the paraphrased sentence to achieve the necessary intensity transfers, as motivated by the original t5 paper @xcite . we train t5-based style transfer models, where the paraphrased sentence and its style buckets are used as input parameters, while the style buckets assigned to the anchor sentence are used as proxy levels of output style transfer. all model-specific details are provided in appendix b. for generating sentences from our trained models, we used beam search @xcite and nucleus sampling @xcite and chose the top 3 sentences from the generations. the following is an example of the input to the joint style transfer model and the expected output. thus, we implemented a multi-style transfer pipeline to test our hypothesis without any finicky modeling paradigms popular in style transfer research, such as variational inference or autoregressive sampling @xcite . ## experimental results and discussion evaluation metrics: style transfer accuracy metrics quantify how nicely output texts match the desired style. however, more than this metric is required. motivated by jin et al., we evaluate style transfer across the three main properties of text style transfer: style transfer accuracy, content preservation, and fluency. we use our custom joint sequence classification models, implemented using huggingface libraries @xcite to evaluate the style transfer success ratio. our definition for the style transfer success s c is the total number of matches between intended and transferred style buckets, divided by the total number of samples. to judge content preserved in style transferred text, we use three metrics: bleu @xcite , embedding-based similarity @xcite using cosine similarity of two sentence embeddings @xcite , and word mover's distance (wmd) @xcite . for fluency, we use measures like perplexity using gpt2 @xcite and an adversarial classifier using the cross-aligned autoencoder model @xcite . experimental setup: in this paper, we illustrate different micro-style combinations in the training data, for a randomly selected case, with each combination in both the "balanced" and "skewed " settings. therefore, we consider 6 cases respectively: 1) formality and arousal in a balanced setting (fa balanced) 2) formality and arousal in a skewed setting (fa skewed) 3) formality, arousal and bias in a balanced setting (fab balanced) 4) formality, arousal and bias in skewed setting (fab skewed) 5) formality, arousal, bias and sentiment in the balanced setting (fabs balanced) 6) formality, arousal, bias and sentiment in skewed setting (fabs skewed). we construct the training data with the appropriate settings and then pass them through our experimental pipeline (illustrated in figure 2 ) and quantitatively evaluate the style transfer results. discussion: figure 3 illustrates that the balanced setup always has a higher success percentage of style transfer (s c ) than the skewed setup. we cannot compare the success percentage across cases because matching the exact target and transferred style buckets becomes difficult as the number of micro-styles increases. we can also observe through in figure 4 , we compare the difference in representation percentage of specific style combinations in the test sample for a specific case where we consider formality, arousal, and bias micro-styles. we observe that a balanced joint distribution leads to more representation in the style combinations that are less likely to occur. this is further accentuated as micro-styles increase, as reported in style transfer goal input text balanced transferred text skewed transferred text ↑ formality ↑ arousal wouldn't it be great if trump went 3rd party and sucked away millions of republican votes lol it wouldn't be nice if trump went to the third party and swooped millions of republican votes. would it not be great if trump went 3rd party and sucked away millions of republican votes? ↑ formality ↓ arousal i didn't know what happiness was until i got married. but by then it was too late. until i got married, i didn't even know what happiness was. i did not know what happiness was till i got married and it was too late. ↓ formality ↑ arousal did you hear about the soldier with 8 limbs? he was army he's an army soldier with 8 legs? did you hear about the soldier with 8 limbs in the army? ↓ formality ↓ arousal yeah, i don't understand all the hate. yeah i'm not gonna understand the hate. yeah i do not understand all the hate. appendix c. in figure 4 , we see that rarer style combinations [ibn, fun, iun] show more representation in the balanced case as compared to the skewed case. this supports our intuition that the style transfer model benefits from learning the representation of all possible style combinations that can occur together. when we consider formality, arousal, and bias micro styles together, the most represented category (30% of samples) is "formal unbiased aroused" (fue). the least represented category (as unlikely to occur together) is "informal unbiased unaroused" (iun) with 1%. we observe that the quantitative evaluation metrics are quite indicative when compared across style combinations. for instance, in we do not claim that our method of balancing multiple styles will work even for entangled microstyle combinations, as that is out of the scope of the current paper. however, balancing considerably affects the multi-style transfer output for the range of micro-style combinations we considered, and that has an application in many nlp tasks. this result could hugely influence future studies exploring better ways to balance even the entangled micro-styles. ## conclusion multi-style text style transfer is a challenging problem predominantly plagued by the need for jointly annotated high-quality datasets. there is a clear need for more research about the marginal and joint distribution of inherent micro-styles present in the training dataset used for style transfer. multi-style text-style transfer typically requires access to large, jointly labeled datasets and many computational resources under typical implementations. more in this paper, we implement a multi-style transfer pipeline that subverts the requirement of a jointly annotated dataset of multiple styles by constructing a pseudo-parallel dataset to which we introduce our contribution of constructing style distributions. we then use the modified pseudo-parallel datasets for multi-style transfer. our modified pipeline effectively allows us to understand the importance of the joint distribution of micro styles in training data and is a substantial contribution. we quantitatively explore the impact of joint micro-style distributions in the training dataset on the style-transferred output sentences. when the joint micro-style distributions are balanced, there is more control over style-transferred output than with a skewed distribution. these findings will likely inform the design of multi-style transfer datasets and encourage us to explore the micro-style relationships in our datasets.