Contextualizing Variation in Text Style Transfer Datasets (2021)

Abstract: Text style transfer involves rewriting the content of a source sentence in a target style. Despite there being a number of style tasks with available data, there has been limited systematic discussion of how text style datasets relate to each other. This understanding, however, is likely to have implications for selecting multiple data sources for model training. While it is prudent to consider inherent stylistic properties when determining these relationships, we also must consider how a style is realized in a particular dataset. In this paper, we conduct several empirical analyses of existing text style datasets. Based on our results, we propose a categorization of stylistic and dataset properties to consider when utilizing or comparing text style datasets.

## introduction the general task of text style transfer involves rewriting source content in a target style. currently, there are a number of text style transfer tasks with available data, such as formality @xcite , bias @xcite , sentiment @xcite , humor or romance @xcite , offensiveness, (nogueira dos @xcite , authorship or time period @xcite , and personal attributes @xcite . while these specific tasks are often modeled in isolation, the general task definition remains consistent. as such, a natural question arises of what the relationship is between the stylistic variation of specific tasks. stylistic variation can arise from a number of factors such as communicative intent, topic, and speaker-receiver dynamics @xcite , yet within the task of text style transfer, our view of a style is constrained to the context of each specific dataset. therefore, understanding the tasks as well as the relationships between different tasks requires considering the stylistic properties and potential contextual and social factors @xcite underpinning them, as well as the dataset characteristics @xcite and intersection of influences giving rise to the realization of style within a dataset. from an application standpoint, considering these influences can provide a more comprehensive understanding of important textual features. there is a body of work already looking at how to identify generic features to increase target task performance @xcite or to compute similarity of textual features to select data for transfer learning @xcite . in the context of text style transfer, these approaches first require understanding what features should be shared across tasks. for example, @xcite leveraged the stylistic features shared between grammatical error correction data and formality to increase model performance on formality transfer datasets. in addition to textual features such as stylistic properties, existing work also suggests that context of dataset creation should be taken into account when identifying compatible data or assessing possible out-of-distribution generalizability. for example, the similarity between how sentiment information is reflected in different domains affects adaptation performance @xcite , and many models can achieve high performance on natural language inference tasks through task-limiting annotation artifacts @xcite . in other words, factors such as data source and annotation method can create underlying textual features that can impact performance and limit generalizability. thus, in combination, these existing works on leveraging inherent stylistic similarities @xcite or similar stylerepresentations in different dataset domains @xcite , as well as identifying task-limiting dataset properties @xcite al., 2018 ) indicate that analysis of both stylistic properties and dataset characteristics, as well as the potential interdependencies between them, is warranted. in this paper, we consider two primary categories of textual variation within the context of text style transfer: stylistic characteristics and dataset characteristics. we perform a series of empirical analyses to demonstrate the visible influence of both style and dataset characteristics on the performance of text style transfer models. then, we present a categorization of style and dataset properties for consideration when utilizing or comparing style transfer datasets. finally, we discuss the downstream applications for contextualizing variation in text style datasets, including multi-task learning, data selection, and generalizability. our work and suggestions fall within the context of and align with recent work on incorporating social factors in natural language processing systems @xcite and characterizing datasets @xcite . ## empirical analyses as an exploratory step, we question whether we can distinguish differences arising from style or dataset properties when comparing empirical results across datasets. we identify a set of aligned english datasets used for supervised text style transfer that exhibit differences ranging from style, annotation method, and domain. we further restrict our selection to datasets in which a single stylistic attribute is transferred between classes. specifically, we look at gyafc-em & gyafc-fr @xcite , shakespeare @xcite , biased-word (bias) @xcite , fluency @xcite , and flickr @xcite . we provide dataset overviews in first impression of data: of the six datasets, four were manually annotated and two were automatically annotated. for manually annotated datasets, gyafc-em and gyafc-fr utilized crowdsourced rewrites, flickr utilized crowdsourced sentences with only visual context shared between annotators, and fluency utilized expert annotations of the target attribute. both automatically annotated datasets (bias, shakespeare) were created through identification of existing data sources. while each style task is unique (other than two domains of gyafc for formality), in terms of style we observe that shakespeare has a significantly different temporal context than all other datasets, and fluency involves a stylistic attribute that, ideally, the sentence pairs in all other datasets should possess. 1 beyond our qualitative observations, we perform an exploratory multi-task learning experiment, described in the following subsection. ## variation from style and data properties our empirical analyses demonstrate the visible influence of both style and dataset properties on how a style is represented in a given dataset. in addition to brief mentions of influences of dataset creation in section 1, we can identify an intuitive reason for these dual influences. while linguistic approaches exist to analyze textual variation (halliday and matthiessen, 2013; @xcite , we suggest that the processes of linguistic-based stylistic analysis and text style transfer typically occur in inverse directions: linguistic analysis may work from human-written text and then analyze stylistic variation, whereas text style transfer may work from pre-existing ideas of targeted stylistic variation and then create datasets of human-written text that meet stylistic expectations. in other words, to create a text style transfer dataset or train a text style transfer model, the researcher should have a notion of the desired style against which to judge the resulting artifact. intuitively, this process can lead to process-attributable we contextualize both style and dataset properties within language and sociocultural context as all language is implicitly reflective of these influences @xcite . based on our results and observations, we consider stylistic properties as properties influencing textual variation that are inherent to a particular style and dataset properties as factors influencing textual variation due to how a particular dataset was created. we detail style and dataset properties in the following subsections and visualize the major distinctions in figure 1 . ## interplay between style and data properties bender and friedman (2018) proposed data statements for documenting dataset contextual factors such as language variety, speaker demographics, annotator demographics, speech situation, and text characteristics (e.g. genre, topic). the style and dataset properties we discuss as potentially contributing to variation in text style transfer datasets show some alignment with those proposed for data statements as such factors contribute to linguistic variation in a general sense. however, our categorization specifically operates within the context of text style transfer datasets for which there are unique considerations and important distinctions between sources of variation and downstream implications or applications. in the previous subsections, we discussed style properties and dataset properties to which variation in text style transfer datasets can be attributed. in this section, we discuss the interdependence of style and data properties in text style transfer datasets in terms of context-dependence of and interactions between sources of variation. style and data property interactions while we previously considered the potential impact of both style and dataset characteristics independently, these characteristics may have underlying interactions and influences on one another. specifically, certain types of stylistic properties may be more or less amenable to certain dataset creation methods or sources, and vice versa. with regard to the stability of stylistic properties, dataset properties such as annotation method may be indirectly influenced when transferring across relatively stable stylistic properties. for example, machine translation models have been found to exhibit stylistic bias through reflecting demographically-biased training data @xcite . while this demonstrates that the demographics of annotators can serve as an important dataset characteristic, it also demonstrates the potential to transfer across relatively stable stylistic properties, such as personal attributes @xcite . however, as the stylistic properties are inherent to the annotator, there may be constraints on dataset creation through manual data annotation, such as potential limitations and additional considerations for using methods such as human judgments. this underscores additional considerations for and potential challenges of selecting data from two styles that may have underlying influences on how datasets are constructed. context-dependence of variation relatedly, contextual considerations come into play with respect to the the shakespeare to modern english style transfer task, a dataset also reflective of transfer across stable, contextual boundaries. the shakespeare to modern english transfer task can be considered as transferring across temporal context, or as the characteristic style of a single author @xcite . in this case, while an influence of sociocultural context is apparent when considering the original data sources, the targeted stylistic variation occurs across such context boundaries. thus, source of variation for textual features arising from external context lies with whether the intent is present for a dataset to represent a transfer across context boundaries, rather than an artifact reflecting specifics of dataset creation. this is illustrated in figure 1 as a dashed line connecting style type to dataset properties. with further regard to dataset creation, it is important to acknowledge that while we consider many properties arising from social influences as dynamic and variable influences giving rise to particular styles, a dataset will indirectly and inadvertently reflect such social context during creation to some degree. as such, we also must consider social factors not related to the actual targeted style, but rather arising from the dataset creation process. as an example of this consideration, we can't simply say two sentiment datasets from the same general domain (such as restaurant reviews) are equivalent if one was constructed with reviewers who had anonymity (in a sense mitigating some of the direct social pressure or influence) and the other was constructed with reviewers who were not anonymous and were thus subject to increased social pressure. by understanding both data and style differences and their interactions within a particular context, these potential differences or hidden influences can be more easily identified. in summary, the interactions between style and data properties are complex. while we have suggested interactions between context and sources of influence, there are likely correlations that exist based on sources of variation which future work can investigate. ## influences and applications in the previous sections, we demonstrated visible influences of style and dataset properties on performance, categorized a set of style and dataset properties for consideration, and discussed the potential interactions between sources of variation. we conclude by discussing several applications of understanding the sources of variation in text style transfer datasets. specifically, we look at multi-task learning, domain adaptation, and generalizability. ## conclusion in this paper, we conducted a set of exploratory analyses to assess the visibility or influence of both style and dataset characteristics on text style transfer. based on these observations, we proposed a categorization of stylistic and dataset properties that can contribute to variation in text style transfer datasets and described the applications in which these properties may be influential, limiting, or leveragable.