Same Author or Just Same Topic? Towards Content-Independent Style Representations (2022)

Abstract: Linguistic style is an integral component of language. Recent advances in the development of style representations have increasingly used training objectives from authorship verification (AV)”:” Do two texts have the same author? The assumption underlying the AV training task (same author approximates same writing style) enables self-supervised and, thus, extensive training. However, a good performance on the AV task does not ensure good “general-purpose” style representations. For example, as the same author might typically write about certain topics, representations trained on AV might also encode content information instead of style alone. We introduce a variation of the AV training task that controls for content using conversation or domain labels. We evaluate whether known style dimensions are represented and preferred over content information through an original variation to the recently proposed STEL framework. We find that representations trained by controlling for conversation are better than representations trained with domain or no content control at representing style independent from content.

## introduction linguistic style (i.e., how something is said) is an integral part of natural language. style is relevant for natural language understanding and generation @xcite as well as the stylometric analysis of texts @xcite . applications include author profiling @xcite and style preservation in machine translation systems @xcite . while authors are theoretically able to talk about any topic and (un-)consciously choose to use many styles (e.g., designed to fit an audience @xcite ), it is typically assumed that there are combinations of style features that are distinctive for an author (sometimes called an author's idiolect). the cav task is to match a 1 with the utterance a 2 that was written by the same author. contrary to the traditional authorship verification task (av), this is complemented by a third "constrastive" utterance that was written by a different author @xcite . in addition to the cav variation to av, we experiment with content control (cc) by selecting b and a 1 to have the same approximate content with the help of a topic proxy. as topic proxies we use conversation and domain information. based on this assumption, the authorship verification task (av) aims to predict whether two texts have been written by the same author @xcite @xcite . recently, training objectives based on the av task have been used to train style representations @xcite @xcite . training objectives on av are especially promising because they do not require any additional labeling when author identifiers are available. similar to the distributional hypothesis, the assumption underlying the av training task (same author approximates same writing style) enables extensive self-supervised learning. style and content are often correlated @xcite : for example, people might write more formally about their professional career but more informally about personal hobbies. as a result, style representations might encode spurious content correlations @xcite , especially when their av training objective does not control for content @xcite . current style representation learning methods either use no or only limited control for content @xcite or use domain labels to approximate topic @xcite . @xcite work with 24 domain labels (here: product categories) for more than 100k amazon reviews to improve generalizability. however, using a small set of labels might be too coarse-grained to fully represent and thus control for content. in this paper, we use "content" and "topic" to refer to different concepts. we assume same content (fulfilled if two utterances are paraphrases of each other) implies same topic (e.g., two utterances that discuss personal hobbies), while same topic does not necessarily imply same content. approach. we introduce two independent variations to the av task (see figure 1 ): adding a contrastive sentence (cav setup) and addressing content correlation with a topic proxy (cc). we train several siamese bert-based neural networks @xcite to compare style representations learned with the new variations to the av task. we train on utterances from the platform reddit but our approach could be applied to any other conversation dataset as well. while previous work mostly aimed for learning representations that represent an author's individual style @xcite @xcite , we aim for general-purpose style representations. as a result, we evaluate the generated representations on (a) whether known style dimensions (e.g. formal vs. informal) are present in the embedding space (section 4.2) and preferred over content information (section 4.3) and (b) whether sentences written by the same author are closer to each other even when they have different content (section 4.1). ## style representation learning we describe the new contrastive authorship verification setup (cav) and our approach to content control (cc) in section 3.1. then we describe the generation of training tasks (section 3.2) and the hyperparameters for model training (section 3.3). ## evaluation we evaluate the learned style representations on the authorship verification task (i.e., the training task) in section 4.1. then, we evaluate whether models learn to represent known style dimensions via the performance on the stel framework @xcite in section 4.2. last, we evaluate representations on their content-independence with an original manipulation of stel (section 4.3). ## style representation analysis we want to further understand what the style representations learned to be similar styles. we take the best-performing style representation (roberta trained on the cav task with the conversation cc and seed 106) and perform agglomerative clustering on a sample of 5.000 cav tasks of the conversation test set resulting in 14,756 unique utterances. we use 7 clusters based on an analysis of silhouette scores (appendix c). out of all utterance pairs that have the same author, 46.2% appear in the same cluster. this is different from random assignments among 7 clusters 10 which corresponds to 20.1% ± .00. as authors will have a certain variability to their style, a perfect clustering according to general linguistic style would not assign all same author pairs to the same cluster. in ## limitations and future work we propose several directions for future research: first, conversation labels are already inherently available in conversation corpora like reddit. however, it remains a difficulty to transfer the conversation cc to other than conversation datasets. moreover, even when using the conversation cc, content information might still be useful for av: if one person writes "my husband" and another writes "my wife" within the same conversation, it is highly unlikely that those utterances have been generated by the same person. with the recent advances in semantic sentence embeddings, it might be interesting to train style representations on cav tasks with a new content control level: two utterances could be labelled as having the same content if their semantic embeddings are close to each other (e.g., when cosine similarity is above a certain threshold). second, for the stel-or-content task, the socalled "triplet problem" @xcite remains a potential problem. consider the example in figure 2 . here, the stel framework only guarantees that a1 is more informal than a2 and s2 is more informal than s1. thus, in some cases a2 can be stylistically closer to a1 than s2. however, we expect this case to be less prevalent: a2 would need to be already pretty close in style to a1, or both s2 and s1 would need to be substantially more informal or formal than a1. in the future, removing problematic instances could alleviate a possible maximum performance cap. third, the representation models may learn to represent individual stylistic variation as we use utterances from the same individual author as positive signals (c.f. zhu and jurgens (2021)). however, because the representation models learn with same author pairs that are generated from thousands of authors, it is likely that they also learn consistencies along groups of authors that use similar style features (e.g., demographic groups based on age or education level, or subreddit communities). future work could explore how different cc levels and training tasks influence the type of styles that are learned. ## conclusion recent advances in the development of style representations have increasingly used training objectives from authorship verification @xcite . however, representations that perform well on the authorship verification (av) task might do so not because they represent style well but because they latch on to spurious content correlations. we train different style representations by controlling for content (cc) using conversation or domain membership as a proxy for topic. we also introduce the new contrastive authorship verification setup (cav) and compare it to the usual av setup. we propose an original adaptation of the recent stel framework @xcite to test whether learned representations favor style over content information. we find that representations that were trained on the cav setup with conversation cc represent style in a way that is more independent from content than models using other cc levels or the av setup. we demonstrate some of the learned stylistic differences via agglomerative clustering -e.g., the use of a right single quotation mark vs. an apostrophe in contractions. we hope to contribute to increased efforts towards learning general-purpose content-controlled style representations.