Can Authorship Representation Learning Capture Stylistic Features? (2023)

Abstract: Automatically disentangling an author’s style from the content of their writing is a longstanding and possibly insurmountable problem in computational linguistics. At the same time, the availability of large text corpora furnished with author labels has recently enabled learning authorship representations in a purely data-driven manner for authorship attribution, a task that ostensibly depends to a greater extent on encoding writing style than encoding content. However, success on this surrogate task does not ensure that such representations capture writing style since authorship could also be correlated with other latent variables, such as topic. In an effort to better understand the nature of the information these representations convey, and specifically to validate the hypothesis that they chiefly encode writing style, we systematically probe these representations through a series of targeted experiments. The results of these experiments suggest that representations learned for the surrogate authorship prediction task are indeed sensitive to writing style. As a consequence, authorship representations may be expected to be robust to certain kinds of data shift, such as topic drift over time. Additionally, our findings may open the door to downstream applications that require stylistic representations, such as style transfer.

## introduction knowing something about an author's writing style is helpful in many applications, such as predicting who the author is, determining which passages of a document the author composed, rephrasing text in the style of another author, and generating new text in the style of a particular author. the trouble is that fully characterizing something as complex as writing style has proven too unwieldy to admit fine-grained hu-man annotations, which leaves the possibility of directly learning explicit and interpretable representations of writing style practically beyond reach. instead, research in this area has largely focused on specific stylistic attributes, such as formality, toxicity, politeness, gender, simplicity, and humor, which are more straightforward to annotate @xcite @xcite @xcite . unfortunately, the reliance on human labels and the narrow focus of such stylistic distinctions severely limit the utility of such representations in tasks related to authorship, such as those listed above. in this paper, we focus instead on the authorship prediction task, which enjoys the benefit of not requiring manually elicited labels, since metadata in many corpora include either explicit author labels or usernames that may serve as proxies for latent authorship. as a result, the vast scale of data available for training authorship prediction models opens the door to learning generalizable authorship representations using deep learning. we specifically consider similarity learning approaches that aim to produce vector representations of documents, where the distance between two vectors is inversely related to the likelihood that the corresponding documents were composed by the same author @xcite . however, achieving high accuracy in the authorship prediction task does not necessarily imply that stylistic features have been successfully learned. for example, in a given corpus, correctly predicting that two writing samples were composed by the same author may be possible on the basis of non-stylistic signal, such as the topic of conversation. therefore, this work is concerned with obtaining a better understanding of the nature of representations learned for the authorship prediction task. unfortunately, because deep learning models behave like black boxes, we cannot directly interrogate a model's parameters to determine what information such representations contain. for example, one might hope to employ attentionbased approaches that provide post hoc explanations through token saliency maps @xcite . however, such methods provide no guarantee of the fidelity of their explanations to the underlying model. furthermore, the subjective interpretation required to deduce the reasons that such methods highlight certain spans of text makes it nearly impossible to systematically draw conclusions about the model. instead, we propose targeted interventions to probe representations learned for the surrogate authorship prediction task. first, we explore masking content words at training time in §5, an operation intended to gauge the degree to which a representation relies on content. then we explore automatic paraphrasing in §6, an operation intended to preserve meaning while modifying how statements are expressed. finally, in §7 we explore the capacity of these representations to generalize to unseen tasks, specifically topic classification and coarse style prediction. taken together, and despite approaching the research question from various points of view, our experiments suggest that representations derived from the authorship prediction task are indeed substantially stylistic in nature. in other words, success at authorship prediction may in large part be explained by having successfully learned discriminative features of writing style. the broader implications of our findings are discussed in §8. ## related work perhaps the work most closely related to our study is that of @xcite and @xcite , who propose measuring the stylistic content of authorship representations through four specific assessments, namely, formality, simplicity, contraction usage, and number substitution preference. our work differs in two main respects. first, we regard style as an abstract constituent of black-box authorship representations rather than the aggregate of a number of specific stylistic assessments. second, the works above deal with stylistic properties of individual sentences, whereas we use representations that encode longer spans of text. indeed, we maintain that the writing style of an author manifests itself only after observing a sufficient amount of text composed by that author. for example, it would be difficult to infer an author's number substitution preferences after observing a single sentence, which is unlikely to contain multiple numbers. the same is true of other stylometric features, such as capitalization and punctuation choices, abbreviation usage, and characteristic misspellings. in another related work, @xcite find that although content-based features may be suitable for datasets with high topical variance, datasets with lower topical variance benefit most from style-based features. like the works mentioned above, sari et al. explicitly identify a number of style-based features, so writing style is more of a premise than the object of study. in addition, experiments in this previous work are limited to datasets featuring a small number of authors, with the largest dataset considered containing contributions of only 62 authors. a number of end-to-end methods have been proposed to learn representations of authorship @xcite @xcite @xcite . a common thread among these approaches is their use of contrastive learning, although they vary in the particular objectives used. they also differ in the the domains used in their experiments, the numbers of authors considered for training and evaluation, and their open-or closed-world perspectives. as discussed in §3, we use the construction introduced in @xcite as a representative neural method because it has shown evidence of capturing stylistic features through both its success in the challenging open-world setting in multiple domains and its performance in zero-shot domain transfer. ## authorship representations in this paper, an authorship representation is a function mapping documents to a fixed euclidean space. the fact that such representations are useful for a number of authorship-related tasks is generally attributed to their supposed ability to encode author-specific style, an assertion we aim to validate in this paper. in this section, we describe how these representations arise and how they are intended to be used. our analysis centers around representations f implemented as deep neural networks and trained using a supervised contrastive objective @xcite . at training time this entails sampling pairs of documents x, x composed by the same author (resp., by different authors) and minimizing (resp., maximizing) the distance between f (x) and f (x ). therefore, we may assume at inference time that f (x), f (x ) are closer together if x, x were composed by the same author than they would be if x, x were composed by different authors. no meaning is ascribed to any attribute of f (x), such as its coordinates, its length, or its direction. rather, f (x) is meaningful only in relation to other vectors. in all the experiments of this paper, f is an instance of the universal authorship representation (uar) introduced in @xcite . 1 notwithstanding the merits of a number of other recent approaches discussed in §2, we argue that because of its typical neural structure and typical contrastive training objective, uar serves as a representative model. the same paper also illustrates that uar may be used for zero-shot transfer between disparate domains, suggesting a capacity to learn generalizable features, perhaps of a stylistic nature, thereby making it a good candidate for our experiments. note that uar defines a recipe consisting of an architecture and a training process that must be carried out in order to arrive at a representation, with the understanding that care must be taken in assembling appropriate training datasets. specifically, we consider a diverse set of authors at training time in an effort to promote representations that capture invariant authorship features, chiefly writing style, rather than time-varying features, such as topic. invariance is a desirable feature of authorship representations because it improves the likelihood of generalization to novel authors or even to novel domains. however, there is no guarantee that invariant features are exclusively stylistic in any given corpus, or that any training process we might propose will result in representations capturing exclusively invariant features. therefore, this work is concerned with estimating the degree to which authorship representations are capable of capturing stylistic features, with the understanding that completely disentangling style from topic may be beyond reach. ## experimental setup mirroring rivera-soto et al. (2021), we conduct experiments involving three datasets, each consisting of documents drawn from a different domain. for the reader's convenience, we present further details and some summary statistics of these datasets in §a.1. to evaluate an authorship representation, we use the common experimental protocol described below. the objective is to use the representation to retrieve documents by a given author from among a set of candidate documents, which are known as the targets, on the basis of the distances between their representations and the representation of a document by the desired author, which is known as the query. to this end, each evaluation corpus has been organized into queries and targets, which are used to calculate the mean reciprocal rank (mrr). following is a friendly description of this metric, with a more elaborate formulation presented in §a.2. an authorship representation may be used to sort the targets according to the distances between their representations and that of any fixed query. in fact, this ranking is often seen as the primary outcome of an authorship representation. because one would need to manually inspect the targets in the order specified by the ranking, it would be desirable for any target composed by the same author as the query to appear towards the beginning of this list. the mrr is the expectation of the reciprocal of the position in the ranked list of the first target composed by the same author as a randomly chosen query. this metric ranges from 0 to 1, with higher values indicating a greater likelihood of finding documents composed by an author of interest within a large collection in the first few search results. following rivera-soto et al., the queries and targets in all our experiments are episodes, each consisting of 16 comments or product reviews contiguously published by the same author in the reddit or amazon domains, respectively, or 16 contiguous paragraphs of the same story in the fanfic domain. in order to conduct a wide variety of experiments in a time-efficient manner, we train all representations on one gpu for 20 epochs, although we acknowledge that better results may be obtained by training with more data, on multiple gpus, or for longer than 20 epochs. ## masking content words our first series of experiments aims to illustrate through a simple training modification that authorship representations are capable of capturing style. specifically, the strategy of masking training data in a way that preserves syntactic structure, something which is known to relate to style, while removing thematic or topical information, has been effective, particularly in cross-domain authorship experiments @xcite . to this end, we propose training authorship representations with restricted access to topic signal by masking varying proportions of content-related words in the training data. evaluating each of these representations and comparing its ranking performance with that of a representation trained on the same unmasked data reveals the capacity of the representation to capture style. words may be roughly divided into two categories: content words and function words. content words primarily carry topic signal. they tend to include nouns, main verbs, adjectives, and adverbs. function words serve syntactic roles and convey style through their patterns of usage. they tend to include auxiliary verbs, prepositions, determiners, pronouns, and conjunctions @xcite . these observations suggest masking words according to their parts of speech (pos), a process we call perturbing linguistic expressions or pertle. ## removing style by paraphrasing in contrast with the experiments in §5 that aim to assess the ability of authorship representations to capture style by removing content, our next group of experiments aims to make the same assessment by instead removing style. for this purpose we consider automatic paraphrasing, which ideally introduces stylistic changes to a document while largely retaining its original content. if an authorship representation avails of stylistic features, then we expect paraphrasing a document to impair its ability to match the document with other documents by the same author. ## generalization to novel tasks our experiments have thus far focused on authorship prediction, a task which is presumably best addressed with a model capable of distinguishing writing styles. we now use authorship representations to directly distinguish writing styles using a corpus of documents furnished with style annotations, namely, the cds dataset @xcite . this consists of writings in disparate 3 we trained the smaller model with the dataset released by @xcite . for the larger model we queried @xcite for comments published between january 2015 and october 2019 by authors publishing at least 100 comments during that period. all three models were trained using the default hyperparameter settings of https://github.com/llnl/luar . the mrrs of uar19, uar, and uar23 evaluated on a test set composed of comments published future to those constituting the three training corpora are 0.482, 0.592, and 0.682, respectively. styles, including writings by two classical authors (shakespeare and joyce), historical american english from different eras, social media messages, lyrics, poetry, transcribed speech, and the bible. with the notable exception of the two classical authors, most styles in cds are not author-specific, but rather represent broad stylistic categories. this means that identifying cds styles is not the same problem as authorship prediction, an important observation we revisit below. in addition, we repeat the experiment with a corpus furnished instead with topic annotations, namely, the reuters21578 dataset @xcite . this is a popular benchmark in text classification consisting of financial news articles, each annotated by one or more topics. we note that certain topics may be associated with particular authors and editors, and therefore style could be a spurious correlate, although we nevertheless expect the authorship representation to perform worse relative to the semantic baseline described below. for each corpus, the experiment consists of simply applying an authorship representation trained on the reddit dataset to two randomly chosen documents from the corpus. we used reddit because it has been shown to yield representations that generalize well to other domains @xcite . we record the dot product of the resulting vectors and pair this score with a binary indicator specifying whether the two documents carry the same labels. noting that predicting the binary indicator from the dot product is a highly imbalanced problem, with most document pairs bearing non-matching rather than matching labels, we simply construct the corresponding receiver operating characteristic (roc) curve, an illustrative device intended to explore the tradeoffs in making that prediction by thesholding. we report the equal error rate (eer), a simple summary statistic of the roc curve. smaller values of this metric are preferable. for good measure, we also report the area under that curve (auc) in §a.4, another summary statistic of the roc curve. finally, because writing style may be difficult to assess without sufficient text content, we also vary the amount of text contributing to the dot products mentioned above. specifically, rather than predicting whether the label of a single document matches that of another on the basis of the dot product of their representations, we more generally predict whether a group of randomly chosen documents of the same label shares that label with another group of randomly chosen documents sharing another label on the basis of the dot product of the means over each group of the representations of their constituents. as a baseline we repeat both experiments using the general purpose document embedding sbert in place of the authorship representation. sbert is commonly regarded as a semantic embedding, but is not typically used to discriminate writing styles without further training. the rationale for the experiment is the following. if the authorship representation primarily encodes stylistic features, then we would expect poor performance relative to sbert on the topic discrimination task since the task presumably does not involve stylistic distinction. however, we would expect better performance from the authorship representation than sbert on the style discrimination task. these expectations are borne out in the results reported in figures 7 and 8 , which show eer against the number of input documents for each task and each model. the generalization performance of uar on these novel tasks relative to sbert is consistent with a representation that is sensitive to stylistic information. namely, sbert consistently outperforms uar on topic classification, while uar consistently outperforms sbert on style classification. we present 95% confidence intervals for each curve as lighter regions of the same color surrounding the curve. although these were calculated using a bootstrap approach, the confidence intervals of the corresponding auc results shown in figures 10 and 11 of §a.4 were calculated using a bootstrap-free calculation. also shown in figures 7 to 11 are the results of the same experiments using the two variations of uar introduced in §6.5. these additional models were included to support an auxiliary argument raised in §8, but also afford an interesting but subtle insight about the current task. namely, although uar19 performs strictly worse and uar23 strictly better than uar on the authorship attribution task, classifying style into broad categories is a different problem than authorship attribution, the latter dealing with fine-grained stylometric features. this accounts for the seemingly contradictory results in figure 8 , in which uar19 performs better than uar, which in turn performs better than uar23. indeed, training uar on more authors produces representations that are more discriminative of individual authors, something which is at odds with identifying broad stylistic categories for the simple reason that being exposed to more authors affords more opportunities for uar to discover stylistic features that distinguish authors. notwithstanding these observations, we remark that within the cds dataset, certain styles are likely to be correlated with particular topics, while in the reuters dataset, certain authors are likely to often write about particular topics. this would suggest that sbert might perform better on cds and uar better on reuters than one might expect, so the absolute performance on both tasks is not particularly informative. ## discussion findings we have examined properties of an exemplary authorship representation construction, finding consistent evidence that the success of the representations it engenders at distinguishing authors may be attributed in large part to their sensitivity to style. first, the masking experiments of §5 show that for sufficiently large training corpora, masking a large fraction of content words at training time does not significantly affect ranking performance on held-out data, suggesting that these representations are largely invariant to the presence of content words. on the other hand, the paraphrasing experiments of §6, which seek to alter writing style while preserving content, confirm that paraphrasing drastically impairs ranking performance. taken together, these experiments suggest that the authorship representations considered are indeed sensitive to stylistic features. this conclusion is corroborated in §7 where we see poor generalization of one of these representations to a topic discrimination task, but better generalization to a style discrimination task, both assessments relative to a semantic baseline. limitations all of the experiments in this paper involve instances of the uar construction. since our primary research question involves testing the capacity of representations trained for the authorship prediction task to capture stylistic features, we select this construction because there is prior evidence that the representations it engenders perform well at zero-shot cross-domain transfer, for certain combinations of source and target domains, which likely requires some degree of stylistic sensitivity @xcite . by design, our analysis is focused on aggregate model behavior. while this addresses the high-level research questions we pose in the introduction, such global analysis does not enable predictions about which specific local features are involved in model predictions. as such, an important avenue for future work lies in developing methods that can faithfully identify local authorship features. to this end, frameworks for evaluating the quality of explanations, such as @xcite , are essential. beyond the usual challenges of explaining the decisions of deep neural networks, explaining author style may pose further challenges, such as the need to identify groups of features that in combination predict authorship. we emphasize that completely disentangling style from content may not be attainable, since certain aspects of writing likely blur the line between style and content @xcite . for example, we notice degradation in ranking performance of the sbert model in broader impact this work contributes to the broader goal of formulating notions of content and style that constitute mutually exclusive and collectively exhaustive aspects of writing that may be embedded in orthogonal subspaces of a euclidean space. not knowing whether this ambition is fully realizable, but hopeful others will explore the question in future research, we resign ourselves in this paper to exhibiting two embeddings that accomplish the objective to a limited extent. specifically, we focus on uar, which we show to mostly encode style, and sbert, which is widely assumed to encode content. this being an imperfect decomposition, the primary goal of this paper is to qualitatively assess the degree to which uar encodes style rather than content. authorship attribution is likely to be a task that benefits from a representation that is relatively stable over time, specifically an encoding capturing primarily writing style. to this end, another open question is whether a representation may be constructed that encodes style to a greater degree than uar, and if so, whether the representation improves performance on the attribution task. if such a representation were proposed in the future, the experiments we propose in this paper could be used to validate the assertion that it encodes style to a greater degree than uar. on the other hand, content features constitute perfectly legitimate discriminators of authorship in some cases. for example, an author who discusses only a narrow range of topics on a particular forum may easily be distinguished from other authors on the basis of topic features. not knowing under which circumstances and to what degree content plays a role in authorship attribution, we maintain that the relationship between the performance of a representation on the attribution task and the degree to which the representation encodes content should be explored fully, something that will again require an estimate of the degree to which a representation encodes style. another promising application of authorship representations is style transfer, where one hopes to rephrase a given statement in the style of a given author. this has been analogously accomplished in the domain of speech, resulting in the ability to have a given statement recited in the voice of a given speaker (see, e.g., @xcite . the primary ingredient in this task is a speaker embedding, which is analogous to an authorship representation. however, by construction, a speaker embedding encodes almost exclusively acoustic features, but encodes content features, namely the specific words spoken, to a very limited degree. the fact that this observation might be the primary reason for the success of speech transfer portends possible difficulties for the style transfer task. however, as with authorship attribution, the relationship between the success of using a representation for style transfer and the degree to which the representation encodes style should also be fully explored, and again, the experiments proposed in this paper would constitute a natural assessment of that degree. the mean mrr computed by three independently trained models, where 0.011 is the maximum sample standard deviation over all experiments reported in the table. we observe that the lite model outperforms the unmasked model in most cases and is also generally better than the grande model, especially in the fanfic domain. once again, the grande model is generally worse than the unmasked model, but only slightly, and even improves on the unmasked model in some settings. if words with high tf-idf scores are indeed primarily content words, then this experiment again suggests that authorship representations rely little on content and more heavily on writing style. as mentioned above, one advantage of the tertle schema is that it obviates pos tagging. to illustrate this potential, and to determine whether similar patterns hold in another language, we repeat our tertle experiment with a chinese dataset scraped from the weibo social network. see §a.1 for more details on this dataset. this requires replacing the sbert component of the uar architecture with a chinese bert pre-trained using whole word masking @xcite . the results of the experiment with the weibo dataset, displayed in figure 9 , show an overall pattern similar to that of the english-focused experiments.