Representation learning of writing style (2020)

Abstract: In this paper, we introduce a new method of representation learning that aims to embed documents in a stylometric space. Previous studies in the field of authorship analysis focused on feature engineering techniques in order to represent document styles and to enhance model performance in specific tasks. Instead, we directly embed documents in a stylometric space by relying on a reference set of authors and the intra-author consistency property which is one of two components in our definition of writing style. The main intuition of this paper is that we can define a general stylometric space from a set of reference authors such that, in this space, the coordinates of different documents will be close when the documents are by the same author, and spread away when they are by different authors, even for documents by authors who are not in the set of reference authors. The method we propose allows for the clustering of documents based on stylistic clues reflecting the authorship of documents. For the empirical validation of the method, we train a deep neural network model to predict authors of a large reference dataset consisting of news and blog articles. Albeit the learning process is supervised, it does not require a dedicated labeling of the data but it relies only on the metadata of the articles which are available in huge amounts. We evaluate the model on multiple datasets, on both the authorship clustering and the authorship attribution tasks.

## introduction authorship analysis is an ensemble of methods that aims to extract useful authorship information of a text by analyzing writing style. the most commonly addressed tasks in this field are classification tasks such as authorship attribution @xcite , authorship verification @xcite and authorship characterization. the authorship attribution is the process of inferring the author of documents among known authors while the authorship verification is the process of deciding whether or not a given document was written by a given author. to this end, most studies rely on feature engineering to represent the input documents in order to improve the performance of machine learning algorithms. feature engineering consists in transforming raw input data into new input data by using domain knowledge with the expectation that the new features will be more suitable for learning an efficient model. for the two aforementioned tasks, the process consists in selecting textual characteristics of documents, then, either use a classifier to predict the author of a document based on these characteristics, or calculate similarities between document representations. one common way to choose these document representation features is by assessing whether or not they can enhance the prediction accuracy. sometimes these features intuitively belong to style such as function words @xcite , sometimes they just correspond to common nlp features such as distributional representations of documents @xcite @xcite . few studies attempt to directly produce unsupervised representations of style in order to project unseen documents in a low dimensional stylometric space @xcite @xcite . feature engineering is designed by humans based on heuristics, a labor-intensive process. however, recent studies show that automatically learning a representation of raw data is useful for many reasons described in @xcite . learned representations are suitable in classification and clustering tasks since these representations manage to select discriminating information from raw data and represent them in a low dimensional vector space @xcite . moreover, this discriminating information is not necessarily captured by humans through heuristics. documents belonging to same authors are generally consistent in their writing style @xcite even for authors covering a large range of topics @xcite . the method we propose relies on this observation. in this paper, we validate the style-generalization assumption which is based on two propositions. first, it states that we can represent unseen documents of unseen authors by generalizing stylometric features from a set of known authors and known documents. second, documents that belong to the same author tend to have similar representations in the stylometric space (given a standard similarity function). in order to validate this assumption, we exploit recent advances in text and sentence representation with deep neural network (dnn) architectures, especially transformers @xcite . we propose a transformer-based dnn model fine-tuned on the authorship attribution task using a large dataset of documents whose authors are known (the reference set). then we assess the model in its ability to capture the similarity between documents of the same authors in a clustering experiment. in this paper, we first give an overview of related work on authorship analysis in section 2. in section 3, we propose a definition of the style and the motivations behind our representation learning method. in section 4, we explain our method and introduce the style-generalization assumption. section 5 presents the experimentation on the main task which is the authorship clustering and on a second, the authorship attribution task. finally, in section 6, we deepen our analysis by studying the second property of our definition of style. recent studies tackle issues of the feature engineering process for authorship analysis by exploiting raw text samples using deep neural networks. @xcite proposed a gated recurrent unit (gru) dnn trained on article and sentences for the authorship verification task. other studies demonstrated the relevance of recurrent neural networks such as gru and long short-term memory (lstm) dnns on the authorship attribution task @xcite . more recently, @xcite proposed a convolutional neural network-based model for the multi-label authorship attribution of scientific publications. besides the use of dnn for classification, very few studies attempt to automatically embed stylometric features from a corpus of documents. @xcite are the first to rely on an external dataset of known authors to pretrain a general model which can compute stylometric similaries between documents. they proposed training a support vector machine-based (svm) model that was re-used for a test dataset containing unseen authors. as far as we know, this study is the first attempt at generalizing a stylometric similarity space while still relying on handcrafted stylometric features. @xcite proposed a model that jointly learns topical and lexical distributional representation of documents in an unsupervised manner to help authorship analysis. @xcite proposed a model that embeds the writing style of english novels. the model is composed of fast-text word embeddings and a stacked lstm. the authors demonstrated that their model performed well on a subset of the pan14 dataset for the authorship verification task. in order to verify authorship of social networks posts, they designed their experiments so that model inputs were short text samples. @xcite recently used a recurrent neural network-based architecture and adversarial learning to tackle the authorship verification task. the model was trained to embed pairs of documents and was assessed on authorship verification task in transfer learning settings, when authors of the test set do not exist in the train set. instead, we propose a representation learning method that aims to embed documents in a stylometric space relative to a large dataset of wellknown authors. we use a modification of a pretrained bert model @xcite in a classification task as proposed in several recent works @xcite . to the best of our knowledge, this paper is the first to propose an authorship attribution-training driven model generalizing stylometric features so that documents of unseen authors can be clustered without fine-tuning. additionally, our method relies on a large dedicated dataset that can be extended. we trained our models on a large amount of data by using news and blog articles benefiting from the wide availability of such data on the web. ## motivation an author can adopt several styles, and one of them can be similar to the writing style of another author. @xcite defined the style as "a consistent and distinguishable tendency to make [some of these] linguistic choices". moreover, @xcite explained that "texts are much more than what they are about". any textual characteristic that is not semantic or topical belongs to stylistic choices of the author. different expressions can have a common meaning, and can refer to the same objects and the same events, but still be made up of different words and different syntax, corresponding to the author's willingness to let a context, an orientation, sometimes an emotion be shown through @xcite . news articles showed to have specific writing style by using, for instance, date as adverb in "the governor thursday announced..." or anthropomorphization in "the 1990s saw an increase in crime...". it is called journalese @xcite . headlines of newspapers also have their own style, called headlinese, such as articles drop @xcite . today's trend towards clickbait also has an influence on headlines @xcite . finally, let's mention style guides of newspapers that not only influence typography (e.g. paragraph structure, quotation, italic) but also the usage of the language @xcite . for instance, a style guide can more or less encourage the use of the active voice against the passive voice. some of the newspapers can have their own style guide that their writers follow, allowing the text to be consistent for the reader while any variation having no purpose could be distracting @xcite . style appears more or less pronounced depending on the text passages, it is difficult to define it precisely and, given a document, to find a set of words (or sequence of words) that will strictly define the style of its author. the text is the combination of a shape -its style -and a content which are intertwinned thanks to the choice of specific words. words or sequence of words in the text can rarely be denoted as belonging specifically to the style or to the content. this is why extracting style features is hard. from documents of a reference corpus, we aim to extract latent structures falling within the scope of writing style. we argue that these latent structures can be identified by dnns, typically rnn models with attention layers which will focus on style-related terms. from a linguistic point of view these latent structures map to lexical, syntactic or structural fragment of sentences or paragraphs. intuitively, when extracting a style representation of a document, we seek to focus on latent structures that will satisfy these two properties : intra-author consistency the property of being consistent in documents belonging to the same author. semantic undistinguishness the property of carrying very little information on what makes the document semantically (e.g. topics, named entities) distinguishable in the corpus. thus, this definition, inspired by @xcite ; holmes (1998), means that the style of a document is represented by linguistic structures which are consistent for individual authors (allowing their identification) but more likely semantically poor regarding the content of the document (e.g. topic, named entities). indeed, what the document is about is a constraint that imposes on the author to use a specific vocabulary. the terms that belong to this specific vocabulary have a strong semantic value with respect to the theme of the document, and on the contrary, are less likely to convey the author's style. the representation learning method is based on identifying consistent latent structures following the intra-author consistency property. next to that, the semantic undistinguishness is a property which can be verified by studying attention weights of a trained dnn models. 4 method let's denote @xmath0, ..., d n } a set of documents and @xmath1, ..., a m } a set of authors so that each document belongs to one and only one author and each author wrote at least one document. and a r is the set of all the authors of the documents in d r . sizes n r and m r are typically large and n @xmath2. let's denote v @xmath3, 1] m r a vector space so that ∀@xmath4, m r @xmath5. thus, softmax vectors for a r belong to v r . we denote g @xmath6, ..., g r m r } the set of one-hot vectors that correspond to the ground-truth vectors of each author in a r . thus, vectors from g r also belong to v r . each document in d r is associated with one and only one vector from g r . let f r : r l × r @xmath7, represented by word vectors, from d r into v r with l the size of the documents (truncated or padded) and w the dimension of the word vectors. the purpose of f r is to project each document d r i , @xmath8, ..., n r } closer to its corresponding vector in g r than to any other groundtruth vector. the main assumption of this paper, the stylegeneralization assumption, is that two representations defined by f r of two unseen documents are more likely to be similar if they belong to the same author than if they do not. intuitively, we assume that any unseen document belonging to an unknown author is similar, in terms of style, to documents belonging to a subset of known authors and that another document of the same unknown author is likely to be similar, in terms of style, to documents belonging to this same subset of known authors. figure 1 illustrates the style-generalization assumption. blue sets in v r correspond to representations of documents from d r . f r allows to project documents from d r to v r such that the representation of a document is close to its corresponding ground truth vector (in red color). green sets correspond to unseen document representations in v r . the assumption states that close representations (green sets) likely belong to the same author. more formally, let's denote u -@xmath9, a u ) a set of unseen documents and authors with the stylegeneralization assumption states that the projection of documents from d u (the u -set) into v r using f r allows to compute similarities such that similar documents from d u are likely written by the same author. thus, learning f r on a reference set allows authorship clustering in the general stylometric space that it defines. this method, which can be called the stylegeneralization learning method, aims to calculate stylometric similarities of documents without considering the author of the document, i.e. without fine-tuning. in section 5, we propose to use a dnn model as the function f r . we train the model on the reference set and generate vector representations of unseen documents by using intermediate weights representation in the dnn. ## in-depth analysis in this section, we intend to assess how well the trained models can, by using the stylegeneralization learning method, focus on terms exposing the second property of our definition of writing style: the semantic undistinguishness. for this purpose, we propose to analyze attention weights of the sna model. in this experiment, our main concern was not the performance in the authorship attribution task but on the use of the attention layer trained with the method described in section 4. thus, we used sna since its training phase is less time-consuming. the semantic undistinguishness suggests that style-related linguistic structures tend to carry little information on content, topics, entities, etc. these style-related structures are often referred to as function words which are frequent in a corpus @xcite . on the other hand, terms with a high semantic value that will identify, for instance, a topic, are those allowing the document to be distinguishable in a corpus. the tfidf weighting is a well established method to estimate how important a word is to a document in a corpus. thus, in order to quantitatively assess the undistinguishness of the sna model, we propose a measure based on the tfidf weighting. the tfidf focus measure allows to compute how well attentions of the model focus on words having lower tfidf weights: a is the attention matrix of size w × d. w is the number of words in a document that we set to 1200 and d is the number of documents. each line of the matrix corresponds to the attention weights in the sna model for a document in a given u-set. an attention vector of a single document is normalized so that the weights sum to 1. the same goes with the normalized tfidf matrix t of size w × d. thus, we defined the tfidf focus as the mean of the attention weight times the tfidf weight of each word. this measure is high when high values of tfidf are in line with high values of attention and low when these high values of tfidf are in line with low values of attention. given a pretrained sna model and a u-set, we generate the matrix a using the model, the matrix t using the tfidf weighting on the target u-set and, finally, the tfidf focus score. the second line shows tfidf focus scores computed by a sna model trained on an external u-set that we randomly chose. the external u-set acts as a short r-set with fewer documents and authors. as we can see, the use of a short r-set is not sufficient for the model to focus on words with lower tfidf weights. the same goes for the three last lines but for blog articles. in this experiment, we quantitatively showed that the original sna model focuses its attention on function words having lower tfidf weights (10% less on average), thus it is more able to capture stylometric features related to specific words exposing the semantic undistinguishness property than other models trained on smaller u-sets. the method proposed in section 4 as well as the use of a large reference corpus have an impact on the results. note that, in the clustering experiment in section 5, good performances of standard baselines such as tfidf can be explained by the fact that documents of same authors have a topic bias (they share same semantic/topic words) because an author generally write on a few topics. this bias helps representations of vocabulary-based models to be close. however, such features fail to identify authors in cross-domain scenarios @xcite , while our model focuses less on topic-and semantic-related words but achieves comparable performance (even better for simrank and the authorship attribution task). ## conclusion in this paper, we proposed a new method for the representation learning of writing style. we have shown that it is possible to generalize the writing style on the basis of a set of reference authors. the method follows a property of the style that we call intra-author consistency. we sought to validate two underlying propositions of the stylegeneralization assumption. first, we can represent unseen documents of unseen authors by using a model generalizing stylometric features from a set of known authors and known documents. second, if two unseen documents have close representations using this model, they are likely to belong to the same author. results show that the dnn model that was trained following our method succeeded in the authorship clustering of unseen documents belonging to unseen authors. it also performs well on the authorship attribution task. moreover, we showed that a model trained with the style-generalization learning method is more able to capture stylometric structures exposing the semantic undistinguishness property. from a practical point of view, our method does not require a tedious labeling effort but relies only on the metadata of a large dataset of articles. we believe that this work provides new perspectives in the field of authorship analysis by proposing a definition of writing style based on distributional properties, as well as a new method aiming to learn stylometric representations. in further studies, we intend to exploit style features in external tasks such as news recommendation. we already have promising results showing that style representations of news articles allow to diversify recommendation lists and to recommend "novel" news articles without loosing prediction accuracy.