Writing Style Author Embedding Evaluation (2021)

Abstract: Learning authors representations from their textual productions is now widely used to solve multiple downstream tasks, such as classification, link prediction or user recommendation. Author embedding methods are often built on top of either Doc2Vec (Mikolov et al. 2014) or the Transformer architecture (Devlin et al. 2019). Evaluating the quality of these embeddings and what they capture is a difficult task. Most articles use either classification accuracy or authorship attribution, which does not clearly measure the quality of the representation space, if it really captures what it has been built for. In this paper, we propose a novel evaluation framework of author embedding methods based on the writing style. It allows to quantify if the embedding space effectively captures a set of stylistic features, chosen to be the best proxy of an author writing style. This approach gives less importance to the topics conveyed by the documents. It turns out that recent models are mostly driven by the inner semantic of authors’ production. They are outperformed by simple baselines, based on state-of-the-art pretrained sentence embedding models, on several linguistic axes. These baselines can grasp complex linguistic phenomena and writing style more efficiently, paving the way for designing new style-driven author embedding models.

## introduction since the early work of @xcite , and more recent models based on the transformer architecture @xcite , continuous word representations have been key in processing and analyzing textual data. it led to a prolific research on learning meaningful representations of largerscale textual entities, such as paragraph, document or even authors. learning author embeddings can be used to solve several downstream tasks, such as user recommendation @xcite and classification @xcite . addi-tionally, metadata (e.g., graph structure @xcite , timestamp @xcite ) are often used to guide the representation learning process, in particular with application to social media. this is referred as user embedding, while author embedding only focuses on the textual production of users. despite a growing literature in both author and user embedding @xcite , it is usually difficult to tell what these representations really capture from its textual production. although most recent models reach more than 90% accuracy in authorship attribution on several datasets @xcite , none of the existing works tried to determine if the embedding space captures topic preferences, topological information, sentiment or stylistic features. getting a better understanding of what these spaces really capture can be a real asset to design new machine learning models. for instance, one could be interested in linking several authors with similar writing style even though they deal with different topics. it is even more important with literacy data or for specific tasks such as authorship attribution for forensic investigation @xcite @xcite . being able to compare and determine which embedding methods are the more relevant in such a context is essential. unfortunately, in most previous works, the evaluation of author embedding relies solely on classification accuracy, which demonstrates the need for a more robust and richer evaluation framework. as stated in @xcite , most evaluation methods for embedding are based on downstream tasks, which does not fully assess the quality of the embedding space, if it really captures what is has been built for. to tackle this issue, we propose a novel experimental scheme to evaluate author representations based on her writing style. even though there is no consensual definition of what the writing style is, many works have tried to identify the most relevant features to characterize it. in this article, we consider low-level structural and syntactical features uncorrelated with topics, and we show that most author embedding methods are in fact essentially driven by semantic. the code is made available at the following address: style embedding evaluation 2 related works in this section, we present an overview of existing author embedding models and evaluation frameworks. we also briefly review prior works on stylistic features selection through authorship attribution. author embedding consists in learning, for each author, a vector representation in a low dimensional space. in this space, the proximity between two vectors should relate to the similarity in authors' textual production. part of the literature focuses on user embedding. while works on author and user embedding may look similar, they should not be mistaken. user embedding usually refers to the context of social media where, in addition to the user textual production, several metadata (e.g., retweet, likes, links) are used to guide the learning process. here, we mostly focus on methods that leverage the textual content only. nevertheless, our experimental protocol method can be used to evaluate user embedding models in a stylistic way, in combination with other metrics. we leave this perspective to future work. ## new proposed framework for author embedding evaluation a classification model usually evaluates whether an embedding method successfully captures an author's topic preferences. here, we want to evaluate how well the embedding captures its way of writing. as stated earlier, simple stylistic features are a good proxy of the authors' writing style. these features can easily be extracted from a corpus and aggregated by author. training a simple regression model (typically linear regression or more complex methods, such as support vector regression) using author embedding to predict these features would allow to compare these representations in their abil- ## conclusion we presented a new evaluation framework for author embedding focusing on the writing style. this evaluation scheme is based on the extraction of stylistic features which represent a good proxy of an author way of writing. we show that simple baselines outperform recent author embedding models in predicting most of those stylistic features. these baselines rely on state-of-the-art sentence embedding models which capture complex linguistic notion thanks to multitask training on several big corpora. this demonstrates the need to develop new author embedding models that can grasp the author writing style. if models relying on doc2vec show clear limit in this task, use with a dan architecture seems to be a way to go.