Learning Universal Authorship Representations (2021)

Abstract: Determining whether two documents were composed by the same author, also known as authorship verification, has traditionally been tackled using statistical methods. Recently, authorship representations learned using neural networks have been found to outperform alternatives, particularly in large-scale settings involving hundreds of thousands of authors. But do such representations learned in a particular domain transfer to other domains? Or are these representations inherently entangled with domain-specific features? To study these questions, we conduct the first large-scale study of cross-domain transfer for authorship verification considering zero-shot transfers involving three disparate domains: Amazon reviews, fanfiction short stories, and Reddit comments. We find that although a surprising degree of transfer is possible between certain domains, it is not so successful between others. We examine properties of these domains that influence generalization and propose simple but effective methods to improve transfer.

## introduction authorship verification offers a useful capability for a number of problems, such as plagiarism detection, moderation of user-generated content, historical authorship attribution, and forensic analysis. authorship verification techniques have traditionally relied on modeling stylometric linguistic properties, such as punctuation and whitespace usage and the frequencies of function words, parts-ofspeech, and character n-grams @xcite . more recently, end-toend models built with convolutional neural networks @xcite and recurrent neural networks @xcite have been proposed. however, neural methods introduce a tradeoff: although they obviate the need of manual feature design by learning relevant features, these features are not explicitly identified. not knowing how a system arrived at its authorship conclusions makes it difficult to assess how closely tied these features are to the domain from which the training data was drawn. in addition, neural methods are less applicable in low-resource domains due to their requirement of large training datasets. on the other hand, if authorship features could be learned in a domainindependent fashion, it would reduce the need for in-domain training sets by exploiting transfer between domains: authorship representations could be learned from a large but out-of-domain corpus and subsequently deployed in a target domain. in prior work, @xcite perform a study on cross-domain author verification in a closed world of 21 authors. in contrast, we consider an open-world setting with several orders of magnitude more authors. with a view towards addressing the deeper question of whether neural approaches are capable of learning universal authorship representations that are effective in unseen domains, we conduct a systematic study involving three disparate domains: amazon reviews, fanfiction, and reddit comments. we propose the combination of an attentionbased architecture and a contrastive training objective in §2 that achieves a new state-of-the-art in ranking performance. in §4.1 we conduct zeroshot transfer experiments among various domains, finding large discrepancies in transfer performance. in §4.2 we evaluate how properties of the training data affect transfer, including the number of training authors and the topic distribution of the data. in §4.3 we train models on the unions of multiple domains, in some cases resulting in improved generalization. finally, in §4.4 we propose a simple masking technique that we find improves transfer. overall, we find that neural models do not in general capture universal authorship features, and that the training data must be carefully controlled in order to learn the desired invariances. ## model objective we learn a function f mapping a collection of documents to r 512 under which any two collections composed by the same author have higher cosine similarity than two collections composed by different authors. we fit f using contrastive learning @xcite , which proceeds as follows. given a mini-batch foot_0 of collections x i indexed by a set i, we denote the l 2 normalization of f (x i ) by z i and maximize architecture we propose the attention-based authorship verification model illustrated in figure 1 , an elaboration of the sbert model developed for learning semantic sentence embeddings @xcite . the following description assumes that the input is a collection of short documents, as in @xcite . if instead the input is a single long document, we regard it as a collection by subdividing into paragraphs. at training time our model randomly samples excerpts consisting of @xmath0, where w varies from 1 to 16 by taking @xmath1, where x ∼ beta (3, 1). previous studies have shown that randomly sampling windows improves model generalizability and provides a simple way to handle very long documents @xcite , while @xcite show that sampling the number w of documents improves a model's ability to handle collections of documents of arbitrary size at inference time. we pass the w excerpts to sbert, which consists of a bert model followed by an attentionweighted mean pooling of the features of each excerpt. this results in w feature vectors, each of dimension 768, the dimension of bert's hidden feature. we apply self-attention to the w vectors, apply maxpooling, and finally project the result to r 512 through a linear layer. we use a pretrained sbert model @xcite ) but update all model parameters during training. prior work has explored self-attention models for authorship attribution @xcite @xcite with mixed success compared to simpler convolutional models. these systems have utilized either the output or the classification token of bert as the basis for learning authorship embeddings. however, we find that sbert's attention weighted averaging of the hidden activations leads to better performing authorship embeddings. at test time we use more of the available text to evaluate the model. namely, we set @xmath2.1, §3.3). for the fanfiction dataset (see §3.2) each short story is regarded as a collection of documents by subdividing into paragraphs; at test time we take w to be the number of paragraphs, which essentially amounts to encoding the entire work. ## data our experiments use the three anonymous domains described below. for each domain we hold out a portion of the domain's data to be used to evaluate the performance of models trained on various combinations of the three domains. the remainder of the data is used for training. for domains that supply timestamps, we stipulate that the held-out evaluation data must be future to the training data in order to assess robustness to ephemeral aspects of writing, such as topic @xcite . the evaluation metrics described in §4 require that the evaluation corpora each be further divided into two disjoint sets, the queries and the targets. for domains that include timestamps, we also stipulate that all the targets be future to all the queries. ## conclusion our main finding is that in general, neural authorship models are not universal, with models trained on some domains exhibiting significantly better transfer than those trained on others. specifically, we find that reddit exhibits consistently good transfer performance and improves performance when incorporated in multi-domain training, which we attribute to both topic diversity and the size of the dataset. in addition, the proposed attention-based model establishes a new state-ofthe-art for the large scale, open-world setting @xcite which to our knowledge, is the first instance where a selfattention model has been shown to consistently outperform simpler, yet highly effective convolutionalbased architectures for authorship verification. in future work, it would be interesting to attempt to explicitly disentangle the learned representations into content and style, for example, by using domain adversarial training objectives @xcite . the data augmentation proposed in §4.4 represents a promising first step towards more sophisticated data augmentation methods. additional data augmentation methods, particularly if applied together during training and incorporated into the loss calculation via multiple independent views of the data @xcite , have the potential to further improve verification performance.