B ert AA : BERT fine-tuning for Authorship Attribution (2020)

Abstract: Identifying the author of a given text can be useful in historical literature, plagiarism detection, or police investigations. Authorship Attribution (AA) has been well studied and mostly relies on a large feature engineering work. More recently, deep learning-based approaches have been explored for Authorship Attribution (AA). In this paper, we introduce BertAA, a fine-tuning of a pre-trained BERT language model with an additional dense layer and a softmax activation to perform authorship classification. This approach reaches competitive performances on Enron Email, Blog Authorship, and IMDb (and IMDb62) datasets, up to 5.3% (relative) above current state-of-the-art approaches. We performed an exhaustive analysis allowing to identify the strengths and weaknesses of the proposed method. In addition, we evaluate the impact of including additional features (e.g. stylometric and hybrid features) in an ensemble approach, improving the macro-averaged F1-Score by 2.7% (relative) on average.

## introduction authorship analysis is the field of natural language processing that studies the characteristics of a text and extracts information on its author. it is made of 3 sub-tasks, which include author profiling, i.e. detecting sociolinguistic attributes such as gender or age, authorship verification which identifies the degree of similarity of texts, and authorship attribution @xcite . authorship attribution (aa) is the process of attributing a text to the correct author among of closed set of potential writers. aa is widely used in plagiarism detection or attribution of historical literature @xcite . this classification task is also well known in forensic investigations @xcite . aa has been studied on short texts @xcite , such as tweets as well as longer texts, such as judgments of a few thousand words on average @xcite . the main challenge in aa is the extraction of relevant features characterising the author's identity. majority of approaches proposed in the past relied on a large amount of feature engineering, in order to reflect both the content and the style of the author @xcite @xcite . in this paper, we propose a method, bertaa, that relies on the fine-tuning of a pre-trained bert language model, to which we add a dense layer and a softmax activation for authorship classification, trained for a few epochs. this is one of the very first attempts to analyze the performances of pretrained language model fine-tuning for in-domain aa, especially for a large number of authors (up to 100). as most deep-learning methods for aa, bertaa does not require text preprocessing nor feature engineering. our method offers state-of-the-art (sota) performances on well-known corpora, with a relative accuracy improvement of up to 5.3%. we also illustrate the strengths and weaknesses of such a system. we also show that building an ensemble architecture, which also incorporates stylometric and hybrid features tends to improve the macroaveraged f1-score. finally, we set a benchmark for the full imdb corpus @xcite for 5, 10, 25, 50, 75, and 100 authors, which, to the best of our knowledge, has never been studied in its full format for aa. the next section discusses the relevant approaches developed in the literature. section 3 presents the corpora used as well as a brief exploration of each of the sources. section 4 details the architectures of bertaa, while section 5 describes the results we obtained, and section 6 discusses the strengths and weaknesses of our method, as well as future work directions. finally, section 7 depicts our conclusions. ## related work traditionally, aa largely relies on the process of extracting features related to content or style of an author @xcite . more recently, some approaches propose to use deep learning methods for aa tasks, whether relying on a previous feature extraction step or not. the following sections briefly describe these various methods. ## authorship attribution corpora several corpora have been studied for the task of aa. in this section, we briefly describe each corpus we used as well as its key features. the aa task we performed focuses on identifying the author of a text among a list of the top n authors for whom we collected the largest number of texts. ## bertaa : bert-based authorship attribution content-related features in aa take into account the topics and the semantics of the text. recent works on language representation models have however shown that transformers such as bert @xcite reach sota performances for various tasks, hence improving glue score as well as several other metrics. it has been extensively used for text classification tasks @xcite , and bert is known to be well-performing at extracting semantic and syntactic information. to the best of our knowledge, no systematic review of the performance of fine-tuned pre-trained language models for aa has been reported yet, and such classifier has never been combined with a stylometric and hybrid features in an ensemble model. hereby, we introduce bertaa, a fine-tuning of bert with a dense layer and a softmax activation, trained for a few epochs for aa. the output dimension of the dense layer corresponds to the number of authors in the corpus. bert is made of 12 transformer blocks and 12 self-attention heads. the input size, i.e. the maximum length of tokens is 512, and the hidden layer representation dimension is 768 @xcite . as described by @xcite , to use bert as a classifier, a simple dense layer with softmax activation is added on top of the final hidden state h of the first token [cls], through a weight matrix w , and we predict the probability of label c the following way: then, all weights, including bert's ones and w , are adapted, in order to maximize the logprobability of the correct label. the training is done using a cross-entropy loss function. we used a pre-trained bert available from the transformer library @xcite , trained on large corpora. the fine-tuning of bert for the aa task was done on a tesla p100-pcie-16gb. additionally, we incorporate stylometric and hybrid features to bertaa, in 2 models called bertaa + style and bertaa + style + hybrid through a lr. thus, our system is able to account for content, stylometric, and hybrid features. the architecture of bertaa + style + hybrid is presented in figure 1 . the stylometric classifier first extracts the lexical stylometric features as proposed by @xcite . the features extracted are the length of text, the number of words, the average length of words, the number of short words, the proportion of digits and capital letters, individual letters and digits frequencies, hapax-legomena, a measure of text richness, and the frequency of 12 punctuation marks. a lr is trained on these features. the hybrid features we extract are the frequencies of the 100 most frequent character-level bi-grams and tri-grams. classification is then done using a lr. finally, the output probabilities of bert classifier, the stylometric, and the hybrid ones are concatenated and classified using an additional lr. ## results parameters we chose for our architectures are presented in bertaa outperforms the tf-idf and lr benchmark on all experiments, with an average relative accuracy gain of 14.3%. it reaches a competitive performance on 5 authors on the enron dataset, since only 2 samples were not classified correctly out of 4104, hence leading to an accuracy of 99.95%. comparing results on enron to other approaches in literature is not trivial since it largely depends on the data preparation that was done. we decided to remove short emails, and remove utterances containing the name of the sender (as a signature for example), but not all papers involving enron data for aa precisely describe their data preparation. furthermore, we found no results in the literature on imdb full-corpus for the top n authors. hence, our results set a benchmark on the full imdb, on average 8.2% above a word-level tf-idf. next, we compare our results with current sota on the imdb62 and the blog authorship datasets. ## discussion our approach can be summarized as an extrapolation of bert's general outstanding scores, for aa. we show that reaching sota results can be achieved using only a single dense layer and a softmax activation on top of a pre-trained bert with a few training epochs. we highlighted that bertaa performs well on rather short texts, few imbalances in the number of texts per author, and a large number of texts per author. previous works @xcite have shown that using stylometric and hybrid features improves the accuracy of aa tasks. we also show that adding such features when leveraging pre-trained language models can improve the macro-averaged f1-score by 2.7% (relative) on average, although impacting the accuracy. the use of bertaa should be limited to cases where bert is itself a good candidate, i.e. when there are sufficient training data per author. this condition might be hard to reach in real applications for police investigations. short texts and few imbalances have also been identified as requirements for better model performances. our model is also currently unable to perform text similarity evaluation in the context of authorship verifica-tion. there are many possible extensions to this work. according to our experiments on enron, blog authorship, and imdb corpora, aa can successfully leverage transformers-based language representation models. so far, we have not performed further pre-training of bert on the target domain, which could also help bertaa. we have not tried yet to use another pre-trained language model. future works should also explore other model architectures like roberta @xcite , or try to extract additional stylometric, hybrid, profiling, or content-related features. including the computation of similarity metrics on embeddings learned through the bert fine-tuning would also be a way to compare the similarity between texts for authorship verification tasks. we will also explore bertaa in aa tasks on asr transcripts, where punctuation and capital letters are not present for example. ## conclusion with the rise of deep learning and transformers in natural language processing, feature engineering and text preprocessing are less needed. in this work, we presented an approach based on fine-tuning of a pre-trained bert for author classification. this is one of the very first attempts to analyze the performances of pre-trained language model fine-tuning for in-domain aa. we showed that our approach, which leverages bert, reaches competitive performances on three wellknown benchmark datasets, even on a large number of authors. the model best performs when sufficient training data per author are available, there is no large class imbalance, and texts remain rather short. we also show that in a large scale aa task, adding stylometric and hybrid features to bertaa in an ensemble model can improve the macroaveraged f1-score by 2.7% (relative) on average. finally, we set a new benchmark on the full imdb authorship attribution corpus for 5, 10, 25, 50, 75, and 100 authors. future works will explore adding features to bertaa, further pre-training bert on target-domain, exploring other pre-trained language models, and extending our approach to authorship verification. 1