Developing a Benchmark for Reducing Data Bias in Authorship Attribution (2021)

Abstract: Authorship attribution is the task of assigning an unknown document to an author from a set of candidates. In the past, studies in this field use various evaluation datasets to demonstrate the effectiveness of preprocessing steps, features, and models. However, only a small fraction of works use more than one dataset to prove claims. In this paper, we present a collection of highly diverse authorship attribution datasets, which better generalizes evaluation results from authorship attribution research. Furthermore, we implement a wide variety of previously used machine learning models and show that many approaches show vastly different performances when applied to different datasets. We include pre-trained language models, for the first time testing them in this field in a systematic way. Finally, we propose a set of aggregated scores to evaluate different aspects of the dataset collection.

## introduction in authorship attribution, various machine learning techniques are used to predict who has written a specific document, given a set of candidate authors. this means that a dataset used for such experiments must be well-controlled for many aspects like topic, length, etc. to ensure that the model detects the writing style of an author rather than something else like the topic of the content @xcite . for example, a model may find it easy to detect the author if each author writes about a single specific topic, and therefore the model would detect topic rather than style. therefore, a well-controlled dataset should cover only one topic and one genre so that the only difference between the authors can be attributed to their writing style. consequently, this makes the results of experiments using these well-controlled datasets prone to data bias, and they become difficult to generalize. one approach to mitigate that bias is to distinguish the style of authors from the content, which allows to use cross-topic datasets. in this subset of tasks, the documents used for training the models are deliberately different from the texts used for validation thereafter. for example, given a set of journalists that write articles in multiple sections, when news articles about politics are used for training and articles about sports written by the same authors are used for testing, the overlap of topical content can be reduced and the model can only detect stylistic features. similarly, cross-genre datasets take this one step further and require different genres of documents for training and testing (e.g., text messages and scientific essays). these datasets reduce the amount of stylistic information that can be used for each author to a subset that is expressive in both genres. in cross-language datasets, the training and testing data are written in different languages, further reducing this overlap. even when using cross-domain (topic, genre, etc.) datasets, the difficulty of generalizing assumptions regarding the authors' writing style remains, as any conclusions can only be stated for the concrete authors in that dataset. this may be sufficient for some applications, in which the style of specific authors or well-controlled groups of authors is analyzed. however, statements that claim to hold up more generally require evaluation with multiple and diverse datasets. in this paper, we present a collection of datasets for authorship attribution which cover a wide variety of aspects, fulfilling these needs. we include datasets with few and many candidate authors, with different numbers of documents per author, with differently sized documents, and cross-topic, crossgenre and cross-language datasets. we provide detailed suggestions on train/test splits and perform evaluation experiments with a wide variety of models to demonstrate how much the choice of the dataset can impact classification results. we provide exemplary attribution results for all datasets for a wide variety of machine learning models. we specifically include several pre-trained language models, as they have shown great success in different nlp fields over the last years, but a systematic analysis of their performance in the authorship attribution field has not yet been performed to the best of our knowledge. lastly, to evaluate how well a model performs for each aspect of the collection, we provide a set of aggregated scores that combines results from different datasets. our contributions in this paper are therefore threefold: (1) we present a collection of selected, highly diverse datasets for authorship attribution that are able to better generalize evaluation results, (2) we benchmark several pre-trained language models on these datasets, providing a previously unavailable baseline in the field of authorship attribution, and (3) we provide a set of scores that evaluate a model based on the different aspects of the datasets. to ensure reproducibility and foster future research, we publish all code online 1 . thereby, we focus on providing tooling that minimizes the efforts required to expand both the dataset collection as well as the evaluation scores. ## related work while many previous studies use either only one dataset or don't specifically increase the diversity of the datasets used, they often fail to address this implicit data bias. even foundational work in this field trying to categorize features in this field in a fundamental way can be prone to this issue. for example, @xcite measure the effectiveness of 39 different feature types for attribution. they address the importance of the dataset being representative for a language and explicitly explain the characteristics of the texts and authors in great detail, but consequently, by using a single dataset, their findings of feature performances are restricted to those very characteristics. nevertheless, findings of such fundamental work are often referenced for research that uses completely different datasets. one idea to mitigate any bias on the content of a dataset is to focus on the separation between style and content. this can be achieved by explicitly modelling the topic @xcite or by using cross-topic or cross-domain datasets, where the training data and the test data have a different genre 1 https://git.uibk.ac.at/csak8736/ authbench or contain texts about different topics @xcite @xcite . for the latter, the key idea is that by minimizing the topic or genre-specific content contained in the overlap of training and testing data, any performances measured must conclude from the stylistic information from the authors. nevertheless, for both approaches, the bias towards those authors remains in the evaluation. even from within a dataset, the choice of training and testing data can have a large impact on the outcome and additionally varies across languages @xcite . additionally, @xcite demonstrated that the amount of text required to reliably attribute an author also depends on the language, and suspects that this result may be depending on the genre of text as well. similarly, @xcite show that while some feature types are more robust to the size of the dataset, the performance of others varies greatly depending on the number of documents per author and the number of authors. in this paper, we want to showcase a collection of diverse authorship attribution datasets and perform attribution experiments with several widely used exemplary machine learning models. the higher goal of our work is that it should be easy to make evaluation results of authorship attribution research easily comparable and also generalizable. using pre-trained language models for authorship attribution has not been researched in great detail. some approaches use them as part of a larger ensemble model @xcite or as a feature extraction step in front of the actual classifier @xcite . however, when it comes to the performance of the unaltered models that are readily available, no overview for comparisons using widely used authorship attribution datasets are available to the best of our knowledge. ## datasets for this benchmark, we have selected a wide variety of authorship attribution corpora. thereby, we focussed on multiple aspects of datasets that may influence the classification process and try to provide a diverse but controlled set of these aspects: ## experiment setup and models the purpose of the evaluation experiments in this paper is to show that the performance of different models varies greatly across different datasets. we therefore perform classification experiments with several classification models to provide an impression of how the choice of a dataset influences the evaluation, but don't claim to provide the best possible configurations of those models for the analyzed datasets. we select several features in combination with a linear support vector machine, as well as several solutions based on pre-trained language models. important parameters for the models are listed in parameter value tf/idf normalization l2 number of features all doc2vec learn rate 0.02 epochs 20 vector size 100 svm c 1.0 penalty l2 loss squared hinge multiclass strategy one-versus-rest max. iterations 1,000 tolerance 1e -4 language model * bert bert-base-cased distilbert distilbert-base-cased roberta roberta-base ## evaluation results for all models presented in section 4, we perform authorship attribution experiments for all datasets covered in section 3 (using the train/test splits as discussed in section 3.2). in ## scores the various datasets allow aggregation of the results according to the different aspects described in the previous sections, for which we formulate scores that are listed in we want to emphasize once more that the aim of this paper is not to provide the best possible results for the tested models, but show how a more expressive evaluation result can be achieved by incorporating multiple datasets into the evaluation process. ## limitations and future work the collection of datasets presented in this paper is by no means exhaustive in terms of covered dataset aspects, but it should provide a solid foundation for this purpose. for example, authorship attribution on a larger scale @xcite ; tschug-scores presented in the previous section. thereby, contributions can be made to existing scores by providing more datasets to reassure them, or add additional scores to the collection. we hope to timely contribute more multilingual datasets to the ml score and expand on different dataset sizes beyond the few thresholds presented by the sm score. ## conclusion in this paper, we present a collection of datasets aimed to increase the expressiveness and generalizability of authorship attribution experiments. the datasets are carefully chosen to include many different aspects of the text, such as document size, number of documents per author, language, genre, or topic. we choose several well-established text classification models and perform attribution experiments on all datasets, for the first time showing results systematically for pre-trained language models in this field. thereby, we demonstrate the importance of including multiple datasets in any evaluation by showing differences in the classification score for similar datasets and train/test splits. we conclude the paper by suggesting an aggregated score for each of the presented aspects to easily distinguish the strengths and weaknesses of different models.