Rethinking the Authorship Verification Experimental Setups (2022)

Abstract: One of the main drivers of the recent advances in authorship verification is the PAN large-scale authorship dataset. Despite generating significant progress in the field, inconsistent performance differences between the closed and open test sets have been reported. To this end, we improve the experimental setup by proposing five new public splits over the PAN dataset, specifically designed to isolate and identify biases related to the text topic and to the author’s writing style. We evaluate several BERT-like baselines on these splits, showing that such models are competitive with authorship verification state-of-the-art methods. Furthermore, using explainable AI, we find that these baselines are biased towards named entities. We show that models trained without the named entities obtain better results and generalize better when tested on DarkReddit, our new dataset for authorship verification.

## introduction identifying the author of a text is one of the most versatile nlp tasks, with applications ranging from plagiarism detection to forensics and monitoring the activity of cyber-criminals. the task spans several decades and was tackled using statistical linguistics @xcite @xcite , and, more recently, machine learning (de @xcite @xcite . due to the typically small data setup of authorship analysis tasks, deep learning methods had a slow start in this domain. nevertheless, inspired by the impressive performance of pre-trained language models, such as bert @xcite , these methods gained traction in authorship analysis as well. @xcite showed that convolutional siamese networks are more robust than a bertbased method over large-scale authorship attribution tasks. @xcite investigated pretrained language models for cross-topic and crossdomain authorship attribution and showed that bert and elmo @xcite achieve the best results while being the most stable approaches. @xcite introduced bert for authorship attribution (bertaa) in which they combine bert with stylometric features for authorship attribution. the authors remarked that their model is unable to perform text similarity evaluation in the context of the more difficult authorship verification problem, which we tackle. one of the main contributors to the active developments in authorship analysis is the pan organizing team, who proposed annual shared tasks since 2009. while the recent pan 2020 and 2021 contests increased the difficulty of the authorship verification task and enabled large-scale model training @xcite , there are still possible generalization issues due to the dataset splits. for instance, models from 2020 trained on the closed-set data surprisingly performed better on the open-set test data (which is arguably more difficult) than on the closed-set test data @xcite . we therefore argue that in order to better assess the generalization capabilities of authorship verification systems, a more fine-grained approach to dataset splitting may be needed. to address these issues, we introduce a set of five carefully designed splits of the publicly available pan dataset, ranging from the easiest setup (closedset) to the most difficult (open-set). our splits progressively alleviate information leaks in the test data, enabling a more confident evaluation. furthermore, we release our splits publicly foot_0 to allow other members of the community to evaluate models on any computing infrastructure, enabling the evaluation of large-scale models. along with the new splits, we introduce a set of bert-based models @xcite to serve as baselines test split o2d2 * o2d2 bert naive † comp. † closed 93.5 96.4 95.6 75.6 72.2 clopen 94.0 96.0 97.4 74.1 71.1 open ua 92.6 92.6 90.2 78.6 68.5 open uf 91.4 95.1 91.6 79.9 79.0 open all 80.6 67.5 88.7 75.6 76.9 pan closed 93.3 93.5 -74.7 74.2 pan open 93.3 94.4 -75.3 74.5 for future research. we show that these language models are competitive with the top scoring o2d2 (out-of-distribution detector) system at pan 2021 @xcite . we also qualitatively inspect the models' predictions and find that they often rely on named entities to verify authorship. we show that by replacing the named entities in the dataset with placeholders, we are able to obtain significant performance gains and better generalization capabilities. in summary, our contributions are threefold: ## datasets we use the pan 2020 authorship verification dataset foot_1 . a document d i belongs to a fandom (topic) f i and is written by an author a i . author verification is a classification task which asks whether documents d i and d j are written by the same author (sa) or by different authors (da). the dataset comes in two sizes: small (52k examples) and large (275k examples). the latter one is better suited for deep learning models. ## experiments training. we fine-tune bert (b) @xcite and character bert (cb, char-bert) @xcite as binary classifiers for authorship verification. given two documents d i and d j , we concatenate and feed them to the transformer encoder. when a document is longer than 256 tokens, we sample a random chunk of length 256. the chunks are resampled at every epoch, hence increasing the variety of the training set. to make predictions, we add a linear layer on top of the h @xcite vector and optimize the entire model via the binary cross entropy loss. we use the same set of hyperparameters across all of the experiments. for the other models (o2d2, naive and compression) we used the provided code and default hyperparameters. ## conclusions we introduced and published five splits of the pan dataset ranging from the easiest closed setup to increasingly more challenging settings. this enables a fine-grained evaluation and model selection. we showed that bert-based baselines are competitive with top-scoring authorship verification methods and significantly outperform non-neural baselines. using integrated gradients, we showed that, distinctly from the closed split, the open splits help generalization at the model selection step by preventing the model from overfitting on named entities of specific train authors or fandoms. we further improved generalization by replacing the named entities, making the models more robust to spurious features. this claim also holds under a strong distribution shift, when cross-evaluating the models on the significantly different darkreddit dataset.