The Topic Confusion Task: A Novel Evaluation Scenario for Authorship Attribution (2021)

Abstract: Authorship attribution is the problem of identifying the most plausible author of an anonymous text from a set of candidate authors. Researchers have investigated same-topic and cross-topic scenarios of authorship attribution, which differ according to whether new, unseen topics are used in the testing phase. However, neither scenario allows us to explain whether errors are caused by failure to capture authorship writing style or by the topic shift. Motivated by this, we propose the topic confusion task where we switch the author-topic configuration between the training and testing sets. This setup allows us to investigate two types of errors: one caused by the topic shift and one caused by the features’ inability to capture the writing styles. We show that stylometric features with part-of-speech tags are the least susceptible to topic variations. We further show that combining them with other features leads to significantly lower topic confusion and higher attribution accuracy. Finally, we show that pretrained language models such as BERT and RoBERTa perform poorly on this task and are surpassed by simple features such as word-level n -gram.

## introduction authorship attribution is the problem of identifying the most plausible author of an anonymous text from a closed set of candidate authors. the importance of this problem is that it can reveal characteristics of an author given a relatively small number of their writing samples. early approaches to authorship attribution depended on manual inspection of the textual documents to identify the authors' writing patterns, and mendenhall (1887) showed that word lengths and frequencies are distinct among authors. since the first computational approach to authorship attribution @xcite , researchers have aimed at finding new sets of fea-tures for current domains/languages, adapting existing features to new languages or communication domains, or using new classification techniques, e.g. @xcite @xcite @xcite @xcite . alternatively, motivated by the real-life applications of authorship attribution different elements of and constraints on the attribution process have been investigated @xcite @xcite @xcite . currently, authorship attribution is being used in criminal investigations where a domain expert would use authorship techniques to help law enforcement identify the most plausible author of an anonymous, threatening text @xcite . explaining both authorship attribution techniques and their results is crucial because the outcome of the attribution process could be used as evidence in the courts of law and has to be explained to the jury members. researchers have investigated same-topic (fig. 1a ) and cross-topic (fig. 1b ) scenarios of authorship attribution, which differ according to whether unseen topics are used in the testing phase. the cross-topic setting is considered more realistic than the same-topic setting, but it causes the performance of well-known authorship attribution techniques to drop drastically. this drop is attributed to the topic-writing style entanglement problem where existing writing style features are capturing the topic variations in the collected documents rather than the authors' writing styles. traditionally, the evaluation of new authorship methods or writing style features for authorship attribution has been based on the difference in the accuracy either on the attribution task or in ablation studies. while this methodology enhanced the performance on the downstream task and helped answer which features perform well, there is a need for methods that can help us understand why certain features are performing better than others. specifically, do these newly proposed features/techniques actually capture the stylistic variations of an author, or are they simply better at picking out sub-topic cues that correlate with each author? in this work 1 , we propose a new evaluation setting, the topic confusion task. we propose to control the topic distribution by making it dependant on the author, switching the topic-author pairs between training and testing. this setup allows us to measure the degree to which certain features are influenced by the topic, as opposed to the author's identity. the intuition is as follows: the more a feature is influenced by the topic of a document to identify its author, the more confusing it will be to the classifier when the topic-author combination is switched, which will lead to worse authorship attribution performance. to better understand the writing style and the capacity of the used features, we use the accuracy and split the error on this task to one portion that is caused by the models' confusion about the topics, and another portion that is caused by the features' inability to capture the authors' writing styles. the primary contributions of this work are the following: ## related work the first work that used a computational approach is @xcite , which used the naïve bayes algorithm with the frequency of function words to identify the authors of the federalist papers @xcite . research efforts have aimed at finding new sets of features for current domains/languages, adapting existing features to new languages or media, or using new classification techniques @xcite @xcite @xcite @xcite . recent attempts have been made to investigate authorship attribution in realistic scenarios, and many studies have emerged where the constraints differ from the training to the testing samples such as @xcite on crosslanguage, @xcite on cross-domain/genre, and finally, @xcite @xcite @xcite on cross-topic. @xcite ; @xcite achieved state-of-the-art results on cross-topic authorship attribution. @xcite proposed a character-and word-level n-grams approach motivated by text distortion @xcite for topic classification. in contrast to @xcite , stamatatos kept the most frequent words and masked the rest of the text. @xcite explored the widely used and massively pretrained transformer-based @xcite language models for authorship attribution. specifically, they trained a separate language model for each candidate author with a pretrained embeddings layer from elmo @xcite , bert @xcite , gpt-2 @xcite and ulmfit @xcite . each model was presented with words from the investigated document, and the most plausible author for that document is the one whose model has the lowest average perplexity. ## dataset we present an extended, curated, and relatively balanced version of the guardian dataset. 3 one motivation is that the articles in the commonly used version of the dataset contained some html artifacts and meta-data from the guardian's website, and had a number of its articles either on the wrong topic, or written by authors that are not in the dataset. because of that, we retrieved the original articles, and added more articles to balance the number of writing samples per author on each topic. we maintained the same upper limit on the number of documents per author as the original dataset. another reason is that as we try to understand the effect of the topic on the attribution process, we need to isolate any external factors that may affect the performance and make the results noisy. for example, in the topic confusion task, we have to use topics with writing samples from all the authors. otherwise, the model could learn to favour one topic versus the other during training, while on test time, it will have author samples that it did not see during training. based on that, it will be hard to tell whether these samples will be misclassified due to lack of training samples or due to a strong topic effect on the attribution process. although datasets in real life can be imbalanced, this issue can be addressed by randomly excluding some writing samples to make the dataset imbalanced or using proper performance metrics for imbalanced datasets such 3 appendix a describes the data collection procedure. as weighted accuracy, precision, recall and f-score. the number of collected articles and additional descriptive statistics are provided in ## authorship attribution models in this section, we discuss two groups of authorship attribution models. the first group contains a set of classical models that use hand-engineered features and a classification algorithm. the second group comprises a set of neurally-inspired models motivated by recent advancements in many natural language processing tasks. such models are considered end-to-end system where the feature representation is learned by the model as opposed to being hand-crafted and provided to the model. ## evaluation procedure for each set of features, we used the setup explained in section 3 to create a 100 different configurations. for each configuration, we randomly ordered the topics, selected 12 out of the 13 available we decided to omit one author and use the remaining twelve out of the available 13 authors to balance the groups. with this split, the probability of picking the correct author is 1 12 , the likelihood of choosing a wrong author in the same group is 5 12 , and the probability of picking a wrong author in the other group is 6 12 . this case applies if the true author was in either group 1 or group 2. however, suppose we were to use all the 13 authors and divide them into two groups of six and seven authors, respectively. in that case, the probabilities will differ depending on whether the actual author is in the group with six authors or seven authors. in that case, we will need to re-weight the errors based on their probability, and that will complicate 7 we trained fs bert and fs roberta only once. after creating the training, validation, and testing sets we train models for authorship attribution. first, the features are extracted from the writing samples. second, a classification model is trained on the training samples, tuned on the validation set to pick the best hyperparameters, and tested on the testing set. note that the classifier does not have access to any information about the setup, such as the groups configuration or the topic labels. ## conclusion in this work, we proposed the topic confusion task, which helps us characterize the errors made by the authorship attribution models with respect to the topic. additionally, it could help in understanding the cause of the errors in authorship attribution. we verified the outcomes of this task on the cross-topic authorship attribution scenario. we showed that a simple linear classifier with stylometric features and pos tags could improve the authorship attribution performance compared to the commonly used n-grams. we achieved a new state-of-the-art of 83.3% on the cross-topic scenario by resurrecting stylometric features and combining them with pos tags and word-level n-grams, 3% over the previous state-of-the-art, masking-based, character-level approach. surprisingly, neurally-inspired techniques did not perform well on the authorship attribution task. instead, they were outperformed by a simple, hand-crafted set of stylometric features and poslevel n-grams and an svm classifier with a linear kernel.