A Call for Clarity in Contemporary Authorship Attribution Evaluation (2021)

Abstract: Recent research has documented that results reported in frequently-cited authorship attribution papers are difficult to reproduce. Inaccessible code and data are often proposed as factors which block successful reproductions. Even when original materials are available, problems remain which prevent researchers from comparing the effectiveness of different methods. To solve the remaining problems—the lack of fixed test sets and the use of inappropriately homogeneous corpora—our paper contributes materials for five closed-set authorship identification experiments. The five experiments feature texts from 106 distinct authors. Experiments involve a range of contemporary non-fiction American English prose. These experiments provide the foundation for comparable and reproducible authorship attribution research involving contemporary writing.

## introduction closed-set authorship attribution picks out the likely author of an unsigned document from a pool of candidate authors. decades of research show that authors leave conspicuous "fingerprints" in their writing @xcite . a small amount of preexisting prose (ca. 2,500 words) is often enough to learn enough about the writing "styles" of a set of candidate authors to correctly identify the author of an unsigned document. authorship attribution techniques have found application in numerous domains. they have been used to resolve uncertainty about authorship in historical research @xcite . for writers living today, widespread use of authorship attribution techniques-and related author profiling techniques @xcite -poses a privacy risk @xcite . a better understanding of how authorship attribution techniques work can inform efforts to improve privacy-enhancing language technologies. while there is no doubt that authorship attribution methods have improved over the past century, recent progress is harder to measure. some of this difficulty is due to the field's success. in recent decades, new methods improve on old ones by small amounts. given small improvements, assessing whether or not the advance may be due to aleatory factors such as preprocessing or a particular dataset becomes difficult. another reason recent progress is difficult to measure is the lack of standard benchmark tasks. of 15 frequently-cited authorship attribution studies examined by @xcite , original corpora could be found for only 4 (27%) and code could be located for 0 (0%). while other fields, notably machine translation and language modeling, excel at organizing research activity around publicly-accessible benchmark tasks, contemporary authorship attribution research has no such tasks. recent experience suggests that without standard benchmarks-and evidence that researchers can consistently reproduce results using them-a field's ability to self-assess progress on well-defined tasks can go astray. the field of recommender systems offers a cautionary tale. @xcite documents a series of papers being published in prestigious journals over a five year period which do not, in fact, improve on earlier results. notably, these papers used a standard dataset for their evaluations (movielens 10m). where these papers fell short was in their reproduction of previous resultsto which their new methods were compared. the papers reported improvements on earlier results which were illusory; models used in earlier research, upon closer examination, outperformed the new methods. analogous cases exist in other fields. in machine translation, although standard datasets were used, inconsistency in applying a key metric (bleu) prevented researchers from easily reproducing or comparing results @xcite . our paper supports reproducible research in authorship identification by introducing five standard benchmark tasks. each task features fixed train and test sets. four of the five tasks have a test set consisting of writing samples on fixed topics, guaranteeing that test set examples do not overlap with training set examples in terms of subject matter. data for all tasks is available for download without any restrictions. having the original data is not enough. the test set, the set of documents whose authorship a model must predict, must also be specified @xcite . if cross-validation is used, the train/test splits must be known. authorship attribution datasets typically feature a small number of authors (8-100) and much of the variability in model performance can be due to the idiosyncratic composition of cross-validation "folds." for an example, consider the task of reproducing the work of @xcite with the enron email corpus. @xcite evaluate different techniques using ten-fold cross-validation with varying number of candidate authors. comparing the performance of a new model with their result requires knowledge of the composition of the folds they used. small improvements in classifica-tion accuracy could be due to different partitions of the set of authors into cross-validation folds. a different partition could, by chance, end up with folds featuring writers who have distinct writing styles, making achieving higher accuracy easier. ## improving authorship attribution evaluation the problems described in the previous section complicate a range of authorship attribution research (e.g., identification, verification, profiling). we propose a suite of five tasks which address the problems for one area of authorship attribution research: closed-set author identification involving contemporary english-language non-fiction prose. lessons learned developing standard benchmark tasks in this area will, we hope, inform the development of analogous tasks in other areas. two arguments support our focus on contemporary non-fiction texts. first, collecting redistributable non-fiction prose from a diverse set of writers is relatively easy. a considerable share of the english-using population writes non-fiction prose. demonstrating (some) competency in english composition is a requirement in secondary education across the english-speaking world. second, many researchers are interested in the efficacy of authorship attribution methods applied to contemporary non-fiction english prose. english is, for the moment, the lingua franca of diplomacy, science, and international business. authorship attribution methods which work on english therefore enjoy broad applicability. the stakes of author profiling research-research informed by authorship attribution research-are also significantly higher for research involving living writers than for writers active in previous centuries. only in the former case is, say, an individual's privacy at risk. ## accuracy of received methods ## discussion perceptions of the importance of having reproducible measures of model performance on wellunderstood tasks have changed over the last decade. previously regarded as something desirable but by no means essential, reproducible benchmarks are increasingly seen as indispensable. experience has shown that without such benchmarks, researchers risk overestimating the reliability of existing results or gaining a false sense of a field's progress on particular problems. our paper contributes a suite of benchmarks which can be used to anchor future authorship attribution research. these five tasks are a start. additional tasks would be welcome. many forms of writing and document types in widespread use today are not featured in the five tasks we introduce here. short text messages and informal e-mails, in particular, are ubiquitous. yet many individuals' habits of composition vary dramatically when writing in such genres. standard benchmarks for cross-register and cross-genre authorship attribution would likely yield new insights into the strengths and weaknesses of existing approaches.