Authorship Attribution of Late 19th Century Novels using GAN - BERT (2023)

Abstract: Authorship attribution aims to identify the author of an anonymous text. The task becomes even more worthwhile when it comes to literary works. For example, pen names were commonly used by female authors in the 19th century resulting in some literary works being incorrectly attributed or claimed. With this motivation, we collated a dataset of late 19th century novels in English. Due to the imbalance in the dataset and the unavailability of enough data per author, we employed the GANBERT model along with data sampling strategies to fine-tune a transformer-based model for authorship attribution. Differently from the earlier studies on the GAN-BERT model, we conducted transfer learning on comparatively smaller author subsets to train more focused author-specific models yielding performance over 0.88 accuracy and F1 scores. Furthermore, we observed that increasing the sample size has a negative impact on the model’s performance. Our research mainly contributes to the ongoing authorship attribution research using GAN-BERT architecture, especially in attributing disputed novelists in the late 19th century.

## introduction authorship attribution identifies authors of a given set of unknown documents @xcite @xcite . conventional techniques and neural networks are the two main authorship attribution methods. the studies on the conventional approaches typically focus on feature engineering and stylometry. the deep learning approaches have been gaining popularity recently due to the superior results compared to the conventional approaches. furthermore, authorship attribution can be tackled in two ways: closed-set and openset attribution. in closed-set attribution, an author is selected from a set of candidate authors, whereas in open-set attribution, the target author may not be included in the candidate authors' list. applications of authorship attribution are employed in various domains, such as digital forensics @xcite , social media analysis @xcite @xcite and digital humanities @xcite . in historical texts, the authorship styles may contain socio-linguistic characteristics due to the century in which the author lived, idea movements inspired by the author, and languagespecific attributes. also, in written texts, the genre and topics are crucial in defining the author's style. several pieces of research have been undertaken in the literature and historical domains, for instance, identifying anonymous or disputed texts @xcite @xcite . the work presented by @xcite analyses the federalist papers, which involves 85 articles and essays written by alexander hamilton, james madison and john jay. another application of authorship attribution in literature is resolving doubted authorships. for instance, @xcite investigate whether c.s. lewis wrote the dark towers. the shakespearean authorship dispute was addressed by @xcite . furthermore, attributing the author is one of many variations in authorship applications, as research directions are in different domains, such as attributing to the publication year and identifying the literary genre and the topic. one such example is @xcite which predicts the date of authorship in historical texts. this research proposes a gan-bert-based model to enhance transformer-based authorship attribution in late 19th-century novels. to our knowledge, this is the first attempt to ensemble gan and bert models and, precisely, the gan-bert model to address authorship attribution in literary texts. in some of the recent works on authorship attribution, the models were trained in a controlled setting and had less elaboration on the data preparation stage, resulting in the poor reproducibility and generalisation of these models. here, we present an end-to-end process from domain selection to dataset collection with insights to experiment planning. an authorship attribution model highly depends on the number of authors represented in the training dataset and the text available per each author. most of the related works emphasise controlled training environments. to improve the model's generalisation and ability to perform well on robust scenarios, it should be identified how much the model depends on the number of authors in the training dataset and the amount of text by each author. we use a normalised dataset of 20 novels per author to avoid dataset imbalance. therefore, to identify how much data provides better model performance, we control the text data sample size drawn from the book text. therefore, the research questions in this study are as follows: rq 1: how to effectively utilise the gan-bert model for authorship attribution? rq 2: how does the number of authors in the dataset impact the gan-bert performance for authorship attribution? rq 3: how does the amount of text data (i.e. sample size) drawn from each novel affect the gan-bert performance for authorship attribution? the remainder of the paper is organised into several sections: section 2 demonstrates a brief literature survey. then section 3 describes the proposed model's architecture, and section 4 presents the dataset collection and preparation. section 5 elaborates on the experiment design, focusing on the research questions, section 6 summarises the results and findings obtained, and finally, section 7 involves the concluding remarks and future directions. ## related work texts vary in terms of topic, sentiment and style. according to @xcite , information about the authors can be extracted from the style of their written documents. the task involves identifying the author from unknown documents, known as authorship attribution, which breaks into two major tasks: authorship identification and authorship verification. authorship identification is identifying a document's author by comparing a set of candidate authors @xcite . authorship identification can be interpreted as a binary classification problem, whereas authorship attribution is a multi-class classification problem. authorship verification is a fundamental problem in authorship attribution which focuses on finding whether the considered person wrote one or more documents or not. authorship verification is comparatively challenging with less data @xcite . with the popularity of deep neural networks for nlp applications, recent authorship attribution research shares a similar trend. the works of bagnall (2015a); hosseinia and mukherjee (2018); @xcite are examples of neural network-based models in authorship attribution. additionally, transfer learning also proved to have astonishing results. @xcite introduce a deep authorship verification using new metrics: dv-distance and dv-projection, which utilise pre-trained language models. their work highlights the utilisation of pre-trained language models in our approach. character and n-gram-based cnn @xcite , syntax-augmented cnn @xcite , and convolutional siamese networks @xcite are some other authorship attribution models which utilise deep learning techniques. these deep learning-based applications provide valuable insights for our approach to utilising the gan-bert model for authorship attribution tasks. language models (lm) used in the authorship tasks can be categorised as n-gram-based and neural network-based @xcite . ge et al. (2016) used a neural network-based language model. the works of @xcite present a character-level rnn-based lm combining a multiheaded classifier. to address the cross-domain problem, barlas and stamatatos (2020) extended bagnall (2015b)'s works for closed-set authorship attribution by combining a multi-headed lm with a pre-trained lm. according to @xcite , having a normalised corpus is crucial for the performance of cross-domain authorship attribution. bertaa @xcite is the recent fine-tuned form of the pre-trained bert model for the authorship attribution task, which presents extensive experiments on various datasets: enron email @xcite , blog authorship @xcite and imdb @xcite . although pre-trained models have gained popularity and promising results in some authorship tasks, the performance of such models highly depends on the training set. generative adversarial networks (gan) are used in authorship-related tasks to prevent adversarial attacks, mainly in the authorship obfuscation problem where one's writing style is masked. @xcite introduce source code authorship verification using gan models and multi-head attention. a 4 n t @xcite is a ganbased style transformation to perform authorship obfuscation learned from data via adversarial training and sequence-to-sequence lms. @xcite presents an lstm-gan classifier to recognise imitations generated by the a 4 n t @xcite model. @xcite presents a data augmentation approach to authorship attribution in weibo text using wasserstein-gan to generate samples of the positive class. the class imbalance problem is hard to avoid in real-world scenarios, particularly in authorship attribution. @xcite introduced a novel strategy to produce synthetic data for the authorship identification task. the approach that stamatatos (2018) mentioned is segmenting the training texts into text samples, considering the training size of the class. the works of @xcite highlight how much data is required to identify authors across different languages and genres. the findings in @xcite show that the minimum sample range is 2500-5000, representing the two ends for latin, english, german, polish, and hungarian datasets. further experiments by @xcite attempt to identify the minimum sample size by removing text one by one from the training set, which yields that 2000 words sample size is appropriate. also, @xcite emphasises that this finding depends strongly on the authors. @xcite propose a hybrid pca and smote approach of oversampling, which reports outperforming the state-of-theart accuracies. the stylometric set similarity (s3) method presents the authorship attribution task as a set similarity problem by considering 3000 novels from 500 authors curated from project gutenberg @xcite . @xcite present a knn-resampling approach to authorship identification by simulating samples from 2 texts. in previous research on authorship attribution, the combination of gan and transformer models has not yet been explored. furthermore, to the best of our knowledge, no attempt has been made to use the gan-bert model specifically for the task of authorship attribution, especially with sampling strategies for many authors and limited data. the critical literature analysis suggests that deep neural networks in authorship attribution would show promising performance with well-designed sampling strategies. here, we propose gan-bert model for authorship attribution along with various sampling strategies, and analyse how transferlearning would support the proposed model in literary domain. ## gan-bert model for authorship attribution let a be a collection of authors of interest, @xmath0, a 2 , . . . , a n }, where n is the total number of authors in a. the document set belonging to each author forms the complete dataset @xmath1, t a 2 , . . . t a n } where t a i is the document set attributed to the author a i in the dataset. given a text, t u of an unknown author u, the proposed model assigns the text to the most likely author from a. gan-bert @xcite combines bert-based models and semi-supervised gan @xcite . figure 1a illustrates the gan-bert model architecture, where discriminator d is utilised to classify examples and generator g generates fake examples f. the discriminator takes the vector representations returned via bert for unlabeled u and labelled l input texts. when training is complete, g is discarded from the model to use the rest of the model for inference. in contrast to gan-bert @xcite , which utilises a semi-supervised gan model @xcite with labelled and unlabeled data, we train the gan-bert model with labelled data only. the discriminator d is trained over n+1 classes to assign the true samples to a class from {1, 2, 3, ..., n }. the fake sample generated from the generator g represents the (n + 1) th class. the discriminator is suitable for detecting authorship obfuscation and forgery since it is trained with fake samples similar to the original author-written texts. figure 1b illustrates the modified gan model. the gan-bert model generally shows superior results for classification tasks with limited labelled data. furthermore, the intuition to use gan-bert for authorship attribution is that, due to the fake data generated in the generator, it considers not only the real writing styles, but also the possible fake writing styles that are synthesised. ## experiment design we conducted experiments on different dataset subsets and different model configurations to address the following: 1. random sampling author combinations 2. the impact of transfer learning 3. number of authors in dataset 4. text sample size per novel we explored the gan-bert model under two dimensions: random sampling and transfer learning. as illustrated in figure 2 , the 20 novels per each author from the 20-author dataset provide different combinations under different numbers of authors. therefore, first, we manually selected authors per each n-author case and then randomly sampled 50 and 100 author combinations. in transfer learning experiments, we compared the performance of manually selected sample sets under standalone training and transfer learning from the 20-author dataset to each n-author case. in a practical scenario of authorship attribution, the number of authors to compare would vary. therefore, we experimented with the gan-bert model response for different numbers of authors in the dataset. also, the text sample size drawn from a novel can be varied when representing the novel text due to varying text lengths. we used the manual sampling of authors to identify any trend towards the text sample size drawn from a novel. in the default setting, unless specified, we used 20 samples per novel drawn sequentially from the book text for training and testing. we first trained the base model on 20-authors for 10 epochs, using adam optimiser, one hidden layer for both generator and the discriminator, a dropout rate of 0.2, batch size of 8, a warm-up proportion of 0.1, and learning rate of 1e-5 for both generator and the discriminator. then the pre-trained 20-author model was used for transfer learning on smaller subsets of each case in {2, 4, 6, 8, 10, 12, 14, 16, 18}-author counts and trained further on these sub-sets for 5 epochs. we compared the proposed gan-bert model with different baseline models such as word-level tf-idf, character n-gram, stylometric features @xcite and @xcite on the 20-authors dataset, 18-authors dataset, imdb, and blog authorship datasets. these baseline experiments provide insights into how the created datasets performed with other baseline models and how other datasets would perform with the proposed gan-bert model. to be consistent with ## conclusion this research proposes a gan-bert-based model for authorship attribution in late-19th-century novels. our primary focus is identifying how the author counts and the text sample size per book affects the model's performance. the manually selected five authors' combinations indicate that the model's performance degrades when the number of authors increases. the declining trend is the same for transfer-learning models, although the overall performance is better than the standalone models. additionally, we experimented with how transfer learning has improved the mean accura-cies over manually selected author sample sets for each n-author case. a future improvement would be an experiment around few-shot and zero-shot tests. furthermore, it would be interesting to experiment with different gan and transformer models replaced in this model architecture.