Forged- GAN - BERT : Authorship Attribution for LLM -Generated Forged Novels (2024)

Abstract: The advancement of generative Large Language Models (LLMs), capable of producing human-like texts, introduces challenges related to the authenticity of the text documents. This requires exploring potential forgery scenarios within the context of authorship attribution, especially in the literary domain. Particularly,two aspects of doubted authorship may arise in novels, as a novel may be imposed by a renowned author or include a copied writing style of a well-known novel. To address these concerns, we introduce Forged-GAN-BERT, a modified GANBERT-based model to improve the classification of forged novels in two data-augmentation aspects: via the Forged Novels Generator (i.e., ChatGPT) and the generator in GAN. Compared to other transformer-based models, the proposed Forged-GAN-BERT model demonstrates an improved performance with F1 scores of 0.97 and 0.71 for identifying forged novels in single-author and multi-author classification settings. Additionally, we explore different prompt categories for generating the forged novels to analyse the quality of the generated texts using different similarity distance measures, including ROUGE-1, Jaccard Similarity, Overlap Confident, and Cosine Similarity.

## introduction early applications of generative models for literary text generation go back to the works by @xcite for automatic poetry generation. moreover, the most recent attempts to generate poems via text generative models were described by @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite . chatgpt and other powerful generative models generated stories by investigating different prompting mechanisms @xcite . in most recent attempts, the researchers have explored human-ai co-creation in literary areas, for instance, in the works of @xcite ; @xcite . also, the work in uludag @xcite performed qualitative and quantitative methods to test the creativity of chatgpt in psychology. uludag @xcite finds that chatgpt has some level of creativity but also imposes limitations, such as a limited understanding of the context and the inability to generate original ideas. with the popularity of generative llms for creative content generation, there have been issues observed on well-known book-selling platforms such as amazon, where ai-generated books are presented for sale under human writers' names with and without the original involvement of the authors @xcite . responding to this situation, platforms such as amazon have taken measures, such as ordering self-publishing authors to explicitly declare whether their content is machinegenerated @xcite . to address these challenges, organisations such as the 'authors guild' and 'the society of authors are actively pursuing legislative protection for human authors from such forged literary works under their names @xcite . as a preliminary step to proposing possible solutions for such authorship issues, particularly considering a use case of machine-generated novels, we explored the ability to utilise gan-bert @xcite to discriminate forged novels generated by chatgpt from the texts of the original novels. the internal architecture of the gan-bert models combines a generator capable of generating fake texts similar to real ones. since the gan-bert model already identifies fake texts @xcite , we want to test the hypothesis that it will perform well in detecting ai-generated novels in a similar style to the original novels. this paper presents the forged-gan-bert model, specifically designed to identify forged novels within the context of authorship attribution. we utilised 20 novels per author during this study, considering 5 randomly selected authors, prompting chatgpt to forge the books' styles with zero-shot prompting. in contrast to a recent study conducted by @xcite , which is primarily on online posts, our research focuses on literary works. we utilise the gan-bert model to conduct a dual analysis of the forged texts, combining forged novels generated within the gan generator and those created by llm, like chatgpt. also, in @xcite , they have used fine-tuning to generate ai text, but instead, we prompted chatgpt to forge or disguise the author's style. to our knowledge, this is the first study using chatgpt prompts to generate similar novelist styles and to utilise the gan-bert model to detect ai-generated novels. our study is steered by the following formulated research questions: rq 1 what are the implications of utilising various text similarity metrics in assessing the quality of forged novels? rq 2 is it possible to distinguish between human novels vs llm-generated novels with the forged-gan-bert? the remainder of the paper is organised into several sections: section 2 provides a brief literature survey. then, section 3 describes the dataset information. section 4 elaborates on the quality analysis of the forged novels against different prompt categories, emphasising the rq1. section 5 outlines the forged-gan-bert model architecture related to the rq2. finally, section 6 adds concluding remarks and future directions. ## related work text generation models, aka natural language generation (nlg), generate text closer or indistinguishable from human text or any other input format, such as image or video, which can be categorised into completion generation, text-to-text generation, and inference. large language models (llm) such as bert @xcite , t5 @xcite , @xcite , @xcite , gpt-4 (openai, 2023b), were trained on larger datasets with billions of parameters, which can process massive loads of data and provide highly accurate results. bertbased models @xcite and t5 @xcite were built on encoder-only or encoderdecoder architectures, respectively, and are flexible for adapting to many tasks by means of finetuning. @xcite investigate the scalability factor of llms in terms of few-shot learning towards multilingual tasks and source code generation tasks. recent llm text generators mainly focus on the models' scalability and increasing the models' capacities compared to the predecessor models. advanced conversational models can be optimised for massive, high-quality data generation via prompt engineering @xcite on the api of interface level. by using prompt engineering in llms such as flan @xcite , chat-gpt (openai, 2023a), llama @xcite , and gpt-4 (openai, 2023b), models can be utilised to curate new datasets @xcite @xcite or as data augmentation strategies @xcite @xcite . @xcite discuss machine learningbased fake news detection techniques with a comparison to deep learning models. tweep-fake @xcite detects deepfake tweets generated by bots based on different text generation techniques such as rnn, markov chains, lstm, and gpt-2. deid-gpt @xcite presents a zero-shot medical text de-identification based on gpt-4 in the domain of clinical notes. in the area of authorship attribution, two main approaches exist for author identification: traditional approaches such as stylometric methods @xcite @xcite , and deep learning-based approaches @xcite @xcite . stylometric approaches focus on stylometric feature identification and utilising them in classification models. moreover, ensemble models such as bacciu et al. (2019) ; @xcite combine stylometric and deep learning mechanisms to enhance the authorship attribution. authorship obfuscation, a sub-discipline of authorship attribution, specifically addresses hiding authors' writing styles and identifying such attempts @xcite . the gan-bert model ( croce et al., 2020 ) integrates bert-based models with the semi-supervised gans, as illustrated in figure 1 . the gan-bert model is being used for a range of applications such as sentiment analysis @xcite , authorship attribution @xcite , text classification @xcite , and multi-task learning @xcite . in contrast, considerable research has been performed on fake news, tweets, medical, and poems, but still limited attention to novels. further analysing original and generated text, specifically for the literary domain, considering ai-generated forged text, has yet to be addressed. ## quality analysis of forged novels it is important to evaluate the quality of the generated forged novels (f ) by comparing them with the original novels (o). different prompt categories (p ) have been considered, with @xmath0, 'similarity', 'identification', 'chapter', and 'temperature'}. although an infinite range of prompts can be used for these experiments, we considered only a finite set of 18 different prompts. since the objective of each prompt is to generate a similar novel to a given original novel, we were interested in the generated text quality and the similarity, hence utilised a range of metrics such as rouge-1, jaccard similarity, overlap confident, and cosine similarity. each averaged similarity score can be calculated as follows, considering a basic averaging approach: with o i an original novel, p j a prompt category and f ij the corresponding forged novel of o i generated using p j . for a given similarity measure m, with @xmath1, jaccard similarity, overlap coefficient, cosine similarity}, the distance d between o i and f ij is represented as d m (o i ,f ij ) . n is the total number of pairs of novels we average over. for each prompt category p , the results of the averaged distribution for each prompt sub-category (see ## forged-gan-bert model in the proposed model architecture as in figure 4 , we are considering two aspects in addressing forged texts in authorship attribution: the proposed forged-gan-bert model differs from the original gan-bert model @xcite by incorporating a dual forged text analysis curated explicitly for the authorship attribution task. in contrast to the original model, only labelled data were used in this approach. across different experiment settings, we provide the model with different ratios of forged novels and original novels and varying numbers of predicted classes via the discriminator (d). the original and forged novels are considered real data, passing through the bert model to generate text embedding. subsequently, using a controlled noise distribution from the latent space, the generator (g) produces fake text resembling the real text, which is used as an input to the discriminator. the calculated loss is then used to adjust the generator and discriminators' training parameters. ## conclusion in conclusion, the introduced forged-gan-bert model addresses the challenges of authorship attribution in machine-generated forged novels, explicitly and implicitly considering a dual forged text analysis approach. the results suggest that the proposed model outperforms other considered baseline models in identifying forged novels in singleauthor and multi-author classifications. additional evaluation on the generated forged novels against different prompts utilised various similarity distance metrics such as rouge-1, jaccard similarity, overlap coefficient, and cosine similarity. the reported results indicated that the 'chapter' configurations have more impact on generating novels similar to the original text. this evaluation can be extended for a probabilistic distribution approach to evaluate the forged novels for all the possible prompts in the infinite series of the prompts.