A B ayesian Approach to Harnessing the Power of LLM s in Authorship Attribution (2024)

Abstract: Authorship attribution aims to identify the origin or author of a document. Traditional approaches have heavily relied on manual features and fail to capture long-range correlations, limiting their effectiveness. Recent advancements leverage text embeddings from pre-trained language models, which require significant fine-tuning on labeled data, posing challenges in data dependency and limited interpretability. Large Language Models (LLMs), with their deep reasoning capabilities and ability to maintain long-range textual associations, offer a promising alternative. This study explores the potential of pre-trained LLMs in one-shot authorship attribution, specifically utilizing Bayesian approaches and probability outputs of LLMs. Our methodology calculates the probability that a text entails previous writings of an author, reflecting a more nuanced understanding of authorship. By utilizing only pre-trained models such as Llama-3-70B, our results on the IMDb and blog datasets show an impressive 85% accuracy in one-shot authorship classification across ten authors. Our findings set new baselines for one-shot authorship analysis using LLMs and expand the application scope of these models in forensic linguistics. This work also includes extensive ablation studies to validate our approach.

## introduction authorship attribution, the process of identifying the origin or author of a document, has been a longstanding challenge in forensic linguistics. it has numerous applications, including detecting plagiarism @xcite and attribution of historical text @xcite . as the digital age progresses, the need for reliable methods to determine authorship has become increasingly important, especially in the context of combating misinformation spread through social media and con-* these authors contributed equally to this work. ducting forensic analysis. the ability to attribute authorship can also lead to challenges around privacy and anonymity @xcite . the field traces its roots back to the early 19th century @xcite , with early studies focusing on stylistic features and human expert analysis @xcite . traditional methods often relied on stylometry, which quantifies writing styles @xcite , and rule-based computational linguistic methods @xcite to deduce authorship. later, statistical algorithms incorporating extensive text preprocessing and feature engineering @xcite were introduced to improve accuracy. however, these methods often struggled with capturing long-range dependencies in text and require careful setup of specific thresholds for various indicators, which can be challenging to select effectively. they also involve designing complex, high-quality features, which can be costly and time-consuming. the advent of deep learning has transformed the landscape of authorship attribution by turning the problem into a multi-class classification challenge, allowing for the capture of more features and addressing more complex scenarios effectively @xcite @xcite . however, these neural network (nn) models often lack interpretability and struggle with generalization in cases of limited samples. despite advancements, the field still faces significant challenges. obtaining large, balanced datasets that represent multiple authors fairly is difficult, and as the number of authors increases, the accuracy of machine learning models tends to decrease. on the other hand, language models, central to modern nlp applications, define the probability of distributions of words or sequences of words and have traditionally been used to predict and generate plausible language. yet, for a long time, these models, including high-bias models like bag-ofwords and n-gram models, struggled to fit the true probability distributions of natural language. deep learning's rapid development has enabled orders of magnitude scaling up of computing and data, facilitating the use of more complex models such as random forests @xcite , character-level cnns @xcite , recurrent neural networks @xcite , and transformer @xcite . the recent rapid evolution of large language models (llms) has dramatically improved the ability to fit natural language distributions. trained on massive corpora exceeding 1 trillion tokens, these models have become highly capable of handling a wide range of linguistic tasks, including understanding, generation, and meaningful dialogue @xcite @xcite . they can also explain complex concepts and capture subtle nuances of language. they have been extensively applied in various applications such as chatbots, writing assistants, information retrieval, and translation services. more impressively, llms have expanded their utility to novel tasks without additional training, simply through the use of prompts and in-context learning @xcite . this unique ability motivates researchers to adapt llms to an even broader range of tasks and topics including reasoning @xcite , theory of mind @xcite and medical scenario @xcite . interestingly, language models have also been explored for authorship attribution @xcite @xcite . recently, research has utilized llms for question answering (qa) tasks within the application of authorship verification and authorship attribution @xcite , though these have primarily been tested in small-scale settings. other approaches have attempted to leverage model embeddings and fine-tuning for authorship attribution, such as using gan-bert @xcite and bertaa @xcite . however, these techniques often face challenges with scalability and need retraining when updating candidate authors. moreover, they require relatively large dataset and multiple epochs of fine-tuning to converge. given the challenges with current approaches, a natural question arises: how can we harness llms for more effective authorship attribution? two aspects of evidence provide insights to an-swer the above questions. first, recent studies on llms have shown that these models possess hallucination problems @xcite . more interestingly, the outputs of llms given prompts may disagree with their internal thinking @xcite . therefore, it is advisable not to rely solely on direct sampling result from llms. second, the training objective of llms is to maximize the likelihood of the next token given all previous tokens. this indicates that probability may be a potential indicator for attributing texts to authors. language models are essentially probabilistic models, but we find the probabilistic nature of llms and their potential for authorship identification remains underexploited. our study seeks to bridge this gap. specifically, we explore the capability of llms to perform one-shot authorship attribution among multiple candidates. we propose a novel approach based on a bayesian framework that utilizes the probability outputs from llms. by deriving text-level log probabilities from token-level log probabilities, we establish a reliable measure of likelihood that a query text was written by a specific author given example texts from each candidate author. we also design suitable prompts to enhance the accuracy of these log probabilities. by calculating the posterior probability of authorship, we can infer the most likely author of a document (figure 1 ). due to the pivotal role of log probability in our algorithm, we coined our approach the "logprob method." our new method has three main advantages: text from unknown author prompt 1 llm logprob 1 author 1's text ... ... ... ... prompt n llm logprob n author n's text softmax figure 1: illustration of bayesian authorship attribution using llm. by applying this technique, we have achieved state-of-the-art results in one-shot learning on the imdb and blog datasets, demonstrating an impressive 85% accuracy across ten authors. this advancement establishes a new baseline for one-shot authorship analysis and illustrates the robust potential of llms in forensic linguistics. ## method our approach to authorship attribution is based on a bayesian framework. given a document whose authorship is unknown, our objective is to identify the most probable author from a set using the capabilities of large language models (llms). we consider a scenario where we have a set of authors @xmath0, . . . , a n } and a set of all possible texts e. given an authorship attribution problem, where each author a i has written a set of texts t i,1 , t i,2 , . . . , t i,m @xmath1, we denote the collection of known texts of an author a i as t(a i ) = (t i,1 , t i,2 , . . . , t i,m i ). for an unknown text @xmath2, we aim to determine the most likely author from the set a. to estimate the author of text u, we use a bayesian framework where the probability that u was written by author a i is given by: here, p (a i ) is the prior probability of each author, assumed to be equal unless stated otherwise, making the problem focus primarily on estimating p (u|a i ). assuming that each author a i has a unique writing style represented by a probability distribution p (•|a i ), texts written by a i are samples from this distribution. to estimate p (u|a i ), we consider the independence assumption: texts by the same author are independently and identically distributed (i.i.d.). thus, the unknown text u is also presumed to be drawn from p (•|a i ) for some author a i and is independent of other texts from that author. notice that although texts are independent under the i.i.d. assumption when conditioned on a particular author, there exists a correlation between the unknown text u and the set of known texts t(a) in the absence of knowledge about the author. this correlation can be exploited to deduce the most likely author of u using the known texts. we then introduce the "sufficient training set" assumption, where: this implies that the training set is sufficiently author 1: tina fey is a successful professional who has missed out on the baby wagon . all her friends have families and she has promotions . desperate for a child she tries a sperm bank but it fails when she is told that she is infertile . in desperation she takes on a surrogate who turns her life upside down . clearly tina fey is the smartest one in the room and she walks through this film seemingly on autopilot and above to everyone around her . what is she doing here ? she is somewhere beyond this film and it shows . its cute and amusing but fey's demeanor promises something on a different plane then the rest of the movie . i think the best way to explain it , or over explain it would be cary grant in a three stooges movie . i think fey can do great things if she wants or can find material that matches her abilities . a good little film . here is the text from the same author: barbet schroeder's portrait of french attorney jacques vergès . you've seen him defending people like klaus barbie , carlos the jackal , pol pot as well as other dictators and terrorists . this is a complex story of a complex man and it essentially tells the tale of the man from world war 2 until today . ( and even at 140 minutes the film leaves a great deal out ) . here is man of his time , who met and defended with many of the famous and infamous people of the last fifty years . he seems to be a man who generally believes in the right of the oppressed to stand up to their oppressors and to have some one to stand up for them . however this is not just the story of a man who fights for the oppressed but it is also the story of a man entangled in things that will cause many to question just how slick a guy is verges . many of the terrorists and dictators he defends are in fact his friends , and he is not doing it for the love of cause but also for the love of the finer things . i liked the film a great deal . to be certain i was lost as to bits of the history and who some people were , but at the same time the film isn't about the history , so much as verges moving through it . this is the story of the man , his causes and to some degree his women . what exactly are we to make of verges ? i don't know , but i sure do think that he and his life make for a compelling tale . i loved that my idea of what verges is changed . i loved that i was completely confused at the end as to what i thought , confused in a way that only a film that forces you to think can do . in the end i don't know what i think . . . in the run-up to the 1972 elections , washington post reporter bob woodward covers what seems to be a minor break-in at the democratic party national headquarters . he is surprised to find top lawyers already on the defence case , and the discovery of names and addresses of republican fund organisers on the accused further arouses his suspicions . the editor of the post is prepared to run with the story and assigns woodward and carl bernstein to it . they find the trail leading higher and higher in the republican party , and eventually into the white house itself . . . . whatever peoples opinions on the watergate ' scandel ' , whether they believe it was a big cover up , or the media got a lot wrong , no one can deny just how powerful and interesting this film really is . pakula directs this very slickly and brings the tension on the two main protagonists very slowly throughout the duration of the movie . redford and hoffman work really well together and are given great support from the rest of the cast . the narration works amazingly well and there is good use of mise en scene and connotations . for example there are a few scenes with the t . v screen in the foreground showing nixon winning his presidential seat again , with . . . here is the text from the same author: barbet schroeder's portrait of french attorney jacques vergès . you've seen him defending people like klaus barbie , carlos the jackal , pol pot as well as other dictators and terrorists . this is a complex story of a complex man and it essentially tells the tale of the man from world war 2 until today . ( and even at 140 minutes the film leaves a great deal out ) . here is man of his time , who met and defended with many of the famous and infamous people of the last fifty years . he seems to be a man who generally believes in the right of the oppressed to stand up to their oppressors and to have some one to stand up for them . however this is not just the story of a man who fights for the oppressed but it is also the story of a man entangled in things that will cause many to question just how slick a guy is verges . many of the terrorists and dictators he defends are in fact his friends , and he is not doing it for the love of cause but also for the love of the finer things . i liked the film a great deal . to be certain i was lost as to bits of the history and who some people were , but at the same time the film isn't about the history , so much as verges moving through it . this is the story of the man , his causes and to some degree his women . what exactly are we to make of verges ? i don't know , but i sure do think that he and his life make for a comcomprehensive to unambiguously differentiate authors, leading to: we use large language models (llms) to estimate p (u|t(a i )), which represents the probability that a new text u was written by the author of a given set of texts t(a i ). the probability nature of language models means that they typically calculate the probability of a token or a sequence of tokens given prior context. for a vocabulary set σ, the input to a language model might be a sequence of tokens x 1 , . . . , x @xmath3, and the model's output would be the probability distribution p llm (•|x 1 , . . . , x m ), typically stored in logarithmic scale for numerical stability. when using an autoregressive language model, we can measure not only the probability of the next token but also the probability of a subsequent sequence of tokens. for instance, if we have a prompt consisting of tokens x 1 , . . . , x @xmath4, and we want to measure the probability of a sequence y 1 , . . . , y @xmath5, we calculate: to estimate p (u|t(a i )) for authorship attribution, we define: the prompt construction can vary, providing flexibility in how we use the model to estimate probabilities. our method involves constructing a prompt steering the llm uses to predict the likelihood that the unknown text was written by the same author (figure 2 ). in summary, our approach is straightforward and simple. by leveraging the capabilities of large language models, we calculate the likelihood that an unknown text originates from a known author based on existing samples of their writing. this probability assessment allows us to identify the most likely author from a set without the need for fine-tuning or feature engineering. ## experiments firstly, we evaluate different methods for author attribution in section 4.1, noting that our logprob method significantly outperformed qa-based methods in accuracy and stability across datasets. then, we study the impact of increasing candidate numbers on performance in section 4.2, where our method maintained high accuracy despite a larger pool of candidates. next, in section 4.3, we analyze prompt sensitivity, concluding that while prompt use is crucial, variations in prompt design did not significantly affect the performance. further, in section 4.4, we explore bias in author attribution and in section 4.5, we measure performance variations across different subgroups. finally, in section 4.6, we compared the efficiency of different author attribution methods. ## conclusion in this paper, we study the problem of authorship attribution. we demonstrate the effectiveness of utilizing pre-trained large language models (llms) for one-shot author attribution. our bayesian approach leverages the probabilistic nature of language models like llama-3 to infer authorship. our method does not require fine-tuning, therefore reduces computational overhead and data requirements. our experiments validate that our method is more effective and efficient compared to existing techniques.