Can Large Language Models Identify Authorship? (2024)

Abstract: The ability to accurately identify authorship is crucial for verifying content authenticity and mitigating misinformation. Large Language Models (LLMs) have demonstrated exceptional capacity for reasoning and problem-solving. However, their potential in authorship analysis remains under-explored. Traditional studies have depended on hand-crafted stylistic features, whereas state-of-the-art approaches leverage text embeddings from pre-trained language models. These methods, which typically require fine-tuning on labeled data, often suffer from performance degradation in cross-domain applications and provide limited explainability. This work seeks to address three research questions: (1) Can LLMs perform zero-shot, end-to-end authorship verification effectively? (2) Are LLMs capable of accurately attributing authorship among multiple candidates authors (e.g., 10 and 20)? (3) Can LLMs provide explainability in authorship analysis, particularly through the role of linguistic features? Moreover, we investigate the integration of explicit linguistic features to guide LLMs in their reasoning processes. Our assessment demonstrates LLMs’ proficiency in both tasks without the need for domain-specific fine-tuning, providing explanations into their decision making via a detailed analysis of linguistic features. This establishes a new benchmark for future research on LLM-based authorship analysis.

## introduction authorship analysis is the study of writing styles to determine the authorship of a piece of text, impacting areas from forensic investigation, such as distinguishing between murders and suicides @xcite , to tracking terrorist threats @xcite . it addresses challenges in digital forensics and cybersecurity, including the fight against misinformation, impersonation, and cyber threats such as phishing and deceptive social media posts @xcite @xcite . authorship analysis is essential for tracing cyber threats to their sources, combating plagiarism to uphold intellectual property rights @xcite , and identifying compromised accounts @xcite . in addition, it helps link user accounts across social platforms @xcite and detect fraudulent activities such as fake reviews @xcite . historically, authorship analysis relied on methods based on human expertise to distinguish between authors @xcite . later, a line of research known as stylometry emerged, which developed various features to quantify writing styles @xcite . the evolution continued with the adoption of rule-based computational linguistic methods @xcite . the development of statistical algorithms provides the capability to handle data with higher dimensions, enabling more expressive representations. these methods relied heavily on extensive text preprocessing and feature engineering @xcite . compared to traditional statistical methods, deep learning techniques require less feature engineering. among these techniques, pre-trained language models (ptms) are widely used for representing authorship @xcite . these models, built predominantly on bert-based architectures @xcite and contrastive learning paradigms, demonstrate efficacy in domainspecific applications. however, they fall short in cross-domain scenarios @xcite . the performance of these methods also declines significantly with shorter query texts @xcite and limited data from the candidate authors. this reduction in performance limits their applicability in real-world situations, where data scarcity and diversity are the norms. while some studies have attempted to overcome these challenges by applying text style transfer to learn content-independent style representations, they have not addressed the cross-domain issue effectively @xcite . these deep learning methods require extensive time and labeled data for training, are not effectively applicable across different data domains, and suffer from limited explainability. despite the rapid development of llms, there has been insufficient analysis and evaluation of their capabilities in authorship analysis @xcite . some initial studies have utilized gpt-3 @xcite for annotating data @xcite before employing a t5 encoder @xcite for learning representations of authorship. llms have demonstrated proficiency in zero-shot learning scenarios within domains lacking extensive resources @xcite . however, their ability to grasp subtle nuances of language and extract critical features for authorship identification has not been extensively examined. consequently, this paper aims to investigate the potential of llms for authorship identification by addressing the following research questions: we also propose a prompting technique named linguistically informed prompting (lip) to guide llms in identifying linguistic features that are used in practice by forensic linguists @xcite . this approach exploits the inherent linguistic knowledge embedded within llms, unleashing their potential to discern subtle stylistic nuances and linguistic patterns indicative of individual authorship. figure 1 demonstrates the application of the lip method in verifying authorship through linguistic feature analysis using gpt-4. it compares two texts from the blog dataset @xcite . analysis can provide specific linguistic evidence such as the use of informal language, punctuation patterns, and typographical errors. our empirical evaluation includes data with different genres and topics to validate the robustness and versatility of llms. the results demonstrate that llms can effectively perform zero-shot authorship verification and attribution, thereby obviating the need for fine-tuning. with the introduction of linguistic guidance, llms are further leveraged for authorship analysis, where our lip technique sets a new benchmark for llm-based authorship prediction. the key contributions of this work are summarized as follows: ## datasets we choose two representative datasets to highlight the importance of user-generated content such as emails and social media posts. the first dataset is the enron email dataset, which consists of approximately half a million messages from senior enron managers. this dataset offers insights into corporate communication, featuring long texts and a high variance in text-per-author @xcite . the second dataset is the blog authorship attribution corpus @xcite . this corpus includes blog articles from 2004 and earlier, collected from blogger.com, comprising over 680,000 text 1: this is a typical pg-13 teen comedy, with a pretty original idea (to the best of my knowledge). the plot is good enough for this type of movie, but apart from the original concept, it seemed too slow and rarely very funny. the characters are believable, although a few come across as total dimwits, and the acting is good, mostly. the humor, while somewhat rare, is mostly pretty funny. there are some very entertaining court scenes near the end, which i enjoyed very much, as i'm a big fan of court dramas. the film is too slow, though, and if i had seen it at the cinema, i might have fallen asleep. i found it worth sitting through for the final court scenes, though, and i'd say that they are easily the funniest thing in the movie. text 2: i have to admit, i usually hate stephen king films. i've seen the shining, carrie and it. i hate all of those. but this one was really good. it was scary and atmospheric, all the way through. the music was great, very moody and scary, the basic theme was pretty good, and the acting was, if not great, sufficient. the effects were mostly good, though i thought the moving corn, was, well, corny (no pun intended). it's interesting, all the way through, and has more likable main characters than your everyday horror movie. both main characters are pretty well developed through the movie, and you actually care about them. this is a movie you should watch, at least once, if you like horror movies, even if you hate stephen king (i do, and i loved the movie). (a) prompting with no guidance (c) prompting with grammar guidance (d) linguistically informed prompting (lip) (b) prompting with style guidance prompt: verify if two input texts were written by the same author analysis: the input texts are not written by the same author. text 1 is a positive review of a horror movie, while text 2 is a review of a teen comedy. answer: false prompt: verify if two input texts were written by the same author. analyze the writing styles of the input texts, disregarding the differences in topic and content. analysis: the writing styles of the two texts are quite different. text 1 focuses on a pg-13 teen comedy movie, discussing the plot, characters, humor, and entertainment value. it uses phrases like 'too slow,' 'rarely very funny,' and 'worth sitting through for the final court scenes.' on the other hand, text 2 discusses a stephen king film, emphasizing its scariness, atmosphere, music, acting, and likable main characters. it uses phrases like 'scary and atmospheric,' 'likable main characters,' and 'if you like horror movies, even if you hate stephen king.' the tone, vocabulary, and focus of the two texts are distinct, indicating different writing styles. answer: false prompt: verify if two input texts were written by the same author. focus on grammatical styles indicative of authorship. analysis: the analysis of the grammar style of the input texts suggests that they are not written by the same author. text 1 uses more descriptive language and has longer sentences, while text 2 uses simpler language and has shorter sentences. additionally, text 1 contains more punctuation marks and uses parentheses for clarification, while text 2 does not. these differences in writing style indicate that the texts are likely written by different authors. answer: false prompt: verify if two input texts were written by the same author. analyze the writing styles of the input texts, disregarding the differences in topic and content. reasoning based on linguistic features such as phrasal verbs, modal verbs, punctuation, rare words, affixes, quantities, humor, sarcasm, typographical errors, and misspellings. analysis: both texts exhibit a conversational and informal tone, with frequent use of personal pronouns and subjective opinions. they both use a mix of simple and compound sentences, and there is a noticeable use of parenthetical statements to add additional commentary. both texts also employ humor and sarcasm, as seen in phrases like 'total dimwits' and 'corny (no pun intended)'. the punctuation style is similar, with a liberal use of commas and occasional ellipses. there are also some typographical errors and informal language, such as 'pretty original idea' and 'pretty well developed'. both texts use modal verbs like 'might' and 'should', and there is a consistent use of phrasal verbs and colloquial expressions. overall, the linguistic features and writing style are quite similar, suggesting that the texts were likely written by the same author. answer: true posts from more than 19,000 authors, averaging 35 posts per author. the texts in this dataset are relatively short, with an average length of 79 tokens for the top five authors. for data preprocessing, we remove duplicate texts and authors with fewer than two texts and filter out non-english texts. if the data formulation is not balanced, we observe that llms tend to predict any two given texts are written by different authors. llms are trained on varied datasets with many authors, which makes them better at detecting differences in writing style than similarities. the lack of multiple works by the same author in training data may prevents llms from learning individual authorial nuances. for our experiments, we organize and sub-sample these datasets for authorship attribution and authorship verification tasks separately. for the authorship verification task, we ensure a balanced distribution of positive and negative cases, meaning half of the texts are from the same author, and the other half are written by different authors. we sample 30 pairs of texts and conduct experiments three times for each dataset. we also make sure that all the authors and texts are unique. for the authorship attribution task, we ensure that the classes and authors are balanced: in each of the three repetitions, every query text is written by a different author, and all candidate authors are unique with only one correct author who also wrote the query text. we randomly select 10 or 20 different texts from different authors for every repetition. all texts in our sampled subsets are unique. we have saved the sampled data and used the same subsets across all baselines. the preprocessing code and sampled data are available on our github repository. ## authorship verification (rq1) authorship verification involves assessing whether a single candidate author is the correct author of a query text. this process can be formulated as a one-class classification problem @xcite . an important variant of authorship verification is to predict whether two or more given pieces of text were written by the same author or not @xcite . the complexity of the author verification problem is amplified when it involves only a pair of documents for comparison. this scenario substantially limits the amount of reference material available for analysis. in this work, we address the problem of determining if two texts were authored by the same author. the difference between "lip", "no_guidance", "little_guidance", and "grammar_guidance", lies in their specificity and focus. "no_guidance" is the prompt that only offers a basic task description. "little_guidance" narrows this down by emphasizing writing style over content or topic differences, while "grammar_guidance" further specifies a focus on grammatical style that reflects authorship. "lip" provides a specific list of linguistic features to analyze, such as phrasal verbs, modal verbs, punctuation, and typographical errors, ensuring a comprehensive and focused analysis. such linguistic guidance allows for a more nuanced assessment, particularly effective in cases where an author writes on varied topics across different domain, thereby making "lip" superior in authorship verification. figure 6 in appendix f shows the prompts we used for this task. comparing the performance of llms with the bert model in our zero-shot setting is not straight in contrast, traditional pre-trained models, such as bert, require fine-tuning along with a prediction head (typically a trained machine learning classifier) to map hidden embeddings to the final output. our experiments indicate that employing cosine similarity with bert embeddings, which are predominantly distributed around 0.9, makes it challenging to distinguish authorship for verification purposes in this zero-shot setting. the experimental results for both tf-idf and bert are provided in appendix d. evaluation metrics for this task include accuracy, precision, recall, and f1 score. accuracy is a fundamental metric that measures the proportion of correct predictions out of the total predictions made. precision focuses on the proportion of true positive predictions within the pool of positive predictions, evaluating the model's ability to avoid false positives. recall assesses the model's ability to identify all actual positives, reflecting its capability to minimize false negatives. the f1 score harmonizes precision and recall by providing a single metric that balances both aspects through the calculation of their harmonic mean. the experiment results are demonstrated in figure 2 and 3. they provide a comparative analysis of the performance of llms in authorship verification tasks across two different datasets. four models are evaluated with four different prompt settings. gpt-4 turbo consistently outperforms the other models in both datasets, indicating its superior capability in understanding authorship. the lip method generally yields the highest scores across all metrics for most models. across both datasets, performance metrics improve as the level of prompt guidance increases from no guidance to lip. this trend underscores the importance of linguistic guidance in leveraging llms for authorship verification. the analysis of these figures reveals that the effectiveness of llms in authorship verification tasks can be significantly influenced by the type of prompt guidance provided. ## authorship attribution (rq2) in this section, we present a comprehensive analysis of experiments conducted to evaluate the efficacy of our proposed models on the zero-shot authorship attribution task. we selected the blog and enron email datasets to ensure a robust assessment across different domains and genres. figure 7 in appendix f shows the prompts of this task, we utilize four prompt similar to the authorship verification task, with lip being the most effective due to its linguistic guidance effect. the experiments were structured to compare the performance of llms not only against each other but also against established benchmarks in the field, such as tf-idf and bert-based models. authorship attribution, the task of determining the most likely author of a given text from a set of candidates, is commonly formulated as a multiclass, single-label text classification problem. tables 1 and 2 provide a overview of the performance of various models. these models were evaluated across two different datasets (blog and email) with varying numbers of candidate authors (10 and 20). weighted f1, micro f1, and macro f1 are used as evaluation metrics. weighted f1 gives an average f1 score weighted by class size. micro f1 calculates the overall average f1 score, combining all classes, and is sensitive to class imbalance. macro f1 computes the unweighted average of f1 scores across classes, treating each class equally, ideal for assessing minority class performance. we also tested llama 2 and mistral 7b. however, input texts from the datasets we used for evaluating other llms are too long and exceed the context limit of llama 2 because of their context length limitations (4k tokens for llama 2 and 8k for mistral, versus 16k for gpt-3.5 turbo and 128k for gpt-4 turbo). therefore the experiment on mistral are shown in a separate table ( ## explainability (rq3) llms can provide explanations in natural language regarding their decision-making processes and provide linguistic evidence that references the original text, which is invaluable for verifying and understanding their conclusions. our lip method improves upon this by generating meaningful analyses of writing styles, which are instrumental in distinguishing between authors. we have made all the input texts and the corresponding authorship analyses, as produced by llms, accessible in our github repository. figure 4 displays word clouds that represent the outputs from gpt-4 during the authorship verification task. these visualizations demonstrate the model's focus and specificity in analysis, where the density and size of terms indicate their prominence in the output. as guidance increases from none to lip, there is a clear shift from general and diverse terms to more specific linguistic features. the word cloud for lip, being the most effective, underscores a thorough analysis by highlighting particular linguistic characteristics. similarly, the word clouds in figure 5 illustrate that llms can offer in-depth explanations for authorship attribution tasks. the effectiveness and focus of these explanations can be significantly improved through explicit linguistic guidance, which directs the model to base its decisions on linguistic attributes used in practice @xcite . the word cloud of the lip method prominently features terms such as "humor", "sarcasm", "casual", and "colloquial." this demonstrates that with lip, the llm is steered towards making decisions grounded in linguistic features, especially high-level and complex features such as humor and sarcasm. the specificity achieved through the lip method highlights the model's ability to provide clear and focused explanations for its authorship decisions, offering a notable improvement over traditional methods that rely on hidden embeddings. the enhanced clarity in the llm's outputs not only facilitates a better understanding of the decision-making process but also has the potential to increase the reliability of the authorship analysis process. ## related work in this section, we review the literature on traditional and contemporary methods of authorship analysis, as well as research on utilizing llms for authorship analysis and related tasks. ## conclusion this paper explores how to leverage llms for authorship analysis. through comprehensive evaluation, it demonstrates that llms, equipped with the novel linguistically informed prompting (lip) technique, excel at identifying authorship without the need for domain-specific fine-tuning. by directly applying our end-to-end methods to authorship attribution and verification tasks, we aim to bypass the intermediate steps of feature extraction and manual annotation. this approach not only surpasses traditional and state-of-the-art methods in performance, especially in zero-shot and lowresource settings, but also enhances the explainability of authorship predictions by illuminating the role of linguistic features. the findings underscore the potential of llms to revolutionize authorship analysis, offering robust solutions for digital forensics, cybersecurity, and combating misinformation. this work paves the way for future research and applications in llm-based authorship prediction.