Who Wrote it and Why? Prompting Large-Language Models for Authorship Verification (2023)

Abstract: Authorship verification (AV) is a fundamental task in natural language processing (NLP) and computational linguistics, with applications in forensic analysis, plagiarism detection, and identification of deceptive content. Existing AV techniques, including traditional stylometric and deep learning approaches, face limitations in terms of data requirements and lack of explainability. To address these limitations, this paper proposes PromptAV, a novel technique that leverages Large-Language Models (LLMs) for AV by providing step-by-step stylometric explanation prompts. PromptAV outperforms state-of-the-art baselines, operates effectively with limited training data, and enhances interpretability through intuitive explanations, showcasing its potential as an effective and interpretable solution for the AV task.

## introduction motivation. authorship verification (av) is a fundamental task in the field of natural language processing (nlp). it aims to determine the authorship of a given text by analyzing the stylistic characteristics and patterns exhibited in the writing @xcite . the importance of av lies in its wide range of applications, including forensic analysis @xcite , plagiarism detection @xcite , and the detection of deceptive or fraudulent content @xcite . the rise of large-language models (llms) @xcite , has facilitated the generation of human-like text where one would now need to factor in the possibility of machinegenerated text @xcite . this development give rise to a new layer of complexity in distinguishing between machine-generated and human-written text, subsequently amplifying the significance of the av problem. over the years, various techniques have been proposed to address the av task. traditional approaches relied on stylometric features, such as n-gram frequencies, vocabulary richness, and syntactic structures, to capture the unique writing style of individual authors @xcite . machine learning techniques, such as support vector machines (svms) and random forests, have been employed to model the relationship between these stylometric features and authorship @xcite . more recently, deep learning approaches have shown promising av results by learning intricate patterns directly from raw text data @xcite @xcite . despite these advancements, existing av techniques have certain limitations. firstly, most existing av methods require a large amount of labeled training data, which can be costly to acquire, especially for scenarios with limited available data. secondly, there is a lack of explainability in the predictions made by these techniques. in many practical applications, it is crucial to understand why a particular decision was reached, which is particularly relevant in legal contexts or situations where interpretability is paramount. ## related work av is a well-established research topic. classical approaches to av have predominantly centered on stylometric features encompassing n-gram frequencies, lexical diversity, and syntactic structures to discern the unique writing styles intrinsic to individual authors @xcite . classical machine learning algorithms were also used to model the associations between these stylometric attributes and authorship @xcite . recently, deep learning models such as rnns and cnns being employed to extract more complex patterns from textual data @xcite @xcite . these deep learning architectures have exhibited promising advancements in av tasks. furthermore, the advent of the transformer architecture @xcite has prompted the development of transformer-based av models @xcite @xcite . ## methodology in an recent work, @xcite have proposed chain of thought (cot) prompting, which employs a series of intermediate reasoning steps to significantly improve the ability of llms in performing complex reasoning. @xcite further extended the concept to propose ps+ prompting, a zero-shot prompting methodology that instructs llms to formulate a strategic plan prior to problem-solving. inspired by these works, we introduce promp-tav, a prompting strategy that incorporates key linguistic features-identified in @xcite as the crucial variables for llms to evaluate. these linguistic features serve as rich, often subtle, markers of an author's distinct writing style and are hence of paramount importance for the av task. for instance, the consistent use or avoidance of certain punctuation marks, or the preference for particular special characters, can be idiosyncratic to an author. similarly, elements like tone and expression, though harder to quantify, can offer significant insights into an author's unique 'voice'. the integration of these linguistic attributes within the promptav framework does not simply add layers of complexity to the model, it fundamentally enhances the model's ability to capture the author-specific nuances within textual data. this capability, in turn, improves the model's accuracy and reliability when attributing authorship to unidentified texts, making promptav a powerful tool in the field of av. figure 1 provides an overview that compares the prompts used in cot, ps+ prompting, and promptav. we also note that conventional prompting techniques that instruct the llm to respond strictly with binary "yes" or "no" answers frequently result in the llm responding 'no" for the majority of instances within an av task. to mitigate this problem, promptav instructs the llm to generate a confidence score ranging from 0 to 1, rather than a binary response. to calibrate these generated confidence scores, we augment the prompt with the additional directive "your answer should reflect a moderate level of strictness in scoring". ## conclusion in this paper, we proposed promptav, a promptbased learning technique that harnesses the linguistics capabilities of llms for av. our experimental results demonstrated promptav's superior performance over the state-of-the-art baselines in zero-shot and few-shot settings. case studies were also conducted to illustrate promptav's effectiveness and explainability for the av task. for future works, we will conduct experiments on more av datasets and explore other linguistic features.