CAVE : Controllable Authorship Verification Explanations (2025)

Abstract: Authorship Verification (AV) (do two documents have the same author?) is essential in many real-life applications. AV is often used in privacy-sensitive domains that require an offline proprietary model that is deployed on premises, making publicly served online models (APIs) a suboptimal choice. Current offline AV models however have lower downstream utility due to limited accuracy (eg: traditional stylometry AV systems) and lack of accessible post-hoc explanations. In this work, we address the above challenges by developing a trained, offline model CAVE (Controllable Authorship Verification Explanations). CAVE generates free-text AV explanations that are controlled to be (1) accessible (uniform structure that can be decomposed into sub-explanations grounded to relevant linguistic features), and (2) easily verified for explanation-label consistency. We generate silver-standard training data grounded to the desirable linguistic features by a prompt-based method Prompt-CAVE. We then filter the data based on rationale-label consistency using a novel metric Cons-R-L. Finally, we fine-tune a small, offline model (Llama-3-8B) with this data to create our model CAVE. Results on three difficult AV datasets show that CAVE generates high quality explanations (as measured by automatic and human evaluation) as well as competitive task accuracy. We have submitted our code and datasets as supplementary material.

## introduction authorship verification (av) @xcite is the nlp task of determining if two input documents were written by the same author. av is used for tasks such as plagiarism detection, forensic analysis (often in support of law enforcement), analysis of the spread of misinformation. given the sensitive nature of this task, it is imperative to develop av methods that are explainable, accessible and secure. explainability ensures that the system's @xcite the likelihood that they were written by the same author is 0.7. here's my reasoning: . . . . overall they may have been written by different authors. decisions can be understood and trusted by the user. accessibility ensures easier downstream usability; for example, a well-structured free-text explanation with clearly defined topic points is more easily understood than unstructured explanations such as long lists of low-level features. finally, security is necessary to safely use the av system for proprietary data; a locally-hosted model on the user's server is more secure than an api call to an online model, which could potentially store the user's sensitive data. av is also an extremely difficult task for humans, in contrast to other nlp tasks such as machine translation; this further necessitates the need for automatic, high-quality av systems. early works performed av by comparing handcrafted features such as n-grams (van dam, 2013), pos tags @xcite , liwc fea-{ punctuation style: both texts use a variety of punctuation, including commas, periods, and quotation marks, but text 1 uses more diverse punctuation such as parentheses and hyphens. maybe special characters style, capitalization style: text 1 uses continuous capitalization for emphasis (e.g., 'wonderful', 'thrilled'), which is not observed in text 2. no , acronyms and abbreviations: neither text makes significant use of acronyms or unusual abbreviations. yes , writing style: text 1 has a more personal, reflective style, sharing personal opinions and feelings about the movie. text 2 provides a more detached, narrative-style review without personal input. no , expressions and idioms: both texts avoid colloquial expressions and idioms, opting for a more formal format. yes , tone and mood: text 1 has a more varied tone, from enthusiasm to disappointment, while text 2 maintains a consistent, somewhat formal and analytical tone. no , sentence structure: text 1 features a mix of short and long sentences with more complex structures, while text 2 tends to use more uniformly structured, intermediate-length sentences. maybe , any other relevant aspect: the approach to movie critique is different; text 1 is more about the impact on the viewer, while text 2 focuses on plot summary and cinematic elements. no , final score: 0.375 , output: no } in this work, we propose our model cave (controllable authorship verification explanations), an in-house model that generates 2 we use terms explanation and rationale interchangeably. since there is no human-written av explanation data that can be used to train our in-house model, we adopt the model distillation strategy: @xcite @xcite we engineer prompt-cave to generate silver-standard train data in the format described above by prompting a large, oracle model (gpt-4-turbo ope-nai (2024a)), and use that to supervised-finetune a smaller language model (llama-3-8b meta-ai (2024)). we also introduce our automatic metric cons-r-l that measures the consistency between the rationale and the model's final label; cons-r-l is used to filter the silver train data before finetuning @xcite , and also assess cave during inference. our experiments with three difficult av datasets show that cave leads to competitive task performances as gpt4-turbo [>>> 8b] (1) cave format (2) cons-r-l filter ⚙ cave prompt-cave + doc-1 + doc-2 silver-standard train-data gold av explanations unavailable ⇒ generate high-quality silver data -accessible rationale zero-shot prompting llama-3 [8b] sft -consistent rationale & label -offline model -high accuracy -post-hoc explainability well as high quality of rationales. ## our method to enable and improve downstream utility of authorship verification systems, we present cave, our trained, offline model that generates accessible free-text rationales (well-structured, and uniform structure across all datapoints). we further propose a novel metric cons-r-l that leverages the structure of cave's rationales to measure consistency between a rationale and its task (av) label. figure 2 demonstrated the visual pipeline to train cave. silver-standard data. there exists no goldstandard, human-written datasets for av explanations; hence, the first step to develop an av model that can generate explanations is to create silverstandard data for it. we propose our zero-shot prompt, prompt-cave, that can be used with llms such as gpt-4-turbo to generate silver training data. we adapt and extend promptav @xcite to engineer prompt-cave. we then use this silver data to train an in-house language model cave to generate av explanations. ## conclusion existing methods for authorship verification face a lack of accessible explainability. in this work, we train an in-house language model cave to generates structured and consistent explanations for av. we present strong experimental results on three difficult av datasets; we also present a human pilot study on the quality of generated rationales. finally, we provide detailed discussions on the robustness of our consistency metric, and explore errors made by cave due to factors such as hallucination, dataset biases, etc. to guide research in this area. as future work, we would like to explore reward metrics and reward-based learning for av explanations.