Residualized Similarity for Faithfully Explainable Authorship Verification (2025)

Abstract: Responsible use of Authorship Verification (AV) systems not only requires high accuracy but also interpretable solutions. More importantly, for systems to be used to make decisions with real-world consequences requires the model’s prediction to be explainable using interpretable features that can be traced to the original texts. Neural methods achieve high accuracies, but their representations lack direct interpretability. Furthermore, LLM predictions cannot be explained faithfully – if there is an explanation given for a prediction, it doesn’t represent the reasoning process behind the model’s prediction. In this paper, we introduce Residualized Similarity (RS), a novel method that supplements systems using interpretable features with a neural network to improve their performance while maintaining interpretability. Authorship verification is fundamentally a similarity task, where the goal is to measure how alike two documents are. The key idea is to use the neural network to predict a similarity residual, i.e. the error in the similarity predicted by the interpretable system. Our evaluation across four datasets shows that not only can we match the performance of state-of-the-art authorship verification models, but we can show how and to what degree the final prediction is faithful and interpretable.

## introduction identifying the author of a text or a collection of texts is a task with many use cases. in forensic investigations, stylometry techniques and authorship identification help link anonymous social media accounts @xcite , narrow down suspects @xcite , and provide supporting evidence in court @xcite . plagiarism and academic dishonesty are cases of intentionally false authorship claims @xcite . figure 1 : demonstration of the task of authorship verification. a forensic linguist is trying to determine if two texts share the same author. they may use either an interpretable system comprising linguistic features faithful to the source text or a neural model, which has good performance but lacks interpretability. our system combines the relative strengths of both by using a neural model to correct the error in the interpretable system's prediction. forensic and related applications of ai, and of authorship analysis models in particular, require explainable ai @xcite . all stakeholders need to be able to verify the authorship claims made by any automated system. furthermore, the explanation must be faithful: it "should accurately represent the reasoning process behind the model's prediction" @xcite . but this is not enough: the reasoning process itself must be based on interpretable features derived from the analyzed texts, i.e., the features must be meaningful to humans. authorship claims from a system using uninterpretable text embeddings cannot be trusted, even if they are used in an explainable reasoning process. but interpretable features are still not enough. even if the input features are interpretable, they need to be traceable. this means that each feature can be traced back to the text(s) being analyzed, and the value of the feature is based on reproducible evidence from the text. a traceable feature cannot be a feature on whose value reasonable observers may disagree after examining the text, such as "formality". as with many nlp tasks, authorship systems that use representations derived from neural language models often achieve better verification performance than interpretable representations do @xcite . however, neural representations are limited in many critical domains because they are not directly interpretable. when attempts are made to interpret predictions, such as by @xcite , the explanations for a model's predictions are not guaranteed to be faithful to how the prediction was made. in this paper, we ask how one can combine the relative strengths of the two methods: the interpretability and traceability of linguistic representations and the high performance of neural models. as the main contribution of the paper, we introduce residualized similarity (rs), which uses the idea of estimating the residual of a predictor i.e., the error in a model's prediction. we approach authorship verification as a similarity task, as is standard in the field. for each pair of documents we obtain some similarity score from a system; if the score is above a certain threshold, we conclude the documents share the same author, and if the score is below the threshold, we say the documents are from different authors. suppose we start with an explainable system using interpretable and traceable features as the initial similarity estimator. we can then train a neural model as a residual predictor, which predicts the error or correction to the interpretable system's similarity score. the final prediction is a simple sum of the interpretable model's similarity score and the predicted residual, i.e., a similarity adjustment made by the neural model. this combined system can achieve the trade-off we desire: (i) when the interpretable model is likely to be correct, the residual should be low, providing interpretability and faithfulness while remaining accurate, and (ii) when the interpretable model is likely to be incorrect, the residual should provide the necessary correction, improving accuracy but reducing interpretability to a degree proportional to the error. this approach is inspired by the residualized control approach @xcite , which trains a residual model for a regression problem, combining numerous linguistic features with a few interpretable health-relevant attributes to predict community health indicators. we describe our approach in detail in section 3. we use gram2vec @xcite as our inter-pretable feature system, which records normalized frequencies of morphological and syntactic features for input texts. we evaluate our rs approach by combining gram2vec with a state-of-the-art av neural model, luar @xcite , finding that rs can match the performance of using luar alone, while introducing faithful explanations using interpretable and traceable features. we perform a case study on how our system retains interpretability, measured by an interpretability confidence (ic) metric, which indicates the extent to which the interpretable system is used for a given input. details of this are in section 7. interpretable methods previous stylometric approaches @xcite often make use of readily interpretable features to train classifiers. some examples include lexical features such as vocabulary, lexical patterns @xcite , syntactic rules @xcite , and others. gram2vec, the interpretable component of residualized similarity falls into this category. neural models authorship verification has benefited from models built upon rnns @xcite , cnns @xcite , bert-like architectures @xcite , and longformers @xcite . more recently, sentence-transformer-based models @xcite have obtained state-of-the-art performance for av tasks. as we are interested in improving the performance of interpretable authorship verification, we focus on these sota av models, particularly luar @xcite . our work uses residual similarity analysis to combine interpretability and neural models' high performance for authorship verification. similar residual approaches have been used previously for improving performance in health outcome prediction, by combining lexical and health-relevant attributes @xcite , and in a recent work that combines statistical and neural methods for machine translation @xcite . other works have focused on generating explanations, often layering other mechanisms on top of interpretable input features @xcite @xcite or doing a post-hoc evaluation on a latent, non-interpretable space @xcite . some recent work also explores prompting large language models to derive interpretable stylometric features for authorship analysis @xcite . however, these features are not traceable in a text as the approaches rely on llms to generate the features, and the generations do not represent the reasoning process behind attributing a set of features to a text. ## experimental setup we perform experiments across a variety of neural models, in order to evaluate robustness and ability to generalize of our technique. we specifically leverage sentence-transformer-based models @xcite , which were developed as a distinct technology in parallel to generative llms. embedding models specialize in creating fixed-length semantic representations optimized for similarity computations-the way authorship verification is evaluated. ## results metrics we evaluate rs against both gram2vec and neural models on the receiver-operating curve area-under-curve (auc), which represents a model's performance across all thresholds. it calculates the true positive rate (tpr) and false positive rate (fpr) at every threshold, and graphs tpr over fpr. we use auc as it is threshold-independent and the data we use is balanced, providing a direct comparison of the various systems. ## robustness of the technique to show that our technique is not dependent on any single interpretable base system, we try another feature extractor in place of gram2vec. we turn our attention towards efficient linguistic feature extraction for natural language datasets, or elfen @xcite . while some features overlap, elfen captures several unique categories of features from gram2vec, and as such, provides a good alternative interpretable system to use in our rs framework. elfen feature areas include dependency, emotion, entities, information, lexical richness, morphological, part-of-speech, readability, semantic, and surface. we provide brief explanations of the different features in appendix c. we perform experiments using elfen in place of gram2vec as the interpretable system, as well as the concatenation of the two interpretable feature sets as the interpretable system. we perform these experiments using luar as the neural model as it is generally the strongest performing neural model of our primary experiments. we find that on its own, elfen is a little less performant or on par with gram2vec, while concatenating the two feature sets tends to improve a bit on either. however, notably, the rs model performs about the same regardless of the interpretable system that it is trained on. we consider this to be a success of the residualized similarity technique as it is able to boost the performance of two separate interpretable systems. 3 ## case study of two pairs of documents we present two cases to illustrate how rs can give a user insight while performing a specific authorship verification task. we present two pairs of documents, one of which is indeed from the same author, and one of which is not. given the range of cosine similarity from -1 to 1, we set a threshold of 0.5, representing a moderately strong alignment in vector space and suggesting that documents need to be more similar than dissimilar to be considered by the same author. we show how our approach can tell the user which gram2vec features were used in the determination, and to what extent they determined the confidence of the prediction. since gram2vec contains over 600 features, we define a criterion to select features to present to the user, depending on whether a pair of documents are predicted to be by the same or different authors. when a pair of documents is predicted to be written by the same author, we want to maximize the absolute values of the feature values (features example pair 1: different author document 1: whirling like a scythe, the saber sliced her upper torso, putting an end to the vengeful sith . dropping to her knees again, jameh crawled to her fallen master , cradling him in her arms. a new darkness grew in her heart now, one like a cold, lonely mist. pilae, obi-wan, and anakin stood nearby, dismayed at the sight that met their eyes: a dismembered former senator , a shorn and wounded padawan , and a jedi master on the verge of death. " master, please, you can"t leave me. i need you; i"m not ready!" document 2: he scanned the field beyond and was dumbfounded when he didn"t see any rats. he pelted back through the entrance and into the clearing. the clan had been alerted by redfur"s yowl of surprise, so they had stopped chatting and lowered their bodies into a crouch, getting ready for the rats. but when they saw the four rats clinging to redfur"s fur, they hissed in astonishment at the size of them. that distinguish these documents from the large set of background documents) while making sure the values are similar for both documents. when a pair of documents is predicted to be written by different authors, we simply find the largest magnitudes of differences in the feature values. thus, for identifying features for same-author pairs, we use the following metrics for ordering features, where val_1 represents the feature's score for document 1, and val_2 represents the feature's score for document 2.: |val_1| + |val_2| -|val_1 -val_2|. for ordering features using different author pairs, we use |val_1 -val_2|. we then choose the top n features; in the examples below, we use @xmath0. ## conclusion we introduce residualized similarity, a method of improving the performance of an interpretable feature set by training a language model to predict the residual, or difference, between the similarity output from an interpretable system and the ground truth. we apply this technique to the task of authorship verification, where interpretability is of the utmost importance. using residualized similarity, we are able to achieve state-of-the-art performance on the task of authorship verification while maintaining a quantifiable degree of interpretability. to measure interpretability, we introduce the interpretability confidence, a measure of how interpretable a prediction from our system is. we then do a case study to observe how using rs, we are able to correct a prediction that was initially incorrect from an interpretable system. in both the case where the prediction was corrected and the case where the prediction from the interpretable system and rs agreed, we show that there is meaningful interpretability in the features, as well as the ability to trace such features back to the text from which they were extracted. we believe this approach to be a promising direction for developing more interpretable and effective nlp systems, bridging the gap between neural methods and interpretable linguistic features while allowing for faithfully explainable systems.