Who’s the Author? How Explanations Impact User Reliance in AI -Assisted Authorship Attribution (2025)

Abstract: Despite growing interest in explainable NLP, it remains unclear how explanation strategies shape user behavior in tasks like authorship identification, where relevant textual features may be difficult for lay users to pinpoint. To support their analysis of text style, we consider two explanation types: example-based style rewrites and feature-based rationales, generated using a LLM-based pipeline. We measured how explanations impact user behavior in a controlled study (n=95) where participants completed authorship identification tasks with our types of assistance. While no explanation type improved overall task accuracy, fine-grained reliance patterns (CITATION) revealed that rewrites supported appropriate reliance, whereas presenting both explanation types increased AI overreliance, minimizing participant self-reliance. We find that participants exhibiting better reliance behaviors had focused explanation needs, contrasting with the diffused preferences of those who overrelied on AI, or incorrectly self-relied. These findings highlight the need for adaptive explanation systems that tailor support based on specific user reliance behaviors.

## background we motivate our study by discussing prior work in explanations for ai-assisted decision-making and authorship analysis. we focus on how explanations can shape human reliance, and then surface how authorship authorship analysis is an ideal candidate for studying how these explanations impact human decision-making. ## study design in designing our study, we ask the following research questions: rq1. how do different explanation types influence reliance patterns in authorship attribution? rq2. what explanation properties do users perceive as most useful for supporting both correct self-and ai-reliance? we explore these questions by designing a study that explicitly prompts users to try the task by hand, before being exposed to some ai support consisting of explanation-by-example and explanationby-feature, when they can choose to revise their decision. task overview figure 1a illustrates the core authorship identification task. participants are presented with a short mystery post and two writing samples from two distinct authors (candidate a, candidate b, where one of these is the ground truth author). the task is formulated as a binary classification: which of these authors authored m ? the domain is constrained on topic so that participants need to rely on stylistic cues (e.g. lexical choice, syntax, punctuation usage, etc.). explanation interventions figure 1b illustrates the explanations we propose: feature-based rationale: a rationale describing stylistic similarities or differences between the mystery post and each candidate's writing. example-based rewrite: a rewrite of the mystery post in the style of each candidate, offering participants an implicit comparison through example. dual explanation: both rationale and counterfactual rewrites. rationales aim to support user reasoning through explicit comparison, while rewrites aim to provide stylistic transformations as examples, aiming to support user reasoning through implicit comparison through which rewrite shifts the style less.we generate these outputs with llms as described in sections 4.3 and 4.2. experimental setup we adopt a 2×4 mixed design. the within-subject factor is ai correctness (correct vs. incorrect prediction), evenly distributed across 10 trials (5 correct, 5 incorrect). the between-subject factor is explanation type, randomly assigned to participants, with four conditions: (1) prediction-only: the ai provides a single prediction for the text's author (author a or author b). (2) dual explanation: the ai provides a prediction accompanied by explanation-by-feature (stylistic rationales) and explanation-by-example (rewrites illustrating how the mystery post might look if each candidate author had written it). (3) explanation-by-feature: the ai provides a prediction accompanied by the selected explanationby-feature. (4) explanation-by-example: the ai provides a prediction accompanied by the selected explanation-by-example. in each trial, participants make an authorship decision in two phases: human-only and ai-assisted. in phase 1 (human-only), they choose an author and describe which stylistic cues influenced their decision, with answer choices drawing on interpretable stylistic feature sets @xcite . in phase 2 (ai-assisted), participants review the model's binary prediction as well as the provided explanation in their assigned explanation condition, and may revise their choice. after completing all 10 trials, participants complete a paraphrased version of nasa-tlx @xcite to assess mental workload and perceived task success. ## task materials we construct task materials used in our user study as described in the following sections: (1) process and select data, (2) generate model predictions, and (3) derive explanations (feature-based rationales and counterfactual rewrites). ## results and discussion our analysis investigates how different explanation types affect user reliance in ai-assisted authorship attribution (rq1), and what users say or show they need to perform the task better (rq2). we organize our findings into four parts: changes in confidence and accuracy, reliance behavior patterns, perceived workloads of the task, and end with a discussion on what participants rely on to do the task manually, and what unmet needs they perceive to have with respect to the presented explanations. ## conclusion we investigate how explanation-by-feature (rationale), explanation-by-example (rewrite), and their combination affect human decision-making in an authorship attribution context. our results show that single-format explanations hit the right balance in introducing positive friction in their decisionmaking process, while combining the two in the dual explanation may have led to unintentionally reducing perceived effort by leading them to short circuit. these findings highlight a challenge in explainable ai: more explanations, even if they do not increase mental burden, can inflate confidence while degrading decision accuracy. our explanation preference analyses suggests a path forwardpeople who exhibit appropriate reliance behaviors tend to prefer more targeted, low-friction explanations. additionally, the single-format explanations we test promote better self-reliance. future work should explore more adaptive explanations that promote correct self-reliance and help users recognize when ai advice warrants revision.