Connecting Concept Layers and Rationales to Enhance Language Model Interpretability (2025)

Abstract: With the introduction of large language models, NLP has undergone a paradigm shift where these models now serve as the backbone of most developed systems. However, while highly effective, they remain opaque and difficult to interpret, which limits their adoption in critical applications that require transparency and trust. Two major approaches aim to address this: rationale extraction, which highlights input spans that justify predictions, and concept bottleneck models, which make decisions through human-interpretable concepts. Yet each has limitations. Crucially, current models lack a unified framework that connects where a model looks (rationales) with why it makes a decision (concepts). We introduce CLARITY, a model that first selects key input spans, maps them to interpretable concepts, and then predicts using only those concepts. This design supports faithful, multi-level explanations and allows users to intervene at both the rationale and concept levels. CLARITY, achieves competitive accuracy while offering improved transparency and controllability.

## introduction language models like bert @xcite and roberta @xcite have transformed nlp, forming the basis of many systems and excelling in various tasks such as sentiment analysis and document classification. despite their impact, these models are black boxes, with complex, opaque outputs, posing challenges in sectors where transparency is crucial, like healthcare and law. to address interpretability, nlp has focused on two main approaches: rationale extraction and con-cept bottleneck models. rationale extraction identifies input text segments that justify model predictions @xcite , promoting interpretability by highlighting essential evidence while using sparsity and regularization for accuracy @xcite . these explanations, however, are often limited to token-level insights and lack broader semantic context. in contrast, concept bottleneck models (cbms) encourage the model to make predictions via interpretable intermediate representations, often aligned with human-defined concepts @xcite . cbms offer several advantages, including the ability to intervene on model reasoning and support post-hoc debugging. however, these models typically assume that the relevant concepts are already known or provided, and they operate on whole examples, without leveraging fine-grained input regions that support those concepts. as a result, they often lack fine-grained textual grounding, making it unclear where in the input a concept arises. despite their complementary strengths, rationale extraction and cbms have largely evolved in isolation. rationale-based approaches offer textual grounding but lack semantic abstraction, while cbms provide interpretable reasoning structures without linking them to specific input regions. this separation is particularly problematic for semantic understanding, where surface forms and deep meanings are intrinsically connected. understanding how linguistic expressions contribute to what semantic concepts is crucial for advancing interpretable semantic processing. crucially, current models lack a unified framework that connects where a model looks (rationales) with why it makes a decision (concepts). to bridge this gap, we propose clarity, a unified framework that tightly integrates fine-grained rationale extraction with concept-based reasoning. by explicitly mapping selected input spans to intermediate concepts before classification, clar-ity produces multi-level explanations that are both textually grounded and semantically meaningful. this approach directly addresses a key challenge in computational semantics: understanding how surface linguistic patterns realize abstract semantic categories. specifically, clarity decomposes prediction into a three-stage process: (1) it identifies sparse, contiguous rationales from the input that correspond to semantically coherent units; (2) it maps these rationales to a low-dimensional vector of interpretable concepts that capture semantic dimensions; and (3) it predicts the final output label using only these activated concepts. the contributions we have made can be described as follows: ## related works interpretability in nlp has advanced rapidly in recent years, moving from simple feature attributions to structured, multi-level interpretability frameworks. we review major work on rationale extraction, cbms, and hierarchical interpretability. rationale extraction rationale-based methods aim to identify input spans that are sufficient to justify predictions. it has advanced significantly since @xcite and the information bottleneck from @xcite . unirex @xcite offers a unified learning framework balancing faithfulness, plausibility, and performance, with a 32.9% improvement in normalized relative gain across five datasets. fid-ex @xcite addresses sequence-tosequence model issues by using sentence markers to encourage extractive explanations. refer @xcite created a rationale extraction framework with a differentiable extractor to enhance task and explanation fidelity through concurrent training. recent work increasingly blends causal reasoning with rationale extraction, addressing confounding factors in rationale models @xcite . however, these models operate purely at the token level, lacking abstraction or semantic generalization. concept bottleneck models cbms guide predictions through a bottleneck of humaninterpretable concepts, first proposed by @xcite for image classification and later adapted to nlp. text bottleneck models @xcite use cbms for text classification, providing global and local explanations via llms discovering concepts without human input. cb-llms @xcite introduced inherently interpretable neurons in llms for text tasks, aligning neuron activations with concept scores for classification and combining interpretable and unsupervised neurons for generation. ct-llms @xcite resolved concept completeness and classification leakage by generating concept labels unsupervised with small language models, removing the need for predefined concepts. while cbms provide global interpretability and allow concept-level interventions, they typically assume concept supervision and operate on entire inputs, making them less suitable for tasks requiring localized justifications. ## methodology we introduce clarity, a framework that combines rationale extraction with concept bottleneck mechanisms. this section formalizes the task and presents the model architecture, interpretability constraints, training objective and intervention procedure. ## experiments to evaluate the effectiveness of clarity, we conduct experiments on five diverse text classification tasks. our analysis focuses on both predictive performance and interpretability, examining how well the model maintains accuracy while generating faithful, semantically meaningful explanations. section 5 provides detailed ablations and rationale quality analyses. ## ablation analysis to better understand the internal behavior of clarity and validate the design choices in its architecture, we conduct a series of ablation studies. we begin with an analysis of training dynamics and follow with targeted evaluations of rationale behavior, concept interventions, and architectural variants. additional analyses are provided in the appendix. ## conclusions and future work we introduced clarity, a modular framework for interpretable text classification that decomposes prediction into rationale extraction, concept mapping, and label prediction. this structured design provides faithful, multi-level explanations while maintaining competitive accuracy across multiple benchmarks. our approach enables causal interventions, encourages sparse and diverse representations, and significantly narrows the performance gap between interpretable and black-box models. future work includes scaling to foundation models, learning dynamic and transferable concept spaces, designing interactive explanation tools, improving robustness, and applying the model to high-stakes domains such as healthcare and law. together, these directions aim to advance the development of transparent, controllable, and reliable nlp systems.