Enhancing the Comprehensibility of Text Explanations via Unsupervised Concept Discovery (2025)

Abstract: Concept-based explainable approaches have emerged as a promising method in explainable AI because they can interpret models in a way that aligns with human reasoning. However, their adaption in the text domain remains limited. Most existing methods rely on predefined concept annotations and cannot discover unseen concepts, while other methods that extract concepts without supervision often produce explanations that are not intuitively comprehensible to humans, potentially diminishing user trust. These methods fall short of discovering comprehensible concepts automatically. To address this issue, we propose ECO-Concept, an intrinsically interpretable framework to discover comprehensible concepts with no concept annotations. ECO-Concept first utilizes an object-centric architecture to extract semantic concepts automatically. Then the comprehensibility of the extracted concepts is evaluated by large language models. Finally, the evaluation result guides the subsequent model fine-tuning to obtain more understandable explanations using relatively comprehensible concepts. Experiments show that our method achieves superior performance across diverse tasks. Further concept evaluations validate that the concepts learned by ECO-Concept surpassed current counterparts in comprehensibility.

## introduction deep neural language models lack explainability. a recent way to tackle this issue is concept-based explanations @xcite , which map the inputs to a set of concepts and measure the importance of each concept to model predictions. by offering human-understandable attributes, concept-based explanations better resemble the way humans reason and explain. some existing concept-based methods apply post-hoc analysis to a trained model, providing concepts that either explain a model's prediction (kim * corresponding author a clean lager that will quench thirst on a hot summer day. the beer is crystal clear golden yellow with a descent head. easy dinking clean and as they say "refreshing"style. a clean lager that will quench thirst on a hot summer day. the beer is crystal clear golden yellow with a descent head. easy dinking clean and as they say "refreshing" style. et @xcite @xcite or elucidate its internal network structure @xcite @xcite . since the model was not exposed to these concepts during training, it may lead to unfaithful interpretations or meaningless concept activations @xcite . self-explaining concept-based methods offer a more reliable approach by providing built-in explanations along with their predictions. however, most of these methods require abundant annotations @xcite for predefined concepts, which shows their limitations in automatically discovering new concepts during training. though there exist self-explaining methods that extract concepts without concept annotations @xcite @xcite , the extracted concepts often lack human comprehensibility due to the ignorance of such constraints to guide concept extraction. consequently, the resulting concept explanations may highlight irrelevant or mislead-ing information, bringing more confusion (demonstrated by previous research @xcite ) and less trust in humans (revealed by our human forward simulatability experiments). the comparison is illustrated in fig. 1 . in this paper, we aim to build a model that can automatically discover comprehensible concepts with no need for concept supervision. inspired by the recent application of the slot attention mechanism @xcite in concept-based interpretability for vision tasks @xcite , we develop a concept extractor that leverages slot attention, enabling each slot to learn a distinct task-specific concept. to ensure that the learned concepts are human-understandable, we leverage llms as human proxies to evaluate concept comprehensibility during model training and use the results to refine the concept extractor. specifically, we propose a metric for concept comprehensibility that measures the capability of a concept's activation map to be summarized and reconstructed using natural language. building on prior work that employs large language models to generate neuron explanations and simulate activations @xcite , our approach introduces a novel feedback loop. this loop utilizes comprehensibility scores to refine the concept extractor, making the extracted concepts more intuitive and human-understandable. experiments on seven tasks and human studies both show the superiority of our method compared with existing concept-based explanation methods. our contributions are summarized as follows: ## related work concept-based post-hoc methods explain an existing model without modifying its internal architecture. some studies predefine task-related con-cepts and assess them by quantifying contributions with linear probes @xcite or analyzing causal effects through proxy methods @xcite . methods without concept supervision extract concepts from intermediate layer representations using k-means @xcite , non-negative matrix factorization @xcite @xcite or concept completeness maximization @xcite . however, these post-hoc methods cannot guarantee that models truly comprehend or employ the adopted concepts, as the models were not exposed to these concepts during training @xcite . additionally, some unsupervised post-hoc methods may extract concepts lacking semantic meaning. concept-based self-explaining methods aim to provide a built-in human-interpretable explanation along with the prediction. some methods adopt a supervised paradigm with experts manually crafting a set of concepts. a representative framework is concept bottleneck model (cbm) @xcite , where an intermediate concept bottleneck layer is introduced to break the standard end-to-end training paradigm. recently, some studies have taken advantage of the capabilities of llms to generate concepts for each class as a replacement for manual annotation @xcite . this insight has also been applied in the text domain @xcite . these methods rely on concepts that are predefined based on priors and cannot discover or adapt to new concepts during training. an incomplete or biased predefined concept set could severely compromise both the model's interpretability and performance. text bottleneck model (tbm) @xcite utilized llms to automatically discover and measure concepts. however, in tbm, the mapping from samples to concepts is directly determined by llms, which remain largely opaque and function as a black box. unsupervised self-explaining concept models autonomously extract concepts during model training without the need for concept annotations. senn @xcite uses self-supervision by reconstruction loss for concept discovery. botcl @xcite and ccts @xcite utilize a slot attentionbased mechanism to extract task-dependent concept slots. while in the text domain, selfex-plain @xcite identifies concepts as the non-terminal leaves of the semantic tree parsing the text. some methods classify samples based on their distance to learnable prototypes @xcite @xcite and introduce distance-based losses to guide prototype learning. without concept annotations, the concepts extracted by these methods pose challenges to human understanding. although existing methods evaluate human interpretability through manual assessments during the experimental phase, no approach has yet incorporated human feedback into the training to optimize concept comprehensibility. ## experiments in this section, we conduct extensive experiments to verify the task performance and interpretability of eco-concept. due to space limitations, parameter sensitivity analysis and other experiments are included in appendices e and f. ## conclusion to automatically extract human-understandable concept explanations with no predefined concept annotations, we proposed eco-concept, an intrinsically interpretable framework. eco-concept employs an object-centric architecture based on the slot attention mechanism to extract task-specific semantic concepts. to ensure the extracted concepts are comprehensible to humans, eco-concept incorporates llms as human proxies to evaluate concept comprehensibility during model training, using their feedback to iteratively refine the concept extractor. eco-concept achieves classification performance on par with black-box models while offering enhanced interpretability. it outperforms existing concept-based approaches in both quantitative metrics and user studies, demonstrating its effectiveness in generating human-aligned, interpretable explanations.