Evaluating Readability and Faithfulness of Concept-based Explanations (2024)

Abstract: With the growing popularity of general-purpose Large Language Models (LLMs), comes a need for more global explanations of model behaviors. Concept-based explanations arise as a promising avenue for explaining high-level patterns learned by LLMs. Yet their evaluation poses unique challenges, especially due to their non-local nature and high dimensional representation in a model’s hidden space. Current methods approach concepts from different perspectives, lacking a unified formalization. This makes evaluating the core measures of concepts, namely faithfulness or readability, challenging. To bridge the gap, we introduce a formal definition of concepts generalizing to diverse concept-based explanations’ settings. Based on this, we quantify the faithfulness of a concept explanation via perturbation. We ensure adequate perturbation in the high-dimensional space for different concepts via an optimization problem. Readability is approximated via an automatic and deterministic measure, quantifying the coherence of patterns that maximally activate a concept while aligning with human understanding. Finally, based on measurement theory, we apply a meta-evaluation method for evaluating these measures, generalizable to other types of explanations or tasks as well. Extensive experimental analysis has been conducted to inform the selection of explanation evaluation measures.

## introduction explainable artificial intelligence (xai) holds significant value in pre-trained language models' mechanism understanding @xcite , visualization @xcite , performance enhancement @xcite @xcite , and security @xcite . previous xai algorithms have been applied to nlp tasks @xcite , vision tasks @xcite and recommendation @xcite . these include natural language explanation @xcite , attention explanation @xcite , and especially attribution methods (lundberg and @xcite @xcite . the attribution methods identify "where" the model looks rather than "what" it comprehends @xcite , typically offering local explanations for a limited number of input samples, restricting their utility in practical settings @xcite . concept-based explanations @xcite @xcite can mitigate the limitations of attribution methods by recognizing highlevel @xcite patterns (see fig. 1 ), which provide concise, human-understandable explanations of models' internal state. despite these merits, the development of concept-based explanations may be hindered due to a lack of standardized and rigorous evaluation methodology. unlike a single importance score assigned on each scalar input by attribution methods, diverse explanation methods approach highdimensional concepts from different aspects. this includes a single classification plane @xcite , an overcomplete set of basis @xcite , or a module designed beforehand @xcite , lacking a unified landscape (c1). moreover, its non-local nature across samples @xcite , combined with the high cost of human evaluation when the number of concepts is large, makes evaluating a concept's readability challenging (c2). for available evaluation measures @xcite , it is difficult to test their reliability and validity (c3). in this paper, we address the challenges above and make the following contributions: first, we provide a unified definition of diverse concept-based explanation methods and quantify faithfulness under this formalization (c1). by summarizing common patterns of conceptbased explanation, we provide a formal definition of a concept, which can generalize to both supervised and unsupervised, post-hoc and interpretableby-design methods, language and vision domains. based on this, we quantify the faithfulness of a concept explanation via perturbation. we ensure adequate perturbation in the high-dimensional space for different concepts via an optimization problem. second, we approximate readability via coherence of patterns that maximally activates a concept (c2). we utilize the formulation defined above to recognize patterns across samples that maximally activate a concept, from both the input and the output side. then, we estimate how coherent they are as one concept via semantic similarity. experimental results have shown this automatic measure correlates highly with human evaluation. third, we apply the classic measurement theory to perform a meta-evaluation on the faithfulness and readability measures (c3). measurement theory @xcite has been long utilized to verify whether a measurement is reliable and valid. approaching via reliability and validity, this meta-evaluation method is useful for evaluating the measures for concepts and can be generalized to analyze the effectiveness of other measures, for example, measures for other types of explanations and other natural language tasks. experimental results have filtered out 4 measures with low reliability, i.e. llm-score, grad-loss, in-uci, in-umass, and verified the remaining faithfulness and readability measures' validity. ## concept formalization in this paper, we primarily focus on explaining llms as black-box models. meanwhile, our method can be generalized to many other deep classification models, including image models (see appx.c). as illustrated in fig. 1 , we consider the black-box model to take an input x from a dataset d and output y, a k-class classification result. in text generation, k is the vocabulary size. for the l-th layer to be interpreted, given a sequence of input tokens x 1 , ...., x t , their corresponding hidden representations are h 1 l , ..., h t l . the output classification logits are g(h). within the context of deep neural networks (dnns), we summarize common patterns of concepts and establish a unified framework. specifically, each concept is represented as a virtual neuron defined by an activation function that maps a hidden representation h into a real value a : r @xmath0, where a positive output signifies activation. for each concept, a semantic expression may be given by humans or llms, depending on the concept explanation methods @xcite . some methods take concepts and semantic expressions predefined by humans as inputs (e.g., @xcite ), while others require additional steps to produce a semantic expression based on highly activated tokens and samples of the extracted concepts @xcite . specifically, given high-activation samples of the concept and the highly activated tokens in these samples (e.g., "internet, computer, networks, . . . "), an llm or a human labeler provides a semantic expression that summarizes their common patterns (e.g., "terminologies related to computer networks") @xcite . our formalization can integrate diverse concept explanation methods, as shown in tab. 1. this includes both supervised methods that require prior information about concepts (e.g., input samples that contain and do not contain the concepts) @xcite and unsupervised methods that do not rely on such prior information @xcite . our method also works for both post-hoc explanation methods that interpret a model after it is trained @xcite and interpretableby-design approaches that integrate interpretability mechanisms directly into the model's architecture before training @xcite . additionally, it applies to image backbone models as well. ## concept evaluation measures we have conducted a literature survey on evaluation measures for concept-based explanations (fig. 5 in appx. a), and decided to focus on two aspects that are of common interest: testing how well they reflect the underlying mechanisms of the machine (faithfulness) and assessing the extent to which explanations can be understood by humans (readability). ## meta evaluation how can we discern the effectiveness among possible measures available for evaluating conceptbased explanations? borrowing metrics from measurement theory @xcite and psychometrics @xcite , our meta-evaluation focus centers on reliability and validity, guided by the methodological framework outlined in @xcite . our metaevaluation methods can generalized to measures of a broader scope, including other xai methods and other natural language tasks like generation. ## conclusion this paper introduced two automatic evaluation measures, readability and faithfulness, for conceptbased explanations. we first formalize a general definition of concepts and quantify faithfulness un-der this formalization. then, we approximate readability via the coherence of patterns that maximally activate a concept. another key contribution of this paper is that we describe a meta-evaluation method for evaluating the reliability and validity of these evaluation measures across diverse settings based on measurement theory. through extensive experimental analysis, we inform the selection of explanation evaluation measures, hoping to advance the field of concept-based explanation.