Paper: Latent Concept-based Explanation of NLP Models (ICLR-2024)

Reviewer comment (Novelty):
The components are loosely connected, which isn't necessarily a bad thing by itself. However, each component is either simplistic or an existing approach.

Author response:
Our work combines the encoded concepts with the attribution analysis to explain the model’s decision in light of the utilized latent space. Our novel formulation allows us to combine what the model has learned or knows with what it uses for making predictions. We argue that it is not straightforward to simply combine other works since the efficacy of every module affects the final output of the model. This is evident from various module-level evaluations presented in the paper.

Reviewer comment (Evaluation):
The evaluation is done with automatically generated labels, and in this particular case, they can be deceptive because of the last layer assumption used to generate them. Also, if they are the targets, one can just adopt the method used to generate labels to replace the proposed method.

Author response:
The evaluation of explanations is a difficult problem. We selected the POS tagging task to design an evaluation that can measure the performance of each module involved in our methodology.

Reviewer comment (Experiments):
The evaluation is limited to simplistic tasks only.

Author response:
While our approach can be applied to explain the prediction of any task, the evaluation of explanation is difficult. We have extended the explanation generation to the MNLI task as well. Moreover, we plan to perform human evaluation to quantify the usefulness of the explanation.

Reviewer comment (Novelty):
How is this different from topic modeling, or many other word sense disambiguation approaches, especially as other components are mostly direct application of existing approaches?

Author response:
It does seem related where a latent concept can be seen as clusters of topics. Similarly, a word with a different sense will have a different latent concept. In our case, the latent concepts are derived directly from the embedding space of the underlying model, and reflect the model's knowledge about the task.

Reviewer comment (Evaluation):
What are the examples of the ambiguity handled with the concepts in the evaluation? Are they apparent with the input text?

Author response:
We assume that the reviewer is referring to ambiguity related to different semantics of a word. Since we consider contextualized representations of a word, the semantic differences will cause different instances of a word to appear in the same or different latent concepts.

Reviewer comment (Experiments):
I think the experimentation could be extended. POS and sentiment are very small and relatively simple. The types of models that you're fine-tuning are also not very broad (no decoder only models like GPT here). What happens when you evaluate on an NLI task? Could you extend this to work on a task like question answering and evaluate that?

Author response:
Our approach is model agnostic and it can easily be applied to decoder only models. We performed experiments using the MNLI task and found latent concepts at the last layer consisting of sentence pairs. The sentences are grouped together in a latent concept based on how the hypothesis relates with the premise e.g. summary, rephrasing the argument, logical relationship, etc.

Reviewer comment (Experiments):
It's really hard for me to figure out the appropriate baselines here, which I think is a potential weakness. It's hard for me to contextualize the results you have.

Author response:
We extended the experiments to the NLI task for the rebuttal. The baseline and evaluation in the case of free form explanation is indeed very hard. We plan to perform a human evaluation to quantify the usefulness of our approach.

Reviewer comment (Experiments):
How would/could you modify your algorithm to work with a pretrained LM without exhaustive fine-tuning on a target task?

Author response:
We assume that the reviewer is referring to prompting LLMs to perform a task without fine-tuning. Our approach relies on the latent concepts of a task with respect to a model. Given training data of a task, say sentiment analysis, one can prompt the model using the training data, extract activations and create latent concepts of the task. At test time, given the generated token(s), one can identify latent concepts and generate an explanation of the prediction.

Reviewer comment (Experiments):
If you have some resource limitations, but wanted to scale up these experiments to a much larger model (e.g 1.5B parameter model), would this be straight forward to do? Could you fine-tune on a subset?

Author response:
We can finetune the model using the full training data. The scaling issue of our approach will come in clustering a dataset of millions/billions tokens. One can consider a random subset of the training data to approximate the latent space. The efficacy of such a solution requires empirical evaluation and is out of the scope of this paper. The current paper aims at providing explanation of a task specific model where generally the training data is not as large as the pretraining dataset.

Reviewer comment (Evaluation):
The quality of the generated natural language explanation is evaluated with case studies only.

Author response:
**A:** The automatic evaluation of the free form explanation is difficult. For this reason, we separately performed automatic evaluation of each component of LACOAT’s pipeline to quantify its correctness. As described in the paper, the sequence labeling task (POS tagging) was deliberately considered in the study to enable a controlled evaluation of each component. We believe that a human evaluation of the explanation will suffice as an evaluation of the system as suggested by the reviewer earlier.

Reviewer comment (Experiments):
The authors could perform human evaluation to assess the utility of explanations, and they could consider baselines and ablation studies such as (1) generating rationales with LLMs (Plausifier in Sec. 2.4) directly based on inputs and predictions, and (2) skipping the conceptmapper and directly generating explanations with extracted salient words (in Sec. 2.2) and LLMs.

Author response:
**A:** Thank you for suggesting various baselines to compare in human evaluation. We will design a study and will perform human evaluation.

Reviewer comment (Evaluation):
There is no quantitative evaluation of faithfulness, which is crucial for explanation algorithms but is not quite intuitive for generative explanations.

Author response:
**A:** LACOAT provides explanation both at latent concept level and as free form explanation. As mentioned by the reviewer, quantifying faithfulness in the case of freeform explanation is hard. However, we argue that the last layer latent concept-based explanation is faithful since the main attributed token at the last layer is the classification head and its latent concept is a reflection of how the model relates it with specific knowledge of the training data.

Reviewer comment (Experiments):
The advantage of concept-based explanation over other attribution-based algorithms is not clear from the experiments.

Author response:
**A:** We will include specific examples to support the claim.

Reviewer comment (Experiments):
The claim in the introduction that a limitation of attribution methods is due to the "multi-facet of words in different contexts" is not supported by case studies in the experiments.

Author response:
**A:** We will include specific examples to support the claim.

Reviewer comment (Experiments):
It is unclear whether concept-based explanations are applicable to more complex reasoning tasks, like reading comprehension and commonsense reasoning, and what explanation outputs would be expected in those cases.

Author response:
**A:** The output of LACOAT varies depending on the task as can be seen from the presented sequence labeling task and the sequence classification task where in the former the latent concepts consist of words while in the latter case they consist of sentences. The Plausifier explanation will also be task dependent. We performed experiments using the MNLI task and found latent concepts at the last layer consisting of sentence pairs. The sentences are grouped together in a latent concept based on how the hypothesis relates with the premise e.g. summary, rephrasing the argument, logical relationship, etc.