Paper: INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection (ICLR-2024)

Reviewer comment (Evaluation):
Regarding the outliers in the activations, it would be nice to see if these outliers really lead to overconfident predictions. For example, is there any correlation between the number of outliers and accuracy of the answer? At least, it could be informative to see a few generated examples corresponding to many outliers or no outliers.

Author response:
**Ans** Thanks for your suggestion. The outliers features increase the likelihood of generating overconfident and self-consistent generations. For example, the top 50 tokens that contain the most outliers have an average confidence of 0.98, while the bottom 50 tokens that contain the least outliers have an average confidence of 0.54. According to your suggestion, we provide several examples that contain many/few extreme features in the **Appendix H.2 (Page19)**. The results show that when there are many extreme features, the model tends to generate consistency hallucination outputs across multiple generations. Instead, when there are few extreme features, the model generates diverse hallucination outputs which can be spotted by different hallucination detection metrics.

Reviewer comment (Evaluation):
For Table 2, it would be interesting to see if the quality of answers is increased after using clipping (as confident mistakes are excluded).

Author response:
**Ans** It is worth noting that the feature clipping approach is proposed to prevent the model from generating overconfident outputs for uncertain questions, but not to improve the quality of answers. To demonstrate the effectiveness of the feature clipping strategy, we show several  cases before and after feature clipping in the **Appendix H.3 (Page20)**

Reviewer comment (Evaluation):
In general, it would be nice to see some qualitative examples of hallucinations that are detected by the proposed method, but not detected by the baselines.

Author response:
**Ans** Thanks for your suggestion. We have provided comprehensive cases in **Appendix H.1 (Page16)**, including good cases and failure cases. Please see the details in our revised paper.

Reviewer comment (Writing):
centralized matrix -> centering matrix

Author response:
**Ans** Thanks，the typos have been revised. We will re-polish our paper carefully.

Reviewer comment (Experiments):
One of the obvious shortcomings of the proposed method is the applicability to blackbox LLMs, e.g., GPT, PaLM and Claude. The method relies on the internal state of the model, e.g., LLaMA and OPT, but the authors may not discuss the limitation well.

Author response:
The reliance on white-box LLMs is a limitation of our work, and thanks for your suggestion, we have discussed the limitations of our work in the revised paper (**Appendix G, Page 16**).
In addition, black-box based [1,8] and white-box based [2,3] methods are equally important for different application scenarios for LLMs. Compared to the methods designed for black-box methods, exploring the internal information for hallucination detection has its own advantages: (1) The internal states retain highly-concentrated semantic information; (2) Exploring the sentence embedding in the internal states is much more efficient compared to extracting the sentence embedding with an additional large model [2,3].

Reviewer comment (Evaluation):
The computational cost is another concern since essentially this method is a sampling-based approach. While the number of samples is not large, how to further reduce the cost for real production is still unsolved.

Author response:
One limitation of our work is that it requires additional computational overhead，which is also the limitation for all consistency-based [2] or prompting-based methods [7]. We have discussed this limitation in our revised paper (**Appendix G, Page16**). 

Besides, we also compare and analysis the computational overhead of our proposal with the base LLM and other compared baselines. The average inference time (second per question) is computed on one NVIDIA-A100 GPU and shown in Table 1. As observed, our EigenScore is about 10 times more efficient than the methods that rely on another large model to measure the self-consistency (such as SelfCheckGPT [2]), and shares the similar computational overhead with the LN-Entropy and Lexical Similarity. It is worth noting that the inference overhead required to generate multiple results is not linearly proportional to the time of generating a single output, owing to the sampling and decoding strategy of the autoregressive LLM model. 

Table 1: Inference cost comparison of different methods in LLaMA-7B and LLaMA-13B.
|   Methods | BaseLLM | Perplexity | LN-Entropy | Lexical Similarity | SelfCheckGPT | **EigenScore** |
|:---------:|:-------:|:----------:|:----------:|:------------------:|:------------:|:--------------:|
|  LLaMA-7B |  0.245s |   0.245s   |   0.803s   |       0.815s       |    10.68s    |     0.805s     |
| LLaMA-13B |  0.309s |   0.307s   |    1.27s   |        1.28s       |    10.26s    |      1.27s     |


Compared to the computational overhead of generating multiple outputs, the cost of feature clipping and EigenScore computation is negligible (0.06s). For more details about the **Computational Efficiency Analysis**, please see the **Appendix D (Page14)** of our revised paper.

Reviewer comment (Theory):
The proposed method is mainly inspired by uncertainty estimation but how to bridge the connection with hallucination detection is still unclear, even though the results look promising.

Author response:
LLMs' knowledge hallucinations primarily arise from uncertainties regarding the source information or uncertainties inherent in the decoding process [5]. Therefore, exploring the uncertainty [5,6] or self-consistency [2,7] of LLMs' generations is one of the most promising direction for effective hallucination detection. According to [7], the self-contradictory hallucinations is one of the most important form of hallucination, and **67%** inconsistent generation indicates hallucination. Therefore, we suggest that our proposal exploring semantic consistency with LLMs' internal information is potentially a promising approach for hallucination detection.

Reviewer comment (Experiments):
Beyond hallucination detection, is that possible to mitigate hallucination through the proposed framework eventually?

Author response:
In most studies, the hallucination detection and mitigatition are two independent modules [3]. Most hallucination mitigatition methods can be categorized into prompting-based, external knowledge based and training/model-editing based methods [9]. In this study, we mainly focus on the hallucination detection. However, the hallucination content can be appropriately suppressed by the well-established prompting-based or external knowledge based approaches after identifying the hallucination content with EigenScore. For exapmle, in [7], the authors use a simple prompt template to correct the detected self-contradiction hallucination. 

In future work, we will delve into how to effectively mitigate hallucinations with our EigenScore.

Reviewer comment (Experiments):
How about the proposed method compared with this work [1] on the TruthfulQA dataset? Both work relies on the internal state of the LLM models.

Author response:
Considering that replicating the ITI [1] would be prohibitively expensive, we test the performance of our proposal and comparison baselines in the TruthfulQA dataset. The results are presented in Table 2. For the ITI, we report the best performance in their paper. As can be seen, our proposal consistently outperforms the baseline methods and achieves comparable performance as ITI when we utilize 50 in-distribution prompts. 

Table 2: Performance comparison of different methods on TruthfulQA dataset. Accuracy is reported.
| #prompt | Perplexity | LN-Entropy | LexialSim | SelfCKGPT | ITI* | **EigenScore** |
|:-------:|:----------:|:----------:|:---------:|:---------:|:---------:|:--------------:|
|    5    |    70.0    |    71.2    |    73.6   |    74.2   |  **83.3** |      76.7      |
|    20   |    76.4    |    77.7    |    77.9   |    76.8   |  **83.3** |      79.5      |
|    50   |    73.1    |    77.9    |    73.6   |   78.3    |  **83.3** |    **81.3**    |

It's worth nothing that the ITI [2] relies on training 1024 binary classifiers in TruthQA datasets. Therefore, their best performance is better than our method which has not been trained on TruthfulQA. However, training on the validation set also limits the generalization of their method on other domains [2].

As TruthfulQA is a very challenging dataset for LLMs, zero-shot inference results in poor performance. Therefore, we follow previous work [10] to utilize different number of in-distribution prompts during inference time. The results show that the performance could be significantly improved when we increase the number of prompts, which also explains why ITI performs good.

For more information about the experiments, please see the **Appendix A (Page13)** of our revised paper.

Reviewer comment (Reproducibility):
Can you provide the computational cost estimation of the whole framework and which type of resources you are using?

Author response:
Please refer to our response to **W2**. We have provided a detailed **Computational Efficiency Analysis** in **Appendix D (Page14)** of our revised paper. All experiments are performed on the NVIDIA-A100 GPU.

Reviewer comment (Experiments):
Since the evaluation metric is AUROC, I am wondering if, can you compare the proposed method with the GPT 3.5 or 4 performance. Or can you add some comparison with the hallucination detection baseline methods, e.g., selfcheckGPT as you mentioned in the paper?

Author response:
To demonstrate the effectiveness of our proposal, we also compare our EigenScore with several competitive methods, including **Semantic Entropy (SemanticEnt)** [3], Shifting Attention to Relevance (**SentSAR**) [4] and SelfCheckGPT (**SelfCKGPT**) [2]. The comparison results in Table 3 demonstrate that our EigenScore significantly outperforms the competitor. 

Table 3: Performance comparison of EigenScore and and several competitive methods on CoQA dataset. AUC$_s$/AUC$_r$ score is reported.
|       Methods      | SemanticEn |  SentSAR  | SelfCkGPT |   EigenScore  |
|:------------------:|:----------:|:---------:|:---------:|:-------------:|
| OPT-6.7B |  63.1/71.7 | 69.8/72.2 | 70.2/74.1 | **71.9/77.5** |
|  LLaMA-7B |  64.9/68.2 | 70.4/65.8 | 68.7/72.9 | **71.2/75.7** |
| LLaMA-13B |  65.3/66.7 | 71.4/64.7 | 68.1/77.0 | **72.8/79.8** |

Note that both SentSAR, SelfCheckGPT and our proposal evaluate the quality of LLMs' generation by exploring the semantic consistency across multiple outputs. In contrast to Semantic Entropy and SelfCheckGPT which rely on another language model for sentence embedding extraction, our approach leverages the internal states of LLMs, which retain highly-concentrated semantic information. 

Additionally, the EigenScore defined by the LogDet of the sentence covariance matrix is able to capture the semantic consistency more effectively compared to the sentence-wise similarity [2]. Furthermore, the proposed feature clipping strategy allows our model to identify the overconfident hallucinations, which has not been investigated by existing methods [2,3,4]

Please see the **Appendix B (Page13)** for the details about the comparison results and analysis.

Reviewer comment (Experiments):
The current baselines are mainly from the authors' choice without open baselines, e.g., semantic uncertainty (ICLR 2023). I noted that many related works are good baselines that can be used for comparison.

Author response:
To demonstrate the effectiveness of our proposal, we also compare our EigenScore with several competitive methods, including **Semantic Entropy (SemanticEnt)** [3], Shifting Attention to Relevance (**SentSAR**) [4] and SelfCheckGPT (**SelfCKGPT**) [2]. The comparison results in Table 3 demonstrate that our EigenScore significantly outperforms the competitor. 

Table 3: Performance comparison of EigenScore and and several competitive methods on CoQA dataset. AUC$_s$/AUC$_r$ score is reported.
|       Methods      | SemanticEn |  SentSAR  | SelfCkGPT |   EigenScore  |
|:------------------:|:----------:|:---------:|:---------:|:-------------:|
| OPT-6.7B |  63.1/71.7 | 69.8/72.2 | 70.2/74.1 | **71.9/77.5** |
|  LLaMA-7B |  64.9/68.2 | 70.4/65.8 | 68.7/72.9 | **71.2/75.7** |
| LLaMA-13B |  65.3/66.7 | 71.4/64.7 | 68.1/77.0 | **72.8/79.8** |

Note that both SentSAR, SelfCheckGPT and our proposal evaluate the quality of LLMs' generation by exploring the semantic consistency across multiple outputs. In contrast to Semantic Entropy and SelfCheckGPT which rely on another language model for sentence embedding extraction, our approach leverages the internal states of LLMs, which retain highly-concentrated semantic information. 

Additionally, the EigenScore defined by the LogDet of the sentence covariance matrix is able to capture the semantic consistency more effectively compared to the sentence-wise similarity [2]. Furthermore, the proposed feature clipping strategy allows our model to identify the overconfident hallucinations, which has not been investigated by existing methods [2,3,4]

Please see the **Appendix B (Page13)** for the details about the comparison results and analysis.

Reviewer comment (Novelty):
The related work comparison is not comprehensive. There is no related work comparison between the internal-state detection approach. For example, [1] investigates mitigating the hallucinations by editing the internal states of llm. They first measure the possibility of generating hallucinations by observing the internal states and then editing those internal states. It might be better if the authors can compare their approach with the internal state-based hallucination detection approaches, e.g., [1].

Author response:
**Ans** Thanks for your suggestion. Considering that replicating the ITI [2] would be prohibitively expensive, we test the performance of our proposal and the baselines in the TruthfulQA dataset [1]. The results are presented in Table 1. For the ITI, we report the best performance in their paper. As can be seen, our proposal consistently outperforms the baseline methods and achieves comparable performance as ITI when we utilize 50 in-distribution prompts. 

It's worth nothing that the ITI [2] relies on training 1024 binary classifiers in TruthfulQA datasets, and they report the best performance (83.3) in the validation set. Therefore, their best performance is better than our method which has not been trained on TruthfulQA. However, training on the validation set also limits the generalization of their method on other domains [2].

Table 1: Performance comparison of different methods on TruthfulQA dataset. Accuracy is reported.
| #prompt | Perplexity | LN-Entropy | LexialSim | SelfCKGPT | ITI* | **EigenScore** |
|:-------:|:----------:|:----------:|:---------:|:---------:|:---------:|:--------------:|
|    5    |    70.0    |    71.2    |    73.6   |    74.2   |  **83.3** |      76.7      |
|    20   |    76.4    |    77.7    |    77.9   |    76.8   |  **83.3** |      79.5      |
|    50   |    73.1    |    77.9    |    73.6   |   78.3    |  **83.3** |    **81.3**    |

As TruthfulQA is a very challenging dataset for LLMs, zero-shot inference results in poor performance. Therefore, we follow previous work [6] to utilize different number of in-distribution prompts during inference time. The results show that the performance could be significantly improved when we increase the number of prompts, which also explains why ITI performs good.

Besides, to demonstrate the effectiveness of our proposal, we also compare our proposal with several competitive methods [3,4,5] that explore self-consistency for hallucination detection. The results in Table 2 demonstrate that our EigenScore consistently outperforms the the competitors by a large margin.

Table 2: Performance comparison of EigenScore and and several competitive methods on CoQA dataset. AUC$_s$/AUC$_r$ score is reported.
|       Methods      | SemanticEn |  SentSAR  | SelfCkGPT |   EigenScore  |
|:------------------:|:----------:|:---------:|:---------:|:-------------:|
| OPT-6.7B  |  63.1/71.7 | 69.8/72.2 | 70.2/74.1 | **71.9/77.5** |
|  LLaMA-7B |  64.9/68.2 | 70.4/65.8 | 68.7/72.9 | **71.2/75.7** |
| LLaMA-13B |  65.3/66.7 | 71.4/64.7 | 68.1/77.0 | **72.8/79.8** |

For more information about the experiments, please see the **Appendix A/B (Page13)** of our revised paper.

Reviewer comment (Experiments):
The llm models involve multiple transformation blocks; which block internal state do the authors investigate? It would make the paper more impactful if the authors could present the ablation on transformation blocks.

Author response:
**Ans** We have demonstrated the performance of exporing EigenScore in different layers (transformer blocks) in **Fig. 3(b)**. And the detailed discussions are illustrated on **Page8** in our submission. The results show that using the sentence embedding in the shallow and final layers yields significantly inferior performance compared to using sentence embedding in the layers close to the middle. Besides, another interesting observation is that utilizing the embedding of the last token as the sentence embedding achieves superior performance compared to simply averaging the token embeddings, which suggests that the last token of the middle layers retain more information about the truthfulness of generations.