Paper: Tuning-Free Accountable Intervention for LLM Deployment - A Metacognitive Approach (ICLR-2024)

Reviewer comment (Presentation):
Lack of captions on some figures can disrupt the flow of understanding for the reader.

Author response:
>**[Q1]**: Lack figure caption.

All figures had captions in the original manuscript. Some figures had brief captions. For better understanding, we have further revised the captions for Figure 3, 4, 5, 7, 8 in the updated manuscript. If you still have doubts for the captions, please kindly let us know the details.

Reviewer comment (Experiments):
The paper emphasizes its effectiveness through experiments on real-world datasets. However, the scope and diversity of these datasets aren't detailed (only text classification tasks considered in the paper), raising questions about the framework's general applicability.

Author response:
> **[Q2]**:  The scope and diversity of these datasets aren't detailed and only text classification tasks considered in the paper

Thank you for highlighting the concerns related to the diversity of datasets used in our study. In the paper, the dataset statistics are given in **Table 1**. The dataset experimented are all text classification tasks. This dataset limitation stems from the constrained variety of available open-source datasets with sufficiently label concepts. This limitation is indeed a broader challenge within the research community, one that has not received adequate attention.

Our work, to the best of our knowledge, has incorporated **all** relevant datasets that are currently accessible and align with our research objectives. As a pioneering work, we understand that the novelty of our problem might contribute to the scarcity of diverse datasets. However, we view this as an opportunity to pioneer new paths in this field. We hope our framework and the promising results we've achieved will inspire more researchers to contribute to the development and curation of more comprehensive and varied benchmark datasets.

We are absolutely open to expanding our research to include additional datasets, should they become available or be recommended to us. Suggestions and contributions in this area are not only welcomed but actively encouraged. We believe that by collaborating with the wider research community, we can collectively enhance the general applicability and robustness of our framework.

Reviewer comment (Experiments):
Considering the increasing scale and complexity of newer models, is the CLEAR framework compatible with larger models such as LLaMA? Furthermore, can the metacognitive capabilities of the CLEAR approach provide benefits when applied to these more extensive and potentially more intricate architectures?

Author response:
> **[Q3]**:  Is the CLEAR framework compatible with larger models such as LLaMA? 

Thank you for your valuable feedback. It is important to clarify that our framework, CLEAR, is **model-agnostic** and is **compatible with different LLMs**. The primary reason for our initial focus on T5-base in Table 2 was the availability of an open-sourced MoE model compatible with it. Unfortunately, up to now, we have not got access to open-sourced MoE models that could be seamlessly integrated with pretrained LLaMA 1/2 or Mistral. This limitation restricted our ability to conduct experiments on these specific models within the scope of our current study.

However, we have recognized the necessity of expanding our evaluation to include other prominent LLMs and have made strides in this direction. Notably, we have added an OPT-MoCE model (with OPT as the LLM backbone), which is a significant step toward applying our methodology to a wider range of LLMs. The results are updated in the manuscript in **Table 2**. We can see CLEAR still works effectively if we change the backbone to OPT. We believe this development demonstrates our commitment to advancing our research to encompass a broader spectrum of models and settings.

Reviewer comment (Experiments):
Can this be extended to learn undefined concepts? Or human-annotated concepts have to be provided.

Author response:
Thank you for your insightful comments regarding the reliance on pre-defined, human-annotated concept labels in our framework. Your concern about the limitations this might impose, particularly in scenarios where such labels are scarce or potentially inaccurate, is indeed valid and warrants further discussion.

Firstly, we acknowledge that the current iteration of our model, CLEAR, primarily utilizes pre-defined concepts, which may limit its applicability in certain contexts. This choice was made to establish a clear and controlled experimental environment to validate the efficacy of our metacognitive approach. However, we recognize the challenges posed by scenarios where human-annotated concepts are either hard to obtain or subject to inaccuracies.

To address these concerns, we are actively exploring the possibility of extending CLEAR to accommodate the learning of undefined or emergent concepts. This extension would involve developing mechanisms within CLEAR that can autonomously identify and categorize new concepts from the data, thereby reducing reliance on human-annotated labels. Techniques such as unsupervised or semi-supervised learning could be employed to facilitate this capability, allowing CLEAR to adapt to a broader range of applications, especially in data-scarce or dynamic environments. Currently, researchers are also trying to use LLMs to label the concepts. 
Furthermore, to mitigate the issue of noisy concept labels, we are considering the incorporation of robust error detection and correction algorithms. These algorithms would aim to identify and rectify inaccuracies within the concept labels, enhancing the overall reliability of the model. These are potential future directions but are currently out of the scope of this work.

In summary, while our current model does utilize human-annotated concepts, we are committed to evolving CLEAR to be more versatile and adaptable to various real-world scenarios. Future iterations of our framework will focus on these aspects, aiming to make CLEAR a more universally applicable and resilient tool in the field of LLMs.

Reviewer comment (Novelty):
[Novelty] The use of Mixture of Experts on LLMs is not new. The authors should at least discuss the differences between previous works [1,2] and their own.

Author response:
>**[Q1]**: Discuss the differences between previous MoE for LLM works [1,2] and the current work.

Thanks for bringing up this question. We have included an additional section in Appendix D for discussing the previous MoE works and emphasize the uniqueness of our work. Specifically, our framework utilizes concept-aware MoE to achieve “tuning-free”, “autonomous”, “accountable” intervention for LLMs, which differs from the previous traditional MoE researches (such as Sheng S. et al. [1] and Mikel A. et al. [2]) that primarily explore MoE for enhancing overall model performance or efficiency. This metacognitive approach equips LLMs with self-aware error identification and rectification capabilities, a departure from the conventional applications of MoE. By focusing on error detection and correction, we address a critical gap in current LLM applications, especially in high-stakes scenarios where trustworthiness and accountability are paramount. We believe these aspects significantly elevate the novelty of our work, and we have elaborate on these distinctions more comprehensively in an extra related work section in the Appendix E to clarify our contribution to the field. We have copied and pasted the section below for convenience:

**Mixture of Experts in Large Language Models.**
The incorporation of Mixture of Experts (MoE) into Large Language Models (LLMs) has evolved significantly, with early research by [4] laying the groundwork. These foundational studies [1-6] focused primarily on improving model performance and computational efficiency  in a black-box manner. On the contrary, in this work, we utilize the design of MoE in LLMs for metacognitive capabilities. This novel approach, distinct from earlier efficiency-focused applications, uses MoE for error detection and correction, a critical step towards solving the interpretability and trust issues in AI decision-making. Our framework, CLEAR, contributes to this evolving landscape by embedding MoE within a metacognitive framework, emphasizing error rectification, transparency, and autonomy in LLMs. This shift marks a significant advancement from traditional MoE applications, positioning CLEAR at the forefront of innovative LLM enhancement strategies.

[1] Sheng S. et al. Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models

[2] Mikel A. et al. Efficient Large Scale Language Modeling with Mixtures of Experts

[3] Shazeer, Noam, et al. "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer." International Conference on Learning Representations. 2016.

[4] Fedus, William, Barret Zoph, and Noam Shazeer. "Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity." The Journal of Machine Learning Research 23.1 (2022): 5232-5270.

[5] Zhou, Yanqi, et al. "Mixture-of-experts with expert choice routing." Advances in Neural Information Processing Systems 35 (2022): 7103-7114.

[6] Du, Nan, et al. "Glam: Efficient scaling of language models with mixture-of-experts." International Conference on Machine Learning. PMLR, 2022.

Reviewer comment (Experiments):
[Complexity and Scalability] The authors do not discuss the computational overhead of adding more experts (LLM backbones) and the trade-off between improving model performance and adding extra experts. Furthermore, in Table 2, the authors only test CLEAR on T5-base. It would be interesting to see the behavior of the proposed method on larger LLMs (like LLaMA 2 13B or Mistral 7B), but with fewer expert layers.

Author response:
> **[Q3]**: Missing discussion on the computational overhead of adding more experts (LLM backbones) and the trade-off between improving model performance and adding extra experts. 

Thanks for pointing this out, we have added an experiment with detailed analysis in **Appendix G** that illustrates the Flops number, which indicates the number of floating-point multiplication operations consumed on one the inference with a sequence of 128 tokens. 

Thank you for pointing out the important aspect of computational overhead in relation to the addition of more experts in our model. To address this, we conducted an additional experiment to quantify the increase in computational complexity, as measured by FLOPs, with the increasing number of experts. The results, as shown in the attached **Figure 11**, clearly indicate an almost linear increase in computational complexity with the number of experts. This is an expected outcome, as each additional expert adds a similar computational load to the model. Note that extra experts are only added to those erroneous samples and does not require extra parameter tuning. 

It is crucial to note that the addition of each expert provides a significant improvement in model performance. We acknowledge that there is a trade-off between the increased computational overhead and the enhanced performance achieved. In practical applications, this trade-off would need to be carefully balanced based on the specific requirements and constraints of the task at hand.

In future work, we plan to investigate techniques for optimizing the computational efficiency of our model, such as pruning less effective experts or implementing more efficient computation strategies. This will allow us to maintain or even enhance model performance while managing the computational costs more effectively.

> **[Q4]**: Can CLEAR work on other LLMs like LLaMA2, Mistral?

Thank you for your valuable feedback. It is important to clarify that our framework, CLEAR, **is model-agnostic** and is compatible with all LLMs. The primary reason for our initial focus on T5 in Table 2 was the availability of an open-sourced MoE model compatible with it. Unfortunately, up to now, we have not got access to open-sourced MoE models that could be seamlessly integrated with pretrained LLaMA 1/2 or Mistral. This limitation restricted our ability to conduct experiments on these specific models within the scope of our current study.

However, we have recognized the necessity of expanding our evaluation to include other prominent LLMs and have made strides in this direction. Notably, we have added an OPT-MoCE model (with OPT as the LLM backbone), which is a significant step toward applying our methodology to a wider range of LLMs. The results are updated in the manuscript in **Table 2**. We can see CLEAR still works effectively if we change the backbone to OPT. We believe this development demonstrates our commitment to advancing our research to encompass a broader spectrum of models and settings.

Reviewer comment (Experiments):
I am curious about how CLEAR ensures that the dynamic adjustment of the expert allocation does not lead to overfitting or catastrophic forgetting when fine-tuning on different tasks on T5-base.

Author response:
> **[Q6]**: How CLEAR ensures that the dynamic adjustment of the expert allocation does not lead to overfitting or catastrophic forgetting when fine-tuning on different tasks on T5-base.

Thank you for your insightful question regarding the safeguards CLEAR employs against overfitting and catastrophic forgetting during fine-tuning on different tasks.
- *Clarification on Overfitting in Direct Fine-Tuning*: As highlighted in Table 2 of our paper, direct fine-tuning on erroneous samples often leads to overfitting, as indicated by increased scores on the test set. This form of overfitting can result in the model neglecting previously acquired, useful knowledge, leading to a generalization error, evidenced by reduced performance on the development set.
- *Selective Expert Allocation to Mitigate Overfitting*: In contrast to direct fine-tuning, CLEAR adopts a selective approach in allocating top-T experts. We retain the minimum necessary number of top-T experts for standard samples while selectively increasing this number for those samples identified as potentially erroneous. This strategy ensures a focused and efficient allocation of computational resources, preventing the model from over-adjusting to specific samples or tasks.
- *Logit Entropy Scrutiny Mechanism*: Our approach utilizes a logit entropy scrutiny mechanism, which plays a crucial role in identifying and targeting the most challenging cases for additional processing. This targeted approach helps in avoiding a blanket increase in model complexity, which is a common cause of overfitting in more generalized training scenarios.
- *Balancing Knowledge Retention and Adaptation*: CLEAR is designed to strike a balance between retaining useful prior knowledge and adapting to new tasks. By increasing expert involvement only where necessary, CLEAR avoids the pitfall of catastrophic forgetting, where a model loses its ability to perform previously learned tasks in the process of adapting to new ones.
- *Empirical Evidence of Effective Balancing*: The empirical results presented in our paper, particularly those contrasting the performance on development and test sets, serve as evidence of CLEAR's ability to effectively balance the dual challenges of avoiding overfitting and catastrophic forgetting.
In summary, through selective expert allocation and the use of a logit entropy scrutiny mechanism, CLEAR effectively navigates the challenges of overfitting and catastrophic forgetting. This nuanced approach allows CLEAR to adapt to new tasks while retaining its proficiency in previously learned tasks.

Reviewer comment (Experiments):
Can the Concept Bottleneck Models (CBMs) developed by CLEAR be effectively generalized across diverse domains and applications, or do they require domain-specific tuning?

Author response:
> **[Q5]**:  Can the Concept Bottleneck Models (CBMs) developed by CLEAR be effectively generalized across diverse domains and applications, or do they require domain-specific tuning?

Thank you for your question regarding the generalizability of the Concept Bottleneck Models (CBMs) developed by CLEAR across diverse domains and applications. Currently, our implementation of CBMs in CLEAR does necessitate domain-specific fine-tuning. This requirement is reflective of the inherent characteristics of current CBM architectures, which are typically optimized for specific domains to ensure maximum effectiveness and accuracy.The development of generalizable CBMs across diverse domains and applications is a great direction to work on, but is out of the scope of this paper.

Reviewer comment (Experiments):
I am confused about the methods shown in Table 2. For instance, for the method ‘prompting’ in the first line, are you simply using prompting on GPT-4 without any help of the MoE and self-correction techniques proposed in CLEAR? If so, it would be much more interesting to see the results of prompting on the LLM backbones for different expert layers within the proposed CLEAR framework without training CBMs.

Author response:
> **[Q7]**: For the method ‘prompting’ in the first line, are you simply using prompting on GPT-4 without any help of the MoE and self-correction techniques proposed in CLEAR? If so, it would be much more interesting to see the results of prompting on the LLM backbones for different expert layers within the proposed CLEAR framework without training CBMs.

Thank you for your perceptive question regarding the methods outlined in Table 2, particularly the use of prompting with GPT-4 in relation to our CLEAR framework.

- *Use of Prompting with GPT-4*: In the case of GPT-4, as mentioned in our paper, we utilized the model in its standard, API-based format. This means that we did not apply the Mixture of Experts (MoE) or self-correction techniques of CLEAR, primarily due to the current unavailability of internal access to GPT-4's architecture. Our use of GPT-4 was intended to provide a baseline for understanding the performance of conventional large language models (LLMs) without the specialized interventions proposed in CLEAR.

- *Limitations with Other Large Language Models*: This limitation is not unique to GPT-4. Other large language models such as LLaMA and Mistral also currently lack open-sourced MoE components that are compatible with them. This restricts our ability to directly apply CLEAR’s methodologies to these models.

- *Model-Agnostic Nature of CLEAR and Future Potential*: It is important to highlight that CLEAR is designed to be model-agnostic. This inherent flexibility suggests that applying CLEAR’s techniques to different LLM backbones, including various expert layers, is not only feasible but could yield significant insights. Your suggestion of exploring the impact of prompting within the CLEAR framework across different LLM backbones is intriguing and aligns well with our objective of enhancing model performance through specialized techniques.

- *Future Exploration Plans*: We are enthusiastic about the potential of integrating CLEAR with more open-sourced LLMs as they become available. Such integration would allow us to more comprehensively evaluate the effectiveness of our proposed techniques across a wider array of models, further validating the versatility and impact of CLEAR.

In summary, while our current application of prompting with GPT-4 and similar models does not incorporate CLEAR’s MoE or self-correction techniques due to accessibility constraints, the prospect of applying CLEAR’s methodologies to various LLM backbones in the future is an exciting avenue for research that we aim to explore.

Reviewer comment (Theory):
The authors identified erroneous prediction by automatically dividing confidence into two groups via K-means clustering (Section 3.2), where the subset of examples with a lower confidence group is flagged as erroneous. While this approach eliminates human involvement, K-means is not robust against outliers. In many real-world scenarios, such as predicting malicious vs. non-malicious users on the web, and predicting bug vs. non-bug features in software security, the binary classification task is imbalanced. Thus, automatically setting the threshold might not be the right way.

Author response:
>**[Q1]**: Using K-means to get logit threshold eliminates human involvement but has limitations. K-means are not robust to outliers, and suffer from imbalance. 

K-means has been proved to be an effective method to detect potentially incorrect predictions / outliers for deep models as shown in many existing works [1,2]. For example, the paper [1] uses K-means to cluster model’s prediction confidence to detect potential noisy labels. Our work draws similar inspiration from these existing works. We have also shown additional results in **Figure 9 (a) (b)** in Appendix E to ensure that our K-means implementation is not unduly influenced by extreme values or imbalance issues. More detailed analyses are as follows:
1. *Robustness Against Outliers in Logits Clustering*. We recognize that K-means can be sensitive to outliers. However, in our specific application of clustering model logits, this sensitivity is mitigated by the nature of the logits themselves. Logits, being the raw output of a model before a final activation function like softmax, typically do not exhibit extreme variances as raw input data might. Moreover, our preprocessing of logits includes normalization steps that further reduce the impact of any potential outliers. 
2. *Addressing Imbalance in Logits Clustering*. The concern about imbalanced datasets is crucial in classification tasks. However, our approach uses K-means for clustering logits, not for direct classification. The objective here is to identify patterns in the logits that correspond to normal and abnormal behaviors, which is a different task from the original binary classification. Our empirical evidence (detailed in Figure 3.) shows that the distribution of logits, even in imbalanced classification scenarios, allows for effective separation into normal and abnormal groups. This separation is less about the balance in the original classes and more about the distinction in model confidence, as represented by the logits.
3. *Rationale for K-means in Logits Clustering*. The use of K-means in our study is driven by the need for an interpretable and straightforward method to distinguish between normal and abnormal model behaviors based on logits. This clustering is not intended to replace the original classification task but to provide an additional layer of analysis. By automatically dividing logits into two clusters, we can identify which instances the model is most and least confident about, and this has proven to be a valuable tool in further understanding and improving our model's performance.

[1] Nematzadeh, Zahra, Roliana Ibrahim, and Ali Selamat. "A hybrid model for class noise detection using k-means and classification filtering algorithms." SN Applied Sciences 2 (2020): 1-10.

[2] Zhu, Zhaowei, Yiwen Song, and Yang Liu. "Clusterability as an alternative to anchor points when learning with noisy labels." International Conference on Machine Learning. PMLR, 2021.

Reviewer comment (Evaluation):
The approaches described in Appendix A involve extracting latent representation Z of text encoder, x, and then adding an extra layer of concept representation R^K and class label. The authors did not report whether their model overfits.

Author response:
> **[Q3]** The authors did not report whether their model overfits for concept predictions.

Thanks for pointing this out. In response to your query regarding the potential for overfitting in our model, we have added a detailed analysis of the model's performance across training epochs (**Appendix F** in the updated manuscript), as visualized in the provided plot of training loss and validation accuracy for concept predictions.

The plot illustrates a consistent decrease in training loss over epochs, indicating effective learning from the training dataset. Simultaneously, the validation accuracy shows an overall increasing trend, reaching a plateau towards the later epochs. This pattern of convergence in loss and stability in accuracy suggests that our model is not merely memorizing the training data but is effectively generalizing to new, unseen data. We adopt the “early stop" strategy to avoid overfitting, where the model with the highest validation accuracy is selected and evaluated on the test set. 

Additionally, the methods described in Appendix A, involving the extraction of latent representation Z and the addition of an extra layer for concept representation R^K, are designed to enhance the model's ability to capture and generalize key concepts without overfitting to specific training examples.

In conclusion, the observed trends in the training and validation metrics provide confidence that our model, while sophisticated in its approach to concept prediction, maintains a strong generalization capability without falling into the trap of overfitting.