Paper: The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction (ICLR-2024)

Reviewer comment (Experiments):
Analysis on other text domain tasks such as reading comprehension could provide further insights.

Author response:
**Results on New Datasets:** We have added analysis on Epistemic Reasoning (logic and reading comprehension), TruthfulQA (factuality), and QA Wiki Data (world knowledge). The Epistemic Reasoning dataset is provided by Big Bench and was developed specifically to measure reading comprehension and reasoning. Please see Table 4 in the Appendix and the general response. All edits are written in red. 

As with other datasets, we find that LASER offers notable boosts in performance across model architectures. We believe that **improvements across 8 datasets and multiple LLMs and tasks provide strong evidence for using LASER.**

Reviewer comment (Presentation):
In Figure 3(c), what is the number of originally correct and answer-corrected datapoints?

Author response:
**Clarification:** We apologize for not defining these terms clearly. “Originally correct” describes samples that are correctly classified even without any intervention.  “Answer-corrected” refers to questions that the model gets correct only after intervening with LASER (i.e., they were not correctly classified before executing LASER). We have edited the text to clarify this in the new version of the paper.

Reviewer comment (Evaluation):
It appears that there is a significant improvement in overall performance with GPT-J compared to LLama2 or Roberta. What could be the underlying cause of the result?

Author response:
**Possible Causes:** Thank you for the question and observation. Although the improvements in GPT-J are more pronounced than in LLAMA2 or Roberta,  there are some domains where improvements in LLAMA2 (FEVER) and Roberta (Bias in bios) are more significant than in GPT-J as well. We consider a thorough analysis of this to be outside the scope of this work, but there are several possible causes that we believe merit investigation, including the capacity of the model, the amount of training data, and the particulars of the optimization procedure. We have added this as an important direction for future research in the current draft.

Reviewer comment (Experiments):
The authors do not provide the final search results of rank reduction, i.e. the layers selected for compressing and the reduced rank, in the final performance in Table 1, 2 & 3. It is very important to provide these results to show that the selected model is indeed in a reduced rank.

Author response:
**Final Search Results:** Thank you for pointing this out. We have added the final search results of the reduced ranks in Table 5 and Table 6 of the updated paper.  As noted in the paper, the largest gains typically occur when intervening in higher layers of the LLM. For reference, Llama2 has 32 layers, GPTJ has 28 layers, and Roberta has 12 layers.

Reviewer comment (Presentation):
I find that the dimension of GPTJ is 4096, which should be $d$ in your notation. So in Fig 2, what is the rank of Reduction 99.5%/99.75% and others with .5%? (4096*0.0025=10.24, not an integer?)

Author response:
**Rounding Clarification:** As mentioned in Section 4, last line of paragraph 1: “We replace it with a rank ⌊ρ · d⌋-approximation.”  Here, “⌋” represents the floor of the number, i.e., the number is rounded down to the nearest integer value.

Reviewer comment (Experiments):
The used CounterFact is very similar to the table-to-text generation task (but for a qa task), which is not a frequently used dataset to test even the QA and factual/world-knowledge reasoning performance of LLM. Any reason for choosing the dataset?

Author response:
**Justification for using Counterfact:** The CounterFact dataset (introduced by Meng et al. 2022 ) is derived from the PARAREL dataset (Elzar 2021) and WikiData. This dataset is well-cited and commonly used for research on LLM understanding, and this was the main reason why we picked it. Including this dataset afforded the measurement of robustness to paraphrases with and without LASER (See Section 5.1 para 3). This study deepened our understanding of LASER’s robustness to paraphrases. Additionally, we provide results on 7 other datasets, including 3 new datasets that we added in this rebuttal. Specifically, we have also included the effect of LASER on WikiData QA, a similar benchmark from Big Bench.

Reviewer comment (Presentation):
There is a typo: Table 3 needs a horizontal line at the bottom.

Author response:
We have added the missing horizontal line in Table 3 and thank you for pointing it out.

Reviewer comment (Experiments):
Seems like the MLP layers are the key components in storing the noise vs storing the “useful inductive biases”. I wonder if some structural choices (e.g., different position embedding methods, different activation functions, number of heads, etc.) can affect Transformer’s low-rank vs high-rank component behavior as well.

Author response:
Examining how structural choices in model design affect the low-rank and high-rank components of the MLP layers is an intriguing question that could advance our understanding of this phenomenon and of transformer models in general. While we consider a thorough investigation of this question to be outside the scope of this work, and it will require training models with different structural choices from scratch, we have included a discussion of this in our paper to contextualize our results in light of this discussion (please see Appendix F).