Paper: How Large Language Models Implement Chain-of-Thought? (ICLR-2024)

Reviewer comment (Writing):
The first basic weakness of this paper is clarity, which is difficult to achieve in this area due to the technical nature of the content.

Author response:
Thanks for pointing out the clarity issue. 
- Following your kind suggestion, we have made adjustments to the content of the **Method** section in the main text (highlighted in blue). 
- We have redrawn the framework diagram (Figure 1) of our method and included it in the main text. We would appreciate it if you could find some time to review our revised version.

Reviewer comment (Evaluation):
The second basic weakness of this paper is the generality of the claims they make.

Author response:
Thanks for your constructive comments. 
- The reason we chose these two models (LLaMA2-7B and Qwen-7B) has two folds: 
  - i) The distinguishing feature of large language models compared to earlier language models is their emergent ability, which starts to emerge with 7B parameters. 
  - ii) The model architectures of large language models are predominantly transformers and text data. The main difference between different large language models lies in their training data. LLaMA2-7B [1] is primarily trained on English corpora, while Qwen-7B [1] is trained on both Chinese and English corpora. Both models [1,2] have shown good performance on NLP tasks. Therefore, we chose LLaMA2-7B as a representative of large English language models and Qwen-7B as a representative of large Chinese-English language models.

> ***Reference***
>
> [1] Llama 2: Open foundation and fine-tuned chat models. In Arxiv 2023. https://arxiv.org/abs/2307.09288
> 
> [2] Qwen technical report. https://qianwen-res.oss-cn-beijing.aliyuncs.com/QWEN_TECHNICAL_REPORT.pdf

Reviewer comment (Experiments):
Regarding the generality of the claims, the weakness stems from using limited data and reference models. The paper makes claims about LLMs at large based on only two example LLMs, without explaining why these two models are representative for LLMs at large or why results from these two models are expected to generalize. Additional claims specifying what classes of models these results are expected to apply to would improve the work.

Author response:
Thanks for your constructive comments. 
- The reason we chose these two models (LLaMA2-7B and Qwen-7B) has two folds: 
  - i) The distinguishing feature of large language models compared to earlier language models is their emergent ability, which starts to emerge with 7B parameters. 
  - ii) The model architectures of large language models are predominantly transformers and text data. The main difference between different large language models lies in their training data. LLaMA2-7B [1] is primarily trained on English corpora, while Qwen-7B [1] is trained on both Chinese and English corpora. Both models [1,2] have shown good performance on NLP tasks. Therefore, we chose LLaMA2-7B as a representative of large English language models and Qwen-7B as a representative of large Chinese-English language models.

> ***Reference***
>
> [1] Llama 2: Open foundation and fine-tuned chat models. In Arxiv 2023. https://arxiv.org/abs/2307.09288
> 
> [2] Qwen technical report. https://qianwen-res.oss-cn-beijing.aliyuncs.com/QWEN_TECHNICAL_REPORT.pdf

Reviewer comment (Experiments):
Limited to analyzing attention heads, does not cover other components like MLPs.

Author response:
Thanks for your constructive comments. 
- In this work, we focus on the attention heads because heads can better represent semantic information compared to neurons in MLP layers, consistent with previous work [1]. 
- Inspired by your insightful comments, we would like to note that heads serve as input units for MLP. If we can categorize identified heads under different MLPs, we would indirectly identify important MLPs, which may align with your insightful comments. 
- We sincerely appreciate your insight, and in our future work, we will directly explore MLPs rather than analyzing them indirectly through attention heads.

Reviewer comment (Experiments):
Did you consider any changes to the model architecture or training process to improve reasoning?

Author response:
Thanks for pointing out the promising direction. 

- We agree with your point that identifying key heads and making modifications is a very promising direction. However, this may involve the field of model editing/modifying, which is a promising direction to explore.
- We would like to note that we mainly focus on how to interpret the reasoning ability of large language models through head localization. Aligning with your insightful question, interpreting models paves the way for model re-training for finetuning to accurately improve model performance, i.e., on a specific task.

Reviewer comment (Evaluation):
Could your analysis approach help detect if a model is just memorizing vs. logically reasoning?

Author response:
It is an insightful question! Thanks for your kind guidance aiming to make our work more solid. 
- Model memorization is a classical/traditional field in deep learning while exploring a model's memorization capabilities is inherently challenging. In the context of large models, there are two challenges: i) determining whether the model's behavior is due to memorization, generalization, or reasoning is quite a challenging task, and ii) under the conditions of large models, we can access the model weights but not the training data, making it even more difficult to determine whether the model is memorizing or reasoning without access to the data.

We have added a discussion about the interesting direction in our revised paper (Appendix F).

Reviewer comment (Novelty):
The methodology used in the paper is not tailored to understanding CoT behaviors. Specifically, the method for identifying "important" attention heads is adopted from prior work (Wang et al. (2023)) and the only task-specific design is constructing the counterfactual sample $x_c$ from a reference sample $x_r$ by replacing the CoT part in the few-shot example prompt with random text generated by ChatGPT. More justification is needed for why important attention heads identified using such $x_c$ contribute to CoT behavior.

Author response:
***Ans for Q1.1):*** 
Thanks for highlighting the potentially confusing issue. We apologize for the misunderstanding, which may result from the inadequate explanation of **path patching**. In response to your comments, we have added corresponding explanations to our work.

- Path patching is a method used to study causal relationships and identify key modules in language models through causal interventions [1,2,3]. The causal interventions are implemented through the design of reference data and counterfactual data $x_r, x_c$, where the key distinction lies in whether they can elicit the model's specific abilities/behaviors. Our focus is on the CoT reasoning ability of large language models (LLMs). In this context, the difference between $x_r, x_c$ lies in whether they can trigger the model's reasoning ability. We believe it aligns with your comments that our $x_r$ triggers the model's CoT ability, and $x_c$ does not. Designing appropriate pairs is challenging and promising for interpreting LLMs.
- According to path patching, the causal relationship between key heads and the model's CoT ability can be investigated by designing appropriate $x_r, x_c$. The importance of data construction is explicitly stated in path patching [1,2,3], and this crucial aspect is also reflected in our study. Poorly designed $x_c$ may inadvertently trigger the model's reasoning ability, making it difficult to locate the model's CoT reasoning ability, as discussed in our ablation study in Appendix E. 
- The construction of $x_c$ is based on our proposed method that leverages in-context learning for interpretation, which effectively controls the model's reasoning ability while not affecting its other capabilities. As shown in Table 2, when we replace the reasoning process in the few-shot example with irrelevant sentences, the model no longer outputs the reasoning process but still produces the answer.

> ***Reference***
>
> [1] Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. In ICLR 2023.
> 
> [2] Localizing Model Behavior With Path Patching. In ArXiv 2023.
> 
> [3] How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. In ArXiv 2023.

Reviewer comment (Experiments):
There are specific concerns about the construction of $x_c$: (i) a simple adversarial change in the prompt could significantly decrease the accuracy; (ii) removing CoT reasoning demonstrations in $x_c$ could cause the LLM to default to non-CoT reasoning, explaining the accuracy drop; (iii) alternative counterfactual designs, such as adding incorrect but relevant CoT demonstrations, could be more informative.

Author response:
***Ans for Q1.2):*** 
- i) The impact of adversarial examples on results differs from the effects of causal interventions. Adversarial examples investigate the robustness of the model, whereas our approach based on causal interventions examines the causal relationship between model components and specific outputs [2]. We would like to note that exploring adversarial robustness is also a promising direction, while we mainly focus on interpreting LLMs in this work. Therefore, we mainly focus on interpreting LLMs through a causal-effect view. 

- ii) We agree with your observation that $x_c$ does not trigger the ability, while $x_r$ does. This is consistent with the motivation to construct these pairs for locating the model's capabilities [1]. Aligning with your insights, our experiments employed $x_c$ that causes limited performance degeneration, i.e., $x_c$ causes a limited drop in the accuracy. Specifically, the accuracy on $x_r$ is $59.5\%$, while the accuracy on $x_c$ is $57.25\%$ (test on 1200 cases of CSQA using LLaMA2-7B).

- iii) We appreciate your valuable suggestion. Accordingly, we conduct experiments using the mentioned approach. In our experiments, this approach fails to exhibit sparsity in identifying the relevant heads. Moreover, the knockout of these heads fails to cause a significant impact on the model's CoT reasoning ability, consistent with the results produced by the random approach. We believe this is a valuable exploration, and thus we have provided additional results in **Section 4.4** and **Appendix E**. These results include multiple sets of different methods for constructing $x_r, x_c$.

> ***Reference***
>
> [1] Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. In ICLR 2023.
> 
> [2] Localizing Model Behavior With Path Patching. In ArXiv 2023.
> 
> [3] How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. In ArXiv 2023.

Reviewer comment (Evaluation):
The score/importance of each attention head is determined by the accuracy drop when substituting its output with the corresponding activations from the counterfactual examples. Since $x_r$ and $x_c$ could differ significantly, the paper does not justify whether direct replacement of activations could itself have deteriorating effects on the LLM due to completely blocking the information flow, making the paper’s conclusions harder to justify.

Author response:
***Ans for Q2):***

We apologize for the misunderstanding, which may results from our insufficient explanation. To address this issue and enhance clarity, we have added the following explanations.


- We have provided a further explanation in Section 3.1 in the revised paper (highlighted in blue). Specifically, the key modifications in Section 3.1 are listed below: 

*Path patching is a method for localizing the key components of LLMs for accomplishing specific tasks or behaviors. The term "behavior" is defined within the context of input-output pairs on a specific dataset, thereby effectively reflecting the model's specific capabilities. To assess the significance of a model component for a particular behavior, a causal intervention is implemented to interfere with the behavior. Assuming that the component holds importance, the causal effect on the model's specific ability should be notable. The causal intervention applied to the model component is achieved through the creation of appropriate reference examples (REs) and counterfactual examples (CEs), which solely vary in their capacity to trigger the model's specific behavior.*

- Additionally, we have included a more comprehensible method framework diagram (Figure 1) to explain our methodology.

- Regarding the operation of replacing activation, we clarify that it involves replacing the output of a specific head with the output obtained when the input is $x_c$. This operation does not block the information flow but rather replaces key information (while keeping the other information unchanged), which is verified on other tasks like "Indirect Object Identification" [1], and "greater than" [2]. Moreover, the experiment in **Section 4.2** shows that replacing the outputs of random attention heads almost has no effect on the models' CoT reasoning ability, indicating the importance of the identified attention heads.

> ***Reference***
>
> [1] Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. In ICLR 2023.
> 
> [2] How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. In ArXiv 2023.

Reviewer comment (Novelty):
The proposed method does not seem to have specific designs for understanding CoT.

Author response:
**Q3**: Method issue:
> "The proposed method does not seem to have specific designs for understanding CoT."

***Ans for Q3):***
We apologize for the misunderstanding. Accordingly, we would like to highlight the novelty and contribution as follows. 

- We have added more explanations (in Section 3.1) to highlight the challenges and two key approaches to address the challenges, aiming to avoid similar issues. Specifically, we propose to leverage in-context learning to construct $x_r, x_c$ (in Section 3.2), and propose a novel metric to evaluate the causal effect, i.e., the proposed Word-of-Interest (WoI) Norm (in Section 3.3). The challenges and our proposed methods in Section 3.1 are shown below, and we believe these changes would make our paper more clear.
<font color=blue style="font-family: 'Times New Roman';">In this work, we aim to localize the key attention head within the model responsible for CoT reasoning. However, we are faced with two primary challenges: i) The complexity arises from the diverse range of abilities that LLMs possess to achieve CoT reasoning, including numerical computation, knowledge retrieval, and logical reasoning, making it challenging to design reference examples (REs) that are accurately paired with counterfactual examples (CEs) differing only in their ability to trigger the CoT reasoning behavior of LLMs. ii) Additionally, the extensive potential word in the LLM's output results in a sparsity of causal effects, as the words directly related to CoT reasoning are significantly limited in number. To overcome these challenges, we propose an innovative approach for constructing paired REs and CEs, accompanied by a word-of-interest (WoI) normalization method aimed at addressing the issue of sparse causal effects.</font>

- We would like to highlight that i) it is challenging to interpret the CoT reasoning ability, accordingly, we propose to leverage path patching to address the challenge; ii) it is challenging to directly employ path patching for the task, because it is unclear how to construct $x_r, x_c$ to realize path patching, accordingly, we propose an in-context learning approach and a novel approach to realize it; iii) it is challenging to locate key heads even using path patching, accordingly, we propose a novel normalization approach to address the challenge. We believe these explorations are specific designs for understanding CoT.   

- Inspired by your valuable comments, we have conducted numerous experiments in data construction and found that utilizing in-context learning is an effective method. Please refer to Section 4.4 for more details about our other attempts to design $x_c$ based on in-context learning. In addition to the in-context learning method, we also attempted to use the zero-shot CoT prompt to design $x_r, x_c$. However, the heads identified through this method are not the key heads for CoT reasoning since knocking out these heads shows little effect on the models' reasoning ability. Detailed results can be found in Appendix E.

Reviewer comment (Experiments):
There is a question about whether directly replacing the activations of certain attention heads could have deteriorating effects on the LLM.

Author response:
**Q4**: Worry about the interference with the attention heads:
> "Will directly replacing the activations of certain attention heads have deteriorating effects on the LLM?"

***Ans for Q4):***
Thank you for your valuable question. 
- To validate your viewpoint, we have performed random perturbations on the heads, as shown in **Figure 3**. The random perturbations on random-selected heads have almost no impact on the CoT reasoning capability of the model while perturbing the key heads significantly affects the model's performance. Please refer to **Section 4.2 Validation of key heads** for more detailed information. 
- Inspired by your question, we plan to include the results of model performance on common NLP evaluation tasks (such as MMLU [1]) when knocking out the key heads. We will add it to our work when we complete the experiments.

> ***Reference***
>
> [1] Measuring Massive Multitask Language Understanding. In ICLR 2021.

Reviewer comment (Writing):
The presentation is at times unclear (see the questions section). I found it quite hard to read and had to spend some time understanding their methodology.

Author response:
**Q1**: Presentation issues:
> "The presentation is at times unclear (see the questions section). I found it quite hard to read and had to spend some time understanding their methodology."

***Ans for Q1):*** 
Inspired by your valuable comments, we have made the following improvements/explanations:

**a)** We have adjusted the misleading Table 1,2/4,5, and the specific modifications will be explained in detail in ***Ans for Q6)***.

**b)** We have redrawn the methodology framework diagram (Figure 1) and included it in the main text instead of the appendix.

**c)** We have made adjustments to the content of the **Method** section in the main text and highlighted the specific changes in blue. We would appreciate it if you could find some time to review our revised version paper, which incorporates these changes.

Specifically, in order to enhance the clarity of the methodology, we have reorganized the **Method** section. We now start by introducing the core framework of the path patching method and have added an introduction to path patching in Sec.3.1:


*Path patching is a method for localizing the key components of LLMs for accomplishing specific tasks or behaviors. The term "behavior" is defined within the context of input-output pairs on a specific dataset, thereby effectively reflecting the model's specific capabilities. To assess the significance of a model component for a particular behavior, a causal intervention is implemented to interfere with the behavior. Assuming that the component holds importance, the causal effect on the model's specific ability should be notable. The causal intervention applied to the model component is achieved through the creation of appropriate reference examples (REs) and counterfactual examples (CEs), which solely vary in their capacity to trigger the model's specific behavior.*

and supplement the challenges of interpreting LLMs' CoT reasoning ability using path patching in Sec.3.1:

*In this study, we aim to localize the key attention head within the model responsible for CoT reasoning. However, we are faced with two primary challenges: i) The complexity arises from the diverse range of abilities that LLMs possess to achieve CoT reasoning, including numerical computation, knowledge retrieval, and logical reasoning, making it challenging to design reference examples (REs) that are accurately paired with counterfactual examples (CEs) differing only in their ability to trigger the CoT reasoning behavior of LLMs. ii) Additionally, the extensive potential word in the LLMs' output results in a sparsity of causal effects, as the words directly related to CoT reasoning are significantly limited in number. To overcome these challenges, we propose an innovative approach for constructing paired REs and CEs, accompanied by a word-of-interest (WoI) normalization method aimed at addressing the issue of sparse causal effects.*

This sets the stage for the subsequent discussion of our method's innovations, namely the $x_r, x_c$ construction, and causal effect evaluation metric design.

Reviewer comment (Novelty):
The literature review section could be more comprehensive: a number of other articles on interpretability rooted in causal interventions exist on models of similar sizes, such as [2,3], the latter also using a 7B model but not being cited. I would recommend the authors contrast the existing approaches in the "_Related Work_" section or an appendix to that section, such as from [3] — but there are also other articles — with their own and comment upon similarities and differences so that the reader is well-informed of how their methods compete with existing ones.

Author response:
**Q2**: Literature review issues:
> "the literature review section could be more comprehensive: a number of other articles on interpretability rooted in causal interventions exist on models of similar sizes, such as [2,3], the latter also using a 7B model but not being cited. I would recommend the authors contrast the existing approaches in the "Related Work" section or an appendix to that section, such as from [3] -but there are also other articles- with their own and comment upon similarities and differences so that the reader is well-informed of how their methods compete with existing ones.
the improvements are nice but somewhat incremental since a single in-context learning technique is analyzed on just two models.


***Ans for Q2):*** 
Thank you for providing these outstanding related works. Based on your suggestions, we have incorporated citations to these papers in our paper and discussed them in Sec.2 Background as follows: 

*[1] takes the first step toward understanding the working mechanism of the 7B-sized large language model. Method Distributed Alignment Search (DAS) [2] based on causal abstraction is applied to align the language model with a hypothesized causal model.*

*Due to the complexity of CoT reasoning tasks, it is intricate to design a unified symbolic causal model for multi-step reasoning tasks. In this work, employing the path patching method from [3,4], we successfully identify the crucial attention heads in LLMs responsible for CoT reasoning ability.*

We have added these contents to Section 2 of our revised paper and modifications are highlighted.

> ***Reference***
>  
> [1] Interpretability at Scale: Identifying Causal Mechanisms in Alpaca. In ArXiv 2023. 
>
> [2] Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations. In NeurIPS 2023.
>
> [3] How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. In ArXiv 2023.
> 
> [4] Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. In ICLR 2023.

Reviewer comment (Novelty):
The improvements are nice but somewhat incremental since a single in-context learning technique is analyzed on just two models.

Author response:
**Q3**: Experiment issues:
> "the improvements are nice but somewhat incremental since a single in-context learning technique is analyzed on just two models."

***Ans for Q3):*** 
We really appreciate your feedback. It is important to support our points with comprehensive experiments. In response to your comments, we have supplemented two additional templates (**Section 4.4 template 1&2**) for constructing counterfactual data adopting in-context learning. The key heads identified using these templates are shown in Figure 5b and c, which are similar to the heads identified using the template we originally proposed (Fig. 5a). 

Additionally, inspired by your valuable suggestion, we have explored using zero-shot CoT to construct $x_r, x_c$ instead of in-context learning (Appendix E). However, we find that the identified heads using this approach are not crucial for the model's reasoning capability, as excluding these heads does not significantly impact the model's reasoning performance (please refer to Appendix E for more results).

Furthermore, drawing inspiration from your suggestion, we conducted another interesting experiment. We noticed that the same key head (Head 18.30 for LLaMA2-7B) is identified on the CSQA and AQuA datasets, but not on the StrategyQA dataset (see Figure 2). The difference between these datasets is that $x_r, x_c$ of CSQA and AQuA have multiple-choice question formats, whereas those of StrategyQA have a true or false question format. Therefore, we converted the $x_r, x_c$ on the StrategyQA dataset into the multiple-choice format. (Please refer to **Template 3** in Section 4.4 for specific examples.) 

As a result, we identified the same key head (Head 18.30) as on CSQA and AQuA datasets (see Fig. 5a, d). Further analysis and details are presented in Section 4.4.

Reviewer comment (Presentation):
Their methodology might also be improved by the use of diagrams to show in a single glance all the relevant information.

Author response:
**Q4**: Methodology issues:
> "their methodology might also be improved by the use of diagrams to show in a single glance all the relevant information."

***Ans for Q4):*** 
Thanks for your suggestion. We have redrawn the method figure and moved it to **Section 3** (**Figure 1**).

Reviewer comment (Presentation):
It is somewhat strange that Appendix B ("_Reference Data Examples_") is empty; why include it in that case?

Author response:
**Q5**: Typo issues:
> "it's somewhat strange that Appendix B ("Reference Data Examples") is empty; why include it in that case?"

***Ans for Q5):*** 
Thanks for your suggestion and careful review. We have fixed this typo problem.

Reviewer comment (Presentation):
Tables 1 and 2: A reader might at first be confused whether what is shown is the complete few-shot that is supplied to the model that is to be tested (e.g., LLaMA2) to facilitate in-context learning, or if what follows after the "A" is the answer of LLaMA2/Qwen. That the former is correct transpires indirectly only from the text: "_The outputs of LLaMA-7B and Qwen-7B on the counterfactual data are shown in Table 4 and 5, respectively._" I would recommend adding at least a caption here, explaining more clearly what can be seen, without having to look in the text for the meaning of the table.

Author response:
**Q6**: Table issues:
> "Tables 1, 2 / 4, 5 are unclear and suffer from a number of issues:
> 1. Table 1, 2: A reader might at first be confused whether what is shown is the complete few-shot that is supplied to the model that is to be tested (e.g., LLaMA2) to facilitate in-context learning - or if what follows after the "A" is the answer of LLaMA2/Qwen? That the former is correct transpires indirectly only from the text: "The outputs of LLaMA-7B and Qwen-7B on the counterfactual data are shown in Table 4 and 5, respectively." I would recommend adding at least a caption here, explaining more clearly what can be seen, without having to look in the text for the meaning of the table.
> 2. In Table 1,2, the last question is "Can Reiki be stored in a bottle?", which led me to believe that this is the question the model should have answered (once with the correct in-context learning text and once with the modified/colored text as indicated from table 2). But a look in the appendix at Table 4 suddenly reveals a different last question, "Would a rabbi worship martyrs Ranavalona I killed?" (all else being the same), which I found confusing. Can the authors explain this?
> 3. minor formatting: "Q" and "A" from Table 1,2 are set in bold but not in Table 4,5, which does not aid readability.
> 4. since these tables seem an essential part of the paper, as the capture the methodology, it is tiring to go back and forth between Tables 1,2 and the Appendix; I would propose to move all Tables to the main body to aid readability."

***Ans for Q6):*** 
Thanks for your careful review. According to your valuable suggestion, we have moved Table 1 and Table 2 from the appendix to Section 3.2. To avoid misleading, each table now includes two parts, one for the model's input sentences and the other for the output sentences. Additionally, we have standardized the questions used for $x_r, x_c$ in Table 1 and Table 2 with the questions from the CSQA dataset. Due to space limitations, we have relocated the examples of $x_r, x_c$ on the StrategyQA and AQuA datasets to Table 3 and Table 4 in Appendix B, and they are labeled in the same format as Table 1 & 2 with model input and output. Concerning the formatting issue with bolding "Q" and "A", we have made the necessary corrections.

Reviewer comment (Presentation):
In Tables 1 and 2, the last question is "_Can Reiki be stored in a bottle?_", which led me to believe that this is the question the model should have answered (once with the correct in-context learning text and once with the modified/colored text as indicated from Table 2). But a look in the appendix at Table 4 suddenly reveals a different last question, "_Would a rabbi worship martyrs Ranavalona I killed?_" (all else being the same), which I found confusing. Can the authors explain this?

Author response:
**Q6**: Table issues:
> "Tables 1, 2 / 4, 5 are unclear and suffer from a number of issues:
> [...]
> 2. In Table 1,2, the last question is "Can Reiki be stored in a bottle?", which led me to believe that this is the question the model should have answered (once with the correct in-context learning text and once with the modified/colored text as indicated from table 2). But a look in the appendix at Table 4 suddenly reveals a different last question, "Would a rabbi worship martyrs Ranavalona I killed?" (all else being the same), which I found confusing. Can the authors explain this?
> [...]"

***Ans for Q6):*** 
Thanks for your careful review. According to your valuable suggestion, we have moved Table 1 and Table 2 from the appendix to Section 3.2. To avoid misleading, each table now includes two parts, one for the model's input sentences and the other for the output sentences. Additionally, we have standardized the questions used for $x_r, x_c$ in Table 1 and Table 2 with the questions from the CSQA dataset. Due to space limitations, we have relocated the examples of $x_r, x_c$ on the StrategyQA and AQuA datasets to Table 3 and Table 4 in Appendix B, and they are labeled in the same format as Table 1 & 2 with model input and output. Concerning the formatting issue with bolding "Q" and "A", we have made the necessary corrections.

Reviewer comment (Presentation):
Minor formatting: "Q" and "A" from Tables 1 and 2 are set in bold but not in Tables 4 and 5, which does not aid readability.

Author response:
**Q6**: Table issues:
> "[...]
> 3. minor formatting: "Q" and "A" from Table 1,2 are set in bold but not in Table 4,5, which does not aid readability.
> [...]"

***Ans for Q6):*** 
Thanks for your careful review. According to your valuable suggestion, we have moved Table 1 and Table 2 from the appendix to Section 3.2. To avoid misleading, each table now includes two parts, one for the model's input sentences and the other for the output sentences. Additionally, we have standardized the questions used for $x_r, x_c$ in Table 1 and Table 2 with the questions from the CSQA dataset. Due to space limitations, we have relocated the examples of $x_r, x_c$ on the StrategyQA and AQuA datasets to Table 3 and Table 4 in Appendix B, and they are labeled in the same format as Table 1 & 2 with model input and output. Concerning the formatting issue with bolding "Q" and "A", we have made the necessary corrections.

Reviewer comment (Presentation):
Since these tables seem an essential part of the paper, as they capture the methodology, it is tiring to go back and forth between Tables 1 and 2 and the Appendix. I would propose to move all tables to the main body to aid readability.

Author response:
**Q6**: Table issues:
> "[...]
> 4. since these tables seem an essential part of the paper, as the capture the methodology, it is tiring to go back and forth between Tables 1,2 and the Appendix; I would propose to move all Tables to the main body to aid readability."

***Ans for Q6):*** 
Thanks for your careful review. According to your valuable suggestion, we have moved Table 1 and Table 2 from the appendix to Section 3.2. To avoid misleading, each table now includes two parts, one for the model's input sentences and the other for the output sentences. Additionally, we have standardized the questions used for $x_r, x_c$ in Table 1 and Table 2 with the questions from the CSQA dataset. Due to space limitations, we have relocated the examples of $x_r, x_c$ on the StrategyQA and AQuA datasets to Table 3 and Table 4 in Appendix B, and they are labeled in the same format as Table 1 & 2 with model input and output. Concerning the formatting issue with bolding "Q" and "A", we have made the necessary corrections.