Paper: DQ-LoRe: Dual Queries with Low Rank Approximation Re-ranking for In-Context Learning (ICLR-2024)

Reviewer comment (Novelty):
Limited Originality: While the paper introduces the DQ-LoRe framework, the methods employed (such as BM25, PCA dimensionality reduction, etc.) are based on prior research. Although the authors have adeptly integrated and applied these methods within their framework, from an innovation standpoint, these techniques are not novel in themselves.

Author response:
Thank you for your constructive feedback. We would like to address some misunderstandings highlighted in the paper and have incorporated additional experiments based on your recommendations to further substantiate the efficacy of our approach.

To begin with, we need to clarify that BM25 is not employed in our principal methodology, DQ-LoRe, as delineated in our publication. Its use is confined to the training phase for generating positive and negative sample pairs. 

As illustrated in Figure 1, the second query retrieval using the smaller model employs the encoder’s output, which is trained on CoT+Question. In this stage, we compute the embedding of the current CoT (derived from the initial query) combined with the Question (from the current test set). We then identify the $M$ samples from the training set’s CoT+Question embeddings that most closely resemble the current test sample's question and Chain-of-Thought. Here, $M$ exceeds the final number of exemplars, $N$. This is because we have observed significant spurious correlation issues in methods relying on the initial retrieval, such as EPR. Therefore, within the initial set of $M$ samples, we seek to eliminate this portion through denoising techniques. We achieve this by employing PCA as a denoising method to project these $M$ samples into a lower-dimensional space and then re-ranking them to obtain the final set of $N$ exemplars, thus mitigating the interference from spurious correlation samples. **Notably, BM25 is not used in this initial sorting phase during inference.** 

While our framework builds upon existing technologies, it stems from a conceptually robust idea: enhancing the precision of the $M$ nearest exemplars (sourced from the dual-query process with embeddings encapsulating both Question and Chain-of-Thought data) by transposing them into a novel vector space and re-ranking them through denoising to pinpoint the Top $N$ most pertinent exemplars. This refinement is vital for improved in-context learning. 

The integration of other technologies is pivotal to this overarching concept. For example, we utilized PCA for enhanced sample differentiation. Although Gaussian kernel methods could theoretically be applied for dimensionality expansion in high-dimensional space to separate linearly inseparable samples, they did not produce satisfactory outcomes in our experiments.

Moreover, to guarantee the comprehensiveness and sufficiency of the embedding data before implementing PCA, we initially query the larger model with a combination of the initial exemplar and the current question to gather a priori information. Subsequently, we merge the CoT and question to query the smaller model. Therefore, the preliminary querying phase with the larger model aims to procure the CoT, ensuring that the embeddings derived from the second query to the smaller model are rich in information, encompassing both CoT and Question elements. This rationale underpins our Dual Query methodology.

Reviewer comment (Experiments):
Methodological Concerns: The first round of sorting in the paper uses BM25 to match the question with the question in the exemplar, aiming to find the exemplar most relevant to the question. The second round aims to find the Chain-of-Thought (CoT) most relevant to the question. This raises a question: Why not directly compute the similarity between the CoT in the exemplar and the CoT of the question itself? Instead, the approach seems to add unnecessary complexity by opting to compute scores for the generated CoT.

Author response:
Thank you for your constructive feedback. 

Firstly, we wish to clarify that BM25 was not utilized for the first-round ranking to obtain sorted samples. Its application was limited to the encoder training phase, where it played a crucial role in constructing positive and negative training samples.

In a similar vein, the computation of scores for the generated Chain-of-Thought (CoT) was instrumental in encoder training, aiding in the creation of positive and negative samples. This scoring mechanism was not employed during the inference stage for re-ranking purposes.

During training, constrained by resources, we leveraged BM25 to select a narrowed subset of samples for scoring. This scoring process was integral to forming the definitive positive and negative samples. More specifically, in the encoder's training, BM25 facilitated a preliminary screening to identify the top n sentences from examples semantically akin to the input question. These chosen sentences, along with the question, were then presented to a smaller model. The confidence level of the smaller model’s response served as an indicator of the sentence’s usefulness in addressing the question.

Post contrastive learning training, our encoder adeptly clusters the most beneficial examples for a question in a high-dimensional space, closely aligning them with the question.

**It's important to note that these techniques were exclusive to the encoder training phase and were not a part of the two-stage ranking process during inference.**

In the initial ranking phase, we utilized encoded representations of the current test sample's question combined with its CoT (from the first query). These were sorted based on their similarity to the encoded representations of question + CoT from all training samples, integrating both CoT and question information.

Furthermore, we embarked on experiments that concentrated exclusively on the similarity between the CoTs of the current test sample and those of the training samples. We also expanded our scope to include additional ablation experiments, which aimed to dissect and understand the distinct contributions of question information and Chain-of-Thought (CoT) information. This was especially pertinent in the context of employing LoRe.

Table 1: Comparing the Use of Trained Encoder and Solely CoT Similarity Retrieval.

|        | w/o trainning |  with trainning  |
|  ----  | :----:  | :----:  |
| question + CoT  | 77.7 | 78.9 |
| question + CoT with LoRe  | 78.8 | 80.7 | 
| only CoT  | 76.1 | 79.4 | 
| only CoT with LoRe  | 78.1 | 80.4 |


* The "w/o training" denotes the direct use of an untrained BERT for encoding sample representations.
* The "with training" denotes the use of a BERT model trained on a training set for encoding sample representations.
* The "question + CoT" denotes the simultaneous use of both question information and the information from CoT.
* The 'only CoT'  denotes to exclusively using 'CoT' information to encode sample representations. **This corresponds to directly computing the similarity between the CoT in the exemplar and the CoT of the question itself.**
* The 'LoRe' denotes the low-rank approximation re-ranking algorithm that we have proposed.

In this experiment, we investigate the effects of training the encoder, not training the encoder, and employing single-round and two-round ranking on the final results. By comparing the 'without training' and 'with training' settings, it has been observed that training the encoder on specific datasets significantly enhances the retrieval task performance in certain datasets. This is particularly evident in the 'only CoT' setting, where there is a notable performance difference of nearly *3.3%*. Furthermore, our approach exhibits robustness, as it achieves consistent improvements even without employing an encoder trained on a specific dataset. Especially in the 'only CoT with LoRe' setting under 'w/o training', we achieve an absolute performance improvement of *2%* without specifically training the encoder. Finally, we discovered that using only CoT similarity yields inferior results compared to utilizing similarity based on both Question and CoT.

Reviewer comment (Experiments):
Experiments Could Be More Robust: While the authors perform commendably in comparative experiments, the ablation studies appear to be lacking. For instance, it would be beneficial to compare the effects of two rounds of sorting versus a single round, and the effects of PCA dimensionality reduction versus no reduction. Such experiments might offer readers more insights into the efficacy of the methods.

Author response:
Thank you for your valuable suggestions. As per your recommendations, we have conducted additional comparative experiments, which we outline below.
To begin, we compared the impact of various re-ranking methods. 'w/o re-ranking' serves as our baseline, where no re-ranking was performed. 'Longest' represents a re-ranking approach that extends the Complex-CoT method. In this method, exemplars are selected for the final inference step based on the longest 8 CoT exemplars from the initial retrieval of 16 or 32 exemplars (the selection of 16 or 32 exemplars is a tunable hyperparameter).
In addition to these, we conducted experiments employing BERT-Whitening[1], MGS (Modified Gram-Schmidt Process)[2], and Gaussian Kernel[3] for re-ranking. The detailed experimental results are presented in the table below:

Table 2: Comparison of Different Inference LLMs and Ablation of Re-Ranking Algorithms 
|    Re-ranking method    | Text-Davinci-003 | GPT3.5-Turbo-16k
|  ----  | :----:  | :----:  |
| w/o re-ranking  |67.9| 78.9 |
| Longest  |68.1| 78.9 |
| BERT-Whitening  |67.9| 78.9 |
| MGS  |68.2| 79.7 | 
|Gaussian Kernel  |66.7| 79.0 |
| PCA  |69.1| 80.7 |


As illustrated in the table above, we establish a baseline by omitting the re-ranking step and employing a single-round ranking model. Subsequently, we undertake a comprehensive analysis of the impact of employing various algorithms for the second-round ranking. It is noteworthy that only two methods, namely MGS and PCA, exhibit a discernible impact on the re-ranking process. This suggests that the effectiveness of the re-ranking process is attributed not only to dimensionality reduction of vectors but also to the orthogonalization of vectors, both playing significant roles.
In contrast, kernel tricks such as the Gaussian Kernel do not appear to contribute significantly to the sample distinction in this context. Notably, in the case of text-davinci-003, we observed a performance decline of 1.2%.

Table 3: Experiments with Different Encoders and Ablation of LoRe Across Baselines
|    Method    | BERT-base-uncased | RoBERTa-base | 
|  ----  | :----:  | :----:  |
| EPR |77.3| 78.0 |
| EPR + LoRe  | 77.0| 77.8 |
| CEIL  |79.4| 77.7 |
| CEIL + LoRe  |78.7 | 77.4 | 
| Dq-LoRe w/o LoRe | 78.9 | 76.4 |
| Dq-LoRe  |80.7| 78.8 |

In the second experiment presented in the table, we investigate the effects of utilizing different base models on the experimental outcomes. Our findings indicate that, in terms of overall performance, the bert-base-uncased model outperforms the RoBERTa-base model, except for an enhancement observed in EPR when employing the RoBERTa-base.
Furthermore, we assess the influence of LoRe on various models and note that the adoption of LoRe leads to a decline in performance for both EPR and CEIL. This decline can be primarily attributed to the absence of the Dual Query process, which hampers the encoding of additional Chain-of-Thought (CoT) information.

**References:**

>[1] Su, J., Cao, J., Liu, W., & Ou, Y. (2021). Whitening sentence representations for better semantics and faster retrieval. arXiv preprint arXiv:2103.15316.

>[2] Longley, J. W. (1981). Modified gram-schmidt process vs. classical gram-schmidt: Modified gram-schmidt process vs. classical gram-schmidt. Communications in Statistics-Simulation and Computation, 10(5), 517-527.

>[3] Scholkopf, B., Sung, K. K., Burges, C. J., Girosi, F., Niyogi, P., Poggio, T., & Vapnik, V. (1997). Comparing support vector machines with Gaussian kernels to radial basis function classifiers. IEEE transactions on Signal Processing, 45(11), 2758-2765.

Reviewer comment (Presentation):
Detailing Issues: Figure 1 in the paper has errors. Specifically, in the "M Q&A Low rank embedding" section, "Embedding 1" appears twice, which might lead to confusion. Authors should ensure that all charts and graphics are accurate and clear to avoid any potential misunderstandings.

Author response:
Thank you for pointing out these mistakes, we will revise them accordingly in the rebuttal version, which will be submitted later this week.

Reviewer comment (Experiments):
Methodological Clarifications: What was the rationale behind opting for two rounds of sorting (first for matching the question with the exemplar and second for finding the relevant CoT) instead of directly computing the similarity between the CoT in the exemplar and the CoT of the question?

Author response:
Thank you for your constructive feedback. 

Firstly, we wish to clarify that BM25 was not utilized for the first-round ranking to obtain sorted samples. Its application was limited to the encoder training phase, where it played a crucial role in constructing positive and negative training samples.

In a similar vein, the computation of scores for the generated Chain-of-Thought (CoT) was instrumental in encoder training, aiding in the creation of positive and negative samples. This scoring mechanism was not employed during the inference stage for re-ranking purposes.

During training, constrained by resources, we leveraged BM25 to select a narrowed subset of samples for scoring. This scoring process was integral to forming the definitive positive and negative samples. More specifically, in the encoder's training, BM25 facilitated a preliminary screening to identify the top n sentences from examples semantically akin to the input question. These chosen sentences, along with the question, were then presented to a smaller model. The confidence level of the smaller model’s response served as an indicator of the sentence’s usefulness in addressing the question.

Post contrastive learning training, our encoder adeptly clusters the most beneficial examples for a question in a high-dimensional space, closely aligning them with the question.

**It's important to note that these techniques were exclusive to the encoder training phase and were not a part of the two-stage ranking process during inference.**

In the initial ranking phase, we utilized encoded representations of the current test sample's question combined with its CoT (from the first query). These were sorted based on their similarity to the encoded representations of question + CoT from all training samples, integrating both CoT and question information.

Furthermore, we embarked on experiments that concentrated exclusively on the similarity between the CoTs of the current test sample and those of the training samples. We also expanded our scope to include additional ablation experiments, which aimed to dissect and understand the distinct contributions of question information and Chain-of-Thought (CoT) information. This was especially pertinent in the context of employing LoRe.

Table 1: Comparing the Use of Trained Encoder and Solely CoT Similarity Retrieval.

|        | w/o trainning |  with trainning  |
|  ----  | :----:  | :----:  |
| question + CoT  | 77.7 | 78.9 |
| question + CoT with LoRe  | 78.8 | 80.7 | 
| only CoT  | 76.1 | 79.4 | 
| only CoT with LoRe  | 78.1 | 80.4 |


* The "w/o training" denotes the direct use of an untrained BERT for encoding sample representations.
* The "with training" denotes the use of a BERT model trained on a training set for encoding sample representations.
* The "question + CoT" denotes the simultaneous use of both question information and the information from CoT.
* The 'only CoT'  denotes to exclusively using 'CoT' information to encode sample representations. **This corresponds to directly computing the similarity between the CoT in the exemplar and the CoT of the question itself.**
* The 'LoRe' denotes the low-rank approximation re-ranking algorithm that we have proposed.

In this experiment, we investigate the effects of training the encoder, not training the encoder, and employing single-round and two-round ranking on the final results. By comparing the 'without training' and 'with training' settings, it has been observed that training the encoder on specific datasets significantly enhances the retrieval task performance in certain datasets. This is particularly evident in the 'only CoT' setting, where there is a notable performance difference of nearly *3.3%*. Furthermore, our approach exhibits robustness, as it achieves consistent improvements even without employing an encoder trained on a specific dataset. Especially in the 'only CoT with LoRe' setting under 'w/o training', we achieve an absolute performance improvement of *2%* without specifically training the encoder. Finally, we discovered that using only CoT similarity yields inferior results compared to utilizing similarity based on both Question and CoT.

Reviewer comment (Experiments):
Experimental Details: In the ablation studies, could the authors provide results comparing the effects of two rounds of sorting versus a single round? Additionally, it would be beneficial to see the effects of PCA dimensionality reduction versus no reduction. How do these individual components impact the overall performance?

Author response:
Thank you for your constructive feedback. 

Firstly, we wish to clarify that BM25 was not utilized for the first-round ranking to obtain sorted samples. Its application was limited to the encoder training phase, where it played a crucial role in constructing positive and negative training samples.

In a similar vein, the computation of scores for the generated Chain-of-Thought (CoT) was instrumental in encoder training, aiding in the creation of positive and negative samples. This scoring mechanism was not employed during the inference stage for re-ranking purposes.

During training, constrained by resources, we leveraged BM25 to select a narrowed subset of samples for scoring. This scoring process was integral to forming the definitive positive and negative samples. More specifically, in the encoder's training, BM25 facilitated a preliminary screening to identify the top n sentences from examples semantically akin to the input question. These chosen sentences, along with the question, were then presented to a smaller model. The confidence level of the smaller model’s response served as an indicator of the sentence’s usefulness in addressing the question.

Post contrastive learning training, our encoder adeptly clusters the most beneficial examples for a question in a high-dimensional space, closely aligning them with the question.

**It's important to note that these techniques were exclusive to the encoder training phase and were not a part of the two-stage ranking process during inference.**

In the initial ranking phase, we utilized encoded representations of the current test sample's question combined with its CoT (from the first query). These were sorted based on their similarity to the encoded representations of question + CoT from all training samples, integrating both CoT and question information.

Furthermore, we embarked on experiments that concentrated exclusively on the similarity between the CoTs of the current test sample and those of the training samples. We also expanded our scope to include additional ablation experiments, which aimed to dissect and understand the distinct contributions of question information and Chain-of-Thought (CoT) information. This was especially pertinent in the context of employing LoRe.

Table 1: Comparing the Use of Trained Encoder and Solely CoT Similarity Retrieval.

|        | w/o trainning |  with trainning  |
|  ----  | :----:  | :----:  |
| question + CoT  | 77.7 | 78.9 |
| question + CoT with LoRe  | 78.8 | 80.7 | 
| only CoT  | 76.1 | 79.4 | 
| only CoT with LoRe  | 78.1 | 80.4 |


* The "w/o training" denotes the direct use of an untrained BERT for encoding sample representations.
* The "with training" denotes the use of a BERT model trained on a training set for encoding sample representations.
* The "question + CoT" denotes the simultaneous use of both question information and the information from CoT.
* The 'only CoT'  denotes to exclusively using 'CoT' information to encode sample representations. **This corresponds to directly computing the similarity between the CoT in the exemplar and the CoT of the question itself.**
* The 'LoRe' denotes the low-rank approximation re-ranking algorithm that we have proposed.

In this experiment, we investigate the effects of training the encoder, not training the encoder, and employing single-round and two-round ranking on the final results. By comparing the 'without training' and 'with training' settings, it has been observed that training the encoder on specific datasets significantly enhances the retrieval task performance in certain datasets. This is particularly evident in the 'only CoT' setting, where there is a notable performance difference of nearly *3.3%*. Furthermore, our approach exhibits robustness, as it achieves consistent improvements even without employing an encoder trained on a specific dataset. Especially in the 'only CoT with LoRe' setting under 'w/o training', we achieve an absolute performance improvement of *2%* without specifically training the encoder. Finally, we discovered that using only CoT similarity yields inferior results compared to utilizing similarity based on both Question and CoT.

Reviewer comment (Novelty):
The return on Investment (ROI) of the method is not super high. The method outperforms SOTA accuracy by 1.7 at the cost of complex implementation. The method involves an additional query of the LLM, a retrieving step, a dimensionality reduction step, and a re-ranking step. One might find it hard to justify the complexity with the 1.7 point improvement.

Author response:
Exactly what we want is to propose a flexible "component" designed to seamlessly integrate with most methods. The experiments in our paper are orthogonal to works such as EPR and CEIL. We conducted ablation studies based on their methodologies to demonstrate a substantial improvement relative to the baselines. In practical applications using GPT-4 on the GSM8K dataset, we achieved a 2.9-point improvement, elevating performance from 91.3 to 94.2. In Table 3 below, we migrate our method to CEIL and conduct ablation experiments on each step to provide insights and address concerns related to the “The return on Investment (ROI) of the method is not super high.”.

Table 3: Ablation Experiments on Individual Components in EPR and CEIL
|   Method    | GPT3.5-Turbo-16k | GPT-4| 
|  ----  | :----:  | :----:  |
| EPR  |77.3| 91.3 |
| EPR + DQ |78.3| 93 |
| EPR + DQ-LoRe  |80.7| 94.2 |
| CEIL  |79.4| 92.5 |
| CEIL + LoRe  |78.7| 92 |
| CEIL + DQ  |79.3| 92.3 |
| CEIL + DQ-LoRe  |79.9| 94.1 |
   * The "Method + DQ" denotes that we use the Dual Queries and simultaneous use of both question information and the information from CoT.
   * The "Method + LoRe" denotes the utilization of Low-Rank Approximation Re-ranking, exclusively relying on question information without the information from CoT.
   * The "Method + DQ-LoRe" denotes that we use the DQ-LoRe.

Reviewer comment (Experiments):
It would be great to include more LLMs to demonstrate the generalizability of the method.

Author response:
Thank you for your valuable suggestions. As per your recommendations, we have conducted additional comparative experiments using Llama2-7b-hf, Llama2-7b-chat-hf and Vicuna, which can be found in Table 4.

Table 4: Experiments on more LLMs
|   Method    | Llama2-7b-hf | Llama2-7b-chat-hf| Vicuna-7b
|  ----  | :----:  | :----:  | :----:  |
| Cot  			|12.6| 22.9 | 19.4 |
| Complex Cot 	|17.7| 29.4 | 23.8 |
| Auto-Cot  	|15.0| 27.0 | 23.9 |
| EPR  			|15.1| 23.0 | 22.0 |
| CEIL  		|15.2| 26.7 | 22.4 |
| DQ-LoRe  		|16.0| 28.9 | 23.8 |

We observe that compared to the retrieval baselines (EPR and CEIL), our method demonstrates significant improvements even on models with smaller open-source parameter sizes. Moreover, the effectiveness of our approach increases with the improvement in the model's instruction-following capabilities. However, we also acknowledge that, when compared to Complex cot and Auto-cot, our method does not show as significant improvements on the LLaMa family 7b-scale models.

Reviewer comment (Writing):
Possible typo on page 9: "a dual-query framework that enhances in-context learning for multi-step reasoning tasks by considering the Contexts of Interest (CoTs) in input questions and exemplars".

Author response:
Thank you for pointing out these mistakes, we have revised accordingly in the rebuttal version, which will be submitted later this week.

Reviewer comment (Writing):
Possible typo on page 2: "Following the acquisition of re-ranked exemplars from the smaller retrieval model, DQ-LoRe subsequently provides the exemplars to the LLMs for inference.,"

Author response:
Thank you for pointing out these mistakes, we have revised accordingly in the rebuttal version, which will be submitted later this week.

Reviewer comment (Evaluation):
It would be nice to include an analysis of latency.

Author response:
Thank you for your valuable feedback. We appreciate the constructive insights you provided. In response to the issues you raised, we have taken steps to clarify certain points in the paper. 

Our methodology encompasses two distinct latency stages: training and inference. In the ensuing discussion, we aim to dispel the notion that our approach is 'not ideal for real-life use cases,' a viewpoint we consider inaccurate.

Firstly, let's consider the latency during the training of the small model.    In practice, a single training iteration often proves sufficient for a given dataset type.    We deem this to be an acceptable cost in both academic and industrial contexts.    Additionally, as illustrated by the experiments outlined in Table 1 of our paper, our small model trained on the GSM8K dataset exhibits outstanding performance when applied to the SVAMP dataset.    This underscores the robustness of our method, offering a notable advantage in real-world applications.

Moving on to the second aspect of latency during the inference stage, we conducted thorough tests to compare the time efficiency of our approach against baseline methods.    Subsequently, we conducted a detailed analysis of the time consumption for each specific module within our method.    For a more comprehensive understanding, please refer to the detailed experimental results provided in the subsequent table.

Table 1: Average Processing Time per Question for DQ-LoRe and Baseline
|   Method    | EPR | CEIL| DQ-LoRe|
|  ----  | :----:  | :----:  | :----:  |
| Time  |0.756s| 0.758s | 1.584s |



Table 2: Detailed Time Consumption of Each Stage in DQ-LoRe
|  DQ-LoRe  | First Query | Second Query |  Re-Ranking |Inference
|  ----  | :----:  | :----:  | :----:  | :----:  |
| Time  |0.728s| 0.028s | 0.101s | 0.727s

From Table 1, it is evident that the time expended by our method exhibits a linear relationship in comparison to the baseline. Further insights from Table 2 reveal that the predominant additional consumption, in contrast to the baseline, occurs during the initial stage of requesting the large model to acquire chain-of-thoughts (CoTs). (In practice, each API key for the 3.5-16k-turbo model can process 180,000 tokens per minute, and with an average request requiring around 2000 tokens, theoretically, by employing numerous keys and parallelizing threads, the time consumption in this stage can be significantly reduced.)

Reviewer comment (Evaluation):
I wonder if the authors conduct significance tests on the improvements of the proposed method over baseline methods.

Author response:
**Question3:I wonder if the authors conduct significance tests on the improvements of the proposed method over baseline methods**

**Response:**
We conducted new significance tests on our method. Due to the need for multiple inferences, this would entail substantial resource consumption. Our model underwent 10 repeated experiments on datasets such as GSM8K, AQUA, and SVAMP using gpt-3.5-turbo to calculate the mean and variance of our model, as well as the mean and variance of our Dual-Query baseline on the GSM8K dataset. The experimental results are as follows:

Table 4: The average performance and variance of our model across various datasets.
 | Statistic | GSM8K | SVAMP| AQUA|
|  :----: | :----:  | :----:  |:----:  |
| Average   |80.9| 88.8 |59.4|
| Variance  |0.017| 0.889 |0.297|

Table 5: The average performance and variance of our model compared to the baseline.
 | Statistic| DQ-LoRe | DQ-LoRe w/o LoRe| 
|  :----: | :----:  | :----:  |
| Average   |80.9| 79.0 |
| Variance  |0.017| 0.043 |

From Tables 4 and 5, we can observe that the fluctuation in the performance of the model is not very significant. It demonstrates the robustness and effectiveness of our method.

Reviewer comment (Experiments):
Following W2 and W3, it would be great if the authors could involve more datasets and baseline methods in the experiments.

Author response:
**Question2:Datasets are limited. All of the datasets are about arithmetic questions. Some mentioned related studies like Auto-CoT (Zhang et al., 2022) are not compared in the experiments, especially while their studies are more general and conducted on more datasets.**

**Response:**
Thank you for your suggestions regarding our experiments. Based on your advice, we have added the Auto-CoT baseline and conducted supplementary experiments on the Q&A task, similar to the related experiments conducted with Auto-CoT and Complex-CoT baselines. The specific experimental results are as follows:

Table 2: Supplementary experiments for the Auto-CoT baseline in text-davinci-003.

| Model | GSM8K | AQUA |SVAMP|
|  ----  | :----:  | :----:  | :----:  |
| CoT     |55.1| 35.8 |77.3|
| Complex-CoT   |66.8| 46.5 |78.3|
| **Auto-CoT**   |60.7| 42.5 |80.0|
| EPR     |64.6| 45.0 |84.6|
|CEIL     |63.7| 47.2 |81.3|
| DQ-LoRe    |69.1| 48.0 |85.0|

Table 3: Supplementary experiments for the Auto-CoT baseline in gpt-3.5-turbo-16k.

| Model | GSM8K | AQUA |SVAMP|
|  ----  | :----:  | :----:  | :----:  |
| CoT     |77.0| 51.9 |82.0|
| Complex-CoT   |79.3| 57.0 |79.3|
|  **Auto-CoT**   |78.4| 50.4 |86.0|
| EPR     |77.3| 57.8 |88.0|
|CEIL     |79.4| 54.7 |87.3|
| DQ-LoRe(PCA)    |80.7| 59.8 |90.0|
| DQ-LoRe(Gaussian kernel)    |79.0| 58.7 |88.3|



Table 4: Supplementary experiments for the QA task in gpt-3.5-turbo-16k.

| Model | StrategyQA | QASC |
|  ----  | :----:  | :----:  |
| CoT     |73.8| 81.8 |
| Complex CoT     |74.5| 75.8 |
|  **Auto-CoT**   |71.2| 74.1 |
| EPR     |73.4| 80.2 |
| CEIL    |73.4| 80.8 |
| DQ-LoRe(PCA)    |74.6| 81.2 |
| DQ-LoRe(Gaussian kernel)    |75.4| 82.7 |

We have newly added experiments on the StrategyQA[1] and QASC[2] datasets, as well as a new baseline, AutoCoT[3].

For the question-answering dataset, we introduce LoRe with Gaussian kernel, where the PCA step is replaced with a Gaussian kernel to compute the similarity between the Exemplar's embedding and the current inference question + CoT (obtained from the first query) embedding. 

The motivation for adopting the kernel method here is to preserve as much information as possible in the embeddings for Q&A tasks. Although this is not a dimensionality reduction approach, the fundamental idea remains unchanged. That is, first, the embeddings of $M$ similar Exemplars are queried, then mapped onto another vector space. After this mapping, the similarity of these embeddings to the current problem's question+CoT embedding is recalculated, followed by re-ranking to yield the final $N$ Exemplars ($M$>$N$).


Based on these experiments, we find that, 

1. Firstly, the performance of AutoCoT is generally inferior to retrieval-based methods such as EPR and CEIL, particularly in Q&A tasks, where its performance is even lower than CoT. 

2. Secondly, in the new Q&A tasks, we are state-of-the-art, and an interesting discovery is that using a kernel trick to map the Exemplar's embedding into a higher-dimensional space for reordering, instead of PCA for dimensionality reduction, can effectively enhance the model's performance. There is even a 1.5% performance improvement on the QASC task compared to PCA.For Q&A tasks, there is a significant difference compared to solving math problems. Specifically, the background knowledge required for Math word problems is presented within the question itself, while the factual information for Q&A tasks needs to be queried from Large Language Models (LLMs). Moreover, this task is sensitive to textual details. Therefore, filtering information in embeddings is not a good choice for Q&A tasks, as dimensionality reduction could remove some vital entity information, which is not suitable for Q&A tasks. Hence, we have adopted a kernel method to transform the embeddings, ensuring that the information within the embeddings is not lost while still becoming separable.


**Reference**
>[1] Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., & Berant, J. (2021). Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9, 346-361.

>[2] Khot, T., Clark, P., Guerquin, M., Jansen, P., & Sabharwal, A. (2020, April). Qasc: A dataset for question answering via sentence composition. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 05, pp. 8082-8090).

>[3] Zhang, Z., Zhang, A., Li, M., & Smola, A. (2022). Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493.

>[4] Xu, S., Pang, L., Shen, H., Cheng, X., & Chua, T. S. (2023). Search-in-the-Chain: Towards the Accurate, Credible and Traceable Content Generation for Complex Knowledge-intensive Tasks. arXiv preprint arXiv:2304.14732.