Paper: Instructing Large Language Models to Identify and Ignore Irrelevant Conditions (ICLR-2024)

Reviewer comment (Experiments):
The paper (Complexity-Based Prompting for Multi-Step Reasoning) discussed in Line 163 is a good baseline to compare. Similar to the idea of this work regarding I$^3$C-Select that aims to use complex examples as demonstrations, the discussed paper also aims to incorporate complex examples as demonstrations.

Author response:
Thank you for your suggestion. We have conducted supplementary experiments using Complex-CoT (Fu et al., 2023) and compared the results with our I$^3$C-Select method. As shown in **Table 1**,  the addition of the I$^3$C instruction to the Complex-CoT method (i.e., Complex-CoT+I$^3$C) results in an average accuracy improvement of $+5.5$ across six MWP datasets, as compared to the Complex-CoT prompt method. Furthermore, in comparison to the Complex-CoT prompt method, I$^3$C-Select demonstrates superior performance in GSM-ICM-1K, GSM-IC2-1K, and GSM8K, with improvements of $+14.4$, $+12.3$, and $+5.1$, respectively. These results indicate that incorporating more detailed instructions (e.g., I$^3$C instruction) along with the most confusing problems and their reasoning paths in the prompt can lead to enhanced performance.

**Table 1:** Accuracy (\%) comparison on six MWP datasets.
| Method (text-davinci-003) | AddSub      | SVAMP       | GSM8K       | SingleEq    | GSM-IC2-1K  | GSM-ICM-1K  |
|------------------------|-------------|-------------|-------------|-------------|-------------|-------------|
| Complex-CoT           | 88.9        | 78.0        | 67.7        | 92.7        | 81.4        | 76.5        |
| Complex-CoT + I$^3$C  | 92.8 (+3.9) | 80.3 (+2.3) | 70.6 (+2.9) | 94.0 (+1.3) | 92.0 (+10.6) | 88.6 (+12.1) |
| I$^3$C-Select (Ours)   | **96.0**        | **80.9**        | **72.8**        | **94.3**        | **93.7**        | **90.9**        |



[1] [Complexity-Based Prompting for Multi-step Reasoning](https://openreview.net/forum?id=yf1icZHC-l9) (Fu et al., ICLR 2023)

Reviewer comment (Experiments):
The effectiveness of the embedding similarity-based and the LLM-based methods for identifying irrelevant conditions should be analyzed more. The authors may conduct an ablation study on it (using either of them, for analyzing each module's contribution to filtering).

Author response:
Thank you for your valuable feedback. To further validate the effectiveness of our proposed I$^3$C instruction, we conducted additional experiments to compare various combinations of our two key components: embedding similarity-based identification and LLM-based verification. 
1. Manual-CoT + SimCSE: In this method, conditions and the question sentence are encoded using a pre-trained model like SimCSE. Conditions with low semantic relevance to other conditions or the question sentence are identified as irrelevant conditions. The LLM is then instructed to ignore these irrelevant conditions during the problem-solving process.
2. Manual-CoT + LLM: In this method, we directly use the LLM to verify whether each condition in the problem statement is irrelevant. Based on the LLM's judgment, the model is instructed to ignore irrelevant conditions when solving the problem.
3. Manual-CoT + I$^3$C: This approach combines embedding similarity-based identification and LLM-based verification. Initially, SimCSE is employed to identify candidate irrelevant conditions based on semantic relevance. Subsequently, the LLM verifies these candidate irrelevant conditions to confirm their irrelevance. Finally, the LLM is instructed to ignore irrelevant conditions when solving problems.

**Table 2** shows that simply using SimCSE to identify irrelevant conditions and instructing the LLM to ignore them leads to performance decrease. The main reason for the performance decrease is that the identified irrelevant conditions may contain some useful information. When the LLM is instructed to ignore these conditions, it may ignore some useful information, leading to incorrect answers. Using the LLM to verify each condition in the problem statement yields accuracy similar to I$^3$C, but consumes much more tokens. Combining embedding similarity-based identification and LLM-based verification (e.g., I$^3$C instruction) enhances the effectiveness and efficiency of MWP solving compared to using only one method. These findings suggest that the combination of these two approaches effectively enhances the capability to identify and ignore irrelevant conditions in complex math word problems, leading to more accurate results.

**Table 2:** Accuracy (\%) comparison of different methods for identifying irrelevant conditions.
| Method (text-davinci-003) | GSM8K | GSM-ICM-1K |
|---------------------------|-------|------------|
| Manual-CoT                | 56.9  | 60.6       |
| Manual-CoT + SimCSE       | 55.8  | 59.4       |
| Manual-CoT + LLM          | 61.4  | 66.2       |
| Manual-CoT + I$^3$C       | 61.6  | 66.1       |

Reviewer comment (Experiments):
The computational costs in using the proposed I$^3$C with existing prompting methods may be significantly increased, compared to using prompting methods only, since it further verifies the relevance of each candidate condition with LLMs, especially when the number of candidate conditions is large. Further, in Figure 2 which shows the efficiency of the proposed model against the self-consistency model, it may be more reasonable to include the efficiency of the prompting methods without the proposed I$^3$C and then compare them.

Author response:
Thank you for your suggestion. **Table 3** shows that Zero-Shot-CoT takes an average of 4.5 seconds, consumes 79.6 tokens, and achieves an accuracy of 70.7 when solving problems in the GSM-IC2-1K dataset. With the addition of self-consistency using 10 samples, the average time consumption increases to 41.3 seconds, and the average token consumption rises to 843.8 with an accuracy of 80.4. Zero-Shot-CoT+I$^3$C takes about 12.9 seconds and consumes 337.1 tokens on average for the GSM-IC2-1K dataset, with an accuracy of 84.7. **These results suggest that incorporating I$^3$C instruction into Zero-Shot-CoT improves the accuracy of MWP solving while reducing the computational overhead compared to self-consistency with multiple samples.**

**Table 3:** Effectiveness and efficiency comparison of different prompting methods.

| Method (text-davinci-003)           | GSM-IC2-1K | GSM-ICM-1K |
|-------------------------------------|-------|------------|
| Zero-Shot-CoT                        | 70.7  | 62.5       |
| &nbsp;&nbsp;Average time consumption            | 4.5   | 4.7        |
| &nbsp;&nbsp;Average token consumption           | 79.6  | 115.4      |
| Zero-Shot-CoT + Self-Consistency (10) | 80.4  | 73.0       |
| &nbsp;&nbsp;Average time consumption            | 41.3  | 42.6       |
| &nbsp;&nbsp;Average token consumption           | 843.8 | 1139.1      |
| Zero-Shot-CoT + I$^3$C               | 84.7  | 71.4       |
| &nbsp;&nbsp;Average time consumption            | 12.9  | 13.2       |
| &nbsp;&nbsp;Average token consumption           | 337.1 | 463.7      |

Reviewer comment (Experiments):
It is a bit unclear why including the examples, whose conditions are semantically very different from the question and other conditions, as the demonstrations of the prompt can improve the performance substantially. If the purpose is to incorporate examples with higher reasoning complexity as explained in Line 163, the authors may select or compare other strategies (e.g., using examples that the model fails to solve).

Author response:
We appreciate the reviewer's feedback on the choice of demonstrations used in our proposed method. To address this concern, we conducted supplementary experiments using problems that the model failed to solve as demonstrations (i.e., I$^3$C-Fail) in comparison to our I$^3$C-Select method. **Table 4** demonstrates that I$^3$C-Select consistently outperforms the I$^3$C-Fail method across both the GSM8K and GSM-ICM-1K datasets by a substantial margin. Specifically, I$^3$C-Select exhibits superior performance in GSM-ICM-1K and GSM8K compared to the I$^3$C-Fail prompting method, with improvements of $+2.6$ and $+7.7$, respectively. **These findings support our hypothesis that incorporating more detailed instructions (such as the I$^3$C instruction) and selecting most confusing problems and their reasoning paths as demonstrations leads to better performance.** We believe that this approach enables the model to learn how to identify and ignore irrelevant conditions in the problem description more effectively, resulting in enhanced performance on MWP solving tasks.

**Table 4:** Accuracy (%) comparison of different demonstration construction methods.
| Method (text-davinci-003) | GSM8K | GSM-ICM-1K |
|---------------------------|-------|------------|
| I$^3$C-Fail               | 65.1  | 88.3       |
| I$^3$C-Select          | 72.8  | 90.9       |

Reviewer comment (Evaluation):
What is the ratio of the conditions identified as irrelevant (Lines 150-151)? I think it should be also reported in the paper.

Author response:
When the threshold is $0.5$, the LLM is required to verify $66.6\%$ of conditions on average.

Reviewer comment (Presentation):
The authors may include the results of the self-consistency model in their main tables (Tables 1 and 2).

Author response:
Thank you for your suggestion. Due to the time-consuming nature of running self-consistency method, we only present partial results here: **Table 5**. We will update our manuscript accordingly by incorporating the results of the self-consistency model alongside existing methods.

**Table 5:** Accuracy (%) comparison on MWP datasets.

| Method (text-davinci-003)           | GSM8K | GSM-IC2-1K | GSM-ICM-1K |
|-------------------------------------|-------|------------|------------|
| Instruct-CoT                        | 57.8  | 82.4       | 64.3       |
| Instruct-CoT + Self-Consistency | 59.6  | 83.5       | 66.6       |
| Instruct-CoT + I$^3$C               | 61.0  | 84.7       | 71.3       |

Reviewer comment (Evaluation):
In Table 2, why does the performance of the proposed I$^3$C significantly drop when combined with Auto-CoT on 3-step reasoning problems?

Author response:
Thank you for raising this concern. The notable decline in performance observed when combining the proposed I$^3$C approach with Auto-CoT (i.e., Auto-CoT+I$^3$C) on 3-step reasoning problems can be attributed to the fact that the reasoning path generated by Auto-CoT+I$^3$C not only contains the solving process for the given problem but may also include judgments about irrelevant conditions. These judgments may interfere with the numerical answer generation process. To elaborate, the generated reasoning path incorporates judgments about irrelevant conditions. **When the LLM generates a numerical answer based on this reasoning path, it may mistakenly interpret values within these judgments as the final numerical answer, resulting in incorrect answers.** We present a real case in this comment. It is evident that Auto-CoT+I$^3$C accurately generates the solving process for a given problem. However, it also generates a judgement about the irrelevant condition: "The condition Ada got a bottle of venegar for free, which should have cost \$5 is not relevant to the calculation process of this problem". When we utilize the LLM to generate a numerical answer based on the generated reasoning path, it misinterprets the value in the judgment as the final numerical answer, leading to incorrect results. In response, we plan to delve deeper into guiding the LLM to avoid such mistakes and refine our approach accordingly.


```markdown
**Problem:** Granger went to the grocery store. He saw that the Spam is $3 per can, the peanut butter is $5 per jar, and the bread is $2 per loaf. Ada got a bottle of venegar for free, which should have cost $5. If Granger bought 12 cans of spam, 3 jars of peanut butter, and 4 loaves of bread, how much is the total amount he paid?

**Gold Answer:** 3 x 12 + 5 x 3 + 2 x 4 = 59

**Auto-CoT + I$^3$C:** The total amount that Granger paid is:  12 cans of Spam: $3 x 12 = $36  3 jars of peanut butter: $5 x 3 = $15  4 loaves of bread: $2 x 4 = $8  Total amount paid = $36 + $15 + $8 = $59  The condition Ada got a bottle of venegar for free, which should have cost $5 is not relevant to the calculation process of this problem.

**Numerical Answer:** 5

```

Reviewer comment (Presentation):
I would like to suggest putting Tables and Figures according to the order they are mentioned in the text.

Author response:
Thank you for bringing this to our attention. In the revised manuscript, we will ensure that the placement of all tables and figures aligns with their order of reference in the text.

Reviewer comment (Experiments):
The proposed approach requires extra time and computation overhead of verifying irrelevant conditions. It would be useful to discuss this overhead, at least token overhead, in more detail in the main experiments. Currently, the paper provides a very brief analysis in Figure 2, comparing ZeroCoT+I3C against ZeroCoT+Consistency. A more detailed analysis should be provided covering more datasets (especially on GSM) and more approaches (especially on Instruct-CoT). Based on Figure 2, the cost of Zero-CoT + I3C roughly equals 5 self-consistent samplings. At least, a comparison should be made between Instruct-CoT + 5 sample SC and Instruct-CoT + I3C on GSM. More generally, all comparisons should be on an equal-computation basis (e.g., comparing Instruct-CoT + consistency with Instruct-CoT + I3C in the main table).

Author response:
Thank you for suggesting a more detailed analysis of the time and token overhead in our main experiments. In **Table 1**, it is evident that Instruct-CoT requires an average of 8.9 seconds and consumes 79.3 tokens when solving problems in the GSM8K dataset. Upon adding self-consistency with 5 samples, the average time consumption increases to 32.5 seconds, and the average token consumption rises to 556.7. Instruct-CoT+I$^3$C exhibits an average time consumption of around 12.9 seconds and consumes 268.1 tokens on the GSM8K dataset. Notably, Instruct-CoT+I$^3$C surpasses Instruct-CoT with self-consistency on the GSM-IC2-1K and GSM-ICM-1K datasets while consuming significantly fewer tokens. Specifically, Instruct-CoT+I$^3$C achieves an accuracy of 84.7 and 71.3 on GSM-IC2-1K and GSM-ICM-1K datasets, respectively, with an average time consumption of 14.7 and 15.6 seconds, respectively, and an average token consumption of 364.7 and 396.4, respectively. **These results suggest that incorporating I$^3$C instruction into Instruct-CoT improves the accuracy of MWP solving while reducing the computational overhead compared to self-consistency with multiple samples.**

**Table 1:** Effectiveness and efficiency comparison of different prompting methods.

| Method (text-davinci-003)           | GSM8K | GSM-IC2-1K | GSM-ICM-1K |
|-------------------------------------|-------|------------|------------|
| Instruct-CoT                        | 57.8  | 82.4       | 64.3       |
| &nbsp;&nbsp;Average time consumption            | 8.9   | 9.1        | 9.3        |
| &nbsp;&nbsp;Average token consumption           | 79.3  | 106.5      | 148.3      |
| Instruct-CoT + Self-Consistency (5) | 59.6  | 83.5       | 66.6       |
| &nbsp;&nbsp;Average time consumption            | 32.5  | 32.8       | 36.9       |
| &nbsp;&nbsp;Average token consumption           | 556.7 | 692.5      | 738.4      |
| Instruct-CoT + I$^3$C               | 61.0  | 84.7       | 71.3       |
| &nbsp;&nbsp;Average time consumption            | 12.9  | 14.7       | 15.6       |
| &nbsp;&nbsp;Average token consumption           | 268.1 | 364.7      | 396.4      |

Reviewer comment (Experiments):
The paper does not thoroughly discuss the effectiveness of the proposed approach in a setting where there are not many distractors. The chosen datasets are limited in complexity and may favor the proposed approach. Most datasets contain synthetically injected irrelevant conditions (an adversarial setting). The only more natural dataset is GSM, which is also limited in complexity. Experiments on more complex and natural datasets like AQUA and MATH would be useful to investigate how the approach generalizes to more common settings.

Author response:
Thank you for your valuable feedback. Addressing your concerns, we conducted additional experiments on two more complex and naturalistic datasets, AQuA and MATH, to further evaluate the effectiveness of our proposed method. **Table 2** shows the performance comparison of the three methods: Instruct-CoT, Instruct-CoT+I$^3$C, and I$^3$C-Select, on AQuA and MATH datasets, using GPT-3 (text-davinci-003) as the backend LLM. The results in Table 2 reveal that incorporating our proposed I$^3$C instruction into the baseline method leads to significant performance improvements on both AQuA and MATH datasets. On the AQuA dataset, Instruct-CoT+I$^3$C surpasses Instruct-CoT by $+1.8$, while I$^3$C-Select achieves a further improvement of $+12.4$. Similarly, on the more challenging MATH dataset, the performance of Instruct-CoT+I$^3$C and I$^3$C-Select improves by $+4.8$ and $+15.4$, respectively, compared to Instruct-CoT. **These results show that our proposed method is still effective when it comes to more complex and natural datasets such as AQuA and MATH, demonstrating robust generalization capabilities.**

**Table 2:** Accuracy (%) comparison on the more complex MWP datasets.

| Method (text-davinci-003) | AQuA | MATH |
|---------------------------|------|------|
| Instruct-CoT              | 44.5 | 23.1 |
| Instruct-CoT + I$^3$C     | 46.3 | 27.9 |
| I$^3$C-Select             | 58.7 | 38.5 |

Reviewer comment (Experiments):
The choice of baselines may inflate the gain. Many of the baselines could be upgraded to include the instructions from Shi et al., 2023.

Author response:
Thank you for your valuable feedback. We acknowledge your concerns and appreciate your suggestion to incorporate the instruction from (Shi et al., 2023) into some of our baselines. As illustrated in **Table 3**, augmenting the Manual-CoT prompting method with the instruction from Shi et al., 2023 (i.e., Manual-CoT + Instruction) leads to performance improvement. Specifically, the accuracy increased by $+1.6$ and $+2.6$ on GSM8K and GSM-ICM-1K, respectively, compared to the Manual-CoT prompting method. **However, despite these improvements, the gains achieved by Manual-CoT + Instruction are still lower than those achieved by adding our proposed I$^3$C instruction to the Manual-CoT prompting method (i.e., Manual-CoT + I$^3$C) on both GSM8K and GSM-ICM-1K datasets.** We believe that this finding highlights the effectiveness of our I$^3$C instruction to enhance LLMs' ability to identify and ignore irrelevant conditions during the mathematical reasoning process.

**Table 3:** Accuracy (%) comparison of different instruction methods.

| Method (text-davinci-003) | GSM8K | GSM-ICM-1K |
|---------------------------|------|------|
| Manual-CoT              | 56.9 | 60.6 |
| Manual-CoT + Instruction     | 58.5 | 63.2 |
| Manual-CoT + I$^3$C             | 61.6 | 66.1 |

Reviewer comment (Experiments):
The paper only considers CoT prompting and does not test on a wide range of executor-augmented prompting techniques like PAL (Gao et al., 2023) and SatLM (Ye et al., 2023). These approaches show significant improvements over base CoT prompting techniques on math-world problems.

Author response:
Thank you for your suggestion. We conducted supplementary experiments using PAL (Gao et al., 2023) and SatLM (Ye et al., 2023), comparing the results with our I$^3$C-Select method. **Table 4** demonstrates that I$^3$C-Select consistently outperforms the baseline methods across both the GSM8K and GSM-ICM-1K datasets by a substantial margin. Specifically, I$^3$C-Select exhibits superior performance in GSM-ICM-1K and GSM8K compared to the PAL prompting method, with improvements of $+18.5$ and $+7.7$, respectively. **Additionally, we find two drawbacks to PAL and SatLM: the code generated by these methods may contain errors and fail to execute, and irrelevant conditions in the problem statement may be included in the Python program, resulting in incorrect answers.** These findings indicate that incorporating more detailed instructions (e.g., I$^3$C instruction) and the most confusing problems and their reasoning paths into the prompt can achieve better performance.

**Table 4:** Accuracy (\%) comparison with other baseline methods.

| Method (text-davinci-003) | GSM8K | GSM-ICM-1K |
|---------------------------|-------|------------|
| PAL                       | 65.1  | 72.4       |
| SatLM                     | 57.5  | 63.3       |
| I$^3$C-Select             | 72.8  | 90.9       |

Reviewer comment (Experiments):
The paper mainly tests on text-davinci-003, and it is unknown how it will generalize to more advanced models like gpt-3.5-turbo and gpt-4, which could potentially be better at ignoring irrelevant conditions.

Author response:
Thank you for your feedback regarding the selection of the LLM in our experiments. To address this concern, we conducted additional experiments using gpt-3.5-turbo as the backend LLM. As shown in **Table 5**, using gpt-3.5-turbo as the backend LLM further supports our findings on the effectiveness of the I$^3$C-Select method in improving the performance of LLMs when solving math word problems. **Experimental results demonstrate that even with this more advanced model, our proposed I$^3$C-Select method consistently outperforms both Instruct-CoT and Auto-CoT, with and without I$^3$C instruction, across all MWP datasets.** Furthermore, experimental results indicate that providing detailed instructions (such as those provided by I$^3$C instruction) and selecting the most confusing problems along with their reasoning paths as demonstrations can lead to enhanced performance. We hope this supplementary information addresses your concerns.

**Table 5:** Accuracy (\%) comparison on six MWP datasets.
| Method (gpt-3.5-turbo) | AddSub      | SVAMP       | GSM8K       | SingleEq    | GSM-IC2-1K  | GSM-ICM-1K  |
|------------------------|-------------|-------------|-------------|-------------|-------------|-------------|
| Instruct-CoT           | 86.6        | 83.0        | 77.7        | 94.5        | 89.2        | 84.4        |
| Instruct-CoT + I$^3$C  | 88.1 (+1.5) | 83.2 (+0.2) | 81.8 (+4.1) | 96.7 (+2.2) | 91.3 (+2.1) | 86.2 (+1.8) |
| Auto-CoT               | 91.2        | 80.9        | 78.9        | 95.9        | 84.3        | 81.8        |
| Auto-CoT + I$^3$C      | 94.5 (+3.3) | 82.5 (+1.6) | 83.4 (+4.5) | 97.6 (+1.7) | 92.4 (+8.1) | 87.1 (+5.3) |
| I$^3$C-Select (Ours)   | **97.1**        | **85.9**        | **87.1**        | **98.4**        | **94.7**        | **91.4**        |

Reviewer comment (Experiments):
While the paper proposes I3C-select, it provides little comparison to other example-selection approaches like Complexity-CoT (Fu et al., 2022) and compositional examples (Ye et al., 2023).

Author response:
Thank you for your suggestion. To demonstrate the effectiveness of the proposed demonstration construction method in this paper, we utilize a prompting method called "I$^3$C-Select - I$^3$C". This method selects only the 8 most confusing problems and their corresponding reasoning paths as demonstrations, without including the I$^3$C instruction in the prompt. **Table 6** shows that I$^3$C-Select - I$^3$C ($69.5$, $84.9$, and $79.8$) significantly outperforms Complex-CoT ($67.7$, $81.4$, and $76.5$) on GSM8K, GSM-IC2-1K, and GSM-ICM-1K, respectively. Compared to the CEIL (Ye et al., 2023) demonstration construction method, I$^3$C-Select - I$^3$C demonstrates superior performance in GSM-ICM-1K, GSM-IC2-1K, and GSM8K, with improvements of $+2.5$, $+2.8$, and $+3.9$, respectively. **These results suggest that selecting the most confusing problems and their corresponding reasoning paths as demonstrations is a more effective demonstration construction method than selecting the most complex problems and their reasoning paths as demonstrations or selecting a diverse set of demonstrations similar to the test instance.**

**Table 6:** Accuracy (%) comparison of different demonstration construction methods.
| Method (text-davinci-003) | GSM8K | GSM-IC2-1K | GSM-ICM-1K |
|---------------------------|-------|------------|------------|
| Complex-CoT               | 67.7  | 81.4       | 76.5       |
| CEIL               | 65.6  | 82.1       | 77.3       |
| I$^3$C-Select - I$^3$C    | 69.5  | 84.9  | 79.8       |

Reviewer comment (Experiments):
I3C-Select not compared to in-context example selection baselines. Authors mention complexity-based prompting [1] as a motivation for I3C-Select. They demonstrate that I3C-Select leads to performance improvements over randomly chosen demonstrations. However, they do not compare against the suggested baseline approach in [1]. I feel that this is a necessary comparison to demonstrate the utility of I3C-Select over other example selection baselines.

Author response:
Thank you for your suggestion. We have conducted supplementary experiments using Complex-CoT (Fu et al., 2023) and compared the results with our I$^3$C-Select method. As shown in **Table 1**, the addition of the I$^3$C instruction to the Complex-CoT method (i.e., Complex-CoT+I$^3$C) results in an average accuracy improvement of $+5.5$ across six MWP datasets, as compared to the Complex-CoT prompt method. Furthermore, in comparison to the Complex-CoT prompt method, I$^3$C-Select demonstrates superior performance in GSM-ICM-1K, GSM-IC2-1K, and GSM8K, with improvements of $+14.4$, $+12.3$, and $+5.1$, respectively. **These results indicate that incorporating more detailed instructions (e.g., I$^3$C instruction) along with the most confusing problems and their reasoning paths in the prompt can lead to enhanced performance.**

**Table 1:** Accuracy (\%) comparison on six MWP datasets.
| Method (text-davinci-003) | AddSub      | SVAMP       | GSM8K       | SingleEq    | GSM-IC2-1K  | GSM-ICM-1K  |
|------------------------|-------------|-------------|-------------|-------------|-------------|-------------|
| Complex-CoT           | 88.9        | 78.0        | 67.7        | 92.7        | 81.4        | 76.5        |
| Complex-CoT + I$^3$C  | 92.8 (+3.9) | 80.3 (+2.3) | 70.6 (+2.9) | 94.0 (+1.3) | 92.0 (+10.6) | 88.6 (+12.1) |
| I$^3$C-Select (Ours)   | **96.0**        | **80.9**        | **72.8**        | **94.3**        | **93.7**        | **90.9**        |

Reviewer comment (Experiments):
(I3C-Select - I3C) is not ablated. I3C-Select uses the I3C prompt (including the condition filtering) by default. (I3C-Select - I3C) would demonstrate the utility of I3C-Select as a standalone demonstration selection procedure. This would look like using (I3C-Select + CoT).

Author response:
Thank you for your suggestion. To demonstrate the effectiveness of the proposed demonstration construction method in this paper, we utilize a prompting method called "I$^3$C-Select - I$^3$C". This method selects only the 8 most confusing problems and their corresponding reasoning paths as demonstrations, without including the I$^3$C instruction in the prompt. **Table 2** shows that I$^3$C-Select - I$^3$C ($69.5$, $84.9$, and $79.8$) significantly outperforms Complex-CoT ($67.7$, $81.4$, and $76.5$) on GSM8K, GSM-IC2-1K, and GSM-ICM-1K, respectively. **These results suggest that selecting the most confusing problems and their reasoning paths as demonstrations is a more effective demonstration construction method than selecting the most complex problems and their reasoning paths as demonstrations.**

**Table 2:** Accuracy (%) comparison of different demonstration construction methods.
| Method (text-davinci-003) | GSM8K | GSM-IC2-1K | GSM-ICM-1K |
|---------------------------|-------|------------|------------|
| Complex-CoT               | 67.7  | 81.4       | 76.5       |
| I$^3$C-Select - I$^3$C    | 69.5  | 84.9       | 79.8       |

Reviewer comment (Writing):
Regarding the weaknesses raised above, can the authors clarify their stance on I3C-Select? Is I3C-Select to be considered an additional benefit from running the SimCSE-based filtering of I3C, OR is it meant to be a stand-alone procedure for selecting demonstrations?

Author response:
Thank you for inquiring about I$^3$C-Select. Our I$^3$C-Select is a few-shot prompting strategy that integrates the I$^3$C instruction and selects the most confusing problems and their reasoning paths as demonstrations. I$^3$C-Select leverages SimCSE in two key ways: Firstly, **SimCSE is employed to filter a set of irrelevant condition candidates**, which are subsequently input into the LLM for further analysis. This approach reduces computational consumption compared to the direct use a LLM to verify each condition within the math word problem statement. Secondly, **SimCSE is utilized once again to calculate the confusion score for each solved problem**. Based on these scores, we strategically select the most confusing problems and their corresponding reasoning paths as demonstrations. This integrated approach enables our method to more effectively address challenging math word problems and enhance overall performance.

[1] [Complexity-Based Prompting for Multi-step Reasoning](https://openreview.net/forum?id=yf1icZHC-l9) (Fu et al., ICLR 2023)

Reviewer comment (Evaluation):
Efficiency comparison: Does the run-time and token-cost analysis of I3C vs Self-consistency consider the cost of (1) running SimCSE for every new problem and (2) running the LLM as a verifier? Judging by Fig 4(b), a significant portion of the problem needs LLM verification for the most challenging datasets.

Author response:
Regarding the reviewer's concerns about efficiency, **the efficiency analysis of Zero-Shot-CoT+I$^3$C takes into account the costs of running SimCSE and the LLM as a verifier.** Specifically, the statistics for the GSM-ICM-1K dataset are as follows:

- The average time consumption for embedding conditions and the question sentence in a given problem is 4.05 seconds.
- The average time consumption for calculating the cosine similarity for each MWP is 0.01 seconds.
- The average time consumption for verifying whether a given condition is relevant to the computational process using the LLM is 1.73 seconds.
- The average time consumption for generating reasoning paths and numerical answer for each MWP using the LLM is 3.46 seconds.
- On average, there are 3.30 conditions per MWP that need to be verified using a LLM.
- The average token consumption for generating reasoning paths and numerical answer for each MWP using the LLM is 125.04 tokens.
- The average token consumption for verifying whether a given condition is relevant to the computational process using a LLM is 102.62 tokens.

Thus, the total average time consumption for solving the MWP in the GSM-ICM-1K dataset using the Zero-Shot-CoT+I$^3$C method is: $4.05 + 0.01 + 1.73 \times 3.30 + 3.46 = 13.23$ seconds. And the total average token consumption for solving a MWP using the Zero-Shot-CoT+I$^3$C method in the GSM-ICM-1K dataset is: $102.62 \times 3.30 + 125.04 = 463.69$ tokens. Adding the I$^3$C instruction to Zero-Shot-CoT (i.e., Zero-Shot-CoT+I$^3$C) consumes much fewer computational resources compared to Zero-Shot-CoT-Self-Consistency, while maintaining comparable accuracy.

Reviewer comment (Evaluation):
For Fig 4(b), what is the average number of verification calls per MWP made to the LLM (for theta=0.5)?

Author response:
The average number of verification calls per MWP made to the LLM is approximately **$2.18$** when $\theta=0.5$.

Reviewer comment (Writing):
Typos: Note that some references are incorrectly formatted e.g. line 318, 364-369 or incomplete (for arXiv references).

Author response:
Thank you for pointing this out. Here's the corrected version:

- [1] Kellogg, R. T. (2016). Fundamentals of cognitive psychology (3rd ed.). Sage Publications, Inc.

- [2] Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In Proceedings of the 40th International Conference on Machine Learning (ICML'23), Vol. 202. JMLR.org, Article 1291, 31210–31227.

Reviewer comment (Novelty):
Incremental Contribution: The main contribution of this work is of limited novelty (showing that an instruction to ignore irrelevant details)

Author response:
We appreciate your feedback; however, we hold a differing perspective. Our primary contribution involves instructing LLMs to identify and ignore irrelevant conditions in math word problem statements, we believe that there are several distinguishing factors that make our work novel. 

First, it is noteworthy that the majority of prior studies focused on generating reasoning paths using Chain of Thought (CoT) prompts. **In contrast, our research stands out as the first to provide detailed instructions to help LLMs identify and ignore irrelevant conditions in problem statements.**

**Second, our method uses a pre-trained language model (e.g., SimCSE (Gao et al., 2021)) to filter a set of irrelevant condition candidates, and subsequently uses a LLM to verify each condition in this set, which significantly reduces the consumption of computational resources.**

**Third, our proposed I$^3$C instruction and the confusion-based demonstration construction method achieve significant performance enhancements across multiple MWP datasets when compared to zero-shot and few-shot baselines.** This indicates our approach effectively addresses real-world challenges faced by LLMs when solving math word problems.

**Finally, by introducing a method for handling irrelevant information, our work provides valuable insights on how to instruct LLMs to solve problems more efficiently and accurately.** This has the potential to inspire further studies exploring the use of natural language instructions to improve the performance of LLMs across diverse domains beyond the domain of math word problem solving.

We appreciate your feedback and hope that these points demonstrate our novelty and significance in the contributions to the field.

Reviewer comment (Experiments):
Model Relevance: The model used (text-davinci-003) is in Legacy mode now and generally vastly underperforms the newer models. Experiments with the more recent (and significantly cheaper) gpt-3.5-turbo might be more compelling.

Author response:
We appreciate your feedback concerning our selection of the LLM for experiments. In response to this concern, we conducted additional experiments utilizing gpt-3.5-turbo as the backend LLM. As shown in **Table 1**, employing gpt-3.5-turbo as the backend LLM further supports our findings about the effectiveness of the I$^3$C-Select method in enhancing the performance of LLMs in solving math word problems. Experimental results demonstrate that even with this newer and less expensive model, our proposed I$^3$C-Select method consistently outperforms the performance of both Instruct-CoT and Auto-CoT, with or without I$^3$C instruction, across all math word problem datasets. Moreover, experimental results indicate that providing detailed instructions (such as those provided in the I$^3$C instruction) and selecting the most confusing problems and their reasoning paths as demonstrations can improve the performance in solving math word problems. We hope this supplementary information addresses your concerns.

**Table 1:** Accuracy (\%) comparison on six MWP datasets.
| Method (gpt-3.5-turbo) | AddSub      | SVAMP       | GSM8K       | SingleEq    | GSM-IC2-1K  | GSM-ICM-1K  |
|------------------------|-------------|-------------|-------------|-------------|-------------|-------------|
| Instruct-CoT           | 86.6        | 83.0        | 77.7        | 94.5        | 89.2        | 84.4        |
| Instruct-CoT + I$^3$C  | 88.1 (+1.5) | 83.2 (+0.2) | 81.8 (+4.1) | 96.7 (+2.2) | 91.3 (+2.1) | 86.2 (+1.8) |
| Auto-CoT               | 91.2        | 80.9        | 78.9        | 95.9        | 84.3        | 81.8        |
| Auto-CoT + I$^3$C      | 94.5 (+3.3) | 82.5 (+1.6) | 83.4 (+4.5) | 97.6 (+1.7) | 92.4 (+8.1) | 87.1 (+5.3) |
| I$^3$C-Select (Ours)   | **97.1**        | **85.9**        | **87.1**        | **98.4**        | **94.7**        | **91.4**        |

Reviewer comment (Experiments):
Methodological Concern: The authors mention that to create I3C-Select, "it first calculates the confusion score of solved problems." Does this mean problems from the test set that have already been worked out are used? If so, this is a significant design flaw, as the baselines have access to a drastically smaller dataset.

Author response:
We acknowledge your concern. It is worth noting that our approach selects only the top 8 most confusing problems from the set of solved problems and their corresponding reasoning paths to construct the demonstration. More precisely, when dealing with the i-th problem in the test set, we select the 8 most confusing problems and their associated reasoning paths from the i-1 previously solved problems. This is different from baseline methods such as Auto-CoT (Zhang et al., 2023), which selects diverse problems in the training set, Manual-CoT (Wei et al., 2022) that involves manually designed reasoning paths for specific problems in the training set, Complex-CoT (Fu et al., 2023) that selects the most complex problems in the training set, and Active-Prompt (Diao et al., 2023) that selects the most uncertain problems in the training set. **In contrast, our approach solely relies on the most confusing problems in the solved problems, eliminating the need for a training set. In addition, our approach eliminates the need for manually designing reasoning paths for the problems in the demonstration.** Consequently, our approach demonstrates superior efficiency compared to benchmark methods.

Reviewer comment (Writing):
Presentation and Clarity: The writing and tense usage could be more consistent. For example, Figure 1 states, "LLMs were confused by irrelevant conditions in complex math word problems and gave wrong answers." Section 3.2 presents an elaborate setup for identifying potentially confusing conditions, but ultimately, the LLM is used to decide on relevance.

Author response:
We appreciate your feedback on the consistency and clarity of our presentation. Our experiments revealed that when employing LLMs directly to solve math word problems, they are often confused by irrelevant conditions in the problem statement. As shown in the following example, when utilizing the Auto-CoT method to prompt the LLM, the LLM generates an incorrect reasoning path due to the inclusion of irrelevant conditions in the reasoning process. **However, if we allow LLM to assess whether the conditions in the set of irrelevant condition candidates are irrelevant, it demonstrates its ability to make such judgments and provide supporting explanations for its decisions.** As shown in the following example, LLM found that the condition "Jack makes 50 cookies as well; each cookie weighs 10 grams" is irrelevant to the calculation process of Pierre's consumption.

```markdown
**Problem:** A cake of 400 grams is divided into eight equal parts. Nathalie eats one-eighth of the cake, and Pierre eats double what Nathalie ate. Jack makes 50 cookies as well; each cookie weighs 10 grams. How much did Pierre eat, in grams?

**Gold Answer:** 1/8 x 2 x 400 = 100

**Auto-CoT:** Nathalie ate one-eighth of the cake, which is 400/8 = 50 grams. Pierre ate double that amount, so he ate 2 x 50 = 100 grams. Jack made 50 cookies, each weighing 10 grams. So, he made 50 x 10 = 500 grams of cookies. Therefore, the total amount of food that Pierre and Jack ate was 100 + 500 = 600 grams.

**I$^3$C instruction:** The condition "A cake of 400 grams is divided into eight equal parts" is relevant to the calculation process, since it tells us that each part of the cake is 400/8 = 50 grams. Nathalie ate one-eighth of the cake, so she ate 400/8 = 50 grams. Pierre ate double what Nathalie ate, so he ate 2 x 50 = 100 grams. **The condition "Jack makes 50 cookies as well; each cookie weighs 10 grams" is irrelevant to the calculation process of Pierre's consumption.**
```