From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This paper proposes a novel technique for improving the performance of language models in tasks that involve arithmetic. The main idea is to do the following steps:

- Substitute the large numbers in the question with smaller numbers:
- Use the language model to generate the final answer.
- Use an algorithm to find out what was the calculation done by the language model.
- Do the same calculation on the original set of numbers.

The proposed method is novel and interesting. Furthermore, authors provide several experiments to evaluate it. The paper is well-written and very easy to follow. However, as also raised partially by some of the reviewers, there are several shortcomings:

- The proposed method is using an external tool to improve the performance of language models in reasoning. Therefore, the most important baseline is comparison with just prompting or fine-tuning language models to use calculator [1,2]. During rebuttal, authors argued that their method would work better than using a calculator when a small language model is used. However, these comparisons should be done and reported in the paper. So my concrete suggestion is to compare with prompting and finetuning language models to use calculator at different model scales. 

- GSM8K dataset [2] is a standard dataset for math-word problem which only involves the main 3 operators. It is a perfect task for this paper and I don't understand why there is not comparison on that dataset. I suggest authors to add GSM8K results.

- There is no guarantee that one can find the exact calculation just given the initial number and the final one. In fact, this probably becomes very difficult once more complex calculations are allowed. This should be clearly discussed or potentially analyzed in the paper.

- In many math word problems, it is not clear that the anchor numbers are feasible in the space of the problem. This is a big issue that cannot be fixed easy and again could get worse as the complexity of the problem/calculations grows. Therefore, it should be discussed in the paper.

[1] https://arxiv.org/pdf/2201.11903.pdf

[2] https://arxiv.org/pdf/2110.14168.pdf

Given the above issues, I recommend rejection but highly suggest authors to resubmit after addressing the above issues.
justification_for_why_not_higher_score: Lack of comparison with the calculator baseline and GSM8K dataset evaluations
justification_for_why_not_lower_score: N/A
summary_of_AC-reviewer_meeting: The main contributions of the paper including its strengths and weaknesses were discussed. The main issue that was raised was that the paper does not compare with the alternative way of using a calculator which is by prompting the model to call a calculator with the right operands when needed.

From: ICLR.cc/2023/Conference/Paper4250/Authors
title: Clarification and more discussion
comment: >**"Nonetheless, the method still relies on the language model to output the correct reasoning steps and the correct implicit computations."**

Thank you for your quick response! However, SOLIS does not require the model to "output the correct reasoning steps". Quite the opposite, we only use the **output number** via implicit calculation and the **input number** to inversely elicit the arithmetic expression, which is systematically derived through SOLIS.

> **"Thus it seems reasonable to expect that these models may still be prompted or finetuned to output an alternative format to represent the reasoning steps they are already doing."**

Thanks again, but we do not think it is trivial for fine-tuned models to "output an alternative format to represent the reasoning steps". Just as you pointed out in the previous sentences of this very comment, smaller models "cannot reliably generate arithmetic expressions". Because of this, we propose SOLIS to elicit the reasoning process inversely, which is exactly the contribution of this work.

Moreover, we have already included an important strong baseline NERD as an alternative format (aim to tackle numerical reasoning in a neural-symbolic manner), which is outperformed by methods that directly generate answers, e.g., the $\rm{POET-SQL_{T5}}$. This indicates that generating expressions is not always the optimal solution in this field, and generating answers directly is a strong alternative.

> **"Providing some examples of the output format you desire would help a lot on grammar correctness"**

Yes, providing examples can help with expression generation, but recent work shows that Codex still produces syntax error when carefully designed few-shot prompting is given, as analyzed by Binder at Table 4 in its paper (5% failed samples from development set have syntax error, which is non-trivial). 

We also appreciate much on the series of work of chain-of-thought, but we believe that SOLIS shows a very different way with different advantages as we stated in the last response letter and thus can motivate future work on reliable numerical reasoning of LLMs from a different perspective.  And importantly, SOLIS is generally effective to LLMs as well as LMs with smaller scale. We hope our answers have resolved some of your concerns. If so, we would appreciate it if you could revise your review or increase the rating. Thanks again for your constructive feedback and suggestions.

From: ICLR.cc/2023/Conference/Paper4250/Reviewer_AWSU
title: Response
comment: Thank you to the authors for your response. 

I do appreciate that this method may be more useful for smaller models that cannot reliably generate arithmetic expressions. Nonetheless, the method still relies on the language model to output the correct reasoning steps and the correct implicit computations. Thus it seems reasonable to expect that these models may still be prompted or finetuned to output an alternative format to represent the reasoning steps they are already doing. Those alternatives are conceptually simpler and more generalizable and I think it is important to study and benchmark against these baselines to establish the value of the proposed method.

Also I'm not convinced by the chatGPT examples. 

For 1a, providing some examples of the output format you desire would help a lot. E.g. using Codex model, we can give it the following prompt:

```
Q: As of the census of 2000, there were 10,477 households and 7,459 families in the county. How many more households are there than families?
A: To see how many more households are there than families, we need to subtract the number of families from the number of households. To solve this question, we use the following expression:

10477-7459=3018.

The answer is 3018.

Q: Tom's income today is 1034532$, and his expense is 53234$ today. David saves three times of money of what Tom does today. How much money did David save today?
```

And we get the following output:

```
A: To solve this question, we need to find out how much money Tom saved today. To do this, we need to subtract his expense from his income. Then, we need to multiply the result by 3. To solve this question, we use the following expression:

(1034532-53234)*3=2907854.

The answer is 2907854.
```

The numerical calculation is wrong but the expression is correct.

Re 1b, that sort of problem can happen in general and is more of a problem with the model's reasoning abilities. I think it can also be alleviated by using better prompts and prompt examples.

From: ICLR.cc/2023/Conference/Paper4250/Authors
title: Response to Reviewer AWSU[1/3]
comment: Thank you for your valuable feedback. We are glad to answer all your concerns and to clarify our points. We hope our answers have resolved some of your concerns. If so, we would appreciate it if you could revise your review or increase the rating. Thanks again for your constructive feedback and suggestions.

> **"This aims to improve the performance of LLMs on mathematical reasoning tasks by increasing the accuracy on arithmetic calculations."**

Thank you, but according to "Summary Of the Paper", we believe that you may misunderstand the scope of our work. In fact, SOLIS is suitable for **not only large language models**, but also any fine-tuned language models which are usually with smaller scale. For instance, we have conducted extensive experiments on models including BART, T5, and GenBERT, and achieved significant gains, as reported in **Table 2 and Table 3**. 

> **W1 “Similar performance gains can be achieved by prompting the LLMs to output its reasoning steps in the format of mathematical expressions, and sending the evaluation to an external calculator.”**

1. The mentioned method which "prompts the LLMs to output its reasoning steps in the format of mathematical expressions" almost works for large language models (e.g., >10B). Importantly, LLMs are not **open source** and their training strategies are not fully public, but SOLIS can be fully studied and applied using smaller fintuned models, making it more accessible and easier to reproduce than prompting the LLMs. We also have to emphasize that the seemingly "simple" method is based on a large model obtained from a large amount of complex data & training engineering, and the value of our work should not be overlooked.
2. Chain-of-thought with external calculator indeed has gains by outputing such reasoning steps, and we have compared SOLIS with chain-of-thought in our experiments, as shown in **Table 4**. Notably, LLMs with SOLIS still outperform chain-of-thought with an external calculator, proving its effectiveness. We think that this attributes to the stable performance of LLMs on implicitly calculating super small numbers than generating reasoning steps. Besides, we believe that generating answers directly and inversely uncovering reasoning steps using SOLIS is complementery to chain-of-thought both conceptly and practically, and we will investigate this in future work.
3. Several very recent works further studied and developed more advanced prompting methods to output step-by-step expressions/code, such as **PAL[1], PoT[2], and Binder[3]**, and we also appreciate these works. But as a kind reminder, these works are either preprinted later than our submission(PAL and PoT), or are concurrently submitted(Binder), so we did not include them as baselines.

## Reference
[1] Gao, Luyu, et al. "PAL: Program-aided Language Models." arXiv preprint arXiv:2211.10435(2022).

[2] Chen, Wenhu, et al. "Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks." arXiv preprint arXiv:2211.12588(2022).

[3] Cheng, Zhoujun, et al. "Binding language models in symbolic languages." arXiv preprint arXiv:2210.02875(2022).

From: ICLR.cc/2023/Conference/Paper4250/Authors
title: Response to Reviewer AWSU[2/3]
comment: > **W2 “this method is a very complicated way to do something that is much simpler in comparison(simple arithmetic calculations)“**
1. We have conducted extensive experiments on the DROP dataset, which contains complex text, often with large numbers, long passages and various numerical scales. Thus, in our opinion, it is a **realistic and difficult** dataset, rather than "simple arithmetic calculations" as you mentioned.

   Moreover, the SOTA performance of DROP is currently obtained from fine-tuned non-LLM model, and the performance of the LLMs is still far from the fine-tune model. SOLIS has been able to achieve significant improvement on these fine-tuned models, which also proves its effectiveness on challenging tasks.

2. The core idea of SOLIS is also quite simple and direct, i.e., using multiple sets of input and output pairs to inversely recover the arithmetic relationship. But to do it systematically and analytically, we carefully defined the current problem space and provided analytical provements. Then we proposed three different algorithms. These algorithms are easy to implement; importantly, the search-based algorithms can be easily extended to a much broader scope, as we responded to Reviewer `Dv6a`.
3. Our methods are not that complex, because multiple sets of input & output pairs can be fed to LMs in parallel. It works well in our experiments (we use A, B, C, D to indicate different sets in the following example):
```
Passage: The Greek census 2011 recorded 9,903,268 Greek citizens(91.56%),
A-14,B-13,C-16,D-16 Albanian citizens(4.44%),
A-9,B-3,C-12,D-11 Bulgarian citizens(0.7%), 46,523 Romanian citizenship (0.43%), 34,177 Pakistani citizens(0.32%),
A-2,B-2,C-10,D-3 Georgia (country) citizens(0.25%) and 247,090 people had other or unidentified citizenship(2.3%).
189,000 people of the total population of Albanian citizens were reported in 2008 as ethnic Greeks from Southern Albania, in the historical region of Northern Epirus.
Question: how many more albanian citizens were there compared to bulgarian and georgia citizens combined?
A: A-8,B-11,C-4,D-2
```
The above example shows that LLMs can give 4 correct outputs simultaneously within one shot. Such cases also indicate that LLMs performs very well using small numbers, again proving our observations.

4. Our framework works in a plug-and-play manner, and requires no extra training, which makes it quite lightweight and easy to use.

From: ICLR.cc/2023/Conference/Paper4250/Authors
title: Response to Reviewer AWSU[3/3]
comment:  >**W3  “does not guarantee that the correct calculations are extracted“**

We agree that SOLIS can not fully guarantee the correctness of the prediction like other methods. But SOLIS can ensure a correct arithmetic relationship in its problem space given that the model performs accurate calculation on small numbers. 

> **W4 “there is no discussion of extending this particular approach to something that cannot be readily achieved by simple methods”**

Thanks for your constructive comments. we believe that generating answers directly and then inversely uncovering reasoning steps is a complementery method to chain-of-thought both conceptly and practically, and there still exists a large research space to explore.

**1.** Directly generating expressions has problems like incorrect grammar, absence of critical information such as unit, and flexible granularity of reasoning steps, while SOLIS does not have such problems. We present some detailed explanation at below as **(1a), (1b),** and **(1c)**.
    
**(1a)** The expression direcetly generated by LMs may **not be executable**. 
    
Take recent chatGPT for example, the generated expression may contain special characters or problematic operations:

**Question:**
```
Tom's income today is 1034532$, and his expense is 53234$ today. David saves three times of money of what Tom does today. How much money did David save today? Please think step by step using mathmetical expressions in python code.
...
```
**Reference answer:** `981298$`

**ChatGPT Answer:**
```
savings = 1034532$ -53234$ 
david_savings = 3 * savings
print(david_savings)
```
    
But this program is not executable due to the dollar character. And even if removing the dollar character, the final answer only has the quantity but lack unit by `print(david_savings)`. In contrast, SOLIS guarantees **the inferred expressions are executable**. And SOLIS can naturally be applied to those answers that contain other critical information (e.g., unit) in addition to pure numbers, like what it has successfully done in the DROP dataset.

**(1b)** For cases where there is not need for calculation, guiding models to generate an expression is very likely to produce a wrong solution, and the following is another example we tested in chatGPT:
    
 **Question:**
```
Tom's income today is 1034532$, and his expense is 53234$ today. How much money did David save today? Please think step by step using mathmetical expressions in python code.
...
```

**Reference answer:** can not be answered based on provided information.
    
**ChatGPT Answer:**
  ```
  income = 1034532
  expenses = 53234
  savings = income- expenses
  ```
    
  But if do not use code generation, chatGPT can easily find that it can not give an answer due to absence of critical information and then will not mislead SOLIS. 

**(1c)** The granularity of reasoning steps of chain-of-thought is also hard to control. But by defining a problem space in ahead, inversely eliciting can control the granularity explicitly.

---
**2.** Through inversely eliciting arithmetic relationship using auto-regressive LMs, SOLIS can help with annotating arithmetic relationships in various corpora, e.g., unlabeled corpus such as crawled finantial reports or weak supervised QA datasets such as DROP and WikiTableQuetion. Also, number substitution will create more samples, which can serve as a data augmentation strategy.

From: ICLR.cc/2023/Conference/Paper4250/Reviewer_AWSU
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors propose a method to infer the arithmetic expressions performed by a language model while solving a mathematical reasoning question with a chain-of-thought style output. The inferred expressions are then calculated outside of the LM. This aims to improve the performance of LLMs on mathematical reasoning tasks by increasing the accuracy on arithmetic calculations. 
strength_and_weaknesses: Strength:

- The method itself appears to be novel. It involves extracting the relevant numbers and substituting large numbers in the chain-of-thought answer with random small numbers where the LM can reliably perform calculations on. By sampling a set of these input and output correspondences, the authors specify and solve a linear system of equations to elicit the arithmetic relationships. 

- The method leads to performance gains in the settings evaluated in this paper.

Weakness:

- The main issue with this work is that this method is a very complicated way to do something that is much simpler in comparison (simple arithmetic calculations). Despite its complexity, the method does not guarantee that the correct calculations are extracted and does not scale well to more complex expressions. It seems that similar performance gains can be achieved by prompting the LLMs to output its reasoning steps in the format of mathematical expressions, and sending the evaluation to an external calculator. This approach was already explored in [1] with some success. One can also imagine finetuning the model to output well-formatted arithmetic expressions in a task-agnostic way. A version of this was explored in [2] on the GSM8k dataset. These alternatives are much simpler and more general than the proposed approach, and the paper has not benchmarked against these simpler approaches to show that it offers a meaningful improvement. 

- The idea of formalizing the reasoning steps of a LM through perturbing its output and inferring the relationships may be an interesting framework, but there is no discussion of extending this particular approach to something that cannot be readily achieved by simple methods. 

[1] https://arxiv.org/pdf/2201.11903.pdf

[2] https://arxiv.org/pdf/2110.14168.pdf
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and it's nice that the paper includes a discussion on its limitations. However, there are important benchmarks missing and thus the evaluation is incomplete. 
summary_of_the_review: The paper presents an interesting idea for eliciting the mathematical expressions that the LM is implicitly computing, and offloading such computations to an external calculator. However, the proposed approach is extremely complicated compared to the problem that it's trying to solve, and does not generalize well to other situations nor scale to more complex arithmetic expressions. There are simpler and more general approaches that are likely to perform well and these alternatives are not benchmarked. Therefore I think the current paper does not convincingly demonstrate the value of the proposed approach.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper4250/Authors
title: Thanks for the Review
comment: Dear Reviewer qbBT,

Thank you again for the efforts! As the deadline for rebuttals is approaching, we would appreciate your feedback on whether our response has addressed your concerns. In the final days, we will **make every effort to address any remaining questions**.

Best,

Paper4250 Authors

From: ICLR.cc/2023/Conference/Paper4250/Authors
title: Thanks for the Review
comment: Dear Reviewer 8HSz,

Thanks again for your insightful comments and suggestions! Since the deadline for rebuttals is approaching, we would appreciate your feedback on whether our responses address your concerns. In the final days, we will **make every effort to address any remaining questions**.

Best,

Paper4250 Authors

From: ICLR.cc/2023/Conference/Paper4250/Authors
title: Gentle Reminder to Reviewer qbBT
comment: Thank you again for your comment.

Since the discussion period is coming to an end, we would like to know if you have any further concerns. From our perspective, we believe we have addressed all of the concerns that you have raised in the initial review. Moreover, we have also updated a revisied version of our paper, which includes more baselines as you requested and discussion of numerical understanding works.

**We would be very grateful if you could re-assess our paper based on our response and let us know if you might find our work more positive.** If you have any further questions, please do not hesitate to inquire and we are available at any time for further clarification.

Best,

Paper4250 Authors

From: ICLR.cc/2023/Conference/Paper4250/Authors
title: Gentle Reminder to Reviewer Lb3T 
comment: Thank you again for your constructive comments. We have improved the wording of experimental setup in the updated paper draft, and have responded to all your concerns at below. Since the discussion period is coming to an end, we would like to know if you have any further concerns.

**Please let us know if there is any additional clarifications we can provide to show the merit of our paper.**

Best,

Paper4250 Authors

From: ICLR.cc/2023/Conference/Paper4250/Authors
title: Gentle Reminder to Reviewer 8HSz
comment: Thank you again for your valuable and constructive feedbacks. They are insightful and have allowed us to improve our work (see updated paper and our response).

Since the discussion period is coming to an end, we would like to know if you have any further concerns. From our perspective, we believe we have addressed all of the concerns that you have raised in the initial review. 

**We would be very grateful if you could re-assess our paper based on our response and let us know if you might find our work more positive.** Please let us know if there is any additional clarifications or experiments we can provide to show the merit of our paper.

Best,

Paper4250 Authors

From: ICLR.cc/2023/Conference/Paper4250/Authors
title: Response to Reviewer qbBT [1/2]
comment: Thank you very much for your valuable comments, and we are delighted that you enjoy the idea and the writing of this paper. We believe, however, that you may have misconceptions about our work due to the absence of certain details, which we shall explain in depth below.

# W1. the offered solution is antithetical to the premise of the paper

 **(1) Response to "The offered solution is antithetical to the premise of the paper''.** 

Our solution **Exactly Matches** the premise of the paper. As **reviewer `Dv6a`** commented, "While most research attempts to use neural-symbolic methods to address numerical instability in neural-based reasoning, this paper figured out a way to identify relevant numbers and outsource the finding solution steps to classical methods.":
- **Our premise** is “LMs are still far from being robust on implicit numerical calculation: as numbers grow bigger and more complex, LMs are more likely to fail” (Section Introduction, Paragraph 1, Line 9-10), but ''understanding'' numbers within LMs is not our focus to make improvements.
- **Our solution** is “leveraging simple numbers as anchors to analytically elicit the implicitly inferred arithmetic expressions from LMs via solving linear systems, and then explicitly applying the expressions on complex numbers” (Section Introduction, Paragraph 2, Line 5-6).
- Experimental results demonstrate that our solution successfully elicits the existing hidden knowledge in LMs and significantly improves their performance on numerical reasoning datasets, especially for complex numbers.

**(2) Response to "Simply appends a system on the top of untouched LMs that does the computations for the LM''.** 

We **do not** think that "touching" language models is the only acknowledged way for trustworthy numerical reasoning.
A flurry of recent works such as GenBERT [1] and POET [2] have explored **a variety of neural based methods** to improve numerical understanding and calculation capabilities of LMs, but they are **substantially challenged** by the infinite space of numbers for implicit numerical calculations, and it is non-trivial to maintain the **versatility** of original LMs for diverse tasks beyond numerical reasoning. 

Like what the reviewer `Dv6a` commented ("this paper figured out a way"), this paper proposes **the first analytical method** to elicit hidden knowledge in LMs through systematic number substitution for reliable numerical reasoning.
**Keeping LMs untouched (training free and plug-and-play)** is exactly what makes this work distinctive from related ones and portable to a variety of existing LMs. As Table 3 shows, in addition to benefiting standard LMs,  our solution even significantly improves the performance of carefully designed LMs for numerical reasoning, including GenBERT[1] and POET[2].
Thank all reviewers for recognizing our novelty, and we do hope that analytically eliciting hidden knowledge from LMs can stimulate a lot of research interests in reliable reasoning of representation learning methods.

**(3) Response to "LM has to understand numerals in the paragraph but the output is textual''.**

In this paper, we **indeed have not** extended SOLIS to numerical reasoning cases where the output is textual, because it is hard to symbolically deal with textual outputs. Fortunately, recent works such as **decomposed prompting [3]** and **least-to-most prompting [4]** demonstrate promising results of automatically decomposing a numerical reasoning problem into several sub-problems. Given this, SOLIS has the potential to improve the final textual answer performance if any of subproblems involves numerical calculation.

For example, to **answer the question** “Given France’s population is 67,813,396 in this year and 67,499,343 in the last year and Germany’s population is 84,323,763 in this year and 83,129,285 in the last year. Which country has the higher population growth rate?”, **the sub-problems** and their corresponding answers can be “1. What is the population growth rate of France? 0.465\%; 2. What is the population growth rate of Germany? 1.014\%; 3. What is the difference of growth rate of France and Germany? -0.465\%; 4. Which country has the higher population growth rate? Germany”.
SOLIS can improve the overall performance by addressing the initial three subproblems.

# Reference

[1]. Mor Geva, Ankit Gupta, and Jonathan Berant. Injecting numerical reasoning skills into language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). 2020

[2]. Pi, Xinyu, et al. Reasoning like program executors. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2022.

[3]. Khot, Tushar, et al. Decomposed Prompting: A Modular Approach for Solving Complex Tasks. arXiv preprint arXiv:2210.02406 (2022).

[4]. Zhou, Denny, et al. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. arXiv preprint arXiv:2205.10625 (2022).

From: ICLR.cc/2023/Conference/Paper4250/Authors
title: Response to Reviewer qbBT [2/2]
comment: # W2. The lack of baselines GenBERT and DICE

**(1) Response to "two essential works that also aim to improve numerical comprehension and also evaluate themselves on the DROP dataset''.**
We really appreciate the pointers on GenBERT [1] and DICE [2]. We agree that GenBERT is a closely related work, and we have discussed the difference between our work and GenBERT in the related work section. As for reasons why we do not choose GenBERT as our baseline, you can find them in the next response.

However, we do not consider the work of DICE an important baseline.First, the method DICE is never evaluated on the DROP dataset.In fact, they just use a DROP-trained model as the baseline in their evaluation tasks. Second, their evaluation tasks are either classification or regression tasks, unlike our focused DROP, which involves flexible answers that go beyond classification (e.g., requiring complex calculation. Last, since we have compared our solution with POET [3] and GenBERT (to be discussed in the next response), we think that it does not hurt our experiments' completeness without comparing DICE, as well as other mentioned works like[4,5,6].

Neverthless, it is necessary to clarify the differences between our approach and the line of numercay understanding. Inspired by your comment, we update the related work to explain them (see below).We hope these actions make sense to you.
> Since our work focuses on numerical reasoning, it is related to previous works on numerical understanding, which has been found to be important for deep learning models (Spithourakis \&Riedel, 2018; Wallace et al., 2019; Naik et al., 2019; Zhang et al., 2020; Sundararaman et al., 2020; Thawani et al., 2021).In general, previous work on numerical understanding aims to develop better numeracy embeddings that accurately reflect their properties, so they typically evaluate numeracy embeddings on synthetic tasks.For example, Wallace et al. (2019) proposes List Maximum, an evaluation task by predicting the index of the maximal number in a list of five numeracy embeddings. When it comes to downstream applications, the majority of previous works(Jiang et al., 2020; Duan et al., 2021) are studied on classification or regression tasks. Differently, numerical reasoning involves flexible answers that go beyond classification (e.g., requiring complex calculation), which is our focus.

**(2) Response to "GenBERT is cited in the paper, but not used as baselines''.**
In the submitted version of this paper, we have compared SOLIS with POET [3], the state-of-the-art numerical-reasoning-oriented language model using pre-training techniques similar to GenBERT. The experimental results show that for POET, SOLIS yields a 2.0\% F1 improvement, which is very impressive. We believe that such a significant improvement to POET is sufficient to verify that our approach can augment reasoning-oriented pre-trained language models (e.g., GenBERT), hence we do not include GenBERT as the baseline in the submitted version.

However, the importance you attach on GenBERT convinces us that it is vital to include GenBERT as a baseline. Therefore, during the rebuttal phase, we have reproduced the GenBERT model and integrated it with SOLIS to verify the effectiveness of our method. The experimental results demostrate that our method does improve GenBERT, with absolute F1 improvements of 4.5\% and 2.6\% on the number and date answers, respectively.
Finally, SOLIS can boost GenBERT by 2.1\% on the DROP validation set, and the details can be found below.
We have also updated our paper accordingly, and hope that alleviates your concerns on the baseline weakness.

| **Models** | **EM** | **F1** |
|:---|:---|:---|
| GenBERT | 68.8 | 72.3 |
| GenBERT w. SOLIS | 70.5 (+1.7) | 74.4 (+2.1)  |
| POET | 76.9 | 80.0 |
| POET w. SOLIS | 78.2 (+1.3) | 82.0 (+2.0) |

# Reference

[1]. Mor Geva, Ankit Gupta, and Jonathan Berant. Injecting numerical reasoning skills into language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). 2020

[2]. Sundararaman, Dhanasekar, et al. Methods for Numeracy-Preserving Word Embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.

[3]. Pi, Xinyu, et al. Reasoning like program executors. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2022.

[4]. Spithourakis, Georgios, and Sebastian Riedel. Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numbers. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL). 2018.

[5]. Jiang, Chengyue, et al. Learning Numeral Embedding. In Findings of the Association for Computational Linguistics: EMNLP 2020. 2020.

[6]. Zhang, Xikun, et al. Do Language Embeddings capture Scales? In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP. 2020.

From: ICLR.cc/2023/Conference/Paper4250/Authors
title: Response to Reviewer Dv6a
comment: Thank you for your recognition and constructive feedback! We are glad to respond to all your concerns.

# W1. SOLIS is limited to linear system

As discussed in the limitation part, currently SOLIS is limited to four fundamental arithmetic operations that can be solved by linear systems.
Fortunately, we believe there are two potential solutions to extend SOLIS to more complex scenarios. On the one hand, recent works such as decomposed prompting [1] and least-to-most prompting [2] demonstrate the possibility of automatically decomposing a complex numerical reasoning problem into several sub-problems. High-school math problems can also be well decomposed to sub-problems and many of them are still linear systems that can be benefited by SOLIS. On the other hand, we also believe the analytical method can be applied to non-linear systems, which will be an interesting follow-up work. For example, by transforming more non-linear systems to linear systems by using virtual variables and even kernel methods, or simply augmenting the search space of the search-based algorithm with non-linear candidate expressions involving operators like `max`, `min`, and `count`.
We will also explore the application of SOLIS on more complex reasoning tasks in our future work.

# W2. About analyzing SOLIS's failure cases

We conduct an analysis of SOLIS' failure cases: we collect a subset of 1,000 cases that SOLIS returns expressions on the Drop dataset, and SOLIS fails on 65 of them.
We are glad to present some of the failure cases at below, and most of them are caused by wrong/instable hidden arithmetic relationships inferred by LMs themselves, e.g., the first case of the table. 
Failure caused by SOLIS' search algorithms has a relatively small percentage.

| questions                                                    | predicted operands |  predicted expression | ground truth expression |analysis |
|---------------------------------------------------------------------------|-------------------|--------|----------------------|----------|
|how many of the personnel were _not civil servants_?| x1=14506, x1=1513| x1-x2 | 14506 - 337 |fail to find correct number |
|how many more ... _longest touchdown_ ... to the _shortest_? | x1=99, x2=2| 100 - x2 | 99 - 2|  fail to search |
|how many are either _military specialists_ or _civil servants_?| x1=1513, x2=520, x3=337 | x1+x2+x3 | 520 + 337 |fail to find correct number & fail to search |

# W3. Not reproducible though as code is not provided

We would like to make all our code and data public upon acceptance of this paper, and we will also create a huggingface space to enable people who are interested in trying and playing SOLIS in the future.

**Update on 11.18:** We have uploaded a zip file, containing code for analytical-based algorithm, and a demo script for search-based algorithm. We hope this can address your concern about the reproducibility.

# Reference
[1]. Khot, Tushar, et al. Decomposed Prompting: A Modular Approach for Solving Complex Tasks. arXiv preprint arXiv:2210.02406 (2022).

[2]. Zhou, Denny, et al. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. arXiv preprint arXiv:2205.10625 (2022).

From: ICLR.cc/2023/Conference/Paper4250/Authors
title: Response to Reviewer 8HSz
comment: # W1. The analysis on majority voting

Thanks for your thoughtful feedback! Following your suggestion, during the rebuttal phase, we completed the analysis on majority voting on the dataset MultiArith. The experimental results demonstrate that even when the baseline is equipped with major voting, SOLIS still works very well. As shown below, SOLIS can boost the baseline by 19.7%, which is very impressive. More details can be found in the Appendix C.3.
| **Backbone** | **MultiArith** |
|:---|:---|
| Zero-shot Chain | 63.8 |
| Zero-shot Chain w. majority voting | 73.5 (+9.7) |
| Zero-shot Chain w. majority voting + SOLIS | 83.5 (+19.7) |


# W2. Any non-synthetic NLP datasets that have systems of linear equations.

In fact, in addition to the DROP dataset, we also experiment with the AddSub and MultiArith datasets, which are often used to evaluate the numerical reasoning capabilities of language models. These two datasets focus on math problems and have similar data distributions as GSM8K. Here is a typical example of MultiArith:
> Megan had 40 homework problems. She finished 26 of them but still had 2 pages of problems to do. If each page has the same number of problems on it, how many problems are on each page?

To correctly answer the question, SOLIS must predict the corresponding expression `(40 - 26) / 2`, which is non-trivial.
The experimental results of language models are as follows:

| **Backbone** | **AddSub** | **MultiArith** |
|:---|:---|:---|
| Zero-shot Chain | 66.6 | 63.8 |
| Zero-shot Chain w. SOLIS | 89.4 (+22.8) | 80.0 (+16.2) |
| Few-shot Chain | 88.4 | 96.7 |
| Few-shot Chain w. SOLIS | 90.9 (+2.5) | 98.7 (+2.0) |

# W3. The path toward applications beyond systems of linear equations.

Thanks for your constructive feedback! We fully agree that our method would be much more appealing if it is applicable to scenarios beyond linear equations. While we focus on the four fundamental operations that can be solved by linear systems, we are willing to share more about the potential ways to make it beyond linear equations. In short, we think there are two ways: 
- When a problem can be decomposed into several sub-problems and some of these sub-problems can be tackled by linear systems, SOLIS can be applied. 
- We also believe the analytical method can be applied to non-linear systems. For example, by transforming more non-linear systems to linear systems by using virtual variables and even kernel methods, or augmenting the search space of the search-based algorithm with non-linear candidate expressions involving operators like `max`, `min`, and `count`.

From: ICLR.cc/2023/Conference/Paper4250/Authors
title: Response to Reviewer Lb3T
comment: Thank you for the comment. We are glad to answer your concerns.

# W1. Experiment setup is a little bit vague.

Following your suggestion, we have revised the experimental setup part of the paper(Section 5.1) and the corresponding titles(Table 3, Table 4, Table 11, Figure 3) to explain our setup more clearly. In short, we evaluate SOLIS on fine-tuning and few-shot settings on DROP, and we evaluate SOLIS on zero-shot and few-shot settings on MultiArith and AddSub.

# W2. Expressiveness of SOLIS

As the first step of inversely eliciting numerical reasoning, SOLIS only considers linear systems, and does not conduct experiments on more complex scenarios (e.g., aggregation).
However, we believe the search-based algorithm can be extended to more operations such as `max`, `min`, and `count`, where we only need to inject more expressions into the search space.

# Q1. Performance under different setup, and time consumption of different solving strategies

We have presented performance comparison in Table 3 and Table 4. 

As for time consumption, we randomly select 1,000 samples from DROP, and test the average solving time.
- The search-based algorithm and analytical-based algorithm are relatively faster: `0.01s` per sample and `0.03s` per sample respectively
- The heuristic-based algorithm is slower mainly because of the constant searching process: `0.45s` per sample.
- Overall, we believe such speed is acceptable.

From: ICLR.cc/2023/Conference/Paper4250/Authors
title: General Response: Summary Of the Update
comment: We thank all reviewers for appreciating our technical novelty and substantial results, and for offering your valuable comments and constructive suggestions. Following reviwers' suggestions, we have submitted an **updated version** of the paper with augmented experimental results and clarified descriptions. All updates are highlighted in purple.

1) We include a new baseline on DROP, as suggested by Reviewer `qbBT` (Table 3), and add an exploratory experiment which combines majority voting with our method(Appendix C.3).

2) We revise the wording of the paper to clarify the experimental configuration, as suggested from Reviewer `Lb3T`. The updates include the caption of Table 3, Table 4, Table 11 and Figure 3.

3) We respond to all weaknesses raised by all reviewers. We hope these responses address the concerns of reviewers.

From: ICLR.cc/2023/Conference/Paper4250/Reviewer_8HSz
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors introduced a clever and simple method that improves the arithmetic capabilities of LMs.
strength_and_weaknesses: The method is easy to understand and implement and offers improvements on various reasoning datasets. It needs to be clarified, though, whether there exists a path toward applications beyond systems of linear equations. I also would like to know whether the method helps when combined with majority voting.

The Drop dataset used for evaluations in this paper is excellent, but its arithmetic is quite restricted. Are there any non-synthetic NLP datasets that have systems of linear equations? How much gain would we see on GSM8K and MATH? Can you perform a qualitative analysis for Drop, GSM8K, and MATH, highlighting the types of errors that were fixed?
clarity,_quality,_novelty_and_reproducibility: The substitution method proposed by the authors is interesting, novel, and easy to reproduce. I am less convinced about its usefulness, though.
summary_of_the_review: The authors introduced a clever and simple method that improves the arithmetic capabilities of LMs. Assessing the full usefulness of this method may require more work.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4250/Reviewer_Lb3T
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The author proposed a novel method combining a large language model with solving linear systems to improve the calculation over large number performance.
Contributions:
1. A novel approach that takes advantage of neural networks' ability to solve a simple math problem, and generalizes the ability through number substitution and linear system solving.
2. Natural language model is agnostic so the approach is generalizable to different models.
3. Have good performance gain even with zero shot

---------

Update about my final recommendation: 
I am convinced with the rebuttal material and happy to accept this paper. 

strength_and_weaknesses: Strengths: 
1. The paper writing is very clear and the story is convincing.
2. The study using different solving strategies demonstrates the approach can be configured toward different needs
3. The experimental studies show it is pretty generic and can improve different neural baselines

Weakness:
1. The experiment setup is a little bit vague. There are multiple setups, including zero-shot, few shots, and fine-tuning. However, it is hard to tell which setup is used to perform one experiment just from the figure and the caption in the evaluation section. 
2. Expressiveness: The system can only handle up to the linear system, and cannot generalize to aggregation, or recursion yet. 
clarity,_quality,_novelty_and_reproducibility: Clarity: 3.5/5 
Pro: The mathematics and paper writeup is clear, and motivates people well in the introduction section.
Con: The experimental presentation can still be improved. For example, label the experiments with ZS, FS, and fine-tune setups.

Quality: 4/5
The author's statements are well supported by both theory and experiments. To make the evaluation more complete, maybe an ablation study can be performed to see how much performance can be obtained under the ZS, FS, and fine-tuning setup, and compare the time consumption and accuracy using different solving strategies. 

Novelty: 4/5
The idea of converting hard numerical problems into easy ones, and using a linear system to abstract the internal reasoning structure is very interesting. 
summary_of_the_review: The author proposed a novel method combining a large language model with solving linear systems to improve the calculation over large number performance. The experiment shows this model-agnostic approach can improve the performance of different natural language models by a non-trivial amount. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper4250/Reviewer_qbBT
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper proposes SOLIS, where during model inference, complex numbers are substituted with simpler numbers (anchors) from 1 to 20, the language model outputs the arithmetic between the anchors, and through these, the original answer of the arithmetic between the complex numbers are derived through solving linear systems. The operand (to be substituted by anchors) is chosen from the input text through the significance of their perturbations on the model performance. Finally, after having the operands and their anchors, the required output is generated through either one of the proposed 3 approaches to solving linear systems - analytical, search-based, and heuristics. 

Although missing some crucial baselines (discussed in the section below), the authors show that LMs can improve their performance for numeral prediction with the incorporation of their SOLIS framework.

## Edit After Reviewing Author Response

After reviewing the author response, I have updated my recommendation to a weak rejection of the paper.  This is primarily due to the clarification of the goals by the authors and incorporating (and demonstrating improvement on) the GenBERT baseline. The reason for non-acceptance despite the paper having some novel contributions and being well-written has to do with my initial comment on the lack of the proposed model to work with both textual and numeric data which is something (also acknowledged by authors in the response) that some previous works are able to do (see references in `summary of review` below).

strength_and_weaknesses: ## Strengths:
1. The ideation is interesting - authors leverage the fact that LMs are good at arithmetic for numerals 1-20 that are highly frequent in the training corpus and use this for eliciting arithmetic operations between more complex numbers.


2. The ideas are clearly expressed in the writing/figures/tables and thus are easy to grasp.

## Weakness:

Among the two weaknesses of the paper, the first is a deal-breaker. The authors establish the premise for the need of numeric comprehension in language models as 1) numeric reasoning forms the crux of understanding text, tables, and knowledge bases 2) extrapolation for numerals outside of the pre-training corpora. Here, the approach that the authors have taken simply appends a system on the top of untouched LMs that does the computations for the LM. This appendage does improve performance when the task is strictly to output a number. However, tracing this back to the original premise of the paper, unless the representations of numbers of these LMs change or adapt, this appendage makes no difference in the understanding of numerals by the LM. Case in point, a DROP dataset instance where the LM has to understand numerals in the paragraph but the output is textual - I believe this appendage provides no significance in the understanding of numerals. This is further corroborated by the paper itself - section 5.3 - where the authors state that for their experimentation, they have specifically chosen instances of the DROP dataset that have numeric outcomes.

Two essential works that also aim to improve numerical comprehension and also evaluate themselves on the DROP dataset (including instances where the output is textual), one of which is cited in the paper, is not used as baselines:

1. Geva, M., Gupta, A., & Berant, J. (2020). Injecting numerical reasoning skills into language models.

2. Sundararaman, Dhanasekar, et al. "Methods for Numeracy-Preserving Word Embeddings." Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020. 


clarity,_quality,_novelty_and_reproducibility: The quality and clarity of the paper is good - it is written and presented well. The proposed solutions are original, however, are not pertinent to the problem statement.
summary_of_the_review: The lack of important baselines I can overlook as the idea is interesting, however, I base my recommendation on the first part of the paper weakness - the offered solution is antithetical to the premise of the paper. If the problem statement is that language models struggle to understand numbers that appear in tables, text, and knowledge bases, the solution cannot be to approximate a calculator as an addition to the inference chain. To elaborate, take a supposed instance of any QA dataset (SQuAD or DROP) where the context is "X scored 4 goals while Y managed to only score 3", the question is "Who scored more goals?" and the answer is textual - "X". The framework suggested can compute arithmetic between 4 and 3, but does not help the LM better understand what 4 and 3 are such that it can make an informed decision. There are several lines of work that work on the premise that this paper takes, but provide interventions that can be applied to both textual and numeric data:

1. Spithourakis, Georgios, and Sebastian Riedel. "Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numbers." Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018.

2. Jiang, Chengyue, et al. "Learning Numeral Embedding." Findings of the Association for Computational Linguistics: EMNLP 2020. 2020.

3. Zhang, Xikun, et al. "Do Language Embeddings capture Scales?." Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP. 2020.

4. Geva, M., Gupta, A., & Berant, J. (2020). Injecting numerical reasoning skills into language models.

5. Sundararaman, Dhanasekar, et al. "Methods for Numeracy-Preserving Word Embeddings." Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020. 

correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4250/Reviewer_Dv6a
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper proposed a method to identify relevant numbers in text, replace them with smaller numbers that are more accurate to the LM, and use the input-output pair to search for a linear system.
strength_and_weaknesses: Strength:
1. Novelty is high in my opinion. Numerical instability is a long-standing problem in neural-based reasoning. While most research attempts to use neural-symbolic methods, this paper figured out a way to identify relevant numbers and outsource the finding solution steps to classicial methods.
2. Experiments are through with good analysis. Comprehensive ablation studies are performed to identify the contributions of variations in LMs and different linear system search methods.

Weakness:
1. Currently the method is limited to linear system, which probably is OK since most arithmetic in natural language text are not complex. But there are areas where non-linear solutions are required (imagine solving high-school math), how can the method be extended to that case?
2. It would be great if the authors can show some failure cases. I wonder which step leads to the most failures? Is it finding the right number in the text, or the search for system solution?
clarity,_quality,_novelty_and_reproducibility: The paper is well written with concise explanations and good amount of illustrations. Quality is high with good novelty. Not reproducible though as code is not provided, and it is not trivial to reproduce the paper from scratch.
summary_of_the_review: The proposed method is novel and thoroughly evaluated, and proven to be effective. It is however not clear if the method can be extended to more complex numerical reasoning tasks.

-----------After rebuttal--------------

I am happy with the rebuttal and I still recommend acceptance for this paper.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper