Solving Quantitative Reasoning Problems with Language Models
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Language models have achieved remarkable performance on a wide range of tasksthat require natural language understanding.
Nevertheless, state-of-the-art modelshave generally struggled with tasks that require quantitative reasoning, such assolving mathematics, science, and engineering problems at the college level.
Tohelp close this gap, we introduce Minerva, a large language model pretrained ongeneral natural language data and further trained on technical content.
The modelachieves strong performance in a variety of evaluations, including state-of-the-artperformance on the MATH dataset.
We also evaluate our model on over twohundred undergraduate-level problems in physics, biology, chemistry, economics,and other sciences that require quantitative reasoning, and ﬁnd that the model cancorrectly answer nearly a quarter of them
Introduction
Artiﬁcial neural networks have seen remarkable success in a variety of domains, such as computervision, speech recognition, audio and image generation, translation, playing games, and robotics.
Inparticular, large language models have achieved excellent performance across a variety of naturallanguage tasks including common-sense reasoning, question answering, and summarization [1, 2, 3,4, 5].
However, these models have struggled with tasks that require quantitative reasoning, such assolving mathematics, science, and engineering problems [6, 7].
Quantitative reasoning problems are an interesting domain of application for language models becausethey test the capability of models on several fronts.
They require the solver to correctly parse anatural language input, potentially recall world knowledge that pertains to the problem, and applyan algorithm or series of computations to the information provided in order to arrive at a correctsolution.
They also require that the solver is able to correctly parse and generate precise sequencesof mathematical tokens, as well as apply a computational procedure to tokens via symbolic andnumerical manipulation.
Finally, such problems are a proving ground for research toward robustquantitative reasoning solvers that are useful in supporting the work of humans in scientiﬁc andtechnical ﬁelds.
Previous research has shown that large language models achieve impressive performance on math andprogramming questions after training on domain speciﬁc datasets [8, 9, 10].
In this work, we applythis approach to quantitative reasoning problems, in which the model is required to provide a completeand self-contained solution, without relying on external tools.
Such tasks include mathematics wordproblems, competition mathematics evaluations, and many problems in science and engineering.
Question:
A line parallel to y = 4 x + 6 passes through (5 , 10) .
What is the ycoordinate of the point where this line crosses the y -axis?
Target: Since the line is parallel to y = 4 x + 6 , we know that its slope is 4.
Thus the equation of the line is y = 4 x + b where b is the y -intercept.
Plugging (5 , 10) into this gives 10 = 4 · 5 + b ) b =  10 which is what we wanted.
Model output: The line is parallel to y = 4 x +6 , which means that it has the same slope as y
= 4 x + 6 .
Since the line passes through (5 , 10) , we can use the point-slope form of a line to ﬁnd the equation of the line:
[Equation]
Now that we have the equation of the line, we can ﬁnd the y -coordinate of the point where the line crosses the y -axis by substituting x = 0 into the equation:
[Equation]
Question: Assume that variance of ﬁrst n natural numbers is 10 and variance of ﬁrst m even natural numbers is 16 .
Compute m + n .
Model output: Variance of ﬁrst n natural numbers is n 2  1 12 .
Variance of ﬁrst m even natural numbers is 2 2 ( m 2  1).
So, n 2  1 12 =and 2 2 ( m 2  1)= 16 .
Solving for n and m , we get n = 11 and m = 7 .
So, m + n = 18 .
Question: Prove that for every a, b , b 6 = a , holds a 2 + b 2>  a + b 2 .
Model output: We have
[Equation]
We present Minerva, a language model that achieves strong performance on many quantitativereasoning tasks. Our model is able to process scientiﬁc and mathematical questions formulated innatural language, and generate step-by-step solutions using correct L A TEX notation.
Figures 1 and 2show example responses from Minerva to mathematics and physics questions.
Minerva is based on a pretrained general language model (which we call BaseModel 1 ) that is furthertrained on a high-quality dataset containing scientiﬁc and mathematical data.
We start with 8B and62B parameter models and continue training them on our technical content dataset.
We achievestate-of-the-art performance on the MATH dataset and strong performance on other math tasksformulated in natural language, including GSM8k, while using less compute than previous models.
Notably, our models can achieve robust performance in a few-shot setting without being explicitlytrained on these evaluation datasets.
The main novelty of this paper is a large training dataset that juxtaposes natural language with thecorrect use of formal language, such as equations and diagrams.
The data is collected from the arXivpreprint server and from web pages that we carefully process to minimise the loss of mathematicalcontent.
In addition to establishing a new SOTA result on the MATH dataset, this work sets a
Question: A particle moves so that it is at  3 sin t 4 , 3 cos t at time t. Find the speed of the particle, measured in unit of distance per unit of time.
Target:
The parametric curve ( x, y ) =
3 sin t 4 , 3 cos t describes a circle with radius 3.
Furthermore, it makes a full revolution at time t = 8 ⇡ .
(ﬁgure)
Therefore, the speed of the particle is 6 ⇡ 8 ⇡ = 3 4 .
Model output: The speed is the magnitude of the velocity vector.
Therefore, we must ﬁrst ﬁnd the velocity vector, which is the derivative of the position vector.
We have
[Equation]
Therefore, the magnitude of the velocity vector is
[Equation]
Therefore, the speed of the particle is 3 4 .
new lower bound for the performance achievable on quantitative reasoning benchmarks simply byincreasing data quality.
Existing benchmarks are limited with respect to quantitative reasoning; for this reason, we build adataset of over 200 undergraduate-level questions in science and mathematics from MIT’s OpenCourseWare (OCW); this provides a measure of our model’s quantitative reasoning abilities in achain-of-thought context beyond a pure mathematical setting.
1.2 Related Works
Solving quantitative reasoning problems expressed in natural language has been an active area ofstudy [11, 12].
Prompting language models using chain-of-thought solutions can lead them to outputstep-by-step reasoning to unseen problems [13].
The GSM8k dataset [7] showed that training veriﬁersto rerank model outputs can lead to improved performance.
The original version of GSM8k includedspecial syntax for algebraic calculations, which were processed by a calculator.
In this work we focuson self-contained models without access to external tools.
The standard method for evaluating language models on generative tasks is to greedily sample onesolution per problem.
Recent works [8, 14, 15, 16] have shown that it is advantageous to samplemultiple solutions per problem, and then ﬁlter those down to a ﬁnal answer.
We ﬁnd that majorityvoting [16] signiﬁcantly improves performance over greedy decoding.
Code generation.
Applying code generating models to mathematical problems has been an activearea of exploration.
PaLM [5] showed that a large language model with code in its training datasetcan achieve good performance on a code version of GSM8k.
Furthermore, by manually constructingprompts [10], the Codex model [8] can generate code solutions to MATH [6] problems.
Thesesolutions often rely on external libraries to perform mathematical operations such as solving equationsor taking limits.
This is a complementary approach to ours, in which we directly probe the model’scapability to arrive at an answer by relying only on its own reasoning capability.
Formal mathematics.
Mathematics developed as a discipline based in natural language, but itsaxiomatic fundamentals make it possible to simulate mathematical thinking.
This can be achievedusing specialized programming languages that facilitate the simulation of logical and mathematicalthinking using a computer, such as Coq [17], Isabelle [18], HOL4 [19], Lean [20], Metamath [21]and Mizar [22].
Work on automation of proof assistants and automated theorem provers such as
E [23], leanCoP [24], and Vampire [25] has substantially beneﬁted from integration with machinelearning methods [26, 27, 28, 29, 30].
Language models applied to formal and synthetic mathematical problems.
Previous worktrained language models to predict mathematical expressions [31, 28, 29, 32, 33].
In turn, such apredictive model can be used to guide a proof search, as done by [29].
Large language models excelin modelling natural language, though in the case of formal languages, models that facilitate retaininginformation about the graph structure of a given mathematical formula, such as GNNs, are still verycompetitive.
Modelling mathematics as a discipline of natural language.
New benchmark datasets [6, 34]cover more advanced mathematical topics.
In this domain language models are facing limitedcompetition from other classes of models.
Training and Evaluation2.1 Mathematical Training Dataset
Our models were trained on a dataset that includes 38.5B tokens from webpages ﬁltered formathematical content and from papers submitted to the arXiv preprint server.
In addition, the datasetincludes a general natural language dataset, which is the same on which BaseModel was pretrained.
Our mathematical webpage dataset was constructed by collecting pages that contain mathematicalexpressions in MathJax format.
The pages underwent a cleaning process that removes most HTMLtags but preserves mathematical notation, including L A TEX symbols and formatting, such that mathematical formulae like e ⇡ i + 1 = 0 or E = mc 2 are presented in full to the model.
This proceduremakes it possible for the model to perform well on tasks that require calculation and symbolicmanipulation.
Table 1 provides a breakdown of the training dataset.
See appendix for more details.
[Table]
2.2 Models and Training Procedure
Our approach is to ﬁnetune pretrained decoder-only Transformer language models on our mathematical dataset using an autoregressive objective.
Table 2 contains the main model and traininghyperparameters.
Additional details can be found in the appendix.
[Table]
2.3 Evaluation Datasets
[Figure] We focus on few shot evaluation. See appendix for a discussion of ﬁnetuning.
For evaluation, wetruncate the inputs from the left to 1024 tokens and we generate up to 512 tokens.
When samplingonce per problem, we sample greedily.
When sampling multiple times per problem we use nucleussampling [35] with temperature T = 0 . 6 , p = 0 .
For generative tasks, the model produces achain-of-thought and demarcates a ﬁnal answer.
We evaluate a solution as correct if the ﬁnal answermatches with the ground truth solution, independent of the quality of the chain-of-thought precedingit.
See appendix for further evaluation details.
To evaluate correctness, we parse the ﬁnal answersand compare them using the SymPy library [36].
This is done in order to correctly identify answersthat are mathematically equivalent such as 1 / p 3 and p 3 / 3 .
See appendix for further details.
The existing datasets on which we focus are:
• MATH: a dataset of 12K middle school and high school mathematics problems [6].
Problem
statements are written in L A TEX.
We prompt the model with a ﬁxed 4-shot prompt (listed in
the appendix).
This prompt includes four random examples from the training dataset whose
ground truth targets are not too long.
• GSM8k: middle school math word problems [7].
Models are evaluated using the chain-of-
thought prompt from Wei et al.
Previous models evaluated on GSM8k made use of an external calculator.
In this work, our model does not have access to any external tools.
• MMLU-STEM: subset of the MMLU dataset [37] focused on science, technology, engineer-
ing, and mathematics (STEM).
We use the 5-shot prompt from the development set for each
task.
It is multiple choice and does not include a chain-of-thought.
2.4 Undergraduate-Level STEM Problems
To evaluate the scientiﬁc reasoning capabilities of Minerva, we harvested a set of STEM problems atthe undergraduate level, most of which involve multi-step reasoning, which we refer to in this paperas OCWCourses.
From publicly-available course materials offered by MIT (OpenCourseWare), wecollected problems with automatically-veriﬁable solutions (either numeric or symbolically veriﬁablevia SymPy ) from courses including ‘solid-state chemistry’, ‘information and entropy’, ‘differentialequations’, and ‘special relativity.’
These were processed by contractors to be self-contained and tohave a clearly-delineated ﬁnal answer (i.e., problems asking for a proof or open-ended short answerwere not included).
In total we curated 272 problems, 191 of which have numeric solutions and81 with symbolic solutions.
In the appendix, we detail the contributions from each course, and the process of converting these course materials into a format suitable for processing by language models.
We also provide the text of all problems.
We plan to release these as part of an open-source datasetwhich will be detailed in an upcoming manuscript.
2.5 Inference-Time Techniques
We ﬁnd that we can considerably outperform greedy decoding by sampling k > 1 solutions (witha non-zero temperature) and selecting one using majority voting [16].
This consists of groupingpredictions with respect to their ﬁnal answer and selecting the most common answer.
Following thenotation of previous work [14], we denote this as maj1@k .
A variation of this algorithm, denotedmajn@k , involves selecting the n most common answers.
Intuitively, the reason this reranking algorithm improves performance is that while there are manyways to answer a question incorrectly, there are typically very few ways to answer correctly.
This is incontrast to pass@k , where a task is considered solved if any single sample solves it.
See Section 4.2for more details about pass@k .
In the appendix, we report how performance depends on k fordifferent metrics.
The results in Table 3 are based on sampling with maj1@k ; we report on maj5@k in the appendix.
Even when pass@k continues to improve as k is increased, majority voting saturates faster: 97%of the large k accuracy is achieved at k = 64 for MATH and k = 16 for GSM8k.
This is becausemajority voting selects the most common answer in the modeled distribution, and the error ofthis estimate decreases with increasing k .
This is in contrast to pass@k where the performanceimprovement comes from the tail of the distribution, which can keep improving as k is increased.
An alternative is to rerank samples with respect to their negative log-likelihood.
We found thatmajority voting performs signiﬁcantly better than log-likelihood reranking.
Results
Table 3 summarizes the results for our 8B and 62B models on the evaluation datasets described in Section 2.3.
The inference techniques we tried were described in Section 2.5, and in Table 3 wepresent only the best-performing technique.
We use k = 64 for OCWCourses, k = 256 for MATH,and k = 100 for GSM8k. Figure 3 presents a breakdown of the MATH dataset results by subtopic.
We present model output samples in Figures 1 and 2.
In addition, we evaluated Minerva 62B on the National Math Exam in Poland and found that it achieves a score of 57% , which happens to be thenational average in 2021 [38, p. 23].
We include results on the latest publicly available language model from OpenAI, davinci-002,evaluated using the OpenAI API with temperature set to the ofﬁcial recommendation ( T = 0 . 2 ).
Note that Minerva 62B outperforms davinci-002 and signiﬁcantly outperforms the state-of-the-arton MATH, one of the hardest of quantitative reasoning datasets to date. GSM8k is an easier dataset,and while our mathematical training helps signiﬁcantly, it does not enable the model to achievestate-of-the-art performance, which was achieved by a model with 9x the number of parameters. MMLU-STEM is a multiple choice dataset, on which Minerva achieves similar performance to Chinchilla [39].
Chinchilla is a considerably stronger base model as it uses roughly twice as muchcompute during pretraining as BaseModel, but our mathematical training helps bridge the gap.
While our main focus is on few shot evaluation, we also tried to ﬁnetune Minerva on MATH. Whilewe did not observe any improvement, we found that ﬁnetuning BaseModel on MATH did give asigniﬁcant improvement, which suggests that the marginal utility of standard ﬁnetuning decreasesas the quality and diversity of the unsupervised ﬁnetuning dataset improves.
Further details can befound in the appendix.
Performance Analysis
4.1 Model Mistakes
To better understand the types of mistakes our models make, we compare the performance of
Minerva 8B and Minerva 62B on 216 problems with high conﬁdence majority decisions of both [Table] models.
Speciﬁcally, we selected examples where the top answer received at least 15% of votes [Table] andthat either Minerva 8B was correct and Minerva 62B was incorrect (15 samples), or vice versa (201samples).
The categories and examples for each category are described in the appendix
[Table] As shown in Table [Table] the prevailing errors of the 8B model were related to incorrect reasoning orcalculations.
Many of the calculation errors were relatively benign mistakes in arithmetic.
Solutionsthat were too short were relatively rare (in these cases, the model immediately produces an incorrectanswer without any intermediate reasoning steps).
Finally, in a few cases, the model hallucinates anequation or mathematical fact that is not real.
In the samples where the 62B model was incorrect, the dominating failure modes were again incorrectreasoning and incorrect calculations.
In summary, we ﬁnd that the 62B Minerva model retains most of the skills of the 8B model andimproves upon both reasoning and calculation robustness.
[Table]
In our approach to solving quantitative reasoning problems, we are able to automatically verifywhether the ﬁnal answer to a problem is correct, but we do not have an automatic way to verify themodel’s chain of reasoning.
This leaves open the possibility of false positives: samples which havethe correct ﬁnal answer, but for which the reasoning is incomplete or incorrect.
We selected 100 random questions from MATH (20 per difﬁculty level), along with answers sampledat zero temperature from the 62B model.
We then manually inspected the answers to determinethe false positive rate, which is the ratio between number of false positive examples and number ofexamples for which the ﬁnal answer is correct; see Table 5.
We found that the overall false positiverate is low, though it does increase with difﬁculty level.
Our focus on pass@1 and majority voting as the primary evaluation metrics is due in part to thefact that they are less susceptible to false positives than pass@k [14].
While the pass@256 accuracyis 84 .
5% for the 62B model, false positives account for part of it.
We inspected the samples thatfailed in majority voting but passed on pass@k due to a single correct answer, and estimate the falsepositive rate for pass@256 to be 30% among samples selected in this way.
After removing falsepositives, we estimate that the pass@256 accuracy to be bigger than 68%; see appendix for details. [Table]
Memorization
A central question in interpreting Minerva’s solutions is whether performance reﬂects genuine analyticcapability or instead rote memorization.
This is especially relevant as there has been much prior workindicating that language models often memorize some fraction of their training data [40, 41, 42].
When examining model solutions, we ﬁnd that memorization of intermediate facts, such as numericalvalues of square roots or trigonometric identities, are crucial elements of model solutions.
Trulystrong performance would combine recall of intermediate facts with genuine solution synthesis.
Wewould like to investigate a strong form of memorization, where model performance is a result ofmemorizing the explicit problems and solutions in our evaluation set, but also a weaker form, wherethe model has memorized alternate answers to the same questions.
In order to evaluate the degree to which our models solve problems by recalling information memorized from training data, we conduct three analyses on the MATH dataset.
First we directly searchfor problems and solutions in our training corpus.
Next, we generate modiﬁed versions of problemsand evaluate our models’ robustness to these changes.
Finally, we measure the degree of overlapbetween the ground truth solutions and solutions generated by our model and measure the effect ofthis similarity on model performance.
Overall, we ﬁnd little evidence that the model’s performancecan be attributed to memorization.
5.1 Training and Evaluation Dataset Overlap
We selected the problems for which our 62B parameter model produced a correct answer, and ﬁlteredthem to the 100 problems with the highest majority vote score, expecting that problems with a highmajority vote score are more likely to have been memorized.
For each of these question-answerpairs, we compute the BLEU score across chunks of 500 characters in our Math Web Pages dataset(a histogram of the BLEU scores is shown in the appendix).
We then manually inspect the 250documents with the highest BLEU scores.
While many of the top matches were from homework helpsites with math questions and solutions, none of the questions matched the questions in the subset of
MATH under consideration.
We have included these 250 segments in the appendix.
We note thatsome problems from MATH can be found on the web.
Nevertheless, this analysis concludes thatthese problems did not make it through our data collection process.
5.2 Performance on Modiﬁed MATH Problems
To further investigate memorization, we randomly selected twenty problems which the 62B modelanswered correctly under majority voting.
We manually modiﬁed each problem either by introducingminor changes to problem wording (framing) or by changing the numbers which appeared in theproblem and modifying the solution accordingly.
We then compared the accuracy over sampledsolutions before and after the modiﬁcation.
Results are shown in Figure 4.
In both cases the accuracybefore and after modiﬁcations are correlated, with no clear bias in favor of the original formulation.
This is suggestive of minimal memorization.
The modiﬁed problems are listed in the appendix.
5.3 BLEU Score Between Ground Truth and Generated Solutions
We seek to detect memorization of solutions by computing BLEU score between ground truth answersand model generated answers.
We use the 62B model and analyze 256 samples per problem in the MATH dataset.
First, we compute overlap statistics for all correct samples.
We ﬁnd that 160 out of5,000 test questions have a sample with a BLEU score greater than or equal to 80 (see appendix).
Wenote that they tend to be short solutions.
To understand the effect of answer similarity on performance,we remove model samples above a certain BLEU score threshold, and recompute the majority voteaccuracy.
We ﬁnd that majority vote performance is robust even down to relatively low similarities(see Figure 4), indicating that performance cannot be attributed to model outputs that are very similarto ground truth answers.
Conclusions and Discussion
In this work, we take an approach to quantitative reasoning that relies on solving problems usingmathematical reasoning expressed in natural language.
We show that by training a large languagemodel on a high quality mathematical dataset, we are able to achieve strong performance on tasksthat require logical reasoning, numerical calculation, and symbolic manipulation.
Our model doesnot make use of external tools, and at inference time relies exclusively on autoregressive samplingto achieve this performance.
Complementary approaches to quantitative reasoning include codegenerating models and formal methods.
These are all different routes toward a common goal: anagent that can reason about and solve quantitative problems.
We believe that such an agent shouldcombine useful elements from all of these approaches.
6.1 Limitations of Our Approach
Our approach to quantitative reasoning has several limitations.
First, we have no automatic wayof verifying the correctness of the model’s answers.
This is in contrast to formal approaches, forwhich automatic veriﬁcation is intrinsic.
Second, our model has no access to external tools such as acalculator or a Python interpreter.
It is therefore limited in its ability to perform quantitative reasoningtasks that require complicated numerical calculations.
Third, because our model was trained on a largeamount of data, we have little direct control over the speciﬁc capabilities that the model acquired.
6.2 Societal Impact
Artiﬁcial neural networks capable of solving quantitative reasoning problems in a general setting havethe potential of substantial societal impact.
Minerva, while a step in this direction, is still far fromachieving this goal, and its potential societal impact is therefore limited.
The model’s performance isstill well below human performance, and furthermore, we do not have an automatic way of verifyingthe correctness of its outputs.
If these issues could be solved, we expect the impacts of this modelto be broadly positive.
A direct application could be an accessible and affordable math tutor whichcould help improve educational inequalities.