M IND ’ S E YE :
G ROUNDED L ANGUAGE M ODEL R EA -
SONING THROUGH S IMULATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Successful and effective communication between humans and AI relies on a shared experience of the world.
By training solely on written text, current language models (LMs) miss the grounded experience of humans in the real-world—their failure to relate language to the physical world causes knowledge to be misrepresented and obvious mistakes in their reasoning.
We present Mind’s Eye, a novel paradigm to ground language model reasoning in the physical world.
Given a physical reasoning question, we use a computational physics engine (DeepMind’s MuJoCo) to simulate the possible outcomes, and then use the simulation results as part of the input, which enables language models to perform grouded reasoning.
Experiments on 39 tasks in a physics alignment benchmark demonstrate that Mind’s Eye can improve reasoning ability by a large margin (27.9% zero-shot, and 46.0% few-shot absolute accuracy improvement on average).
Smaller language models armed with
Mind’s Eye can obtain similar performance to models that are 100 × larger.
Finally, we conﬁrm the robustness of Mind’s Eye through ablation studies.
I NTRODUCTION
“ In questions of science, the authority of a thousand is not worth the humble reasoning of a single individual. ”
——Galileo Galilei,“Do objects fall proportionately to their weight?”
This famous question was once controversial until
Galileo’s Leaning Tower of Pisa experiment 1 —Galileo dropped two balls of different masses from the same height (i.e., experiment ) and concluded that their time of descent was independent of their mass (i.e., reasoning ).
Such an experiment-reasoning paradigm has been used by humans for centuries to ground reasoning on complicated problems (Newell, 1980) and transfer learned knowledge to unfamiliar domains (Novak & Gowin, 1984).
Current language models (LMs) follow a different path—by training on natural language, they attempt to reverse engineer the physical world, so that they are able to perform reasoning about it.
Large-scale pre-trained LMs have achieved revolutionary performance on many tasks, such as solving math word problems (Roy & Roth, 2015;
Ling et al., 2017; Cobbe et al., 2021) and commonsense reasoning (Talmor et al., 2022; Geva et al., 2021).
However, these models do not experience ﬁrsthand the situations that are described by the language (McClelland et al., 2020), and lack the ability to ﬁnd the correct answers by performing actual experiments like humans.
As a consequence, when asked the same free fall question, one of the most widely-used LMs, GPT-3 2 (Brown et al., 2020)—though achieving superhuman performance in many reasoning tasks—will generate the wrong answer: “The heavier object will fall faster.”
(as shown in Figure 1).
Due to the lack of grounded reasoning, current
LMs also have issues in truthfulness (Lin et al., 2021) and factuality (Petroni et al., 2020).
To tackle these problems, existing remedies include using improved prompting techniques, such as inserting hand-written decomposed reasoning steps in few-shot demonstrations (Wei et al., 2022;
Question:
TwobaseballsXandYarereleasedfromrestatthesameheight.
XisheavierthanY. Whichbaseballwillfalltothegroundfaster?
AnswerfromVanillaLM: Answer:
Theheavierbaseballwillfalltothegroundfasterbecauseithasmoremassandthereforemoregravity.
Zero-shotReasoning
Question:
TwobaseballsXandYarereleasedfromrestatthesameheight.
XisheavierthanY. Whichbaseballwillfalltothegroundfaster?
[Figure]
Question:
TwobaseballsXandYarereleasedfromrestatthesameheight.
XisheavierthanY. Whichbaseballwillfalltothegroundfaster? +
[Figure]
of [Figure] ThoughtonSimilarQuestions
[Figure]
Sincetheaccelerationofanobjectcanbecomputedasa=F/
[Figure]
,theheavieronewillhavemoregravity,
[Figure]
: XandYhavethesameacceleration.
[Figure]
Vanilla
Zhou et al., 2022).
These methods are inherently limited as their reasoning ability completely relies on the knowledge perpetuated in the LM—their performance could suffer if the knowledge learnt by the LM is incorrect (Petroni et al., 2019) or outdated (Dhingra et al., 2022).
To incorporate external knowledge, retrieval-augmented LMs such as REALM (Guu et al., 2020), RAG (Lewis et al., 2020) and RETRO (Borgeaud et al., 2022), retrieve relevant documents as additional evidence for given questions, and may also ﬁne-tune the LM on the question-document-answer triplets.
However, the knowledge presented in written language is known to have reporting bias (Bisk et al., 2020), whereby some everyday unspoken facts or rarely seen (but practically possible)
compositions are commonly missing in text (Paik et al., 2021).
Correct and complete understanding of properties and interactions in the physical world is not only essential to achieve human-level reasoning (Lake et al., 2017), but also fundamental to build a general-purpose embodied intelligence (Huang et al., 2022).
In this work, we investigate to what extent current LMs understand the basic rules and principles of the physical world, and describe how to ground their reasoning with the aid of simulation.
Our contributions are three-fold:
• We propose a new multi-task physics alignment dataset, U TOPIA , whose aim is to benchmark how well current LMs can understand and reason over some basic laws of physics (§2).
The dataset contains 39 sub-tasks covering six common scenes that involve understanding basic principles of physics (e.g., conservation of momentum in elastic collisions), and all the ground-truth answers are automatically generated by a physics engine.
We ﬁnd that current large-scale LMs are still quite limited on many basic physics-related questions (24% accuracy of GPT-3 175B in zero-shot, and 38.2% in few-shot).
• We explore a paradigm that adds physics simulation to the LM reasoning pipeline named Mind’s Eye (§3), to make the reasoning grounded within the physical world.
Speciﬁcally, we ﬁrst use a model to transform the given text-form question into rendering code, and then run the corresponding simulation on a physics engine (i.e ., MuJoCo (Todorov et al., 2012)).
Finally we append the simulation results to the input prompts of LMs during inference.
Our method can serve as a plug-and-play framework that works with any LM and requires neither handcrafted prompts nor costly ﬁne-tuning.
• We systematically evaluate the performance of popular LMs in different sizes on U TOPIA before and after augmentation by Mind’s Eye, and compare the augmented performance with many existing approaches (§4.2).
We ﬁnd Mind’s Eye outperforms other methods by a large margin in both zero-shot and few-shot settings.
More importantly, Mind’s Eye is also effective for small LMs, and the performance with small LMs can be on par or even outperform that of 100 × larger vanilla LMs.
U TOPIA B ENCHMARKING
Humans are able to understand their physical environment and intuit rules about the world from embodied experience.
The rules and principles behind the real world have been discovered and concluded as scientiﬁc laws—we humans have ingrained them as knowledge or intuition (Kaiser et al., 1986) to make reliable predictions on how observed events will unfold in day-to-day life (Kubricht et al., 2017).
For example, when driving, we can anticipate when to brake when approaching a stop sign, using intuition or knowledge from Newton’s second law of motion.
We also know it would be a disaster to collide with a heavy truck, not only in terms of our knowledge on the conservation of momentum (i.e., the lighter object will have a greater velocity after collision), but also from our embodied experience of collision in everyday life.
We are thus inspired to design a physics alignment dataset that covers this knowledge, aiming to benchmark to what extent current LMs understand basic physical concepts and rules.
As shown in
Table 1, we choose six representative scenes, mainly from textbooks (e.g., high-school Physics).
The sub-tasks are deﬁned based on the composition of observed and queried concepts.
For example, one task in a motion scene could be: given the observed acceleration of the two objects with the same mass, please answer what is the relationship of forces applied on them.
In total we have 39 sub-tasks across different scenes, and each sub-task contains various hand-designed questions whose language style is similar to that of textbooks as well
[Table]
[Table]
[Table]
velocitymassmomentum
[Table]
massheightfriction
Table 1 exempliﬁes some samples in U TOPIA .
We deliberately choose to use relative comparison (e.g., “greater than” , “smaller than” , “the same as” ; text in purple) rather than actual numbers to describe the physical properties, since we are thus able to disentangle the effects from numeracy (i.e., the gain on reasoning is not attributed to better memorization on numbers, which has been reported as “shortcuts” used by LMs (Patel et al., 2021)).
This setting is also different from those in mathematical reasoning tasks (e.g., GSM8k (Cobbe et al., 2021)), where the decomposed reasoning path is typically the procedure of plugging different values into equations—the LM might be able to solve these problems by symbolic manipulation (Razeghi et al., 2022) rather than actual reasoning.
Most existing physics alignment datasets use vision as the primary modality, such as images (Zellers et al., 2019), animations (Wu et al., 2017), or videos (Piloto et al., 2022), which loses the ﬂexibility to run on LMs which only takes text input.
PIQA (Bisk et al., 2020) and MMLU-Physics (Hendrycks et al., 2021) are popular physics reasoning datasets used for LM benchmarking; however, their sizes are naturally limited because of required human annotations (e.g., only 206 samples are on physics in MMLU, with college and high school level questions combined).
U TOPIA differs from all these datasets as it leverages a physics engine to generate data—in theory we can obtain unlimited samples—and each sample has reliable ground-truth supported by actual simulation.
Although in the present work we only take the text-form data for LM benchmarking, the corresponding simulation videos during data generation have been recorded as data for future multi-modality research.
M IND ’ S E YE
As shown in Figure 1, Mind’s Eye comprises three main components, a text-to-code LM as the frontend, a physics simulation engine (i.e., MuJoCo) as the back-end, and a foundation model (Bommasani et al., 2021) for general reasoning.
We detail the implementation of Mind’s Eye as below:
Text-to-Code Converter.
The objects and dynamics of the simulation is manifested by the rendering code fed into MuJoCo.
The rendering code is written in a type of XML ﬁle named MCJF 3 , where the physics properties can be easily controlled by changing some key-value pairs.
For example, to change the mass of an object to 10, the line of rendering code needed is geom.set(’mass’, ’10’) .
We use actual values to express the relative relationships in U TOPIA (e.g., “greater” will be translated to 10 and 1 for the values of the properties to be set).
We create rendering templates for each sub-task of U TOPIA , and use programs to generate a dataset with 200,000 text-code pairs.
In each pair, the question in text is appended to the top of the XML code as comments.
We then train decoder-only LMs from scratch to learn how to generate the rendering code given the question in comments auto-regressively.
We leverage the BPE vocabulary set from GPT-2 (Radford et al., 2019) and extend it by several special tokens to represent repeating tabs or spaces.
Besides ﬁne-tuning on the dataset with text-code pairs, we also pre-train the model on the C4 dataset (Raffel et al., 2019a) to enhance the model’s understanding on natural language.
All the training is on TPU-v3 Pods and the resulting models have 0.3B and 1.5B parameters (used as default).
See §4.1 for training details.
Simulation Augmented Prompting.
Once receiving the rendering code, the physics engine will run the corresponding simulation to get the ground-truth outcome.
The program that triggers the simulation will also parse the outcome into text-form prompt injections (e.g., “Hints: X and Y have the acceleration.” , as shown in Figure 1).
The injection combined with the question will be fed to the foundation model, with which LMs can ground their reasoning with the physical world rendered by the physics engine.
We present more details of this procedure in §A.1.
The intuition behind Mind’s Eye is to imitate the experiment-reasoning paradigm; however, we leverage quick and cheap physics simulation as an alternative to actual experiments in physical world.
The cognitive analog for Mind’s Eye might be the mental visualization process, also known as “the mind’s eye” (Battaglia et al., 2013; Hegarty, 2004), which often relates to motor processes (Wexler et al., 1998) during embodied reasoning (Nathan et al., 2021; Tamari et al., 2020).
Discussion: Why does Mind’s Eye work?
Table 2 shows the comparison between Mind’s Eye and two other methods in the formulation of the grounding process during LM inference.
Assuming knowledge of the physical world aligns with the distribution p World , the Zero-shot Reasoner (Kojima et al., 2022) which uses “Let’s think step by step.”
in prompts can be extended to any number of new tasks.
However, its reasoning ability will be compromised if the knowledge in LMs is incorrect or outdated.
Similarly, incorporating handcrafted reasoning steps rather than a generic phrase, Chainof-Thought (Wei et al., 2022) is able to elicit LM reasoning in a more explicit way; however, its
[Table]
Mind’s Eye overcomes these issues by including a simulator into the reasoning pipeline.
Given the question, the simulator (i.e., the physics engine) returns the most likely outcome based on its encoded world knowledge.
Since the simulator is accurate enough to approximate the physical world, the prompt injection of Mind’s Eye basically serves as a scoring machine, which puts probability mass on the answer that is best aligned with the rules of physics—the LM reasoning over the injected rationales is thus grounded.
Mind’s Eye is also scalable since the whole pipeline is automated.
Besides scalability and grounded reasoning, Mind’s Eye is also efﬁcient.
Mind’s Eye delegates domain-speciﬁc knowledge to external expert models (i.e., the MuJoCo engine for expert physics knowledge) (Du et al., 2022; Shazeer et al., 2017), which decouples general reasoning from domain speciﬁc knowledge.
The size of the LM can thus be signiﬁcantly shrunk since it removes the burden of memorizing all the domain-speciﬁc knowledge.
Experiments ﬁnd that 100 × smaller LMs augmented with Mind’s Eye can achieve similar reasoning capabilities as vanilla large models, and its promptingbased nature avoids the instability issues of training mixture-of-expert models (Zoph et al., 2022).
The compatibility with small LMs not only enables faster LM inference, but also saves time during model saving, storing, and sharing.
E XPERIMENTS 4. E XPERIMENTS S ETTINGS
Data and Model.
For the convenience of benchmarking on huge LMs, we prepare 100 samples for each sub-task, resulting in a dataset with about 3,900 samples.
We use this version of U TOPIA for evaluation across the paper.
The MuJoCo simulations can achieve 171 fps on one A6000 GPU, and generating 100 simulations of a 2 seconds collision scene takes 0.67s.
For LMs, besides GPT-3, we have also tested Pathway Language Model (PaLM) (Chowdhery et al., 2022) on 8B, 62B, and 540B checkpoints.
All experiments for PaLM are run on TPU-v4 Pods.
Training Details.
Training of the JAX-based text-to-code LMs runs on TPU-v3 Pods.
The learning rates we use for training 0.3B and 1.5B LMs on C4 are {3.0e-4, 1.8e-4}, which are switched to {1.8e-4, 0.5e-4} when ﬁne-tuning on the text-code pairs.
We use cosine annealing to control learning rate over time with ﬁxed warm-up steps (3k).
As mentioned in §A.2, we have also ﬁne-tuned 62B
PaLM to study task generalization, which takes about 25 minutes on 64 TPU-v4 chips for each task.
4. B ENCHMARK R ESULTS ON U TOPIA
As shown in Figure 4.2, we run experiments on U TOPIA with GPT-3 and PaLM of different sizes, ranging from 340M (GPT-3 Ada) to 540B (PaLM).
Although larger LMs perform consistently better than smaller LMs in both zero-shot and few-shot settings ( n = 5 ), reasoning ability seems to plateau after a certain size, especially in the few-shot setting.
In other words, the scaling curve of vanilla few-shot is nearly ﬂat.
One interpretation could be that few-shot demonstrations have managed to trigger effective in-context learning (e.g., for learning the answer format), but the lack of grounded reasoning becomes the bottleneck for further improvement.
[Figure]
Smaller LMs augmented with Mind’s Eye can achieve on par or even outperform larger vanilla LMs in both zero-shot and few-shot settings (e.g., 29.8% for GPT
3B + Mind’s Eye vs. 29% for vanilla
GPT-3 175B in zero-
In few-shot settings ( n 5 ), the scaling potential is unlocked when
LMs are augmented with Mind’s Eye is nearly ﬂat in vanilla few- mode (orange line), which demonstrates the power of incorporating knowledge from simulations.
On average across all sizes, the gain from Mind’s Eye is greater in large LMs than in small ones, and greater in few-shot (34.2%, absolute) than in zero-shot settings (18.4% absolute).
Mind’s Eye , however, unlocks the ability to scale (red line in Figure 4.2) by adding knowledge from physics engine simulations.
Since the correctness of the simulation results is guaranteed by the physics engine, the reasoning of the large foundation model is thus well grounded.
Compared with solely using knowledge perpetuated in LMs (results with vanilla LMs), Mind’s Eye is able to boost the reasoning ability of LMs by 18.4% in zero-shot and 34.2% in few-shot settings.
Interestingly, smaller LMs augmented with Mind’s Eye can achieve similar or even better performance than vanilla larger LMs.
For example, the accuracy of GPT-3 Babbage 1.3B with Mind’s Eye in zero-shot is 29.8%, while that of vanilla GPT-3 Davinci 175B in zero-shot is 29%.
This ﬁnding demonstrates the effectiveness of the Mind’s Eye paradigm decoupling experimenting from reasoning —where the domain speciﬁc tool is responsible for providing ground-truth results, and the LM can mainly focus on general reasoning, whose size can thus be largely decreased.
Comparison with other reasoning enhanced techniques.
In Table 3, we compare Mind’s Eye with other methods that improve the reasoning abilities of LMs (i.e., GPT-3 175B).
In addition to a)
Chain-of-Thought (Wei et al., 2022), we consider other prompt-based methods such as b) Zero-shot
Reasoner (Kojima et al., 2022), which uses “Let’s think step by step.”
in prompts to induce the decomposed reasoning steps of LMs; c) Self-Consistency Decoding (Wang et al., 2022), which is an ensemble technique that decodes multiple reasoning paths concurrently to improve the performance of Chain-of-Thought; the recent study d) DiVerSe
(Li et al., 2022b) achieves new SotAs on many reasoning tasks, by using pre-trained veriﬁers to weight good answers more than bad ones in each step decoding.
Besides RAG (Lewis et al., 2020) that retrieves knowledge from the memory of 75GB documents, we also consider other ﬁne-tuned LMs which attempt to optimize reasoning ability from different perspectives, such as e) task-scaling T0 (version pp)
(Sanh et al., 2021), which ﬁne-tunes
T5 (Raffel et al., 2020) on thousands of prompts to make LMs better understand input prompts, especially in zero-shot settings, and f) model-scaling Minerva (Lewkowycz et al., 2022), which ﬁne-tunes PaLM (Chowdhery et al., 2022) 540B on a newly collected dataset that contains scientiﬁc and mathematical documents (e.g., arXiv papers) to improve quantitative reasoning.
Though most experiments are running on GPT-3 175B, we also explore the scaling effect by using the 1.3B Babbage model.
The ‘grounding gain’ is the absolute accuracy difference for the 175B model augmented with and without Mind’s Eye.
Unless otherwise stated, we use the default parameter settings recommended by competitor methods.
Results shows that Mind’s Eye outperforms other methods in both zero-shot and few-shot settings signiﬁcantly, even if using a relatively smaller LM (i.e., GPT-3 Babbage 1.3B).
Comparing results on
[Table]
GPT-3 175B and 1.3B, we can conclude that 1) larger LMs can better leverage Mind’s Eye, especially in few-shot settings (46.0% vs. 21.3% average grounding gain), probably because larger LMs are more capable of general reasoning, and 2) solely scaling-up models is not adequate for reliable grounded reasoning performance.
For example, when using a 100 × larger LM (1.3B → 175B), the few-shot accuracy is merely boosted by 1.8% (absolute; 36.4% → 38.2%) if using vanilla LMs, but that can be boosted by 37.5% (absolute; 46.7% → 84.2%) if the LMs are augmented by Mind’s Eye.
Note that Instruct-GPT(Ouyang et al., 2022) augmented with Mind’s Eye is able to achieve nearly perfect performance in few-shot settings.
This result is promising because it demonstrates the ideal alignment is achievable if the LM is given proper reasoning evidence and has good understanding of the questions (as Instruct-GPT is optimized for instruction following).
The improvement from simply better prompting or decoding methods is limited, since their reasoning completely relies on the internal knowledge perpetuated in the LMs, while the knowledge induced from the LMs could be factually wrong.
Among augmented LMs, 540B Minerva is the best performing one but still falls behind Mind’s Eye + 175B GPT-3, mainly due to the lack of grounded experience for reasoning.
We also ﬁnd the retrieved evidence of RAG sometimes cannot answer the question, even though it includes entities mentioned in the question, and RAG cannot function well in few-shot settings since the retriever module (i.e., DPR (Karpukhin et al., 2020)) was not pre-trained to handle context that contains multiple question-answer pairs.
We have also discussed the domain generalization ability of
Mind’s Eye in §A.2 by ﬁne-tuning PaLM on each scene seperately.
4. A BLATION S TUDY
What type of simulation do we need?
In Table 4, we show the performance if 1) we randomly alter the simulation results in the prompts to mismatch the physics property asked (e.g., asking velocity but including acceleration results in prompts), and 2) we delete the trigger words at the beginning of [Figure] the prompts (i
[Figure] e., “Hints:” ).
We also test what would happen if we ground the reasoning on incorrect simulation results (e
[Figure]
g., Simulation indicates greater but we use smaller in prompts). We ﬁnd reasoning on mismatched simulation results causes substantial performance drops, while including wrong simulation results will further deteriorate the performance, probably because in this case the reasoning is misguided even if the vanilla LM can answer the question correctly.
We take these results as evidence that the correct simulation results are crucial for grounded LM reasoning.
Missing trigger words have marginal inﬂuence on the performance, which conﬁrms again that reasoning performance mainly depends on the existence and correctness of the simulation results.
simulation results and the trigger words of Mind’s Eye to understand their effects.
We also study whether the correctness of simulation will affect the reasoning performance.
[Table]
[Table]
[Table]
Can few-shot replace simulation?
Given enough in-context demonstrations, can the LM learn how to make its reasoning grounded internally without relying on external simulations?
In Figure 3, we present the dynamics of reasoning performance by gradually including more and more in-context demonstrations in few-shot settings on both vanilla LMs and Mind’s Eye augmented LMs.
We also design a semi-augmented baseline, Semi-Mind’s Eye, which has the Mind’s Eye style in-context demonstration, but the ﬁnal shot omits the simulation result—in other words, the LM has to perform reasoning on the generated grounding evidence by itself in the ﬁnal step.
This baseline differs from vanilla few-shot as it incorporates some simulation results from the physics engine, and it differs from Chain-of-Thought (CoT) since the reasoning steps of CoT are written by humans.
The result demonstrates neither vanilla few-shot (red line) nor semi-augmented few-shot (yellow line) can provide as good an improvement as Mind’s Eye (blue line).
We also ﬁnd the few-shot reasoning performance of CoT (green line) and Semi-Mind’s Eye (yellow line) has some instabilities which depends on whether the few-shot demonstrations happen to have similar reasoning steps as the given question (ﬁnal step).
The effectiveness of Mind’s Eye seems to echo the ﬁndings in Zhao et al.
(2021), which conﬁrms that the LM tends to ground its reasoning on “recent context”, which is coincidentally the simulation results of the given question.
Using a smaller text-to-code LM?
Text-to-code LMs convert text-form questions into rendering code.
To study its scaling effect, we explore several combinations of using text-to-code LMs and pretrained LMs in different sizes.
As shown in Table 5, we ﬁnd using a smaller text-to-code LM (0.3B) has slightly smaller impact for larger pretrained LMs, as the accuracy decreases by 4.7% for 1.3B pretrained LM but 3.6% for 175B in zero-shot.
We also ﬁnd the conversion error is somehow mitigated by few-shot demonstrations (e.g., the accuracy decreases by 3.6% in zero-shot but by 2.1% in few-shot when the 175B model uses a smaller T2C LM), indicating the beneﬁts on robustness when using a larger pretrained LM.
R ELATED W ORK
Grounded Reasoning.
Language models trained only on text descriptions of the world are often criticized for the lack of grounding in their reasoning abilities–—that is, they have never physically observed the objects or experienced the events that appear in their generations, so that they are reported to be still limited in spacial understanding (Patel & Pavlick, 2021; Janner et al., 2018), action planning (Valmeekam et al., 2022;
Ruis et al., 2022), and even simple object counting (Dai &
Bosma, 2022). Early attempts to relate language understanding (Winograd, 1972) or learning (Roy & Pentland, 2002) to the physical world mostly rely on manually created linguistic and physical rules (Hermann et al., 2017; Berant et al., 2013).
Recent studies have pivoted to learn such relations with neural models, including implicitly learning from text (Li et al., 2021), and explicitly learning from animation (Wu et al., 2017), games (Fan et al.), or actual videos of the world (Ding et al., 2021;
Chen et al., 2020; Xue et al., 2018).
Instead of learning a discrete network for physics knowledge, Mind’s Eye directly uses tools that contain embedded intelligence (i.e., the physics engine) to obtain grounded reasoning abilities. On the other hand, the idea of simulation augmented prompting is more scalable compared with other grounding improvement methods, such as adding decomposed reasoning steps (Wei et al., 2022; Zhou et al., 2022), hand-written task descriptions (Raffel et al., 2019b) and targeting formats (Marasovi´c et al., 2021) to the prompts.
Augmented Language Models.
Inspired by the evidence that humans require sensory perception for grounded reasoning (Sachs et al., 1981), previous work has tried to augment text inputs to LMs with audio signals (Tsai et al., 2019) or visual perception (Bakhtin et al., 2019;
Yi et al., 2018), for improved game playing (Lee et al., 2022), faster language learning (Hill et al., 2020), or better decision making in general (Reed et al., 2022).
Our approaches seemingly echoes these ﬁndings as we leverage the simulation results as the extra sensory input.
To endow LMs with updated knowledge, TALM (Parisi et al., 2022) ﬁne-tunes LMs interactively with augmented inputs from API calls, which is rather costly compared to prompt-based Mind’s Eye.
Other methods have also explored to train veriﬁers for math reasoning (Cobbe et al., 2021), unit tests for program synthesis (Chen et al., 2022), and human feedback predictors for safe (Glaese et al., 2022) or satiesfying (Nakano et al., 2021) dialogue agents. Mind’s Eye differs from these methods as we use off-the-shelf simulator without training.
PaLM-SayCan (Ahn et al., 2022) uses the pre-trained PaLM 540B to help robots better understand complex instructions that require reasoning—Mind’s Eye can be viewed as the reverse: it infuses knowledge from external tools (i.e., the MuJoCo physics engine) into LMs, to further unlock the the reasoning capabilities of LMs.
Modeling the World.
Learning by trial and error, humans seem able to learn enormous amounts of common sense about how the physical world works (Lerer et al., 2016)—such observation has inspired the idea of developing a neural model to model the world (LeCun, 2022; Craik, 1967).
World models (Ha & Schmidhuber, 2018) should be capable of planning (Li et al., 2022a), predicting (Amos et al., 2018), and reasoning (Ullman et al., 2017) through interactions with the world.
Similarly, Mind’s Eye proposes a paradigm that reasons over the experimental results predicted by simulation, and the experiments are planned beforehand by the text-to-code LM.
Using simulated environments to help learning has been widely adopted by research in robotics (e.g., MineRL (Guss et al., 2019),
VIMA (Jiang et al., 2022)) or computer vision (e.g., Kubric (Greff et al., 2022), ThreeDWorld (Gan et al., 2020)), while our focus is grounded reasoning in the form of natural language.
Apparently, Mind’s Eye can be limited by the ability of the backend simulator: For example, currently there is no support for ﬂuid simulations in MuJoCo 4 , so that Mind’s Eye still cannot solve fuild related physics questions in current version. Also, the simulation results are summarized as sensory outputs (e.g., “ X and Y will have the same acceleration. ”), the available types of which heavily depend on whether MuJoCo enables capturing certain types of information.
We believe these issues can be partially solved by replacing the open-source MuJoCo with more-advanced industry-level physics/game engines, such as Unity 5 or Unreal 6 , and we ﬁnd how to realistically simulate softbody is still an active research topic in computer vision (Huang et al., 2020; Xia et al., 2018). Integrating these new simulation techniques might enable the the physics engine to render more complex scenes.
We also perform error analysis for Mind’s Eye. As shown in Figure 4, there are three major types of error: Ignorance error refers to the reasoning apparently ignores the given grounded rationale. Recency bias means the LM tends to extract the ﬁnal answer from nearby words (e.g., the object Y).
“I don´t know” error are the cases where the LM simply generates non-sense when unable to reason about the question.
All these cases are much less common when we run experiments on larger LMs, probabily because the larger LMs are more capable of understanding the context, where we inject the simulation based rationales.
[Figure]
[Table]
TherearetwopiecesofwoodXandYmovingtowardseachotherhorizontallyatcertainvelocities.
ThehorizontalvelocityofXhasasmallermagnitudethanthatofY. Whichonewillmovefasteraftertheirelasticcollision?
Answer
[Figure]
Hints
[Table]
Xhasagreatermagnitudeofvelocityafterthecollision.
Sotheansweris
[Table]
Y. Question
[Table]
TherearetwoboxesXandYslidinghorizontallyatcertainvelocities
[Table]
[Figure]
[Table]
C ONCLUSION AND F UTURE W ORK
We have proposed Mind’s Eye, a novel paradigm that enables LMs to ground their reasoning with simulations of the world.
We conclude that Mind’s Eye is not only effective and scalable but also efﬁcient, as it is able to boost the reasoning performance of small-scale LMs signiﬁcantly, requiring neither handcrafted prompts nor costly ﬁne-tuning.
We believe the idea of including simulation into the reasoning pipeline can be easily extended to other domains or applications, especially where simulation engines already exist.
For example, instead of the physical world, one can use a simulation of economic changes or thermodynamics, aiding policy making or engineering.
The design of Mind’s Eye that decouples the domain speciﬁc knowledge from the general reasoning module offers potential beneﬁts for real-time systems, where the update on simulators will not interrupt the running service.
More importantly, the dynamic nature of Mind’s Eye where we generate grounding evidence further unlocks the scaling potential of language models, since the model can obtain ground-truth rationales on-the-ﬂy rather than memorize everything.
E THICS AND R EPRODUCIBILITY S TATEMENT
The goal of Mind’s Eye is to present a general-purpose paradigm that incorporates knowledge from simulation by professional tools into LM reasoning pipeline.
Though we have already seen signiﬁcant improvement on reasoning with Mind’s Eye grounding, the performance can be affected by how accuracy the simulation is.
Unqualiﬁed simulators may result in wrong simulation results, grounding with which will lead to even worse results than reasoning without grounding.
Furthermore, our experiments and analysis are done in English, and therefore we do not claim that our reasoning paradigm will generalize across all languages, although our framework has potential to be extended to other languages by using multilingual LMs.
For reproducibility, we run experiments mainly with publicly available LMs (e.g., GPT-3) and choose baseline methods that have open-sourced implementation.
To aid reviewing, we have included the benchmarking version of U TOPIA as well as model outputs in the supplementary materials.
A. S CENE G ENERALIZATION WITH M IND ’ S E YE
By injecting simulation based prompts during LM inference, Mind’s Eye does not need ﬁne-tune the LM every time it encounters new contexts.
This feature is appealing as we expect Mind’s Eye can serve as a powerful tool for building autonomous intelligence that can ground its reasoning or decision making even in unfamiliar contexts.
To clearly show the advantages of Mind’s Eye on scene generalization, we compare the performance of ﬁne-tuned PaLM 62B with that of vanilla PaLM 62B armed with Mind’s Eye.
As shown in Figure A.2, we ﬁnd the ﬁne-tuned model can achieve good performance on in-domain data, but its performance is quite limited in scenes it has not been ﬁne-tuned on.
For example, in
Figure A.2 (a), ﬁne-tuned on the Motion scene of U TOPIA can obtain 70% in domain alignment accuracy, but the accuracy decreases to 24% if we test this model on the Projection scene (> 50% performance drop).
However, as shown in Figure A.2 (b), Mind’s Eye enables the LM to perform consistently well, even if the LM has never seen the scene before.
[Figure]
In Table A1 we show more detailed benchmark results by including PaLM 540B performance and variance of accuracy for each method.
In general, we ﬁnd few-shot has lower variance than zero-shot, while Mind’s Eye has relatively lower variance than others, partially because the injected rationale makes the reasoning more determined.
We also ﬁnd self-consistency can effectively decrease the variance since its majority-voting nature.
[Table]