On the Learning Mechanisms in Physical Reasoning
Abstract
Is dynamics prediction indispensable for physical reasoning?
If so, what kindof roles do the dynamics prediction modules play during the physical reasoningprocess?
Most studies focus on designing dynamics prediction networks andtreating physical reasoning as a downstream task without investigating the questionsabove, taking for granted that the designed dynamics prediction would undoubtedlyhelp the reasoning process.
In this work, we take a closer look at this assumption,exploring this fundamental hypothesis by comparing two learning mechanisms:
Learning from Dynamics (LfD) and Learning from Intuition (LfI).
In the firstexperiment , we directly examine and compare these two mechanisms.
Resultsshow a surprising finding: Simple LfI is better than or on par with state-of-the-art(SOTA)
LfD. This observation leads to the second experiment with Ground-truth
Dynamics (GD), the ideal case of LfD wherein dynamics are obtained directly froma simulator.
Results show that dynamics, if directly given instead of approximated,would achieve much higher performance than LfI alone on physical reasoning; thisessentially serves as the performance upper bound.
Yet practically, LfD mechanismcan only predict Approximate Dynamics (AD) using dynamics learning modulesthat mimic the physical laws, making the following downstream physical reasoningmodules degenerate into the LfI paradigm; see the third experiment .
We note thatthis issue is hard to mitigate, as dynamics prediction errors inevitably accumulate inthe long horizon.
Finally, in the fourth experiment , we note that LfI, the extremelysimpler strategy when done right, is more effective in learning to solve physicalreasoning problems.
Taken together, the results on the challenging benchmark of
PHYRE [3] show that LfI is, if not better, as good as LfD with bells and whistlesfor dynamics prediction.
However, the potential improvement from LfD, thoughchallenging, remains lucrative.
Introduction
Humans possess a distinctive ability of understanding physical concepts and performing complexphysical reasoning.
The literature on humans learning mechanisms for solving physical reasoningproblems can be categorized into two schools of thoughts [27]: (i) physical intuition at a glancewithout much thinking, such as judging whether a stacked block tower will collapse [6], and (ii) moreextensive unfolding of states under the assumed physical dynamics when facing complex physicaltasks [7].
Such a disparity between System-1-and-System-2-like problem-solving strategies [23]motivates us to think over the learning mechanisms for physical reasoning in machines:
When performing physical reasoning, is it better for machines to learn from intuition by simply analyzing the static physical structure, or to learn from dynamicsby predicting future states?
Recent physical reasoning benchmarks for machine learning are mostly physics engines focusing onevaluating the task-solving abilities of models.
For example, PHYRE [3] and Virtual Tools [1] contain
Generate dynamic sequences from predictor
Initial scene
Classifier
Obtain dynamic sequences from simulator
Simulator
Predictor
Classifier Classifier
Simply use initial scene without any dynamic information or prediction
Solved
Unsolved
Solved
Unsolved
Solved
Unsolved
D egene r a t e
I dea li z e LFD
LFI
GD
AD physical scenes with long-term dynamics and complex physics interactions.
These environments havevarious tasks with explicit goals, such as making the green object touch the blue object by placinga new red ball into the initial scene.
An artificial agent is tasked to predict the final outcome, e.g .,whether the placed red ball will successfully solve the given task.
Existing problem-solving methods approach such physical reasoning problems by designing a varietyof future prediction modules [18, 32].
These modules are devised under the assumption that humanbrains inherently possess a simplified physics engine (called intuitive physics engine) [14], akin to acomputer game simulator, capable of predicting objects’ future states and changes.
Although the intuitive theory claims that humans are able to predict physical outcomes in a rapidfashion, it does not directly guide at the computational level more than the hypothesis that we mighthave a physics-engine-like mechanism in our brain [14, 27].
Critically, although humans can predictdynamics under the intuitive theory, dynamics prediction might not be necessary for all types ofphysical reasoning tasks.
This hypothesis is largely left untouched, especially at the computationallevel.
Just as noted by Lerer et al.
[29], directly learning by intuition without dynamics predictionis sufficient for various physical reasoning tasks.
Of note, this hypothesis does not contradict theintuitive theory, but rather provides a new perspective at the computational level.
In this paper, we conduct a series of experiments to empirically answer the above questions.
To thebest of our knowledge, ours is the first to systematically compare the LfI and LfD paradigms.
Inthe first experiment , we verify the simple approach of LfI by training a classifier [12] to predictwhether an action would lead to success in problem-solving.
Surprisingly, in the preliminary study,such a model already reaches the SOTA performance and even outperforms existing LfD methodsin unseen scenarios, which indicates better generalization.
Inspired by this counter-intuitive result,we conduct more experiments on the two learning mechanisms; see Fig. 1 for an illustration.
Inthe second experiment , we first set out to investigate whether LfD could work better than LfI intheory by measuring the performance of a video-based classifier [8] using the ground-truth dynamics,setting an upper bound for the route of dynamics prediction.
In the third experiment , we replace theground-truth dynamics with predicted dynamics from an advanced future prediction model [39] to seehow LfD performs in practice.
The results from these two experiments reveal that precise dynamicssignificantly improve model performance, but the predicted approximations fail to work as expected:
Approximation brings disturbing biases and causes the performance to degenerate into LfI or evenworse in certain tasks.
In the fourth experiment , through a series of experiments on various LfImodels, we conclude that LfI could be a simpler and more practical paradigm in physical reasoning.
However, making breakthroughs in physical prediction is still promising though challenging.
Wehope that our discussions shed light on future studies on physical reasoning.
Related Work
Intuitive Physics and Physical Reasoning
Since Battaglia et al.
[6], the computational aspects ofintuitive physics have attracted research attention [27, 44].
The progress enables machines to learnto judge (i) which object is heavier after observing two objects collide [16, 33, 37], (ii) whether astacked block tower will fall [6, 19, 29], (iii) whether water in two different containers will pour atthe same angle if titled [25, 35] or liquid in general [5], and (iv) behaviors of dynamics with variousmaterials [26].
However, this line of work primarily focuses on physical tasks without long-termdynamics, either by knowledge-based approaches [6, 33] or learning-based approaches [19, 29].
More complex physical reasoning problems [1, 3, 42], including those involving question answering[9–11, 21, 43], have also been studied.
In particular, Allen et al.
[1] propose to use knowledge-basedsimulation [1]; Xu et al.
[41] adopt a Bayesian symbolic method [41]; Battaglia et al.
[7], Girdhar et al.[18], and Qi et al.
[32] recruit the graph-based interaction network.
However, none of these methodsfully justify the necessity of dynamics prediction.
In this work, we challenge this fundamentalassumption and point out a simpler and efficacious but overlooked solution.
PHYRE and other relevant environments Bakhtin et al.
[3] introduce the novel physical reasoningtask of PHYRE, wherein an agent is tasked with finding action in an initial scene to reach the goalunder physical laws.
Current methods for solving PHYRE range from reinforcement learning ( e.g ., DQN [3]) to forward prediction neural networks with pixel-based representation [18] and objectbased representation [18, 32].
Notably, Girdhar et al.
[18] adopt different kinds of forward predictionarchitectures to perform PHYRE tasks, but fail to obtain significant performance improvement; Qiet al.
[32] design convolutional interaction network to learn long-term dynamics, achieving SOTAperformance by leveraging ground-truth information about object states in the physical scenes.
In fact, the physical reasoning task of PHYRE can be regarded as an image classification task onjudging whether an initial action would lead to a successful outcome, or a video classification task bytaking into consideration the dynamics after the initial action is performed.
The success of the formersat on deep convolutional neural networks [20, 24] and has now shifted to Transformer-based models[4, 12, 30].
The change from convolutional architectures to attentional models also inspires recentadvances in video classification.
Models such as TimeSformer [8] and ViViT [2] in this domain alsoexpand into fields such as action recognition [17] and group activity recognition [15].
Unlike PHYRE which directly focuses on physical reasoning, some benchmarks include physicalreasoning in their environments more implicitly, such as the ones in autonomous driving [13] andembodied AI [34].
The robotic controller based on physics engines [38] and navigation tasks on 3Dphysical scenes [40] may also need physical perception modules in the system.
Dynamics Prediction Predicting dynamics into the future is one of the most extensively studiedtopics in the vision community.
One common method is to extract image representation and incorporate an RNN predictor [36, 39] or a cycle GAN-based approach [28].
However, most approachescannot extract robust representation from the pixels and incur exponentially growing errors in longterm prediction.
To tackle this problem, Janner et al. [22] and Qi et al.
[32] focus on object-centricrepresentation.
However, inductive biases ( e.g ., spatial information, the number of objects) lead totask-specific solutions, and the performance drops when they are dealing with multiple objects withocclusions [31].
The Two Learning Mechanisms
In this section, we define the two learning mechanisms for solving physical reasoning problems.
Henceforth, we denote all objects’ states at time t as X t .
Given an initial background image I of aphysical setup and a random distribution of actions A , the model needs to learn a distribution of thefinal outcome P ( y | X 0 ) , where X
=
{A , I } , and y denotes the possible outcome.
Mechanism 1.
Learning from Intuition (LfI) In LfI, the outcome y is learned directly from theinitial images and actions using a task-solution model f :
[Equation]
where θ denotes the parameters of the task-solution model f ( · ) .
We call this mechanism LfI becausethe task-solution model f ( · ) can be viewed as an intuitive map from the initial conditions to theoutcome.
Mechanism 2.
Learning from Dynamics (LfD)
The nature of physics is inherently dynamic.
Therefore, in LfD, y is no longer directly learned from the initial scenes; instead, this approach firstlearns the underlying dynamics D
= { X t | t = 0 , 1 , . .
, T } within a time window T using a dynamicsprediction module g ( · ) , and then predicts the outcome from the predicted dynamics.
Formally:
P ( y | X 0 ) =
f ( D ; θ ) , where D = g
( X 0 ; ϕ ) , (2) where ϕ represents the parameters of the dynamics prediction model g ( · ) .
Usually, g ( · ) is implemented as either an auto-regressive module based on pixel presentation or a graph-based interactionnetwork from object-based representation.
In this work, we consider two optimization schedules forg ( · ) : parallel optimization during joint learning of f ( · ) or serial optimization by learning and fixingg ( · ) first; please refer to Supplementary Material for detailed algorithms.
While the dynamics prediction step and the outcome prediction step can be integrated together, it isworth noting that in LfD, additional architecture changes and supervisory signals are necessary tolearn the underlying dynamics, without which the paradigm degenerates into LfI. Ideally, a physicsengine or a simulator plays the role of future prediction.
However, inversely learning the physicallaws is very demanding due to the intrinsic challenges in long-term prediction.
Experimental Setup
In this section, we briefly introduce the challenging physical reasoning benchmark of PHYRE-B andthe setup of our experiments.
PHYRE-B is a goal-driven benchmark consisting of 25 different task templates of physical puzzlesthat can be solved by placing a red ball (hence the “B”).
Each template has 100 similar tasks,which enables two different evaluation settings.
(i) Within-template setting: train on 80% of tasksof each template and test on the rest 20% of each template.
(ii) Cross-template setting: train onall tasks of 20 from the 25 templates and test on the rest 5 previously unseen templates.
Theperformance is evaluated by AUCCESS [3], a weighted sum of the accuracy rate related to thenumber of attempts.
To encourage as few attempts as possible to solve a puzzle, the weights arecalculated as: ω k = log( k + 1) − log( k ) , where k ∈ { 1 , 2 , 3 ... 100 } , represents the range of attempts.
Formally, AUCCESS =
(cid:80) k ω k · s k (cid:80) k ω k , where s k is the success rate with k attempts [3].
This physicalreasoning benchmark is deemed challenging due to the following three reasons: (i)
The environmentinvolves a variety of objects of different shapes and mass distribution, such as balls, bars, standingsticks and jars.
(ii)
The solution set takes up only a small part of the whole action space which makesrandomly sampled actions hardly ever work out.
(iii) Before reaching the goal, a series of complexphysical interactions would happen, involving falling, rotation, collision and friction.
To the best ofour knowledge, no success has been claimed on the environment.
Experiment 1: LfD vs. LfI
In the first experiment, we compare the SOTA LfD model on PHYREand a transformer-based classifier for LfI. Specifically, we pick the RPIN model [32] to represent
LfD and the ViT model [12] for LfI.
The RPIN model leverages a convolutional interaction networkbased on object-centric representation and predicts the object states (bounding boxes and masks) intothe future.
When solving a PHYRE task, the model first predicts 10-time steps into the future for anaction and then recruits an MLP-based task-solution model to predict the outcome.
In comparison,
ViT is a transformer-based classifier initially designed for image classification.
The model patchesan image, decomposes it into non-overlapping regions and performs a series of self-attention.
Theimage-level representation is finally extracted by appending a CLS token.
Experiment 2: LfD under GD The second experiment serves as a diagnostic test for the efficacyof dynamics in the task of physical reasoning.
To this end, we directly extract simulator results andfeed the image sequence into a video classifier model, setting an upper bound for the LfD paradigmby replacing dynamics prediction with ground truth.
Due to its superior performance, in this part, weadopt the TimeSformer model [8].
The model patches the image sequence and replaces traditionalconvolution architectures with interleaving spatial attention and temporal attention.
Experiment 3: LfD under AD We dive deep into the ineffectiveness of LfD methods like RPIN byfixing the task-solution model with TimeSformer but replacing the input with trajectories predictedby a learned module.
In particular, we train a PredRNN [39] to learn the dynamics.
PredRNN is anadvanced pixel-based video predictor extending the inner-layer transition function of memory statesin LSTMs to zigzag memory flow that propagates in both bottom-up and top-down directions.
Acurriculum is set up during learning for better long-term prediction [39].
Experiment 4: More on LfI
In the final set of experiments, we verify the performance of different
LfI methods.
In addition to ViT, we consider BEiT [4] and Swin Transformer [30] which provebeneficial in image classification.
Comparison of Learning from Intuition and Learning from Dynamics
In this section, we start our discussion with a preliminary experiment and consequently explorewhether dynamics contribute to better judgments in physical reasoning tasks both in ideal conditionsand in reality by comparing the performance of LfI and LfD.
5.1 LfD vs. LfI: An interesting finding
To tackle the demanding physical puzzles in PHYRE-B, previous studies use pixel-based or objectbased future prediction modules to boost problem-solving performance [18, 32].
The SOTA LfDmethod RPIN uses a convolutional interaction network to predict long-term dynamics.
Inspired byprevious studies on intuitive physics, we set out to see whether a decision can be made directly by aclassifier of ViT. Counter-intuitively, while the supervision from the dynamics should ideally help amodel in LfD look into the future, the performance of the LfI model reaches better than or on parwith SOTA as shown in Tab. 1.
The AUCCESS of ViT in the cross-template setting even significantlyoutperforms RPIN’s, which demonstrates its high generalization ability when solving unseen physicalpuzzles.
Driven by this intriguing finding, we conduct additional follow-up experiments to find out ifthe dynamics prediction component helps perform physical reasoning.
[Table] [Table]
LfD
[Table]
GD:
[Table]
ground-truth dynamics help make better decisions than intuition?
[Figure]
The surprising discovery motivates us to ask why the LfDmodel has similar or even worse performance than LfI. Asa first step, we question the merit of dynamics:
Do dynamics help in solving these physical puzzles?
To answerthis question, we supply the model with the best-case—ground-truth—dynamics and provide the information toour task-solution model of TimeSformer, due to the videoclassification-like nature.
We vary the number of timeframes supplied to the model: We consider the input oflengths 1, 2, 4, and 8 extracted directly from the simulator,with a time interval of 1 second.
It is worth noting thatusing GD, we assume an ideal dynamics prediction modelg ( · ) that accurately predicts into the future.
The performance of LfD with GD is shown in Fig.
While the model underperforms ViT with fewer than 3frames, which is out of the scope of our work, we seethat AUCCESS is significantly boosted with 4 or more frames.
Specifically, with 8 input frames,
AUCCESS can be improved by 23.11 in the within setting and 48.16 in the cross setting, reachingabout 95% in both settings.
The observation answers the above question: Dynamics do help toproblem-solve in physical reasoning.
Such an observation aligns with our intuition that dynamics cantell more about the future states, thus making learning decisions easier for the task-solution model.
With sufficient dynamics information, the problem could be “solved” to a large extent.
However, if dynamics do help, why do current LfD methods fail to outperform LfI as shown in
Sec. 5.1, even considerably lower than the simple baseline in generalization?
Is it because ofinaccurate dynamics prediction or because of the outdated design in task-solution models?
In thefollowing set of experiments, we answer this question.
5.3 LfD under AD:
How do approximate dynamics perform?
Given the fact that GD does help problem-solving, we explore the reason why AD—the dynamicsprediction in practice—does not take effect as expected.
Specifically, we fix the task-solution modelf ( · ) with TimeSformer [8] as in Sec.
5.2 and instantiate the dynamics prediction model g ( · ) with anadvanced video prediction model PredRNN [39].
We train the LfD pipeline using both of the twooptimization schedules mentioned in Sec.
3 to investigate which schedule can better help in learninguseful dynamics information.
The PredRNN first takes one initial frame as the input and outputs three predicted future frames.
Next,the TimeSformer takes all the frames as input, including the first initial frame and the three predictedframes, and outputs the final outcome.
(i)
For parallel optimization, we train an end-to-end frameworkby integrating the output from PredRNN with the input of TimeSformer.
The model parametersϕ and θ are updated simultaneously by backpropagating both the dynamics-learning loss and thefinal cross-entropy loss.
(ii) For serial optimization, the PredRNN is trained first independentlyusing ground-truth future dynamics as supervision and kept fixed, after which the TimeSformer isoptimized using the output from the pre-trained PredRNN, supervised by the ground-truth decisions.
For both optimization schedules, the PredRNN only takes raw RGB images as the input without extrainformation, such as bounding boxes or masks of each object, and directly generates the predictedfuture frames without further rendering.
We report the AUCCESS of both optimization schedules on PHYRE-B in Tab. 2.
These results showthat independent of the optimization schedule used, AD falls far behind from GD and performs equallyor even worse than LfI, indicating that approximate dynamics do little help for the task-solution modelin making better judgments.
Specifically, by comparing the performance of parallel optimization to
TimeSformer with a single frame, we observe a tendency of LfD pipeline’s performance degenerationto LfI, ending up with similar AUCCESS.
Moreover, Tab. 3 shows that for both within and crosssettings, the test dynamics loss in parallel optimization is much higher than the corresponding onein serial optimization, which suggests that the representation learned by LfD pipeline in paralleloptimization has difficulty in capturing accurate dynamics and is severely impacted by the crossentropy loss, as is evident in the visualization results in Fig.
Although serial optimization gives thedynamics prediction model ample opportunities to learn better pixel-wise approximations of futureframes, it still fails to make better decisions than parallel optimization.
We hypothesize that in theserial optimization, the task-solution model eventually uses the noisy predicted dynamics, whichhinders the performance due to the inaccuracy; in parallel optimization, the model can avoid thisproblem by paying less attention to learning future dynamics and focusing more on directly learningthe outcome in a manner similar to LfI. Taken together, these experimental results pinpoint thereasons for the inefficacy of current LfD methods with AD: (i)
It is the poor dynamics learned, ratherthan the task-solution model, that causes the performance degradation.
Dynamics inaccurate preventaccurate reasoning.
(ii)
Existing LfD methods with AD degenerate into LfI in the best case.
[Table] [Table]
By comparing the models above and answering the questions [Table] we conclude that dynamics prediction,in theory, does help solve physical puzzles by providing more temporal information, while strugglingto show the full strengths in reality due to the challenges in handling accumulated uncertainty ina long horizon.
The results in serial and parallel optimization also reveal the fact that dynamicsprediction plays an important role regardless of the task-solution model, but inaccurate dynamics willeventually harm downstream reasoning; in the best case, LfD degenerates into LfI.
T=
T= T=
T= T=
T= T=
T= T=
T= T=
T=
T= c r o ss S e r
i a l
T=
T=
C r o ss P a r a ll e l
C r o ss
G r ound
T r u t h
T=
W it h
i n S e r i a l
T=
T=
W it h i n
P a r a ll e l
W it h i n
G r ound
T r u t h
C r o ss S e r
i a l
5.4 More on LfI: How does LfI perform?
Observing the notable success of ViT in the physical reasoning task, we consider testing additionalvisual classification models to verify the effectiveness of the LfI paradigm.
Among a myriad ofmodels, we pick ViT, Swin Transformer, and BEiT for experiments due to their demonstratedsuperiority in image classification.
Of note, TimeSformer with a single input frame can also beconsidered as a model in the LfI paradigm.
All of the models take only the first frame as the input.
For equal comparison, we train all of the LfI models with an equal number of actions per task as in
RPIN.
Nevertheless, the LfI models take no extra prior object information as the input and use onlyfinal outcome labels as supervision signals, significantly simplifying the learning process.
The results of these LfI models are presented in Tab. 4.
The performance of LfI models in the withintemplate setting is competitive with the SOTA RPIN.
In addition, all of the LfI models outperform
SOTA in the cross-template setting in AUCCESS; ViT even outperforms by almost 6 points onaverage.
By visualizing the distribution heat maps of the possible solution set learned by ViT and
[Table]
W it h i n
G r ound
T r u t h
C r o ss
G r ound
T r u t h C r o ss P r e d i c ti on W it h i n
P r e d i c ti on
ViT learns an accurate solution set distribution as the ground truth comparing them with the ground truth in within-template and cross-template settings (see Fig.
4),we observe that the ViT model not only achieves high performance but also accurately recoversthe underlying solution distributions.
We hypothesize that intuitive models, though trained withoutdynamics information, can still extract high-level spatial knowledge and physical commonsense thatgeneralizes well in unseen scenarios by looking at the data only; e.g ., the most suitable red ballposition should be close to the moving objects in order to exert influence.
However, we believe thatthe performance of LfI still has a lot of room for improvement since the inductive biases and structuredesigns for physical reasoning are not explicitly considered.
Besides the promising performance, LfI models also demonstrate the following three advantages: (i)
They are design-efficient without the complex hand-crafted tweaks for dynamics prediction modules,which in some cases only introduce harmful distraction and noisy representation for reasoning asdiscussed in Sec.
(ii)
They only take initial scenes as input and require no extra task-specificprior knowledge about the objects as used in object-centric dynamics prediction.
(iii) LfI modelscan be easily pre-trained on other computer vision tasks, e.g ., image classification on ImageNet,incorporating general domain-agnostic representation that avoids task-specific overfitting.
In summary, we view LfI as a simpler, more effective, and more general paradigm to solve physicalreasoning problems.
Conclusion and Discussion
We introduce the concepts of two learning mechanisms in physical reasoning, i.e ., Learning from
Intuition (LfI) and Learning from Dynamics (LfD).
While it is generally believed that learningthe dynamics of physics could help downstream reasoning, a beginning trial of RPIN and ViT on
PHYRE-B challenges this fundamental assumption: A ViT model effectively learns to performphysical reasoning without any additional supervision from the ground-truth dynamics, object-centricor not.
This counter-intuitive discovery motivates us to ask whether dynamics play an essential role in physical reasoning.
We proceed to answer this question by using Ground-truth Dynamics (GD) from
PHYRE’s simulator and feeding the sequence to a TimeSformer model.
Experimental results showthat accurate dynamics can boost problem-solving performance.
We further explore why Approximate
Dynamics (AD) from dynamics predictors perform unfavorably in physical reasoning.
We note thatthe task-solution model is not the one to blame.
In addition, despite increasingly accurate dynamicsprediction over the years, we notice that noisy dynamics prediction still has a negative impact on theoverall performance in reasoning; during parallel optimization, the LfD paradigm collapses into LfI.
We hypothesize that in the long run, uncertainty in dynamics prediction unavoidably accumulates,leading to inferior final performance.
Finally, we dig deeper into the LfI paradigm and check theperformance of various classification models in PHYRE.
The experimental results show that thesemodels achieve much better cross-template generalization while remaining competitive in withintemplate generalization.
It turns out LfI can still perform well even if accurate dynamics are hard topredict, providing a route for a simpler, more natural, and less task-specific framework for physicalreasoning.
However, the LfD route, though challenging, still remains a lucrative approach to thisproblem if dynamics could be predicted in a significantly more accurate manner.
Why do dynamics-based models struggle to make accurate predictions?
Analyzing the experimental results, we try to provide preliminary explanations on why current dynamics predictionmodels struggle.
We summarize the following three possible reasons: (i)
The dynamics predictionitself is challenging, especially in unseen scenarios.
For one thing, prediction into the far futureis intrinsically difficult as unforeseeable events might steer the dynamics in another direction.
Foranother, the errors will accumulate from earlier frames, leading to exploding uncertainty and noise inthe future.
Unfortunately, current dynamics prediction methods cannot produce a robust model topredict accurate dynamics in physical scenes.
(ii) Pixel-based dynamic representation has moreinformation than object-based representation, while object-based representation is more concise.
Arguably, pixel-based representation potentially incorporates all necessary information, such asshapes of objects, potential points of collision, and angular velocity.
However, such a representationis extremely noisy, and useful information embedded is hard to be extracted.
In comparison, objectbased representation is by design concise and follows the general principles in the laws of physics.
Nevertheless, object-centric methods lose essential clues in the scenes especially when it comesto collision and its aftereffects.
The fact that there is not yet a feature representation method thatsummarizes all necessary information for physical modeling further complicates physical reasoning.
(iii) Task-solution model design might play a role, though not significant.
The self-attention
mechanism has
constantly proven rewarding
in a variety of tasks.
The strong performance of LfI
could also benefit from
this architectural design.
Limitations and Future Work Building intelligence with the physical reasoning ability is anever-ending journey.
However, there are still many aspects left to be studied in the future.
• In contrast to the significance of the topic is the lack of rich environments for experiments.
Therefore, we
only conduct experiments on PHYRE to support our insights.
In the future wehope to develop a suite of physical reasoning tasks and further investigate the insights presented
in this work.
In
addition, we would also like to incorporate the latest advancements in the suite
to see if LfI is a more feasible approach to physical reasoning
• In the experiments, we use original LfI models in our experiments without extra design for
physical reasoning tasks.
We hypothesize that better perceptual modules and more useful spatial
analyzing modules related to physical reasoning could further improve the performance of LfI.
In particular, the most challenging physical dynamics in PHYRE involve collision that is a
non-smooth and angular movement that does not lie in the traditional linear Euclidean space.
We hope to incorporate insights for physical modeling into LfI models for further refinement.
• Analyzing the experiments in Sec. 5, an additional question to ask is how accurate the dynamicsprediction needs to be in order for it to be beneficial.
We did not figure out a measure for this
problem in this work.
Carefully masking or introducing noise to specific image areas as well as
testing physics predictors at different levels may lead to an answer.
• While difficult, the dynamics prediction route remains appealing as indicated by its upper
boundand the status quo.
We are curious about how dynamics will benefit other reasoning processessuch as counterfactual reasoning and hypothetical reasoning.
We argue the dynamics predictionroute is still worth additional efforts and look forward to advances made in this field.
Societal Impacts
No foreseeable negative societal impacts in this work.
Checklist
For all authors
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See abstract and introduction in Sec. 1.
(b) Did you describe the limitations of your work?
[Yes] See Sec. 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Sec. 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] We have read the ethics review guidelines carefully and ensured that out
paper conforms to them.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We will includethem in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] We will include them in the supplemental material.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e
g.,
type of GPUs, internal cluster, or cloud provider)?
[Yes] We will include them in the
supplemental material.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See our experiments.
[Table]
(b) Did you mention the license of the assets?
[Yes] They are open-source for research.
[Table] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable? [N/A] [Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount [Table] spent on participant compensation? [N/A]