ESPRIT : Explaining Solutions to Physical Reasoning Tasks (2020)

Abstract: Neural networks lack the ability to reason about qualitative physics and so cannot generalize to scenarios and tasks unseen during training. We propose ESPRIT, a framework for commonsense reasoning about qualitative physics in natural language that generates interpretable descriptions of physical events. We use a two-step approach of first identifying the pivotal physical events in an environment and then generating natural language descriptions of those events using a data-to-text approach. Our framework learns to generate explanations of how the physical simulation will causally evolve so that an agent or a human can easily reason about a solution using those interpretable descriptions. Human evaluations indicate that ESPRIT produces crucial fine-grained details and has high coverage of physical concepts compared to even human annotations. Dataset, code and documentation are available at https://github.com/salesforce/esprit .

## introduction humans learn to understand and reason about physical laws just by living in this world and doing everyday things. ai models, on the other hand, lack this ability and so are unable to generalize to new scenarios that require reasoning about abstract physical concepts like gravity, mass, inertia, friction, and collisions @xcite . we propose explaining solutions to physical rea-soning tasks (esprit), a framework for explaining qualitative physics reasoning using natural language. neural networks with knowledge of qualitative physics would have commonsense reasoning abilities about the way the world works @xcite . in turn, this could, for example, improve performance on tasks that involve interacting with humans and make human-robot interactions more efficient and trustworthy. figure 1 : an example from the phyre dataset @xcite consisting of a goal, an initial scene, a solution -the action of adding a red ball, and the resulting simulation rollout. each object color corresponds to an object type. red: user-added dynamic object; green and blue: dynamic goal object; gray: dynamic scene object; black: static scene object. ideally, ai systems would reason about and generate natural language commonsense explanations of physical concepts that are relevant to their behavior and prediction. a key intuition is that natural language can provide an efficient low-dimensional representation of complicated physical concepts. to equip ai systems with this ability, we collected a set of open-ended natural language human explanations of qualitative physics simulations. the explanations include descriptions of the initial scene, i.e., before any physics is at play, and a sequence of identified pivotal events in a physics simulation. three physical concepts are crucial for our simulation to reach a specified goal state: gravity, collision, and friction. our work attempts to build an interpretable framework for qualitative physics reasoning with strong generalization abilities mirroring those of humans. esprit is the first-ever framework that unifies commonsense physical reasoning and interpretability using natural language explanations. our framework consists of two phases: (1) identifying the pivotal physical events in tasks, and (2) generating natural language descriptions for the initial scene and the pivotal events. in the first phase, figure 2 : the end-to-end esprit framework for identifying pivotal physical events, extracting the features from pivotal events in a table, and explaining solutions using a table-to-text model for natural language generation. the purple bar is a static goal object. the input to esprit is a sequence of frames from a physics simulation and the output is a natural language narrative that reflects the locations of the objects in the initial scene and a description of the sequence of physical events that would lead to the desired goal state, as shown in figure 2 . the first phase of the framework uses a neural network classifier to identify salient frames from the simulation. for the second phase we experimented with table-to-text models @xcite as well as pre-trained language models @xcite . we evaluated our framework for natural language generated reasoning using several automated and human evaluations with a focus on the understanding of qualitative physics and the ordering of a natural sequence of physical events. we found that our model achieves very high performance for phase one (identifying frames with salient physical events) and that, for phase two, the table-to-text models outperform pre-trained language models on qualitative physics reasoning. ## evaluation metrics we evaluate our models using both automatic metrics and human evaluations. ## experimental results and discussion from the decision tree, we extract feature importance values for each of the 13 input variables described in section 3. the most important variable is the time step of the collision, with a weight of 0.178. the most important features for clas-sification were an object's collision position, its velocity, and then its angle of rotation. given such strong results for identifying pivotal events, we were able to predict the salient events of previously unseen simulations and that helped in the next step of generation descriptions of salient events. human evaluation -validity. while the gpt model can achieve scores comparable to the datato-text models using automated metrics, its performance using human evaluation is as good as chance, as shown in human evaluation -coverage. ## conclusions and future directions esprit uses a two-step approach for qualitative physical reasoning. to train models that can describe physical tasks, we collected open-ended natural language text descriptions of initial states and pivotal physical events in a 2d simulation from human annotators. we then trained a model to identify these pivotal events and then fine-tuned on pre-trained table-to-text generation and language models without using the image representations of the actual simulation frames. our results indicate that table-to-text models perform better than language models on generating valid explanations of physical events but there is a lot more room for improvement compared to human annotations. we hope that the dataset we collected will facilitate research in using natural language for physical reasoning. reinforcement learning (rl) agents may be able to solve physical tasks much more efficiently by leveraging natural language reasoning as opposed to model-free approaches that are often highly sample-inefficient. an rl agent that leverages natural language descriptions of physical events to reason about the solution for a given goal (similar to @xcite ) or for reward shaping (similar to @xcite ) could be a compelling line of future research. more importantly, having a model that can meaningfully reason about commonsense qualitative physics could be interpretable and more robust, as they might focus on the parts of physical dynamics that are relevant for generalization to new scenarios. such systems are widely applicable to self-driving cars or tasks that involve human-ai interactions, such as robots performing everyday human tasks like making coffee or even collaboratively helping with rescue operations.