Probing Physical Reasoning with Counter-Commonsense Context (2023)

Abstract: In this study, we create a CConS (Counter-commonsense Contextual Size comparison) dataset to investigate how physical commonsense affects the contextualized size comparison task; the proposed dataset consists of both contexts that fit physical commonsense and those that do not. This dataset tests the ability of language models to predict the size relationship between objects under various contexts generated from our curated noun list and templates. We measure the ability of several masked language models and encoder-decoder models. The results show that while large language models can use prepositions such as “in” and “into” in the provided context to infer size relationships, they fail to use verbs and thus make incorrect judgments led by their prior physical commonsense.

## introduction humans possess physical commonsense regarding the behavior of everyday objects. physical commonsense knowledge is relevant to their physical properties, affordances, and how they can be manipulated @xcite . while a significant amount of physical commonsense can be expressed in language @xcite , direct sentences describing facts such as "people are smaller than houses" rarely appear because of reporting bias @xcite . recent language models have succeeded in tasks that do not require contextual reasoning, such as size comparison and prediction of event frequency @xcite . however, what about inferences that are contextdependent? whether a language model can make correct inferences in various contexts is important because physical reasoning is highly contextdependent @xcite . several studies on contextual physical reasoning @xcite @xcite template:<*> is @xmath0. a house is in an eletctric bulb. is the electric bulb bigger than the house? an electric bulb is in a house. ## experiment task definition we measure the ability of masked language models and generative models to recognize size relationships by providing sentences for each architecture. these sentences are generated from templates (section 3). we also see how the language model's behavior changes when context sentences follow or do not follow a general common-size relationship. ## result and analysis tables 2 and 3 exhibit the performance of the language model on our datasets. gpt-3 outperforms other models in ordinary and nocon setups. roberta-large and albert-xxlarge show better reasoning ability than the other masked language models in the ordinary dataset. however, for the ccommon dataset, the performance of the pretrained language model decreases, particularly in albert-xxlarge. this result suggests that commonsense built into the model hinders its ability to make accurate judgments. other models struggle to capture size relationships. these results without context (nocon) are generally consistent with the findings of a previous investigation of the nocontext size comparison task conducted by @xcite . in some ccommon examples, bert performs better than roberta. this may be because bert is less equipped with commonsense, allowing it to make simpler judgments without being influenced. ## conclusion we develop a method providing a countercommonsense context to measure physical reasoning ability. our proposed contextualized physical commonsense inference dataset reveals that current language models can partially predict size relations but do not perform as well as humans in contexts that contradict commonsense. these judgments are possible to a limited extent in the presence of certain prepositions such as "in" and "into." while we focused on size comparison tasks in this study, the importance of context in physical reasoning is not limited to this task. increasing the size and scope of the datasets for contextual commonsense inference is necessary to build language models that more closely resemble humans and differentiate between general commonsense and the facts at hand.