Large Language Models Are Challenged by Habitat-Centered Reasoning (2024)

Abstract: In this paper we perform a novel in-depth evaluation of text-only and multimodal LLMs’ abilities to reason about object *habitats* or conditions on how objects are situated in their environments that affect the types of behaviors (or *affordances*) that can be enacted upon them. We present a novel curated multimodal dataset of questions about object habitats and affordances, which are formally grounded in the underlying lexical semantics literature, with multiple images from various sources that depict the scenario described in the question. We evaluate 16 text-only and multimodal LLMs on this challenging data. Our findings indicate that while certain LLMs can perform reasonably well on reasoning about affordances, there appears to be a consistent low upper bound on habitat-centered reasoning performance. We discuss how the formal semantics of habitats in fact predicts this behavior and propose this as a challenge to the community.

## introduction discussion of large language models (llms) in both research and public media often gives the impression that they are capable of much more humanlike reasoning than in they really are @xcite . this makes it even more important to rigorously examine the limitations of llms. when it comes to multimodal large language models (mllms), particularly in the domain of integration with robotic systems, significant attention has been paid to affordance recognition and reasoning @xcite , particularly as it pertains to the ability to reason about changes enacted over objects in a scene. however, toward goals such as llm-enabled robots, there remain many open problems left to be solved, from migrating out of tabletop scenarios to navigating dynamic and changing environments. one such under-addressed area is habitats @xcite . these are locally-scoped environments that contextualize an object and condition the behaviors it can participate in. they are closely related to affordances @xcite , in that habitats enable or disable certain behaviors that may be executed with an object @xcite . for example, in order to screw in a screw with a screwdriver, the tip of the screwdriver must be inserted into the head of the screw. in this paper, we thoroughly evaluate the habitat and affordance reasoning capabilities of 16 openweight text-only and multimodal llms. we find that performance on affordance-centering questions frequently outstrips performance on habitatcentering questions, although all models make significant errors on multiple classes of problems. we also find that while image information provided to mllms can help performance, they are highly sensitive to perturbations in the image input; often, image information actually hurts performance, indicating weaknesses in the reasoning capabilities of mllms. our findings indicate how habitats, which are typically implicitly indicated in linguistic descriptions, are more challenging for llms and mllms to reason about than affordances, which are typically selected for by the matrix verb of sentences involving object-action descriptions. more generally, we show how llms' reasoning is susceptible to strong biases toward typical or "canonical" object orientations, because these are the collocations that typically occur in free text and common image datasets, even if they do not reflect the reality of in situ object reasoning problems. our evaluation data is collated into habitatqa, a novel reasoning and question answering dataset focused on the habitats and affordances of common objects, with accompanying images of the situations described, from various sources, including crowd-sourcing, simulation, and generative text-to-image models. our data and code is available at https://github. com/csu-signal/habitat-reasoning . there is a wealth of datasets and llm evaluations that address areas related to but nonoverlapping with habitats. a sampling are discussed below. the space dataset @xcite consists of synthetic video in a 3d environment, of containment, stability, and contact events. physob-jects @xcite is a large dataset centered on common household objects automatically annotated with physical concepts that capture human priors from the physical appearance. the handal dataset @xcite consists of images annotated with 6-dof category-level pose and scale for robotic manipulation, annotated for affordances as manipulable parts of objects. the physiclear dataset @xcite ) contains both physical/object property reasoning tasks and annotated tactile videos obtained using a gelsight tactile sensor. @xcite presented a dataset to investigate the ability of language models to predict size relationships between objects, which is a component of the habitat concept but does not cover it completely. @xcite focused on multi-modal machine comprehension (m3c) tasks given text, diagrams, and images. @xcite introduced piqa, a multiple choice question-answering dataset for physical interaction wherein the model must choose the best solution for a physical goal expressed in natural language. aroca-ouellette et al. (2021) used multiple choice questions to cover 10 basic concepts: direction, mass, height, circumference, and various object affordances. @xcite showed visual reasoning models underperform humans on part-based conceptual, relational, and physical reasoning. @xcite showed that embeddings trained over representations of affordances can be used to analogize between similar objects, such as in finding a substitute tool for a task. similarly, @xcite explored the creative problem solving capabilities of llms, with an emphasis on unconventional tool usage and how objects can be combined to achieve a complex goal. @xcite 's benchmark compares humans and distributional llms in planning and explanation generation. @xcite benchmark multimodal and text-only llms on physical attributes such as malleability, elasticity, and stiffness, using multiple choice questions. @xcite perform an assessment of llm problem solving and find them inferior to human children when it comes to choosing objects for a task based on their affordances. @xcite assessed machine common sense reasoning in mllms and non-generative models over soft bodies and fluids using video question answering, wherein the model must have a deep understanding of physical scenes and their dynamics to succeed. ser-manet et al. (2024)'s robovqa is a large dataset for robotics-oriented pick-and-place tasks. @xcite present openeqa, for embodied question answering in an open environment, focused on locational and action questions. williams and huckle (2024) present a benchmark of "easy problems that llms get wrong," including spatial reasoning. they astutely observe that the prevalence and proliferation of large scale benchmarks encourages optimization toward the benchmark, rather than a focus on holistic evaluation. in the spirit of holistic evaluation, we note that their spatial reasoning benchmark does not focus on habitats and affordances, as ours does. the above works indicate how many previous approaches address problems that are adjacent to habitat-based reasoning in language models. however, there remains a gap in directly evaluating the problem of habitats themselves, as we do here, on a wide variety of models. ## data collection our data, termed habitatqa, consists of multiplechoice questions, each with associated images that depict the scene or scenario described in the question. there are a total of 210 questions and 617 image/question pair samples (consisting of a question paired with an associated image, on which multimodal models are evaluated). each question has at least 2 associated images, and some images may be associated with more than one question, because they appropriately depict multiple scenarios described. the data was collected/constructed using the methods below. ## evaluation we performed zero-shot evaluations of habitatcentered and affordance-centered reasoning on 16 llms, including 11 text-only and 5 multimodal models, and a random guessing baseline. the models we test include members of the llama 2 (touvron et al., 2023) and llama 3 (ai@meta, 2024) families, the flan family @xcite , and the llava 1.5 and 1.6 families @xcite , with different parameter sizes, as well as unifiedqa-v2-large @xcite and blip @xcite . the default evaluation settings were used for all models, and results come from a single evaluation run. experiments were run on 2 nvidia rtx a6000 48 gb device. input to the text-only models included the question as written with the answer choices. input to the multimodal models (blip and llava) included the question plus an image resized to 512 × 512 pixels. because multiple images were associated with each question, we input the question with each image, and the accuracy of the model on that question was considered to be the proportion of times the model got the question correct. foot_2 ## results fig. 3 shows accuracy of each evaluated model on the habitat-centering and affordance-centering questions. text-only models are shown in blue and multimodal models are shown in orange. results shown are from a single evaluation run using each model's default settings. we also present values for what random guessing would achieve (in green). this was computed by performing 1,000 iterations of randomly selecting an answer for each question, and averaging accuracy over all iterations. random guessing on affordance-centering questions results in 32% accuracy, and on habitat-centering questions, 48%. in the habitat scenario particularly, all llms are performing around the level of random guessing. performance on affordances is widely variable, ranging from 22% accuracy (blip) to 77% accuracy (llama 3-70b). on affordances, larger parameter sizes within the same family of models displays at least a weak correlation with performance. at 77% accuracy, llama 3-70b almost reaches performance that is likely to be sufficient for many reasoning tasks involving affordances. there does not exist a direct comparison on affordance reasoning, but one can consider reported human performance on foundational attribute comprehension by @xcite , which hovers around 80% agreement with majority human response. meanwhile the best performing multimodal model was llava 1.6-34b at 69% accuracy, which is a small improvement on contemporaneous text-only models, like llama 2-70b, implying that images concerning affordances provide a modest performance boost to a sufficiently large model. by contrast, we actually see very consistent, if mediocre, performance on habitats. all models, even the newest ones, hit a performance plateau with an upper bound of 57%. the bestperforming models, unifiedqa-v2 and llava 1.6-34b, achieve only 57% and 55% accuracy, respectively, which is only about 7-9% better than random chance. we also see that the llava family of models, which, as multimodal models, are evaluated using image inputs, performs consistently worse on habitats than affordances. performance by image type when evaluating llava 1.6-34b, the best-performing multimodal model, according to the type of image used in the input (natural, generated, or simulated), we observe that images of different provenance may perform differently. habitats image type mean performance failure rate natural 54% (.46) 38% generated 57% (.49) 38% simulated 75% (.50) 25% affordances image type mean performance failure rate natural 63% (.42) 26% generated 82% (.33) 20% simulated 62% (.48) 35% llama 3-70b misses the rolling affordance of the multifunctional glass object. if we consider the afforded behavior roll in the context of its verbnet class, roll alternates with slide, which can happen to a glass in its default orientational configuration (habitat), and so the collocation "glass" + "slide" is likely more common in the training data of even a very large model in contexts in which "roll" might also occur. similarly, llama 3-70b also failed on questions like the following: which of the following objects has a flat surface that would allow it to slide? a) a beach ball b) a book c) a sphere d) a cylinder e) b and d are correct choices knowledge is needed of the flat or round parts of objects to accommodate the corresponding action. in general we observe that models have difficulty selecting innovative or non-traditional uses of objects, like using a bowl or glass to scoop. this indicates that challenging affordance reasoning typically involves some knowledge of object orientation, configuration, and habitat. the best performing habitat models, unifiedqa-v2 and llava 1.6-34b, fail most frequently on questions involving resultant states under transformations, including counterfactuals: there is an upright coffee pot on a table. how would its bottom be positioned if the coffee pot were rotated 180 degrees? a) coffee pot bottom would face upward b) coffee pot bottom would touch the table unifiedqa-v2 fails on questions that isolate a counterfactual or resultant state and habitat chains, and appears to confuse concavity and convexity: a cup is placed mouth-down in the pot. how is the cup's concave side oriented? a) facing toward the opening of the pot b) facing toward the bottom of the pot when we examine questions that llava 1.6-34b never answered correctly with any corresponding image, we see patterns relative to the categories in vision-language grounding blip is notably the worst-performing model on affordances, but generally on par with all other on habitats. given this discrepancy, an examination of some outputs of blip's habitat-centered reasoning is illustrative of where multimodal model failures may be occurring. in both cases, the visualization of blip attention over "legs" and "downward" have a very similar distribution (over parts of the legs). this highlights the strong bias toward "downward" that is introduced by "legs," even when the image obviously depicts the opposite. the image input introduces noise given the biases inherent in the model. that is, a telic role, as would be expressed in, e.g., qualia structure @xcite : ## conclusion in this paper, we presented a thorough evaluation of llms' and mllms' reasoning capabilities focused specifically on the concept of "habitats", compared to affordances. we also developed a novel challenge dataset for this task. our data collection and collation was theoretically-grounded in the lexical semantics underlying habitats and affordances. to our knowledge, this is the first dataset and evaluation designed specifically for the underaddressed problem of habitat-centered reasoning and for both text-only and multimodal evaluation. we found that llms'/mllms' performance on affordance-centered reasoning was highly variable but in the best cases approached 70-80% accuracy, which previous research has indicated is roughly aligned with human performance @xcite . by contrast, performance on habitatcentered reasoning consistently plateaued no higher than 57%, across all models. this indicates that habitat-centered reasoning remains a challenge for llms and their multimodal variants. in many cases, we found that image information actually adversely impacted performance on habitats. this can be attributed to biases toward canonical orientations that previous research @xcite has found to be pervasive in image training datasets. further, @xcite 's formalism of the telic role suggests why artifacts are overwhelmingly likely to be discussed in terms of their canonical uses and associated orientations, such that when habitats are exploited to perturb that canonical alignment, it poses a unique challenge to modern models that future work must addressed to achieve true common-sense multimodal reasoning. specific approaches may include augmenting llms with object and counterfactual reasoning and making them more common-sense oriented towards these types of questions using knowledge distillation @xcite . while the consistent low performance of habitat reasoning highlights this challenge, the variability of the better-explored affordance reasoning performance, and sensitivity to small differences, highlight a need for better methods of guaranteeing or predicting model performance.