T ext2 A fford: Probing Object Affordance Prediction abilities of Language Models solely from Text (2024)

Abstract: We investigate the knowledge of object affordances in pre-trained language models (LMs) and pre-trained Vision-Language models (VLMs).A growing body of literature shows that PTLMs fail inconsistently and non-intuitively, demonstrating a lack of reasoning and grounding. To take a first step toward quantifying the effect of grounding (or lack thereof), we curate a novel and comprehensive dataset of object affordances – Text2Afford, characterized by 15 affordance classes. Unlike affordance datasets collected in vision and language domains, we annotate in-the-wild sentences with objects and affordances. Experimental results reveal that PTLMs exhibit limited reasoning abilities when it comes to uncommon object affordances. We also observe that pre-trained VLMs do not necessarily capture object affordances effectively. Through few-shot fine-tuning, we demonstrate improvement in affordance knowledge in PTLMs and VLMs. Our research contributes a novel dataset for language grounding tasks, and presents insights into LM capabilities, advancing the understanding of object affordances.

## introduction object affordance refers to the properties of an object that determine what actions a human can perform on them @xcite . gaining the knowledge of object affordances while learning textual representation from large corpora maybe hard; as in nlp, we lack corresponding images (or videos) which provides necessary visual cues such as shape, color, and texture to predict affordances. this lack of mapping or rather grounding ability has been noted by many researchers in the context of large pretrained language models (ptlms). authors in @xcite have pointed the lack of symbol grounding to be a fundamental factor behind ptlms failing to grasp meaning from form (surface form text). the authors argue that language models which are exposed to only text (surface form) may never truly understand meaning, as ptlms are unaware of possible groundings of the surface text. most current nlp datasets and tasks are not designed to evaluate grounding, as it is hard to evaluate grounding without any visual context. here, we aim to quantify the ability of pretrained models to learn affordances -which in turn requires the ability to ground symbols in text to real-world objects. in other words, grounding ability from text can enable understanding and reasoning about the physical properties of an object, which may help predict affordances. as another example, for the sentence "an apple in the tree", we should infer that the "apple" can be eaten, and is rollable. however we cannot roll an "apple logo". in computer vision and robotics efforts, an accompanying image (or video) often provides necessary information about shape and physical properties of the entity, which can be used to predict affordances @xcite . however such information is absent in nlp tasks. to capture this nuance, we annotate crowdsourced text intended for other tasks (such as nli) with the dataset train size dev size test size reasoning type source imagedependent targeted affordance publicly available paco @xcite 5,580 1,860 4,960 preconditioned commonsense crowd-sourced ✗ ✗ ✔ winoventi @xcite --4,352 commonsense with exceptions crowd-sourced ✗ ✗ ✔ pvlir @xcite 34,000 preconditioned visual commonsense other dataset ✔ ✗ ✗ normlense @xcite --10,000 defeasible visual commonsense crowd-sourced ✔ ✗ ✔ winoviz @xcite objects and affordances. we use 15 affordance classes from @xcite . through extensive pilot studies, we train a set of annotators using the toloka.ai platform. we choose 25 highly-skilled annotators who annotated a total of 2368 sentenceobject pairs with 15 affordance classes, on a 0-3 likert-like scale. for each sentence-object pair and each affordance class we ensure annotations from three annotators to enable majority votings. we name this novel dataset text2afford. we use the dataset for zero-shot evaluations of small lms, open-source llms and also some vlms by forming different task setups. figure 1 presents an example from text2afford and the derived tasks (detailed in section 4). we evaluate the effect of few-shot fine-tuning on few ptlms and vlm. our contributions can be summarized as follows. reasoning about object affordances. object affordances has been extensively studied in computer vision and robotics @xcite ; @xcite ). recent methods employ deep learning approaches to detect object affordance. nguyen et al. (2017) applies an object detector, cnn and dense conditional random fields to detect object affordance from rgb images. persiani and hellström (2019) extracts object-action pairs from web corpora using semantic role labelling. in contrast, we propose a crowd-sourced text only affordance dataset to audit the strength of sota llms and vlms to reason about contextualized object affordance. probing methods. talmor et al. (2020) utilizes probing and employs multi-choice mlm (masked language modelling) and multi-choice qa (question answering) setup to capture reasoning capabilities of pre-trained language models. yang et al. (2022) examines zero-shot prediction performances on different tasks by llm through novel visual imagination. aroca-ouellette et al. (2021) highlights the shortcomings of state-of-the-art pretrained models in physical reasoning, with a further performance decline observed when incorporating option shuffling and superlatives in reasoning questions. liu et al. (2022) proposes a novel spatial commonsense probing framework to investigate object scales and positional relationship knowledge in text-based pre-trained models and models with visual signals. joshi et al. (2020) uses probing methods to investigate a more fine-grained logical reasoning capabilities of pre-trained models. reasoning tasks and dataset. reasoning about object affordance require a sort of commonsense reasoning. a series of works ( singh et al. (2021) 2018 )) study the text based commonsense knowledge of language models. dataset such as δ-nli @xcite focuses on defeasible inference of commonsense knowledge; paco @xcite and pinks @xcite deal with preconditioned commonsense inference of language models. pvlir @xcite , normlense @xcite use images as precoditions to reason about defeasible commonsense norms. however, none of these specifically focus on reasoning of object affordance. @xcite proposes a benchmark of object-attribute pairs plus a diverse set of questions to reason object's physical properties. aroca-ouellette et al. (2021) tackles physical and affordance reasoning from an object-centric approach. @xcite attempts to extract common object-action pairs from web corpora. in ## text2afford dataset construction we select 20, 000 sentences from a crowdsourced english dataset (xnli english) @xcite 2 and extract the noun phrases using the stanford corenlp tool. as we restrict to the affordances that humans can directly perform, we filter the phrases which do not represent a tangible object (using conceptnet). we manually filter out objects that cannot be acted upon directly by humans (such as school, building). after this preprocessing, we obtain a set of sentence-object pairs (⟨x i , o i ⟩), where the sentence acts as the context for the corresponding object. each sentence on average has 2-3 such objects. we use the 15 predefined affordance classes from @xcite to label each sentence-object pair for annotation. we utilize the toloka platform foot_1 for conducting the data annotation. we design an interface on this platform, containing clear instructions and examples for annotating the data. we conduct two rounds of pilot studies along with additional ama (ask me anything) sessions to analyze the subjective understanding of the annotators and, thereby, only select the high quality, serious annotators. a total of 114 annotators participated in the pilot study, and out of that we finally engage 25 skilled annotators to annotate a total of 2,368 sentence-object pairs each containing 15 affordance classes. each datapoint (i.e., sentence-object pair along with an affordance class) has been annotated by three different annotators. we provide the details of the pilot studies & annotator training in appendix a.1. by evaluating the complexity of the task for the annotators from the pilot studies, we intentionally consider a relatively small number of datapoints at a point for the annotation. this leads us to a total of 10 phases to complete the final annotation. we carefully reviewed each annotation and provided feedback with guidance in case of mistakes. for instance, annotators initially got confused with the affordance 'watch' as human can watch any visual objects. in another instance, some annotators asked whether 'throw' can be valid affordance for the object 'kittens' as humans can perform 'lift', 'throw' to the object 'kitten'. we discussed these types of ambiguities with the annotators after each phase. throughout each of the data annotation phases, we put scrupulous attention to quality con- agreement category affordance classes objects object-affordance pair high agreement (>0.75) row, feed, ride, fix the horse, striped white shirts, a brown paper sack, chinese lanterns, adrin's sword, the movie breakfast-feed, a horse-watch, crops-fix, sports-grasp, sports-lift, sports-push, the phone-feed, football-ride medium agreement (>0.4 & <0.75) throw, pourfrom, writewith, lookthrough, lift, grasp, play, push a red flag, an arrow, art, automatic weapons, babies, black-and-white tv computers-writewith, cats-feed, football-play, book-writewith, the door-push low agreement (<0.4) watch, siton, typeon brandy from spain, stone circles, iron, batteries, his fist, historical artifact, gift, olive oil, outdoor tables, bumper sticker on a car weapon-push, the table-lift, boat-fix, paintings-lookthrough, cats-throw trol, including iterative annotation refinement, and manual evaluation. the overall statistics for this currently constructed dataset -text2afford is in ## task description our first objective is to audit the strength of large language models in identifying the pre-defined affordance classes of objects from text in zero-shot settings. given a textual context, and the object, the task is to predict whether a particular affordance class is applicable to that object conditioned on the context. we majorly leverage 4 types of task setup for the experiments. for the encoder based models (e.g., roberta, bert) we choose masked language modelling (mlm) and natural language inference (nli) based setup, and for the generative models we adopt 2 types of probing setup (text only, text+image) to formalize the task. ## experiments we explore various state-of-the-art baselines using pre-trained language models (roberta-large, bart-large), instruction-fine-tuned large language models (e.g., flan-t5, falcon, chatgpt, llama-3), pre-trained multi-modal vision and language architectures (clip-vit, vilt, instructblip, idefics, llava). we observe whether these models gain the knowledge of affordances through their pre-training, fine-tuning on commonsense tasks (nli, piqa), or few-shot fine-tuning scenarios. ## benchmarking text2afford prediction evaluation metric. to assess the performance of the zero-shot affordance prediction, we calculate accuracy in the following way. each affordance class is treated as a binary classification problem, where a value of 1 represents a positive class indicating that the affordance can be performed on the object, and a value of 0 represents a negative class indicating that the affordance cannot be performed. for each positive @xmath0, p 2 , ..p n }, we compare the predicted scores of that affordance class with the predicted scores of the negative @xmath1, n 2 , ..n m }. if the predicted score of the positive class is higher than the predicted score of all the negative classes (i.e., p(p i ) > p(n j ) ∀j ), we increment the correct count by 1 5 . conversely, if the predicted score of the negative class is higher, we increment the wrong count by 1. the final accuracy is calculated by dividing the total number of correct counts by the total number of the instances. to rank the affordance classes based on the predicted score, we also report the mean average precision (map@k, where k is the number of affordance classes). zero-shot performance. ## text2afford for physical reasoning apart from benchmarking llms and vlms, we observe whether text2afford can be used as a source of affordance knowledge. we choose the physical commonsense reasoning as a target as the 'object affordance' represents an innate physical property of an object, and we believe that any language model having strong affordance reasoning capability can enhance the physical reasoning capability. to explore this, we perform an 'instruction fine-tuning' on the text2afford dataset (although it is not meant for training) using few opensource llms (llama-3-8b-instruct, flan-t5), and test on two physical reasoning dataset -( 1 ) prost (aroca-ouellette et al., 2021), which contains 10 types of different physical properties of an object (including 6 affordance properties -rolling, breaking, stacking, grasping, sliding, bouncing) along with complex reasoning questions, and ( 2 ) piqa @xcite which, focuses on selecting appropriate option given a situation that requires physical commonsense. for prost, using llama-3 the accuracy boosts from 0.36 to 0.42 after instruct fine-tuning with text2afford. moreover, out of the 6 affordance properties from prost, the accuracy got boosted for the reasoning of 5 affordance properties. for the piqa, the same llm gives a maximum of 4% accuracy boost. the full result is shown in ## additional details reason for choosing xnli. we select xnli to incorporate object references from less conventional and commonly explored scenarios. unlike typical object identification datasets, xnli offers sentences derived from novels, thus presenting a more in-the-wild textual context, which adds complexity and diversity to our dataset. specifically, we choose the hypothesis portion of the xnli sentences due to its shorter context length. this choice intentionally poses a challenge to llms, allowing us to better evaluate their reasoning capabilities, especially when dealing with minimal contextual information. non-explicit mention about contextual object affordance in the instruction. the instructions shown in appendix figure 3 represent the initial guidance provided to annotators as an introduction to the task. since understanding contextual object affordance can be challenging for non-expert annotators, this initial step was designed to give a basic idea of the task. however, we follow this up with a comprehensive training process and conduct two ama (ask me anything) sessions to ensure that annotators fully understood the need to base their judgments on the provided context. these efforts are key in ensuring high-quality annotations throughout the dataset creation. reason for choosing 0-3 likert scale in data annotation. we opt for a 0-3 likert scale (4point) to minimize the potential for neutral or noncommittal responses, which can often arise when a midpoint option is available. our initial observations indicated that some annotators tended to select an "average" value without fully considering the contextual affordance of the objects, which diminished the depth of their evaluations and limited the discussion around ambiguities. by adopting a 4-point scale, we aim to encourage more decisive judgments. in addition, we provide a textbox (see appendix figure 3 ) for annotators to express any uncertainties or ambiguities they encountered, which has helped us in capturing more nuanced feedback. reason for choosing visual genome. we chose visual genome as a primary source for real images due to its rich, complex scenes, which are widely used in visual reasoning tasks. the complexity of the images in visual genome provides diverse contexts that align well with the goals of our study, which focuses on contextual object affordances. while other methods, such as using search engines like bing, have been employed in prior work to retrieve images, we opt for visual genome to ensure that the images contain sufficient contextual and visual detail to support affordance prediction, even if there are minor limitations in reasoning. reason for choosing stable diffusion. regarding the use of stable diffusion, we have been inspired by its demonstrated capability to generate high-quality, realistic images, particularly in prior studies where it was effective in reasoning tasks. while clip is primarily trained on real-world images, we hypothesize that stable diffusion could generate contextual images with sufficient accuracy to complement the real images. the generated images provide additional diversity, which helps us explore the affordance prediction task from a different angle. the benefit of using stable diffusion lies in its ability to create controlled, context-specific images that may not always be available in existing datasets, providing a broader range of testing scenarios for our models. reason for framing generative tasks as a binary decision problem. in the generative setting, we opt for a binary yes/no classification to evaluate the affordance of individual context-object-affordance triples. we decide this based on the observation of the tendency of smaller llms to hallucinate, which can make direct affordance prediction challenging, particularly in zero-shot scenarios. by framing it as a binary classification task, we aim to simplify the evaluation and obtain more reliable results. in addition, our approach allows for a comprehensive evaluation of both positive and negative affordances. this is critical for our dataset, as it is designed to assess affordances that are applicable, as well as those that are not, in a given context. ## conclusion in this paper we introduced a novel text-based affordance dataset text2afford to investigate the affordance knowledge of ptlms and pre-trained vlms in different zero-shot settings. our findings suggest that, the state-of-the-art language models, particularly text-to-text models, still exhibit limitations in their ability to reason about object affordances. in this seemingly easy task, we observe how context can introduce various levels of ambiguity and difficulty. we also observe, that even in the presence of such difficulty, human performance is superior and llms/vlms still face difficulty in gaining such knowledge during their pretraining. additionally, we observe how our dataset provides some additional knowledge that can be useful for physical commonsense reasoning -stressing its orthogonality more with respect to the pretraining knowledge llms and vlms possess.