Identifying Physical Object Use in Sentences (2022)

Abstract: Commonsense knowledge about the typicalfunctions of physical objects allows people tomake inferences during sentence understanding.For example, we infer that “Sam enjoyedthe book” means that Sam enjoyed reading thebook, even though the action is implicit. Priorresearch has focused on learning the prototypicalfunctions of physical objects in order toenable inferences about implicit actions. Butmany sentences refer to objects even when theyare not used (e.g., “The book fell”). We arguethat NLP systems need to recognize whether anobject is being used before inferring how theobject is used. We define a new task called ObjectUse Classification that determines whethera physical object mentioned in a sentence wasused or likely will be used. We introduce a newdataset for this task and present a classificationmodel that exploits data augmentation methodsand FrameNet when fine-tuning a pre-trainedlanguage model. We also show that object useclassification combined with knowledge aboutthe prototypical functions of objects has thepotential to yield very good inferences aboutimplicit and anticipated actions.

## introduction physical objects play an important role in daily life. people use them for different purposes, for example we use knives for cutting, cars for transportation, and books for reading. recent work by @xcite argued that most humanmade physical artifacts were created for a specific purpose, and that commonsense knowledge about an object's prototypical function is essential for natural language understanding. for example, "she finished the puzzle" and "she finished the cigarette" implicitly refer to different actions associated with puzzles (solving) vs. cigarettes (smoking). similarly, humans interpret "he used a gun" as a shooting but "he used a knife" as a stabbing based on our knowledge of guns and knives. @xcite developed a method to learn the prototypical functions for physical artifacts from text corpora, with the goal of producing a commonsense knowledge resource for physical objects. however an open question is how to apply this knowledge for sentence understanding. it would be risky to assume that objects are always used in the most typical way because objects can be used in atypical ways too. for example, "max used the knife to open the bottle" probably means that max popped the top off the bottle with the knife, not that max cut the bottle. but as we will discuss in section 4, our study found that physical artifacts are used in the prototypical way most of the time (96%), so it is a very good assumption. a much bigger problem for applying knowledge of prototypical functions is that physical objects are often mentioned when they are not used at all! for example, the sentences below mention a knife, but the knife is not being used: in addition, some sentences suggest that a physical object will be used, although it has not been used yet. for example, mary aims to acquire a knife through various actions in the sentences below: when reading these sentences, people naturally infer that mary intends to use the knife, most likely in the typical way (i.e., to cut things). we believe that nlp systems should also make these predictive inferences to "read between the lines" during narrative text understanding. for example, consider the sentence "the fish was too big for the freezer, so mary got a knife.". human readers would assume that mary used the knife to cut the fish into smaller pieces, even without any explicit mention of cutting. we propose a new nlp task, object use classification, to classify the usage status of physical objects mentioned in sentences with respect to three categories: used, anticipated use, and no use. our first goal is to identify sentences that state or imply that an object was used (used) to enable prototypical function inferences when the action is implicit. our second goal is to identify sentences that describe actions which suggest someone's probable intent to use the object (anticipated use) to enable second-order prototypical function inferences. finally, identifying sentences where there is no use of a mentioned object (no use) is important to recognize when prototypical function inferences should not be applied. we introduce a new object use dataset for this task with gold standard human annotations of sentences that mention physical objects. we found that all three use categories are common: our annotators labeled 45% of the sentences as used, 28% as anticipated use, and 27% as no use. we explored several methods to tackle this task. first we applied prompting methods using two large language models to evaluate a zero-shot generalization approach, but the results were mediocre. next, we fine-tuned a transformer-based model, which yielded much better performance. finally, we added two data augmentation techniques, synonym replacement and back translation, and also provided exemplar sentences associated with the object's prototypical function frame. our experimental results show that the complete model achieves good performance for this task. ## related work commonsense knowledge has long been recognized as an essential part of natural language understanding @xcite @xcite . some work specifically argued that commonsense knowledge about physical objects is often used to make inferences and plays an important role in narrative text understanding @xcite . recently, a variety of projects have focused on acquiring knowledge about physical objects, including relative physical knowledge @xcite , relative spatial relations @xcite , location knowledge @xcite , and object affordance @xcite . @xcite developed a method to learn the most typical way that people use human-made physical artifacts, and they used framenet frames as a representation for common object functions. for their work, they created a dataset of physical objects annotated with their prototypical functions. our research builds upon that work by developing a model for identifying the usage status of physical objects mentioned in a sentence, which we argue is a necessary precursor to applying prior knowledge about prototypical functions. recently, there have been efforts aimed at learning implicit information with pre-trained language models. @xcite explored using pretrained masked language models to capture implicit knowledge elicited from humans, which are so-called stereotypical tacit assumptions. @xcite created a question answering dataset consisting of questions that require implicit multistep reasoning skills, such as "did aristotle use a laptop?". they show that a large language model fine-tuned on related datasets without retrieval of relevant knowledge performs far worse than humans, and high-quality retrieval makes the model more effective in the reasoning process. @xcite trained language models with automatically generated data sampled from existing knowledge sources. they show that language models can combine implicit knowledge encoded in their parameters with explicit rules and facts, and further perform reasoning. our work also has ties to frame semantics @xcite , a theory of how we associate words and phrases with cognitive structures called frames. frame semantic parsing @xcite is the task of automatically extracting frame semantic structures from sentences, based on the berkeley framenet project @xcite . the process begins with identifying frameevoking words in the sentence (target identification) and identifying the evoked frame for each target (frame identification @xcite ). however, one limitation of this setting is that it typically relies on the predicate to predict the frame. for example, the sentence "sam enjoyed the book" would not trigger a reading frame because "enjoy" is not associated with read-ing in framenet. our work strives to identify this implicit action by recognizing that the book was used and then applying the prototypical function associated with books. ## motivation actions involving physical objects are often left implicit in natural language. in many cases, these actions do not need to be explicitly stated because they can be easily inferred by people using our knowledge about physical objects. early nlp research recognized this need for commonsense knowledge about physical objects (e.g., burstein (1979)) and some efforts have been undertaken to compile such knowledge, including conceptnet @xcite , which contains a "usedfor" relation that captures possible uses for an object expressed in natural language, and recent work by @xcite that learns to associate physical objects with framenet frames describing their prototypical uses. however, a crucial question is when to apply this knowledge in sentence understanding. in this paper, we claim that nlp systems must be able to distinguish between (1) sentences that mention a physical object and state or imply that the object was or will be used, and (2) sentences that mention a physical object but the object was not used. for example, the sentences "mary read the book" and "mary enjoyed the book" both imply that the book was used (read), but "mary dropped the book" does not mean that the book was used, only that mary was carrying it. we found that about 73% of sentences that mention a physical object in our data set (see section 4) suggest that the object was (45%) or will be (28%) used. for the other 27% of sentences that mention a physical object, the object was not used at all. consequently, we argue that an important task for understanding sentences about physical objects is object use classification. a second question relates to the applicability of "prototypical" functions for objects: when an object is used, how often is it used in the prototypical way? in our dataset (section 4), we found that when a sentence mentions or implies the use of an object, 96% of these sentences correspond to the prototypical use for the object. only 4% of these sentences suggest that an object was used in an atypical way. these results indicate that an effective object use classifier can go a long way toward enabling nlp systems not only to infer whether an object was used, but also how an object was used, even when that action is not explicitly stated. in this paper, we tackle the problem of object use classification and define three categories of object use: used, anticipated use, and no use. in the next section, we define these three categories and explain our motivation for them, and we present a new object use dataset for this task. ## dataset creation since we are tackling a new task, we created a new toucan (textual object use classification) dataset foot_0 with gold standard human annotations. our primary goal was to obtain human judgements for sentences that mention a physical object indicating whether the object was or will be used, or not. but a second goal was to better understand how often objects are used in a prototypical way, as opposed to an atypical way. so we obtained additional human judgements for the sentences in which an object was or will be used and asked the annotators to determine whether the use corresponds to the object's prototypical function. we leveraged the results of prior work that studied physical objects and their prototypical functions so as not to reinvent the wheel. physical objects: we use the list of physical objects produced by @xcite . they extracted human-made physical objects with sense definitions from wordnet @xcite , and used a concreteness dictionary @xcite to filter out abstract terms. their freely available dataset foot_1 contains 938 human-made physical object terms. sentences: then we extracted sentences containing these physical objects from the spinn3r corpus @xcite , which consists of 44 million blog posts. in order to get a uniform distribution of different physical objects, we randomly sampled 4 sentences (or fewer if there are not enough) for each physical object. this produced a set of 2,460 sentences in total. ## object use classification models we explored several approaches to tackle this task. we first present a transformer-based model finetuned solely on our gold standard training sentences. then we present a method that takes advantage of two commonly used data augmentation techniques, synonym replacement and back translation. finally, we also show that our model further benefits from prior knowledge of the object's prototypical function by incorporating the exemplar sentences associated with its function frame. ## evaluation we split our gold standard data set into roughly 70% for training, 15% for development and 15% for testing. we also made sure that the objects in the test set do not appear in the training set. our fine-tuning framework is based on the robertabase model @xcite . for the hyperparameters, we used a max sequence length of 192, a batch size of 8, learning rate initialized as 2e-5, and train for 15 epochs. each result is averaged over three runs with different random seeds. we report overall accuracy as well as precision, recall and f1 scores macro-averaged over the 3 classes. ## conclusion we introduced a new nlp task, object use classification, which identifies whether an object mentioned in a sentence has been used or likely will be used. we introduced a gold standard dataset for this task and showed that all 3 categories (used, anticipated use, and no use) are common in real sentences. then we presented a transformer-based architecture for this task that uses two types of data augmentation techniques (synonym/hyponym replacement and back translation) and also exploits exemplar sentences from framenet that correspond to an object's prototypical function. the resulting classification model achieves reasonably good performance for this task, although there is room for improvement that we hope will inspire future work on this problem.