Humans Meet Models on Object Naming: A New Dataset and Analysis (2020)

Abstract: We release ManyNames v2 (MN v2), a verified version of an object naming dataset that contains dozens of valid names per object for 25K images. We analyze issues in the data collection method originally employed, standard in Language & Vision (L&V), and find that the main source of noise in the data comes from simulating a naming context solely from an image with a target object marked with a bounding box, which causes subjects to sometimes disagree regarding which object is the target. We also find that both the degree of this uncertainty in the original data and the amount of true naming variation in MN v2 differs substantially across object domains. We use MN v2 to analyze a popular L&V model and demonstrate its effectiveness on the task of object naming. However, our fine-grained analysis reveals that what appears to be human-like model behavior is not stable across domains, e.g., the model confuses people and clothing objects much more frequently than humans do. We also find that standard evaluations underestimate the actual effectiveness of the naming model: on the single-label names of the original dataset (Visual Genome), it obtains −27% accuracy points than on MN v2, that includes all valid object names.

## introduction research on object naming @xcite @xcite , such as the linguistic analysis of the naming behavior of humans and computational models, requires natural and reliable object naming data. such data should contain naturally occurring naming variation, i.e., that the same object can very often be called by different names. for instance, a duck can be called duck, bird, animal, etc. at the same time, in cases where a naming dataset provides multiple names for a given object, it is important to verify that this apparent naming variation is not a mere consequence of, for instance, lexical errors or different people naming different objects in the same scene. in this work, we assess the factors that affect the collection of object naming data in the typical language & vision (l&v) setup, and test whether these factors impact the accuracy or apparent accuracy of a l&v object labeling model. existing work in l&v has relied on data collection methods that prompt natural language users to freely talk about or refer to particular objects in an image @xcite @xcite . common to these methods is the use of images as a proxy for the actual context of language use (i.e., the real world), and the use of bounding boxes drawn onto the image to indicate the target object that is to be named. these two common aspects, essentially simulating real-world linguistic reference, have enabled large-scale data collection leveraging existing computer vision datasets. however, both aspects also introduce potential confounding factors, such as referential uncertainty, where a bounding box may fail to uniquely identify the target object, or visual uncertainty, i.e., an object may be more difficult to recognize from a still image than in a real-world scenario. in this work we analyze, after expansion with new meta-annotations, an existing dataset that was created using the method just described, namely manynames @xcite . 1 it builds upon object annotations in visual genome @xcite , and provides up to 36 name annotations for each of 25k objects in images, where the objects belong to 7 different domains (e.g., animals, people, food). to crowdsource the names, @xcite subjects with an image and asked them to freely name the target object that was marked by a bounding box. the resulting dataset appeared to exhibit naming variation, with an average of roughly 3 distinct names per object, but, as mentioned in @xcite , the presence of naming errors in the data prevent one from drawing conclusions about naming behavior, in humans and in computational models. indeed, upon manual inspection we found errors of the kinds we mentioned above: some subjects named a different object from the one highlighted by the bounding box, or intended to name the correct object but used a clearly incorrect name (see figure 1 for examples). our first contribution is to assess the factors that affect the collection of valid naming data in the typical l&v setup. to this end, we collect and analyze verification annotations for manynames via crowdsourcing. our analysis shows a) that the predominant type of error in manynames is a referential mistake (where subjects name an object other than the target); b) that naming variation in manynames is still substantial after correcting for errors and uncertainty, with objects having an average of 2.2 distinct names; and c) that there are differences between domains in the confounding factors, and in the degree of genuine naming variation. we publish a verified version of manynames (called manynames v2) that we derived through our analysis, to serve as a reliable, verified object naming dataset for future work. 1 as our second contribution, we assess if the confounders identified by means of our analysis impact the accuracy of a l&v object labeling model. we introduce a diagnostic evaluation based on manynames v2. it moves beyond the single-label setup that is common in computer vision and l&v while avoiding the aforementioned confounds. that is, the verification annotations of manynames v2 enable us to establish whether a name predicted by the model coincides with the most frequent human response, a less frequent but still valid name, or a mistake (and, in the latter case, of which kind). to the best of our knowledge, to date there exists no work in l&v that performs a systematic investigation of the kind we do here for object naming. we showcase the potential of this evaluation method by analyzing the performance of a popular l&v model, bottom-up @xcite . we show a) that a single-label evaluation greatly underestimates the naming capabilities of bottom-up, which actually come close to our estimated human upper bound; and b) that, however, its effectiveness varies across domains. we furthermore find c) that the model is confused by the same aspects that confuse humans (as categorized in manynames v2), but not always to the same degree, which suggests that the gap between humans and models regarding actual human object naming is still larger than the overall accuracy on its own would suggest. ## related work object naming objects are members of many categories and can be called by many names (e.g., a duck can be called duck, bird, animal, etc.). the task of object naming-generating a formally correct and also appropriate, natural name for an object-has been studied in psycholinguisticsand l&v research, and is related to object recognition tasks in computer vision. we briefly discuss each area. psycholinguistic studies have traditionally focused on object categories instead of individual objects, e.g., the category duck as opposed to a particular duck, and have typically used prototypical or schematic depictions to represent a category (e.g., @xcite . such studies have found that humans, when naturally naming an object, have a preference towards a particular name, defined as the entry-level name @xcite @xcite . in contrast, work in l&v is mostly concerned with naming particular object instances situated in naturalistic images. in such a setting, naming preferences are more nuanced: humans may prefer different names for instances of the same class (e.g., fig. 1a-b ), and even disagree in their choice for the same instance @xcite . while @xcite modeled object naming in a controlled setting, @xcite and @xcite used imagenet data @xcite , where images show more realistic, yet still isolated objects annotated with wordnet synsets @xcite . @xcite train a classification-based model for names produced in referring expressions in the refcoco data @xcite , where objects are situated in complex scenes and names might be affected by context. in contrast to our work, existing approaches did not have access to name annotations from many different annotators ## free object naming: verification collection and analysis of manynames v2 3.1 obtaining consistent response sets from manynames v1 silberer et al. (2020) created the dataset of object names, manynames, by collecting on amazon mechanical turk (amt) approximately 36 names each for 25k target objects o. each object o j was presented visually by a picture containing a single bounding box b j to delineate the target object, see figure 1 for examples. the manynames annotations are structured as follows: for a box b i that delineates an object o i in image @xmath0, we have a response set r i := {(n 1 , p 1 ), ..., (n k , p k )} of k name-frequency pairs. unless otherwise stated, by name-object pairs we refer to name types given an individual object (i.e., there are k names). for each object, there are m name tokens, with m := k @xmath1. for each name n j in the response set, p j is the relative frequency of that name in the response set: ## diagnosing model effectiveness in human-like object naming in section 3.4 we obtained manynames v2, which provides a reliable categorization of object names: the top name, alternative names for the same object (i.e., in the consistent response set), adequate names for other objects (i.e., outside the consistent response set), and inadequate names of various types. we now use it to define a diagnostic evaluation method for object naming models, one which is more fine-grained than the predominant single-label evaluation. it also allows to assess whether models are affected by the same issues as humans, in particular referential and visual uncertainty. we apply our evaluation to bottom-up @xcite as a representative l&v object naming model, which has been widely used for transfer learning in l&v @xcite @xcite @xcite . in contrast to existing works in computer vision research @xcite on diagnosing the effects of object or image characteristics on model performance, manynames allows us to compare the model against an upper bound of the human performance in object naming, estimated via the verification annotations. our analysis focuses on two questions: first, can an object detector that was trained in a single-label setting (i.e., towards predicting unique ground truth names) account for the naming variation inherent in human object naming behavior? second, does the model exhibit a similar sensitivity as humans to the interaction between domains (humans, clothes, etc.) as well as the visual characteristics of target objects? ## conclusions modeling how humans use language in the visual world is at the core of l&v research. we have focused on object naming, the choice of a noun (or compound noun) to refer to an object which is marked with a bounding box in a real-world image. this setup is typical of computer vision and l&v for tasks such as object classification, referring expression interpretation/generation or visual dialogue. our findings underline the importance of modeling naming as a phenomenon of its own: a woman, for example, can be named skier, person, or woman, in different images or by different people. at the same time, there are clear preferences about how to name a particular object: overall naming agreement in humans is around 80%. to boost research on object naming, we provide a high-quality naming resource, manynames v2, which is a verified version of a previous dataset with 36 names for objects in 25k images. for dataset collection, our analysis of the verification data strongly supports a collection methodology that elicits names from many speakers, in order to capture the variation in possible naming choices, and to reliably estimate the preferred name of an object. it furthermore suggests that referential issues, the main cause of noise introduced through the collection setup, can be greatly reduced by a simple verification step that assumes that the object named by the most frequent response is the target object, and asks subjects to select the alternative names for this object. for model development and evaluation, both naming variation and its domain dependence need to be taken into account. we have shown that manynames provides a very different picture of the performance of a state-of-the-art naming model, compared to a resource that only provides one single gold name per object (visual genome). our analysis also shows that the model's naming behavior differs from that of humans particularly in the people, clothing and food domains, that is, in domains that are very familiar and relevant in human daily life, and in which we have found that humans exhibit the highest language variation. this is based on a model that provides a single answer per object; future work should seek to do even more justice to language variation, by predicting the whole probability distribution of names for objects @xcite . we hope that our work will spur further research on object naming and, more generally, how humans use language to talk about the world.