Linking W ord N et to 3 D Shapes (2018)

Abstract: We describe a project to link the Princeton WordNet to 3D representations of real objects and scenes. The goal is to establish a dataset that helps us to understand how people categorize everyday common objects via their parts, attributes, and context. This paper describes the annotation and data collection effort so far as well as ideas for future work.

## introduction the goal of this project is to connect word-net @xcite to 3d representations of real objects and scenes. we believe that this is a natural step towards true grounding of language, which will shed light on how people distinguish, categorize and verbally label real objects based on their parts, attributes, and natural scene contexts. our main motivation is to establish a dataset connecting language with realistic representations of physical objects and scenes using 3d computeraided design (cad) models to enable research in computational understanding of the human cognitive process of categorization. categorization is the process by which we group entities and events together based on salient similarities, such as shared attributes or functions. for example, the category "furniture" includes tables, chairs and beds, all of which are typical parts of a room or house and serve to carry out activities inside or around the house. subcategories are "seating furniture," which includes chairs and sofas and "sleeping furniture," which includes beds, bunkbeds and futons. note that some categories have a simple verbal label (a name, like "furniture"), but often category names are compounds (like "sleeping furniture"). compounding, a universal feature of human language that accounts in part for its infinite generativity, allows us to make up names on the fly whenever we feel the need to distinguish finer-grained categories, such as "college dorm room furniture." of course, not all languages share the same inventory of simple labels. we form and label categories all the time. categories help us to recognize never-before-seen entities by perceiving and assessing their attributes and functions and, on the basis of similarity to known category members, assign them to a category @xcite . young children in particular learn to form categories by being exposed to an increasing number of different category members and gradually learning whether they belong to one category or another. importantly, categories allow us to reason: if we know that beds are made for lying down on and sleeping, encountering a new term like "sleigh bed" will tell us that such a bed is likely to have a flat surface on which a person can lie down. conversely, seeing a sleigh bed for the first time and identifying this salient feature will prompt us to call it a "bed." categorization is so fundamental to human cognition that we are not consciously aware of it; however, it remains a significant challenge for computational systems in tasks such as object recognition and labeling. parts, attributes, and natural contexts of objects are all involved in category formation. objects are made of parts, and parts often imbue functionality, especially in the broad category of "artifacts." thus, seat surfaces are a necessary part for functioning chairs. parts and the functionality they enable are fundamentally intertwined with categorization @xcite . beyond their concrete parts, objects are perceived to have a general set of attributes. for example, the distinction between a cup, a goblet and a mug relies less on the presence or absence of specific parts and more on the geometric differences in aspect ratio of the objects themselves. lastly, real objects occur in real scenes, meaning that they possess natural contexts within which they are observed. in language, context reflects a variety of aspects beyond functionality, including syntactic patterns and distributional properties. in contrast, physical context is concrete and defined by the sets of co-occurring objects and their relative arrangements in a given scene. to study these three aspects of category formation, we will connect wordnet at the part, attribute, and contextual level with a 3d shape dataset. the longer term goal of this project is to ask how people distinguish and categorize objects based on how they name and describe the parts and attributes of the objects. we will focus primarily on the category "furniture." ## existing datasets there has been much prior work linking word-net @xcite to 2d images. the most prominent effort in this direction is ima-genet @xcite , which structures a large dataset of object images in accordance with word-net hierarchies. our project differs from ima-genet because even though imagenet is based on the wordnet hierarchy, the focus of our project is on annotating parts and attributes on 3d models rather than the labeling of images. following this, the sun @xcite dataset focuses on scene images, and the visualgenome @xcite dataset defines object relations and attributes in a connected scene graph representation within each image. another line of work focuses on detailed annotation of objects and their parts -a prominent recent example is the ade20k dataset @xcite . however, a fundamental assumption of all this work is that objects and their properties can be adequately represented in the 2d image plane. this assumption does not generally hold, as many object parts, spatial relations between objects and a full view of object context are hard to infer from the limited field of view of a 2d image. more recently, there has been some work that links 3d cad models to wordnet. the shapenet @xcite dataset is a large collection of cad representations (curating close to 65,000 objects in approximately 200 common wordnet synsets), whereas the suncg @xcite dataset contains cad representations of 45,000 houses, composed of individual objects (in 159 wordnet synsets). these two datasets are both "synthetic" in the sense that the 3d cad representations are designed virtually by a human expert. a different form of 3d representation is obtained by scanning and 3d reconstruction of real world spaces. recent work introduced scannet @xcite and the matter-port3d @xcite dataset, which both contain 3d reconstructions of various public and private interior spaces (containing 409 and 430 object synsets respectively). though both synthetic and reconstructed 3d data are increasingly available, no effort currently exists to connect such 3d representations to word-net at the part, attribute, and contextual levels. such a link of wordnet entries to 3d data can provide much richer information than 2d image datasets. naturally, 3d representations allow us to reason about unoccluded parts and symmetries, arrangement of objects in a physically realistic three dimensional space, and to account for empty space, a critical property of real scenes which is not observable in 2d images. moreover, 3d representations are appropriate for computationally simulating real spaces and the actions that can be performed within them. the ability to do this is a powerful tool for investigating and understanding actions @xcite . therefore, our project aims to annotate 3d models in one of the existing datasets and link them to the appropriate synset in the wordnet database at the part, attribute, and contextual level. ## project description our project has so far focused on annotating part and attribute information on 3d cad objects in suncg @xcite and linking them to the corresponding wordnet synsets. we are working with a preliminary categorization of the objects performed in prior work, which establishes their connection to wordnet synsets denoting physical objects. however, we plan to refine the granularity of this categorization by introducing finer-grained categories -e.g. partitioning "doors" into "garage doors" and "screen doors" among others. we chose this dataset because the 3d objects in suncg (approximately 2,500) are used across a large number of 3d scenes (more than 45,000). this means that for each object, we can automatically establish many contextual observations. this property of the suncg dataset differs from other common 3d cad model research datasets such as shapenet @xcite where each object is de-contextualized. for the latter dataset we would have to additionally compose scenes using available objects in order to acquire observation contexts for the objects. we first augment the suncg objects with part annotations that are linked to wordnet synsets. we defer the assignment of attributes to the same objects as it is an easier annotation task in terms of interface design. to perform the part annotation, we designed an interface with a "paint and name" interaction where the user paints parts of the surface of an object corresponding to a distinct part and assigns a name to that part. the details of the interface and annotation task are described in the following section. ## annotation interface our interface is designed to allow for efficient annotation of parts by inexperienced workers on crowdsourcing platforms such as amazon's mechanical turk. the interface is implemented in javascript using three.js (a webgl-based graphics library). it can be accessed on the web using any modern browser, and does not require specialized software or hardware. figure 1 shows a series of screenshots from the interface illustrating the annotation process. the user first sees a rotating "turn table" view that reveals the appearance of the object from all directions. the user then types the name of a part in a text panel and drags over the surface of the object to select the regions corresponding to the part. the process is repeated for each part and the final result is a multi-colored painting of the object with corresponding part names for each color. the partitioning of the object's geometry into different parts is saved and submitted to a data server upon completion. figure 2 shows a close-up of the interface while an object is in the process of being annotated. in order to enable an efficient multi-level painting interaction, the size of the paint brush can be adjusted by the annotator. the object geometry is pre-segmented for several levels of granularity: segmentation into surfaces with the same material assignment, a segmentation with a loose surface normal distance criterion, and finally a segmentation into sets of topologically connected components of the object geometry. this multi-level painting allows the speed of labeling to be adjusted to accommodate both small parts (e.g., door handles) and large parts (e.g., countertops on kitchen islands). the annotators use freeform text for the part names, requiring that we address the problem of mapping this part name text to a wordnet synset. we implemented a simple algorithm that restricts the candidate synset set to physical objects in the wordnet hierarchy, preferring furniture (since we are dealing with indoor scenes that are predominantly composed of furniture). given the object category, we can additionally use the meronym relations in wordnet to suggest and rerank possible part synsets. after applying this algorithm to connect each part name's text to a wordnet synset, we manually verify and if needed fix the inferred link using an interface that displays the wordnet synset assignment for the given part (in the same view as the part annotation view) and allows the user to select a different synset. figure 4 : list of 3d models to be annotated. this is the interface through which one selects a model to annotate or can examine a previously annotated model. ## initial annotations and statistics five persons have worked on the annotation thus far. annotating one 3d model takes roughly five minutes on average, with time being mostly a function of the complexity of the particular object. to maintain consistency while annotating the objects in the database, the annotators were instructed to name the geometric and functional parts of the object, not decorations or stylistic elements (e.g., a picture of a fish on a lamp shade). using the interface described above, we have so far collected more than 100,000 part instance annotations. an example of a car annotated by a crowd worker is shown in figure 3 . suncg includes a total of 2547 models, of which we have so far annotated 1021 during the prototyping process for developing our interface (figure 4 ). figure 5 shows several other example objects with their part annotations visualized. ## limitations the initial stages of the annotations were limited by two major factors: the quality of the 3d models, and language in general. a handful of the models in the database had segmentation issues, i.e., any error in how the model was broken up into regions. for example, a segmentation issue for a table is encountered when more than one leg is connected into one region, thus making it impossible even with the smallest brush size to label the individual legs. to solve this problem, the segmentation algorithm must be improved upon. a more widespread problem with the database, however, was the lack of internal structure in many of the models. for example, in book cases with doors, only the doors would be present in the model, while the internal structures -in this case the shelves -were omitted. some linguistic factors can cause the annotation to be less than straightforward. for example, should the link be to the british or the u.s. word? if we annotate a coffee cup, should we annotate the piece of cardboard around the cup that is designed to protect our hand from the heat of the coffee using its specific but rare name "zarf," or should we choose a more common but less specific term such as "sleeve" or even "piece of cardboard?" moreover, some parts do not have proper labels: what should we call the beams that help stabilize some tables other than "support beams?" see figure 6 for an example. 7 future work our work so far has focused on developing the infrastructure and annotation interfaces to collect 3d object part annotations at scale. this part data linked to wordnet is of tremendous potential value, which we plan to investigate as our project continues. a very interesting direction of work is in building contextual multimodal embeddings. many object parts and attributes are rarely mentioned in language. for example, a stool doesn't have a back, but people don't refer to stools as "backless chairs." neither do speakers encode the fact that chairs often have four legs or 5 wheels; only non-default exemplars might be labeled in an ad hoc fashion as "five-legged chairs," for example. furthermore, the physical contexts of objects (see figure 7 ) provides richer information than is found in text. in this regard, the text and 3d modalities are complementary and provide an excellent target for building multimodal distributional representations @xcite . multimodal embeddings are a promising semantic representation which has been leveraged for various natural language processing and vision tasks @xcite @xcite . another direction for future work is to leverage the object part and attributes and their correspondences to wordnet to go beyond the set of word-net synsets and automatically induce new senses, along the lines of recent work on sense induction @xcite . for example, we have found that wordnet synsets do not have good coverage of some fairly modern categories of objects that we observe in figure 7 : an example of the same nightstand object (outlined in blue) in two different 3d scene contexts. a contextual embedding afforded by the full 3d representation of the scene within which the nightstand is observed would be a powerful way to analyze and disentangle different usage contexts for common objects.