Paparazzi: A Deep Dive into the Capabilities of Language and Vision Models for Grounding Viewpoint Descriptions (2023)

Abstract: Existing language and vision models achieve impressive performance in image-text understanding. Yet, it is an open question to what extent they can be used for language understanding in 3D environments and whether they implicitly acquire 3D object knowledge, e.g. about different views of an object. In this paper, we investigate whether a state-of-the-art language and vision model, CLIP, is able to ground perspective descriptions of a 3D object and identify canonical views of common objects based on text queries. We present an evaluation framework that uses a circling camera around a 3D object to generate images from different viewpoints and evaluate them in terms of their similarity to natural language descriptions. We find that a pre-trained CLIP model performs poorly on most canonical views and that fine-tuning using hard negative sampling and random contrasting yields good results even under conditions with little available training data.

## introduction recent advancements in pre-training large-scale language and vision (l&v) models, such as clip @xcite , have led to exceptional performance on benchmarks and leaderboards in 2d image-text retrieval @xcite @xcite . however, the image-text data in these benchmarks have specific properties and biases @xcite that may limit the language grounding capabilities of existing l&v models and their robustness in realword scenarios @xcite . a fundamental bias in existing l&v data comes from the fact that images generally show single, human-centric views of different objects. this raises a simple but intriguing question: to what extent can a model acquire knowledge about the concept of viewpoints and identify different views on the same object? figure 1 illustrates this challenge, showing the top-3 images retrieved by clip for two basic viewpoint descriptions, car/airplane from the bottom, in the laion-5b @xcite data set: the airplane images mostly correspond to the correct view, but none of the car images shows a bottom view. it suggests that the model does not generalize the meaning of viewpoint descriptions across different objects, foot_0 and may fail to acquire visual-linguistic knowledge that would be needed in more realistic 3d scenarios, such as when instructing a drone to take a picture of an object from a specific viewpoint @xcite . this opens the door for a systematic examination of the capabilities of l&v models for grounding viewpoint descriptions, delving into the question of why, despite their excellent zero-shot capabilities, a model like clip struggles when it comes to representing perspectives of the same object. in this paper, we investigate whether language understanding in pre-trained l&v models generalizes to simple text-viewpoint descriptions of common objects. we propose a new task -textviewpoint retrieval -and a framework for analyzing and scaling image-text models with 3d data. we implement a paparazzi agent that circles a spherical camera around a 3d object, samples images, and scores pairs of image-viewpoint descriptions using a pre-trained image-text matching model. in this framework, we evaluate and analyze whether clip, as a representative image-textmatching model with excellent zero-shot capabilities, systematically retrieves images of views of 3d shapes, regardless of potential reporting biases in 2d l&v data sets. to successfully interpret viewpoint descriptions like car from the bottom, models need to connect concepts in natural language to visual representations and basic knowledge of object geometry. to investigate this, our approach is deliberately simple: we use 3d shapes from five categories of common objects in shapenet that have visually distinct canonical views @xcite . based on @xcite , that divide a sphere into hexagonal shapes, we analyze whether clip provides an adequate embedding for the viewpoint space around an object. our analysis suggests that basic viewpoint understanding is indeed a systematic gap in the pre-trained clip model, as it achieves very poor performance in scoring view-description pairs and even retrieves nonsensical, non-human-centric views. furthermore, we find that this problem is not fixed by standard fine-tuning. thus, we propose a procedure for fine-tuning clip that extends the contrastive learning approach to viewpoints and descriptions generated from 3d visualizations. we find that a small amount of training data and extended fine-tuning is successful in scaling clip to basic viewpoint understanding in 3d. language grounding in 3d. @xcite present pioneering work in this area, with a referring expression data set designed for learning the language of shape for chair objects in shapenet, the most well-known resource for 3d object models @xcite . they build a neural resolution model that predicts which chair is referred to by a given shape description. their encoder combines an autoencoder for point clouds of 3d shapes and a pre-trained image encoder for a single view of the object. as @xcite collected descriptions of the 3d objects in a static environment with a fixed camera perspective, their approach does not account for dynamic viewpoints in 3d. @xcite present a larger data set for expressions referring to shapenet objects and build a model that relies on image-text matching via the clip architecture, similar to ours. their model takes images of eight fixed viewpoints of the object as input and integrates a component that estimates the viewing angle of an image. they evaluate on resolution accuracy and do not explicitly test viewpoint understanding in the clip model. in contrast to these existing works, the input to our model does not specify a fixed set of camera positions, and the output is an explicit, specific viewpoint of an object represented as an image. ## text-viewpoint retrieval task we study viewpoint understanding from descriptions and describe a framework for text-viewpoint retrieval. we present a task definition, the set-up of the 3d environment and the camera, and our approach to evaluation and analysis. ## analysis this section takes a closer look at how well the textviewpoint embeddings capture understanding of different viewpoints. specifically, we will explore whether the scoring functions correctly identify viewpoints that align with the linguistic description, while providing lower scores for those that do not. ## conclusion we developed a new framework to assess the capabilities of l&v models to ground viewpoint descriptions. through our research, we discovered that a standard clip model struggles to distinguish between different viewpoints. to address this, we explored a combination of different loss objectives on synthetic data to make it easier to retrieve viewpoints from language descriptions. our experiments revealed that incorporating random contrasting leads to a more accurate and seamless scoring function, as compared to using only text and human-centric images. our framework thus offers a promising approach to scale l&v models trained on large-scale image-text datasets for applications that involve interaction in the 3d world.