UOUO : Uncontextualized Uncommon Objects for Measuring Knowledge Horizons of Vision Language Models (2024)

Abstract: Smaller-scale Vision-Language Models (VLMs) often claim to perform on par with larger models in general-domain visual grounding and question-answering benchmarks while offering advantages in computational efficiency and storage. However, their ability to handle rare objects, which fall into the long tail of data distributions, is less understood. To rigorously evaluate this aspect, we introduce the “Uncontextualized Uncommon Objects” (UOUO) benchmark. This benchmark focuses on systematically testing VLMs with both large and small parameter counts on rare and specialized objects. Our comprehensive analysis reveals that while smaller VLMs maintain competitive performance on common datasets, they significantly underperform on tasks involving uncommon objects. We also propose an advanced, scalable pipeline for data collection and cleaning, ensuring the UOUO benchmark provides high-quality, challenging instances. These findings highlight the need to consider long-tail distributions when assessing the true capabilities of VLMs. Code and project details for UOUO can be found at https://zoezheng126.github.io/UOUO-Website/.

## introduction the advent of vision-language models (vlms) has marked a revolutionary leap in the integration of natural language processing and computer vision, largely due to the capabilities of the selfattention mechanism and the transformer architecture @xcite . these technologies allow vlms to effectively process and fuse information from both text and images, leading to significant advancements in tasks that require multimodal understanding, such as visual question answering and image captioning @xcite @xcite @xcite . vlms, trained on large-scale datasets, typically boast high performance on general tasks involving everyday objects and common scenarios @xcite @xcite . however, models of smaller scale, defined here as having fewer than 70 billion parameters, often claim to match the capabilities of their larger counterparts on general domain tasks @xcite @xcite @xcite while offering advantages in computational efficiency and storage. despite these claims, the no-free-lunch theorem @xcite suggests that these smaller models may compromise on their ability to handle less common or more complex scenarios that lie in the long tail of data distributions. one natural and intuitive hypothesis is that they are sacrificing their fitness to the elements on the long tail of the distribution. empirical observations of real-world data frequently align with zipf's and power law @xcite , which indicates that while some objects and concepts are exceedingly common, a vast number of them are rare and fall into the long tail of the distribution . understanding how well vlms handle these rare and uncommon instances is crucial for assessing their true robustness and applicability across diverse and nuanced contexts. despite the importance of this evaluation, there is currently a lack of dedicated benchmarks that systematically test vlms on objects and concepts that are significantly outside the everyday norm. to address this gap, we introduce the "uncontextualized uncommon objects" (uouo) benchmark. the object class distribution of uouo is systematically out of common image sources such as im-agenet @xcite , coco @xcite , and open image dataset @xcite . our goal is to rigorously test and quantify the performance of both large-scale and small-scale vlms on elements from the long tail of the distribution to showcase their knowledge gap. the contribution of our work is three-fold. (1) we compile a million-scale dataset specifically designed to include uncommon and uncontextualized objects, which are rarely encountered in everyday contexts but are significant in specialized domains. much to our surprise, it turns out that the image sources of gqa, refcoco, ok-vqa, mme coarse-grained recognition, vqa-v2, and a significant proportion of mmbench are all direct random samples from coco. only nocaps features novel object classes (sourced from the 600categories open image dataset @xcite ) outside coco's less-than-100 common classes. this showcases the significant limitation of categorical diversity of extant vqa datasets. the knowledge and performance gap between the small-and large-scale vlms might be concealed in such low coverage and diversity. existing datasets with uncommon object labels in extant datasets, stanford cars @xcite , cub-bird @xcite , deepfish @xcite , rocov2 @xcite , fgvc-aircraft @xcite also features rare object labels. some non-academic mine & stone datasets, and chemical objects datasets can also be found on internet. however, the typical emphasis of these datasets is either fine-grained subtype or subspecies of common objects, or domain-specific expert knowledge. in realistic use cases such as autonomous car or embodied robotics, such knowledge might have limited generalizability. ## test instances generation background removal and decontextualization connectionist neural networks (including vlms) are notoriously known for their tendency of overfitting to spurious correlations present in the training data. for instance, in our collected data, bulldozers are often seen in construction scenes laden with materials such as sand, concrete, and bricks. this high co-occurrence can lead models to rely on these contextual cues rather than truly understanding and recognizing the bulldozer itself. to mitigate this issue and ensure that models focus on the objects rather than their typical environments, we implement a robust background removal process to decontextualize all candidate objects in our dataset. to achieve effective background removal, we utilize a stateof-the-art, off-the-shelf background removal model (bria-ai, 2024). testing instances generation to assess the performance of vision-language models on our uouo benchmark, we generated challenging test instances designed to probe the models' capabilities beyond common knowledge. specifically, we employ the clip embeddings combined with the maximum mean discrepancy (mmd) with a gaussian rbf kernel @xcite to identify and retrieve hard negative examples. let x and y be the sets of clip embeddings for two different object categories, each of shape (n, d), where n is the number of embeddings and d is the embedding dimension. the maximum mean discrepancy (mmd) between sets of embeddings x and y is calculated as follows: for our calculations, we set σ = 10. we use the mosaic image augmentation technique @xcite to generate testing data in a scalable way. each testing data point is created from four images, each background-removed. the four images contain objects of different categories but share some similar visual properties such as structures, colors, or textures. the selection of these images is determined by the maximum mean discrepancy (mmd) distance between the categories they belong to. the closer the mmd distance, the more similar in features they might appear. we create an 800x800 canvas large enough to accommodate all four images. then, each of the four images is augmented and positioned on the canvas's top-left, top-right, bottom-left, or bottomright. the ground-truth bounding box for the object grounding is generated from the segmentation mask of background removal and normalized to be dimension-insensitive, accounting for potential differences in the vlm's rescaling process. figure 2 showcases an exemplar test instance. vlms that are trained to perform grounding, including: llava-v1.5-7b, llava-v1.5-13b @xcite , cogvlm-v1.5vicuna-7b @xcite , and propriety vlms including: gemini-1.5-pro @xcite , gpt-4turbo, gpt-4o (openai, 2024). we test vlms' performance on both randomly generated test instances and the mmd-augmented hard instances. we employ two metrics to quantitfy the performance: miou -mean iou (intersection over union), a standard metric for object segmentation; and accuracy , which we prompt the vlm to output one positions from "top-left, top-right, bottom-left, bottom-right", and directly evaluate whether the answer matches the ground truth. the prompts used in this experiment can be found in appendix a. ## conclusion in our work, we introduced the uouo benchmark to assess vlms on objects out of everyday distributions. our findings show that while smaller vlms perform well on tasks of common objects, they struggle significantly with uncommon objects, unlike larger models which handle these challenges much better. this highlights the need to consider long-tail distributions in evaluations. the systematic data curation, filtering, and hard test instance generation pipeline for uouo construction has high extensibility, paving the road of future research of long-tail distribution objects. uouo itself could also be expanded in this way, extending beyond the domain of manufacturing and to other broad category of objects.