OLIVE : Object Level In-Context Visual Embeddings (2024)

Abstract: Recent generalist vision-language models (VLMs) have demonstrated impressive reasoning capabilities across diverse multimodal tasks. However, these models still struggle with fine-grained object-level understanding and grounding. In terms of modeling, existing VLMs implicitly align text tokens with image patch tokens, which is ineffective for embedding alignment at the same granularity and inevitably introduces noisy spurious background features. Additionally, these models struggle when generalizing to unseen visual concepts and may not be reliable for domain-specific tasks without further fine-tuning. To address these limitations, we propose a novel method to prompt large language models with in-context visual object vectors, thereby enabling controllable object-level reasoning. This eliminates the necessity of fusing a lengthy array of image patch features and significantly speeds up training. Furthermore, we propose region-level retrieval using our object representations, facilitating rapid adaptation to new objects without additional training. Our experiments reveal that our method achieves competitive referring object classification and captioning performance, while also offering zero-shot generalization and robustness to visually challenging contexts.

## introduction despite the popularity, many existing vlms such as llava @xcite , minigpt4 @xcite , and mplug-owl @xcite handle the entire image for visual understanding, leading to two major shortcomings. first, these vlms use a visual transformer to split an image into a grid of image patches and embed them into a lengthy array of image patch embeddings that have object level features scattered around different positions of the array. this leads to the different granularity between the image patch tokens and text tokens, further creating difficulty in aligning and grounding visual objects to text concepts. second, feeding all image patch embeddings to the large language model (llm) decoder is problematic due to the resulting long context and inefficiency of including in-context examples from multiple images. to improve fine-grained visual alignment, recent region-based vlms are pre-trained to integrate object level information into the llm decoder. gpt4roi @xcite pre-trains llms to understand roialign features @xcite extracted from bounding boxes. other similar methods such as shikra @xcite or kosmos-2 @xcite ground and refer to objects using text in multimodal referential dialogues. ferret @xcite and vip-llava @xcite further support free-form shapes as referring input by summarizing visual features sampled within the region of interest. although these methods provide improvement to object level reasoning, they still fail at recognizing unseen/rare objects and are sensitive to spurious background features, as shown in §5. even powerful closed-source multimodal models such as gpt4v are unreliable to deploy in high-stakes domain-specific situations such as the medical domain @xcite . a straightforward way to handle generalization to unseen visual content is to integrate a retrieval component. methods such as reveal @xcite and murag @xcite provide retrieved multimodal facts as supplementary context to help vlms generalize to new concepts without further training. however, these models do not consider object level retrieval and in-context prediction. models such as flamingo @xcite and qwen-vl @xcite allow for in-context examples from multiple images, yet do not support object level retrieval and reasoning. to address the above issues, we propose to encode object level in-context visual embeddings (olive) to enhance llms with region-level reasoning capabilities. critically, we omit lengthy image patch features and encode visual object embeddings by a lightweight encoder of 20 million parameters, allowing for faster training and direct connection to existing llms. this preserves the full functionality of the original llms, while also introducing novel multimodal reasoning abilities. furthermore, our object level retrieval module allows for more precise queries and retrieved information to help the model adapt to domain-specific tasks with limited training data. our contributions are summarized below and in ## preliminaries generative vlm architecture recent generative vlms (e.g., llava, blip-2) adopt a similar architecture that connects a pre-trained visual encoder ϕ v and a pre-trained language model decoder ϕ t through a lightweight fusion neural network, denoted as ϕ c . specifically, the fusion module first uses a projection function to map a visual feature @xmath0, and then fuse the visual and text embeddings as input to language model decoder. formally, given an image v and a text prompt x, the decoder takes in the combined feature x to autoregressively predict the output y. different from prior fusion modules (e.g., linear projection in llava, gated cross-attention in flamingo, and q-former in blip-2) that project the whole image features, we propose an object level encoder ( §3.1) that captures fine-grained region features and speeds up training and inference. ## method this section as well as figure 1 highlights the main components of our method. we first design an object encoder ( §3.1) to learn visual object embeddings in a shared vision-text space, then apply a similarity search over object embeddings to retrieve relevant visual objects ( §3.2), and finally construct a code-switch multimodal prompt to integrate the retrieved object information for generation ( §3.3). ## experimental settings in this section, we first describe two main objectlevel tasks for evaluation ( §4.1) together with the datasets used ( §4.2). finally, we describe three variants of our model ( §4.3), the training details §4.4), and the other baselines in comparison ( §4.5). ## related work grounding in language and vision a popular approach for aligning vision and language embeddings is contrastive learning methods such as clip and align @xcite . however, these methods align the entire image representation, leading to poor reasoning on image details for downstream vision language tasks. region-clip @xcite and glip @xcite address this issue by proposing fine-grained alignment with region-text pairs during pretraining. glipv2 @xcite further improves the pretraining and alignment by introducing localization, detection, and other tasks. another recent popular approach involves training models on automatically curated region-level data from image-caption pairs @xcite . many other works focus on region-level alignment during pretraining for greater vision-language understanding @xcite @xcite . more generally, a recent study @xcite shows that vlms with fine-grained object-level pretraining such as x-vlm @xcite have better reasoning ability. other works align vision and language using regularization or loss to create relation aware cross attention between modalities @xcite . visual resampling visual resampling is a popular technique to compress long sequences of image features into a few rich vector representations. this is achieved by constructing a fixed amount of learnable vectors that attend to the visual features through cross-attention layers. models such as blip @xcite first explore this idea to connect frozen vision features to llms efficiently by summarizing the content of the image. other methods including x-decoder @xcite or seem @xcite use resampling to encode various types of prompts or intents which improve the llm decoding ability. additionally, works such as flamingo @xcite and qwen-vl @xcite show that multiple images can be inserted in-context to the prompt by compressing image features with resamplers, enabling few shot capabilities. our work visually resamples object representations for object-conditioned text generation, and only uses a single vector for the representation. this allows for more fine-grained reasoning and longer in-context prompting. ## conclusion we present a simple approach to insert object level visual embeddings into large language model decoders, enabling object level reasoning with flexible prompt structure. our object encoder compresses fine-grained region level information into a single vector, enabling in-context prompting with objects from multiple images and more efficient training and inference. in addition, we introduce the idea of region retrieval, which allows for precise queries free of image background noise and rapid generalization to rare and unseen objects with no parameter updates. we hope our method may help researchers design vision language models which can adapt to their needs by simply updating the retrieval set or object encoder, while also being responsive to varying user intents using llm prompting techniques.