Beyond Embeddings: The Promise of Visual Table in Visual Reasoning (2024)

Abstract: Visual representation learning has been a cornerstone in computer vision, involving typical forms such as visual embeddings, structural symbols, and text-based representations. Despite the success of CLIP-type visual embeddings, they often lack access to world knowledge critical for visual reasoning. In this work, we propose Visual Table, a novel form of visual representation tailored for visual reasoning. Visual tables are constructed as hierarchical descriptions of visual scenes, featuring a scene description and multiple object-centric descriptions covering categories, attributes, and knowledge. Thanks to the structural and textual formats, visual tables offer unique properties over mere visual embeddings, such as explainability and controllable editing. Furthermore, they deliver instance-level world knowledge and detailed attributes that are essential for visual reasoning. To create visual tables, we develop a generator trained on the dataset with collected, small-scale annotations. Extensive results on 11 visual reasoning benchmarks demonstrate that the generated visual tables significantly outperform previous structural and text-based representations. Moreover, they consistently enhance state-of-the-art multi-modal large language models across diverse benchmarks, showcasing their potential for advancing visual reasoning tasks. Our code is available at https://github.com/LaVi-Lab/Visual-Table.

## introduction visual representation learning has been a fundamental and long-standing topic in computer vision. at the early stage, the learning was supervised by expensive human-annotated labels @xcite @xcite . this paradigm recently evolved to learn visual embeddings by aligning image-text pairs from the internet @xcite . beyond visual embeddings, symbolic and structured visual representations (e.g., scene graph) @xcite exhibited advantages across domains, such as vision-language tasks @xcite @xcite , video and 3d scene understanding @xcite @xcite , and robotics @xcite @xcite . more recently, some works have strived to convert visual scenes into text-based representations (e.g., image captions) @xcite @xcite , triggering the reasoning capability of large language models (llms) @xcite @xcite . among these visual representations, clip-type visual embeddings @xcite , learned from image-text pairs, have dominated many vision tasks. their success can be attributed to robust generalization in encoding visual attributes (e.g., visual appearance @xcite @xcite , visual relations @xcite @xcite ). however, lacking external world knowledge (e.g., object affordance, background knowledge of named entities), these visual embeddings lead to sub-optimal performance in complex reasoning tasks. consider the example at the left top of fig. 1 . while clip embeddings can effectively capture visual attributes (e.g., recognizing a person with a beard), they still struggle to answer questions that necessitate critical world knowledge beyond the image itself @xcite @xcite . to this end, we propose visual table, a novel form of visual representation designed for visual reasoning. as shown in the left-bottom of fig. 1 , a visual table is presented as hierarchical text descriptions for visual scenes, and comprises a scene de-knowledge from our visual table : currency: used as a medium of exchange, legal tender in the united states, a historical figure important to the nation's history portrait: represents abraham lincoln, the 16th president of the united states, known for leading the country through the civil war and issuing the emancipation proclamation questions: knowledge from llms or knowledge bases: currency: a system of money used in a particular country or region for transactions, serving as a medium of exchange portrait: typically a painting, photograph, or other artistic medium, that depicts the likeness of a person or group, often focusing on their facial features and expressions ## visual table we introduce visual table, a new form of visual representation, constructed in hierarchical text. given an image i, a visual table v @xmath0. v t consists of a scene description and multiple object-centric descriptions that encompass categories, attributes, and knowledge. thanks to structural and textual formats, visual tables support interpretability to humans and llms, and enable controllable table editing. more-visual encoder (clip) connector (mlp) large language model (vicuna) "based on the given image, generate the visual table that follows the following json format: ……" input image input instruction tokenization visual table generator scene description: this is an image of a us five-dollar bill featuring the portrait of abraham lincoln. category: currency; attribute: paper material, features a portrait, textual inscriptions, security threads, and symbols; knowledge: used as a medium of exchange, legal tender in the united states, and bears the likeness of a historical figure important to the nation's history. … category: portrait; attribute: depicts a male with a beard, wearing a black suit and a bow tie; knowledge: represents abraham lincoln, the 16th president of the united states, known for leading the country through the civil war and issuing the emancipation proclamation. multi-modal large language model with generated visual tables visual table annotations designed prompt input image humans / foundation models llm input image visual embeddings (optional) generated visual table input instruction text response object list: train figure 2: an overview of learning a visual table generator and its application on mllms. left: we design a prompt to collect visual table annotations on a small scale of images. these annotations are used to train our visual table generator which consists of a frozen visual encoder, a vision-language connector, and a pre-trained llm. right: our generator is employed to generate visual tables for the images in downstream tasks. with the generated visual table either as standalone or as additional visual representations, llm performs reasoning to output text response. fig. 2 provides an overview of visual table generation and its application on image reasoning using mllms. we first collect a small number of visual table annotations (sec. 3.1) and then train a visual table generator g (sec. 3.2). the trained generator is able to automatically produce visual tables for any input images in downstream tasks. we evaluate the efficacy of our generated visual tables on diverse benchmarks where visual tables serve as visual representations (sec. 3.3). ## experiments in this section, we first introduce our implementation details, benchmarks, and evaluation protocols, and then present our results, including the comparison with traditional text-based representations, the comparison with sota mllms, the ablation study, the case study, and the human study. implementation details. we provide implementation details for our visual table generator and the mllm with visual tables. for visual table generator, we adopt the same architecture of llava-1.5 @xcite , consisting of clip vit-l/14@336px @xcite as the visual encoder, vicuna-13b @xcite as the llm, and a two-layer mlp as the connector. it is initialized by the llava-1.5-13b model pretrained by excluding vqa data, and further finetuned on 61k visual table annotations for 3 epochs, with batch size as 128, learning rate as 2e-5, and the optimizer as adamw. for the mllm with visual table, we adopt the same training pipeline as llava-1.5, except that during the second training stage, our model is fine-tuned with the generated visual tables as additional visual representations. benchmarks. to evaluate visual tables, we conduct experiments across a diverse set of 11 evaluation benchmarks, providing a comprehensive assessment of visual reasoning capability. our @xcite ; l-7b/13b: llama-7b/13b @xcite ; q-7b/13b: qwen-7b/13b @xcite . bold values refer to the best results within each group. visual table largely outperforms previous text-based representations and is the only text representation that can consistently enhance sota mllms across diverse benchmarks. llava-bench evaluates model capabilities on conversation, detailed description, and complex reasoning tasks that usually require world knowledge to answer accurately. mmmu evaluates models on multi-discipline tasks demanding college-level subject knowledge and deliberate reasoning. mm-bench evaluates the perception and reasoning capabilities. mmvp measures visual understanding capabilities by collecting image pairs that clip perceives as similar despite their clear visual differences. pope assesses the object hallucination problem. vizwiz requires models to answer questions from individuals with visual impairments. scienceqa spans questions from subjects of natural science, language science, and social science. gqa and vqa-v2 are traditional vqa bench-marks that evaluate reasoning ability, while gqa focuses more on visual attributes. textvqa evaluates the ocr reasoning capability. ## conclusion in this paper, we propose visual table -a new visual representation presented in structured text. using the collected dataset with visual table annotations, we learn a generator to produce highquality visual tables for arbitrary input images. beyond visual embeddings, visual tables support interpretability to humans and llms, enable controllable editing, and meanwhile, offer rich descriptions of attributes and knowledge for each object, thereby facilitating visual reasoning. according to extensive experiments, the resulting visual tables exhibit superior performance than previous text-based representations, and demonstrate consistent improvements over the sota mllms across diverse benchmarks. we believe our study has showcased visual tables as robust and generalizable visual representations, laying the groundwork for future research concerning visual reasoning. limitations. when training visual table generator, we exclude the vqa annotations to minimize the biases in vqa tasks. however, our generator might still inherit undesired biases from the training datasets and the collected annotations. besides, visual tables require computation during generation and downstream tasks. a more efficient way to generate and utilize visual tables can be further explored in future research.