V i S truct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation (2023)

Abstract: State-of-the-art vision-language models (VLMs) still have limited performance in structural knowledge extraction, such as relations between objects. In this work, we present ViStruct, a training framework to learn VLMs for effective visual structural knowledge extraction. Two novel designs are incorporated. First, we propose to leverage the inherent structure of programming language to depict visual structural information. This approach enables explicit and consistent representation of visual structural information of multiple granularities, such as concepts, relations, and events, in a well-organized structured format. Second, we introduce curriculum-based learning for VLMs to progressively comprehend visual structures, from fundamental visual concepts to intricate event structures. Our intuition is that lower-level knowledge may contribute to complex visual structure understanding. Furthermore, we compile and release a collection of datasets tailored for visual structural knowledge extraction. We adopt a weakly-supervised approach to directly generate visual event structures from captions for ViStruct training, capitalizing on abundant image-caption pairs from the web. In experiments, we evaluate ViStruct on visual structure prediction tasks, demonstrating its effectiveness in improving the understanding of visual structures. The code will be made public to facilitate future research.

## introduction vision-language models (vlms) exhibit impressive performance across various multimodal downstream tasks @xcite @xcite , such as visual question answering @xcite , image captioning @xcite , visual event extraction @xcite , chart understanding (zhou et al., 2023), and multimedia misinformation detection @xcite . the recipe lies in the pre-training on web-scale data, enabling substantial enhancements in vision-language representation @xcite . however, recent evaluations expose the limitations of existing vlms models in capturing visual structural information @xcite @xcite , rendering them ineffective for visual structural knowledge extraction. for example, @xcite argue that vlms represent images as bag-of-words, disregarding relations among objects. in this work, we present vistruct, a training framework for vlms to improve their visual structural knowledge extraction ability. there are two key challenges in multi-granularity visual structural knowledge learning: (1) capturing the structures at each granularity, for which we propose code-vision representations, and (2) recognizing the dependencies between different granularities, for which we propose a curriculum pyramid to guide the knowledge learning. the motivation of employing code-vision rep-resentations is to mitigate the negative impact of commonly employed techniques such as contrastive learning @xcite and imageconditioned language modeling @xcite , which limit vlms to focus solely on primary objects, resulting in "bag-of-words" representations. we draw inspiration from prior successful application of programming language in representing linguistic structures @xcite . we propose to extend this "code for structure representation" principle to multimodal settings for explicit representation of visual structures. initial attempts to employ code representation in visual reasoning @xcite focus solely on object information and lack perception abilities, but relying on the off-the-shelf object detectors as tools. as a result, we establish a multi-granularity code representation framework to capture structured visual information, including visual concepts, attributes, relations, and events (see figure 1 ). to further capture the interdependencies among various granularities, we propose the curriculum guided visual structural knowledge learning. our intuition is that foundational knowledge (such as concepts) may enhance comprehension of more complex visual structures (such as visual events). thus, unlike previous work that involves pretraining vlms on a mixture of tasks, we build a visual structural knowledge pyramid (see figure 2 ), aligning with the pedagogical strategy of progressively addressing simpler to more complex tasks during training @xcite . however, in contrast to conventional curriculum learning methods that focus on arranging data samples to facilitate training for a specific task @xcite , our objective is to enable vlms to understand visual structures comprehensively at all levels. to accomplish this, we incorporate a replay buffer mechanism that selectively reserves a small amount of data at each level to preserve lower-level knowledge throughout the training process. moreover, we curate an extensive collection of datasets for vistruct, called vistruct suite. this suite includes multi-level structural knowledge, aligned with the curriculum pyramid. we first collect existing labeled datasets and large-scale unlabeled image-caption pairs. then, we adopt a weakly-supervised approach to facilitate the creation of visual event structure labels from captions in a scalable manner. to ensure consistent class label definitions, we automate the alignment of all concepts from various datasets to wordnet @xcite and framenet @xcite synsets, with minimal manual assistance. for downstream task evaluation, we find that vistruct consistently outperforms various baselines on three downstream structure prediction tasks (visual relation detection, scene graph classification, and situation recognition). in addition, we find that code representation and curriculum learning both provide consistent gains on downstream tasks and demonstrate their effectiveness on zero-shot visual structure prediction. parsing visual data into various levels of detail has been a longstanding problem. to the best of our knowledge, we are the first to extract multigranularity visual structured knowledge via codevision representation learning, which effectively captures the hierarchy among different levels of visual knowledge, offering a comprehensive and compositional understanding of overall structure. ## vistruct we propose vistruct that learns a vision-language model that takes images as input and generates a set of structural knowledge, such as concepts, relations, and events involved in the images (see figure 2 ). we first define a code-vision representation (sec. 3.1), which serves as the output to decode visual structural knowledge as code blocks. we then propose a curriculum pyramid to guide the training objectives from low-level towards high-level semantics (sec. 3.2). to train the model, we construct a comprehensive multi-granularity dataset vistruct suite (sec. 3.3), and optimize the model only on information that describe relavant visual strcutural information (sec. 3.4). ## experiment we evaluate vistruct on three structure prediction tasks, including visual relation detection @xcite , scene graph classification @xcite , and situation recognition @xcite . we refer to appendix a for implementation details. ## conclusion we present a visual structural knowledge extraction framework vistruct as a new step towards multi-granularity semantic parsing of visual data. first, we utilize the inherent structured of programming language to represent visual structural knowledge of different levels in a unified way. to capture the interdependencies among various granularities, we then leverage linguistic cues and propose a curriculum learning pyramid that guides the models to effectively acquire visual structural knowledge of each level. results show significant and consistent improvements on structure prediction tasks across all levels.