Advancement in Graph Understanding: A Multimodal Benchmark and Fine-Tuning of Vision-Language Models (2024)

Abstract: Graph data organizes complex relationships and interactions between objects, facilitating advanced analysis and decision-making across different fields. In this paper, we propose a new paradigm for interactive and instructional graph data understanding and reasoning.Instead of adopting complex graph neural models or heuristic graph-to-text instruction design, we leverage Vision-Language Models (VLMs) to encode the graph images with varying structures across different domains. This paper first evaluates the capabilities of public VLMs in graph learning from multiple aspects. Then it introduces a novel instruction-following dataset for multimodal graph understanding and reasoning in English and Chinese. Besides, by fine-tuning MiniGPT-4 and LLaVA on our dataset, we achieved an accuracy increase of 5%-15% compared to baseline models, with the best-performing model attaining scores comparable to Gemini in GPT-asissted Evaluation. This research not only showcases the potential of integrating VLMs with graph data but also opens new avenues for advancements in graph data understanding.

## introduction graph is an important form of structured data, which is capable of storing and representing the complex relationships between objects. downstream tasks performed on graph data include node classification @xcite , link prediction @xcite , graph reasoning @xcite etc. in early days, graph learning is commonly modelled using graph networks, with a wealth of seminal literature supporting this approach. in recent years, graph neural networks @xcite @xcite @xcite ) provide a more flexible and effective means of dealing with * equal contribution † corresponding author(jianghy66@mail.sysu.edu.cn) the diversity and complexity in graph structures. by propagating and aggregating node information within the graph structure, gnns capture complex graph features. with the rise of large language models (llms), there has been extensive research in the field of natural language-guided interactive graph data understanding, showcasing two main strategies. the first approach constructs prompts by explicitly representing graph structures in a sequential format. constructing prompts for llms involves innovative techniques such as self-prompting @xcite , graph-syntax trees @xcite , natural language descriptions of graphs' structures and features @xcite , and graph-structure prompting in various modalities @xcite . the second strategy transforms gnn-learned features into llm-comprehensible tokens. models like git-former @xcite gimlet @xcite and molca @xcite consider graph as a mode and integrate all modality data into a unified latent space. however, due to the inherent differences between graph structures and language models, current integration methods face significant limitations. first, sequence-based prompt representations inevitably lose crucial graph structural information, which is vital for complex graph understanding and reasoning @xcite . second, integrating gnn-learned features into llms presents alignment challenges, as the representation learning space of gnns cannot be easily mapped onto the token space of llms through simple function mappings @xcite . considering these limitations, this work introduces a new paradigm for graph data understanding based on vlms. our basic idea is to transform graph data into images, either through their natural representations like maps or visualization methods † . this is followed by utilizing an image encoder to interpret the visual information, thereby understanding the semantic information of nodes and relationships within the graph data. the final step of this process involves the fusion of image and text encoders, which is inherently supported in specific vlms @xcite @xcite , offering a novel approach to graph data comprehension. figure 1 shows an example of graph data access based on visual language models. utilizing visual language models for graph data comprehension offers friendly and natural interaction, reducing manual labor costs, and unifies diverse graph structures globally, handling various graph-related tasks through natural language instructions. leveraging state-of-the-art visual language models trained on extensive datasets facilitates knowledge transfer and promises high performance potential in this paradigm of graph understanding and reasoning. this work introduces a novel dataset on various graph formats for multimodal graph understanding and reasoning in english and chinese. instructions involve simple queries and complex multi-hop reasoning on the graphs, with difficulty levels varying from simple to medium and difficult. additionally, based on this dataset, we conducted a comprehensive evaluation of current open models from various perspectives. in conclusion, the current models exhibit significantly better performance in english compared to chinese. even the most powerful models like genimi or gpt-4v achieve accuracy rates below 30%. finally, in fine-tuning llava and minigpt-4 on our dataset, experimental results revealed that the fine-tuned llava model on english&chinese datasets outperformed base- † https://www.ownthink.com/ line llava on both english and chinese dataset and achieved comparable performance with genimi in chinese. to enhance the open-source models' ability in recognizing chinese characters, we also proposed incorporating ocr data transformed into instruction data into our chinese datasets. the model refined through this approach exhibited significant improvements on the chinese dataset after fine-tuning. our main contributions are as follows: ## dataset we constructed two multimodal instructionfollowing datasets on graphs in english and chinese respectively. each dataset includes five types of graphs: knowledge graph, route map, mind map, flowchart, and gantt chart. the images were crawled from search engines and filtered for relevance and accuracy by humans. the construction of multimodal instructions involves two steps. the first step prompts gpt-4v to generate candidate instructions and responses for each image. the second step involves human evaluation and annotation to ensure the validation of instructions and accuracy of responses. for images containing mixed language content, such as english text in chinese images or vice versa, we have already discarded these during the manual filtering process to ensure language consistency in our dataset. ## conclusion and future work this study focuses on natural language-guided interactive graph data understanding, distinguishing itself from traditional graph network modeling by utilizing vlms to enable flexible interactions with graph data. we constructed a benchmark dataset to address the lack of available data in this domain, further supporting the development and evaluation of models in this field. by fine-tuning llava and minigpt-4 with our dataset, we observed notable improvements in performance on both chinese and english datasets, marking a significant advancement in the field of graph data understanding. in the future, we will advance this work from the following two directions. (1) designing dedicated im-age encoders to capture the pixels related to nodes and edges more sensitively. (2) supporting large graph (with thousands of nodes or edges) understanding by splitting a big image into a sequence of sub-images. it requires that the vlms have the ability to understand the internal content in a single image as well as the contents across different sub-images, for a more accurate understanding of the original large graph content.