Which Modality should I use - Text, Motif, or Image? : Understanding Graphs with Large Language Models (2024)

Abstract: Our research integrates graph data with Large Language Models (LLMs), which, despite their advancements in various fields using large text corpora, face limitations in encoding entire graphs due to context size constraints. This paper introduces a new approach to encoding a graph with diverse modalities, such as text, image, and motif, coupled with prompts to approximate a graph’s global connectivity, thereby enhancing LLMs’ efficiency in processing complex graph structures. The study also presents GraphTMI, a novel benchmark for evaluating LLMs in graph structure analysis, focusing on homophily, motif presence, and graph difficulty. Key findings indicate that the image modality, especially with vision-language models like GPT-4V, is superior to text in balancing token limits and preserving essential information and comes close to prior graph neural net (GNN) encoders. Furthermore, the research assesses how various factors affect the performance of each encoding modality and outlines the existing challenges and potential future developments for LLMs in graph understanding and reasoning tasks. Our code and data are publicly available on our project page - https://minnesotanlp.github.io/GraphLLM/

## introduction large language models (llms) are increasingly utilized in areas with inherent graph structures like social network analysis @xcite , drug discovery @xcite , and recommendation systems @xcite , but they face limitations due to their reliance on unstructured text and challenges in incorporating new data post-training @xcite @xcite . graphstructured data can address these issues, providing a nuanced and flexible representation of real-world relationships. while there has been progress in interpreting multi-modal information @xcite , integrating graph understanding into llms remains a developing area. current research typically employs limited setups with small real-world graphs @xcite or synthetic ones @xcite , exposing a gap in effectively incorporating large real-world graphs into llms, owing to their context window constraints. this suggests that text-only encoding may not be optimal for complex, large graph structures. other challenges include llms' difficulty directly processing complex graph-structured data, necessitating innovative input encoding and prompt design @xcite for various graph tasks. designing effective text representations of graphs requires extensive research from the human prac-titioner's perspective, which raises the question of alternative encoding modalities for graphs. this paper investigates the impact of different modalities for encoding global and local graph structures, focusing on node classification tasks, and compares three modalities: text, motif, and image (see figure 1 ). text modality offers detailed local insights but becomes verbose for large graphs @xcite , often exceeding the input limits of models like gpt-4. the motif modality is suggested to address this, capturing essential patterns in a node's vicinity for a balanced localglobal perspective. additionally, image modality is proposed, utilizing fewer tokens to convey a more global view of the node's neighborhood, a method enhanced by the vision capabilities of the newly released @xcite . finding the optimal prompt input format is a notably complex challenge, with text modality encoding requiring extensive exploration compared to the simpler, more human-readable image modality. in our evaluations, we balance informativeness and prompt conciseness across all modalities using a combination of metrics. our main contributions are as follows: some key findings: 1) when balancing the constraint of token limits while preserving crucial information, the image modality is more effective than the text modality for graph-related tasks. 2) the choice of encoding modality for graph task classification depends on the task's difficulty, assessed by homophily and motif counts, with image modality being optimal for medium-difficulty tasks and motif modality for harder ones. 2 setups ## proposed encoders with different modalities graph encoding is crucial for converting graphstructured data into a sequence format that language models can process. as shown in figure 2 , the experimental setup involves using a modality encoder to input the graph structure and a graph query, such as predicting a node's label in node ## graphtmi benchmark creation our study reveals that the ease of node classification in graphs varies across different modalities, depending on the graph's "difficulty," determined by motif count and homophily. homophily @xcite , based on network theory, suggests that nodes are more likely to connect with similar nodes; thus, graphs with higher homophily (more nodes sharing the same label) are simpler to classify than those with more heterophily (diverse labels). this is illustrated through cora dataset examples in figure 4 . graphs are categorized as "easy," "medium," or "hard" based on the diversity of labels. additionally, graphs with more network motifs are considered more complex and challenging for classification @xcite . the "task difficulty" is defined across eight categories (2 3 = 8), with the final difficulty level determined by the higher of the two criteria, homophily or motif count. this led to the creation of graphtmi, a new benchmark dataset that includes various graph structures along with their respective modalities (text, motif, and image), prompts, and llm classifications, thereby providing deeper insights into how different graphs affect llm prompting. specific statistics are given in appendix a. model cora citeseer pubmed gnn baselines gcn 0.7584 ±0.121 0.6102 ±0.087 0.7546 ±0.076 gat 0.7989 ±0.092 0.6583 ±0.074 0.7490 ±0.060 graphsage 0.7719 ±0.124 0.6017 ±0.103 0.7193 ±0.076 llms + encoding modality text 0.81 ±0.04 [0.07 ±0.03] 0.75 ±0.05 [0.07 ±0.01] 0.83 ±0.01 [0.08 ±0.01] * motif 0.73 ±0.06 [0.06 ±0.01] 0.59 ±0.01 [0.32 ±0.02] 0.77 ±0.006 [0.13 ±0.04] image 0.77 ±0.05 [0.04 ±0.02] * 0.71 ±0.09 [0.06 ±0.0] * 0.79 ±0.03 [0.19 ±0.01] ## conclusion and future work this study explores the application of llms in graph-structured data, evaluating their strengths and weaknesses in node classification using various input modalities like motif and image for effective data representation. introducing the graphtmi benchmark highlights the image modality's efficiency in token limit management, and the potential of motif modality in complex graphs. although llms have progressed in graph data processing, they still don't match the performance of gnns in practical settings. the research advocates for future work combining different modalities to improve node classification, combining llm-based methods with gnns, applying these techniques to complex, text-dense graphs, and delving into link prediction and community detection to broaden applications and insights across multiple domains.