Visualization Recommendation with Prompt-based Reprogramming of Large Language Models (2024)

Abstract: Visualization recommendations, which aim to automatically match proper visual charts for specific data tables, can significantly simplify the data analysis process. Traditional approaches in this domain have primarily relied on rule-based or machine learning-based methodologies. These methods often demand extensive manual maintenance and yet fail to fully comprehend the tabular data, leading to unsatisfactory performance. Recently, Large Language Models (LLMs) have emerged as powerful tools, exhibiting strong reasoning capabilities. This advancement suggests their substantial promise in addressing visualization recommendation challenges. However, effectively harnessing LLMs to discern and rationalize patterns in tabular data, and consequently deduce the essential information for chart generation, remains an unresolved challenge. To this end, we introduce a novel Hierarchical Table Prompt-based reprogramming framework, named HTP. This framework aims to integrate multi-dimensional tabular data into LLMs through a strategically crafted prompt learning method while keeping the LLMs’ backbone and weights unaltered. The HTP framework uniquely incorporates a four-level prompt structure, encompassing general, instance, cluster, and column levels. This multi-level approach is engineered to provide a comprehensive understanding of both general distribution and multifaceted fine-grained features of tabular data, before inputting the tabular data into the frozen LLM. Our empirical studies confirm that the HTP framework achieves state-of-the-art performance, marking an advancement in the field of data visualization and analysis. The code and data will be made publicly available upon acceptance.

## introduction data visualization, which facilitates effective decision-making via better responding the human * this work was done when the first author was an intern in baidu research under the supervision of jingbo zhou. last year has witnessed the prosperity of large language models (llms), whose advanced capabilities of llm in processing and interpreting 1 complex data unlocks new possibilities in the field of data visualization. however, significant challenges persist in effectively harnessing the potential of an llm for visualization recommendation. firstly, llms are typically designed to process sequences of discrete tokens in an unstructured format, whereas tables used for visualization often display highly structured and predominantly numerical characteristics, which leads to a mismatch. second, the inherent pre-training of llms does not naturally encompass the comprehension abilities required to interpret complex tabular data. third, rather than fine-tuning the entire model, it is often preferable to keep the llm backbone "frozen". this approach ensures the model's versatility in supporting a variety of tasks without compromising its fundamental capabilities. to this end, we propose a novel hierarchical table prompt-based reprogramming framework (htp), to adapt llm for visualization recommendation without altering the backbone structure. the essence of htp lies in leveraging data-driven multilevel prompts to adaptively reprogram the tabular data across various dimensions, thereby bridging the gap between the structured nature of tabular data, as well as the inherent capabilities of llms for textual processing. specifically, htp utilizes four levels of prompts as follows: based on these four prompts, as illustrated in figure 1 , we concatenate prompts at general, instance, and cluster levels with a serialized table before feeding them into llms, thereby facilitating table-specific knowledge extraction and inte-gration. meanwhile, the column-level prompts are prepended to encoded inputs to retain the structural information of tables, thereby improving the distinction among columns, and fostering a contextual understanding between them. the contribution of this paper could be summarized as follows: ## methodology in this section, we will introduce the proposed framework in detail. as shown in figure 2 , our framework consists of four primary prompts, namely general, instance, column and cluster level prompts, to inject table insights into the llm. technical details of each component are discussed in the following subsections, step-by-step. ## conclusion table prompt-based reprogramming framework, called htp, to enhance the visualization recommendation process through large language models (llms). the htp harnessed potential of llms through four level of prompts, aiming at extracting semantic information from table through comprehensive perspectives. in this way, htp effectively unlocked the potential of llm for transforming table into insightful charts. extensive experimental results demonstrated the superior performance of htp framework.