LIDA : A Tool for Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Models (2023)

Abstract: Systems that support users in the automatic creation of visualizations must address several subtasks - understand the semantics of data, enumerate relevant visualization goals and generate visualization specifications. In this work, we pose visualization generation as a multi-stage generation problem and argue that well-orchestrated pipelines based on large language models (LLMs) and image generation models (IGMs) are suitable to addressing these tasks. We present LIDA, a novel tool for generating grammar-agnostic visualizations and infographics. LIDA comprises of 4 modules - A SUMMARIZER that converts data into a rich but compact natural language summary, a GOAL EXPLORER that enumerates visualization goals given the data, a VISGENERATOR that generates, refines, executes and filters visualization code and an INFOGRAPHER module that yields data-faithful stylized graphics using IGMs. LIDA provides a python api, and a hybrid user interface (direct manipulation and multilingual natural language) for interactive chart, infographics and data story generation. Code and demo are available at this url - https://microsoft.github.io/lida/

## introduction visualizations make data accessible by reducing the cognitive burden associated with extracting insights from large tabular datasets. however, visualization authoring is a complex creative task, involving multiple steps. first the user must build familiarity with the dataset (content and semantics) and enumerate a set of relevant goals or hypotheses that can be addressed using the data. next, users must select the right visualization representation (marks, transformations and layout) for each goal. finally, the user must implement the visualization either as code or using available direct manipulation interfaces. each of these steps require expertise, and can be tedious as well as error prone for users with limited visualization experience (novices). existing research has sought to address these challenges by automating the visualization (autoviz) creation process, given a dataset @xcite . automation may occur in two modes: i.) fully automated -the system automatically generates visualizations relevant to the data ii.) semi-automated -the user specifies their goals and the system generates visualizations that address these goals. the former mode is valuable for users unfamiliar with the data and the latter is valuable for users with some familiarity with the data and the visualization task. consequently, a successful autoviz tool must excel at each of several subtasks -understand the semantics of the data, enumerate relevant visualization goals and generate visualization specifications that meet syntax, design, task and perceptual requirements of these goals @xcite . furthermore, given the target demographic (novice users), such a tool must support the user by offering nl (nl) interaction modalities @xcite @xcite , affordances to control system behavior and sense making tools to understand and debug/verify system behavior. while related work has addressed aspects of the autoviz task, there are several known limitations @xcite ) such as they: (i) rely on heuristics that are limited in coverage, challenging to craft and tedious to maintain @xcite . (ii) require significant user interaction to generate visualizations @xcite . (iii) implement automated approaches that offer limited control over system input and output @xcite (iv) require grammar (or chart type) specific training data and model architectures @xcite for each sub task, (v) do not consider alternative chart representation formats such as infographics. compared to existing autoviz approaches, lida proposes an implementation that is simplified (eliminates the need for subtask-specific models), general (can be adapted to generate visualizations in any programming language or grammar), flexible (individual modules can be optimized) and scalable (the system performance will improve with advances in the underlying llm). taken together, these contributions provide building blocks towards complex workflows such as visualization translation, chart question answering (with applications in accessibility of charts), automated data exploration and automated data stories. to the best of our knowledge, lida is the first tool to formulate visualization/infographic generation as a multi-step generation task and demonstrate an end-to-end pipeline that addresses a variety of subtasks. ## related work lida is informed by research on large foundation models applied to creative tasks across modalities such as text and images, and advances in automated generation of visualizations and infographics. ## the lida system lida comprises of 4 core modules -a summa-rizer, a goal explorer, a visgenerator and an infographer (see fig 1 ). each module is implemented in the lida github repo as a python library with an optional user interface (see appendix a). ## conclusion in this work, we formulate visualization generation as a multi-stage text (and code) generation problem that can be addressed using large language models. we present lida -a tool for the automatic generation of grammar-agnostic visualizations and infographics. lida addresses limitations of current automatic visualization systems -automatic generation of hypothesis/goals given datasets, conversational interface for controllable visualization generation and refinement, support for multiple visualization grammars using the same pipeline and the ability to generate infographics. lida is effective compared to state of the art systems (see example gallery of generated visualizations); it offers a simplified system implementation and leverages the immense language modeling and code generation capabilities of llms in implicitly solving complex visualization subtasks. finally, we introduce metrics for assessing reliability (visualization error rate -ver) and visualization quality (self-evaluated visualization quality -sevq) for llm-enabled visualization tools. we hope modules implemented in lida will serve as useful building blocks in enabling complex creative workflows such as visualization translation, chart question answering(with applications in accessibility of charts), automated data exploration and automated storytelling.