Do LLM s Generate Creative and Visually Accessible Data visualisations? (2024)

Abstract: Data visualisation is a valuable task that combines careful data processing with creative design. Large Language Models (LLMs) are now capable of responding to a data visualisation request in natural language with code that generates accurate data visualisations (e.g., using Matplotlib), but what about human-centered factors, such as the creativity and accessibility of the data visualisations? In this work, we study human perceptions of creativity in the data visualisations generated by LLMs, and propose metrics for accessibility. We generate a range of visualisations using GPT-4 and Claude-2 with controlled variations in prompt and inference parameters, to encourage the generation of different types of data visualisations for the same data. Subsets of these data visualisations are presented to people in a survey with questions that probe human perceptions of different aspects of creativity and accessibility. We find that the models produce visualisations that are novel, but not surprising. Our results also show that our accessibility metrics are consistent with human judgements. In all respects, the LLMs underperform visualisations produced by human-written code. To go beyond the simplest requests, these models need to become aware of human-centered factors, while maintaining accuracy.

## introduction when evaluating ai systems, we typically focus on accuracy. however, generative ai systems, such as language models, are being applied to tasks where other, human-centered, factors are important too. an output can be accurate, but not accessible, e.g., if the colours chosen make a data visualisation hard to read, or there is not enough space between labels, see figure 1 . similarly, an output can be accurate, but not creative, e.g., if the data visualisation always has a linear scale, when in some cases a log-scale would reveal additional patterns. creativity is present in a range of human activities, from structured goal-oriented tasks like writing code or creating recipes @xcite , to more open-ended tasks like writing a story @xcite or painting @xcite . in the context of data visualisation, a creative visualisation presents the data in an unexpected way that more effectively communicates the data to the viewer @xcite . creativity can be defined in terms of value, novelty and surprise @xcite . even though llms can produce valuable artifacts, achieving novelty and surprise is still a challenge @xcite . recent studies concluded that at the individual level, systems are better than some people. however, at the collective level, systems tend to produce homogenous outputs @xcite , which raises concern about the potential impact on creativity when these tools are used by people. accessibility is another critical human-centered aspect of various tasks. it is a key part of inclusive design, which aims to make tasks available to everyone @xcite . for data visualisation, there are many potential pitfalls, such as colours that are hard to distinguish, or text that is difficult to read. most tasks with llms do not have to consider accessibility directly, as the output is text, and accessibility is then the concern of the text-rendering system. data visualisation is an interesting exception, where accessibility is crucial, and (unlike tasks like image generation), it may be measureable. this work investigates the creativity and accessibility of llm generated data visualisations through a human study conducted with 57 people. we show people outputs from two llms and examples from the documentation of libraries for data visualisation. the questions probe the notion of creativity in several ways, with absolute judgements (e.g., asking if any data visualisation was surprising) and relative judgements (e.g., selecting the best data visualisation from a small set). we apply several standard approaches to encourage greater llm creativity, including demonstrative prompts (issak and varshney, 2023), e.g., "using your imagination", and variation in configuration hyperparameters, e.g., different temperature values. for accessibility, we define two new metrics, one focused on the spacing of text and the other focused on color choices. our metrics are automatic, and we use questions in our study to verify their consistency with human perception. we find that the llms can generate data visualisations that are novel, but not surprising. our visual accessibility metrics are consistent with human perception, indicating that they can be used in future work. applying the metrics to a large sample, we see that llm outputs span a far wider range of scores than human created data visualisations do. do llms generate creative and visually accessible data visualisations? no, while the data visualisations being produced today are effective for simple tasks, there is scope for improvement in creativity, which must occur without sacrificing accessibility or accuracy. ## related work metrics for code generation existing work has evaluated the correctness of programs generated in response to a natural language query. for example, @xcite proposed variations in evaluation of text-to-sql, and @xcite considered more general programming questions. in both of these cases, there are multiple solutions included in the dataset, but they are only considered in the evaluation for measuring accuracy. in the former case, the results of executing the code are also considered, partly because the authors point out that there are multiple correct solutions. measuring partial matching of code is similar to measuring partial matches in tasks such as machine translation. metrics like bleu and bertscore have been adapted to code, e.g., in codebertscore @xcite , which was more accurate than prior metrics on the conala dataset @xcite . in all of these cases, the focus is on accuracy, rather than the additional human-centred factors we focus on here. prior work has considered creativity in code @xcite , arguing as we do that they are more than just a task-solving process. however, their focus was on the code itself, whereas we are also interested in the creativity of the output it generates. metrics for creativity looking beyond code, there has been some work considering creativity in the output of generative models. @xcite considered image generation, arguing that standard loss functions for these models encourage them to produce "more of the same", rather than more unusual out-of-distribution outputs. they point to prior work in computational creativity measurement as a potential avenue for guiding model development. some have argued that for a system to be creative, it should integrate creativity in the process of self-exploration and self-modification @xcite . we do not subscribe to this view. instead, we see creativity in output as a property that can be judged by humans, regardless of the process that generated it. in the case of llms, inherently creative domains, such as recipes, may seem like a promising space for creativity, but in practise, researchers have needed to set low temperatures in order to achieve consistency between ingredients and instructions @xcite . one example of an effort to measure creativity is deepcreativity @xcite . the system weights three factors of creativity: value, novelty, and surprise. valuable is a binary label judged by a trained model. novelty is the euclidean distance between a vector representing style and one of typical values. surprise is the difference between the prior and posterior distribution of a sequential predictive model. this approach is effective at modeling creativity in poetry over time, but the use of a sequential model means there is a strong assumption of time-based variation, and it is unclear how to generalise their methods to the code setting. human evaluation for creative tasks we study creativity and calibrate our accessibility metrics by conducting a study in which people judge data visualisations. human evaluation is often a critical part of evaluating human-centred factors like creativity. @xcite evaluated open-ended text generation from the wikitext-103 dataset using contextualized embedding metrics such as mauve. by comparing automatic and human judgements from two annotators, they identified a range of issues with automatic metrics, emphasising the importance of human evaluation. @xcite measure creativity using the consensing assessment technique and propose the torrance test of creativity writing (ttcw). ten experts rated human and ai stories considering fluency, flexibility, originality, and elaboration in writing. they found that 84% of human stories passed the rubric, while only 9% passed for gpt-4, and 30% for claude. like @xcite , they found disagreements between human judgements and automatic metrics. outside of text, humans have also been used to evaluate a range of other creative tasks. for example, mechanic miner @xcite is a game generation system, which was evaluated by getting over 5,933 people to play generated levels and rate the enjoyment and difficulty of the level. all of this past work supports the idea that human evaluation is critical in creativity judgement. ## experiments this work has three key components: (1) creating examples of llm generated data visualisations, (2) writing metrics for accessibility, and (3) a human study foot_0 in which we collect judgements of creativity and accessibility. ## analysis we obtained responses from a total of 57 participants. their experience with visualisations varied significantly 12.3% indicated low experience, 64.9% indicated medium experience, and 22.8% indicated high experience. in terms of tools, all but two had used excel, and 36.8% had never used tableau. in terms of programming languages, 42.1% had used just python, and 34% had used both python and r. 61.4% also reported using, at least one time, a programming language other than python and r for visualisations. this range of expertise indicates that our sample is not biased towards people with a specific background in terms of tools. ## conclusions do llms generate creative and visually accessible data visualisations? regarding creativity, the code itself is not particularly creative, and the outputs are sometimes novel, but not surprising. for accessibility, generated data visualisations are typically effective, but can span a wide range of effectiveness. overall, this work shows that data visualisation remains a challenging space for llms to generate creative outputs. that is not a major issue for generating simple data visualisations, but more work is needed to be able to handle more personalized or complex requests.