By My Eyes: Grounding Multimodal Large Language Models with Sensor Data via Visual Prompting (2024)

Abstract: Large language models (LLMs) have demonstrated exceptional abilities across various domains. However, utilizing LLMs for ubiquitous sensing applications remains challenging as existing text-prompt methods show significant performance degradation when handling long sensor data sequences. In this paper, we propose a visual prompting approach for sensor data using multimodal LLMs (MLLMs). Specifically, we design a visual prompt that directs MLLMs to utilize visualized sensor data alongside descriptions of the target sensory task. Additionally, we introduce a visualization generator that automates the creation of optimal visualizations tailored to a given sensory task, eliminating the need for prior task-specific knowledge. We evaluated our approach on nine sensory tasks involving four sensing modalities, achieving an average of 10% higher accuracy compared to text-based prompts and reducing token costs by 15.8 times. Our findings highlight the effectiveness and cost-efficiency of using visual prompts with MLLMs for various sensory tasks. The source code is available at https://github.com/diamond264/ByMyEyes.

## introduction large language models (llms) have shown remarkable performance in tasks across diverse domains, including science, mathematics, medicine, and psychology @xcite . the recent advent of multimodal llms (mllms), e.g., @xcite , has further expanded their capabilities to images and audio inputs, broadening their use in fields such as industry and medical imaging @xcite . meanwhile, sensor data-including measurements from smartphones, wearables, iot @xcite , and medical equipment (pantelopoulos and bourbakis, 2009)-holds potential for ubiquitous applications when effectively integrated with mllms. sensory tasks involve extensive and our method 2,020 tokens 52,910 tokens figure 1 : an example of solving a sensory task using an mllm with visual prompts. the visualization generator generates an appropriate visualization for the given sensor data, and the visualized data is provided as an image to the mllm for solving the task. recent studies explored leveraging pre-trained llms to solve general sensory tasks @xcite @xcite . one approach extracts task-specific features from sensor data and composes them as prompts @xcite . however, designing such prompts requires specific domain knowledge. alternatively, incorporating raw sensor data as text prompts @xcite has been a widely used method to handle sensory data with llms as a more generalizable solution. yet, we empirically found that providing raw sensor data with text prompts shows poor performance in real-world sensory tasks with long-sequence inputs and incurs high costs due to an extensive number of tokens. to address these challenges, we propose providing visualized sensor data as images to mllms that support visual inputs. leveraging mllms' growing ability to interpret visual aids @xcite , we explore their effectiveness in analyzing plots generated from sensor data. we designed a visual prompt comprising visualized sensor data and task-specific instructions to solve sensory tasks. in addition, we present a visualization generator that enables mllms to independently generate optimal visualizations using tools available in public libraries. this generator filters potential visualization methods based on the task description and assesses the resulting visualizations of each method to determine the best visualization. figure 1 compares the existing text-only prompts with our method for sensory tasks. evaluations on nine sensory tasks involving four different modalities showed that visual prompts generated from the visualization generator significantly improved performance by an average of 10% while reducing token costs by 15.8× compared with the existing baseline. our findings highlight the effectiveness and efficiency of visualized sensor data with mllms in various applications. we summarize our contributions as follows: llms with sensor data. sensory tasks involve sequences of numbers indicating values over time. initial research for handling sequential data focused on time-series forecasting @xcite . converting time-series data into text prompts for forecasting has been proposed in promptcast @xcite and llmtime @xcite . other studies @xcite used specialized encoders to create embeddings compatible with pre-trained llms. beyond forecasting, llms have been explored in healthcare for their ability to answer questions using physiological sensor data @xcite . for example, llms have been used for ecg diagnosis @xcite by integrating ecg-specific features and knowledge from ecg databases. penetrative ai @xcite and health-llm @xcite have used raw sensor data in text prompts to solve health problems without taskspecific processing. our study examines whether existing methods can generalize to broader sensing tasks with high-frequency, long-duration data. building upon these works, we propose visualizing sensor data for mllms to improve their performance and cost efficiency. multimodal large language models (mllms). advancements in mllms @xcite have equipped popular models such as chat-gpt (openai, 2022) with vision capabilities @xcite . recent studies explored the in-context learning @xcite abilities of mllms, showing that they can understand images with the interleaved text and few-shot examples @xcite . this capability has been applied in medical diagnostics, including analyzing radiology and brain images with accompanying text instructions @xcite . our work explores using mllms to analyze visualized sensor data for broader applications. using tools with llms. recent research has shown that augmenting llms with external tools can extend their capabilities. toolformer @xcite enables llms to access public apis and search engines, while visual programming @xcite uses llms to generate and execute codes. hugginggpt @xcite and chameleon @xcite integrated multiple expert models to enhance functionalities. recently, data interpreter @xcite llms to analyze data and build taskspecific models for data interpretation. building on them, our work leverages mllms to utilize sensor data visualization tools. importantly, unlike existing approaches that rely on external tools as the pri-mary task solvers, we propose positioning mllms as the main solvers for sensory tasks, leveraging their in-context learning abilities. our approach aims to eliminate the need for additional data collection and training, thereby addressing the scarcity of public resources for sensory tasks. furthermore, we introduce a design in which mllms perform demonstration-based assessments to evaluate their task-solving effectiveness, ensuring optimal visualization for specific tasks. ## limitations of representing sensor data as text-based prompts existing approaches for grounding language models with sensor data primarily rely on text-based prompts @xcite @xcite . one approach uses prompts with specialized features extracted from sensor data for specific tasks, such as r-r intervals for ecg-based applications @xcite . while this approach effectively handles known sensory tasks, feature-based prompts often require domain knowledge, which is not generalizable for non-expert users. instead, a more common approach @xcite @xcite incorporates raw sensor data sequences directly into prompts without data processing. however, most studies focus on short sequences (e.g., fewer than 100 elements) @xcite and simple tasks (e.g., binary classification) @xcite . real-world sensor data often entail long numeric sequences with high sampling rates and long durations. for example, arrhythmia detection @xcite requires ecg data sampled at 100hz over 10 seconds, resulting in 1,000 elements. this section investigates the limitations of using text-based prompts to represent such complicated sensor data in language models. we focus on the capability to interpret sensor data and the token consumption costs associated with long numeric sequences. language models struggle to interpret long numeric text sequences. language models interpret simple numeric sequences by performing arithmetic operations @xcite and understanding sequential data @xcite . however, we empirically revealed that their performance declines significantly with longer sequences inside the prompt, such as those exceeding 100 numbers, common in sensor data. we conducted experiments with two specific tasks: mean prediction to evaluate arithmetic capabilities @xcite and wave classification to assess pattern recognition @xcite in sequences. the defined tasks represented the basic functionalities for sensor data interpretation, serving as typical feature extraction methods. using randomly generated sine and sawtooth waves with varying lengths, we asked a language model, gpt-4o (openai, 2024), to calculate mean values and classify wave types using one-shot examples for each task. each task was repeated 30 times to ensure robustness. figure 2 shows the results. in arithmetic operations, error rates consistently increased with the length. in pattern recognition, performance declines significantly for sequences longer than 100 elements, approaching the performance of a random classifier at 500 elements. while recent models such as gpt-4o, with its 128k context window, support long input lengths, our results indicate that interpreting sensor data with long numeric sequences still remains challenging. sensor data in text is costly. the computational and financial burdens for api users of language models scale with the number of tokens in the prompt. representing sensor data in textual format leads to extensive token usage, thereby increasing costs. for instance, performing passive sensing to track activities with smartphone accelerometers @xcite uses sensor data sampled at 100 hz. collecting this data over a minute results in a prompt of 18k numbers, translating to 90k tokens. this leads to a huge cost of $450 per hour when using gpt-4o api to classify six activities with one example for each. higher sampling rates or longer durations further increase the costs, making such applications infeasible. ## method we introduce our method for handling sensory tasks by providing sensor data as image inputs to mllms. section 4.1 overviews our prompt design strategy. section 4.2 introduces our visualization generator, which automatically generates suitable visualizations for heterogeneous sensor data. ## experiments we evaluate the applicability of our approach with mllms by conducting experiments on a range of sensory tasks. ## conclusion we addressed sensory tasks by providing visualized sensor data as images to mllms. we designed a visual prompt to instruct mllms in using visualized sensor data, provided with textual descriptions of the task and data collection methods. additionally, we introduced a visualization generator that automatically selects the best visualization method for each task using visualization tools available in public libraries. we conducted experiments across nine different sensory tasks and four sensor modalities, each with a distinct task. our results suggest that the visual prompts generated by our visualization generator not only improve accuracy by an average of 10% over text-based prompts but also significantly reduce costs, requiring 15.8× fewer tokens. this indicates that our approach with visual prompts and a visualization generator is a practical solution for general sensory tasks.