Penetrative AI : Making LLM s Comprehend the Physical World (2024)

Abstract: Recent developments in Large Language Models (LLMs) have demonstrated their remarkable capabilities across a range of tasks. Questions, however, persist about the nature of LLMs and their potential to integrate common-sense human knowledge when performing tasks involving information about the real physical world. This paper delves into these questions by exploring how LLMs can be extended to interact with and reason about the physical world through IoT sensors and actuators, a concept that we term “Penetrative AI”. The paper explores such an extension at two levels of LLMs’ ability to penetrate into the physical world via the processing of sensory signals. Our preliminary findings indicate that LLMs, with ChatGPT being the representative example in our exploration, have considerable and unique proficiency in employing the embedded world knowledge for interpreting IoT sensor data and reasoning over them about tasks in the physical realm. Not only this opens up new applications for LLMs beyond traditional text-based tasks, but also enables new ways of incorporating human knowledge in cyber-physical systems.

## introduction large language models (llms) have made remarkable strides @xcite @xcite . a particularly revolutionary milestone is chatgpt (openai, 2023b), which excels in fluid, human-like conversations, marking a new era in human-ai interactions. these latest llms cultivated on extensive text datasets have showcased remarkable capabilities across diverse tasks, including coding and logical problemsolving @xcite . these out-of-thebox capabilities have demonstrated that they already comprise enormous amounts of world knowledge 2 . this paper is motivated by an essential and intriguing question: can we enable llms to complete tasks in the real physical world? we delve into this inquiry and explore extending the boundaries of llms' capabilities by directly letting them interact with the physical world through internet of things (iot) sensors. a basic example of this process is depicted in figure 1 , where different from the conventional way of llms, an llm is expected to analyze sensor data which are indeed projections from the physical world. we conjecture that llms, having been trained on vast amounts of human knowledge, learned the physical world which can be directly harnessed for analysis of such sensory information to derive deep insights that traditionally require background knowledge from human experts and/or bespoke machine learning models trained with large amounts of labeled sensor data. as illustrated in figure 1 , we formulate such a problem from a signal processing's point of view, and specifically explore the llms' penetration into the physical world at two signal processing levels with the sensor data: i) with the textualized signals derived from underlying sensor data, and ii) with the digitized signals, essentially numerical sequences of raw sensor readings. we term this endeavor "penetrative ai" -where the embedded world knowledge in llms serves as a foundation model, seamlessly integrated with the cyber-physical systems (cps) for perceiving and intervening in the physical world. our methodology is exemplified through two illustrative applications at two different levels, respectively -user activity sensing where textualized signals from smartphone accelerometer, satellite, and wifi data are analyzed to discern user motion and environment conditions, and human heartbeat detection where digitized electrocardiogram (ecg) data are utilized to derive the heartbeat rate. preliminary findings are encouraging, showcasing llms' proficiency in interpreting iot sensor data and performing perception tasks in the physical world. our exploration also underscores that existing llms like chatgpt-4 may already possess the capability to establish intricate connections among world knowledge and can be guided to tackle cps tasks. section 2 and section 3 will elaborate on the design and experiment results of these two illustrative applications. section 4 sets the scope of penetrative ai and shares our insights on the foreseeable challenges to advance this burgeoning research frontier. we present related works in section 5 and conclude this paper in section 6. ## penetrative llm with textualized signals this section describes tasking llms to comprehend sensor data at the textualized signal level. ## penetrative llm with digitized signals this section describes our effort that goes beyond the general expectations of the textualized signal processing ability of llms. we specifically study the potential of llms in comprehending digitized sensor signals. ## penetrative ai while not achieving perfect accuracy, llms exhibit surprisingly encouraging performance, even when dealing with pure digital signals. this presents an enticing opportunity to leverage llms' world knowledge as a foundation model to derive insights from sensory information while requiring no or little additional task knowledge or data, i.e., in zero or few-shot settings. such a capability may be equipped with iot sensors and actuators to build intelligence into cyber-physical systems -a concept we term "penetrative ai". ## conclusion we present penetrative ai and explore the potential of leveraging large language models as world models to accomplish real-world tasks with iot sensors. our findings illuminate a promising path for the integration of artificial intelligence and cps, offering insights into the future of ai-powered solutions.