Automating e HMI Action Design with LLM s for Automated Vehicle Communication (2025)

Abstract: The absence of explicit communication channels between automated vehicles (AVs) and other road users requires the use of external Human-Machine Interfaces (eHMIs) to convey messages effectively in uncertain scenarios. Currently, most eHMI studies employ predefined text messages and manually designed actions to perform these messages, which limits the real-world deployment of eHMIs, where adaptability in dynamic scenarios is essential. Given the generalizability and versatility of large language models (LLMs), they could potentially serve as automated action designers for the message-action design task. To validate this idea, we make three contributions: (1) We propose a pipeline that integrates LLMs and 3D renderers, using LLMs as action designers to generate executable actions for controlling eHMIs and rendering action clips. (2) We collect a user-rated Action-Design Scoring dataset comprising a total of 320 action sequences for eight intended messages and four representative eHMI modalities. The dataset validates that LLMs can translate intended messages into actions close to a human level, particularly for reasoning-enabled LLMs. (3) We introduce two automated raters, Action Reference Score (ARS) and Vision-Language Models (VLMs), to benchmark 18 LLMs, finding that the VLM aligns with human preferences yet varies across eHMI modalities. The source code, prompts, Blender scenarios, and rendered clips are available at https://github.com/ApisXia/AutoActionDesign.

## introduction automated vehicles (avs) promise to redefine transportation systems by eliminating human driving errors and optimizing traffic flow @xcite . however, the absence of a human operator disrupts road interactions, as drivers no longer exchange contextual cues (e.g., eye contact and gestures) to negotiate ambiguous scenarios @xcite . to bridge this gap, external human-machine interfaces (ehmis) have emerged as mediators, conveying av intent (e.g., yielding, turning) to other road users, such as pedestrians, cyclists, and human drivers @xcite . these interfaces use diverse forms, such as displays @xcite , led strips @xcite , projections @xcite , and attached robots @xcite , to convey vehicle intentions by text, signals, or non-verbal motions. current ehmis are usually designed and analyzed with predefined text messages (e.g., "please stop.", "i am worried.") with scenario information (e.g., "pedestrian is crossing the road", "robot is stuck in snow") and manually designed actions to perform these messages, as shown in figure 1 . this restricts the real-world deployment of ehmis because dynamic interactions demand adaptable communication strategies @xcite . therefore, developers must design actions for all possible messages that avs might need to communicate to other road users. this process is time-intensive, costly, and significantly limits the scalability of ehmis in practical scenarios @xcite . recently, large language models (llms) demonstrate generalizability and versatility in multiple tasks, such as reading and answering questions @xcite , as well as pattern following @xcite , suggesting that they may serve as suitable automated action designers for ehmis. however, it is unclear whether the application of llms for ehmis is feasible and useful, leading to our research question (rq): to what extent do pretrained llms achieve parity with human designers in designing ehmi actions that are understandable to road users? answering our rq involves three key challenges. first, there is no systematic pipeline for translating specified messages into executable action sequences for ehmis. second, there is a lack of highquality datasets for testing and improving the translation of ehmi messages into action sequences. third, there is no commonly used benchmark to compare different methods for designing and evaluating ehmi actions fairly. therefore, first, we propose a pipeline that integrates llms and 3d renderers. to adapt llms for controlling ehmis, we draw inspiration from llmbased robot action planning @xcite , which utilizes llms as action designers to generate a series of executable actions to actuate robotic motors. second, we introduce a user-rated action-design scoring dataset. the dataset comprised eight intended messages for the ehmi to convey by analyzing traffic scenarios, and selected four representative ehmi modalities frequently discussed in ehmi research. we collected messages from previous ehmi studies @xcite @xcite and designed new ones based on message types @xcite to enrich the variety. for each message-modality pair, we generated ten actions: eight produced by llms (gpt-4o @xcite , sonnet 3.5 @xcite , gemini 2 flash (deepmind, 2024), and gpt-o1 (ope-nai, 2024b)) and two designed by human experts. these actions were rendered using blender version 4. @xcite , resulting in 320 video clips. we conducted a video-based user study with human participants. they evaluated the understandability of the llm-designed actions by measuring the consistency between the intended messages and perceived meanings. the action-design scoring dataset provides averaged human scores for each action, enabling a comparative benchmark for existing llms. third, we introduce the action reference score (ars), which uses the similarity between the newly designed actions and those rated in our dataset. additionally, we discussed the potential of vision-language models (vlms) to serve as human-like raters. then, we benchmark 18 llms on this task. contribution statement: this work proposes the first complete pipeline, along with a comprehensive dataset and benchmark for evaluating ehmis. beyond these core contributions, we also share several noteworthy insights as follows: ## methodology this section outlines our responses to three key challenges: i) the llm-blender fusion pipeline, ii) the action-design scoring dataset, and iii) the automated evaluation system for benchmarks. these designs serve as a proof of concept for our rq and offer a systematic approach to developing and evaluating newer llms or ehmis modalities. ## experiments in this section, the experiments are designed to achieve three specific purposes. ## discussion challenges in both the auto-design and rating processes. for llm designers, we encountered two main problems. i) at the early stage of our prompt design, some models were "too lazy" to ex-plore creative alternatives and copied the patterns of our examples, which are not always suitable. encouraging more in-depth reasoning in the prompt helped mitigate this. ii) we noticed that llms tend to include expressions of gratitude, but human designers prefer effectiveness, which made llmgenerated actions much longer than those written by humans. one possible remedy is to instruct the llm to omit emotional expressions, but since emotion can be an essential part of some messages, finding the right balance between clarity and emotional tone -so that the actions feel human-like and satisfy human raters -remains a future direction. for vlm raters, we identified two main challenges: differences from human annotators and limited recognition capability for small changes within an image series (section 4.2). to address the first issue, we believe that collecting additional annotations from diverse human groups and finetuning vlms to better align with human preferences could be effective. regarding the second challenge, future improvements may involve utilizing more advanced vlm architectures. ## conclusion in conclusion, this work proposes the first llm-blender fusion pipeline to design ehmi actions. alongside this, we introduce the action-design scoring dataset. our findings suggest that pretrained llms can attain a nearly human-level capability in action design. additionally, we provide a benchmark that can be used to evaluate the capability of other llms. our work establishes a solid foundation for llm-based action design and the real-world application of ehmis.