Autonomous Workflow for Multimodal Fine-Grained Training Assistants Towards Mixed Reality (2024)

Abstract: Autonomous artificial intelligence (AI) agents have emerged as promising protocols for automatically understanding the language-based environment, particularly with the exponential development of large language models (LLMs). However, a fine-grained, comprehensive understanding of multimodal environments remains under-explored. This work designs an autonomous workflow tailored for integrating AI agents seamlessly into extended reality (XR) applications for fine-grained training. We present a demonstration of a multimodal fine-grained training assistant for LEGO brick assembly in a pilot XR environment. Specifically, we design a cerebral language agent that integrates LLM with memory, planning, and interaction with XR tools and a vision-language agent, enabling agents to decide their actions based on past experiences. Furthermore, we introduce LEGO-MRTA, a multimodal fine-grained assembly dialogue dataset synthesized automatically in the workflow served by a commercial LLM. This dataset comprises multimodal instruction manuals, conversations, XR responses, and vision question answering. Last, we present several prevailing open-resource LLMs as benchmarks, assessing their performance with and without fine-tuning on the proposed dataset. We anticipate that the broader impact of this workflow will advance the development of smarter assistants for seamless user interaction in XR environments, fostering research in both AI and HCI communities.

## introduction the advent of "industry 4.0", centered on the concept of smart manufacturing, presents a landscape with both opportunities and challenges for enhancing production efficiency @xcite (a) industrial car assembly. mixed reality (mr), encompassing both virtual reality (vr) and augmented reality (ar), spans a spectrum from fully real environments to "matrixlike" virtual environments, showing promise for industrial manufacturing assembly tasks @xcite @xcite . these multimodal, interactive, user-centric environments provide a solution for trainees who experience significant cognitive workload for training @xcite @xcite . however, the assistance of a senior figure 2 : the proposed autonomous workflow, involving an ai agent interacting with an mr application. the ai agent comprises a core cerebral language agent, which interacts with a vision-language agent to interpret the multimodal context into metadata, which can be utilized by the cerebral language agent iteratively. the mr application interacts with ai agents by serving functions as external tools. to advance intelligent virtual assistants, traditional work leverages natural language processing (nlp) techniques @xcite @xcite and reinforcement learning @xcite to promote human-machine interactions. large language models (llms), as the new era of prevalent nlp techniques, have been observed to elicit diverse interaction patterns across tasks, demonstrating their versatility and feasibility @xcite . however, (i) tailoring assistant services by grounding interactions; and (ii) understanding users' situated multimodal contexts remain challenging and under-explored @xcite . to this end, we introduce an autonomous workflow (see figure 2 ) tailored for integrating ai agents into mr applications for fine-grained training. we present a demonstration of a multimodal fine-grained training assistant within a toy mr application for lego brick assembly. specifically, we design a cerebral language agent that integrates the llm with memory, planning, and interaction with mr serving functional tools and a visionlanguage agent, enabling agents to decide their actions based on experiences. then, we introduce lego-mrta, a multimodal fine-grained assembly dataset synthesized automatically by a commercial llm. this dataset comprises 65 multimodal instruction manuals, 1,423 conversations with visionlanguage pairs, serving usages of 18 functional tools in an mr environment. additionally, several prevailing open-resource llms are presented as benchmarks, assessing their performance with and without fine-tuning on the proposed dataset. furthermore, we anticipate that the broader impact of this workflow will advance the development of smarter assistants for seamless user interaction in mr environments, fostering research in both ai and hci communities. we summarize our contributions as follows: • we design a workflow, which integrates autonomous ai agents for fine-grained assembly assistance in an mr demonstration. ## related work we summarize previous research concerning multimodal datasets and virtual dialogue assistants within the realm of mr. ## fine-grained training workflow in this section, we describe the proposed workflow (see figure 2 ) that advances ai agents towards mr guided fine-grained training. ## dataset creation in this section, we introduce how to use the proposed workflow (see §3) to create a multimodal dialogue dataset in the mr environment. ## discussion of broader impact the research presented in this paper offers a fully new environment to advance how workers are trained and get help by using mr technologies. by integrating ai assistants into mr environments, workers can tackle complex tasks more effectively. this innovation not only enhances worker productivity but also reduces training costs for companies, as it eliminates the need for expert instructors to be physically present for employee training sessions. ## conclusion in this work, we introduce an autonomous workflow to develop smarter multimodal fine-grained training assistants in mr environments. specifically, we have designed a cerebral language agent that integrates llm with memory, planning, and interaction with mr tools, along with a visionlanguage agent. this integration enables agents to make decisions based on past experiences, thereby addressing the challenge of tailoring assistant services through grounded interactions. we have designed a vision-language agent to better understand users' situated multimodal contexts. notably, we have created a dataset for fine-grained training in mr. we have compared the performance of openresource llms before and after fine-tuning using this dataset. we aim to facilitate the development of smarter assistants for seamless user interaction, fostering research in ai and hci communities.