O pen O mni: A Collaborative Open Source Tool for Building Future-Ready Multimodal Conversational Agents (2024)

Abstract: Multimodal conversational agents are highly desirable because they offer natural and human-like interaction.However, there is a lack of comprehensive end-to-end solutions to support collaborative development and benchmarking.While proprietary systems like GPT-4o and Gemini demonstrating impressive integration of audio, video, and text with response times of 200-250ms, challenges remain in balancing latency, accuracy, cost, and data privacy.To better understand and quantify these issues, we developed OpenOmni , an open-source, end-to-end pipeline benchmarking tool that integrates advanced technologies such as Speech-to-Text, Emotion Detection, Retrieval Augmented Generation, Large Language Models, along with the ability to integrate customized models.OpenOmni supports local and cloud deployment, ensuring data privacy and supporting latency and accuracy benchmarking. This flexible framework allows researchers to customize the pipeline, focusing on real bottlenecks and facilitating rapid proof-of-concept development. OpenOmni can significantly enhance applications like indoor assistance for visually impaired individuals, advancing human-computer interaction.Our demonstration video is available https://www.youtube.com/watch?v=zaSiT3clWqY, demo is available via https://openomni.ai4wa.com, code is available via https://github.com/AI4WA/OpenOmniFramework.

## introduction large language models (llms) @xcite demonstrated remarkable capabilities in understanding user intentions and following instructions. however, text-only human-computer interaction (hci) is often insufficient @xcite . openai recently released their new flagship model, gpt-4o, which can reason across audio, video, and text in real time. the impressive performance is achieved with response times between 200-250ms, which is acceptable for large-scale applications foot_0 . google soon followed with their latest multimodal competitors, indicating a clear trend towards multimodal generative models and applications foot_1 . llava @xcite is one of the early publicly available solutions for multimodal large models integrating text and images. however, there is currently no open source end-to-end conversational agents implementation and demonstration publicly available online. the ideal form of multimodal hci should mirror human interactions, incorporating video and audio inputs with audio outputs. despite the availability of various modular components, there is no comprehensive integrated open-source implementation to foster research and innovation in this field. integrating existing models, such as audio speech recognition (speech2text), multimodal large models @xcite , and text-to-speech synthesis (tts)-into a multimodal conversation system reveals significant challenges in balancing latency and accuracy. historically, accuracy has been a major hurdle; however, advancements in large language models (llms) have substantially improved contextual relevance. the main challenge is reducing end-to-end latency while maintaining accuracy. while openai and google have shown it's possible, the open-source community lacks alternatives that replicate this performance. another issue is data privacy. the gpt-4 family of solutions also raise concerns about cost and data privacy. since gpt-4 is closed-source, users must upload their data to the server via a paid api, raising privacy issues. the privacy policy of gpt foot_2 informs users that various forms of personal information, including account details, user content, communication information, and social media data, response audio storage client: file sync client: file sync client: file sync relationaldb graphdb client: push client: push client: pull speech2text emotiondetection llm text2speech rag annotator benchmark dashboard human evaluation client api interface researchers annotator agent storage figure 1: architecture design for openomni framework to support the rapid and responsible development of this new hci format, establishing robust evaluation and benchmarking protocols is essential. for instance, if a user initiates a conversation in a sad and urgent tone, the system should respond appropriately with patience. evaluating this interaction is crucial and challenging for widespread adoption. our project aims to bridge these gaps by: to achieve this goal, we propose the openomni framework, an open-source, end-to-end multimodal pipeline that integrates advanced technologies such as speech-to-text (speech2text), emotion detection, retrieval augmented generation (rag), large language models (llms), and text-to-speech (tts). the framework gathers video and audio data from cameras and microphones, processes it through a customizable agents pipeline, and responds via a speaker, as illustrated in figure 1 . openomni can be deployed on a local server, ensuring secure data management and addressing privacy concerns. for research purposes, it includes tools for easy annotation and benchmarking, offering real-time monitoring and performance evaluation of latency. users can annotate individual components and entire conversations, generating comprehensive benchmark reports to identify bottlenecks. the open-source nature of openomni allows for adaptation across different application domains, such as aged care, personal assistant, etc. each pipeline component can be enabled or disabled based on specific use cases, facilitating flexible and efficient deployment. additionally, the framework supports the easy addition of extra models, enabling comparisons and further experimentation. the openomni framework allows researchers to focus on solving critical bottlenecks without reinventing the wheel, fostering innovation in multimodal conversational agents. it enables rapid proof-of-concept development, such as indoor conversational robots assisting visually impaired individuals. text audio sequence of images image2text chunk external data vision llm figure 2: traditional divide-and-conquer end-to-end multimodal conversation system modal conversation systems, as shown in figure 2 , typically use a divide-and-conquer strategy, splitting the process into sub-tasks: speech-to-text (automatic speech recognition), image-to-text, text generation, and text-to-speech @xcite . speech-to-text converts spoken language into text, while image-to-text generates textual descriptions of images. text generation, often powered by large language models, produces contextually appropri-ate responses, and text-to-speech converts these responses back into spoken language. these core components form the backbone of the conversational pipeline. image-to-text adds essential context, enhancing natural human-computer interaction, and additional tasks like emotion detection tailor responses to the user's emotional state. a safe guard module can optionally be integrated to ensure responses are appropriate, non-harmful, and controllable, maintaining interaction integrity, especially in sensitive scenarios. while this modular approach allows for optimization of individual components, the accumulated latency and accuracy errors can render the end-to-end system impractical for real-world use. ## conclusion multimodal conversational agents offers a more natural human-computer interaction, exemplified by models like gpt-4o. however, real-world constraints necessitate balancing cost, latency, and accuracy, which may explain why gpt-4o's full capabilities are not yet accessible. there are several technical options to achieve this, including traditional divide-and-conquer methods, fully end-to-end models like gpt-4o, and hybrid approaches. the fully end-to-end approach inherently allows for lower latency, while the divide-and-conquer method faces latency issues when coordinating multiple components. both approaches must address the challenge of handling large data i/o. if models are deployed locally, local network i/o issues can be more manageable. however, openai's models are closed-source, making local deployment impractical. while deploying other vision models locally is feasible, achieving high accuracy may be limited by local computational resources. hybrid solutions provides alternative approaches: pre-processing or compressing large data locally and then utilizing cloud-based models, or converting video to text and integrating it into the end-to-end voice model. we developed the openomni framework to enable researchers to integrate their work into an endto-end pipeline. the framework supports various solutions, allows for pipeline customization, generates latency performance reports, and provides an annotation interface for accuracy review. these features facilitate the creation of benchmark reports to identify and address key issues. testing with the us presidential debate scenario highlighted latency as a critical issue, particularly 51 with large video data. integrating external knowledge remains a challenge, emphasizing the need for efficient retrieval-augmented generation (rag). for applications like indoor assistance for the visually impaired, latency improvements and model adaptation are both essential. the openomni framework can significantly benefit the research community by facilitating the collection and management of new datasets, integrating various conversational agents approaches, and generating automatic latency benchmarks. its annotation interface aids in accuracy performance review, making openomni production-ready for suitable application scenarios and fostering further development in multimodal conversational agents.