C oll EX – A Multimodal Agentic RAG System Enabling Interactive Exploration of Scientific Collections (2025)

Abstract: In this paper, we introduce CollEx, an innovative multimodal agentic Retrieval-Augmented Generation (RAG) system designed to enhance interactive exploration of extensive scientific collections. Given the overwhelming volume and inherent complexity of scientific collections, conventional search systems often lack necessary intuitiveness and interactivity, presenting substantial barriers for learners, educators, and researchers. CollEx addresses these limitations by employing state-of-the-art Large Vision-Language Models (LVLMs) as multimodal agents accessible through an intuitive chat interface. By abstracting complex interactions via specialized agents equipped with advanced tools, CollEx facilitates curiosity-driven exploration, significantly simplifying access to diverse scientific collections and records therein. Our system integrates textual and visual modalities, supporting educational scenarios that are helpful for teachers, pupils, students, and researchers by fostering independent exploration as well as scientific excitement and curiosity. Furthermore, CollEx serves the research community by discovering interdisciplinary connections and complementing visual data. We illustrate the effectiveness of our system through a proof-of-concept application containing over 64,000 unique records across 32 collections from a local scientific collection from a public university.

## introduction the exploration of scientific knowledge is a cornerstone of human progress. however, the vast and rapidly growing body of scientific literature presents significant challenges for educators and learners, who often find themselves overwhelmed by the sheer volume and complexity of information. despite advancements in information retrieval and knowledge discovery @xcite @xcite , existing search systems for rich and complex data often lack the interactivity, intuitiveness, and cross-modal search capabilities @xcite @xcite to engage diverse audiences, such as students, teachers, or researchers. this limitation negatively affects educational settings where fostering curiosity is essential. ## the collex system this section describes the collex system, i.e., its architecture and core components, as well as the data to be explored. ## system demonstration in the following, we demonstrate collex showcasing some general functionality and two exemplary user stories depicted by screenshots of the app foot_11 . due to the limited space to display the screenshots and the thereby induced readability issues because of the small image sizes, we provide highresolution screenshots in appendix d. ## conclusion in this work, we introduced collex, an innovative multimodal agentic rag system aimed at facilitating interactive and intuitive exploration of extensive scientific collections. leveraging stateof-the-art lvlms, collex provides a powerful yet user-friendly interface for diverse audiences, such as pupils, students, educators, or researchers. our proof-of-concept implementation, covering over 64,000 scientific items across 32 diverse collections, successfully demonstrates the system's potential, showcasing capabilities such as cross-modal search, advanced semantic retrieval, and agentdriven interactions. additionally, collex serves as a versatile blueprint that can be straightforwardly applied to other scientific collections. in conclusion, with collex, we presented an innovative system to interactively explore scientific collections, enhancing educational and researchoriented applications, thereby positively contributing to the broader scientific community. despite the promising potential of our introduced system, we acknowledge several limitations summarized in the following: firstly, user experience when using collex heavily depends on the capabilities of the underlying lvlms. if a model misinterprets the user intent, invokes incorrect or irrelevant tools, misuses parameters, misunderstands tool responses, or fails to communicate results clearly and engagingly, the application's usability and user satisfaction significantly suffers. such issues might lead to frustration among users, diminishing their excitement in the tool and thereby scientific exploration which is the opposite of our intention. secondly, collex performs optimally with proprietary lvlms, which can create dependency and privacy issues including substantial ongoing costs and reliance on external model providers. although the system supports integration with open-source lvlms, the overall user experience often suffers, as open-source alternatives generally lag behind in accuracy, responsiveness, and general robustness. thirdly, collex currently integrates an extensive range of tools that, while offering powerful capabilities, sometimes overwhelms or confuses the lvlm. this complexity can lead to inappropriate or inefficient tool use, further impacting the overall user experience negatively. a potential solution would involve reorganizing the system from a single agent into multiple specialized agents managed hierarchically by an orchestrator agent. this would simplify decision-making processes and tool invocation more effectively. however, since we currently do not rely on any agentic frameworks or libraries to implement collex, this introduces several challenges such as optimizing the intercommunication between the agents. lastly, the current implementation of collex lacks formal evaluation of both the overall system and its individual components. this is primarily due to the considerable investment in computational and human resources required for comprehensive user studies and empirical assessments. without systematic evaluations, it remains challenging to quantify the true effectiveness, usability, and scalability of the system in real-world contexts. therefore, conducting extensive evaluations to validate the system's performance and identify areas for improvement is a priority for future work.