A Conversational Agent Framework for Multimodal Knowledge Retrieval: A Case Study in FHWA I nfo H ighway Web Portal Queries (2025)

Abstract: The rapid proliferation of heterogeneous data in government and industry presents increasing challenges for users seeking to retrieve actionable insights across both structured and unstructured sources. To address this, this paper presents InfoTech Assistant, a novel multimodal conversational framework that enables natural language interaction with both semantic document retrieval and structured database querying. The system integrates Large Language Models (LLMs) with Retrieval-Augmented Generation (RAG) and schema-aware Text-to-SQL capabilities, enabling dual-mode processing of user input for unstructured explanations and relational analytics. The architecture features a modular, locally deployed backend built with Flask and optimized for Graphics Processor Unit (GPU) acceleration, supporting low latency, privacy preserving inference. User queries are dynamically routed through an intent-aware processing pipeline, leveraging sentence embeddings, schema metadata, and prompt engineering strategies. A pilot deployment using infrastructure datasets from the Federal Highway Administration (FHWA) InfoHighway portal demonstrates the system’s effectiveness in real-world domain-specific retrieval. The assistant ingests FHWA technology documents and National Bridge Inventory (NBI) text records, tables, and images organized in a hybrid schema supporting both semantic and SQL-driven interaction. Evaluation results show 95% accuracy in RAG-based semantic tasks and 88.6% success in translating natural language into executable SQL queries. These findings underscore the potential of hybrid LLM-based agents for scalable, secure knowledge access in critical public-sector and industrial applications.

## introduction public infrastructure management increasingly depends on multimodal data sources, including structured databases and unstructured documentation, to support tasks such as maintenance planning, inspection analysis, and policy development. however, deriving actionable insights from these sources often requires technical expertise in structured query language (sql), relational schema design, and the ability to interpret domain-specific documents. these challenges pose significant barriers to nontechnical stakeholders such as policymakers and professionals outside of database or data science domains, including engineers from civil or infrastructure disciplines. this highlights the growing need for unified systems that simplify access to both structured and unstructured data through natural language interaction. recent advancements in natural language processing (nlp), particularly with large language models (llms) and retrieval-augmented generation (rag) @xcite , offer promising solutions to bridge this gap. these approaches enable intuitive, conversational interfaces capable of synthesizing information from diverse sources. yet, most existing systems are tailored for open-domain use cases and primarily focus on either unstructured document retrieval or structured sql generation, rarely supporting multiple modalities within a unified framework. to address this limitation, this paper introduces infotech assistant, a multimodal knowledge retrieval framework that integrates llms within a dual processing architecture. the system supports both document grounded semantic retrieval via rag @xcite and structured querying through a schema-aware text-to-sql module. this design facilitates seamless interaction with infrastructure data text, tables, and image references through a single natural language interface. the system is deployed on a locally hosted, gpu-accelerated backend using a modular flask architecture, enabling low latency and privacy preserving inference. a case study on the federal highway administration's (fhwa) infobridge portal (federal highway administration, 2024a) demonstrates its effectiveness, with high accuracy observed across semantic and sql driven tasks. output formats include html tables and reference-linked responses, enhancing both usability and domain adaptability. the main contributions of this paper are as follows: ## related work structured data querying and unstructured knowledge retrieval remain key challenges in domainspecific question answering (qa). prior research in this space generally falls into two categories: text-to-sql techniques for translating natural language into executable database queries @xcite , and llmbased systems with rag for grounding responses in external documents @xcite . this section reviews representative work across both directions. ## system architecture framework the infotech assistant is a modular, extensible conversational system that integrates rag and text-to-sql capabilities. built on locally deployed llms and gpu acceleration, the system supports high performance, multimodal knowledge retrieval as shown in the figure. 1 illustrates the overall architecture. ## case study this section evaluates the infotech assistant in a real-world infrastructure context. the system, powered by meta llama 3.1 8b and hosted on a gpu-enabled vm, was tested using fhwa in-fobridge content (federal highway administration, 2024b) and a structured sqlite database built from fairfax county bridge datasets (federal highway administration, 2024a). evaluation spans two modes: rag for unstructured semantic queries and schema-guided text-to-sql for structured analytics. this dual setup supports multimodal, knowledge grounded qa over technical infrastructure data.