Evaluating the Performance of RAG Methods for Conversational AI in the Airport Domain (2025)

Abstract: Airports from the top 20 in terms of annual passengers are highly dynamic environment with thousands of flights daily, and they aim to increase the degree of automation. To contribute to this, we implemented a Conversational AI system that enables staff in an airport to communicate with flight information systems. This system not only answers standard airport queries but also resolves airport terminology, jargon, abbreviations, and dynamic questions involving reasoning. In this paper, we built three different Retrieval-Augmented Generation (RAG) methods, including traditional RAG, SQL RAG, and Knowledge Graph-based RAG (Graph RAG). Experiments showed that traditional RAG achieved 84.84% accuracy using BM25 + GPT-4 but occasionally produced hallucinations, which is risky to airport safety. In contrast, SQL RAG and Graph RAG achieved 80.85% and 91.49% accuracy respectively, with significantly fewer hallucinations. Moreover, Graph RAG was especially effective for questions that involved reasoning. Based on our observations, we thus recommend SQL RAG and Graph RAG are better for airport environments, due to fewer hallucinations and the ability to handle dynamic questions.

## introduction amsterdam airport schiphol, one of the top 20 airports in the world, ranked by annual passenger numbers, handles thousands of flights each day. these airports rely on staff like gate planners and apron controllers to access and update data across systems. for these employees, traditional database queries can be complex and time-consuming for some employees who are not query experts when they need flight information. a conversational ai system with a natural language query (nlq) interface allows all employees to interact with systems naturally, asking questions like, "which fights are at ramp d07?" and receiving instant answers. this improves productivity, and streamlines workflows, especially in high-pressure areas like at the gate, where less educated workers require access to upto-date information. by replacing strict query formats with intuitive, real-time responses, conversational ai enhances decision-making and efficiency, making it a suitable solution for dynamic environments such as airports. building such a system is challenging because flight data is stored by experts in tables using aviation abbreviations. we need our system to understand these datasets to answer questions from the airport domain. additionally, ensuring aviation safety is a major concern; the system must be safe and enable employees to perform accurate operations. we address those challenges using two research questions. the first question is how to handle flight data so that our system can answer different questions. we divided the questions into three types: the second research question is about how to reduce hallucinations @xcite for the safety of aviation operations. hallucinations occur when llms generate information not based on facts or their training data. in high-safety environments such as airports, however the output should be factual and not imaginative @xcite . for example, if the system gives wrong gate numbers, flight schedules, or safety instructions, this might disrupt aviation operations, cause delays, or even risk passenger safety. thus, accurate responses are important. in this case study, we examine three retrieval-augmented generation (rag) techniques for the airport environment: traditional rag @xcite retrieves relevant information from the flight database and uses llms to generate answers based on the retrieved data and original questions. sql rag @xcite stores all datasets in an sql database and converts natural language questions (nlq) into structured sql queries. knowledge graph-based retrieval-augmented generation (graph rag) @xcite aims to improve the performance of llm tasks by applying rag techniques to knowledge graphs (kgs), requiring the original datasets to be stored in the knowledge graph. a key challenge is retrieving the correct flight information from thousands of flights while minimizing hallucinations. the paper is structured as follows: we first survey related work (sec. 2), then present our dataset (sec. 3), followed by a high-level description of our experiments (sec. 4). we then present the results for the research questions (sec. 5), and lastly conclude (sec. 6). in the appendix a, we provide further details, especially on the question generation and classification, next to our prompting. ## dataset and questions our flight information dataset is tabular containing thousands of flights with key details such as flight number, aircraft category, bus gate, bus service needed, flight uid, ramp, expected on-ramp time, connecting flight number, etc. to evaluate the effectiveness of different retrieval methods, we classified the questions, and then based on these questions, we created two ground truth datasets: a straightforward dataset and a complicated, ambiguous dataset. the straightforward dataset consists of unambiguous questions that can be directly answered from flight information. examples include: "what category of aircraft is designated for flight kl1000?" and "which ramp is assigned for flight kl1000?". such questions are easily handled by retrieval methods to select the most relevant information. this dataset contains thousands of questionanswer pairs, with around 100 to 200 pairs selected for the rag methods comparison. the complicated and ambiguous dataset contains questions with variables that may be unclear or missing from the flight information which cannot be directly queried from the tabular dataset. examples are: "which flight is at gate b24?" or "which gate is assigned to the 0164 flight?", "when is delta landing?" here, 'b24' might relate to multiple flights or meanings (bus gate or ramp number), and '0164' is not a complete flight number, 'delta' also needs clarification. this dataset also contains thousands of question-answer pairs, with 185 pairs randomly selected for comparison. more information on question generation and question classification is provided in the appendix a. ## experiments to handle the flight tabular dataset, our conversational ai should understand the meaning of these flight terms, it also needs to understand specific jargon and terminology. we explore three rag methods for a conversational system on flight data. figure 1 shows the traditional rag method. when a user asks a question, various retrieval methods are employed to retrieve the correct flight data from the flight information dataset. these methods are mainly divided into three categories: keyword search, semantic search, and hybrid search. after retrieving the relevant flight information, large language models (llms) generate answers to the user's questions based on this data. several llms were tested to assess their performance, including gpt models, llama-3-8b-instruct, bert, and bert-related models. figure 2 shows the sql rag method, which begins with users asking natural language questions. an llm processes these questions using the sql database schema to generate appropriate sql queries. the queries retrieve relevant information from the sql database, which the llm then interprets and reformulates into human-readable answers. following the approach in @xcite , we experimented with code representation prompt (crp) and openai demonstration prompt (odp) to fine-tune the prompts and improve the sql rag results. more details of sql rag prompts are provided in appendix a. figure 3 shows the graph rag method, which also starts with users asking natural language questions. an llm processes these questions using the graph schema from the graph database to generate graph queries. we use neo4j's apoc plugin to extract the schema by executing 'call apoc.meta.schema() yield value return value' and include it in the prompt. and the llm interprets this data to formulate human-readable answers. the graph structure enables context-aware retrieval and reasoning, more details of graph rag prompts are provided in appendix a. the three rag methods described above can handle straightforward datasets easily because the answers all exist in the flight tabular, we will add some explanations about flight row names' meanings to the prompts, so that llms can generate better more accurate answers. however, questions about jargon and short sentences from complicated or ambiguous datasets need to be classified using a question classification prompt, as shown in figure 4 . after classification, each question is directed to different prompts to answer jargon and abbreviations. figure 2: sql rag method figure 3: graph rag method figure 4: method on ambiguous question dataset 5 results in this section, we present the experimental results, structured using our research questions. ## conclusion our evaluation of three rag methods shows that of the traditional rag methods, bm25+gpt-4 is more efficient than other methods, because of the terminology used in the airport. however, traditional rag can produce hallucinations, which poses a safety risk. sql rag and graph rag produce fewer hallucinations, and graph rag on top has higher accuracy. our overall system effectively handles specialized airport terminology through question classification and prompt engineering; specifically, we address airport jargon and abbreviations. graph rag is particularly effective in handling reasoning tasks and questions about dynamic data, making it efficient in the airport domain.