Empowering Air Travelers: A Chatbot for C anadian Air Passenger Rights (2024)

Abstract: The Canadian air travel sector has seen a significant increase in flight delays, cancellations, and other issues concerning passenger rights. Recognizing this demand, we present a chatbot to assist passengers and educate them about their rights. Our system breaks a complex user input into simple queries which are used to retrieve information from a collection of documents detailing air travel regulations. The most relevant passages from these documents are presented along with links to the original documents and the generated queries, enabling users to dissect and leverage the information for their unique circumstances. The system successfully overcomes two predominant challenges: understanding complex user inputs, and delivering accurate answers, free of hallucinations, that passengers can rely on for making informed decisions. A user study comparing the chatbot to a Google search demonstrated the chatbot’s usefulness and ease of use. Beyond the primary goal of providing accurate and timely information to air passengers regarding their rights, we hope that this system will also enable further research exploring the tradeoff between the user-friendly conversational interface of chatbots and the accuracy of retrieval systems.

## introduction air travel in canada has seen many challenges when it comes to passenger rights. canada's deficient regulations lag behind the standards adopted by other western countries such as members of the european union @xcite . canada also lacks meaningful enforcement of passengers' existing rights by the federal regulator, whose cozy relationship with the airline industry and impartiality has been questioned by a parliamentary committee @xcite and by the 1 the code is available at https://github.com/mak sym-taranukhin/apr_chatbot judiciary @xcite . this situation has led to a high number of questions from passengers trying to understand their rights and find solutions. a group of dedicated volunteers foot_0 is handling these questions, providing information on the rights and options available to affected passengers. however, the growing number of inquiries calls for a more efficient, automated solution to ensure quick and accurate responses. to address this issue, we propose a chatbot (figure 1 ) that can adeptly understand narratives detailing air travel concerns and extract pertinent information from relevant sources. our goal is to streamline the process of informing and educating passengers about their rights and options, ultimately empowering them to make informed decisions. this will reduce the workload of the human volunteers, allowing them to focus on the more complex questions from users. crucially, this application has a very low tolerance for errors and hallucinations, which may cost passengers time and money. tellingly, in a recent incident, a chatbot developed by air canada provided incorrect information, leading to negative consequences for both the passenger and the airline. 3 our chatbot is designed to mitigate such risks via retrieval from a reliable collection of documents. our approach simplifies complex questions, ensures systematic coverage of different aspects, and enhances search efficiency. furthermore, to prevent hallucinations, we do not generate a response to the user based on the extracted information, as in the traditional rag approach. instead, we present the generated queries and the relevant passages from the source documents to the user. this method allows users to directly view the authoritative information that is relevant to their input, which they can then apply to their specific circumstances. we conducted an extensive user study to evaluate the chatbot's performance across several dimensions: usefulness, user satisfaction, ease of use, and ease of learning. the results indicated that the chatbot was highly effective at providing pertinent information quickly and efficiently. the participants also reported that the chatbot's interface was more convenient than a manual google search. also, we compared our system to a standard rag-based system and found that the latter had a hallucination rate of 27.5%, which exceeds the acceptable threshold. in contrast, our chatbot produced zero hallucinations, highlighting its reliability in delivering accurate information. in terms of the application itself, the proposed chatbot is first a prototype. given the users' preference for the chatbot over a google search, we are encouraged to develop future versions of the chatbot that are more conversational and that further contextualize the answers, while maintaining a strict zero-hallucination policy. the importance of this research goes beyond assisting air passengers; it introduces a way of using recent advances in nlp to provide legal information with greater accessibility and accuracy, especially in areas with complex regulations such as law and medicine. 4 2 chatbot architecture our chatbot architecture, which is depicted in figure 2 , is composed of 2 main components. the query understanding component (sec 2.1) is responsible for interpreting the user input and generating a series of simpler queries. these queries go into the document retrieval component (sec 2.2) which is tasked with extracting relevant information from the knowledge base. finally, the extracted information is formatted and the answer is presented to the user (sec 2.3). ## related work in recent years, research has focused on ai systems designed to aid individuals, especially those without legal expertise, in navigating complex legal procedures, bridging the gap between legal information and laypeople. the proposed chatbot operates within the domain of document-grounded dialogue systems (dgds) that enable more trustworthy and informed user interactions. in this section, we overview the evolution of access to justice tools alongside the datasets and methods relevant to dgds. ## conclusion we developed a chatbot that provides accurate and timely information about canadian air travel regulations and passenger rights, supporting the manual process currently handled by a group of volunteers. the chatbot utilizes retrieval augmented generation and in-context learning to interpret complex user inputs and extract relevant information from a comprehensive knowledge base. instead of generating a synthesized response, it provides users with a direct presentation of the formulated queries and corresponding passages from the source documents, reducing the risk of hallucination. a user study comparing the chatbot to a google search demonstrated its ability to accurately interpret and respond to user queries and successfully inform passengers of their rights. in future work, we plan to improve the chatbot's usefulness by contextualizing the answer for the user query, and reasoning over multiple extracted passages to synthesize an answer. we will explore how to achieve these properties without compromising the answers' accuracy.