ALOHA : Empowering Multilingual Agent for University Orientation with Hierarchical Retrieval (2025)

Abstract: The rise of Large Language Models (LLMs) revolutionizes information retrieval, allowing users to obtain required answers through complex instructions within conversations. However, publicly available services remain inadequate in addressing the needs of faculty and students to search campus-specific information. It is primarily due to the LLM’s lack of domain-specific knowledge and the limitation of search engines in supporting multilingual and timely scenarios. To tackle these challenges, we introduce ALOHA, a multilingual agent enhanced by hierarchical retrieval for university orientation. We also integrate external APIs into the front-end interface to provide interactive service. The human evaluation and case study show our proposed system has strong capabilities to yield correct, timely, and user-friendly responses to the queries in multiple languages, surpassing commercial chatbots and search engines. The system has been deployed and has provided service for more than 12,000 people.

## introduction the advancement of large language models (llms) greatly changes the methods of information retrieval. by leveraging the powerful reasoning and generative capabilities of llms @xcite , users can input complex and descriptive instructions to obtain required information from conversations, which results in llms replacing traditional search engines in consultation scenarios. despite the impressive performance of llms, they can still make mistakes due to their lack of domain-specific knowledge. previous works adopt retrieval-augmented generation (rag) to refrain from hallucinated responses in the domain of medicine @xcite and law (hu et al., * equal contribution. 2024). commercial service such as gemini (gemini team, 2024) also enhances llms with web search to provide reliable outputs in general domain. however, these solutions still cannot address the challenges faced in developing consulting agent for university campus. as a large community engaged in education, research, and daily life, the university can encompass a wide variety of information sources. faculty and students, especially the freshmen, may fail to determine where to search for information, highlighting the urgent need of consulting agent for campus. the agent should be capable to select evidence from the documents with diverse formats, which is different from retrieving medical guidelines or law articles. furthermore, the information on campus can be frequently updated, and users have high demands of timeliness. however, search engines often fail to provide the latest web pages. additionally, we hope the agent can be especially beneficial for the orientation of new students. the agent needs the capabilities to serve people who speak distinct languages, while search engines also demonstrate poor performance to provide cross-lingual results. as an application used daily by tens of thousands of people, we also need to ensure the agent supports human-centric interaction. to address the issues mentioned above, we propose aloha, an automatic multi-lingual agent for university orientation with hierarchical retrieval. we collect a vast number of campus-specific documents and divide them into distinct subsets according to their structure and granularity. (1) to enhance the correctness of agent response, we employ hierarchical retrieval @xcite ) to find precise evidence. (2) we also prompt the llm to be aware of the timestamp of each evidence for timeliness. (3) we introduce language identification @xcite and 2 overview of aloha our system, aloha, provides interactive information consultation services for users across multiple languages. we divide the pipeline of aloha into three stages: preliminary analysis of user queries, hierarchical retrieval, and postprocessing of responses. we illustrate the workflow of our system in figure 1 . more demonstrations of the interface are shown in appendix a. preliminary analysis of query aloha can provide service to students and staff who communicate in distinct languages. therefore, we first employ fasttext @xcite to identify the language of the query. we then translate the query into chinese if it is in other languages, which can enhance the recall of retrieval, since almost all our documents are in chinese. aloha also conducts a pre-analysis of user intent. we find that the queries regarding schedules and financial standards typically require retrieval of semi-structured documents (i.e., containing tabular data) for accurate information. in contrast, other queries may need merely plain-text evidence. thus, we finetune qwen2-1.5b-instruct @xcite for user intent classification. hierarchical retrieval to mitigate the lack of campus-specific knowledge in llms, we con-struct a collection of semi-structured and unstructured documents for retrieval-augmented generation. aloha can select evidence from specific subsets of documents based on the results of intent classification. we further divide the unstructured documents into three categories: concepts, question-answer pairs, and filtered web pages. our system conducts retrieval sequentially across these categories. since inaccurate documents may lead to hallucinations, if the system acquires evidence from a certain category, it will skip the subsequent pipelines of retrieval. ## evaluation in this section, we conduct an automatic evaluation to examine the effectiveness of our heuristic intent classification. we also use human evaluation to examine the response quality of aloha. ## case study we conduct case study to investigate aloha's performance in several key areas, including multilingual support, user-friendliness, capability to handle complex information, and timeliness. oùse trouve la cantine xinyuan? la cantine xinyuan est situé e dans le nouveau campus de changping. ce restaurant s'é tend sur deux é tages. au premier é tage, vous trouverez une varié té de plats en libre-service avec diffé rentes saveurs. au deuxiè me é tage, il y a un comptoir dé diéàla cuisine ethnique. l'environnement de la cantine est é lé gant et confortable, et les plats sont ré guliè rement mis àjour. vous pourriez é galement dé couvrir des surprises agré ables avec les pâ tisseries et plats faits maison. navigation: 馨园食堂 reference: 昌平新校区生活指北--馨园食堂 user chatbot figure 2: a case of location inquiry in french. see english translation in appendix c. ## deployment based on the aloha system, we develop the oncampus ai assistant xiaobei for peking university. this application has been integrated into the portal of peking university and has provided consultation service around 33,000 times for 12,000 faculty, staff, and students, including more than 3,000 new undergraduate students. ## conclusion in this work, we propose aloha, enhancing llm agent with hierarchical retrieval to select evidence from extensive document with diverse structures and granularities. our system performs better than existing chatbots or search engines to generate correct, timely and user-friendly responses in various languages. aloha facilitates campusspecific information retrieval and greatly benefits the orientation of new students. we hope aloha can inspire the development of more agents serving for education and public affairs.