Ancient Wisdom, Modern Tools: Exploring Retrieval-Augmented LLM s for A ncient I ndian Philosophy (2024)

Abstract: LLMs have revolutionized the landscape of information retrieval and knowledge dissemination. However, their application in specialized areas is often hindered by limitations such as factual inaccuracies and hallucinations, especially in long-tail knowledge distributions. In this work, we explore the potential of retrieval-augmented generation (RAG) models in performing long-form question answering (LFQA) on a specially curated niche and custom knowledge domain. We present VedantaNY-10M, a dataset curated from extensive public discourses on the ancient Indian philosophy of Advaita Vedanta. We develop and benchmark a RAG model against a standard, non-RAG LLM, focusing on transcription, retrieval, and generation performance. A human evaluation involving computational linguists and domain experts, shows that the RAG model significantly outperforms the standard model in producing factual, comprehensive responses having fewer hallucinations. In addition, we find that a keyword-based hybrid retriever that focuses on unique low-frequency words further improves results. Our study provides insights into the future development of real-world RAG models for custom and niche areas of knowledge.

## introduction generic llms have proven to be highly effective for broad knowledge domains. however, they often encounter challenges in niche and less popular areas, suffering from issues such as factual inaccuracies and hallucinations for long-tail knowledge distributions @xcite . moreover, the inability to verify responses against authentic sources is particularly problematic in these long-tail domains, where llms can generate highly inaccurate answers with unwarranted confidence @xcite . in response to these limitations, there has been a growing interest in retrieval-augmented generation (rag) models @xcite @xcite @xcite . these models, which integrate external datastores to retrieve relevant knowledge and incorporate it into llms, have demonstrated higher factual accuracy and reduced hallucinations compared to conventional llms @xcite @xcite . additionally, updating these external datastores with new information is more efficient and cost-effective than retraining llms. recent studies on the societal impact of llms @xcite have highlighted the increasing significance of llms as cultural technologies akin to libraries for search and retrieval. analogous to earlier technologies like writing, print, libraries and internet search, the power of llms can be harnessed meaningfully to preserve and disseminate human knowledge @xcite . in this vein, we argue that rag models show immense potential for supplementing study in diverse knowledge domains. hence, there is a growing need to examine the applications of rag models for unconventional, custom knowledge domains that are often niche and scarcely represented in pre-training data. the capability of rag models to provide verified, authentic sources when answering questions is particularly advantageous for end-users. in this work, we develop and evaluate a ragbased language model specialized in the ancient indian philosophy of advaita vedanta (upanishads, >3000 b.c.e.; bhagavad gita, 3000 b.c.e.; shankaracharya, 450 b.c.e.). to ensure that the llm has previously not been exposed to the source material, we construct vedantany-10m, a custom philosophy datastore comprising transcripts of over 750 hours of public discourses on youtube from vedanta society of new york. we evaluate standard non-rag and rag models on this domain, and find that rag models perform significantly better. however, they suffer from a number of issues, including irrelevant retrievals, sub-optimal retrieval passage length, retrieval-induced hallucinations, and reliance on outside knowledge. in early attempts to mitigate some of these issues, we find that traditional sparse retrievers have a unique advantage over dense retrievers in niche domains having specific terminology-sanskrit terms in our case. hence, we propose a keyword-based hybrid retriever that effectively combines sparse and dense embeddings to upsample low-frequency or domainspecific terms. in addition, a simple keyword-based retrieval refinement serves to shorten or lengthen retrievals to further refine context. we conduct an extensive evaluation comprising both automatic metrics and human evaluation with 5 computational linguists and 3 domain experts, assessing the models along three dimensions: transcription, retrieval, and generation. our findings are twofold. first, the standard rag model significantly outperforms the generic non-rag model along all axes, offering more factual, comprehensive, and specific responses while minimizing hallucinations. user preference for the rag model over the generic counterpart is evident, with a preference rate of 81%. second, the keyword-based hybrid rag model further outperforms the standard deep-embedding based rag model in both automatic and human evaluation metrics. our study also includes detailed long-form responses from the evaluators, with domain experts specifically indicating the likelihood of using such llms to supplement their day-to-day study. our work offers a step toward building and evaluating real-world rag models for niche and esoteric ancient knowledge domains, highlighting the opportunities and challenges arising thereof. ## related work language models for ancient texts @xcite recently conducted a thorough survey of machine learning techniques applied to the study and restoration of ancient texts. spanning digitization @xcite , restoration, @xcite @xcite and representation learning @xcite , a wide range of use cases have benefitted from the application of machine learning techniques to study ancient texts. recently, @xcite released a digital corpus of romanized buddhist sanskrit texts, training and evaluating embedding models such as bert and gpt-2 on them. however, the use of llms as a question-answering tool to enhance understanding of ancient esoteric knowledge systems has not yet been systematically studied. to the best of our knowledge, ours is the first work that studies the effects of rag-based models in the niche knowledge domain of ancient indian philosophy. ## the vedantany-10m dataset we first describe our niche domain dataset creation process. the custom dataset for our study needs to satisfy the following requirements: (1) niche: must be a specialized niche knowledge domain within the llm's long-tail distribution. (2) novel: the llm must not have previously encountered the source material. (3) authentic: the dataset should be authentic and representative of the knowledge domain. (4) domain experts: should be available to evaluate the model's effectiveness and utility. ## in-context rag for niche domains we now discuss the methodology adopted to build an in-context retrieval augmented chatbot from the custom dataset described above. we first define a generic chatbot c g that does not use retrieval as follows: c g : @xmath0. now, let d t represent the textual data corpus from our knowledge domain and r be the retriever. our goal is to build a retrieval-augmented generation chatbot c r : q × r(d t , q) → a r that will generate answer a r for the query by retrieving relevant context from d t using r. an overview of our approach is illustrated in fig. 2 . we first build d t from 765 hours of public discourses on advaita vedanta introduced in sec. 3. when deployed, the system processes q by first using retriever r to identify the top-k most relevant passages p from d t using a similarity metric. subsequently, a large language model (llm) is prompted with both the query and the retrieved passages in-context, following @xcite , to generate a contextually relevant response. we now describe each of the components in detail. we follow a four-stage process as follows: transcription. we first need to create a dense textual corpus targeted at our niche domain. since our dataset consists of youtube videos, we first employ a transcription model to transcribe the audio into text. our video corpus d v consists of 612 videos totaling 765 hours of content, with an average length of 1.25 hours per video. we extract audio content from d v and transcribe it using openai's whisper large-v2 model @xcite . this step converts the spoken discourses into a transcribed textual corpus d t consisting of 10m tokens in total. since whisper is a multilingual model, it has the capacity to support the dual-language nature of our dataset. we evaluate the transcription quality of whisper in sec. c.1. ## evaluation we now evaluate the model along two axes: automatic evaluation metrics and a human evaluation survey. to ensure a broad and comprehensive evaluation, we categorize the questions into five distinct types, each designed to test different aspects of the model's capabilities: ## challenges the evaluation in sec. 5 shows that the rag model provides responses that are not only more aligned with the source material but are also more comprehensive, specific, and user-friendly compared to the responses generated by the generic language model. in this section, we discuss the challenges we encountered while building the retrieval-augmented chatbot for the niche knowledge domain of ancient indian philosophy introduced in this work. transcription. our requirement of using a niche data domain having long-tail knowledge precludes the use of source material that the llm has previously been exposed to. to ensure this, we construct a textual corpus that is derived from automated transcripts of youtube discourses. these transcripts can sometimes contain errors such as missing punctuations, incorrect transcriptions, and transliterations of sanskrit terms. a sample of such errors is shown in appendix tab. 3 . a proofreading mechanism and/or improved transcription models can help alleviate these issues to a large extent. spoken vs written language. unlike traditional textual corpora that are compiled from written sources, our dataset is derived from spoken discourses. spoken language is often more verbose and less structured than written text, with the speaker frequently jumping between concepts midsentence. this unstructured nature of the text can be unfamiliar for a language model trained extensively on written text, which expects a more coherent and structured input. a peculiar failure case arising from this issue is shown in appendix fig. 5 . this can be addressed by converting the spoken text into a more structured prose format with the help of well-crafted prompts to llms, followed by human proofreading. context length. the passages retrieved in the standard model are of a fixed length and can sometimes be too short for many queries, especially for long-form answering. as an example, the retrieved passage may include a snippet from the middle of the full context. as a result, the chatbot response may be incomplete or incoherent (fig. 10 ). this motivated us to employ a keyword-based contextexpansion mechanism to provide a more comprehensive context. while this results in much better answer generation, the retrieved passage may contain too much information, making it difficult for the generator to reason effectively. moreover, the increase in the number of tokens increases processing time. future work can explore more advanced retrieval models capable of processing longer contexts and summarizing them effectively before input to the llm. ## conclusion in this work, we integrate modern retrievalaugmented large language models with the ancient indian philosophy of advaita vedanta. toward this end, we present vedantany-10m, a large dataset curated from automatic transcriptions of extensive philosophical discourses on youtube. validating these models along various axes using both automatic and human evaluation provides two key insights. first, rag models significantly outperform non-rag models, with domain experts expressing a strong preference for using such rag models to supplement their day-to-day study. second, the keyword-based hybrid rag model underscores the merits of integrating classical and contemporary deep learning techniques for retrieval in niche and specialized domains. while there is much work to be done, our study underscores the potential of integrating modern machine learning techniques to unravel ancient knowledge systems.