Building an Efficient Multilingual Non-Profit IR System for the Islamic Domain Leveraging Multiprocessing Design in Rust (2024)

Abstract: The widespread use of large language models (LLMs) has dramatically improved many applications of Natural Language Processing (NLP), including Information Retrieval (IR). However, domains that are not driven by commercial interest often lag behind in benefiting from AI-powered solutions. One such area is religious and heritage corpora. Alongside similar domains, Islamic literature holds significant cultural value and is regularly utilized by scholars and the general public. Navigating this extensive amount of text is challenging, and there is currently no unified resource that allows for easy searching of this data using advanced AI tools. This work focuses on the development of a multilingual non-profit IR system for the Islamic domain. This process brings a few major challenges, such as preparing multilingual domain-specific corpora when data is limited in certain languages, deploying a model on resource-constrained devices, and enabling fast search on a limited budget. By employing methods like continued pre-training for domain adaptation and language reduction to decrease model size, a lightweight multilingual retrieval model was prepared, demonstrating superior performance compared to larger models pre-trained on general domain data. Furthermore, evaluating the proposed architecture that utilizes Rust Language capabilities shows the possibility of implementing efficient semantic search in a low-resource setting.

## introduction dense retrieval is an advanced approach in ir that utilizes embeddings to identify semantically similar text, known as semantic search. llms are a key component in creating text embeddings and performing dense retrieval @xcite . one of the first challenges in building a non-profit multilingual domainspecific ir system is that the use of publicly available multilingual large language models (mllms) pre-trained on a general domain could deteriorate performance due to domain shift when applied to new domains @xcite . to overcome this, we begin with pre-training an mllm for the islamic domain to address this issue. however, pre-training a domain-specific mllm brings two additional challenges. firstly, assembling a multilingual domain-specific corpus for pre-training a mllm requires a large amount of domain-specific data that is often difficult to find in different languages. secondly, multilingual models are heavyweight, frequently exceeding 1gb, making them challenging to deploy. to effectively tackle the issue of pre-training domain-specific mllm, we employ a continued pre-training approach and incorporate domain-specific vocabulary to accommodate the domain shift better @xcite . to deal with the challenge of the large size of mllm, we perform language reduction and remove languages not needed in the current deployment. this method helps us reduce the model's size by more than half, even after introducing new domain-specific vocabulary. we use this lightweight domain-specific mllm as a backbone for the retrieval. evaluation of this model on an in-domain ir dataset found that our model significantly outperforms general-domain multilingual and monolingual models even after performing language reduction. moreover, deploying non-profit ai systems implies operating on a limited budget, which makes it challenging to use embedding apis or libraries that rely on gpu acceleration to perform search reasonably fast. to tackle this challenge and meet the requirements of implementing an ad hoc ir system on a public website, we utilize the multiprocessing capabilities of rust language to create an efficient and secure semantic search based on cpu architecture @xcite @xcite . our system's evaluation and comparison against others, such as faiss, indicates that our implementation of semantic search with underlying rust multiprocessing architecture can significantly accelerate search without compromising performance. our main contributions are: ## domain-specific ir to prepare the retrieval model, we utilize a dense retrieval approach @xcite ) that employs dual-encoder architecture @xcite . we use the sentence transformer framework that adds a pooling layer on top of llm embeddings and produces fixed-sized sentence embedding @xcite . the loss function is formulated in the framework of contrastive learning that enables learning an embedding space that brings closer queries and their relevant passages and pushes further queries and irrelevant passages (van den @xcite . for efficient training, we use in-batch negatives @xcite @xcite . the transfer language of the xlm-r base is english, while xlm-r4-id was adapted for the islamic domain, predominately using arabic. we experiment with both english and arabic as transfer languages to compare their transfer potential for solving the ir task at hand. we utilize the ms marco ir dataset, which contains more than half a million queries and a collection of 8.8m passages in english @xcite metric mrr@10 (ms marco's official metric). in ## deploying domain-specific ir system using gpus to train transformer-based llms and retrieval models is often a necessity. however, gpus for inference in a production environment are cost-prohibitive, especially in non-profit organizations. additionally, given supply availability to ensure the right size of cloud machines with gpus often imposes a fixed set of resources in predefined bundles of size, which typically leads to vast overprovisioning and grossly underutilized resources. our goal is to maximize software performance and resource efficiency on widely-used, cost-effective cpu servers. we argue that leveraging the ubiquity and flexibility of cpu servers makes it possible to build a system and improve efficiency independently of the underlying substrate, allowing deployment even on serverless infrastructure, which is predominately cpu-based. ## related work there is a substantial amount of work written on the topic of pre-training domain-specific llm; some of them describe more costly approaches like pre-training a new llm from scratch @xcite al. (2020) 2022 ) covers in detail the topic of dense retrieval, discussing different types of models' architecture and training approaches, including the selection of high-quality negatives. there is a growing body of research on rust language memory-safe features that came to be known as fearless concurrency @xcite @xcite . ## conclusion this work outlines the development of a non-profit multilingual ir system for the islamic domain. we also address the challenges it presents and propose potential solutions for handling these challenges in low-resource settings. our research demonstrates that utilizing continued pre-training and integrating new domain-specific vocabulary can help mitigate domain shift, even when pre-training on a small corpus. the retrieval model we built using a domain-adapted mllm as a foundation exhibited better performance compared to general domain models. additionally, we found that implementing language reduction can significantly decrease the model size without deteriorating performance. furthermore, we showed that leveraging the multiprocessing capabilities of the rust language can decrease inference time without compromising performance or requiring expensive acceleration hardware like gpus.