Improving V ietnamese- E nglish Cross-Lingual Retrieval for Legal and General Domains (2025)

Abstract: Document retrieval plays a crucial role in numerous question-answering systems, yet research has concentrated on the general knowledge domain and resource-rich languages like English. In contrast, it remains largely underexplored in low-resource languages and cross-lingual scenarios within specialized domain knowledge such as legal. We present a novel dataset designed for cross-lingual retrieval between Vietnamese and English, which not only covers the general domain but also extends to the legal field. Additionally, we propose auxiliary loss function and symmetrical training strategy that significantly enhance the performance of state-of-the-art models on these retrieval tasks. Our contributions offer a significant resource and methodology aimed at improving cross-lingual retrieval in both legal and general QA settings, facilitating further advancements in document retrieval research across multiple languages and a broader spectrum of specialized domains. All the resources related to our work can be accessed at huggingface.co/datasets/bkai-foundation-models/crosslingual .

## introduction document retrieval systems play a crucial role in question-answering (qa) frameworks by identifying relevant documents that provide the necessary information to answer a given query. however, the majority of existing document retrieval systems @xcite @xcite @xcite and datasets @xcite @xcite @xcite are designed to operate within a single language, typically targeting resource-rich languages like english or chinese. this monolingual focus limits the effectiveness of these systems in multilingual contexts, where users may pose queries in one language while the relevant documents are in another. some studies have tried to explore cross-lingual information retrieval @xcite @xcite , yet these efforts have largely concentrated on highresource languages and general domain knowledge, leveraging extensive resources and pre-existing knowledge bases. meanwhile, vietnamese remains largely underexplored in this context, primarily due to the limited availability of datasets necessary for pretraining and fine-tuning representation models in this language. for example, vietnamese accounts for less than 1% of the total pretraining data in the bge m3 model @xcite . additionally, general domain datasets @xcite @xcite are frequently derived from open-domain sources such as wikipedia, web documents, or news articles, that typically involve quite short documents. while this is valuable for general qa, it fails to address the complexities of specialized domains, where documents are often lengthy and domain-specific, such as legal documentation. consequently, there is a need to develop cross-lingual document retrieval systems that can effectively handle low-resource languages and specialized domains, ensuring more comprehensive and contextaware qa solutions. to address these gaps, we present a novel benchmark aimed at evaluating cross-lingual information retrieval (clir) between vietnamese and english. in addition to general knowledge question answering, our dataset enables the investigation of retrieval systems within the legal domain using the vietnamese law library. from this resource, we develop a retrieval model that demonstrates strong performance across both general knowledge and legal domains. in summary, our contributions are as follows: ## related work recently, information retrieval has attracted considerable attention, with document retrieval emerging as one of the central focuses. several methods have been proposed to address this task, which can generally be classified into three approaches: dense retrieval @xcite @xcite , lexical retrieval (dai and callan, 2020; gao et al., 2021), and multi-vector retrieval @xcite . numerous datasets have also been developed to evaluate these systems @xcite @xcite . however, these methods and datasets primarily focus on monolingual scenarios. recently, several studies have explored cross-lingual settings, where queries and documents are in different languages @xcite @xcite . in contrast, some prior studies have investigated specific domains, such as the legal, extending beyond general knowledge qa, but still focusing on monolingual scenarios @xcite @xcite @xcite . ## experiments and results experiment setup: we trained the models for 4 epochs using the adamw optimizer (loshchilov and hutter, 2019) with a base learning rate of 2 × 10 -5 . a cosine learning rate scheduler with the warm-up ratio set to 5% of the total training steps was applied. the temperature τ was set to 0.05. besides, we employed smart batching @xcite to group samples with similar sequence lengths. for symmetrical training, the sampling rate p sym was set to 0.3. the trained models are subsequently evaluated on the evaluation sets from vnlawqc and zalolegal2021 for the legal domain, as well as zalowikipediaqa for the general knowledge domain. we conduct evaluations using various training datasets, retrieval modes, and loss functions. evaluation metric: following prior work in document retrieval @xcite @xcite ; khattab and zaharia, 2020), we leverage four metrics recall@k, mrr@k, map@k and ndcg@k for evaluation. specifically, we use @xmath0. we observe that the average scores exhibit a strong correlation with individual metrics, making them a suitable representation of overall performance. detailed results for each specific metric are provided in tables 6 and 7 . experimental results: while models with symmetrical training demonstrate significant performance in legal retrieval, models trained with auxiliary loss achieve the highest performance in the general knowledge domain. these results align with our motivation for employing auxiliary loss, as queries in the general domain tend to be short and ambiguous. ## conclusion in summary, we introduce a novel dataset for crosslingual information retrieval (clir) between vietnamese and english, covering both general knowledge and the legal domain. additionally, we develop a clir model by finetuning cross-lingual and synthetic data while proposing an auxiliary loss function and training strategy to enhance performance. our contributions provide valuable resources and methods for advancing cross-lingual retrieval in specialized fields.