N ativ QA : Multilingual Culturally-Aligned Natural Query for LLM s (2025)

Abstract: Natural Question Answering (QA) datasets play a crucial role in evaluating the capabilities of large language models (LLMs), ensuring their effectiveness in real-world applications. Despite the numerous QA datasets that have been developed and some work done in parallel, there is a notable lack of a framework and large-scale region-specific datasets queried by native users in their own languages. This gap hinders effective benchmarking and the development of fine-tuned models for regional and cultural specificities. In this study, we propose a scalable, language-independent framework, NativQA, to seamlessly construct culturally and regionally aligned QA datasets in native languages for LLM evaluation and tuning. We demonstrate the efficacy of the proposed framework by designing a multilingual natural QA dataset, MultiNativQA, consisting of approximately ~64K manually annotated QA pairs in seven languages, ranging from high- to extremely low-resource, based on queries from native speakers from 9 regions covering 18 topics. We benchmark both open- and closed-source LLMs using the MultiNativQA dataset. The dataset and related experimental scripts are publicly available for the community at: https://huggingface.co/datasets/QCRI/MultiNativQAand https://gitlab.com/nativqa/multinativqa.

## introduction recent advancements in llms have revolutionized the landscape of artificial intelligence, significantly pushing the state-of-the-art for a broad array of natural language processing (nlp) and speech there are limited multilingual region-specific cultural benchmarks designed to evaluate the llms' performance across different cultures and languages. as a result, multilingual and non-english llms have been evaluated by using mt, lang q/a example (native) english translation arabic q ‫ﺽ؟‬ ‫ﻭ‬ ‫ﻝ‬ what is the area of qatar length and width? a . ‫ﺍ‬ 200 ‫ﻝ‬ ‫ﻭ‬ 100 ‫ﺍ‬ ‫ﺽ‬ its area is about 100 km in width and extends 200 km in the gulf. assamese q েকান জন িবখয্াত ৰাজৈনিতক বয্ি েয় েশহতীয়াৈক অসমত িবেজিপৰ পৰা কংে ছৈল েযাগদান কিৰিছল? which famous political person recently joined from bjp to congress in assam? a আিমনু ল হক ল েৰ েশহতীয়াৈক অসমত িবেজিপৰ পৰা কংে ছত েযাগদান কিৰিছল। aminul haque laskar recently joined congress from bjp in assam. bangla q েশালািকয়া মােঠর আয়তন কত? what is the area of sholakia field? a বতর্ মান েশালািকয়া ঈদগাহ মােঠর আয়তন ৭ একর। the current area of sholakia eidgah field is 7 acres. english q does udst offer scholarships? na a public schools in qatar receive government funding and provide free tuition to all citizens. na hindi q नवराित्र में कलश रखने का शु भ मु हू तर् क्या है ? what is the auspicious time to keep kalash in navratri? a कलश क स्थापना चै त्र शु क्ल पक्ष क प्र तपदा त थ को क जाती है . इस बार चै त्र नवराित्र क घटस्थापना का सबसे अच्छा मु हू तर् सु बह 6 बजकर 2 िमनट ले कर सु बह 10 बजकर 15 िमनट तक है | the kalash is established on the pratipada date of chaitra shukla paksha. this time the best time for chaitra navratri is from 6.02 am to 10.15 am. nepali q ने पालको सबै भन्दा ठू लो ताल कु न हो which is the biggest lake in nepal? a ने पालको सबै भन्दा ठू लो ताल कणार् ली प्रदे शको रारा ताल हो। the largest lake in nepal is rara lake in karnali province. turkish q istanbul'da göl var mı? is there any lake in istanbul? a i̇stanbul'da dört doğal göl bulunmaktadır. bunların yanı sıra, baraj gölleri de vardır. there are four natural lakes in istanbul. in addition, there are also reservoir lakes. figure 2 : examples of questions and answers in different languages with their translation from our dataset. with or without human involvement, to translate the existing english datasets into corresponding languages @xcite . however, translation often misses the cultural and regional nuances of target languages, making humanannotated datasets a better alternative. in a recent study, @xcite developed 1.5k culturespecific qas by gathering questions from community web forums and employing native speakers to manually write questions. similarly, myung et al. (2024) produced 52.5k multiple-choice and shortanswer questions, with both question collection and answer writing being fully manual. within 5 mi open now top rated endangered animals wild life in qatar 6/15/24, 9:28 pm wild life in qatar -google search in this study, we propose a framework, native qa (nativqa), specifically designed to seamlessly develop regionally-and culturally-specific qa datasets following a human-machine collaborative approach. datasets developed through nativqa serve two primary functions: (i) evaluating the llm performance over real users' information needs and interests expressed in their native languages, and (ii) facilitating fine-tuning of llms to adapt to cultural contexts. moreover, to show the efficacy of the nativqa framework, we developed a natural multilingual native question-answering (qa) dataset, multinativqa, including ∼ 64k qa pairs in seven extremely low to high resource languages (see in figure 1 ), covering 18 different topics from nine different regions (see examples in figure 2 ). we further demonstrate the usefulness of multinativqa dataset by fine-tuning llama-3.1. • we benchmark over multinativqa with 2 open and 2 closed llms. in addition, we report experimental results of a fine-tuned llama-3.1 model across all languages. a summary of our findings is as follows: gap -high vs. low resources languages. we observed the highest performance for english and lowest for assamese on average across models, which clearly indicates that the performance correlates to the representation and/or richness of digital content of the language used in the models. this finding corroborates the findings reported in several parallel works @xcite . gap in close vs. open models. close models outperforms open models. gpt-4o (bleu: 0.230) and gemini (bleu: 0.226) perform similarly among closed models. among open models, llama-3.1 (bleu: 0.186) outperforms mistral (bleu: 0.162). capability enhancement with fine-tuning. finetuning (i) improves performance for extremely low resource languages such as assamese and nepali, (ii) for medium resource languages, it helps dialect-rich languages like arabic compared to other medium resource-languages (e.g., hindi). cultural benchmarking. our findings emphasize the importance of well-crafted benchmarks efforts for studying regional/cultural awareness in llms. the results support the hypothesis that under-represented regions, and dialectal-rich languages (e.g., arabic) benefit more from incorporating native and culturally aware information in the llm. this highlights the value of the proposed language-independent framework nativqa, which efficiently creates multilingual, region-and culturespecific resources with minimal human effort. ## related work llms have demonstrated remarkable capabilities across various disciplines and tasks, leading to efforts to evaluate their performance on standard nlp tasks @xcite @xcite . while several initiatives have developed resources to benchmark llms, most focus primarily on english. for other languages, evaluations often rely on translated data @xcite @xcite . existing qa datasets. question answering has been a standard nlp task for decades, pushing the development of many qa datasets in different languages. @xcite and @xcite proposed two extractive qa datasets including natural questions (nq), both containing longform large-scale question-answer pairs. @xcite developed triviaqa dataset, which consists of 650k question-answer-evidence triples. these triples are created by merging 95k question-answer pairs. @xcite developed squad, which is a collection of 100k crowdsourced qa's paired with shortened wikipedia articles. help-steer @xcite is another qa dataset, which comprises a 37k sample dataset with multiple attributes of helpfulness preference. the most closest work in the literature to ours is blend @xcite which is a hand-crafted benchmark consisting of 52.6k multiple choice and shortanswer qa pairs for 13 different languages in total, focusing cultural aspects of languages. evaluations of llms for qa. for llm evaluation, there are notable datasets covering world knowledge @xcite , commonsense reasoning @xcite , reading comprehension @xcite , factuality @xcite , and others. these datasets are usually transformed into multiple-choice questions. in addition, standard qa datasets have also been used for llm evaluation @xcite . @xcite performed the analysis of different open-domain qa models, including llms by manually judging answers on a benchmark dataset of nq-open @xcite . their investigation shows that llms attain state-of-the-art performance but fail in lexical matching when candidate answers become longer. in ## nativqa framework figure 4 presents the nativqa framework with three inter-connected modules described below. ## multinativqa dataset we demonstrate the effectiveness and scalability of the nativqa framework by creating a largescale, multilingual multinativqa dataset. the multinativqa dataset spans over seven languages -from high-to extremely low-resource and nine different location/cities. our choice of languages for multinativqa was guided primarily by the authors' native proficiency, which allowed for more accurate annotation and evaluation. multinativqa captures linguistic diversity, by including several dialects for dialect-rich languages like arabic. we also added two linguistic variations of bangla to reflect differences between speakers in bangladesh and west bengal, india. furthermore, we included english queries from dhaka and doha, where english is often used as a second language. qa collection using qac module we enriched queries and qa pairs for each language and its respective city. we ran our collection algorithm for 3-7 n iter per region based on the convergence rate. we collected ∼ 154k qa pairs across all languages (see ## conclusions in this paper, we propose the nativqa framework, to enable constructing culturally and regionallyaligned natural qa datasets with minimal humaneffort. the proposed framework is scalable and language-independent, which not just facilitate creating region-and culture-based benchmarking efforts, but also resources that can be used in continual learning or fine-tuning the llms. we show the efficacy of the nativqa, by designing and developing a multilingual native qa dataset, multinativqa -from 9 regions (7 languages) encapsulating the scenario of high-low resource representation. we benchmark the multinativqa with 2 open and 2 closed llms. our results indicate the superiority of closed models over open llms, and the performance gaps between high-and low-resource languages. by utilizing the multinativqa dataset for fine-tuning, we can potentially inject cultural and regional knowledge into the llms, as evidenced by the improved performance of arabic, a midresource language, and assamese, an extremely low-resource language. our future work includes extending the nativqa framework with additional search engine capabilities, image and video search options, and releasing it to the community for seamless use in research @xcite .