BE n QA : A Question Answering Benchmark for B engali and E nglish (2024)

Abstract: In this study, we introduce BEnQA, a dataset comprising parallel Bengali and English exam questions for middle and high school levels in Bangladesh. Our dataset consists of approximately 5K questions covering several subjects in science with different types of questions, including factual, application, and reasoning-based questions. We benchmark several Large Language Models (LLMs) with our parallel dataset and observe a notable performance disparity between the models in Bengali and English. We also investigate some prompting methods, and find that Chain-of-Thought prompting is beneficial mostly on reasoning questions, but not so much on factual ones. We also find that appending English translation helps to answer questions in Bengali. Our findings point to promising future research directions for improving the performance of LLMs in Bengali and more generally in low-resource languages.

## introduction large language models (llms) like gpt-4 have shown impressive performance in many complex natural language processing tasks including reasoning and question answering, all of which have been subject to intense research in recent years @xcite . however, most of this research has focused on english and other highresource languages, often neglecting medium-to low-resource languages. to make matters more difficult, most of the benchmarks designed for measuring progress in llm research are only available in english and a handful of other languages, making it very hard to compare different llm skills in low-resource languages. this issue is particularly critical in light of the growing use of llms like chatgpt in the context of education @xcite . the absence of llms that perform equally well in non-english languages would lead to further divide in access to education and technology, highlighting a critical need for more inclusive language model development. addressing this gap, our work specifically targets bengali, a language spoken by 272 million speakers worldwide @xcite , yet considered a low-resource language. we make the following contributions: ## benqa dataset our dataset, benqa, is a parallel corpus of english-bengali science questions in a multiplechoice format from 8th, 10th, and 12th grade exams. these questions are sourced from nationwide board exams in bangladesh and are officially available in both english and bengali, giving us a high-quality parallel corpus. ## experiment setup to assess the performance of existing llms on our newly created dataset, we evaluate them using the benqa dataset across several open-source and proprietary models. following the recommendation from prior work on @xcite , we keep the system prompt in english for both bengali and english datasets. most of the benchmark results presented in the main paper are conducted in a zero-shot setting to save tokenization costs in proprietary models @xcite . additionally, experiments in 3-shot and 5-shot settings were performed as supplementary analyses, details of which are provided in appendix c. ## additional experiments in this section, we conduct additional experiments to explore whether we can improve the performance in bengali using better prompting. all the experiments described in this section were conducted on gpt-3.5, which provides a good value of cost-efficiency vs. capability in bengali. 6.1 can the performance be improved by appending english translation? our hypothesis for the models' subpar performance in bengali might be attributed to two main factors: their unfamiliarity with bengali scientific terminology and potential difficulty in understanding non-latin scripts @xcite . we expect that having access to a translation of the question would likely help the model to understand the context better. to confirm our hypothesis, we experiment with a randomly selected subset of 115 data points from each subject within the 10th-grade exam questions. questions from 10th-grade exams were chosen because they provide a middle ground in terms of difficulty compared to 8th and 12th grade. prompt details as illustrated in figure 8 , in addition to providing the model with the bengali question, we append the english translation to the prompt. this results in a prompt that consists of both the original data and its translation. experiment results as shown from the results in figure 9 , appending english translation appears to have a positive impact in all subjects. throughout our work, all system prompts have been consistently in english. in this experiment, the questions were provided in bengali along with the english translations. hence the performance gain can be concluded to be coming from the appending of the english translations. the most improvement is shown in biology, where scientific terminology can be found in the majority of questions. in contrast, for math, the improvement is not as significant as in the other subjects. we also experimented with the case where we did not have access to gold english translations and replaced them with llmgenerated translations. our initial experiments suggest that appending llm-generated translations might work just as well as appending human translation. more details can be found in appendix c.3. 6.2 does the translation appended prompting method generalize to other datasets? we extend our experiments to other datasets, such as copa and big-bench-hard, to evaluate the applicability of our prompting strategy across various datasets. for copa, we ask the model to answer bengali copa problems taken from indiccopa @xcite . copa as shown in figure 10 , a noticeable improvement is observed in bengali copa when english translations are appended alongside the bengali question. the improvement is more pronounced when we use the actual english version of the copa dataset (marked as gold). however, even using gpt-generated translation, accuracy improves the performance by 7 to 8%. we also try to see if translation appended prompting can be helpful for the other languages included in the original x-copa (ponti et al., 2020) languages. the results are shown in the figure 11 . appending english translation increases accuracy in all but one language, with the effects most pronounced in low-resource languages. further research in utilizing this prompting technique is an interesting direction left for future work. big-bench-hard bengali for this experiment, we curate big-bench hard bengali (bbh-bn) dataset using gpt-4. since only a small proportion of tasks require reasoning skills in benqa (11% of the total questions), we chose big-bench hard as an additional dataset alternative. we selected some tasks in big-bench hard that require reasoning and also have equivalence to bengali to add more questions in this category. when we say equivalence to bengali, we mean ignoring some tasks, like word sorting, because the alphabetical order may not be the same upon translation to bengali. in this experiment, two native bengali speakers generated prompts for each task iteratively and then generated the prompt. our experiment shows that on average, there is a 5.8% increase in bbh-bn performance in bengali when we append gold english translation, while appending gpt-4-generated english translation provides an average of 4.2% improvement. details about big-bench-hard task selections and results can be found in appendix e. ## discussion unlike most other bilingual or multilingual nlp datasets, which are usually first curated in english and then translated into other languages, questions in our benqa datasets are curated in bengali first and then translated into english by school teachers. this approach could also provide an interesting outlook on how l2 english speakers might use english and whether llms are robust to such linguistic variations. besides, our benchmark results highlight that llms, such as chatgpt, still have some way to go in terms of catching up with performances in low-resource languages like bengali. one interesting aspect we noticed is that, in bengali, it is much harder to make the model output in a predefined format for automated evaluation. future research should focus on improving the instructionfollowing ability of multilingual models. another key finding is the substantial performance gap between open-source and proprietary language models. the open-source models, which are more accessible in the context of developing countries, must stride ahead to catch up with the proprietary models to make sure the benefits of ai, particularly llms, are not limited to only some demographics. a recently published open multilingual llm, aya @xcite , represents a promising step in this direction. furthermore, we demonstrated the feasibility of having llms respond in the target language, while leveraging the advantage of high-resource languages, such as english, in the inference pipeline by utilizing translation of the query. the benefit of this approach is that we do not need to have access to the manually-created gold translation (as we had in this case): translation using the same model or more powerful/specialized models would work too. however, the choice of translation method, whether it is another more proficient llm or a domain-specific fine-tuned translation model, can influence the performance. this finding necessitates further research to optimize and refine the use of llms, especially in the context of low-resource languages. ## conclusion and future work in this paper, we introduced benqa, a locally sourced dataset from bangladesh, comprising middle-school and high-school level exam questions in both english and bengali. our dataset is provided in parallel format, allowing us to benchmark the performance discrepancy between both languages in a fairer way. upon benchmarking several large language models (llms) with this dataset, we found that performance in bengali lags behind english, even for the best-performing models; and current open-source models significantly lag behind proprietary models. we also explored whether the performance in bengali questions can be improved by appending english translations to the prompts. this approach resulted in performance enhancements across most subjects in the benqa dataset on gpt-3.5 model. notably, this improvement also extends to other datasets, such as copa and big-bench-hard. these findings pave a promising direction for future research in improving the llms performance in low-resource languages, specifically in bengali. our findings open up several promising directions for future research. our results have highlighted a possible gap in the current models' understanding of bengali terminology, leading us to develop a very simple prompting method that helps the performance in bengali data significantly. we also show the efficacy of this prompting approach on bengali copa, x-copa, and bengali big-bench-hard dataset. future research should also focus on to see to what extent such prompting methods utilizing high-resource languages generalize to other datasets and languages.