PARIKSHA : A Large-Scale Investigation of Human- LLM Evaluator Agreement on Multilingual and Multi-Cultural Data (2024)

Abstract: Evaluation of multilingual Large Language Models (LLMs) is challenging due to a variety of factors – the lack of benchmarks with sufficient linguistic diversity, contamination of popular benchmarks into LLM pre-training data and the lack of local, cultural nuances in translated benchmarks. In this work, we study human and LLM-based evaluation in a multilingual, multi-cultural setting. We evaluate 30 models across 10 Indic languages by conducting 90K human evaluations and 30K LLM-based evaluations and find that models such as GPT-4o and Llama-3 70B consistently perform best for most Indic languages. We build leaderboards for two evaluation settings - pairwise comparison and direct assessment and analyse the agreement between humans and LLMs. We find that humans and LLMs agree fairly well in the pairwise setting but the agreement drops for direct assessment evaluation especially for languages such as Bengali and Odia. We also check for various biases in human and LLM-based evaluation and find evidence of self-bias in the GPT-based evaluator. Our work presents a significant step towards scaling up multilingual evaluation of LLMs.

## introduction large language models (llms) have made tremendous progress recently by excelling at several tasks @xcite @xcite . however, it is not always clear what capabilities these models possess, leading to an increased interest in evaluation. benchmarking is the defacto standard for evaluating llms, with several popular benchmarks used to validate the quality of models when they are released. however, standard benchmarking suffers from the following issues: many popular benchmarks are available on the web and have already been consumed in the training data of llms, rendering them unsuitable for fair evaluation. this phenomenon is known as test dataset contamination, and recent works @xcite @xcite @xcite have suggested that contamination may occur not only during pre-training, but also during fine-tuning and evaluation @xcite . this calls for dynamic benchmarking with the help of humans @xcite . although, human evaluation is considered the gold standard, it can be expensive and time consuming. due to this, the use of llm-evaluators, where an llm itself is used to evaluate the output of another llm (sometimes the same llm) has become very popular. most studies on llm training and evaluation focus on english. recent works have shown that llms perform worse on non-english languages, particularly those written in scripts other than the latin script, and under-resourced languages @xcite @xcite . studies on cultural values in llms have also shown that frontier models such as gpt-4 align more closely to western, rich, and industrialized norms @xcite . this has led to a proliferation of models being built for specific languages, cultures and regions such as indic, arabic, african, chinese, european, and indonesian @xcite @xcite @xcite @xcite . multilingual evaluation is challenging due to the small number of multilingual benchmarks available, the lack of language diversity in them @xcite and the evidence of possible contamination of many of these benchmarks @xcite . additionally, many multilingual benchmarks are translations of benchmarks originally created in english, leading to loss of linguistic and cultural context. in this work, we perform 90k human evaluations -the largest scale multilingual human evaluation of llms as per our knowledge. we perform evaluation on a new set of general and culturally-nuanced prompts created independently by native speakers for each language. we use a setting similar to the lmsys chatbotarena @xcite and ask human evaluators employed by karya foot_0 , an ethical data company, to perform two evaluation tasks: comparative evaluations between models, and individual evaluations or direct assessments of 30 models. karya employs workers from all states of india, with a focus on rural and marginalized communities, making our study the first effort as per our knowledge that includes these communities in the evaluation process. in addition to performing human evaluations, we build upon prior work on llms as multilingual evaluators @xcite to perform the same evaluations using llms as judges. we also use llms to perform a preliminary safety evaluation, for which we do not engage karya workers due to ethical concerns. our contributions are as follows: (1) we perform 90k human evaluations across 10 indic languages, comparing 30 indic and multilingual models using pairwise and direct assessment on a culturallynuanced dataset. 2 related work multilingual evaluation benchmarks @xcite ; @xcite conduct comprehensive multilingual evaluations of open-source and proprietary models on a large scale across various available multilingual benchmarks. @xcite release a multilingual generative test set that can assess the capability of llms in five different languages. other popular multilingual nlu benchmarks include xglue @xcite , xtreme @xcite , xtreme-r @xcite . @xcite release the first indic nlu benchmark, in-dicglue, for 11 languages. @xcite build on top of the former and release in-dicxtreme, spanning all 22 languages. on the nlg side, @xcite offer indicnl-gsuite, covering 5 tasks across 11 languages. @xcite release a machine translation benchmark, in22, for both conversational and general translation evaluation across all 22 languages. recently, @xcite put forth indicgen-bench, a collection of diverse generation tasks like cross-lingual summarization, machine translation, and cross-lingual question answering. ## methodology our evaluation setup is summarized in figure 1 . ## discussion rq1. are indic llms able to compete with proprietary models in respective languages? from our evaluations, we find that smaller indic models perform better than the open-source models they are trained on, and larger frontier models such as gpt-4o perform best on indic languages. however, newer medium-sized open-source models such as llama-3 show great potential in our evaluations. our evaluation not only provides a ranking of llms but also indicates which open source models (like llama-3) are potentially promising starting points for fine-tuning language specific indic models. rq2. can llm evaluators be used as a substitute for human evaluators in the multilingual setting? which task format shows higher promise? we find that llm evaluators agree fairly well with humans on the pairwise evaluation task in comparison to the direct assessment task. the llm evaluator has low agreement with humans on marathi, bengali and punjabi in the pairwise task and very low agreement for all languages particularly bengali and odia in direct assessment task. we also get feedback from the native human annotators and find direct assessment to be a harder task. on manually going through some examples, we find that humans tend to prefer outputs which are more friendly in nature, i.e., have good formatting, use colloquial language and explain with the help of examples. we find that llm-evaluators agree less with humans on evaluating responses with culturalnuances, suggesting that they do no possess enough cultural context to do these kinds of evaluations well. however, llm evaluators are still able to capture general trends at a higher level as seen from the τ scores. this suggests that a human-in-theloop or hybrid evaluation system is necessary for performing multilingual, multi-cultural evaluation. rq3. what are the biases that affects the evaluators' judgements? we look for position bias by looking at evaluator behaviour on option flipping and find no such biases. llm evaluators are not able to detect hallucinations and pick a response even when both are hallucinated in 87% cases compared to 53% by humans. they are also found to be over-optimistic in nature. this leads to llm evaluators having higher scores in the direct assessment task as well as fewer ties (more decisive) in pairwise evaluations task. we also look for correlations between response length and a winning response and find slight bias towards a longer response by both evaluators. lastly, we check for any self-bias in the gpt evaluator and find evidence of it preferring its own output. ## limitations language coverage our work is subject to some limitations. our study covers 10 indic languages, however, there are several other indic languages that we do not cover yet in this study, which we hope to do in future iterations. our choice of languages is based on the availability of languagespecific indic models. prompt diversity the prompts used for evaluation in our study are limited, and we plan to scale the number of prompts used in future iterations. however, due to the nature of pairwise evaluations, where every model is evaluated in battles with every other model, scaling to hundreds of prompts for human evaluation becomes intractable. we plan to modify our design to have fewer battles per prompt and also source more prompts from native speakers. ## ethics statement we use the framework by bender and friedman (2018) to discuss the ethical considerations for our work. institutional review all aspects of this research were reviewed and approved by the institutional review board of our organization and also approved by karya . data our study is conducted in collaboration with karya , that pays workers several times the minimum wage in india and provides them with dignified digital work. workers were paid rs. 10 per datapoint for this study. each datapoint took approximately 5 minutes to evaluate. annotator demographics all annotators were native speakers of the languages that they were evaluating. other annotator demographics were not collected for this study. annotation guidelines karya provided annotation guidelines and training to all workers. the guidelines and training were modified based on experiences from a pilot study we conducted before the evaluation round described in this paper. we once again highlight that no human annotators were employed for the safety/toxicity analysis of our work. ## mle elo in the context of llms, the models have fixed weights and their performance does not change over time unless further training is done. therefore, the order of battles does not matter. to estimate the log-likelihood of the underlying elo, we use the bradley-terry (bt) model (bradley and terry, 1952), which assumes a fixed but unknown pairwise win-rate. like elo rating, the bt model also derives ratings of players based on pairwise comparison to estimate win-rate between each other. the main difference between the bt model and the standard elo system is that the bt model assumes that the player's performance does not change (i.e., game order does not matter). we use a logistic regression implementation to calculate the maximum likelihood estimate (mle) elo ratings.