METAL : Towards Multilingual Meta-Evaluation (2024)

Abstract: With the rising human-like precision of Large Language Models (LLMs) in numerous tasks, their utilization in a variety of real-world applications is becoming more prevalent. Several studies have shown that LLMs excel on many standard NLP benchmarks. However, it is challenging to evaluate LLMs due to test dataset contamination and the limitations of traditional metrics. Since human evaluations are difficult to collect, there is a growing interest in the community to use LLMs themselves as reference-free evaluators for subjective metrics. However, past work has shown that LLM-based evaluators can exhibit bias and have poor alignment with human judgments. In this study, we propose a framework for an end-to-end assessment of LLMs as evaluators in multilingual scenarios. We create a carefully curated dataset, covering 10 languages containing native speaker judgments for the task of summarization. This dataset is created specifically to evaluate LLM-based evaluators, which we refer to as meta-evaluation (METAL). We compare the performance of LLM-based evaluators created using GPT-3.5-Turbo, GPT-4, and PaLM2. Our results indicate that LLM-based evaluators based on GPT-4 perform the best across languages, while GPT-3.5-Turbo performs poorly. Additionally, we perform an analysis of the reasoning provided by LLM-based evaluators and find that it often does not match the reasoning provided by human judges.

## introduction recent large language models (llms) like @xcite , gpt-3.5-turbo @xcite , palm2 @xcite , gemini-1.5 @xcite , mistral @xcite etc. have shown impressive performance on a variety of standard nlp tasks across languages @xcite @xcite figure 1 : pipeline of metal framework. @xcite . however, there are several challenges in fair and accurate assessment of these models, such as the contamination of existing datasets in llm pre-training data, lack of multilingual datasets @xcite , lack of benchmarks that represent real-world usage of these models, lack of frameworks for consistent subjective evaluations, and budget and access issues for native speaker evaluation. therefore, there is a growing need for frameworks and resources that address the above challenges and allow us to systematically evaluate llms across several dimensions and languages. further, evaluating the text generation capabilities of these models is even more challenging @xcite @xcite . natural language generation (nlg) capabilities are traditionally evaluated using automated metrics such as rouge @xcite or bleu @xcite scores. these metrics have several known drawbacks, such as reliance on exact matches and over-emphasis on length. secondly, these metrics do not account for subjective evaluations such as quality, coverage, and coherence @xcite @xcite . thirdly, these metrics are reference-based i.e. they need a comparison baseline, which can be expensive to collect and can sometimes have a low correlation with human judgments. this has led to work on reference-free and subjective evaluation @xcite @xcite . using llms as evaluators presents several challenges. recent works @xcite @xcite have shown that while llms can produce evaluations with human-like accuracy, these evaluations are often inconsistent and can easily be influenced. llms also show position bias or scale region bias and are unable to distinguish between candidates that are close to each other @xcite . llms are sensitive to instructions and their capabilities vary for different metrics @xcite @xcite . another significant challenge when using llms as evaluators is a limited assessment of their abilities in multilingual settings. studies have shown that llms have inferior performance even on some high-resource languages and cannot be assessed extensively on low-resource languages due to a lack of benchmarks @xcite . therefore, it is still unclear if llms can replace human evaluations in multilingual settings. in this paper, we introduce the metal framework for a robust assessment of llms as evaluators in multilingual scenarios. figure 1 shows an outline of our framework. the metal framework is an end-to-end pipeline, that starts with first creating a rich meta-evaluation dataset that contains a variety of samples across the metrics of interest. we do this by systematically prompting gpt-4 to generate a wide range of sample data points, that are then evaluated by native speakers. in the next step, we compare llm judgments with human judgments. for this, we draw on our previous work @xcite to prompt llms for evaluations and subsequently compare the scores with human judgments. in particular, for the task of summarization we create a dataset of 1000 summaries covering 10 languages, with human ratings across 5 metrics. 1 next, we obtain llm evaluations from gpt-3.5-turbo, gpt-4, and palm2 across these 1000 summaries and 5 metrics. our findings show that the gpt-3.5-turbo-based evaluator does not perform well across languages and metrics, while evaluators based on gpt-4 and palm2 perform better. we find that the evaluation ability of llms significantly across languages, motivating the creation of a meticulously crafted metaevaluation dataset covering all target languages before using llm-based evaluators. lastly, our qualitative analysis shows that while gpt-4 and palm2 can achieve accuracy close to humans, the reasoning behind their evaluations is often flawed. while we study the applicability of the metal framework for the task of summarization, it is extensible to other tasks as well, by creating meta-evaluation datasets for the other tasks. evaluation datasets human-verified goldstandard datasets are crucial for being able to evaluate llms. @xcite release risum, an english-centric dataset of documentinstruction-output triplets, with the llms generating instructions and outputs, and a human evaluation is adopted to score the triplets, with a focus on "instruction-following". in our work, we manually curate the instructions and create a dataset covering 10 languages. seahorse @xcite is a multilingual and multifaceted data for summarization with 96k summaries, and metrics related to grammar and output quality. they fine-tune t5 @xcite and palm @xcite models on the train split of the dataset for generating a spectrum of outputs, whereas we work with black-box models and tune our prompts to generate "good" and "bad" summaries. ## the metal dataset the metal dataset contains a total of 1000 summaries across 10 languages. the dataset is specially curated to investigate the capabilities of llms as evaluators in different languages along 5 dimensions. in this section, we describe how the dataset was created and annotated. ## qualitative analysis while results in section §5 show that detailed instructions to gpt-4 and palm2 give evaluations very close to humans, it is unclear how humans and llms reason about their scores. we qualitatively analyse the comments from the annotators for their ratings and justifications produced by the llms at the time of scoring the summaries. an analysis of some interesting examples is discussed in this section. as discussed in section §4.4, the annotations from humans can be divided into three categories: when all annotators agree, when two annotators agree, and when no annotators agree. the first example is where no annotators agree on tq for an english sample. both gpt-4 and palm2 assign a 2 in this case. while all three annotators point out a few problems with the summary, both gpt-4 and palm2 ignore some key elements for tq such as "omission of important information", and "poor discourse" and say that the summary "captures main points of the passage". the next two cases in the table are when two annotators agree. in the second case, two annotators give the sample a score of 1 for la, however, no annotators point towards any grammatical issues with the summary. their comments are more relevant for tq and ocq. this indicates that for humans their judgment of one metric might affect their judgment of other metrics. both llms give a high score of 2 to the sample, even though the reason from palm2 says "lacks natural rhythm and intonation". this shows that llms' reasons might not always be aligned with their scores, in line with findings from @xcite . in the third case, the annotator aggregate for ocq is 2, however, both llms assign a score of 0. annotators mention problems such as "hallucinations" in their comments, while gpt-4 says the summary is an inaccurate representation of the main text, and palm2 complains of incoherence. the last two examples in the table are where all three annotators agree but the llm scores are different. in the fourth case, all annotators assign a score of 2 for tq, however, both llms assign a score of 0. even though a2 complains about the quality of the summary, they assign a score of 2, indicating some error in judgment. both llms assign a score of 0 and reason that that the summary consists of hallucinations. it is interesting that humans still assign the summary a score of 2 indicating that there can be subtle differences in how humans interpret these metrics. in the last case, all annotators assign the sample a score of 2 for ocq and do not mention any issues with content quality in their comments. interestingly, palm2 assigns a score of 1 and the justification states "it does not provide any new information or insights beyond what is already in the passage". since this was a summarization task no new information is expected in the summary. this again indicates that the judgment and justification might not always be aligned. language metric human score and comments gpt-4 score and justification palm2 score and justification 1 english tq 'a1': 1, 'some grammatical errors present. some misaligned elements and omission of important information.', 'a2': 2, 'high quality summary, except missing verbs so medium linguistic acceptability', 'a3': 0, 'summary contains grammatical errors, has poor discourse and is incoherent.' 2, the summary adheres to the instructions and captures the main points of the passage. therefore, the summary demonstrates a precise understanding and adherence to the prompt's instructions. overall, our analysis indicates that there are several challenges in the alignment of human evaluations with llm evaluations. while the scoring by llms on several metrics and languages might come close to humans, it is difficult to understand how they come up with these scores, necessitating further research. ## conclusion we presented the first framework for end-to-end evaluation of llms as evaluators in multilingual scenarios. we created a dataset of 1000 summaries across 10 languages rated by native speakers on 5 different metrics. our dataset covers a range of summaries in terms of linguistic acceptability, output quality, task quality, and others. we do this by systematically prompting gpt-4 to generate summaries of varying quality. the human ratings obtained for these summaries are of high quality with κ > 0.6 and f 1 > 0.75. we plan to make the metaldataset available to the research community. using our dataset, we investigate the capabilities of three llms as evaluators: gpt-3.5-turbo, gpt-4, and palm2 using two prompting strategies and compare their evaluation with the metalhuman evaluations. our results show that gpt-4 with detailed instructions performs closest to humans, while gpt-3.5-turbo is not a suitable multilingual evaluator but surprisingly does better than gpt-4 and palm2 in some metrics for bengali. we also show that gpt-4 with detailed instructions does best when there is disagreement amongst human annotators. we compare the overlapping summaries between seahorse and metal and show how our metrics and prompting methods can be used to compare generations from different models. finally, we analyze human and llm reasoning and observe that llms often provide incorrect justifications for their scores, thus showing that more research is needed to be able to use llm-based evaluators with confidence in the multilingual setting.