S ea E val for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning (2024)

Abstract: We present SeaEval, a benchmark for multilingual foundation models. In addition to characterizing how these models understand and reason with natural language, we also investigate how well they comprehend cultural practices, nuances, and values. Alongside standard accuracy metrics, we investigate the brittleness of foundation models in the dimensions of semantics and multilinguality. Our analyses span both open-sourced and closed models, leading to empirical results across classic NLP tasks, reasoning, and cultural comprehension. Key findings indicate (1) Many models exhibit varied behavior when given paraphrased instructions. (2) Many models still suffer from exposure bias (e.g., positional bias, majority label bias). (3) For questions rooted in factual, scientific, and commonsense knowledge, consistent responses are expected across multilingual queries that are semantically equivalent. Yet, most models surprisingly demonstrate inconsistent performance on these queries. (4) Multilingually-trained models have not attained “balanced multilingual” capabilities. Our endeavors underscore the need for more generalizable semantic representations and enhanced multilingual contextualization. SeaEval can serve as a launchpad for more thorough investigations and evaluations for multilingual and multicultural scenarios.

## introduction over the past years, there has been rapid development of large language models (llms), also known as a type of foundation models (fms) @xcite , demonstrating their generalizability and adaptability across various downstream tasks @xcite @xcite @xcite figure 1 : seaeval for multilingual foundation models. english is represented by the color blue, chinese by red, and a mix of multiple languages by yellow. seaeval includes the datasets within the dotted-line circle. compared to other species on earth, humans are adept in using language to symbolize and encode thoughts, represent and document knowledge, and express emotions. whether it is spoken or written form, language has become a default means for humans to conduct and communicate our reasoning process and logic @xcite . three variables are language, region, and culture. our cultural behavior, rituals, and values are often embodied and represented through lan-guage @xcite . therefore, the capabilities of language models go beyond language per se. evaluating multilingual language model should incorporate deep cultural understanding and reasoning. to this end, we expand the current evaluation criteria to cover more linguistic and cultural contexts @xcite , which are under-explored in prior research. another important yet under-explored aspect of evaluating multilingual foundation models is their knowledge transferability in language dimensions. multilingual foundation models are expected to demonstrate consistent performance across languages in the context of region-invariant common knowledge @xcite . monolingual or bilingual benchmarks cannot adequately capture this aspect. therefore, we introduce cross-lingual consistency evaluation with tailored datasets and specialized metrics. seaeval aims to assess the capabilities of multilingual foundation models in four dimensions: (a) classic nlp tasks that are centered around language understanding and generation; (b) complex reasoning; (c) cultural understanding and reasoning; and (d) cross-lingual knowledge transfer and contextualization. seaeval encompasses a total of 28 datasets, including 6 new datasets constructed for cultural reasoning and cross-lingual consistency assessments. key findings from our investigations and experimental results indicate: (1) most models show varying responses to rephrased instructions. (2) exposure bias (e.g., positional bias, majority label bias) of label arrangements still prevails. (3) most models give inconsistent answers when the same fact-based questions are asked in different languages. this counter-intuitive observation suggests semantic representations are not generalized in the multilingual context. (4) multilingually-trained models fall short of achieving "balanced multilingual" proficiency. to sum up, our contributions are multifold: ## essential properties of multilingual foundation models and benchmarks in this section, we delve into the desired properties of multilingual foundation models and explore the ideology for crafting a comprehensive benchmark. ## seaeval in this section, we present our seaeval benchmark from task selection, data curation, to evaluation protocols. besides the evaluation of fundamental capabilities and complex reasoning, we also include the evaluation tasks on cultural understanding and cross-lingual alignment. the datasets are summarized in ## evaluation results and discussion we show the evaluation results on five datasets for cross-lingual consistency and cultural reasoning in figure 5 and our key findings are as follows. firstly, gpt-4 demonstrates outstanding performance on most datasets, surpassing others by a substantial margin across cultures and languages, demonstrating its superior capability in handling multilingual tasks. second, becoming an expert in cultural knowledge necessitates extensive pre-training with a di20 40 60 80 ac3 accuracy consistency english-acc chinese-acc spanish-acc indonesian-acc vietnamese-acc malay-acc baichuan-2-13b-chat llama-2-70b-chat bloomz chatgpt gpt4 figure 6: evaluation results on cross-mmlu. both overall and language-specific scores are shown. last, when assessing models' accuracy in indi-vidual languages, it is evident that their problemsolving capability in english usually surpasses that in other selected languages. this illustrates that the proficiency of models varies unevenly across different languages (our finding 4). compared to high-resource languages, the performance of low-resource languages is inferior. for example, english, chinese and spanish rank within the top 5 out of 46 languages present in bloomz corpus. therefore, the multilingual foundation model's capability in low-resource languages needs to be further improved to match the more centralized languages. the disparity in cross-lingual consistency highlights the need for more robust alignment efforts. particularly under low-resource constraints, enhancements in this aspect have the potential to elevate the overall performance across all languages through effective knowledge transfer, facilitating further development of multilingual language models @xcite @xcite . ## conclusions we introduced seaeval benchmark for multilingual foundation model evaluation, grounded in comprehensive experimentation across languages, models, tasks, and datasets. seaeval encompasses 29 datasets, including 7 new ones for cultural understanding and cross-lingual consistency. our empirical analysis demonstrates four key findings on the capabilities of multilingual foundation models: