CIF -Bench: A C hinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models (2024)

Abstract: The advancement of large language models (LLMs) has enhanced the ability to generalize across a wide range of unseen natural language processing (NLP) tasks through instruction-following.Yet, their effectiveness often diminishes in low-resource languages like Chinese, exacerbated by biased evaluations from data leakage, casting doubt on their true generalizability to new linguistic territories. In response, we introduce the Chinese Instruction-Following Benchmark (**CIF-Bench**), designed to evaluate the zero-shot generalizability of LLMs to the Chinese language. CIF-Bench comprises 150 tasks and 15,000 input-output pairs, developed by native speakers to test complex reasoning and Chinese cultural nuances across 20 categories. To mitigate data contamination, we release only half of the dataset publicly, with the remainder kept private, and introduce diversified instructions to minimize score variance, totaling 45,000 data instances.Our evaluation of 28 selected LLMs reveals a noticeable performance gap, with the best model scoring only 52.9%, highlighting the limitations of LLMs in less familiar language and task contexts.This work not only uncovers the current limitations of LLMs in handling Chinese language tasks but also sets a new standard for future LLM generalizability research, pushing towards the development of more adaptable, culturally informed, and linguistically diverse models.

## introduction the landscape of natural language processing (nlp) has been dramatically reshaped by the emergence of large language models (llms), which have demonstrated an ability to generalize across unseen nlp tasks @xcite @xcite the enemy destroyed the city. your stance tells me. how about a nice, big hug for grandma? she kept getting distracted by her own face in the camera viewfinder. i appreciate the opportunity i've had here, and i hope you guys have a great summer. you need to be able to justify why you're wearing yellow socks in a formal environment. dancing with you is so carefree. i miss my best friend. i had so much hope in the beginning. it' s fascinating that you know all the planets. 2023, 2024), often showcased through the framework of instruction-following @xcite @xcite . despite these advances, skepticism remains regarding the transferability of this instruction-following capability, particularly in multilingual contexts. the models perform worse when switching to chinese due to the prevalence of english training data @xcite , as figured in fig. 1 . this concern is exacerbated by observations that benchmarks designed to assess the capabilities of llms may inadvertently suffer from biased evaluations due to data leakage @xcite , particularly when web-scale datasets are employed to enhance model generalizability @xcite . such observations raise a critical question: while the generalizability of llms appears intriguing, do these models face significant challenges when evaluated on private and diversified instruction-formatted tasks in less common language contexts? to answer this question, we introduce the chinese instruction-following benchmark (cif-bench), a novel benchmark designed for the zero-shot generalizability evaluation of llms, with chinese serving as an insightful example for multilingual transferred instruction-following tasks. our benchmark comprises 150 tasks and 15, 000 input-output pairs, with the assistance of native speaker annotators, ensuring the inclusion of human-authored tasks that are not only challenging but also naturally expressed. a significant portion (38.7%) of these tasks are designed to test a model's complex natural language inference (nli) and reasoning capabilities, as well as drawing upon chinese culture spread across 20 distinct categories. in an effort to mitigate future evaluation biases from data leakage, we decide to publicly release only half of the data instances, reserving the rest as a private dataset to maintain an impartial benchmark. furthermore, cif-bench enhances its robustness by introducing 5 variations of instructions per task, using these to diminish score variance in private split evaluations as discussed in §5. cif-bench also pioneers a model-based automatic pipeline designed to tackle the inherent challenges of evaluating open-ended natural language generation outputs @xcite . by selecting a range of popular llms that support chinese for evaluation, we aim to depict the limits of current instruction-following capabilities in language transfer contexts as the many models follow an english-oriented pre-training paradigm @xcite . our findings reveal that even the best-performing model achieves a score of only 52.9% on cif-bench, underscoring the gap that exists when llms are confronted with tasks in a less-familiar language and unseen data instances. we find that this performance decrement is particularly noticeable in scenarios involving unseen tasks and unseen input-output pairs, contrasting with the models' performance on existing chinese datasets and translated english-language tasks. such results suggest that while llms exhibit impressive generalizability in a context more aligned with observed data, their effectiveness diminishes when faced with the dual challenges of unacquainted languages and novel tasks. to summarize our contributions, we: chinese llm benchmarks. there have been important efforts, such as clue @xcite and cuge @xcite , made to evaluate the pre-trained language on extensive tasks in the chinese context, which consider the traditional taxonomy of natural language understanding and generation. as these benchmarks are restricted in the prediction formats and could not fully measure the cross-task generalization of llms in the free-form outputs, more recent studies @xcite propose to reformat the tasks into multi-choice question answering, mostly examining the knowledge-base abilities in chinese. however, such a strict format could impede the models from fully generalizing to more complex reasoning and creative tasks. thereby, we argue that there is a lag in evaluating llms instruction-following capacity in the chinese language. ## the challenging chinese instruction-following benchmark the challenging chinese instruction-following benchmark unifies the nlp tasks in the promptbased instruction-following schema @xcite and evaluates the llms in a zero-shot manner, which is to say that the models are expected to directly provide the correct output given the concatenation of the task instruction and data input texts. formally, for each data sample in cif-bench, the three components we refer to are: ## experiments baselines. we compare the performance of existing llms that have been trained on chinese corpora. we select chatgpt, for which we use gpt-3.5-turbo-instruct, foot_1 which we believe corresponds to instructgpt text-davinci-002. then we select a series of open-source llms, including chatglm @xcite , aquilachat-7b. 5 baichuan @xcite , deepseek-llm-67b-chat (deepseek-ai, 2024), qwen @xcite , yi, foot_3 tigerbot-7b-chat @xcite , telechat @xcite , cpm-bee-10b, foot_4 , and moss-moon @xcite ## results analysis broadly speaking, we aim to investigate the performance capabilities of current representative chinese llms in a diverse set of nlp tasks to ascertain how well the annotated data with human is cif-bench challenging? to ensure the reliability of our benchmark, the scores in the private split with the diversified instructions are referred to as the main results for discussion, as shown in in addition, we provide finer-grained score aggre-gation to further analyze the challenging task categories (n.b., most bilingual llms perform poorly on tasks in code, summarization, and translation categories). in the code category, the models might misunderstand the semantics expressed in chinese for the newly defined variable or function. specifically, models usually perform poorly in a new "programming language" environment that requires the model to understand restricted actions. as for summarization tasks, models could misinterpret the instruction, eg. models sometimes consider the instruction "modify the input into a more friendly expression to non-native speakers" as a chinese-english translation task and might provide redundant explanations even if not required by the instructions and hence will cause large semantic distances to the golden reference. we point out that chinese-commented code corpora and parallel translation data of chinese and other languages are still scarce resources, which might lead to their poor performance on cif-bench's code and translation categories. additionally, we assume that chinese and english bilingual llms, although a major branch of multilingual llm, do not significantly benefit llms' capacity to deal with minor-language-related tasks. part of the tasks in cif-bench's summarization category are very challenging, combining counterfactual reasoning and empathy estimation (i.e., task 125 and task 131 referring to appendix a.1). thereby, the bilingual llms' poor performance on cif-bench's summarization category is understandable. detailed category-based scores on the public split are available in language transferability. we select the public split to investigate llm language transferability in instruction-following. in the cif-bench public split, a set of 70 tasks from sni @xcite are used as representative samples of english nlp tasks equipped with directly translated input-output pairs in chinese. we select the top-5 performing models on the public split to show the performance comparison between sni and our 37 original curated chinese tasks in ## conclusion in summary, cif-bench not only exposes the limitations of current llms in navigating the complexities of chinese language instruction-following tasks but also provides a foundational platform for future advancements in llm generalizability research. through this work, we aim to facilitate the development of more adaptable, culturally aware, and linguistically diverse language models, capable of truly understanding and interacting with the global tapestry of human language.