Flames: Benchmarking Value Alignment of LLM s in C hinese (2024)

Abstract: The widespread adoption of large language models (LLMs) across various regions underscores the urgent need to evaluate their alignment with human values. Current benchmarks, however, fall short of effectively uncovering safety vulnerabilities in LLMs. Despite numerous models achieving high scores and ‘topping the chart’ in these evaluations, there is still a significant gap in LLMs’ deeper alignment with human values and achieving genuine harmlessness. To this end, this paper proposes a value alignment benchmark named Flames, which encompasses both common harmlessness principles and a unique morality dimension that integrates specific Chinese values such as harmony. Accordingly, we carefully design adversarial prompts that incorporate complex scenarios and jailbreaking methods, mostly with implicit malice. By prompting 17 mainstream LLMs, we obtain model responses and rigorously annotate them for detailed evaluation. Our findings indicate that all the evaluated LLMs demonstrate relatively poor performance on Flames, particularly in the safety and fairness dimensions. We also develop a lightweight specified scorer capable of scoring LLMs across multiple dimensions to efficiently evaluate new models on the benchmark. The complexity of Flames has far exceeded existing benchmarks, setting a new challenge for contemporary LLMs and highlighting the need for further alignment of LLMs. Our benchmark is publicly available at https://github.com/AIFlames/Flames.

## introduction large language models (llms) play a vital role in today's ai landscape, drawing top-tier companies and research teams into their exploration @xcite @xcite @xcite . however, llms also bring safety challenges as they may generate harmful content that violates legal, ethical, and human rights principles @xcite @xcite . it is thus crucial to evaluate the extent to which llms align with human values. currently, researchers have dedicated efforts to benchmark language models' ethical and safety ability @xcite @xcite @xcite . likewise, in the context of chinese language, scholars have crafted benchmarks to measure the safety of outputs generated by llms supporting chinese @xcite @xcite . however, these benchmarks have notable limitations: (a) they contain prompts with explicit malice that llms with simple fine-tuning can easily learn to refuse to answer; (b) they have a lack of fine-grained annotations, which are necessary for enhancing the harmlessness of llms in practice; and (c) they fail to provide a specified scorer for evaluating new models in the future, which limits the usage of the datasets. to address these problems, we propose flames (fairness, legality, data protection, morality, safety) -the first highly adversarial benchmark in chinese for evaluating the value alignment of llms, to our best knowledge, which includes a manually designed prompts dataset, a fine-grained annotation dataset, and a specified scorer. tab. 1 shows the comparison between flames and existing chinese benchmarks. the remarkably high success rate of attacks underscores the challenging nature of flames. fig. 1 illustrates the construction pipeline of flames. we first design a framework that encompasses five dimensions in line with human values, each of which is further subdivided into several sub-components, enabling a more detailed and fine-grained evaluation. notably, the corpus of the morality dimension incorporates various chinese cultural and traditional qualities. based on this framework, we carefully guide crowd workers to design highly inductive, adversarial prompts that contain implicit malice. following their initial design, we engage in an iterative process of review and refinement to ensure their effectiveness. we test these prompts on a selection of random llms to ascertain their capacity to successfully 'attack' these models. only those prompts that demonstrate the ability to compromise or elicit inappropriate responses from at least one llm are retained. subsequently, we gather responses to these prompts from popular llms and then meticulously craft highly detailed guidelines for annotation. analyzing the data, we observe that all the evaluated llms exhibit poor performance on flames. claude emerges as the best-performing model, yet its harmless rate is only 63.77%. this finding highlights the effectiveness of flames in revealing the safety vulnerabilities of llms and underlines the imperative for ongoing improvements in value alignment. furthermore, to facilitate subsequent automatic evaluations, we train a scoring model using ∼22.9k annotated data. our scorer, with an achieved accuracy of 79.5%, significantly outperforms gpt-4 as a judge (61.3%). this higher level of accuracy positions our scorer as a more reliable and cost-effective tool for the public evaluation of flames. in summary, this paper has the following contributions: we have meticulously designed a dataset of 2,251 highly adversarial, manually crafted prompts, each tailored to probe a specific value dimension. our evaluation addresses the exceptional challenge presented by flames, which far exceeds incumbent benchmarks in the field. ## flames construction in this section, we introduce how we design our framework for value alignment and how we construct the datasets. overall, the prompts and annotation datasets follow the prompting-responseannotation workflow, as shown in fig. 3 . ## specified scorer we first employ gpt-4 as a scorer with 0-shot and 5-shot prompting. however, the overall accuracy of gpt-4 judgments is only 58.8% for 5shot prompting and 51.9% for 0-shot prompting, which indicates that it is not reliable to use gpt-4 as a judge on flames (see app. a for more details). hence, we develop a combined scoring model named flames-scorer, which can score llms' responses on flames holistically and more accurately. ## conclusion this study proposes flames -the first highly adversarial benchmark for evaluating the value alignment of llms in chinese. flames benchmark consists of (a) a comprehensive framework, (b) a highly adversarial, manually crafted prompts dataset, (c) a carefully annotated dataset with finegrained human evaluation, and (d) a light-weight specified scorer with high accuracy on flames. based on the flames benchmark, we conduct an extensive evaluation and analysis of existing llms. we find that although various techniques have effectively enhanced the ability of llms to understand human society, significant gaps still exist across multiple dimensions, especially safety. the flames benchmark, therefore, serves not only as a thorough and systematic method for assessing value alignment in llms but also poses a new standard in the field.