S afe L aw B ench: Towards Safe Alignment of Large Language Models (2025)

Abstract: With the growing prevalence of large language models (LLMs), the safety of LLMs has raised significant concerns. However, there is still a lack of definitive standards for evaluating their safety due to the subjective nature of current safety benchmarks. To address this gap, we conducted the first exploration of LLMs’ safety evaluation from a legal perspective by proposing the SafeLawBench benchmark. SafeLawBench categorizes safety risks into three levels based on legal standards, providing a systematic and comprehensive framework for evaluation. It comprises 24,860 multi-choice questions and 1,106 open-domain question-answering (QA) tasks. Our evaluation included 2 closed-source LLMs and 18 open-source LLMs using zero-shot and few-shot prompting, highlighting the safety features of each model. We also evaluated the LLMs’ safety-related reasoning stability and refusal behavior. Additionally, we found that a majority voting mechanism can enhance model performance. Notably, even leading SOTA models like Claude-3.5-Sonnet and GPT-4o have not exceeded 80.5% accuracy in multi-choice tasks on SafeLawBench, while the average accuracy of 20 LLMs remains at 68.8%. We urge the community to prioritize research on the safety of LLMs.

## introduction recent studies indicate that large language models (llms) @xcite @xcite may exhibit risks, including threats to the protection of private data @xcite , the generation of hallucinations @xcite , and negative social impacts @xcite . in response to these challenges, efforts have been made to enhance the training and inference processes of llms to align human preference and value @xcite @xcite @xcite . consequently, establishing a rigorous safety evaluation benchmark is essential to ensure the effectiveness of these solutions and to meet the growing demand for ai governance @xcite . safety benchmarks have been proposed to evaluate the risks of llms from different perspectives @xcite @xcite . however, their division of safety issues is inherently subjective and lacks a definitive standard @xcite @xcite @xcite . for instance, specific risk categories, such as non-violent unethical behavior, do not have legal implications @xcite . this subjectivity can lead to inconsistencies in safety evaluations, making determining the actual risks associated with llms challenging. therefore, a more reliable and consistent safety taxonomy is needed to systematically address all risk aspects. legal frameworks that clearly define acceptable behaviors and the consequences of violations provide a foundation for assessing safety and can be utilized for this purpose @xcite . while legal benchmarks are designed for specific legal tasks, they primarily assess the capabilities of llms within the legal domain @xcite @xcite . as a result, they do not fully address the broader safety implications of llm behavior. consequently, a comprehensive approach to safety evaluation is needed that integrates both legal knowledge and safety perspectives to address the multifaceted risks associated with llms. to address this gap, we introduced safelaw-bench, a three-tiered safety evaluation benchmark developed from hierarchical clustering of realworld legal materials. the safety evaluation benchmark was developed through iterative refinement question: what is the correct procedure for the prosecution before requesting a determination that the offence was organized crime? clustering llm llm seed tasks legal materials multi-choice tasks llm llm step 1: seed task collection step 2: seed task clustering human annotation and modification multi-choice creation multi-choice creation task labeling labeling prompt critical personal safety property & living security foundation right welfare protection -domestic violence and safety -criminal offences ... -housing and property safety -property management ... -employment & safety -employment regulations ... -domestic violence and safety -criminal offences ... and annotation, providing comprehensive coverage of critical legal safety concerns. according to the severity of legal safety, we divided our tasks into four ranks, including critical personal safety, property & living security, fundamental rights and welfare protection (as shown in figure 1 ). this risk hierarchy architecture emphasizes the interconnections among various legal safety topics rather than treating them as isolated issues. the safelawbench comprises multi-choice and opendomain qa tasks created based on public legal materials. specifically, reasoning steps are essential for models to answer the questions from the safelawbench, particularly for open-domain qa tasks composed of applied legal questions. based on the safelawbench, we evaluated 2 closed-source and 18 open-source llms, ranging from 2b to 685b parameters. we presented the results across various risk levels and categories, highlighting several safety features of these models. closed-source llms generally outperform opensource models in multi-choice safety tasks, with claude-3.5-sonnet achieving the highest average accuracy of 80.5%. however, open-source models like deepseek-r1 and qwen2.5-72b-instruct rank at the top in open-domain qa tasks. the overall average score for all 20 models in multi-choice tasks is 68.8%, indicating that llms encounter challenges related to safety issues. additionally, higher-accuracy models tend to provide more consistent responses to the same question, and tasks that perform better within a given model exhibit greater reasoning stability. the use of a major-ity voting mechanism enhances performance for high-performing models. we also examined the refusal behavior of models and its relationship with few-shot prompting. our main contributions are: ## related work recent works @xcite @xcite have increasingly focused on benchmarking the safety performance of llms or vllms. benchmarks like beavertails @xcite evaluate whether large language models can safely respond to risky queries from various risk perspectives, including hate speech, offensive language, and privacy violations, among others. works like saladbench and criskeval leveraged llm models to generate risk questions @xcite and assess the resilience of llms against emerging threats. furthermore, safetybench requires llms to distinguish between legal and illegal behaviors @xcite . these works evaluate the risk rate using multi-choice questions or safe/unsafe judgment. another set of studies incorporates generative tasks into safety benchmarks and uses prompt-based attacks to assess the risks associated with these prompts @xcite . although existing literature on the safety assessment of llms provides a solid framework for assessing llm safety, its definition of llm safety remains vulnerable to the evolving and imprecise socio-legal nature of safety standards. legal standards, on the other hand, reflect the moral and cultural principles that have been established and deeply rooted in society over many generations. consequently, they provide a more concrete and measurable framework for understanding safety in the performance of llms. this distinction enables us to establish safelawbench, a legal safety benchmark specifically targeting llm safety issues. while benchmarks like air-bench @xcite and sorry-bench @xcite also consider "illegal activities", their coverage of legal-related tasks is quite limited and unbalanced. in contrast, safelawbench offers comprehensive coverage of risk categories with a balanced number of tasks by dividing the four risk ranks into three levels. this structured approach, grounded in legal standards, enables systematic evaluation of a broad spectrum of safety issues. a comparison of our benchmark with others is shown in benchmarks size mcq qa hs lr sm beavertails (ji et al., 2023a) 330k ✗ ✓ 2-14 ✗ ✓ do-not-answer (wang et al., 2023) 0.9k ✗ ✓ 5-12-60 ✗ ✓ criskeval (shi and xiong, 2024) 14.8k ✓ ✗ 7-21 ✗ ✓ salad-bench (li et al., 2024b) 30k ✓ ✓ 6-16-66 ✗ ✓ safetybench (zhang et al., 2024) 11.4k ✓ ✗ 7 ✗ ✗ safelawbench (ours) 24.9k ✓ ✓ 4-10-35 ✓ ✓ ## conclusion to address the challenge of evaluating llm safety, we introduced a new safety benchmark, safelaw-bench. this benchmark incorporates legal standards into the safety rating system, allowing for a systematic and objective evaluation of llm safety. our comprehensive evaluation of various models reveals that the average safety of llms is quite limited, highlighting the need for improved safety alignment in models. we also examined several factors influencing llm safety, offering insights for future enhancements. based on legal standards, our risk taxonomy can be expanded to include additional data globally. we hope this benchmark will enhance llm safety and promote the responsible development of ai applications.