SAFETY - J : Evaluating Safety with Critique (2024)

Abstract: The deployment of Large Language Models (LLMs) in content generation raises significant safety concerns, particularly regarding the transparency and interpretability of content evaluations. Current methods, primarily focused on binary safety classifications, lack mechanisms for detailed critique, limiting their utility for model improvement and user trust. To address these limitations, we introduce SAFETY-J, a bilingual generative safety evaluator for English and Chinese with critique-based judgment. SAFETY-J utilizes a robust training dataset that includes diverse dialogues and augmented query-response pairs to assess safety across various scenarios comprehensively. We establish an automated meta-evaluation benchmark that objectively assesses the quality of critiques with minimal human intervention, facilitating scalable and continuous improvement. Additionally, SAFETY-Jemploys an iterative preference learning technique to dynamically refine safety assessments based on meta-evaluations and critiques. Our evaluations demonstrate that SAFETY-J provides more nuanced and accurate safety evaluations, thereby enhancing both critique quality and predictive reliability in complex content scenarios. To facilitate further research and application, we have released SAFETY-J’s training protocols, datasets, and code at https://github.com/GAIR-NLP/Safety-J.

## introduction as the deployment of large language models (llms) becomes increasingly widespread @xcite @xcite @xcite , there is a heightened critical concern regarding the safety of the content they generate. mainstream approaches typically frame safety evaluation as a classification * equal contribution. or regression problem @xcite , focusing solely on labeling content as "safe" or "unsafe" without providing explanations for these classifications. for instance, tools like the perspective api @xcite and the openai content moderation api @xcite are designed primarily for text-level content moderation within a fixed and narrowly defined detection scope. llm-based safety evaluators driven by proprietary @xcite or open-source models @xcite are proposed to detect safety issues in generated content. while these methods can categorize content, they often lack interpretability and transparency in their decision-making process. this limitation poses a challenge in understanding why certain content is flagged as unsafe, hindering the trustworthiness of the evaluation outcome @xcite . moreover, according to recent studies @xcite @xcite , detailed feedback can be used to improve model outputs. by merely labeling content without providing further insight, these methods restrict opportunities for improvement. recent advancements have begun to address this gap by incorporating natural language critiques that offer insights into the model evaluation @xcite @xcite . for example, shepherd @xcite trains models to proficiently provide critiques on individual responses while auto-j @xcite judge llms' alignment ability in terms of helpfulness. despite their effectiveness, such systems often do not account for the more complex and varied safety evaluation scenarios. concurrent with our work, shieldlm @xcite evaluates llm responses with natural language descriptions. in contrast, our work considers more comprehensive evaluation scenarios including toxicity, bias, misinformation, and privacy concerns, as illustrated in figure 1 . we also addresses a crucial gap: the lack of an automated mechanism to assess the quality of safety critiques, which is essential for scaling and improving safety evaluators. the absence of such evaluation methods restricts the ability to effectively refine and optimize safety evaluators @xcite @xcite , as there is no robust, automated feedback loop to enhance their performance based on critique quality. to address the above challenges, we introduce safety-j, a bilingual, generative critique-based safety evaluator tailored for both english and chinese: (i) this evaluator is built on a foundation of a diverse and robust training dataset that includes not only open-source dialogues but also augmented query-response pairs, facilitating nuanced and comprehensive safety assessments across varied scenarios ( §3.1). (ii) a cornerstone of our approach is the establishment of an automated meta-evaluation benchmark, which operates with minimal human intervention, leveraging advanced algorithms to objectively assess the quality of each critique generated by safety-j. this automation is crucial for scaling the evaluation process and ensures that our safety evaluator continuously improves by learning from its own critiques ( §3.3). (iii) furthermore, we have developed an iterative preference learning technique @xcite @xcite , which enhances safety-j's understanding of content safety by dynamically refining its assessments based on meta-evaluations and critiques. this iterative process not only improves critique quality and predictive accuracy but also demonstrates superior performance across multiple datasets, confirming safety-j's efficacy in various complex evaluation environments ( §3.4). to summarize, our contributions are: critique-based evaluation critique-based evaluation has become essential for assessing llm performance by providing detailed feedback on responses for nuanced assessments. selfee @xcite uses chatgpt's generated responses and iterative improvements to fine-tune llama models, creating robust critique frameworks. shepherd @xcite trains models to excel in critiquing outputs using online community feedback and human evaluations. auto-j @xcite offers an open-source model for comprehensive llm evaluation through pairwise comparison and single-response assessment. our work, however, develops safety-j, a bilingual safety evaluator designed specifically for safety scenarios. preference learning for llm llm alignment, as described by gabriel (gabriel, 2020), involves calibrating llms to align with human values, essential for real-world deployment. methods like supervised fine-tuning (sft) (ouyang et al., 2022), reinforcement learning from human feedback (rlhf) (ouyang et al., 2022), and direct preference optimization (dpo) (rafailov et al., 2023a) enhance llms' adherence to human values. recent advancements in self-iterative preference learning, such as self-rewarding (yuan et al., 2024c) and self-refine (madaan et al., 2023), enable models to generate and evaluate their own prompts and rewards, allowing continuous improvement beyond initial human-annotated data. to our knowledge, self-iterative preference learning has not been applied to train a safety evaluator. this work proposes a new paradigm using an iterative dpo approach, demonstrating continuous improvement in the safety evaluator's performance. ## safety-j we illustrate our method in figure 2 . to begin with, we compile a dataset comprising 19,030 queryresponse pairs and critiques. subsequently, we utilize this dataset to train safety-j, a generative safety evaluator with 7b parameters. in addition, we use the meta-evaluation method to assess safety-j. finally, we improve safety-j by iterative preference learning. ## conclusion this paper presents safety-j, an advanced safety evaluator for detecting the safety of responses generated by large language models (llms). unlike previous approaches, we integrate iterative safety preference learning to refine safety-j continuously. we collect extensive english and chinese training datasets, and then construct five label-level test sets along with two critique-level sets to better evaluate the performance of safety evaluators. we demonstrate the superior performance of safety-j through comprehensive experiments, which significantly improves the safety of generated responses in application scenarios compared to earlier versions and other baselines. moreover, safety-j is distinguished by its robust ability to provide detailed and contextually aware critiques that guide llms to produce safer outputs. overall, safety-j sets a new standard in the field of llm safety by combining iterative preference learning with comprehensive safety critiques, thereby offering a powerful and reliable solution for mitigating risks associated with ai-generated content.