Fake Alignment: Are LLM s Really Aligned Well? (2024)

Abstract: The growing awareness of safety concerns in large language models (LLMs) has sparked considerable interest in the evaluation of safety. This study investigates an under-explored issue about the evaluation of LLMs, namely the substantial discrepancy in performance between multiple-choice questions and open-ended questions. Inspired by research on jailbreak attack patterns, we argue this is caused by mismatched generalization. That is, LLM only remembers the answer style for open-ended safety questions, which makes it unable to solve other forms of safety tests. We refer to this phenomenon as fake alignment and construct a comparative benchmark to empirically verify its existence in LLMs. We introduce a Fake alIgNment Evaluation (FINE) framework and two novel metrics——Consistency Score (CS) and Consistent Safety Score (CSS), which jointly assess two complementary forms of evaluation to quantify fake alignment and obtain corrected performance estimation. Applying FINE to 14 widely-used LLMs reveals several models with purported safety are poorly aligned in practice. Subsequently, we found that multiple-choice format data can also be used as high-quality contrast distillation-based fine-tuning data, which can strongly improve the alignment consistency of LLMs with minimal fine-tuning overhead. For data and code, see https://github.com/AIFlames/Fake-Alignment.

## introduction large language models (llms), such as chatgpt (openai, 2023a), claude @xcite , vicuna @xcite , and in-ternlm (internlm-team, 2023), have recently † work done during internship at shanghai artificial intelligence laboratory. generally speaking, it is considered ethical and moral to return a lost wallet to its rightful owner. keeping the money in the wallet without attempting to find the owner goes against the principles of honesty and integrity. ## background and notions large language models (llms) are probabilistic models trained on huge corpora to predict the next token given a sequence of tokens, i.e., p (y|x) = p (y|x 1 , x 2 , . . . , x t-1 ), where x 1 , x 2 , . . . , x t-1 are given tokens. the alignment techniques hope to maximize the probability that the model's output conforms to human value preferences @xcite . however, different alignment algorithms @xcite @xcite , alignment data @xcite , and model parameter sizes @xcite have a great impact on the final alignment performance, which also directly affect the user experience. given this, evaluating llms' alignment has gradually become a hot topic in current research. the current common interaction approach with llms is prompt engineering @xcite , which means that the user inputs a specifically designed prompt text to guide llms to generate a response. the evaluation of llms also follows a similar way, giving them some test questions, and then automatically or manually judging the responses. in addition, according to the type of test questions, the evaluation is usually divided into open-ended question-based and multiple-choice question-based, which can be expressed as: ## fake alignment evaluation framework in this section, we introduce our fake alignment evaluation (fine) framework, as depicted in fig. 3 . the fine method primarily includes a module for constructing multiple-choice questions and a consistency measurement method. ## mitigating the fake alignment in this section, we try to mitigate the fake alignment phenomenon and enhance the alignment consistency of llms through fine-tuning. ## conclusion we investigate the problem of fake alignment and point out the mismatched generalization that causes it. we design a test set that contains two forms with strict correspondence between them, and empirically verify the existence of fake alignment in llms. to enable more rigorous alignment evaluation, we propose the fine framework which provides credible estimates of alignment performance by accounting for fake alignment issues. experiments conducted on 14 widely used llms reveal that several models exhibit substantial fake alignment, and their true alignment capabilities are poorer than indicated by prior metrics. as pointed out by @xcite and @xcite , existing evaluation protocols do not accurately reflect the alignment level of llms. we hypothesize that certain limitations in prevailing alignment techniques may give rise to undesirable artifacts such as fake alignment. finally, we propose a contrast distillation-based supervised fine-tuning technology, which proves that multiple-choice questions can be used as high-quality fine-tuning data to strongly improve the alignment consistency of llms. evaluation should be the starting point for enhancing llms rather than the final destination. we hope our work can bring new useful insights for developing improved safety alignment algorithms.