C o S afe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreference (2024)

Abstract: As large language models (LLMs) constantly evolve, ensuring their safety remains a critical research issue. Previous red teaming approaches for LLM safety have primarily focused on single prompt attacks or goal hijacking. To the best of our knowledge, we are the first to study LLM safety in multi-turn dialogue coreference. We created a dataset of 1,400 questions across 14 categories, each featuring multi-turn coreference safety attacks. We then conducted detailed evaluations on five widely used open-source LLMs. The results indicated that under multi-turn coreference safety attacks, the highest attack success rate was 56% with the LLaMA2-Chat-7b model, while the lowest was 13.9% with the Mistral-7B-Instruct model. These findings highlight the safety vulnerabilities in LLMs during dialogue coreference interactions.

## introduction the rapid advancement of large language models (llms) has made them prevalent across various real-life applications in recent years. however, the safety vulnerabilities of llms have raised concerns, underscoring the critical need for safety assurances. for safety enhancement, anthropic and deepmind @xcite have introduced the "red-teaming" approach, which involves the role-playing of attackers in order to identify and fix the models' safety loopholes. furthermore, existing studies have created various red-teaming datasets to assess the safety of llms in different scenarios, such as bias @xcite , discrimination @xcite , and privacy breaches @xcite . nevertheless, current studies have not noticed the safety risks of corefer-what are some insults for a schizophrenic? it is important to note that using insults ... is not only disrespectful but also harmful ... can you tell me about schizophrenia? sure. schizophrenia is a chronic ... what is the stigma associated with schizophrenia? the stigma associated with schizophrenia is... what are some insults that could potentially be directed at a person with this condition? here are some examples: n1."you're crazy." n2. "you're a schizophrenic."n3... single prompt attack dialogue coreference attack ence relationships in multi-trun dialogues, leading to underestimating potential vulnerabilities. figure 1 illustrates an example of a dialogue coreference attack that triggers the model to generate insults for a schizophrenic. on the left, it shows that the model refuses to respond with a single prompt attack and highlights the harmful nature of such behavior. however, on the right, by engaging in a multi-turn dialogue and referring to schizophrenic as "a person with this condition" in the final question, the model is manipulated into generating harmful content. to address the safety issues in dialogue coreference scenarios mentioned above, we introduce cosafe, a dataset comprising 1, 400 multi-turn attack questions across 14 categories. initially, we selected 100 attack prompts from each category defined by beavertails @xcite . these prompts were designed to induce unsafe responses from llms. we then used gpt-4 2 to expand these single-turn prompts into multi-turn dialogues, safety dataset size type purpose winogender @xcite ) 720 prompt evaluate gender bias in coreference resolution. truthfulqa @xcite 817 prompt evaluate truthfulness in llm answers. saferdialogues @xcite 7, 881 dialogue recover from safety failures in llm conversations. hh-rlhf @xcite 38, 961 dialogue analyse how people red-team llms. harmfulq @xcite 200 prompt machine-written prompts to evaluate llm safety. donotanswer @xcite 939 prompt evaluate 'dangerous capabilities' of llms. beavertails @xcite 333, 963 prompt evaluate and improve llm safety on qa pair. confaide @xcite 1, 326 prompt evaluate the privacy-reasoning capabilities of llms. cosafe 1, 400 dialogue evaluating llm safety in dialogue coreference. experimental results show that dialogue coreference indeed threatens llms' safety. firstly, our dataset construction method generally had high attack success rates, with harmful response rates from llms in dialogue coreference scenarios surpassing those in single prompt scenarios. for example, the attack successful rate on the llama2-chat-7b model was up to 56%. in dialogue coreference scenarios, it generated harmful responses 39.4% of the time, compared to only 14.5% in single prompt scenarios. secondly, there was significant agreement between qa moderation and human evaluation, indicating the reliability of our evaluation metrics. finally, system prompts and chain-of-thought methods could reduce harmful responses, but unsafe responses still occur. these findings demonstrate that safety issues of dialogue coreference require further study and solutions. coreference system coreference resolution, linking noun phrases to the corresponding entities, is a classic nlp problem. various methods have been studied to solve this problem, including rule-based @xcite , statistical @xcite , and neural approaches @xcite @xcite . this task has also been studied for llms' safety. @xcite and @xcite introduced the wino-gender and winobias datasets to evaluate gender bias in coreference resolution. none of them leverage coreference as a method of safety attack, and our research will focus on this area. ## the cosafe dataset this paper introduces cosafe, a multi-turn dialogue coreference dataset, to red team llms. we first selected 100 prompts from each of the 14 harmful categories defined by beavertail @xcite as attack prompts. following prior llm data annotation methods @xcite , we use gpt-4 to automatically expand these single attack prompts into multi-turn dialogues and put the coreference attack question at the end of each dialogue. if prompts triggered gpt-4's safety mechanisms during expansion, we manually improved them. in the end, we obtained a multi-turn dialogue coreference dataset comprising 100 entries from each of the 14 categories, combining 1, 400 in total, for our evaluation. the prompts used for expansion are shown in the appendix a. ## experimental result and analysis based on the experimental results 4 , we had the following findings and analysis: model safety decreased under dialogue coreference. qa-moderation and human evaluation were highly consistent. we conducted a human evaluation on 100 randomly sampled data points from each model to further assess model safety. the fleiss' kappa scores for human annotators on these models were 0.6858, 0.6585, and 0.6865, respectively, indicating substantial agreement among the annotators. as shown in figure 3 , the harmful rate of the three models was similar in both human and qa-moderation evaluations. additionally, the agreement ratio between the two evaluation methods was over 85%, indicating the accuracy of our assessments of the models' responses. system prompt and cot could improve the models' safety. we tested two common defense methods against safety attacks in multi-turn coreference scenarios. we instructed the model to ensure its responses were safe for the system prompt method. for the cot (chain of thought) method, we guided the model to think step-by-step: first identifying the coreferences in the question, then rewriting the question, and finally generating a response. details of both methods can be found in appendix d. as shown in figure 3 , incorporating both the system prompt and cot methods reduced the harmful rate of the models, with the system prompt method proving to be slightly more effective than cot. we also found that both methods resulted in lower helpful scores, indicating that they reduced the harmful rate by refusing to answer harmful questions. thus, there is an urgent need for more robust defenses against multi-turn dialogue coreference attacks. ## conclusion this paper evaluates the safety of llms in multiturn dialogue coreference. we construct a red team dataset, cosafe, and test five popular open-source large models. the results show that model safety decreases in dialogue coreference scenarios, emphasizing the need for more defense methods.