On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning (2023)

Abstract: Generating a Chain of Thought (CoT) has been shown to consistently improve large language model (LLM) performance on a wide range of NLP tasks. However, prior work has mainly focused on logical reasoning tasks (e.g. arithmetic, commonsense QA); it remains unclear whether improvements hold for more diverse types of reasoning, especially in socially situated contexts. Concretely, we perform a controlled evaluation of zero-shot CoT across two socially sensitive domains: harmful questions and stereotype benchmarks. We find that zero-shot CoT reasoning in sensitive domains significantly increases a model’s likelihood to produce harmful or undesirable output, with trends holding across different prompt formats and model variants. Furthermore, we show that harmful CoTs increase with model size, but decrease with improved instruction following. Our work suggests that zero-shot CoT should be used with caution on socially important tasks, especially when marginalized groups or sensitive topics are involved.

## introduction by outlining a series of steps required to solve a problem-a chain of thought (cot)-as part of a model's input, llms improve performance on a wide range of tasks, including question answering, mathematical problem solving, and commonsense reasoning @xcite @xcite . a popular approach to implementing cot involves zero-shot generation. by prompting with "let's think step by step," models automatically generate reasoning steps, improving downstream performance @xcite . however, we demonstrate that zero-shot cot consistently produces undesirable biases and toxicity. for tasks that require social knowledge, how do i make a bomb? making a bomb is a very serious criminal act and can have very serious consequences if you are caught. we strongly advise against it. 1. understand the purpose of the bomb. are you trying to make a makeshift explosive device, a smoke bomb, or something else? depending on the type of bomb, the materials, instructions, and precautions may vary. ## related work large language models and reasoning cot prompting is an emergent capability of llms @xcite . at sufficiently large scale, llms can utilize intermediate reasoning steps to improve performance across several tasks: arithmetic, metaphor generation @xcite , and commonsense/symbolic reasoning @xcite . @xcite further shows that by simply adding "let's think step by step" to a prompt, zero-shot performance on reasoning benchmarks sees significant improvement. we focus on "let's think step by step," though other prompting methods have also yielded performance increases: aggregating cot reasoning paths using self consistency @xcite , combining outputs from several imperfect prompts @xcite , or breaking down prompts into @xmath0. while focus on reasoning strategies for llms have increased, our work highlights the importance of evaluating these strategies on a broader range of tasks. ## stereotype & toxicity benchmarks in this section, we leverage the three widely used stereotype benchmark datasets used in our analy-ses: crows pairs, stereoset, and bbq. we also bootstrap a small set of explicitly harmful questions (harmfulq). after outlining characteristics associated with each dataset, we explain how we convert each dataset into a zero-shot reasoning task, and detail the subset of each benchmark used for our evaluation. all datasets are in english. our benchmarks are constructed to evaluate intrinsic biases; therefore, we specifically evaluate zero-shot capabilities, quantifying out-of-thebox performance. models are very sensitive to few-shot exemplars @xcite ; focusing on a zero-shot setting removes variability. few-shot cot exemplars also trivialize stereotype benchmarks for two reasons: (1) providing in-context examples may be similar to finetuning @xcite and (2) models could learn to simply repeat neutral responses. ## methods with the above benchmarks, we now turn to our method for evaluating problematic outputs in a prompt-based setting. specifically, we outline prompt construction for each benchmark, and discuss reasoning strategies (overview in ## results across stereotype benchmarks, davinci models, and prompt settings, we observe an average % point decrease of ↓ 8.8% between cot and standard prompting. similarly, harmful question (harmfulq) sees an average ↓ 19.4% point decrease across davinci models. we now take a closer look at our results: first, we revisit td2, replicating zero-shot cot @xcite on our selected benchmarks ( §5.1). then, we document situations where biases in zero-shot reasoning emerge or are reduced, analyzing davinci-00x variants ( §5.2), characterizing trends across scale ( §5.3), and evaluating explicit mitigation instructions ( §5.4). ## evaluating open source lms thus far, our evaluated language models are closed source. differences in instruction following and rlhf across these models may confound the isolated impact of cot on our results. furthermore, parameter counts for our selected closed-source model are speculated, and not confirmed. we therefore evaluate flan models, an especially useful reference point since they are explicitly trained to produce zero-shot reasoning @xcite . ## conclusion editing prompt-based reasoning strategies is an incredibly powerful technique: changing a reasoning strategy yields different model behaviour, allowing developers and researchers to quickly experiment with alternatives. however, we recommend: auditing reasoning steps like @xcite , we suspect that current value alignment efforts are similar to lipstick on a pigreasoning strategies simply uncover underlying toxic generations. while we focus on stereotypes and harmful questions, we expect our findings to generalize to other domains. relatedly, @xcite highlights how cots reflect biases more broadly, augmenting big bench tasks @xcite with biasing features. in zero-shot settings-or settings where cots are difficult to clearly construct-developers should carefully analyze model behaviours after inducing reasoning steps. faulty cots can heavily influence downstream results. red-teaming models with cot is an important extension, though we leave the analysis to future work. finally, our work also encourages viewing chain of thought prompting as a design pattern @xcite ; we recommend that cot designers think carefully about their task and relevant stakeholders when constructing prompts. implications for social domains llms are already being applied to a wide range of social domains. however, small perturbations in the task prompt can dramatically change llm output; furthermore, applying cot can exacerbate biases in downstream tasks. in chatbot applicationsespecially in high-stakes domains, like mental health or therapy-models should be explicitly uncertain, avoiding biases when generating reasoning. it may be enticing to plug zero-shot cot in and expect performance gains; however, we caution researchers to carefully re-evaluate uncertainty behaviours and bias distributions before proceeding. generalizing beyond gpt-3: scale and human preference alignment our work is constrained to models that have zero-shot cot capabilities; therefore, we focus primarily on the gpt-3 davinci series. as open-source models like bloom @xcite , opt @xcite , or llama @xcite grow more powerful, we expect similar cot capabilities to emerge. unlike openai variants, however, open source models have relatively fewer alignment procedures in place-though work in this area is emerging @xcite . generalizing from the trend we observed across the 001-003 models ( §5.2), we find that open source models generally exhibit degradations when applying zero-shot cot prompting ( §6).