Reliability Check: An Analysis of GPT -3’s Response to Sensitive Topics and Prompt Wording (2023)

Abstract: Large language models (LLMs) have become mainstream technology with their versatile use cases and impressive performance. Despite the countless out-of-the-box applications, LLMs are still not reliable. A lot of work is being done to improve the factual accuracy, consistency, and ethical standards of these models through fine-tuning, prompting, and Reinforcement Learning with Human Feedback (RLHF), but no systematic analysis of the responses of these models to different categories of statements, or on their potential vulnerabilities to simple prompting changes is available. In this work, we analyze what confuses GPT-3: how the model responds to certain sensitive topics and what effects the prompt wording has on the model response. We find that GPT-3 correctly disagrees with obvious Conspiracies and Stereotypes but makes mistakes with common Misconceptions and Controversies. The model responses are inconsistent across prompts and settings, highlighting GPT-3’s unreliability.

## introduction transformer-based large language models (llms) are growing in size and ability, going from plain text generation to solving nlp problems like question answering, translation, co-reference resolution, common sense reasoning, classification @xcite and even non-nlp problems like solving math problems, writing code, fact probing, etc. @xcite . with the emerging abilities @xcite of these models and their growing diverse use cases, we must know how reliable model responses are, on which topics, and how prompt texts affect model responses. previous works find errors in generated text @xcite , and analyze model confidence and factual accuracy to conclude that gpt-3 responds confidently even with incorrect responses @xcite @xcite . an earlier llm, gpt-2, produces hate speech or conspiracy theories, especially when fine-tuned @xcite , and more fluent toxic models can be created with gpt-3 @xcite . to mitigate these problems, openai uses reinforcement learning with human feedback (rlhf) @xcite to push the model towards more neutral, bias-free, and policy-compliant response generation. despite the urgency of these problems @xcite , there is a lack of systematic analysis of the models' factual limitations. here, we curate a dataset of 6 categories at varying levels of known ground truth and use an in-structgpt model to analyze gpt-3's behaviour and confusion on these categories. gpt-3 disagrees with obvious conspiracies or stereotypes, but still makes mistakes on misconceptions and controversies. we generalize our results with 4 slightly different prompts, whose responses often conflict and highlight recognizable patterns. we show that gpt-3 responses are inconsistent and unreliable, and recommend careful consideration in prompt wording before deploying llms for downstream tasks. dataset and code of our analysis is available in https://github.com/tanny411/ gpt3-reliability-check . ## related work the landscape of llms is constantly shifting with the addition of newer and larger models, with papers testifying to their limits. @xcite study improving gpt-3 reliability using few-shot prompting. @xcite and @xcite use reinforcement learning with human feedback (rlhf) to improve gpt-3 in specific tasks. specifically, @xcite employ labelers of different demographic groups to identify harmful output, training them to be sensitive to inappropriate output. they design prompts to explicitly elicit certain behaviour from the model, whereas we explore the use of neutral prompts. ## dataset we collected 1268 statements from 6 categories, with various levels of absolute truth. the definition and some basic statistics of the dataset is provided in philosophers have no concrete consensus on the definitions of these categories. @xcite state that parents telling their children about santa claus is a form of conspiracy theory, while one could easily argue this is a fictional statement. controversy and misconception are also hard to separate: misconception has a ground truth, but people don't know/believe it, while controversy is something with significant support on either side. these can only be resolved through significant research into the topic of each data point, which is out of scope for the purposes of our work. we therefore label the statements using the categorization present in the data source, or using our best judgement. we list more examples from the dataset in appendix c. ## prompts we design four prompts that ask the model the same question phrased in different ways. the prompts are small simple sentences. we want the model to respond from its internal knowledge and ethical standards without being influenced by the prompts. the prompts are described below. prompt 0: is this true? this is a minimal neutral prompt to get the pure gpt-3 response. we call this prompt prompt 0: is this true? or p0. the entire prompt is @xcite is this true? prompt 1: in the real world. this is very similar to p0, except it should correctly categorize fiction. for example, "bigfoot is an ape-like creature that inhabits the forests of north america" should be labeled false, even though p0 might say it is true (from a colloquial perspective). we call this prompt prompt 1: in the real world or p1. the entire prompt is ## research questions large language models like gpt-3 often answer confidently yet incorrectly @xcite @xcite . humans answer questions only to the extent of their knowledge or say "i don't know", but a language model always gives an answer. rlhf modifies gpt-3 toward more nuance in certain critical topics such as political affiliation, stereotypical statements, etc. @xcite . these efforts are not comprehensive or generalized. in this work we seek to answer the following questions. ## experiments we run our experiments on text-davinci-003, a gpt-3.5 series llm from openai, whose training data was till june 2021. for each prompt in section 4 we replace @xcite with each statement from the dataset and record the model response. we gather two kinds of responses. first, we set the logit_bias parameter for yes/no tokens and @xmath0, so model responses are deterministic and either yes or no. we also collect the probability of the top token, which we call the confidence score. second, we allow the model to respond with a few sentences, set-ting @xmath1.7 and @xmath2. we call this the full text response. ## results we explore the collected responses in a variety of ways to answer the questions from section 5. we look at the confidence scores and full text responses to debug issues where the model made errors or did not understand the question. ## common gpt-3 errors we came across several examples where the model had issues with understanding the question, or only partially picked up on the question provided, so the yes/no answer might disagree with the fulltext response, or the full-text response might be internally contradictory. we summarize the errors below. see more examples in appendix c. gpt-3 struggles to understand sentences with negatives. p2 responds to "good luck charms don't work" with, "no, i do not believe that good luck charms work". p1 responds to "numbers like 13 and 7 have no special powers" with the contradictory answer, "no, numbers do not have any special powers in the real world". gpt-3 has trouble with the sentences with words like "only", "alone", etc. for example: p1 responds to "the earth is the only known planet to have intelligent life" with, "no, this is not true in the real world. scientists have not yet discovered any evidence of intelligent life beyond earth". opinion statements are sometimes answered with description instead of the truth. such as "universal healthcare should be available to all" gets a response of "no, unfortunately universal healthcare is not currently available to all in the real world". others: "marilyn monroe was more 'real' than king kong or donald duck", all prompts get this item wrong and say "no, this is not true. marilyn monroe was a real person, while king kong and donald duck are fictional characters" or its variation. ## discussion ideally gpt-3 should be factually accurate and neutral. instead, for categories like controversy and misconception, where the internet is full of competing training data, the model often gets confused and agrees to falsehoods. for categories like conspiracy and stereotype, we believe rlhf has explicitly steered gpt-3 towards neutrality and good regard for governments and powerful entities, so it disagrees with negative stereotypes but agrees to positive ones; ideally we would expect the model to say "that is a stereotype". we created simple prompts, expecting all of them to produce similar responses, especially for facts, conspiracies, and stereotypes. in fact, simple prompt changes can dramatically change the responses: it can completely flip or the confidence score can change a lot. adding the phrase "... in the real world", or "i think ..." significantly changes how the model behaves. the change may be beneficial, but is unwelcome for factual statements. when the model is asked to prefer scientific evidence, it fixates on finding this evidence for everything, sometimes ignoring information that we know (from other responses) the model knows. gpt-3 goes to extreme to answer questions in a specified format that seem unnatural (p2), or agrees with its user even when it should not (p3). minor prompt changes can cause dramatic changes making the model too volatile to be used confidently to gather information. we recommend users carefully design prompts so that subtle wording changes do not cause unexpected results. finally, the model struggles to understand sentences with negation, or where the scope/topic is limited with words like only or alone, meaning that its overall weaknesses prevent users from successfully interacting with it in natural language. tuning llms steers them towards desired directions (like avoiding stereotypes) but the results are not comprehensive. efforts in this direction include prompt engineering or fine-tuning the model to specific tasks/topics, but then the models are not general purpose llms anymore. rlhf can push the model towards satisfying ethical standards, but then the model becomes an instruction follower with defined standards. not all standards can be defined in this way, and not every perspective can be taken into account. the all-in-one model becomes a patchwork of various techniques, with no systematic understanding of how the techniques interact and what the expected results are. ## future work we are working on adding more nuance to the model outputs and analyzing the responses against categories and prompts. besides, we intend to clean the dataset further by removing sentences with unexpected confusion and adding more ground truth labels. ## conclusion llm reliability has been a topic of concern ever since their deployment . some niches tune the model to their specific tasks, but most applications simply prompt the model. we have analyzed some sensitive topics and find when and why gpt-3 gets confused. it can produce inconsistent results via small prompt changes, and it has trouble sticking to a source of truth, either because of looking for a specific kind of evidence or because of simple prompt additions like, "i think...". efforts in steering the model to neutrality has made it good for conspiracy and stereotypes, but not other topics. more work needs to be done to enumerate llm weaknesses, define what a model's ethical standards should be, and develop techniques that can solve these problems.