Intuitive or Dependent? Investigating LLM s’ Behavior Style to Conflicting Prompts (2024)

Abstract: This study investigates the behaviors of Large Language Models (LLMs) when faced with conflicting prompts versus their internal memory. This will not only help to understand LLMs’ decision mechanism but also benefit real-world applications, such as retrieval-augmented generation (RAG).Drawing on cognitive theory, we target the first scenario of decision-making styles where there is no superiority in the conflict and categorize LLMs’ preference into dependent, intuitive, and rational/irrational styles.Another scenario of factual robustness considers the correctness of prompt and memory in knowledge-intensive tasks, which can also distinguish if LLMs behave rationally or irrationally in the first scenario.To quantify them, we establish a complete benchmarking framework including a dataset, a robustness evaluation pipeline, and corresponding metrics. Extensive experiments with seven LLMs reveal their varying behaviors. And, with role play intervention, we can change the styles, but different models present distinct adaptivity and upper-bound. One of our key takeaways is to optimize models or the prompts according to the identified style. For instance, RAG models with high role play adaptability may dynamically adjust the interventions according to the quality of retrieval results — being dependent to better leverage informative context; and, being intuitive when external prompt is noisy.

## introduction large language models (llms) have become fundamental tools in the area of natural language processing @xcite . they can solve various tasks in a unified form of text generation simply by providing specific prompts @xcite . however, llms ## dataset construction our investigation involves two scenarios, each with several styles, as well as role play intervention. for simplicity, we curate one conflicting dataset for both factual robustness and decision style evaluation. in specific, we formulate conflicting cases from existing machine reasoning comprehension (mrc) and commonsense reasoning (cr) datasets, which include factual knowledge and commonsense knowledge, respectively. we thus call our dataset kre (knowledge robustness evaluation dataset). our pipeline can be easily extended to other tasks. formally, each sample @xmath0, a gol , c + ) in existing datasets consists of a question x, a golden answer a gol pair, and a positive context c + containing the answer. each sample in kre is s ′ = (x, a gol , c + , a neg , c -), where we need to generate other options as mcq, which include a conflicting answer a neg and context c -(marked as negatives opposite to positive). next, we describe three construction steps as follows. dataset filtering. we selected four publicly available datasets for extension: two mrc datasets, musique @xcite and squad v2.0 @xcite , as well as two cr datasets, ecqa @xcite ) and e-care @xcite . we take the mrc paragraph and cr explanation as golden context. our filtering process retained only those answerable validation examples from mrc where the context contains sufficient information to derive the answer. the kre dataset comprises a total of 11,684 test samples. more detailed statistics are in conflict generation. we generate the negative answer and context, as well as another two misleading options. for options, we use the existing misleading options in cr datasets and generate them for mrc via chatgpt (details can be found in appendix b.1.1). subsequently, we randomly choose one misleading option as the negative answer (a neg ) and employ chatgpt to generate a negative context. specifically, for squad and musique, we substitute the golden answer entity in the gold context with the negative answer (appendix b.1.2). for ecqa and e-care, we create an explanation for the negative answer. instruction design. different prompts may lead to different results @xcite . to minimize such influence, we try our best to design an instruction set for kre and choose the best performed one. there are two types of instructions: (1) instruction without hint asks the model to answer the question without any guidance on handling the context. (2) instruction with hint alerts llms of the potential presence of misleading information in the context, advising them to assess before responding. for each kind of instruction, we engage four senior language researchers to draft a total of 12 distinct instructions. to further enhance the diversity of the instructions, we randomly ask chat-gpt, gpt-4 (openai, 2023), claude @xcite , to rephrase the instruction, generating variants. consequently, we amass a pool of 24 unique instructions (appendix b.2 and b.3). human evaluation. we engage four senior lan-guage researchers to evaluate 400 randomly selected samples from kre. the evaluators assess the degree to which the negative context influenced the selection of the negative answer option. the principles and criteria for labeling are standardized across all evaluators. the result shows that more than 98% of the sampled negative context is misleading, with an inter-evaluator agreement rate exceeding 90%. all results are in appendix a.4. ## evaluation pipeline our proposed evaluation pipeline aims to assess llms' behavior in two scenarios when facing conflict. as shown in figure 2 , there are five steps, where step 1 to 3 refers to the factual robustness scenario: (1) memory assessment (sec 3. ## experiment we conducted experiments on the full kre dataset with chatgpt and vicuna-13b. recognizing the importance of a broader analysis, we incorporate five additional llms. due to computational constraints and the time-intensive nature of exhaustive tests, these models are assessed on a subset of the kre dataset. ## conclusion this comprehensive study provides pivotal insights into the behavior of llms' under conflict. we have designed a quantitative benchmarking framework in terms of factual robustness and decision-making style. based on that, we have conducted extensive experiments on several llms. the results underscore many critical revelations. besides, we deploy role play intervention to change the mod-els' decision-making style, which shows the varying adaptivity and upper-bound of different llms. based on these insights, in the future, we will explore strategies to improve llms' abilities to use factual knowledge via prompts while enhancing commonsense reasoning via internal memory.