Can You Follow Me? Testing Situational Understanding for C hat GPT (2023)

Abstract: Understanding sentence meanings and updating information states appropriately across time—what we call “situational understanding” (SU)—is a critical ability for human-like AI agents. SU is essential in particular for chat models, such as ChatGPT, to enable consistent, coherent, and effective dialogue between humans and AI. Previous works have identified certain SU limitations in non-chatbot Large Language models (LLMs), but the extent and causes of these limitations are not well understood, and capabilities of current chat-based models in this domain have not been explored. In this work we tackle these questions, proposing a novel synthetic environment for SU testing which allows us to do controlled and systematic testing of SU in chat-oriented models, through assessment of models’ ability to track and enumerate environment states. Our environment also allows for close analysis of dynamics of model performance, to better understand underlying causes for performance patterns. We apply our test to ChatGPT, the state-of-the-art chatbot, and find that despite the fundamental simplicity of the task, the model’s performance reflects an inability to retain correct environment states across time. Our follow-up analyses suggest that performance degradation is largely because ChatGPT has non-persistent in-context memory (although it can access the full dialogue history) and it is susceptible to hallucinated updates—including updates that artificially inflate accuracies. Our findings suggest overall that ChatGPT is not currently equipped for robust tracking of situation states, and that trust in the impressive dialogue performance of ChatGPT comes with risks. We release the codebase for reproducing our test environment, as well as all prompts and API responses from ChatGPT, at https://github.com/yangalan123/SituationalTesting.

## introduction understanding the meaning of language inputs and their impact on information states is essen-tial for building a communicative human-like ai agent (e.g., @xcite ). this capability requires an agent to know truth conditions @xcite of a given input, and how that input updates the context over time @xcite . for instance, the natural language input "open box-4, obtain key-1" in a game environment should assert updates "opened(box-4) = true" and "obtained(key-1) = true" in the agent's representation of the environment state. we describe this ability as situational understanding (su), as the agent needs to ground language in situations and understand situational changes. recent research indicates that despite the tremendous success of large language models (llms) (e.g., @xcite ), these models still fail to understand situational changes, and cannot serve as human-like agents to accomplish realworld tasks. for example, evidence suggests that models fail to detect sarcasm expressed in underlying context @xcite , infer incorrect logic states when context changes in games @xcite and fail to track entity state changes in a discourse @xcite . while these works have shown important limitations in llms, it remains unclear why these models show these limitations, and there is less work that shows the extent to which these limitations persist in more recent, chat-trained models like chatgpt. in this work we seek to shed light on both of these questions. to test the situation tracking ability of chatgpt and other chat models, we design a synthetic environment for controlled testing of models' ability to follow instructions and maintain consistent understanding of situation states. we design a multi-stage testing framework, and include various controls to avoid models relying on mappings memorized from pre-training rather than doing real-time state tracking. our environment allows for flexible testing under different conditions, as well as controlled follow-up analyses to enable closer examination of causal factors. we apply our tests primarily to chatgpt, which we find to outperform other models fine-tuned on chats. our results show that chatgpt's performance reflects a failure to retain coherent and correct environment states across time, despite the simplicity of the task and the fact that chatgpt has access to the full dialogue history in its input window. our follow-up analyses suggest that a major contributor to this degradation is failure to retain prior states in memory, as well as susceptibility to spurious hallucinated updates (which can also contribute to accidentally correct outputs). our findings suggest overall that chatgpt lacks the ability to track situation state changes robustly. ## related work situational understanding ability is essential for building agent-like real-world intelligent systems. in comprehensive benchmarks, big-bench-hard @xcite ) finds that for gptfamily models and palm 540b models @xcite , even equipped with the stateof-the-art chain-of-thought prompting, still fail on tasks that require situational information (e.g., to detect sarcasm). helm @xcite also points out llms can lack state tracking ability based on evaluation results for 30 models on babi @xcite environment. in synthetic and controlled environments, @xcite ; @xcite ; @xcite does probing on bart @xcite , t5 @xcite and gpt-3 using the scone @xcite and textgame @xcite datasets, and show that these models lack the ability to track and infer information states. @xcite re-analyze the results in @xcite and find that gpt-3/3.5 and flan-t5 @xcite cannot track entity state changes in a discourse. @xcite instruct gpt-2 @xcite to play chess and find it difficult to track board states per move. we build on these previous works in two primary ways. first, these works mainly use llms as feature extractors (e.g., train probe models over intermediate representations or final-layer representations), and none of them discuss whether situational understanding limitations still exist in the recent powerful and widely used chatgpt system. we investigate here whether chatgpt has the crit-ical underlying situational understanding to generate coherent, consistent and effective dialogue @xcite @xcite . second, tests used in previous work are susceptible to interference from confounds present when fine-tuning linear probes, or shortcuts in the testing environment that can be utilized to bypass situational understanding tests @xcite . we aim to address some of these concerns by introducing a number of additional controls in our tests. finally, we also carry out a more in-depth exploration of model update dynamics, to better understand causes for patterns of performance. ## building a situational testing environment what we want to test is models' ability to process situational changes conveyed by language inputs, and to maintain internal representations of the corresponding situation states. to test this, we use a synthetic box-moving environment like textgame @xcite , where we have full access to underlying game states, but eliminate complicated reward pursuing and route branching. having this kind of full-information synthetic environment is helpful to test models' evolving understanding of environment states as the input describes progressively more situational changes. ## experiments: testing model sensitivity to instructions and situation change we use our environment to test chatgpt in its ability to track situation changes, and we report the results below. we also compare against the performance of additional chat models, which we find to underperform chatgpt. those results can be found in appendix e. we try 2-shot, 3-shot and 5-shot settings (in which 2, 3 and 5 example steps with enumerated states are provided). we use 50 samples with the number of additional steps randomly sampled from {1, . . . , 10 -#(num-shots)} for each setting. we follow the official openai cookbook foot_1 to prepare the request and parse the response when interacting with chatgpt api. details are in appendix c. ## analysis of model performance in the previous section we tested the capacity of chatgpt in tracking and enumerating states within our environment, and we found that the model showed clear limitations. in this section, we analyze model performance further, to better understand the source of these limitations. ## conclusion in this paper, we propose a novel synthetic testing environment for testing situational understanding capabilities, which we apply to test chatgpt, the state-of-the-art chatbot. we instruct chatgpt to process a series of sparse environment updates across time in a dialogue history. with careful environmental designs, we reduce the possibility of data contamination and other artifacts typically introduced by traditional probing methods. we find that despite the simplicity of our task, and even with chatgpt having full access to the complete dialogue history within the input window, the model fails to retain coherent and correct environment states over time. further analysis suggests that this failure is largely because chatgpt does not have persistent in-context memory, and is susceptible to hallucinated updates. these findings indicate overall that chatgpt does not have robust situational state tracking ability. our proposed synthetic environment and the findings that it generates can have noteworthy realworld implications. first, it can diagnose the potential limitations of current chatbot systems in multi-round interactions. second, our findings also reflect a potential problem for the model's ability to follow instructions and remain consistent with rules/norms established in its context, which is especially important for responsible ai safety and human-ai alignment research. austin), hao zhu (cmu) and zhaofeng wu (mit) at the early stage of this work (names are not listed in particular order). we also thank the anonymous emnlp reviewers and chairs for providing insightful and constructive feedback to make this work more solid.