Towards Benchmarking Situational Awareness of Large Language Models:Comprehensive Benchmark, Evaluation and Analysis (2024)

Abstract: Situational awareness refers to the capacity to perceive and comprehend the present context and anticipate forthcoming events, which plays a critical role in aiding decision-making, anticipating potential issues, and adapting to dynamic circumstances. Nevertheless, the situational awareness capabilities of large language models have not yet been comprehensively assessed. To address this, we propose SA-Bench, a comprehensive benchmark that covers three tiers of situational awareness capabilities, covering environment perception, situation comprehension and future projection. SA-Bench provides a comprehensive evaluation to explore the situational awareness capabilities of LLMs. We conduct extensive experiments on advanced LLMs, including GPT-4, LLaMA3, Qwen1.5, among others. Our experimental results indicate that even SOTA LLMs still exhibit substantial capability gaps compared to humans. In addition, we thoroughly analysis and examine the challenges encountered by LLMs across various tasks, as well as emphasize the deficiencies they confront. We hope SA-Bench will foster research within the field of situational awareness.

## introduction situational awareness (sa) is crucial for facilitating decision-making, foreseeing possible problems, and adjusting to changing environments. it involves perceiving elements in the environment, comprehending their meaning, and projecting their future status @xcite . previous researches category sa into three levels: @xcite @xcite perception identifies and records relevant information and dynamics within the environment, such as targets, events, and entities. comprehension synthesizes this information to understand relationships and impacts on the current situation * corresponding author. through data analysis and correlation. projection predicts future changes and impacts of these elements, aiding in effective decision-making and planning. recently, with the rapid scaling of pretraining, llms have demonstrated outstanding performance in various downstream tasks @xcite , possessing excellent text comprehension and logical reasoning capabilities @xcite @xcite . with their exceptional capabilities in information extraction, reasoning and decisionmaking, llm-based situational awareness has attracted a lot of reaserch interest. previous research typically focuses on a single level of sa. for example, information extraction @xcite focuses on environment perception, link prediction and logical reasoning @xcite @xcite examine situation comprehension, and scenario prediction @xcite @xcite explores future projection. in summary, existing studies focus narrowly and lack comprehensive evaluations of the overall capabilities in sa, thereby presenting certain limitations. to address this, we propose a comprehensive situational awareness benchmark, sa-bench. specifically, we category situational awareness into evaluation and forecast. evaluation corresponds to level 1 and level 2 of sa, encompassing two sub-tasks with four diverse scenario categories. we design multi-level knowledge bases and situational questions for various maritime vessels, using a question answering format to evaluate the model's capabilities in information extraction, environmental awareness, and information comprehension. forecast corresponds to the third level of sa, featuring five sub-tasks from different spectrum. we provide collections of real news texts from economic, technological, and other domains in a qa format, requiring models to predict various elements such as time, location, events, and actions based on historical background news. to balance task difficulty, we combine multiplechoice questions (mcq) and multiple-select questions (msq) in different proportions, simulating the decision-making process in real sa tasks as closely as possible. additionally, we conduct rigorous expert design, multiple independent annotations, and unified verification of the dataset. we validate the data annotations based on fleiss' kappa, which indicates that our annotation process is consistent with high annotation quality. we conduct extensive experiments with advanced llms to quanty their capabilities in situational awareness, including gpt3.5 @xcite , gpt-4/gpt-4o @xcite , gemini-1.5-pro @xcite , llama3 @xcite , claude-3 @xcite , deepseek-v2 @xcite , qwen1.5 @xcite , and yi @xcite . we incorporate chain-of-thought reasoning techniques and evaluate the models under zeroshot and few-shot settings. as shown in figure 1 , the experimental results indicate that gpt-4 and gpt-4o continue to lead, demonstrating excellent capabilities in information extraction, knowledge reasoning, and reasonable prediction. despite these advancements, a notable disparity persists when compared to human performance. other advanced models exhibit varying levels of performance across different tasks, but overall, the differences are minor, with all models slightly outperforming gpt-3.5. these findings suggest that while models have room for improvement in information extraction, reasoning, and prediction when handling complex, longer texts, significant progress is still needed. additionally, we observe that chain-of-thought prompts do not significantly enhance model performance and may even cause slight performance degradation. finally, we conduct a thorough analysis, providing a detailed examination of the challenges encountered by llms in situational awareness tasks. we summarize our contribution as follows: ## definition of situational awareness situational awareness is defined as the perception of elements within a specific time and space, the comprehension of their meaning, and the projection of their future status @xcite . based on previous research, situational awareness can be conceptually divided into three levels @xcite @xcite . environmental element perception (l1) involves identifying and perceiving relevant information and dynamics within the current environment, encompassing the detection of various elements such as targets, events, and surrounding entities. the system must accurately detect and record information about these elements. current situation comprehension (l2) requires the system to synthesize the perceived information from various elements, understand the relationships between these elements and their impact on the current situation. this procession necessitates data analysis and correlation. future status projection (l3) involves predicting future changes and impacts of elements based on the understanding of the current situation. the system must predict the future states and behaviors of environmental elements using current context and historical data, which is crucial for effective decision-making and planning. task sub-task sub-angle # format evaluation single proactive 270 mcq+msq protective 134 mcq+msq multi collaborative 404 mcq+msq comparative 202 mcq+msq forecast time -286 mcq cooperation -126 mcq location attack 100 mcq arrive 214 mcq route -117 mcq threat -113 msq in total 1966 ## conclusion situational awareness is fundamental for effective decision-making and response to dynamic situations, yet the capabilities of llms in this area remain largely unexplored. to fill this gap, we introduce sa-bench, a comprehensive benchmark de-signed to assess llms across three levels of situational awareness: environment perception, situation comprehension, and future projection. we evaluates various advanced llms, such as gpt-4, claude-3, llama3, and qwen1.5, revealing significant gaps in their abilities compared to humans. additionally, we provide a further analysis about obstacles and challenges in situational awareness, providing insights for future research.