Enhancing Situation Awareness through Model-Based Explanation Generation (2024)

Abstract: Robots are often deployed in remote locations for tasks such as exploration, where users cannot directly perceive the agent and its environment. For Human-In-The-Loop applications, operators must have a comprehensive understanding of the robot’s current state and its environment to take necessary actions and effectively assist the agent. In this work, we compare different explanation styles to determine the most effective way to convey real-time updates to users. Additionally, we formulate these explanation styles as separate fine-tuning tasks and assess the effectiveness of large language models in delivering in-mission updates to maintain situation awareness. The code and dataset for this work are available at:———

## introduction automation offers significant advantages in our society, particularly in critical sectors like manufacturing and offshore applications, as recognized in prior studies @xcite . fostering transparency and accountability within robotics is imperative to bolster trust and wider adoption @xcite . one pivotal cognitive process influencing trust and adoption is situational awareness, characterized by three essential stages: perception (understanding a robot's decision-making), comprehension (discerning the rationale behind these decisions), and projection (anticipating future automated behaviours). recent research has shown that textual explanations presented visually within human-in-the-loop applications, such as autonomous driving, positively impact all facets of situational awareness @xcite . in this work, we include two user studies focusing on situation awareness and explanation generation. we share a dataset, licensed under creative commons attribution (cc-by), which contains categorical events related to maritime autonomous missions, user queries, and corresponding explanations. additionally, we demonstrate the performance of multiple large language models on three downstream tasks derived from our dataset. through the fine-tuning process and the user studies, we aim to answer the following research questions: the remainder of this paper is structured as follows: section 2 reviews prior research that has influenced our approach. section 3 describes the data collection and annotation processes. section 4 outlines the fine-tuning process for the large language models and details the experiments conducted to identify the best-performing model. in section 5, we describe the tasks included in our study to address research questions 2 and 3, as well as the participant groups that completed the questionnaire. section 6 presents the performance of the large language models and our findings from the user studies. finally, section 7 examines the implications of our findings, and section 8 discusses potential future experiments and concludes the paper. ## related work explainable agents and robots have become a crucial research area due to the increasing demand for transparency and interpretability in autonomous systems @xcite . these systems must effectively communicate their decision-making processes to users, preferably through user-friendly modalities such as natural language @xcite . typically, natural language explanations are presented as causal explanations, which are easy to understand and clearly justify automated behaviours @xcite . other types of explanations, such as counterfactual explanations (answering "what if" questions) and contrastive explanations (answering "why not" questions), also facilitate the interrogation of black-box systems @xcite . generating these explanations faithfully involves sophisticated methods for content selection, such as using bayesian networks or surrogate models @xcite . the selected content can then be communicated through controllable template-based approaches @xcite . additionally, end-to-end approaches using encoder-decoder architectures have shown promise in conveying agent rationale and improving failure and solution identification @xcite . the advent of causal language models with transformer-based encoder architectures @xcite has significantly advanced the field of text generation. these models excel at replicating domain-specific knowledge due to extensive training on vast amounts of humangenerated text @xcite . despite their substantial size and complexity, new techniques such as qlora @xcite have made fine-tuning more computationally efficient, facilitating the adaptation of pre-trained models to specific downstream tasks. to evaluate the semantic accuracy of models, researchers frequently compare the outputs with their corresponding inputs @xcite . this evaluation approach is particularly important in applications where the accuracy and reliability of natural language explanations are critical. commonly used metrics for this purpose include bleu @xcite , rouge @xcite , and me-teor @xcite , which measure quality based on n-gram overlap between reference labels and model-generated responses. however, these metrics have limitations, as they often fail to capture the true semantic similarity or desired verbosity of the outputs @xcite . to address these shortcomings, combining n-gram-based metrics with additional metrics that perform verbatim comparisons can provide a more comprehensive evaluation of model outputs, particularly in highstakes applications. given the robustness of large language models in data-to-text generation and their capability to perform multiple tasks, various domains have leveraged these models for diverse applications. for instance, they have been used for action selection in embodied tasks @xcite and for text summarization to infer sets of rules for object manipulation based on user preferences @xcite . in the realm of explainable robotics, language models are combined with retrieval augmented generation @xcite to transform robot logs and user queries into natural language explanations, thereby enhancing human-robot interaction. ## data generation to collect a dataset for autonomous maritime vehicles, we deployed an agent that follows a preexisting plan and attempts to complete its objectives by visiting a set of waypoints using different patterns (e.g., lawnmower, loiter). the agent prioritizes the integrity of the robotic platform and replans its behaviour in case of unexpected events. at each simulation timestep, we recorded the robot's behaviour and the states affecting that behaviour, such as objectives and sensor-derived events (e.g., obstacle or vessel detection). for this dataset, we utilised moos-ivp, an opensource behavioural agent designed for maritime robots @xcite . this simulator offers a variety of pre-built scenarios, from which we selected and refined three specific missions. we further modified the mission plans, creating four distinct configurations for each scenario. we limited the logged vehicle states to those impacting the agent's behaviour activations. finally, we performed post-mission parsing of the log files to extract the relevant vehicle states and the corresponding activated behaviours. in figure 2 , we illustrate three scenarios, each with four distinct task and environment configurations. scenario a involves an unmanned surface vehicle (usv) conducting a survey, avoiding obstacles, and returning to its starting point for retrieval. scenario b features an unmanned underwater vehicle (auv) loitering around predefined waypoints and transitioning to a designated survey area upon receiving instructions. scenario c has two vehicles loitering around a random polygon, occasionally switching sides and restarting their routine while avoiding collisions and obstacles. with each scenario, the task difficulty increases by adding more behaviours and introducing complex tasks such as collision avoidance. ## fine-tuning before attempting any fine-tuning, we evaluated the performance of existing instruction fine-tuned large language models to assess their capability in generating explanations from autonomous vehicle states. specifically, we employed three transformerbased decoder models, each with an identical number of parameters: llama2-7b-chat, mistral-7b-it, and falcon-7b-it, using 2-shot inference @xcite @xcite . this preliminary experiment revealed that all three models demonstrated strong results in terms of semantic accuracy and precision, with mistral and llama2 slightly outperforming falcon in these aspects. however, when evaluated using machine translation metrics, all three models exhibited significant shortcomings, with mistral performing slightly better. upon further inspection of the model outputs, we found that these models often compensated by increasing verbosity and adding supplementary tokens. these additions were unnecessary and may increase the cognitive load for users reading the explanations. the higher semantic accuracy and precision scores can be attributed to our metric's focus on the output mentioning spatial elements, behaviours, and entities present in the input representation. in contrast, the lower scores in rouge-l, bleu, and meteor metrics were due to the generated outputs not closely resembling the dataset labels. the results of this initial experiment are illustrated in figure 4 . our downstream task, we defined a fine-tuning setup where each explanation type is treated as a separate task. in figure 3 , we represent our finetuning tasks, where a task instruction, along with a representation and a user query, are provided as input. the model output is the corresponding explanation, including permutations for counterfactual and contrastive explanations. using our annotated dataset, we trained the three large language models on all explanation tasks utilizing the huggingface and peft @xcite libraries. ## user study to estimate the effect of explanations on users and determine user preference for model-based explanations, we designed two user studies. a total of 21 participants were recruited from the robotics industry and academia, including 9 individuals very familiar with autonomous vehicles, 9 who were familiar, and 3 who were not familiar. user study on situation awareness: this study builds upon prior work @xcite to investigate the effect of different explanation styles on users' situation awareness. we used recorded videos from the aforementioned maritime robot simulator, where an agent attempts to accomplish a set of objectives while considering its environment and inner state, particularly during unexpected events that require replanning. participants encountered three different conditions, each with a different explanation type (causal, counterfactual, and contrastive), along with a tutorial video describing the task beforehand. in the first condition, explanations were presented with captions. in the second and third conditions, participants selected user queries to generate corresponding explanations that clarified alternative outcomes. after the explanations were displayed, the interface asked users about events taking place in the video at predefined timesteps. their responses were used to estimate a performance metric representing their situation awareness per condition, thus assessing the effect of each explanation style on their mental models. ## results in this section, we present the results of our finetuned models and user study, addressing the research questions outlined in section 1. ## discussion our evaluation of inference performance using existing instruction fine-tuned large language models revealed that, despite their inherent capabilities and domain knowledge from pre-training, these models fall short in generating explanations with the verbosity and style expected by domain experts in autonomous vehicles. consequently, additional fine-tuning on specific downstream tasks is nec-essary. our fine-tuned models showed significant improvements in machine translation metrics, indicating a strong n-gram overlap between predictions and reference labels. notably, our best model performed exceptionally well on counterfactual and contrastive explanations, followed by causal explanations and mixed styles when using a balanced dataset. furthermore, the generated outputs exhibited high semantic accuracy and precision, underscoring the effectiveness of the fine-tuning process. the results from the first user study on the effect of different explanation styles on situation awareness demonstrated that users significantly benefited from our explanations compared to random chance, as there were three potential answers per question. specifically, users gave the most correct answers using causal explanations, followed by contrastive and counterfactual explanations. for causal explanations, users excelled in answering questions about decision-making, as the justification behind the exhibited behaviour was clear and did not require further queries. conversely, counterfactual and contrastive explanations allowed users to interrogate the system and learn more about the spatial elements of the mission, resulting in an almost equal percentage of correct answers. while causal explanations helped users answer spatial questions with the third-best success rate, they did not provide enough time to digest the information, potentially increasing cognitive load. for intrinsic questions concerning the robot's inner states, such as sensor readings, causal explanations demonstrated the best performance, indicating that a straightforward approach to explaining a robot's inner states is the most effective strategy. considering these findings, future work could tailor the explanation styles presented to users based on the type of content needing explanation. the results from the second user study indicated a clear preference among participants for domain expert template-based explanations, though some participants preferred model-based explanations, and others expressed no strong preference, showing equal satisfaction with both types. this preference may have been influenced by the presentation format: each scenario featured two template-based explanations characterised by high completeness (the breadth of justifications behind an outcome) and low to medium soundness (the level of detail for each justification), which directly reflected the vessel's current state @xcite . in contrast, only one model-based explanation was provided per scenario. the use of model-based data annotation for labelling the dataset may have also impacted the study's outcomes. future work should focus on aligning language model outputs more closely with the response styles of domain experts and further refining modelbased data annotation techniques, particularly for critical applications. template-based explanations, while effective, are not scalable, require significant time to develop, and lack robustness, especially when dealing with complex or evolving scenarios. these limitations highlight the need for a data-driven approach using large language models, which offer greater adaptability, efficiency, and the potential to generate contextually relevant explanations at scale. ## conclusion and future work this work has successfully demonstrated the impact of different explanation styles on situational awareness across various aspects of a mission, such as decision-making, spatial elements, and inner vehicle states, within the context of human-in-theloop applications for autonomous vehicles. additionally, we assessed user preferences between template-based and model-based explanations. we also showcased the capabilities of our large language model in performing data-to-text tasks, transforming the states of autonomous vehicles into natural language explanations across three different styles. the fine-tuned models have shown satisfactory performance in generating coherent and contextually appropriate explanations. for future work, several avenues for enhancement and exploration remain open. experimenting with a broader range of explanation types could provide deeper insights into user preferences and effectiveness. additionally, integrating additional modalities, such as map or chart-based user interfaces, would be a valuable extension. these interfaces are commonly used in conjunction with autonomous agents and could offer a more comprehensive and interactive explanatory experience for users.