Foveate, Attribute, and Rationalize: Towards Physically Safe and Trustworthy AI (2023)

Abstract: Users’ physical safety is an increasing concern as the market for intelligent systems continues to grow, where unconstrained systems may recommend users dangerous actions that can lead to serious injury. Covertly unsafe text is an area of particular interest, as such text may arise from everyday scenarios and are challenging to detect as harmful. We propose FARM, a novel framework leveraging external knowledge for trustworthy rationale generation in the context of safety. In particular, FARM foveates on missing knowledge to qualify the information required to reason in specific scenarios and retrieves this information with attribution to trustworthy sources. This knowledge is used to both classify the safety of the original text and generate human-interpretable rationales, shedding light on the risk of systems to specific user groups and helping both stakeholders manage the risks of their systems and policymakers to provide concrete safeguards for consumer safety. Our experiments show that FARM obtains state-of-the-art results on the SafeText dataset, showing absolute improvement in safety classification accuracy by 5.9%.

## introduction intelligent systems provide increased accessibility and convenience but come with potential new risks, particularly for susceptible groups such as children or marginalized communities. these risks have been exhibited by large language models, with issues relating to social biases, misinformation, and user safety @xcite @xcite . regarding user safety, situations may arise, such as a child asking a smart device for medical advice and receiving incorrect information that can lead to harm @xcite . as unsafe language becomes increasingly *equal contribution. 1 https://github.com/alexmeigz/farm more common @xcite , building systems that can identify, reason, and prevent such language is critical to reducing physical harm. previous work in natural language safety has primarily focused on explicitly violent text and typically expressed through violent keywords @xcite . recently, researchers have studied another form of unsafe text, which is instead implicitly unsafe. @xcite discusses how this covertly unsafe text, language that contains actionable physical harm, but requires further reasoning to identify such harm, remains an underexplored area and needs to be prioritized by researchers, stakeholders, and policymakers. @xcite presents safetext, a dataset comprised of this type of unsafe text, with different user situations and accompanying pieces of safe and unsafe actions. while previous research in covertly unsafe text introduces the specific area and related datasets, there is no work beyond general benchmarking of this text across various models and tasks. furthermore, these experiments only identify and measure the likelihood of generating unsafe text -it is also crucial to qualify the knowledge required to reason about the safety of such text to increase awareness and preventability regarding potentially unsafe situations and aid system operators in better understanding the risks of their systems concerning different user groups. our work aims to provide users with human-readable trustworthy rationales to explain why given text may be identified as safe or unsafe, which will benefit both the system users with new supplemental safety knowledge and model creators with more interpretable risk analyses regarding incorrect reasoning. to qualify and reason about knowledge regarding text safety, we explore the following research question in this paper: can language models correctly identify and justify whether various actions are safe or unsafe in different scenarios? to achieve such desiderata, we propose farm, the foveation attribution rationalization methodology (figure 1 ). by definition of covertly unsafe text, additional knowledge is required to reason about the safety of such scenarios. as a result, we first leverage few-shot prompting to fixate on foveations of the additional knowledge needed from external sources. then, we query these foveations and retrieve external knowledge with attributions to trustworthy sources to minimize the potential for misinformation in such sensitive domains. finally, we use this attributed knowledge to generate rationalizations for whether an action for a given scenario is safe or unsafe. our work proposes the following contributions: few-shot prompting. to improve natural language generation, researchers leverage few-shot prompting -providing examples as a prompt for a target task @xcite . while few-shot prompting tends to increase task-specific performance, explicitly prompting large language models to generate a chain-of-thought, a series of intermediate reasoning steps, during the inference process outperforms generic demonstrations on several tasks @xcite . introducing explanations after answers in these prompts can also effectively improve performance @xcite . sampling generated rationales from the output space in an ensemble method can help improve robustness @xcite . our paper builds upon these techniques by proposing the novel foveation task to help guide few-shot prompting for rationale generation. ## problem formulation we investigate whether large language models have safety reasoning capabilities and can correctly determine whether texts are safe or unsafe. as language models are not time-agnostic and do not have a complete overview of world knowledge, we investigate a model's safety reasoning skills when given access to external knowledge. specifically, given scenario s, the goal is to generate trustworthy rationale r to explain whether the advice given in s from text generation model m is safe or unsafe. by definition of covertly unsafe text, additional knowledge k is needed to generate r; however, since k is unknown, we must define an intermediate task to approximate the additional knowledge with k using an approximator a (equation 1). then, given k, the ultimate task is to generate r through some generator g (equation 2 ). the quality of a rationale r is evaluated using judgement function j, with the optimal rationale being the maximum judgement value (equation 3 ). we define the intermediate optimization problem to solve for the optimal estimator kopt , the knowledge added to maximize the quality of a rationale compared to when no external knowledge is added foot_0 (equation 4 ). in §4, we tie our foveation and attribution steps to the intermediate task to find an approximator a to estimate k and our rationalization step to generate a trustworthy rationale r. ## farm for covertly unsafe text to proceed with our problem formulation, we propose a time-agnostic methodology consisting of three steps in a pipeline (algorithm 1): ## future work while our research focuses on an engineering approach to mitigating physical harm, we call for an interdisciplinary solution to ai safety. specifically, a user-centered method focusing on informing communities regarding the risks of intelligent systems (e.g., hallucination) can be beneficial to ensure users will diligently verify attributed sources to prevent potential endangerment rather than naively trusting ai systems' outputs; all systems always have the malfunction potential regardless of guarantees, creating risk for physical harm. additionally, while we explore farm in the context of ai safety, a natural future research direction is to apply farm to other applications in intel- ## conclusion in this paper, we propose farm, a problem-solving paradigm that identifies missing information, retrieves and attributes it to trustworthy sources, and utilizes it for few-shot prompting for humaninterpretable rationale generation. farm is a time-agnostic solution that seeks to increase interpretability and confidence during text generation through foveation and attribution insights, empowering users to easily verify the factuality of these rationales, thereby improving the reliability of our system, increasing users' physical safety in the context of covertly unsafe language. our experiments show that farm improves upon the current safety benchmark for covertly unsafe text, safetext, by 5.9 points and generates rationales with improved entailment, factuality, faithfulness, and confidence. we release our generated foveations and rationales alongside the existing safetext dataset to promote future work in this area. by generating trustworthy, human-interpretable rationales, we hope to progress toward qualifying the knowledge required to reason through a safety scenario to inform stakeholders of systems' risks to different user groups. these rationales provide insight to help system designers and operators manage their system's safety risks, policymakers define concrete laws to reinforce consumer safety, and end-users with the knowledge to guard themselves and their community against the potential risks of ai. we encourage stakeholders, policymakers, and end-users to proactively prioritize user safety by leveraging these rationales to make informed decisions regarding ai physical safety.