Personalized open world plan generation for safety-critical human centered autonomous systems: A case study on Artificial Pancreas (2025)

Abstract: Design-time safety guarantees for human-centered autonomous systems (HCAS) often break down in open-world deployment due to uncertain human interaction. In practice, HCAS must follow a user-personalized safety plan, with the human providing external inputs to handle out-of-distribution events. Open-world safety planning for HCAS demands modeling dynamical systems, exploring novel actions, and rapid replanning when plans are invalidated or dynamics shift. No single state-of-the-art planner meets all these needs. We introduce an LLM-based architecture that automatically generates personalized safety plans. By itself, the LLM fares poorly at producing safe usage plans, but coupling it with a safety verifier—which evaluates plan safety over the planning horizon and feeds back quality scores—enables the discovery of safe plans. Moreover, fine-tuning the LLM on personalized models inferred from open-world data further enhances plan quality. We validate our approach by generating safe usage plans for artificial pancreas systems in automated insulin delivery for Type 1 Diabetes patients. Code: https://github.com/ImpactLabASU/LLMOpen

## introduction human centered autonomous systems (hcas) are often safety critical @xcite , where actions taken by a reactive module (rm) in response to percepts from the environment can cause harm to the human. the human in a hcas can assume two roles (fig. 1 ): a) human in the loop (hil), where the human is observing the environment and can provide additional inputs to the environment or rm to change their states, and b) human in the plant (hip), where the human acts as a passive physical entity with time variant dynamics that gets directly affected by the actions of the rm with the intent to reach a goal state. as such assurance of safety under human inputs is an essential property to establish trust and widespread acceptance of hcas such as artificial pancreas (ap) systems for glucose management in type 1 diabetes (t1d) or autonomous cars. given that the human is inherently coupled with the operation of hcas, any safety assurance process will need to accurately model rm-human interaction @xcite . the complexity of human interactions with the hcas makes it impossible to accurately enlist all possible scenarios @xcite . as such, in the design phase, safety critical systems utilize certain limiting assumptions on the distribution of hil actions and hip properties; a human who satisfies the assumed distribution is referred to as the "average human". then a set of safety plans, hil action sequences of the "average user", is developed under which the hcas is tested to be safe. some of the major problems in assuring that a tested safe hcas with a set of safety plans still operates safely in deployment are: p1: out of distribution hil -the set of safety plans may not optimize performance for a human user who may not fit the assumptions of the average human user. as a result, the user in its hil capacity may operate the hcas under out of distribution (ood) usage plans that are not tested to be safe and hence can potentially cause safety violations in deployment. p2: long term time varying hil-hip contexts -the hcas is often intended for long term usage and in the course, user contexts such as hip dynamics or hil action space can fundamentally change. such changes may lead to human interactions with open world planning problem for insulin management text prompt general llm insulin management plan approach 1: llm as autonomous planner approach 2: llm + safety verifier open world insulin glucose data type 1 diabetes simulator to generate training data fine-tuned llm open world planning problem for insulin management text prompt model recovery (sindy-mpc: bergan minimal model, bmm) model recovery (sindy-mpc) embodied prompt rlhf chat interface fine-tuned llm 2 2 1 type 1 diabetes simulator as verifier that computes time in hypoglycemia insulin management plan 1 back prompt offline part online part hcas that may not adhere to the set of safety plans or the safety plans themselves may be invalidated due to the changed hip dynamics. p3: short term unexpected plan invalidation -human contexts are characterize by unexpected events that may affect hip or may require novel hil actions for safe and effective performance, resulting in deviation from the safety plan. in state-of-art commercialized hcas, the problems p 1 through p 3 are tackled by exploring novel usage plans in the open world through consultation with experts, detecting safety violations through safety monitors, and preventing hazards through fail safe modes that may entail shutting off hcas autonomy and requiring emergency hil action. in this paper, an open world @xcite is characterized by occurrences of one or more instances of the problems p 1 through p 3. in the standard modus operandi of hcas, the exploration and execution of novel plans in the open world is a burden on the user. as such increased plan management burden: a) is often a deterrent to usage or adherence to safety plans, or b) may restrict the hil actions which may in turn affect hip dynamics through the psycho-physiological feedback pathways. in this paper, we develop a framework to automatically generate usage plans in the open world for a hcas that are: a) personalized for a given human user, in that it optimizes certain efficacy criteria for the human user, b) safe for human use even if they are originally not included in the set of safe usage plans, and c) reduces plan management burden by decreasing the number of inputs required from the human user. ## formal problem definition let the state of the hcas be expressed by the state variable x which follows the dynamics in eqn 1. for convenience in expressing the planning problem and with restricted focus on the case study of ap, we will assume that the environment state is accurately expressed by the hcas state and hence x @xmath0. in a hil-hip architecture, the input to the environment is given by: @xmath1, s p ) + u ex , where u @xmath2, and s p can be manually changed by the hil. a usage plan, pl, consists of a temporal sequence of b external inputs (u ex (q i )) at times 0 ≤ q @xmath3, t h is the planning horizon. we denote the set of all possible plans as p∞. safety is defined as a logical predicate saf e(x, f ω , pl) on the state of the hip x, for a given dynamics f ω , and a given usage plan pl. a plan pl is a safety plan if the predicate saf e(x, f ω , pl) is satisfied at all times @xmath4, t h ]. the user burden for executing a plan @xmath5. ## usage plan in open world case study ap is an exemplary safety critical hil-hip hcas with the open world problem. aps automatically infuse insulin, known as micro bolus, to control blood glucose levels around a set point s p , while preventing hypoglycemia when blood glucose level falls below 70 mg/dl. all ap systems that are approved for human use by food and drug administration (fda) require the human user to provide external insulin in addition to the ap controller input to manage glucose variability due to meal intake, also called meal bolus (u meal ). this meal bolus is proportional to the carbohydrate content of the meal c, with carb insulin ratio (cir) as the proportionality constant. while administering meal bolus any residual insulin in the body due to past insulin infusion, characterized by insulin on board (iob) is subtracted (eqn. 2). in addition, external insulin, correction bolus (u corr ), unrelated to meal can also be administered if the cgm reading is greater than the set point. u corr is proportional to the difference between the current glucose value g(t) and the set point, with insulin sensitivity factor (isf) as the proportionality constant shown in eqn. 3. the residual insulin or iob depends on the insulin pharmacokinetics, (eqn. 4), obtained from bergman minimal model (bmm) (bergman, 2021), and is difficult for a human to guess. here, we assume that y and z are internal state variables of the bmm and are not measurable. a safe usage plan for the ap hcas is a sequence of set point changes, carbohydrate, meal bolus, and correction bolus intake actions such that the safety criteria of percentage time below 70 mg/dl in 24 hrs is less than 4% is satisfied. in stl the safety condition can be written as g @xmath6, where g is the globally true stl operator. an example safe usage plan is as follows: "set point is 110 mg/dl at 6 am; set isf to 50 at 6 am; set cir to 15 at 6 am; breakfast with 20 g of carbohydrate at 8:30 am with 1 u of meal bolus; if @xmath7, take correction bolus from eqn. 3; lunch with 40 g of carbohydrate at 1 pm with meal bolus in eqn. 2; if @xmath8, take correction bolus from eqn. 3; dinner with 30 g of carbohydrate at 6 pm with meal bolus in eqn. 2, set set point at 90 mg/dl at 10 pm" this is deemed safe by a sample patient from the virtual patient registry available in the fda approved type 1 diabetes simulator developed at uva padova @xcite . this is an example of the average user that follows the constrained distribution manifested by the virtual patient registry. the outcome is measured using four metrics: a) percentage time in range (tir), 70mg/@xmath9, and b) time below range (tbr), when @xmath10. ## detailed solution we provide details of the two approaches. ## evaluation the open world scenario is introduced to each llm using the following prompt sequence. ## conclusions this paper demonstrates using llms to plan the personalized operation of an hcas. it highlights a key property of llm planners: they can explore novel actions and reason about dynamical systems with rapid replanning. our main observations are that llms can plan control tasks when two steps are carefully designed: (a) contextualization of the chat rl, and (b) fine-tuning the llm's internal weights via embodied training, where textual instructions and interpretations intertwine with traces from the real-world system. our method applies to any open-world planning over a dynamical system with lipschitz-continuous dynamics and temporally concatenated inputs-e.g., artificial pancreas with human in the loop operation, or semiautonomous cars that drive under autonomy and hand off control to the driver in critical scenarios or perception failures. future work: safety assured llm based open world planners can be integrated into hil control design @xcite and assistive technologies such as aiim @xcite to seamlessly integrate open world planning capabilities in safety critical systems.