FRIDA to the Rescue! Analyzing Synthetic Data Effectiveness in Object-Based Common Sense Reasoning for Disaster Response (2025)

Abstract: During Human Robot Interactions in disaster relief scenarios, Large Language Models (LLMs) have the potential for substantial physical reasoning to assist in mission objectives. However, these reasoning capabilities are often found only in larger models, which are not currently reasonable to deploy on robotic systems due to size constraints. To meet our problem space requirements, we introduce a dataset and pipeline to create Field Reasoning and Instruction Decoding Agent (FRIDA) models. In our pipeline, domain experts and linguists combine their knowledge to make high-quality, few-shot prompts used to generate synthetic data for fine-tuning. We hand-curate datasets for this few-shot prompting and for evaluation to improve LLM reasoning on both general and disaster-specific objects. We concurrently run an ablation study to understand which kinds of synthetic data most affect performance. We fine-tune several small instruction-tuned models and find that ablated FRIDA models only trained on objects’ physical state and function data outperformed both the FRIDA models trained on all synthetic data and the base models in our evaluation. We demonstrate that the FRIDA pipeline is capable of instilling physical common sense with minimal data.

## introduction which of the following would be most dangerous if it collapsed? this question, as seen in figure 1 , is fairly trivial for humans to answer, but requires several types of semantic knowledge. one must know the general size of these items and their other functions to fully assess the danger the item poses. a collapse is also a change of state that fundamentally shifts the use of these objects; a collapsed chair could be more likely to cut or scrape someone, but it could also mean the chair can now be carried if the chair folds. all of this knowledge is needed to answer this question, and all of it is embedded in our semantic understanding of objects that can cause danger and objects that can collapse, both intentionally and unintentionally. the ability to reason about objects is especially important in the context of human-robot interaction in disaster relief scenarios @xcite . for example, during search and rescue after an earthquake, a robot needs to know how to navigate partially collapsed buildings and how to use the many tools required to free people from the rubble. however, using robots to aid in disaster relief introduces many constraints. because of the destruction a disaster can wreak, consistent internet connectivity cannot be assumed. for human safety, robots must be handled via radio in a secure location. this low-bandwidth communication means limited image data can be transmitted to the handlers, which rules out remote piloting @xcite . we therefore need an autonomous system that can reason about its environment and the relief tasks required. as llms have improved dramatically, their abilities at semantic reasoning about objects have improved as well. llms have long been proven able to encode physical world knowledge @xcite , and their embeddings can improve physical understanding of an environment and its objects both within and beyond a fine-tuned domain @xcite . however, much of this improvement is found only in larger models trained on more data @xcite . this makes these essential semantic capabilities less accessible to our use case. our robot cannot rely on an internet connection to make api calls. we instead must utilize the robot's limited on-board computing power, which can be as little as 16 gb of virtual ram on an array of gpus @xcite . that amount of gpu ram can only reasonably run inference on a 13 billion parameter model given the heuristics described in @xcite . furthermore, this heuristic assumes that our robot is not running other processes in parallel, which is fairly unreasonable. we thus wanted to answer: given our constraints, how can we imbue all the physical common sense and semantics needed for smaller llms to be more capable at understanding a disaster environment? to answer our research question, we first tested the effectiveness of fine-tuning smaller models on disaster relief data. however, available data proved to be an additional constraint. most publicly available data on disasters is social media-based reactions @xcite , which do not pertain much to our subdomain of disaster relief efforts. furthermore, the specific knowledge (and to a lesser extent, the general knowledge) required for each mission varies by disaster. for example, after an earthquake, a robot needs to find survivors, while after a chemical spill, a robot needs to sample the environment for hazardous materials. therefore, we need a method for generating training data for specific disasters, and we need to evaluate which data are most effective at improving robot performance. we present a pipeline to create field reasoning and instruction decoding agent (frida) models as a proof of concept for llm viability in the disaster relief domain. for frida, we leveraged both disaster and linguistic expertise to create goldstandard instructions that, in turn, are used as a basis for synthetic data generation, as seen in fig -ure 2. these synthetic data are then used to finetune smaller models that fit our memory constraints. like its rescue dog eponym, foot_0 our frida models were initially developed and tested for earthquake disaster relief, based on expert knowledge pertaining to the february 6th, 2023 earthquakes in turkey and syria @xcite . thus, the resulting models are small enough to effectively operate onboard a robot and are fine-tuned on specialized and inexpensive data, satisfying all of our use case constraints. to investigate which synthetic data most influenced model performance, we ran an ablation study where we fine-tuned the same small llms on subsets of our synthetic data corresponding to specific types of object-based reasoning. we call these resulting models the ablated frida (afrida) models. we found that afrida models trained on general semantics and physical common sense had stronger overall performances than models trained on only domain-specific knowledge. additionally, the best performing afrida models scored better than their corresponding base models and frida models trained on the entire synthetic dataset. we posit that frida succeeds in improving objectrelated general common sense, but that small llms struggle with disaster-specific equipment usage. our contributions are as follows: our datasets, code, and a complete walkthrough of the frida pipeline are currently available. 2 2 related work ## results as seen in we assessed each model's capability on each type of reasoning tested in the evaluation dataset. to show the overall trend across models, we present the semscore results for the frida and afrida models in figure 4 . overall, when observing model performance in the figure 4' s columns, models model semscore accuracy (%) ministral 8b instruct 93.5 frida 94.6 ablated model semscore fine-tuning data subset accuracy (%) relative sizes and shapes 95.0 object functions 94.7 object differences 93.4 objects causing harm 93.3 specialized equipment 93.8 non-functional obj facts 93.2 earthquake knowledge 91.7 instruction understanding 85.0 looking at evaluation data types represented in the rows, it is clear that the more difficult evaluations are "specialized equipment", the category querying about the specialized objects used in earthquake search and rescue, and "earthquake", the category evaluating scientific knowledge about earthquakes. both of these evaluations are highly specific and technical. the easier evaluation categories are "object functions" and "differences", which pertain to understanding the basic semantics of objects' abilities and the differences between objects, respectively. another key observation from figure 4 can be found by comparing evaluation performance between frida and ministral 8b. frida has figure 4 : semscores (embedding-vector cosine similarity scores) for the frida suite for each type of evaluation. across all models, performance is better in evaluation data corresponding to physical common sense (object functions, differences) and worse in evaluation data corresponding to specialized object knowledge (earthquake, specialized equipment). ## discussion & error analysis it is particularly surprising that the "afrida relative size and shape" and the "afrida object function" models outperformed all other models across the board, even though the physical semantics expressed in those fine-tuning data are not complex. we hypothesize that clarifying the basic properties and affordances of objects provided a better basis for the model to have stronger physical reasoning across all categories. another surprise was that the "relative sizes and shapes" evaluation subset was a challenge for the frida suite. although one may think that simpler object properties like its "relative sizes and shapes" might be relatively prevalent in the base models' pre-training data, it is also plausible that reporting bias in web text leads to under-representation of highly commonplace facts @xcite . we hypothesize that this lack of pretraining data is partially why the ablation model trained on "relative shapes and sizes" synthetic data performs so strongly. however, this does not answer why the ablated models trained on data pertaining to other challenging categories in our evaluation, namely "afrida: earthquake" and "afrida: specialized equipment", did not receive the same overall performance bump. we suspect that the reason "afrida: earthquake" and "afrida: specialized equipment" did not similarly improve performance is that our synthetic data for the more specific objects and tasks tended to be longer and have lower rouge scores. these data therefore had more diversity. the sample size of the earthquakes and specialized equipment synthetic data subsets may have been too small for the model to be correctly biased by fine tuning. conversely, larger models may have ingested operators' manuals for specialized equipment, facilitating parroting answers for questions on this topic. we note that our research highlights the general difficulty of analyzing the precise effects of fine-tuning given opaque pre-training data. error analysis of both the frida model (finetuned using all synthetic data) and the "afrida relative size and shape" model revealed that both models got the same instances and number of the "relative size and shape" evaluation data incorrect. for example: a) with the camera plugged in b) with the camera unplugged gold: b) with the camera unplugged frida: a) with the camera plugged in the base ministral model generally gets the same "relative size and shape" evaluation instances incorrect as the frida models. however, it also answers incorrectly for over half of the instances of test items that relate to answering which item is bigger and which item will fit into another item. for example: thus, we conclude that the fine-tuning contributed to improvement in understanding which items are bigger and which items fit into others in particular. this improvement may translate to improvement in other related categories. specifically, we also see dramatic improvement over the base model for the "objects causing harm" evaluation data. this could be further boosted by a general understanding of which objects are larger. when it came to reasoning about the complex equipment used, error analysis revealed that both vanilla and fine-tuned models scored perfectly when asked to choose the correct role for an object in an event. for example: the task of choosing the correct object to use for a task proved more challenging. fine-tuning on related data seemed to unnecessarily bias the model toward choosing the most complicated object, while fine-tuning on unrelated data maintained results. for example: 5. "select the equipment needed for breaking rubble into smaller pieces after an earthquake." a) axe b) pickaxe c) hydraulic lift d) hard hat e) hammer gold: b) pickaxe ministral: b) pickaxe frida: c) hydraulic lift afrida relative sizes: b) pickaxe in the most complex reasoning task of ordering steps to complete to use an object, fine-tuning had no clear effect, with all models providing random answers. we thus conclude that fine-tuning for required equipment did not effectively bias the models to understand the use cases of these complex objects. at its worst, it incorrectly biases the model to choose complex objects when simpler ones would be more effective. overall, the frida pipeline improves small llm object reasoning when said models are finetuned on more general physical common sense and object reasoning data. the frida suite models are lightweight enough to fit within our constraints, and can even achieve comparable performance to a much larger gemini model. in comparison to the ablated models, the performance of the full frida model trained on all synthetic data demonstrates that more work needs to be done to improve the synthetic dataset distribution to be ideal for improving frida model performance on reasoning for earthquake search and rescue. ## conclusion we introduce a pipeline to create expert-in-theloop-based synthetic data that is then used for fine-tuning to create frida models. we found our pipeline improved performance over our base model. we performed an ablation study and found that data generated from templates based in basic physical common sense reasoning about objects improved performance most; ablated models trained on those data scored higher than frida models trained on all synthetically generated data and higher than gemini-1.5-flash, the llm that generated the synthetic data. this pipeline is an important step in understanding and improving llm object reasoning for practical use. even if some of our problem constraints are eventually alleviated by technology that facilitates very large models with smaller compute requirements, there will remain problem spaces for which web-based pre-training data simply does not exist. our research demonstrates an effective pipeline to specialize models fine-tuned on data that is not well-represented in typical web text pre-training data.