Improving Industrial Safety by Auto-Generating Case-specific Preventive Recommendations (2024)

Abstract: In this paper, we propose a novel application to improve industrial safety by generating preventive recommendations using LLMs. Using a dataset of 275 incidents representing 11 different incident types sampled from real-life OSHA incidents, we compare three different LLMs to evaluate the quality of preventive recommendations generated by them. We also show that LLMs are not a panacea for the preventive recommendation generation task. They have limitations and can produce responses that are incorrect or irrelevant. We found that about 65% of the output from Vicuna model was not acceptable at all at the basic readability and other sanity checks level. Mistral and Phi_3 are better than Vicuna, but not all of their recommendations are of similar quality. We find that for a given safety incident case, the generated recommendations can be categorized as specific, generic, or irrelevant. This helps us to better quantify and compare the performance of the models. This paper is among the initial and novel work for the preventive recommendation generation problem. We believe it will pave way for use of NLP to positively impact the industrial safety.

## introduction in this paper, we propose a novel application of natural language processing (nlp) to improve industrial safety and thereby take a step towards creating positive impact on the society in general. industrial incidents refer to unplanned events or accidents that occur in industrial settings and frequently lead to injuries, property and material loss, and may also cause loss of life or environmental damage. industrial accidents continue to be a major global concern. according to the international labour organization (ilo), there are millions of work-related deaths and injuries annually @xcite . incidents can lead to direct costs such as medical expenses, equipment repairs, and legal fees. further, they also entail indirect costs like lost productivity and the hidden costs of long term damage to environment (jayapriyanka j, 2023) as well as reputation of an organization (e.g., the fallout of boeing 737 max @xcite ). the economic burden of occupational injuries and illnesses in the u.s. alone is estimated to be more than $100 billion annually foot_0 . these costs have remained high across different geographies @xcite , and also observed over long duration @xcite . preventing incidents not only saves invaluable lives and avoids injuries but also is more costeffective than dealing with their consequences. by systematically identifying and controlling risks, organizations can better protect their assets, personnel, and operations from unforeseen events. further, many industries are subject to strict regulatory requirements related to occupational health and safety foot_1 . implementing preventive recommendations ensures compliance with these regulations. it also helps to avoid penalties and legal liabilities. a critical component of an overall risk management strategy is to prevent recurrence of similar incidents @xcite . industrial environments often involve complex processes, machinery, and technologies. identifying potential hazards @xcite and developing effective preventive measures requires in-depth knowledge of these operations and specialized expertise. further, different industries and workplaces face diverse risks and hazards. developing comprehensive preventive recommendations requires a thorough understanding of industry-specific risks, regulatory requirements, and best practices tailored to each environment. acquiring and retaining such qualified personnel with the necessary skills can be costly for organizations. hence, obtaining good preven-tive recommendations for industrial safety can be challenging and costly. nlp in general and large language models (llms) in particular can be valuable tools for generating preventive recommendations for industrial safety. llms can analyze vast amounts of data quickly, including past incident reports, safety regulations, and industry best practices. this enables them to identify patterns and insights that might be missed even during manual reviews. llms can process and generate recommendations for large datasets of industrial incidents without the need for proportional increases in human resources. thus, nlp tools can have a positive impact on the society by improving industrial safety. contributions: in this paper, we propose a novel application to improve industrial safety by generating preventive recommendations using llms. using a dataset of 275 incidents representing 11 different incident types sampled from real-life osha incidents @xcite , we compare three different llms to evaluate the quality of preventive recommendations generated by them. we also demonstrate that while large language models (llms) hold significant promise, they are far from being a comprehensive solution for generating preventive recommendations. despite their capabilities, llms have inherent limitations and are prone to producing responses that may be inaccurate or off-topic. their performance can be inconsistent and they can generate recommendations that do not always align with the intended goals or context. we show that for a given incident case, the generated recommendations may be (i) specific and directly relevant to the case, or (ii) generic, i.e., are useful for as a broad preventive measure which need not be focused on the current incident for which recommendation is sought from llm, or (iii) the recommendations may be completely irrelevant for the current incident and it may even be hallucination by the llm. rest of the paper is organized as follows. in section 2, we first describe the details of the proposed study. section 3 covers the experimental setup, results and analysis. in section 4, we give a brief overview of the related work. limitations and ethical considerations for generating preventive recommendations using llms are discussed in section 5. finally, we conclude in the section 5. ## proposed approach in this work, we use generative power of large language model (llm) to generate preventive recommendations for industrial incidents. figure 1 shows the steps in the recommendation generation and their evaluation process. in this study, we examine three different llms: (a) vicuna-13b-v1.5-16k foot_2 , (b) mistral-7b-instruct-v0.2 foot_3 , (c) phi-3-mini-4k-instruct foot_4 . we chose the vicuna (chiang et al., 2023), mistral (jiang et al., 2023), and phi3 @xcite models for this task because they are representative of recently released open source models. they all give us the important benefit of easy customization and are freely available to the community. they also require less computing power compared to larger or more expensive models. further, their relatively smaller sizes allow us to experiment with moderate compute resources. the small sized models also are easier for integration in larger solution. all these factor make them an easy-to-use and cost-effective proposition. we prompt each llm with the incident report text in special delimited format (triple quotes used as the delimiter) and ask it to generate the preventive recommendations to avoid recurrence of similar incidents in future. we also tweaked the parameters to improve the quality of output from the llms as follows: (i) @xmath0: the maximum number of tokens to generate, ignoring the number of tokens in the prompt. (ii) @xmath1: this parameter helped to prevent the model from generating repetitive sequences by restricting the repetition of ngrams in the generated text. (iii) @xmath2.0, 0.3, 0.7}: the temperature parameter of an llm helped to regulate the amount of randomness or diversity in the output. lower temperature helped to reduce the hallucination. the generated recommendations were evaluated in two stages. first, we check and quantify the acceptability of the generated text. for this purpose, we checked the basic criteria such as: (i) the output is readable by a human, e.g., the words in a sentence should be properly separated by white-space and punctuation etc. the sentences are clearly separated using period or punctuation or bullet points so that further automated analysis at sentence level is possible. (ii) there is no verbatim copy of the input incident text, (iii) to check if the generated text contains hallucinations, i.e., text which is completely unrelated to the input incident report. for this purpose, we sampled output from each llm for about 20% of the incidents and checked it with respect to the above mentioned basic sanity check criteria. at the end of this stage, we found that about 65% of the output from vicuna model was not acceptable. hence, vicuna was eliminated from further evaluation. for the remaining models, viz. mistral and phi-3, we observed that not all recommendations of same quality. in particular, we noted that some of the recommendations are very specific and directly useful for the given incident. other recommendations were general suggestions and a few were irrelevant. hence, in the second stage of evaluation, we soliciated help from human experts to categorize the generated recommendations into one of the following three categories, viz. {specif ic, generic, irrelevant}. this categorization helps us to quantitatively benchmark the utility of the llms for the task of preventive recommendation generation. ## related work this section describes the related work to prevent incidents occurring in the construction industry. @xcite discuss about manual analysis of report to identify cause of the incident and suggests preventive actions based on the findings and on the literature. the work in @xcite surveys 395 construction workers (cws) and identifies different organizational stressors, personal and emotional stress, and safety behaviors using factor analysis to prevent injury incidents and enhance safety behaviors of cws. manual analysis of reports or surveys to identify cause of the incident is a cost intensive and time-consuming activity. to solve the issue, @xcite study the cause of incident using data mining techniques but provides suggestions for a few specific cases. @xcite examine whether the introduction of an incident reporting scheme with feedback in two industrial plants had an effect on the number of major incidents. @xcite talks about incidents prevention through design (ptd)/ design for safety (dfs) concept. such techniques require collaboration of all stakeholders, development of new design standards and regulations, and improved availability of ptd/dfs tools. more importantly, these technologies are country specific and may not be available in other countries. to the best of our knowledge there is no prior work on providing recommendations and categorizing them for a large-scale dataset such as occupational safety and health administration (osha) using large language models (llms). ## conclusion industrial incidents can disrupt operations and production schedules, causing downtime and delays. preventive recommendations help maintain continuity by minimizing disruptions and ensuring that work can proceed safely and efficiently. in this work, we proposed to use llms to improve industrial safety. based on the comparative analysis of different llms, we identified their strengths and weaknesses. we found that vicuna model is not suitable for this task. phi-3 and mistral models perform much better than vicuna. even with these two models, only about half of the recommendations generated are specific to the particular safety incident. other recommendations tend to be generic and a small fraction of the recommendations is irrelevant. we also highlight that the llm technology still needs significant improvement for this task and the preventive recommendations from llms need to be reviewed by safety professionals before actual implementation to ensure they are valid, practical, and aligned with industry standards. as part of future work, we plan to explore how investigation of causes and generating questions to probe relevant temporal aspects @xcite can be used to improve the recommendations. further, we plan to improve alignment of the models so that the hallucinations and irrelevant recommendations are reduced and the fraction of specific recommendations in the output generated is improved.