Walking a Tightrope – Evaluating Large Language Models in High-Risk Domains (2023)

Abstract: High-risk domains pose unique challenges that require language models to provide accurate and safe responses. Despite the great success of large language models (LLMs), such as ChatGPT and its variants, their performance in high-risk domains remains unclear. Our study delves into an in-depth analysis of the performance of instruction-tuned LLMs, focusing on factual accuracy and safety adherence. To comprehensively assess the capabilities of LLMs, we conduct experiments on six NLP datasets including question answering and summarization tasks within two high-risk domains: legal and medical. Further qualitative analysis highlights the existing limitations inherent in current LLMs when evaluating in high-risk domains. This underscores the essential nature of not only improving LLM capabilities but also prioritizing the refinement of domain-specific metrics, and embracing a more human-centric approach to enhance safety and factual reliability. Our findings advance the field toward the concerns of properly evaluating LLMs in high-risk domains, aiming to steer the adaptability of LLMs in fulfilling societal obligations and aligning with forthcoming regulations, such as the EU AI Act.

## introduction large language models (llms) have revolutionized how the world views nlp @xcite . their astonishing performance on many tasks has led to an exponential increase in real-world applications of llm-based technology. however, llms have a tendency to generate plausible but erroneous information, commonly referred to as hallucinations @xcite . this phenomenon proves to be particularly detrimental within high-risk domains, underscoring the importance of accurate and safe model outputs @xcite . in addition, with upcoming regulations, such as the eu ai act @xcite , is not yet finalized, it is expected that llms will fall into the high-risk category in specific domains, such as medical and legal. 1 the necessity of properly analyzing and evaluating llms is further addressed. eu ai act is expected to become the first law worldwide that regulates the deployment of ai in the european union, therefore, set a precedent for the rest of the world. according to the current draft, ai systems in high-risk domains, e.g. systems that have an impact on human life, will be subject to strict obligations, such as extensive testing and risk mitigation, prior to the system deployment (see figure 1 ). in the era of llms, instruction-tuning @xcite has been proposed to efficiently solve various tasks like question answering (qa), summarization, and code generation @xcite . however, these models, trained on heterogeneous internet data, lack domain-specific knowledge crucial for accurate and reliable responses in high-risk domains, including up-to-date regulations, industry practices, and domain nuances @xcite . furthermore, the quality of the training data is seldom quantified @xcite . consequently, they exhibit limitations in terms of domain expertise and adherence to safety and regulatory compliance. in the study conducted by @xcite , a comprehensive perspective was introduced, advocating for the consideration of multiple facets in assessing generalization across diverse data distributions and scenarios. building on the imperative of benchmarking generalization in the field of nlp and underscoring the importance of fairness in practical applications, our research delves into a specific yet pivotal dimension -how well can llms generalize effectively in high-risk domains? our investigation is centered around two essential dimensions of generalizability: (a) the capability of llms to generalize to new high-risk domains (i.e., general vs. high-risk domains) and new tasks (i.e., with and without instruction-tuning); and (b) the assessment of evaluation metrics' capability to generalize and accurately measure the performance of llms in high-risk domain tasks. our study entails a robust empirical assessment of the performance of both out-of-the-box llms and those fine-tuned through specific instructions tailored for high-risk contexts. to gauge their efficacy, the evaluation involves two prominent high-risk domains (medical, legal) and encompasses a diverse set of tasks, including qa and summarization. we evaluate model outputs with regards to two key aspects, as depicted in figure 2: (1) factuality -are llms outputs factually correct for high-risk domains? (2) safety -do llms successfully avoid producing harmful outputs? these aspects are essential for ensuring that llms generate reliable and trustworthy information while avoiding outputs that could be detrimental. to evaluate this, we employ existing metrics for factuality @xcite and safety (hanu and unitary team, 2020; dinan et al., 2022) concerns. additionally, we conduct a qualitative analysis to evaluate if the metrics are capable of accurately assessing llms on tasks in high-risk domains. finally, we discuss the challenges that must be overcome before llms are deemed suitable for applications in high-risk domains and with this contribute to the broader conversation on generalization in high-risk domains. contributions. our contributions are summarized as follows: (i) we robustly evaluate the outputs of out-of-the-box and instruction-tuned llms in two high-risk domains on 6 datasets across qa llm output factuality safety quality benchmark legal medical figure 2: overview of the evaluation framework of evaluating llms in high-risk domains. we evaluate how well llms with and without instruction-tuning perform in high-risk domains: legal and medical. the quality of the outputs is assessed using existing metrics to measure factuality and safety. and summarization tasks in terms of safety and factuality concerns; (ii) we demonstrate a qualitative investigation to identify shortcomings of existing metrics; (iii) we discuss open challenges that need to be solved in order to solidify trust to the generalization capability of llms in high-risk domains; (iv) we advocate for the need of human-centric nlp systems that are capable of giving the final control to human users in order to build trustworthy applications in high-risk domains. ## domain-adaptive instruction-tuning the emergence of gpt @xcite has led to a multitude of generative llms. one line of improving llm performance has been proposed to increase the number of model parameters @xcite . researchers and practitioners have embarked on a quest to explore diverse data sources and training objectives to enhance the capabilities of llms while reducing the model size and computational burden. another focus is leaning toward training smaller foundation models (e.g., gpt-j @xcite , llama @xcite , mpt (mosaicml nlp, 2023)). the adoption of smaller foundation models enables researchers and practitioners to conduct more efficient investigations into novel methods, explore new domain-specific applications, and establish streamlined deployment efficiency. crucially, the emphasis on smaller models is in accordance with the utilization of the instruction-tuning @xcite ) method, enabling efficient customization and adjustment of llms for particular domains or tasks @xcite . in our experiments, we rely on a series of smaller size llms for efficiency and cost concerns, and effectively incorporate domain knowledge for high-risk domains via instruction-tuning. by leveraging explicit instructions during the training process, instruction-tuning has proved to enhance the model's ability for generalization @xcite and domain adaptability @xcite . the domain-adaptive instruction-tuning approach explores the capability of how smaller models can effectively adapt to high-risk domains @xcite . to efficiently incorporate domain knowledge, we employ qlora @xcite , a method based on lora @xcite , which compresses models using 4-bit quantization while maintaining performance parity. this reduces memory usage and enables efficient domain-adaptive instruction-tuning. ## experimental setup instruction-tuning data. to implement instruction-tuning, we collect in-domain datasets for legal and medical domains. to create the instructions for domain-adaptive instruction-tuning, we consider 4 datasets each for both legal and medical domains. an overview of the collected datasets is shown in ## evaluation results factuality. results for the factuality metrics can be found in comparing the models, results of the instructiontuned model are better than those of the baselines, indicating that domain-adaptive instruction-tuning can lead to improvements in results generated for high-risk domains. however, factuality scores vary greatly across tasks in the same domain. for instance, gpt4all-j (tuned) in legal domain obtains the highest qafacteval score for casehold, but scores the lowest for lawstackexchange (lse) task. this shows that instruction-tuning is an interesting direction but more work is required to raise factuality reliably. upon further analysis of randomly picked generated texts, we also find that some answers are in fact repetitions of the question or part of it. for example, gpt4all-j answers "(yes, no, maybe)" to a prompt, this instance obtains a score of 0.5 from qafacteval and 0.946 from unieval. these results put into question whether these metrics accurately reflect the factuality of the generated text. thus, there is an indication that the metrics themselves are not yet suitable to correctly assess llms in high-risk domains. ## implications the need for factual and secure outputs of llms is crucial for their deployment in high-risk domains. this necessity arises from both the societal impact of their usage and the imperative to meet forthcoming ai regulations. based on the outcomes of our empirical investigation, it is evident that llms are not yet ready for deployment in high-risk domains @xcite . in light of this, we address three key implications that can guide us towards a more suitable course of action: (1) models enhancement: a pressing need to improve the llms themselves is crucial to ensure they generate accurate and reliable responses; ## related work llms in high-risk domains. recent work has demonstrated the efficacy of leveraging llms in high-risk domains, and has been achieved either by training the model using a substantial volume of domain-specific data @xcite , or by employing instruction-tuning techniques to harness the benefits of fine-tuning llms with relatively smaller sets of in-domain instruc-tions from diverse tasks @xcite . domain-adaptive instruction-tuning approach has proven effective in high-risk domains, such as finance @xcite , medicine @xcite , and legal @xcite . @xcite proposed med-palm2 model and evaluated on several medical domain benchmarks, but it has been demonstrated that even with extreme llms, the model remains inferior to the expertise of clinicians. similar findings are also suggested in legal domain @xcite , where llms have yet to attain the proficiency levels of experienced tax lawyers. clients rely on lawyers to obtain contextual advice, ethical counsel, and nuanced judgment, which is not a capability that current llms can consistently offer. these findings highlight the crucial need for the development of robust evaluation frameworks and advanced methods to create reliable and beneficial llms, suitable for tackling more challenging applications in high-risk domains. assessing llms. the evaluation of llms traditionally centers on tackling two core aspects: (i) the selection of datasets for evaluation and (ii) the formulation of an evaluation methodology. the former focuses on identifying appropriate benchmarks for assessment, while the latter involves establishing evaluation metrics for both automated and human-centered evaluations @xcite . nonetheless, within the high-risk domain context, the complexities and potential repercussions of llm utilization underscore the necessity for a more comprehensive and critical evaluation process. specific challenges arise when assessing llms within particular domains @xcite . for instance, domains like law demand continuous updates in information to remain relevant @xcite . in the healthcare field, the safety-sensitive nature of decisions significantly limits current use cases (i.e., the possibility of hallucinations could be detrimental to human health) @xcite . to mitigate risks in high-risk domains, enhancing the model's factual grounding and level of certainty is essential @xcite . recent research has emphasized a shift toward humancentered evaluation @xcite . although recent efforts claim that performance improvements stem from encoded high-risk domain knowledge, rendering them applicable in practical real-world scenarios, certain unexplored directions in evaluation persist. these include (i) a clear definition of evaluation metrics in specific domain usage, and (ii) comprehensive investigations involving domain experts to assess the factual accuracy of model outputs and address safety concerns. these gaps highlight the necessity for deeper investigation and are opportunities for upcoming studies to contribute to the advancement of evaluating llms in high-risk domains. ## conclusion as llms have taken the world by storm, the benchmarking generalization concern in nlp gains significance. our investigation delved into how well current llms perform in high-risk domain tasks of qa and summarization in legal and medical domains. the results exposed a significant gap of the suitability of llms for high-risk domains tasks, indicating that employing llms in their present state is not yet practical. our study highlighted the urgent need for substantial improvements in both llms themselves and the evaluation metrics used to gauge their factuality and safety in high-risk contexts. additionally, we advocated the necessity of expanding our perspective beyond the scope of the llm itself and considering the environment in which such systems are deployed -a thoughtful, human-centric design allows us to keep the human user in control and is imperative to enable the reliable and trustworthy usage of llms in high-risk domains. overall, our findings and discussions accentuate the importance of a close collaboration with stakeholders and therefore collaboratively address open critical concerns. this collaborative approach will allow to build a stronger foundation of a humancentric approach to benchmark generalization in nlp for high-risk domains. measure factuality and safety. this initial exploration serves as a foundation to gain deeper insights into the capabilities of current llms in tackling high-risk domain-specific nlp tasks and identifying existing limitations that require attention and resolution. the current setup has a series of shortcomings that should be reduced in future work, namely: (1) the collected datasets currently only focus on english; (2) the instruction templates are designed manually and might lead to variable outcomes; (3) other instruction-tuned models trained on generalpurpose instructions might offer different capabilities, depending on the specific context of domains and tasks; (4) other metrics should be explored and considered, such as robustness @xcite and explainability @xcite ; and (5) users should be aware that the metrics used are automatic and therefore themselves might also make mistakes and misrepresent model performance (i.e., the metrics require separate benchmarking themselves). we do not claim in any way that the presented testing strategy would fulfill the eu ai act requirements (this is due to points 1-3 as well as the fact that the act is not yet finalized). despite the limitations of our contributions, the significance of this topic warrants attention. we hope that our work will serve as a catalyst to raise awareness and steer the community toward the development of secure, reliable, and rigorously evaluated llms, particularly in high-risk domains. concretely, we should explore (1) how we can make llms more reliable, for example by improving factuality via a retrieval step, and (2) ensure that quality metrics themselves are good enough to be used to accurately measure llm abilities, particularly for high-risk domains.