Uncertainty Estimation in Large Language Models to Support Biodiversity Conservation (2024)

Abstract: Large Language Models (LLM) provide significant value in question answering (QA) scenarios and have practical application in complex decision-making contexts, such as biodiversity conservation. However, despite substantial performance improvements, they may still produce inaccurate outcomes. Consequently, incorporating uncertainty quantification alongside predictions is essential for mitigating the potential risks associated with their use. This study introduces an exploratory analysis of the application of Monte Carlo Dropout (MCD) and Expected Calibration Error (ECE) to assess the uncertainty of generative language models. To that end, we analyzed two publicly available language models (Falcon-7B and DistilGPT-2). Our findings suggest the viability of employing ECE as a metric to estimate uncertainty in generative LLM. The findings from this research contribute to a broader project aiming at facilitating free and open access to standardized and integrated data and services about Costa Rica’s biodiversity to support the development of science, education, and biodiversity conservation.

## introduction the signatory countries of the convention on biological diversity (cbd) of the united nations (un) have committed to safeguarding and sustainably using the planet's biodiversity @xcite . however, countries lack comprehensive data and the application of biodiversity knowledge in decision making has been limited (secretariat of the convention on biological diversity, 2020). much of the required data to address this need are in text format and are part of the globally available taxonomic literature. taxonomic literature keeps records of the planet's biodiversity and gives access to the knowledge needed for research and sustainable management. the number of publications generated is quite large: the corpus of biodiversity literature includes tens of millions of figures, taxonomic treatments, and other technical documents. unfortunately, most of the taxonomic literature is in text format. the biodiversity heritage library @xcite , the world's largest open access digital library for biodiversity literature and archives, integrates more than 61 million digitized pages. additionally, our local project, that works with data on the biodiversity of costa rica, has a database with over 32 thousand records containing information such as scientific name, morphological description, common names, species distribution, life cycle, feeding, reproduction, demography, habitat, phenology, behavior, interactions, conservation status, and legislation, among other types of content. obtaining highly structured records from digitized text has been shown to be complex and very expensive @xcite . also, there is much left to document. the scientific community has described over 1.2 million species, but studies suggest that 86% of existing species on earth and 91% of species in the ocean still await description @xcite . the published descriptions synthesize observations ## background recent efforts to assess uncertainty in generative llm have introduced various algorithms. one approach, as proposed by @xcite and @xcite , involves training a model to generate an uncertainty estimate alongside the generated text. another strategy, suggested by @xcite , utilizes clustering techniques to estimate uncertainty. in their work, an 3 unsupervised method known as semantic entropy is introduced. this method leverages linguistic invariances to group texts based on shared meanings, offering an uncertainty estimation using clustering. in addition, @xcite have proposed a method that takes into consideration the relevance of tokens within a sentence or the relevance of sentences within a text. their approach involves implementing an attention shift mechanism that adjusts attention based on the importance of tokens or sentences, ultimately influencing how uncertainty is estimated in the generated language model. ## results ours findings indicate the feasibility of using ece as a metric for estimating uncertainty in generative llm. the outcomes obtained through perplexity and mcd highlight the disparity in text quality between distilgpt-2 and falcon-7b (figure 4 and figure 5 ). across all experiments, distilgpt-2 demonstrates a higher calibration error compared to falcon-7b, with an average ece using perplexity of 0.2539, as opposed to falcon-7b, which registers an error of 0.0898 ( the calibration of the biodiversity project models not only involves parameters such as dropout rate but also parameters related to text generation. the quality of the texts generated by llm rely on randomness in the decoding method, in particular through penalty alpha, top-k, and topp variables that guide the selection of next word. the next word generated by a model is selected from the top k most probable choices in the model's vocabulary given a set of previously generated words or from the smallest set of tokens for which the cumulative probability exceeds a specified value, p. @xcite . the biodiversity project already has a portal that provides free and open access to biodiversity data. the fine-tune llm will complement the services available on the portal to support the development of science, education and biodiversity conservation. for conducting tests with data from the biodiversity project, we have a cloud-based server (4x gpu nvidia® t4 and 192 gb ram or some similar configuration) at our disposal, and we plan to use an open multilingual llm with state-of-theart performance (e.g. falcon-40b) for the process. ## 5 conclusion ensuring model calibration is vital as it guarantees not only accurate but also reliable probability estimates. our exploratory study gives preliminary data on uncertainty estimation in generative language models using mcd, perplexity methods for uncertainty quantification, and bertscore. according to the results yielded for the two evaluated models, the performance of both uncertainty scoring methodologies can be thought to be similar. however, more testing is necessary, with more replicas and a calibration process using a holdout dataset. identified areas requiring future scrutiny and enhancement include: a) model selection -the choice of falcon-7b and distilgpt-2 was constrained by computational resources. however, there is a necessity for a superior-performing, multilingual, open-source language model like falcon-40b, particularly for testing texts written in spanish. b) the results obtained are encouraging; however, it is necessary to involve humans in the following experiments of the project. c) evaluating model confidence using metrics beyond perplexity, as suggested by @xcite , who propose a framework of metrics based on language laws. d) prompt engineering evaluation -assess diverse prompt strategies designed to improve text generation.