Quantification of Biodiversity from Historical Survey Text with LLM -based Best-Worst-Scaling (2025)

Abstract: In this study, we evaluate methods to determine the frequency of species via quantity estimation from historical survey text. To that end, we formulate classification tasks and finally show that this problem can be adequately framed as a regression task using Best-Worst Scaling (BWS) with Large Language Models (LLMs). We test Ministral-8B, DeepSeek-V3, and GPT-4, finding that the latter two have reasonable agreement with humans and each other. We conclude that this approach is more cost-effective and similarly robust compared to a fine-grained multi-class approach, allowing automated quantity estimation across species.

## introduction long-term observation data plays a vital role in shaping policies for preventing biodiversity loss caused by habitat destruction, climate change, pollution, or resource overexploitation @xcite . however, these efforts depend on the availability of reliable and relevant historical data and robust analytical methods, a significant challenge due to the heterogeneity of records representing such data. the available biodiversity data varies widely in resolution, ranging from detailed records (e.g., point occurrences, trait measurements) to aggregated compilations (e.g., floras, taxonomic monographs) @xcite . many projects, such as the global biodiversity information facility (gbif), focus largely on the disaggregated end of the spectrum, particularly with presence/absence data @xcite . furthermore, despite their utility, longitudinal data is largely confined to records from after 1970 @xcite , leaving significant historical gaps. natural history collections and records from the archives of societies present valuable opportunities to extend data further back in time @xcite . such sources are rich, but typically unstructured and require sophisticated extraction tools to produce meaningful quantitative information. recent advances in nlp have shown promising potential for retrieval-based biodiversity detection from (mostly scientific) literature @xcite @xcite . this paper focuses on evaluating methods for biodiversity quantification from semi-structured historical survey texts. to achieve this, we test tasks to distill meaningful metrics from textual information found in survey records. a particular focus lies on the feasibility of best-worst scaling (bws) with a large language model (llm) as an annotator, which promises greater efficiency and cost-effectiveness compared to manual annotation @xcite . in the following, we describe the data, outline the tasks and machine learning methods, and finally present a case study. ## data in 1845, the bavarian ministry of finance issued a survey to evaluate biodiversity in the bavarian kingdom, a region that encompasses a variety of different ecosystems and landscapes. to that end, 119 forestry offices were contacted to complete a standardized questionnaire. namely, trained local foresters recorded in free text how frequently 44 selected vertebrate species occurred in the respective administrative territory, and in which habitats and locations they could be found. figure 1 shows the facsimile of a digitized survey page. it features a header containing instructions and a number of records describing animal species with their respective responses. these historical survey documents are preserved by the bavarian state archives (cf. @xcite . the archival sources were digitized, transcribed from the handwritten original and enriched with metadata, including, among others, taxonomic norm data according to the gbif-database 1 (telenius, 2011) and geographical references to forestry offices. this data set is freely available on zenodo @xcite . in total, the data set contains 5,467 entries 2 among which are also a number of empty (striked out) or 'see above'-type responses. the unique set we used for our experiments contains 2,555 texts. we find that the foresters' replies vary considerably in length where most texts contain 3 to 10 tokens and only a few texts more than 20 tokens. ## tasks & experiments the main task in this paper is to assign a quantity label to a text, indicating the frequency with which an animal species occurs in a specific area. this can be operationalized in various ways, either 1 gbif.org 2 including species that were not explicity prompted. we evaluate three different task setups, foot_0 as detailed in ## case study for a proof of concept, we map the predictions of the regression model (labse transfer regression model based on gpt-4 bws) to the multi-class human annotation. figure 3 shows a strong relationship between multi-class labels and regression scores for the entire dataset (four species), but also that the extinction class is not properly represented in the regression, and furthermore that higher values are challenging to predict. ## conclusion & future work this study demonstrates that information of occurrence frequencies from semi-structured historical biodiversity survey texts can be adequately modeled with best-worst scaling through llms. while a simple classification approach performs well with minimal training data, a more complex classification struggles with design decisions and imbalanced data. bws meets this by eliminating rating scale design decisions. in addition, it is cognitively and computationally less expensive, since no manual annotation of training data is necessary, while still offering similarly accurate results with much finer granularity through regression. the robustness of methods and models should be further tested, not exclusive to biodiversity surveys, lending itself to a number of tasks. yet, similar data to ours likely exists, e.g., on 19th century bavarian flora, württembergische oberamtsbeschreibungen (1824-1886), or data in biodiversitylibrary.org, making our methods valuable.