Large Language Models as Annotators of Named Entities in Climate Change and Biodiversity: A Preliminary Study (2025)

Abstract: This paper examines whether few-shot techniques for Named Entity Recognition (NER) utilising existing large language models (LLMs) as their backbone can be used to reliably annotate named entities (NEs) in scientific texts on climate change and biodiversity. A series of experiments aim to assess whether LLMs can be integrated into an end-to-end pipeline that could generate token- or sentence-level NE annotations; the former being an ideal-case scenario that allows for seamless integration of existing with new token-level features in a single annotation pipeline. Experiments are run on four LLMs, two NER datasets, two input and output data formats, and ten and nine prompt versions per dataset. The results show that few-shot methods are far from being a silver bullet for NER in highly specialised domains, although improvement in LLM performance is observed for some prompt designs and some NE classes. Few-shot methods would find better use in a human-in-the-loop scenario, where an LLM’s output is verified by a domain expert.

## introduction analysing the language of climate change is an important step in following and understanding ongoing debates in this field. in a corpus linguistics setting, an important precondition for performing such an analysis is the access to corpora that have been annotated with morpho-syntactic and semantic features at the token level. named entities (nes) belong to the latter category and constitute an important part of linguistic analysis @xcite underline that linguistic choices in terms of decisions to explicitly name or leave out a certain entity or concept is an important notion in analysing political speeches. this line of thinking can easily apply to texts of various genres from the climate change domain, too. in many instances, available corpora for corpus linguistics research, such as those hosted on english-corpora.org, foot_0 rarely offer token-level annotations that extend beyond lemma and part-ofspeech (pos) tags, and eventually syntactic dependency tags. these corpora can be obtained as pre-tokenized data; preserving existing tokenlevel features and enriching them with custom ne annotations is contingent upon having (a) an annotation tool capable of processing tokenized @xcite having sufficient data to train a custom ner component within the tool. depending on the annotation tool, such training data must usually be annotated in the iob or bioes/bilou format foot_1 or contain character span information about the ne instance. challenge (a) is alleviated by the fact that (1) some known annotation tools, such as stanza @xcite and trankit @xcite , can accept pre-tokenized input, and (2) obtaining high-quality morpho-syntactic features by reannotating the corpus is generally unproblematic. 3 challenge (b) is more complex, especially concerning the annotation of specialised corpora. in the context of adding climate-change-related token-level ne annotations that would be relevant to analysing the scientific climate change discourse, which could involve ne categories such as greenhouse-gases or climate-datasets, the first step would be to define a set of relevant categories, and the second to obtain a high-quality annotated corpus of sufficient size to train an ner compo-nent of a tool for linguistic annotation. creating richly annotated specialised corpora is thus a timeand resource-intensive activity. meanwhile, large language models (llms) have been seen as possible "destabilizers" of "inequalities of academic research", as they might allow moderately-funded labs to perform analyses that were previously accessible to well-funded institutes only @xcite . motivated by the positive results in llm-powered few-shot ner for specialised domains reported in @xcite , this paper employs a number of few-shot experiments to investigate whether this "destabilization effect" also transfers onto the annotation of nes in scientific literature on climate change and biodiversity. experiments undertaken in the scope of this study should answer two questions: (q1) can llms be used as reliable "annotators" of named entities at the token and sentence levels in the domains of climate change and biodiversity? and (q2) does providing tokenized input affect an llm's performance when identifying named entities in these domains? descriptive information about the datasets used in the study and extensive supplementary materials related to the experiments and the results are available in a dedicated github repository. 4 finally, an effort is made to refrain from using anthropomorphic language when disucssing llms @xcite , as long as this does not hinder the description of llm-based systems, methodologies and functionalities. the advent of transformer-based lms has put the limelight on transfer learning and finetuning, methodologies that demonstrate robust results with fewer manually labelled training examples. in fine-tuning, the architecture of an lm is modified in line with the task requirements: @xcite present a methodology for learning an lm to understand language structure, and then test its performance on downstream tasks including ner. many of the tools developed in this way, such as biodivbert @xcite , are models that have been developed for an ner task only and merging their output with the morpho-syntactic token-level features obtained from a linguistic annotation library is not always a seamless process due to variations in tokenization approaches. 5 the increased availability of open-source and paid text-generation and question-answering models, alongside reports of pre-trained llms performing well on nlp tasks in zero-and few-shot settings in data-poor contexts @xcite , have fuelled the interest in experimenting with zero-shot and few-shot ner approaches. in most instances, this means that ner is defined as a question-answering task, where the llm is expected to generate an answer based on a prompt sent to the system. epure and hennequin (2022) perform zero-shot and few-shot ner using gpt-2. before prompting the model, they ensure a low ambiguity level between ne categories by merging possibly confusing ne labels into a single, unambiguous label. they also simplify the task by prompting the model to recognise one ne category at a time. @xcite ensure that the input sequence from which the model is expected to extract nes is semantically similar to the example sequence in the prompt template by retrieving the k nearest neighbour of the input sequence. they also prompt the model to enclose the ne into special tokens, which should allow for span retrieval. @xcite have presented an intuitive approach to ner, where they propose a prompt template that can easily be customized to any project using own ne categories and definitions. their approach has been implemented in spacy-llm's ne annotation pipeline, where users can define ne categories on the fly and annotate their data with an lm of their choice. 6 this study builds on existing work in the field of few-shot ner and conducts experiments using different prompt templates and a varying number of task examples. it differs from previous methods in (1) the format of the input given to the model and the requested output, and (2) the use of highlyspecialised ner datasets, which, to the best of my knowledge, have not been used in a few-shot ner setting previously. ## data basic descriptive information about the two ner datasets that are used in the experiments described in section 4 is provided below; a comprehensive dataset description involving definitions of each ne class, information about the distribution of ne instances per category and per data split, descriptive statistical sentence-and token-level information, as well as the ten most and least frequent ne instances per each ne class, are provided in the dataset documentation available in the dedicated github repository referred to in section 1. climate-change-ner is a publicly-available dataset foot_6 for english-language ner in scientific texts on climate change, developed in an ibm research ai foot_7 -led initiative, involving nasa foot_8 @xcite among other organisations. the dataset has 13 climate-specific ne classes, which originate from complex taxonomies used in climate-related literature. these are: climate-assets, climate-datasets, climate-greenhouse-gases, climate-hazards, climate-impacts, climate-mitigations, climatemodels, climate-nature, climate-observations, climate-organisms, climate-organizations, climate-problem-origins, and climate-properties. seed keywords, such as wildfire and floods, had been used to collect a total of 534 abstracts from the semantic scholar academic graph @xcite , which were then manually annotated with the iob tagging scheme, with the help of a set of class-specific dictionaries @xcite . the train and test data splits, which are used in the experiments of this paper, contain 985 and 177 sentences and 3029 and 555 ne instances respectively. biodivner is a publicly-available dataset foot_9 for english-language ner in the biodiversity domain @xcite . the dataset has 6 biodiversity-related ne classes: organism, phenomena, matter, environment, quality, and location. the annotated corpus comprises of abstracts, tables, and metadata files collected by using a set of keywords from semedico, foot_10 bef-china, 12 and data.world 13 and manually annotated with the iob tagging scheme. biodivner's train and test data splits contain 1828 and 229 sentences and 6709 and 1277 ne instances respectively. ## methodology this section presents the steps taken to preprocess the data, the prompt design, the llms used in the experiments, the evaluation approach, and the baseline against which the llms' performance is compared. ## results and analysis tables 3 and 4 summarize the f1 scores for experiments conducted on the test data split of climate-change-ner and biodivner involving the prompts described in section 4.2. one iteration was performed on each prompt set and on each model. in the tables, k stands for the number of te pairs included in the prompt. prior to calculating the results, each model's output was cleaned from misspelled or non-existing categories (e.g. organsim instead of organism). tables 5 and 6 present the percentage of spanand-category matches between a model's predicted nes and the gold standard. span-andcategory matches measure instances where the model correctly identifies the span of an ne instance and the ne class. for token-based input and output, this means that the token indices are not taken into account when calculating the percentage of span-and-category matches, while for string-based input and output, the model is not expected to generate token indices at all. therefore, these two tables allow one to gauge the degree to which a model is affected by the input-output format. ## discussion and future work the experiments reveal that few-shot ner methods are not a turnkey solution for highlyspecialised ne annotation at token-and sentencelevel, which answers q1 and further highlights the importance of reflecting on and reporting llms' limitations on domain-specific tasks, especially at a time of benchmark-centric research. nevertheless, the results also reveal possible use cases for llms in the context of ner, which include testing the robustness of datasets and further simplifying the task by focusing on isolated ne classes and extensive task descriptions; both of these are discussed in subsection 6.1. regarding the input-output format investigated within q2, the experiments show that llms achieved slightly better performance on tokenbased than on string-based input. a plausible explanation for this might be that repeated ne instances in a single sentence are more likely to be identified with a token-based approach, as the llm processes each token individually. in future iterations, it would be useful to investigate whether the prompt for string-based processing could benefit by including an instruction for the llm to extract repeated occurrences of the same ne instance. since llms' performance could improve with more context, it would be worthwhile investigating whether redefining the string-based prompt as a document-level ner task would yield better performance. finally, it was noticed that the biodivner dataset contained many tokens that were remnants of pdf parsing, which might also have affected the llms' output for string-based prompts. in many cases, there was an overlap in the classes on which the llms performed well or poorly. the experiment results seem to hint that the complexity of the task could be rooted in the llms not having been exposed to sufficient data about the specialised domains. it would be interesting to test this approach on a domain-specific llm developed for climate change questionanswering, such as models belonging to the cli-mategpt family @xcite . unfortunately, this was not realistic for this study due to infrastructure constraints. ## ethical considerations this study uses publicly available datasets. the experiments do not require specialised infrastructure and can be reproduced using an api and the prompts provided in the dedicated github repository. the costs for all experiments, per language model family are: ca. 40 eur for openai's gpt4 models, and ca. 20 eur for llama's 3.1 models.