Evaluating LLM -Prompting for Sequence Labeling Tasks in Computational Literary Studies (2025)

Abstract: Prompt engineering holds the promise for the computational literary studies (CLS) to obtain high quality markup for literary research questions by simply prompting large language models with natural language strings. We test prompt engineering’s validity for two CLS sequence labeling tasks under the following aspects: (i) how generalizable are the results of identical prompts on different dataset splits?, (ii) how robust are performance results when re-formulating the prompts?, and (iii) how generalizable are certain fixed phrases added to the prompts that are generally considered to increase performance. We find that results are sensitive to data splits and prompt formulation, while the addition of fixed phrases does not change performance in most cases, depending on the chosen model.

## introduction large language models (llms) have taken over the field of natural language processing (nlp) in the past years. llms implement the transformer architecture and are fine-tuned to follow instructions @xcite , which also led to the introduction of a new paradigm: 'prompting'. 1 in contrast to pre-training, fine-tuning or classical machine learning, prompting does not actually update the weights of the model itself. instead, prompt strategies aim at producing the best possible prompt for a given task @xcite , thus providing a textual context for the model to generate reasonable replies. llm-prompting is a promising development for digital humanities in general, because task descriptions can be expressed in natural language, presumably making it easier to connect to classical, non-digital research in the humanities. this may also apply to the model's output, if it is in natural language or can be verbalized (correctly) as such. a distinction can be made between two prompting scenarios: i) interactive prompting, as with a chatbot, is the scenario in which most people currently experience llms, as it is easily available even without technical background. it is characterized by a direct application and associated implicit validation, often used in an exploratory manner. note that results obtained must not be perfect or even correct to be useful, and in following gricean conversation maxims @xcite , human users put in interpretation effort to make sense of the results. ii) batch-use comes into play if prompts are applied to a large(r) quantity of data, and the llm is used for automatic detection of some textual concept. this paradigm is closely related to established machine learning scenarios, and thus needs to follow established machine learning best practices. the remainder of this article is about this batch-use of llm prompting. evaluation of llms can also be separated into two areas: i) with the goal of evaluating llms as such (and unrelated to a specific task), they are usually confronted with test items from multiple benchmark data sets that cover a certain range of tasks. arc @xcite , for instance, defines 7787 natural science questions with four possible answers, out of which one is correct. the model is tasked to provide the identifier of the correct answer. models can then be ranked according to their (average) performance on such benchmarks, resulting in rankings such as the huggingface open llm leaderboard 2 . ii) for a task-specific evaluation, reference data for the specific task is needed, and allows comparing system and reference output as is established in machine learning. in both evaluation setups, it is important to realize that what is evaluated is not (only) the model itself, but a tuple of model, task formalization, parameters and prompt, and that an exhaustive evaluation of all possible settings is usually not possible. this paper, as do many others, selects a number of parameters for the experiments and this selection has theoretical and pragmatic reasons. this paper explores the use of llm-prompting in computational literary studies (cls). cls analyzes literary texts and text corpora using methods of statistics, machine learning and nlp. in doing so, cls draws partly on traditional literary studies, but does so with the help of data-driven approaches and methods. past studies in cls focused on authorship attribution, drama and genre analysis, literary-historical questions, narratological and gender analysis and questions of canonicity (cf. @xcite @xcite . non-computational literary research questions are typically highly complex, context-dependent and embedded in a deep theoretical framework, that is often expressed somewhat vaguely. addressing such questions thus requires a multitude of tools and methods that form components in an argumentation that uses manual and automatic work steps. the tasks we discuss in this paper are representative for such components. concretely, this paper's contribution is the systematic evaluation of a number of llms and prompts on two different cls-relevant sequence classification tasks for which manually annotated reference data sets exist. sequence classification in nlp is the task of assigning a categorical label to each element in a sequence of data, such as words in a sentence or characters in a word. such tasks are complex as they combine two potentially separate work steps in one: the selection of a token span to be classified and the classification of this span. such tasks are common in cls as manual annotation tasks. 3 an important methodological aspect of such an evaluation is that as soon as prompting strategies make use of manually or automatically optimizing prompts on a data set ("prompt engineering"), this needs to be treated as a training process, even if no weight updates are performed: selecting the best prompt on a data set and evaluating its performance on the very same data set is a case of overfitting and the measured performance is not indicative of its performance on new data. this does not mean that performance on unseen data must be lower in every case -if the model-promptcombination has generalized properly, it may even achieve similar performance on unseen data. we suspect that in practice this optimization process is usually based on a small, hand-picked selection of examples, and often not evaluated on an independent test set. accordingly, to avoid overfitting, we propose to follow established best practices and make a (documented) split into train and test data, with similar roles as in classical machine learning: train data is used to optimize a prompt and test data to evaluate it. ## related work several studies in nlp use llms for classic classification tasks. balkus and yan (2023) use gpt-3's api to classify the topics of short texts and use both the generative completion capabilities as well as a dedicated classification end point of the api. @xcite use chatgpt to classify agriculture-related texts with regards to sentiment, prediction of natural disasters and text topic. @xcite test gpt-3.5, gpt-4 and llama 2 on, among others, sentiment analysis of tweets. in addition to this, @xcite show that in the binary classification of qualification requirements for job advertisements, llms such as openai's text-davinci-003 model clearly outperform classical ml approaches such as svm but also smaller 'foundational models' such as debertav3. many studies investigate the influence of prompts for prediction performance @xcite @xcite @xcite . all come to the conclusion that the form and quality of manually crafted prompts is highly influential on performance and often suggest methods for automatically generating prompts or using methods such as prompt tuning to circumvent the shortcomings of hard prompts. many studies distinguish systematically between different prompt components, such as "definition", "things to avoid", etc. @xcite . @xcite investigate which words are most important in a prompt by systematically replacing words in prompt components like "let's think step-by-step" and measure the change in performance via a newly introduced metric. they find that nouns are consistently among the most important words regarding prediction and that the most important word varies according to the task performed. @xcite demonstrate how single prompts lead to chance-based outcomes and suggest using a suite of prompts and averaging over their performance (this strategy is called 'prompt ensemble' in @xcite ). lastly, @xcite suggest that the proclaimed emergent abilities of llms disappear once appropriate evaluation metrics are used. the largest study on the usages of llms for classification tasks in a computational humanities context to date comes from @xcite . they work in the context of computational social science and perform zero-shot learning on a wide variety of tasks on different textual levels such as sarcasm and ideology detection, misinformation detection, empathy classification, politeness, event detection and roles and tropes. the study uses one prompt template per task and does not address the potential impact of prompt brittleness on the evaluation. they find that, except for certain minor tasks, llms in a zero-shot setting are not able to outperform fine-tuned classifiers or replace the work of human annotators @xcite . @xcite come to a similar conclusion in the context of an icl-experiment in the cls, in which they investigate the extent to which openai's text-davinci-003-llm can reproduce the performance of smaller older models used by @xcite in the course of a classification task based on complex knowledge from literary theory, namely the determination of domain specific generalizing statements in literary studies. @xcite tested several open and closesourced llms in zero and few-shot setups on the task of identifying knowledge transfers about family relations in german dramas. they also conclude that, in the current state, llms are not suitable to sufficiently perform high-level cls classification tasks out-of-the-box. @xcite , recently published as a pre-print, arrives at differentiated results. the study identifies ten tasks from computer-assisted text analysis, characterized as cultural analytics, for which annotated reference data is available, and investigates how well these tasks can be solved by llms compared to pretrained language models (plms). the choosen llms are gpt-4o, llama 3 70b and mixtral 8x22b, which are prompted with a single prompt template containing 10 examples but no chain-of-thought-prompts. they find that "llms offer competitive performance through prompting alone for established tasks, while traditional supervised methods excel for newly constructed phenomena (even in scenarios with limited training data)". in a further comparison, for which the models were fine-tuned on the task-specific reference data, the performance differences between masked plms and llms are even smaller. issues of prompt brittleness and prompt generalizability are not addressed. ## sequence classification tasks and data this section describes the two sequence classification tasks (emotion and event) and data sets used in our experiments. note that the event dataset is in german, while the emotion dataset is in english language. regarding the issue of data leakage @xcite , please also note that both the emotion and event dataset are publicly available. it can therefore not be excluded that (parts of) the public data sets and their annotated labels have been included in the pre-training of our models. emotion the dataset for the emotion task is coming from work by @xcite and is called reman (relational emotion annotation for fiction). they provide annotations of 200 english texts from project gutenberg 5 and annotate the emotions anger, anticipation, disgust, fear, joy, sadness, surprise and trust plus a category other emotion for cases that do not fall into one of the above. annotated is either a single word or phrase with a preference for shorter spans. for instance, the annotated span for the sentence "his smile was distinctly attractive." is "smile" and was given the joy-label. in a multi-step process, all spans that do not match exactly between annotators, but overlap, were adjudicated by an expert. kim and klinger provide baseline experimental results on predicting emotions on their dataset, using dictionary and bag-of-words-based baselines, a conditional random field (crf) model as well as a long short-term memory model (lstm) architecture with a crf classification on top. the lstm-crf performs best with an f1 score of 43 % in a strict setting where all spans have to match exactly, but the authors note that recall is low for both models. they report inter-annotator agreement scores for their annotations, ranging from an average cohen's κ of 0.11 for anticipation to a κ value of 0.35 for joy. see the original annotation task consisted of three parts: in a first step, the annotation span had to be identified, in a second step it had to be marked with the corresponding labels, and then in a third step subordinate property tags had to be assigned. following this procedure, they achieved an agreement for these event types of krippendorff's α between 0.57 and 0.75, depending on the text. to our knowledge, there are currently no published studies on automatic annotation of the dataset. examples for annotation spans for each of the four categories look like the ones in ## formalization in this section, we describe which measurement techniques we use to answer the three research questions introduced above. in general, our prompts consist of a frame structure describing the role of the llm, the task, the expected output format, and the labels to be used, with slots for variable components and the text to analyze: a prompt is thus defined as a complete input sequence that realizes one of 8 possible combinations of so-called prompt components, where prompt components are elements that can be switched on and off. the implementation of one of these 8 possible combinations as a prompt, we call prompt configuration. additionally, there are 3 paraphrases (semantically equivalent reformulations) of each prompt. these were generated automatically by using gpt to generate 10 alternative reformulations based on an initial manually created prompt that follows current prompt engineering recommendations, from which we then manually selected three. all in all, this leads to 4 * 8 = 32 different prompt configurations -for each model and each task -which results in a grand total of 64 different prompts and 256 model runs. ## experiments we carry out experiments on all tasks described above, using the following llms: gpt foot_5 (gpt-4o foot_6 ), llama (llama3.1-8b-instruct (ai@meta, 2024) foot_7 , mixtral (mixtral-8x7b-instruct (jiang et al., 2024) 11 ), and sauerkraut (sauerkrautlm foot_9 ). the models provide a balance of close and (semi-)open source systems and with sauerkraut there is a model that was especially re-pretrained on german language texts. furthermore, all models displayed high scores on popular nlp benchmarks and should therefore generally be able to tackle the two cls tasks. due to the computer resources available, we quantified llama and sauerkraut into a 4-bit version using huggingface's bitsandbytes library. ## discussion dividing the test data into two sub-data sets (rq1) shows a clear tendency: all four models perform in a statistically significant way differently on the two data sets. this is arguably not specific to prompting or large language models, but a general property of machine learning approaches, although we are not aware of work that systematically investigates this. we believe this to be a consequence of how test data is sampled, how much variety of the phenomenon it covers, and, ultimately, how representative the selected test sample is for other test samples or the 'population' in general. in particular the latter question is not easy to answer, given that we are dealing with historical and cultural data, which is subject to a number of highly intransparent selection processes (cf. @xcite . still, as it has been hinted that large language models "understand" a prompt @xcite foot_11 (which nobody has claimed for classical machine learning algorithms), it can be argued that if the models would have understood those prompts, they would not show a statistically significant difference on different test data sets. the fact that different prompts lead to different responses (rq2) is not surprising per se. what @xcite have uncovered is that meaning-preserving prompt variants (e.g., spelling variation or paraphrases) also lead to different responses, and that -when ranking models for their performance -the exact prompt formulation has tremendous influence on the ranking of such models. they therefore recommend to use the mean performance over multiple prompts. generally, we also observe a difference in f1 score depending on the exact prompt. while model ranking is not our prime goal here, different model rankings can be established from our experiments as well -which makes the search for the 'best model' for a given task more complex. however, the differences we observe are rather modest, with standard deviations over various prompt variants between 2 and 6 points in f1 score. still, if the overall absolute performance results were better, a difference in this range could very well have impact on the applicability of such a model in practice. to address specific tasks, there is no alternative to having annotated reference data and experimenting with different formulations and parameters. at the same time, exhaustively searching the best setting is impossible. finally, we have investigated recommendations that are often given for manually constructed prompts (rq3), on what to include in the prompt. our results support these recommendations only partially. first of all, we see different results for different tasks. across the two tasks discussed in this paper, we can only extract three clear trends: a) llama seems to benefit from using the stepscomponent (asking the model to think step by step). b) the same component seems to be detrimental for the sauerkraut model. c) sauerkraut, on the other hand, benefits from the bribe-component for both tasks. for all other components and models, no tendency can be discerned. in general, across the three research questions in this paper, there seems to be a generalizability issue (which is also discussed in recent papers in philosophy of science, cf. @xcite . generalizing from any scientific experiment to the 'real world' (or, more technically, from lab data to application data) rests on certain assumptions about model behavior and data sets. this applies first to the performance measures that have been achieved on a test set -assuming representativity of the test set, performance will be roughly similar during application. this is, in practice, impossible to control and verify. secondly, as the actual performance of a model-prompt-pair varies substantially depending on prompt variations, it is impossible to recommend a model or prompt formulation that is in general beneficial to the performance results. this holds not only to the formulation variants of a prompt, but, thirdly, also to the selection of prompt components. while there is no reason to believe that the same prompt component will always be beneficial (or detrimental) to the results -properly establishing prompt components that often lead to better results would require either a huge project or a number of meta studies that investigate many different existing publications. conversely, the scientific use of llms and prompting as a ml technique is usually not about general chat functionality (as is a smart personal assistant or "general artificial intelligence"), but about very specific questions and tasks. the general performance of a llm (measured on some benchmark) may not be indicative for the specific tasks that a researcher from cls has as their goal. for solving specific tasks, using reference data as train/test data still is the only way to systematically search for the best performing combination of model, prompt and parameters. ## conclusions we were able to show that (i) llm models are sensitive to data splits (ii) the choice of prompt-model combination determines the success in performance to a high degree and (iii) the helpfulness of fixed components in the prompts to increase performance can not be corroborated for all models for the given tasks. overall, it could also be shown that all tested models have problems to reach satisfying results on both tasks (emotion and event sequences classification), casting doubt on the immediate usefulness of in-context, zero-shot llm-sequence-classification for the given cls tasks.