Evaluating In-Context Learning for Computational Literary Studies: A Case Study Based on the Automatic Recognition of Knowledge Transfer in G erman Drama (2024)

Abstract: In this paper, we evaluate two different natural language processing (NLP) approaches to solve a paradigmatic task for computational literary studies (CLS): the recognition of knowledge transfer in literary texts. We focus on the question of how adequately large language models capture the transfer of knowledge about family relations in German drama texts when this transfer is treated as a classification or textual entailment task using in-context learning (ICL). We find that a 13 billion parameter LLAMA 2 model performs best on the former, while GPT-4 performs best on the latter task. However, all models achieve relatively low scores compared to standard NLP benchmark results, struggle from inconsistencies with small changes in prompts and are often not able to make simple inferences beyond the textual surface, which is why an unreflected generic use of ICL in the CLS seems still not advisable.

## introduction computational literary studies (cls) is a subfield of digital humanities. cls attempts to expand the traditional methods of literary studies to include quantitative approaches with the help of statistical methods and machine learning or natural language processing (nlp) @xcite . the latter has made enormous progress in recent years: at the latest since the development of the transformer architecture @xcite large language models (llms) based on it have been breaking traditional nlp-benchmarks. until 2020, the development and domain-adaption of these models was dominated by an approach that emerged as a result of the bidirectional encoder representations from transformer models (bert, @xcite , which has been termed the pretraining and fine-tuning paradigm. this practice has now also arrived in cls, as a recent survey on machine learning in cls shows @xcite . however, another paradigm shift in nlp appeared when the developers of @xcite , an autoregressive llm with around 175 billion parameters, showed in 2020 that with the help of a few examples ('few-shot learning') without fine-tuning and exclusively through natural language interaction, the model not only corresponded to the performance of predecessor models in numerous nlp tasks, but even outperformed them. this type of conditioning of a llm to perform a specific task using only a natural language input and no gradient update is called 'in-context learning' (icl, @xcite . with the publication of chatgpt, icl has gained popularity among the general public, but its potential and limits for cls has yet to be determined. ## related work while prompt engineering, which is often equated with icl, recently gained a lot of traction, studies based on it are still rare in dh in general and cls in particular. initial reflections and experiments can be found in computational science studies (css). @xcite investigate the potential of llms for css by investigating the viability of zero-shot-learning for sociological research. overall, they posit that llms perform well in certain zero-shot classification tasks within the context of social studies research, but "do not match or exceed the performance of carefully fine-tuned classifiers" @xcite 2). across their experiments, models demonstrate optimal performance in tasks related to misinformation classification, stance detection, and emotion classification. the authors attribute this success to the presence of either a ground truth (as in the case of misinformation) or an annotation schema that corresponds to (implicit) definitions of everyday concepts. 1 however, they also note that models perform worse in tasks requiring intricate expert taxonomies. this difficulty arises from the complex nature of expert-informed annotation guidelines, which may not align semantically with much of the llm's training data. 1 these assumptions correlate with the initial research into why and how icl works, summarized and brought together by @xcite and xie and @xcite who interpret this ability as bayesian inference of a latent concept conditioned on the prompt -a capability that arises from structure in the pretraining data. in the realm of nlp, several papers have explored icl for solving sets of diverse tasks (cf. @xcite @xcite @xcite , investigated methods for constructing reliable prompts (cf. @xcite @xcite and choosing suitable evaluation metrics @xcite . in terms of related tasks, the implicit_relations task from the big-bench benchmark @xcite seems to be closest to our setup, with @xcite achieving an accuracy score of 49.4% using a 70b parameter chinchilla model and @xcite an accuracy score of 36.4% using a 280b parameter gopher model. ## task we work on a paradigmatic task for computational literary studies: the transfer of knowledge about family relationships in german theatre plays. this type of task is paradigmatic for cls insofar as research questions from literary studies often focus on particular realizations of concepts in specific literary contexts or specific particularities of genres and single texts which are difficult to generalize and thus data sparsity becomes an issue. we conduct two experiments within the framework of icl in order to gain insights into how llms can be applied here: in the last scene of the last act of lessing's nathan the wise, for instance, nathan reveals that recha and the templar are siblings (see appendix a for the relevant segment). in this case, the classification goal would be to assign the class siblings. ## data we make use of one pre-existing dataset, a corpus of knowledge transfer annotations on german theatre plays pertaining to family relations @xcite . the authors understand knowledge in a broad way to be beliefs that are thought to be true by a certain character at a certain point in time but might later turn out to be wrong during the advance of the plot. the corpus contains 30 texts sampled from the german drama corpus @xcite . for each family relation that a character learns about, an annotation marks the source and target of the knowledge transfer, which family relation is being transferred and who is part of this relation, as well as additional, optional properties like lies or uncertainty. while the dataset in total contains 1277 annotated text passages, we removed the infrequent relation types as well as all annotations that do not represent knowledge transfer. this yields 89 annotations for our experiments, which are divided into four categories: parent of (29), child of (26), siblings (23) and spouses (11). while this is a rather small dataset, it is suitable for our premise of evaluating a typical cls task, since data sparsity is a common -and perhaps inherent -feature of cls tasks, as also mentioned earlier. the same dataset is used for experiments 1 and 2, whereby in the case of experiment 2, 39 instances are changed so that they are classified as propositions that are not entailed by the text snippet. for instance, if an annotation contains a parent of relationship, the proposition is changed to "[character x] is not the parent of [character y]". ## results llama 2 even explicitly justifies why it classifies actually entailed sentences as non-implied, as the following example shows: "the text does not explicitly state that iphigenie and orest are siblings. while it mentions "schwestern" (sisters) and "bruder" (brother) in the same sentence, it does not explicitly state their relationship. therefore, the proposition that iphigenie and orest are siblings is not entailed by the given text." foot_4 ## discussion our experiments show that the performance of an llm is strongly affected by its potential lack of 'understanding' of the task, or particular words in the prompt as well as minor changes in the prompt templates. this is insofar not surprising, as the dramas date from the period 1750-1910 -with the majority dating from the turn of the 18th and 19th centuries -and therefore use a language that is also literary in style and in some cases versified, for which there were most likely no examples in the training data of the models. recent studies investigating icl argue that both semantic priors and input-label mapping in the prompts influence the icl competence of llms, although the latter only applies to very large llms @xcite @xcite . with regard to our experiments, one can assume that in the training data of llama 2 the connection between the classification of an implicit knowledge transfer of a family relation and the fact that the same text snippet entails this family relation formulated in the form of a proposition, which is self-evident for a human speaker, was not represented. ## conclusion it is worth noting that one of the dangers of icl (and generative models in general) is the seemingly straightforward use of their output. we believe that using natural language output of such models is a regression compared to properly defined, symbolic output, as it requires interpretation and naturally contains ambiguities and terminological vagueness. "the worst dangers may lie in the humanist's ability to interpret nearly any result" @xcite , and this holds even more when the result comes in the form of natural language. it also follows from our experiments that an unreflected and generic out-of-the-box use of icleven with open-source llms -for the automation of analytical sub-steps in the cls is not yet recommended. the accuracy and f1-scores, although respectable per se, are still too low for this. methodologically, it follows that for each task-specific use of icl in the cls, it must be clarified in each specific case whether and how the selected llms represent the subject-specific vocabulary for this task validly and with a high degree of accuracy. for certain tasks, it should also be considered whether the breakdown of a complex concept into simpler everyday concepts potentially mastered by the modelan llm-specific operationalization of the complex concepts -does not achieve better scores. in the background of this methodological recommendation lie the following open research questions: is it really the case that llms perform better with icl if the models already have semantic prior knowledge of the task-specific concepts/vocabulary? what does this mean for the domain-specific vocabulary of cls? can this be generically categorized in such a way that a distinction is made between everyday concepts, which are likely to be represented by llms or can be represented in principle, and complex concepts, which are likely to be difficult to represent by llms? how suitable are more recent low-resource 'fine-tuningmethods' such as peft @xcite or lora @xcite for cls? the highly successful instruction-fine-tuning paradigm is rarely applicable in cls due to a lack of available data, but alternatives such as peft have so far only been tested on standard nlp benchmarks like super-glue. the extent to which these methods increase the accuracy of llms in cls tasks will have to be examined in the future.