Testing and Adapting the Representational Abilities of Large Language Models on Folktales in Low-Resource Languages (2024)

Abstract: Folktales are a rich resource of knowledge about the society and culture of a civilisation. Digital folklore research aims to use automated techniques to better understand these folktales, and it relies on abstract representations of the textual data. Although a number of large language models (LLMs) claim to be able to represent low-resource langauges such as Irish and Gaelic, we present two classification tasks to explore how useful these representations are, and three adaptations to improve the performance of these models. We find that adapting the models to work with longer sequences, and continuing pre-training on the domain of folktales improves classification performance, although these findings are tempered by the impressive performance of a baseline SVM with non-contextual features.

## introduction arguably the biggest development in natural language processing in recent years has been the use of pre-trained large language models (llms) such as bert @xcite to transfer the linguistic knowledge from models trained on massive corpora to other tasks, without the need for retraining. these llms have unlocked new avenues of investigation for many fields of research, such as digital folklore @xcite . progress has been particularly rapid for high resource languages, such as english, spanish, mandarin, as the high threshold of data required by llms is easily met by what is available online. however, despite multiple models claiming to represent a selection of low-resource languages, the relative scarcity of data available to train on may have a substantial impact on the model's ability to output a faithful representation of the language. this in turn can restrict digital folklore research to using older technologies, or to working on corpora from higher-resource languages. the current project is an ambitious contribution to folklore preservation and analysis which initially focused on digitally rendering thousands of handtranscribed tales in irish and gaelic @xcite . with this step complete, we now aim to use cutting-edge language technology to uncover cultural and linguistic links between the folklore traditions of ireland and scotland. our dataset presents two challenges: firstly that it comprises two low-resource languages -irish and scottish gaelic. we are interested in measuring how well llms represent the languages in our dataset, using the base version of the model with no adaptations. we also experiment with continuing fine-tuning of these base models with a bilingual corpus of folktale -i.e. domain adaptation. the second challenge is that many of our tales exceed the maximum context window allowed by common llms. this leads us to the following research questions: we operationalise these questions by setting up two classification tasks -to predict the type of folktale and the gender of the person who told the folktale. ## related work although the field of computational folkloristics is relatively new, predicting the tale type of a folktale has been approached a number of times. @xcite classified the tale type of dutch folktales using the learning to rank algorithm. the features to their system included measures of lexical similarity using tfidf representations and jaccard similarity, and subject-verb-object triplets extracted from the text. their nearest neighbours approach easily outperformed their baselines. @xcite focused not on atu prediction, but on predicting one of the elements which determines which atu number is appropriate -locations within the tale. they annotated a corpus of german folktales for location, with moderate inter-annotator agreement. they then used rule-based and statistical approaches to classify the locations. their best-performing system was a hybrid naive bayes system with some rule-based features. lô et al. (2020) experimented on a corpus of 742 west african and western european folktales, all translated into english. amongst their experiments, they trained classifiers to distinguish between the african and european tales, using a bow as input to a multi-layer perceptron, as well as word embeddings as input to a long short-term memory (lstm) system. the much simpler bow approach performed significantly better than the more complex lstm system, and the authors speculate that it was the small size of the dataset that led to this. @xcite used svms to predict the 10 most populated classes of the english-language atu-annotated tale collections available online (1,518 texts annotated with 182 atu tale types). they used tfidf features as input, and reported fscores ranging between 0.8 and 1.0 for the ten best populated classes. however they did not report any baseline model, so it is difficult to contextualise their achievements. related work on irish and scottish gaelic speech and language processing more widely can be found in @xcite , including work on handwriting recognition of the folktale data used for this project @xcite . the current paper is the first study applying computational text classification methods to irish and scottish gaelic folktales. ## dataset the irish and gaelic folktales dataset is a collection of 4,692 folk tales collected from two national folklore archives. the irish language data derive from a 40k page subset of the national folklore collection of ireland's main manuscript collection, hosted at university college dublin. this collection comprises over 700k manuscript pages of transcribed folklore interviews made by the irish folklore com-mission in the mid-20th century. ó raghallaigh et al. (2022) describe the digitisation and semiautomatic handwriting recognition (hwr) of our subset of this material, which amounts to 2091 transcription mss and 3,829,559 words.the average length of the irish folktales is 1,831.86 words (@xmath0,827.26). the gaelic data come from the university of edinburgh's school of scottish studies archives. @xcite describe how the dataset was created by scanning and semi-automatically recognising 2601 transcriptions and published versions of scottish gaelic folklore, amounting to 3,048,348 words. the average length of the scottish folktales is 5,869 @xcite . this dataset includes extensive metadata, such as the type of tale, and the gender of the interviewee who contributed the tale -these two variables are the focus of the below classification tasks detailed in section 4. the type of tale is determined by the atu number assigned to it @xcite . the atu scheme is a taxonomy used specifically for international folktales. each number represents a distinct tale type and was assigned to our tales by professional folklorists. atu numbers run from 1-2499 and have three main divisions: 1-299 (animal tales), 300-1199 (ordinary folktales) and 1199-2499 (jokes, anecdotes and formula tales). ## methodology we set up two classification tasks -gender and atu prediction. there were over 300 atu types represented in our dataset, so in order to avoid issues of data sparsity, we binned the tale types in our corpus into these four broad categories: 'animal' (atu 0-299), 'magic' (atu 300-745), 'ordinary (non-magic)' (atu 746-1,199), and 'jokes' (atu 1,200-2,499), in addition to one for tales of magic (aka 'fairy tales': atu 300-749). while all of the irish tales have been labelled with an atu number, only 451 gaelic tales were appropriate to label this way; those without an atu number were omitted from the classifier. the second classification task was to predict the storyteller's gender. the gender of the narrator was strongly skewed towards male in both the irish data (83.6% male) and the gaelic data (83.2% male). the gender was reported as 'unknown' for 58 of the tales in the dataset and these were omitted from the classifier. ## discussion in order to explore which multilingual llms give a faithful representation of two low-resource languages -irish and gaelic -we set up two classification tasks on a small folklore dataset, and measured the models' performance under four conditions: base model, domain-adapted, length-adapted and both adaptations. of the base models, the languageagnostic labse and the irish-only gabert models performed best. roberta and mbert did not beat a very simplistic baseline on one of the tasks, and performed poorly at the other. surprisingly, one of the best results came from an svm model with non-contextual features. of the two adaptations we presented, length augmentation improved our results more than domain adaptation, except in the case of labse. the combination of both adaptations gave the best results for gabert, mbert and roberta. several interesting points arise from our experiments. firstly, roberta and mbert are two of the most well-known transformer models, and their performance on our dataset indicates that they may not represent low-resource languages as well as well as they claim. even with adaptations for domain and length, they achieved minimal improvements on the classification tasks. secondly, it was not anticipated that gabert would perform so well, particularly in light of the fact that it is not trained on any gaelic data. however, the model's input data was tokenized with a sentencepiece tokenizer, and it is possible that there was positive transfer between irish and gaelic due to a high number of shared sub-word tokens between the two celtic languages. it is also notable that domain adaptation did not improve the model's results much, it was the length augmentation that improve performance. however, as finally, the competitive performance of the svm is an important takeaway from this paper. although transformers have yielded state of the art results in nlp for some years now, the have a number of limitations, such as interpretability and energy consumption. on the latter point, the svm classifier model trained in seconds, compared to 4 minutes for a base or adapted llm and 30+ minutes for an lsg model. researchers should consider an svm as their first choice of model due to its performance, ease of implementation and lower energy footprint. in order to further understand the performance of our best performing models, we examined the training curves of the svm and gabert models to check for overfitting. we trained the models on subsets of 10%, 30%, 50%, 70% and 100% of the data. after training on each subset, the model is evaluated on the same training data, in order to determine how well it performs on seen data. the validation scores measure the model's performance on unseen data. figure 1 indicates that both the svm and gabert are overfitting to the training data for atu prediction, as we can see early on that the training score is high. the disparity between this score and the cross-validation scores for both models also shows that this overfitting means that the models do not generalize as well as hoped to unseen data. furthermore, the transformer model performs almost as well on 10% of the data as it does on the full dataset. similarly, in the gender task, the svm overfits the training data, as does the transformer to a lesser extent. however, the transformer's validation score indicates that it generalises better to unseen data. the performance of the transformer seems to plateau with about 50% of the data, while the svm could possibly improve performance with some additional data. the use of regularization techniques, or a different kernel may lessen the extent of the overfitting. ## conclusion we present work examining the performance on four llms on a classification task, along with three augmentations to the models. the base models do not perform as well as the augmented ones, with mbert and roberta failing to beat a dummy baseline. the best performing model was trained on irish data only, was domain-adapted with continued finetuning and used lsg attention to increase the size of its context window. however, our linear baseline -an svm with bag of words and tf-idf features gave the second best performance. we conclude that the low-resource setting is challenging for llms, and while augmentations do help, classical machine learning models are still a competitive choice.