Contemporary LLM s and Literary Abridgement: An Analytical Inquiry (2024)

Abstract: Within the framework of this study, several contemporary Large Language Models (ChatGPT, Gemini Pro, Mistral-Instruct and BgGPT) are evaluated in relation to their ability to generate abridged versions of literary texts. The analysis is based on ’The Ugly Duckling’ by H. C. Andersen as translated into English, French and Bulgarian. The different scenarios of abridgement experimented with include zero-shot, one-shot, division into chunks and crosslingual (including chain-of-thought) abridgement. The resulting texts are evaluated both automatically and via human evaluation. The automatic analysis includes ROUGE and BERTScore as well as the ratios of a selection of readability-related textual features (e.g. number of words, type-to-token ratio) as pertaining to the original versus automatically abridged texts. Professionally composed abridged versions are regarded as gold standard. Following the automatic analysis, six selected best candidate texts per language are then evaluated by volunteers with university education in terms of textual characteristics of a more qualitative nature, such as coherence, consistency and aesthetic appeal.

## introduction the current work seeks to provide an overview of the ability of contemporary large language models (llms) to generate abridged versions of literary works. as per the merriam-webster dictionary, 'abridged' means 'shortened or condensed, especially by the omission of words or passages'. notably, abridgement makes literary texts accessible to audiences that would find it hard to read or work with the corresponding original texts, such as young children, foreign language learners or people with learning disabilities. the term will not be used as synonymous to 'summary' due to both its particular relevance to the literary domain and its focus on overall simplification rather than merely reduction in size. ## background although literary abridgement by llms is not yet an established research topic, it implies several sets of abilities pertaining to the technology that are currently of marked academic interest, notably the use of long context, summarisation, and creative/literary writing. ## discussion although bulgarian texts received comparatively lower scores in the conducted human evaluation, they were shown to be mostly competitive to counterparts in more highly resourced languages. interestingly, they also tended to demonstrate different shortcomings compared with texts in english and french, such as excessive textual length. crosslingually derived texts were rated very highly by participants, notably occupying first place in the cases of french and english. texts derived through the 'chunks' scenario were judged to have problems in relation to information and transitions, which leads us to hypothesise that an application of chang et al. (2024)'s method of hierarchical merging would be of significant benefit. the chatgpt and gemini models performed better than the smaller but instruction-tuned mistral and bggpt; however, the gap was not striking, mistral-generated texts notably performing high in the conducted automatic evaluation. the fact that agreement between participants in the survey is low speaks of high subjectivity, which in turn implies that the texts were mostly lacking obvious, objective drawbacks. ## conclusion and future directions four contemporary llms of different sizes and statuses of use were evaluated for their ability to provide abridged versions of a literary text. three discrete languages were regarded: english, a relatively high-resourced language (french) and a relatively low-resourced language (bulgarian). whilst english-language texts expectedly demonstrate superior quality, models such as chatgpt and gemini also perform competitively in other languages, whilst demonstrating different weaknesses in relation to different generation scenarios. particularly, in a relatively low-resource language like bulgarian, high quality text can be achieved if the models' limitations in terms of context length are overcome. this study is an analytical inquiry into the current abilities of llms to generate abridged versions of literary texts on the basis of their original training data. these abilities are likely to be improved following additional training on relevant datasets as well as more elaborate prompting techniques. a natural continuation of the presented study would be the exploration of abridgement by llms in relation to a variety of texts; this time proceedings of clib 2024 with a focus on the models and abridgement scenarios that proved strongest.