Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writers (2024)

Abstract: We evaluate recent Large Language Models (LLMs) on the challenging task of summarizing short stories, which can be lengthy, and include nuanced subtext or scrambled timelines. Importantly, we work directly with authors to ensure that the stories have not been shared online (and therefore are unseen by the models), and to obtain informed evaluations of summary quality using judgments from the authors themselves. Through quantitative and qualitative analysis grounded in narrative theory, we compare GPT-4, Claude-2.1, and LLama-2-70B. We find that all three models make faithfulness mistakes in over 50% of summaries and struggle with specificity and interpretation of difficult subtext. We additionally demonstrate that LLM ratings and other automatic metrics for summary quality do not correlate well with the quality ratings from the writers.

## introduction narrative is a fundamental part of how we communicate and make sense of our experiences. as @xcite describes, ''narrative roots itself in the lived, felt experience of human or human-like agents interacting in an ongoing way with their surrounding environment...'' understanding narrative is, therefore, a necessary skill large language models (llms) need in order to engage with the subtleties of human experience and communication. we test how well llms understand the subtleties of narrative through the task of narrative summarization, with a focus on interpreting themes and meaning in addition to capturing plot. for our narrative form, we use fiction short stories as they present several interesting challenges. fiction writing does not follow a clear intro-bodyconclusion format and instead may follow a non-linear timeline, hint at things only abstractly, or deliberately mislead the reader. fiction may use multiple language varieties and creative language. for example, consider this quote from toni morrison's beloved: ''the pieces i am, she gather them and give them back to me in all the right order'' @xcite . this sentence uses complex metaphor and african american language @xcite from the late 1800s to express a beautiful relationship between two characters. finally, fiction short stories can be longer than an llm's context window. here, we consider stories up to 10,000 tokens long, which fit within the context window of the paid llms we use but are too long for the context window of the open-source llm we use. evaluations of narrative summarization on long documents have been scarce due to several key challenges: 1) narrative text is generally either in the public domain (and therefore likely in llm training data) or under copyright, and 2) holistic summary evaluation has been prohibitively difficult due to a lack of reliable automatic metrics @xcite and complications with human evaluation. for example, it can take someone over an hour to thoroughly read and evaluate just one story and summary, which quickly becomes expensive. @xcite make progress on these issues by purchasing recent books (which are less likely to be in models' training data but may still be discussed in the training data), and developing an llm-based metric for evaluating summary coherence. we take this a step further by working directly with experienced creative writers, and thus are able to: 1) use stories that are not discussed or present in training data, 2) expand beyond evaluation of coherence to aspects of narrative understanding like faithfulness and thematic analysis, and 3) use human rather than model judgments. we evaluate three llms -gpt-4, claude-2.1, and llama-2-70b -on 25 short stories. we ask authors to evaluate the summaries of their own unpublished stories foot_0 since they are experts in what they have written, focusing on coherence, faithfulness, coverage of important details, and useful interpretation in the summaries. we then present an analysis of their evaluation. the key features of our work are: ## writers and data to avoid using stories that may have content or analysis available online, we find skilled writers with unpublished stories they are willing to share. we recruit writers through mfa listservs, posts on x, and direct emails. once writers express interest, we screen for skill by writer education we also explore how similar the stories may be to the llm training data by measuring their perrplexity. we cannot compute perplexity using the gpt-4 and claude apis, so we get an approximate number by using llama-2-7b (float16), which we can run on our own compute. this model assigns perplexity scores between 9.2 and 21.6 to the stories. for reference, the first chapter of pride and prejudice (a story in llama-2's training data) has a perplexity score of 1.3, which is just under the model's training loss. these scores therefore validate that the stories are unfamiliar to the models. ## summary generation we evaluate automatic summarization using three recent llms: gpt-4 (november 2023 update), claude-2.1, and llama-2-70b-chat (instructiontuned for chat). gpt-4 (openai, 2023) and claude-2.1 foot_3 have very long input contexts and can ingest an entire short story. llama-2 @xcite serves as a comparison point for smaller open-source models. @xcite also used these three llms in their work on automatic coherence evaluation in long narrative summarization. protecting the writers' unpublished work is paramount when using each of these three models. we do not want their stories to be saved or trained on by openai, anthropic, or hugging-face. we inform the writers of this risk in the consent form (see appendix a) and complete the available forms/settings with each of these companies to request our data not be stored for long periods or used for training. for each story, we then generate three summaries -one from each of the three models. for gpt-4 and claude, we use one prompt (see figure 1 ) as they both have a context length long enough to process a whole short story. claude refuses to summarize two of the stories which do not meet its content restrictions. 5 one of these stories is about a shooting and the other involves sex and robbery. given llama's short context window, we use hierarchical merging of chunk-level summaries @xcite for stories in the medium and long length buckets. depending on length, we chunk each story into 1-4 chunks using section or paragraph breaks. llama summarizes each of these chunks separately and then summarizes the concatenation of these summaries (see figure 1 ). for all three models, we access them from december 2023 through january 2024, and we use simplified versions of @xcite prompts (see figure 1 ). for gpt-4 foot_5 and claude, foot_6 we use max @xmath0.3. for llama, 8 we use the default settings in huggingchat. 9 in in addition to these commonly used metrics, we further investigate how much the models copy directly from the stories in