Unveiling Narrative Reasoning Limits of Large Language Models with Trope in Movie Synopses (2024)

Abstract: Large language models (LLMs) equipped with chain-of-thoughts (CoT) prompting have shown significant multi-step reasoning capabilities in factual content like mathematics, commonsense, and logic. However, their performance in narrative reasoning, which demands greater abstraction capabilities, remains unexplored. This study utilizes tropes in movie synopses to assess the abstract reasoning abilities of state-of-the-art LLMs and uncovers their low performance. We introduce a trope-wise querying approach to address these challenges and boost the F1 score by 11.8 points. Moreover, while prior studies suggest that CoT enhances multi-step reasoning, this study shows CoT can cause hallucinations in narrative content, reducing GPT-4’s performance. We also introduce an Adversarial Injection method to embed trope-related text tokens into movie synopses without explicit tropes, revealing CoT’s heightened sensitivity to such injections. Our comprehensive analysis provides insights for future research directions.

## introduction large language models (llms) @xcite @xcite have shown their significant reasoning skills in few-shot manners with billion-scale parameters. chain-ofthoughts (cot) @xcite extends llms' reasoning capabilities by introducing stepwise reasoning. this approach resembles "slow thinking" in cognitive science @xcite , enabling llms to break down complex tasks into a sequence of simpler tasks, facilitating progressive processing and integration of results. it enhances performance across various tasks, including arithmetic calculations @xcite and domain-specific reasoning @xcite . unlike existing llm analyses, which primarily focus on factual reasoning and have demonstrated significant advances in llms and cot processes for factual reasoning, this work presents the first investigation of llms using tropes in movie synopses. it reveals that llms and cot lack narrative reasoning reasoning skills, as shown in figure 1. unlike factual reasoning, which is based on logical deductions and objective data @xcite , narrative reasoning presents distinct challenges by requiring a deep understanding of event sequences and extensive world knowledge @xcite . since narratives encapsulate human behavior, beliefs, and motivations beyond displayed contexts @xcite , the trope in movie synopses task introduces several novel challenges and perspectives beyond previous work, as shown in figure 2 . first, trope understanding requires llms to understand concepts that are not physically present or directly observable. for example, "heroic sacrifice" involves a character choosing to give up their own life or well-being for the greater good or to protect others. the concept goes beyond the physical act of sacrifice to encompass the thematic implications of altruism, selflessness, and the value of individual life versus collective well-being. this trope is often portrayed as the death of a character, but the death of the character does not entail the trope (it could even be the exact opposite, as shown in figure 2 ). existing benchmarks such as math reasoning @xcite or natural language inference (nli) @xcite provide concrete and observable context. for math reasoning, concepts and operations are transparent. while nli involves abstract thinking to some degree to infer relationships between sentences and understand implied meanings, the scope is confined to linguistic and logical reasoning within specific textual contexts. nli tasks do not demand the same level of thematic interpretation, symbolic analysis, or connection of ideas across disparate narrative elements that trope understanding does. second, comprehending tropes requires llms to make connections between seemingly unrelated ideas. "heroic sacrifice" requires viewers or readers to connect the act of sacrifice with broader themes or messages of the narrative, such as freedom, love, or redemption. these themes and messages not only are hard to be observed but also appear to be completely unrelated. for example, a character might subtly express their love for freedom in a casual conversation, which later informs their ultimate sacrifice. however, randomly grasping several "seemingly unrelated ideas" without carefully reasoning between the ideas could result in hallucination. for example, "a character a loves b" and "the character a dies in an accident in front of b" could be unrelated to heroic sacrifice despite seeming to have the elements of death and love. meanwhile, existing benchmarks do not require such a capability. for example, while logic reasoning @xcite and commonsense reasoning @xcite ) assess ai's logical reasoning and commonsense knowledge, they do not require the nuanced thematic interpretation and narrative analysis needed to understand tropes like heroic sacrifice. "heroic sacrifice" involves integrating complex narrative themes and character motivations, a level of abstract thinking and interpretation beyond the structured challenges of logical reasoning and the common knowledge queries of commonsense reasoning. exploring narrative reasoning capabilities is beneficial for both llm research and application development. it challenges modern llms by requiring abstraction from narrative contexts, a realm beyond previous research. this exploration sheds light on llm behaviors under specific circumstances and encourages the development of more reliable applications to mitigate hallucinations. in light of this, we first investigate llms' reasoning capability by revisiting an existing trope in movie synopses (timos) dataset @xcite and discover that it remains a very challenging task even for current llms with a well-engineered prompting pipeline. as figure 1 shows, advanced llms such as gpt-4, chatgpt, and fine-tuned llama-2 perform poorly, only achieving f1 scores at the level of random guessing even when they are equipped with cot @xcite prompts. this suggests that state-of-the-art llms, despite dominating various benchmarks, do not carry reasoning skills for challenges in trope understanding tasks. furthermore, we address the challenge by reframing the timos task as trope-wise querying, where each llm query inputs a single trope, significantly enhancing llm performance. as a result, performance improves by 11.8 points on the f1 score, surpassing the supervised state-of-the-art results proposed by timos @xcite . this strategy opens a new pathway for tackling complex reasoning tasks by decomposing multiple concepts into a single concept within an llm query. in addition to assessing trope reasoning capabilities, we reveal cot's tendency for hallucination and show that it does not always enhance reasoning compared to vanilla prompting. furthermore, cot can increase llms' susceptibility to adversarial inputs. specifically, inspired by prior reading comprehension studies @xcite , we devise an adversarial injection that inserts trope-related text tokens without explicit trope introduction into a movie synopsis, aimed at gauging whether llms are misled. the propensity for hallucination tendency is underscored by: (1) a stark decline in llm precision when employing cot, (2) adversarial injection significantly misleads llms through keyword and pattern recognition, especially when cot is equipped, and cot generates accurate responses with erroneous rationales. we also provide a comprehensive analysis, highlight the challenges associated with timos, and offer insights for future llm research and applications. the contributions of this paper are as follows: ## conclusion we examined llms' reasoning using movie synopsis tropes. despite previous research highlighting llms' strengths in complex tasks, including gpt-4, our study reveals challenges in narrative reason-gpt-4 vs. chatgpt chatgpt + cot vs. chatgpt gpt-4 vs. supervised ing, with llms achieving only random guessing performance on the timos dataset. we addressed these challenges by utilizing trope-wise querying, which significantly improved performance. additionally, we found that cot diminishes gpt-4's performance and proposed adversarial injection to assess llms' hallucination tendencies, discovering that cot exacerbates this issue. these findings underscore the gap in llms' capabilities and raise concerns about their safety and trustworthiness. we are optimistic that our findings will pave new paths for future llm research and applications.