C o RRPUS : Code-based Structured Prompting for Neurosymbolic Story Understanding (2023)

Abstract: Story generation and understanding—as with all NLG/NLU tasks—has seen a surge in neurosymbolic work. Researchers have recognized that, while large language models (LLMs) have tremendous utility, they can be augmented with symbolic means to be even better and to make up for many flaws that neural networks have. However, symbolic methods are extremely costly in terms of the amount of time and expertise needed to create them. In this work, we capitalize on state-of-the-art Code-LLMs, such as Codex, to bootstrap the use of symbolic methods for tracking the state of stories and aiding in story understanding. We show that our CoRRPUS system and abstracted prompting procedures can beat current state-of-the-art structured LLM techniques on pre-existing story understanding tasks (bAbI Task 2 and Re³) with minimal hand engineering. This work highlights the usefulness of code-based symbolic representations for enabling LLMs to better perform story reasoning tasks.

## introduction stories are a complex form of writing that involve many inter-dependent components, such as causal reasoning @xcite , temporal ordering @xcite , social commonsense @xcite , and the consistent portrayal of facts and events as the story unfolds @xcite . despite the recent uptick in research involving story understanding and reasoning @xcite @xcite @xcite @xcite @xcite @xcite , there is still a significant gap in the abilities of models to actually understand (or generate) stories at a reasonable level of commonsense. @xcite provide a recent overview of the area. meanwhile with natural language prompts, gpt-3 fails to keep track of the laptop through the character amy's movement. it can follow that amy has her laptop and that she brought it to the dorm, but it "loses track" of the laptop after amy goes to the cafeteria. although large language models (llms) like gpt @xcite by themselves can produce text that can be indistinguishable from human-written stories, these models still struggle to generate coherent long-form text @xcite & solve simple commonsense reasoning tasks (see figure 1 for an example.) and therefore, end up only writing at human-level quality less than three-quarters of the time @xcite . it is also likely that the majority of this generated text is actually memorized and generated verbatim from human-written stories @xcite . over the past few years, researchers have begun to see value in integrating symbolic ai methods-that are consistent and logical-with neural networks-that are flexible and unpredictable @xcite . these neurosymbolic techniques try to balance the best of both worlds. @xcite draws a parallel between neurosymbolic ai and the system 1/system 2 cognitive science theory @xcite : neural sequence models are a fast and intuitive system (system 1) that can be improved by adding a logical reasoning system (system 2). @xcite and @xcite use gpt-2/3, respectively, to produce candidate story continuations and then use a symbolic state representation to compare against in order to remove suggestions that would be inconsistent. we extend this work with the use of the code-llm codex @xcite for structuring and tracking state changes for story understanding. our system, which we call corrpus (code representations to reason & prompt over for understanding in stories) foot_0 , extracts structured information (the story world model) which is then used for reasoning. we compare to neural-only, symbolic-only, and prior works' neurosymbolic systems on two pre-existing story understanding tasks: babi and re 3 , which we will describe in section 4. our contributions are as follows: in the rest of the paper, we will outline a brief history of neurosymbolic work in storytelling and recent use of code-llms. we will then go over corrpus's three prompting methods, followed by a description of the two story understanding tasks babi and re 3 and our experiments with corrpus on both. ## the corrpus prompting system in this work, we examine the types of prompts for tracking symbolic story state representations using openai's code-llm, codex @xcite . all experiments that use the corrpus prompt system are conducted with code-davinci-002 using openai's api (which was free for research purposes). following @xcite , we use nucleus sampling with top-p value equals to 0.95. we set the temperature to 0 when no diverse generation is needed (i.e., answering multiple choice questions from the babi task, section 4.2) and set the temperature to 0.7 when diverse generation is needed (such as in the re 3 task, section 4.3). all the experimental results come from a single run. we provide the codex model a collection of classes in python to represent a model of characters and objects that we want to track in the story, in addition to an initialization of the specific characters and objects for the current story, which is presented in a world class. see figure 2a and appendix 8.1 for examples of these classes. in addition to this information for the current story, we also provide codex a 1-shot example of the full process we want it to complete. formally, we define the problem as: given a story @xmath0: s 1 : ... : s n ] and a story world state initialization w 0 , we want the model to update the story world state w i until the story is complete w n . after each sentence s i of the story, the model updates the world state using some black-boxed update function u i . thus, the final world state note that, with the except of w 0 and w n , these intermediate world states are also opaque and unseen from the user's perspective. this system, which we call corrpus, extracts structured information using code-based chain-ofthought-type prompting in order to more accurately track the underlying story state and detect any inconsistencies. we experiment with three different prompting techniques. the following examples show how the story sentence "sandra journeyed to the bedroom" would be modified for each prompt type. thus, it has to rely directly on the comments from the prompt and the story world state initialization. this shows us how well the code-llm can infer and fill in information with very little guidance. example: given the comment ## sandra journeyed to the bedroom., the model should generate self.sandra.@xmath1. the model then does not have to figure out what it means when a particular event happens but still has to map which function is appropriate for a given story sentence and how to fill in the arguments of the function. example: given go(character, location) and other functions, generate go(@xmath2, @xmath3. full examples of these prompts can be shown in figure 2 and appendix 8.1. ## experiments to show how corrpus can improve story understanding via maintaining a world model, we evaluate our system on two tasks: (1) task 2 of the babi set of tasks @xcite -which tests multi-step reasoning, and (2) re 3 @xcite , for detecting story inconsistencies in complicated real-world story examples. ## conclusion we present corrpus, a code-based prompting system that can extract structured information from complicated stories using 1-shot prompting. we conducted experiments on babi's task 2 to show that code-based prompting can leverage symbolic information to perform multi-step logical reasoning better than natural-language-based prompting, regardless of whether a code-llm or regular llm is used. we also evaluate corrpus on detecting story inconsistencies using the re 3 task @xcite , showing that code-llms can extract relevant information from story sentences better than llms. we emphasize that a careful prompting procedure that provides relevant low-level semantic abstractions can greatly improve the accuracy and generalizability of neurosymbolic reasoning but the type of prompt needed is entirely task-dependent.