Using Planning to Improve Semantic Parsing of Instructional Texts (2023)

Abstract: We develop a symbolic planning-based decoder to improve the few-shot semantic parsing of instructional texts. The system takes long-form instructional texts as input and produces sequences of actions in a formal language that enable execution of the instructions. This task poses unique challenges since input texts may contain long context dependencies and ambiguous and domain-specific language. Valid semantic parses also require sequences of steps that constitute an executable plan. We build on recent progress in semantic parsing by leveraging large language models to learn parsers from small amounts of training data. During decoding, our method employs planning methods and domain information to rank and correct candidate parses. To validate our method, we evaluate on four domains: two household instruction-following domains and two cooking recipe interpretation domains. We present results for few-shot semantic parsing using leave-one-out cross-validation. We show that utilizing planning domain information improves the quality of generated plans. Through ablations we also explore the effects of our decoder design choices.

## introduction recent advancements in natural language processing (nlp) have successfully combined large language models with external symbolic reasoning capabilities. toolformer @xcite enables the use of external tools to perform tasks such as arithmetic and factual lookup, which are currently challenging for large language models. @xcite created a system that performs multi-step reasoning in planning tasks specified in natural language. this is achieved by using an external symbolic planner for domain-specific reasoning, providing guarantees that plans are logical and satisfy environmental constraints. by integrating external symbolic reasoning capabilities, many of the shortcomings of large language models can be addressed while still capitalizing on their strengths. motivated by this line of work, we develop a novel decoding procedure that uses symbolic planning to improve the few-shot semantic parsing of longform instructional texts. we map instructional texts to formal action sequences and validate our method on two recipe semantic-parsing datasets @xcite . semantic parsing of instructional texts with few-shot learning poses several challenges to natural language processing (nlp) techniques. current nlp methods are dominated by large language models. these are based on transformer @xcite architectures and leverage large scale pretraining to enable high, few-shot and zero-shot task performance @xcite . building on work from @xcite , we investigate semantic parsing in the few-shot setting using openai's codex language and code llm @xcite . learning occurs in-context, by prompting the llm with a few input-output task examples. pretrained llm representations allow for more sample efficient learning than non-pretrained methods; but data scarcity still introduces performance limitations @xcite . data efficiency is advantageous when working with long-form instructional texts. the datasets we consider are small and the cost of annotating long texts with ground-truth semantic parses is high. in many semantic parsing tasks, context dependencies, input natural language strings, and output parses are relatively short. these tasks fit easily within the available context size of llm models and consist of at most several input and output statements. modeling long input-output dependencies poses a number of challenges for current models, as shown by their degraded performance on tasks designed to leverage long contexts @xcite . semantic parsing of long-form instructional texts, like the recipe datasets we evaluate, can require learning representations for hundreds of words, and outputting tens of steps, each with multiple arguments and complex syntax. instructional texts also exist within an implicit planning domain. these texts describe plans for achieving a goal by manipulating objects that comprise a world-state. executable semantic parseplans must consist of valid transitions within this world-state. for example, to bake an item the oven must be on and preheated. these requirements constitute preconditions for the bake action. longcontext dependencies pose challenges to generating executable sequences of actions that are also relevant to the task instructions. to form a valid parse-plan, instructions must be translated into a sequence of executable actions. all requisite actions must be represented in the plan, potentially including actions not explicitly mentioned. complicating matters, the common-sense knowledge needed to reason about valid plan sequences in a domain is only very implicitly represented within the llm and few-shot examples. to address these challenges, we propose planning augmented semantic parsing. our method leverages a formal symbolic planning representation to rank and correct candidate plans. plans are corrected by searching for sequences of actions that satisfy the preconditions of all output actions. ranking selects plans which best meet the domain's planning and syntactic constraints by ranking plans highly if they have fewer inadmissible actions, require fewer additional actions to correct, and have fewer steps with invalid syntax. after ranking, planning errors are fixed using symbolic planning methods. the result is an effective neuro-symbolic approach that combines the strengths of deep-learned llms and classical ai planning. we validate our approach using leave-one-out cross validation across each dataset and provide ablations for various aspects of our model choices. results show that using planning augmented semantic parsing results in more valid plan sequences that still maintain high relevance to the natural language task instructions. overall we make the following contributions: ## methods for the llm we use davinci codex @xcite based on the gpt-3 architecture @xcite . like gpt-3, the model was trained on a large web-sourced text corpus, but includes code in the dataset. while openai does not publish the sizes of the codex models available through their api, @xcite empirically estimate the size of text-only davinci at 175b parameters. ## experiments we use leave-one-out cross-validation to evaluate performance of various models on our two recipe datasets. for the shorter recipes in the @xcite ) dataset, we evaluate using both one and five training plans in the prompt. we evaluate the longer recipes of the @xcite dataset using only a single prompt example due to context length limitations. to evaluate the correctness of each output plan we compare the generated plan to the ground truth annotation from the dataset. we use metrics that measure the similarity between the output and ground truth plans. however for each recipe there are potentially many admissible plans and subjective judgements about the level of detail of the annotation and about which attributes to include. to address these potential ambiguities, we evaluate models using several diverse metrics to capture different aspects of plan accuracy. ## results we report results for the @xcite dataset in evaluations show that our method improves plan validity as measured by the mean number of precondition errors, syntax errors, and accuracy of steps returned (f1) in each plan. lcs remains fairly constant across our evaluations and ablations. the lcs metric reflects both the content of planning steps and their sequencing. by contrast, f1 only assesses the accuracy of steps in the generated plans. perhaps there exists a trade-off wherein the process of inserting corrective plan steps reduces the amount of alignment of the generated and ground truth plans (lowering lcs), but increases the accuracy of included steps (raising f1). of all the metrics considered, our method results in largest reduction in the number of precondition errors (pe). we achieve these improvements without singificant reductions in lcs and with an increase in f1. this is an important validation for our method, as @xcite finds that there exists a trade-off between the executability and semantic correctness (measure by lcs) of generated plans. it is straightforward to increase executability (fewer precondition errors) by ignoring the instructional text content and only outputting valid actions. for any downstream applications, plans must be executable and while also reflecting the content of the instructions. therefore is is important to reduce the number of precondition errors while maintaining content similarity to ground-truth plans. ## conclusion we develop an approach to semantic parsing for long-form instructional texts that leverages planning domain information to generate more valid plans in a low-data, few-shot setting. our method significantly reduces the number of precondition errors present in semantically parsed plans for two recipe datasets. these results highlight the benefit of a neuro-symbolic approach that utilizes the stateof-the-art code-generation llm codex to produce relevant steps for recipe execution and refines these plans using classical symbolic planning. in quantitative and qualitative evaluations, our approach generates plans that reflect the relevant steps of the natural language recipe. the symbolic planning component corrects precondition errors that arise from omitted or implied instructional steps and the challenges of learning with long contextdependencies from limited examples. @xcite rank rank + plan ground truth pour(flour) mix() mix() preheat( 350 ) bake( 20 ) pour(flour) mix() mix() preheat( 350 ) scrape() bake( 20 ) pour(flour) mix() scrape() preheat( 350 ) bake(20) create_ing("1 (10 ounce) package frozen mixed berries") create_ing("1 (15 ounce) can sliced peaches, drained") create_ing("2 tablespoons honey") create_tool("blender") combine("1 (10 ounce) package frozen mixed berries", "1 (15 ounce) can sliced peaches, drained", "2 tablespoons honey", "fruit and honey", "") put("fruit and honey", "blender") mix("fruit and honey", "blender", "smoothie", "blend") chefcheck("smoothie", "smooth") @xcite easy cake mix cookies 1 (18 1/4 ounce) box chocolate cake mix 1/3 cup vegetable oil 2 eggs combine cake mix, oil and eggs. mix well. bake at 350f for about 10 minutes. remove from oven and let cool on pan for several minutes before removing to rack to finish cooling. ingredient(["cake_mix"], "1 (18 1/4 ounce) box", @xmath1, "1/3 cup", @xmath2, "2", @xmath3, pour(oil), pour(eggs), mix() mix() scrape(), preheat(350), bake(10) noop()