Incorporating Task-Specific Concept Knowledge into Script Learning (2023)

Abstract: In this paper, we present Tetris, a new task of Goal-Oriented Script Completion. Unlike previous work, it considers a more realistic and general setting, where the input includes not only the goal but also additional user context, including preferences and history. To address this problem, we propose a novel approach, which uses two techniques to improve performance: (1) concept prompting, and (2) script-oriented contrastive learning that addresses step repetition and hallucination problems. On our WikiHow-based dataset, we find that both methods improve performance.

## introduction a goal-oriented script refers to a sequence of events that describe some stereotypical activities for achieving a specified goal @xcite . it is important to study how to automatically construct instructional scripts because it enables many high-impact applications such as robot action planning @xcite , causal reasoning @xcite , and task-oriented dialogue generation @xcite . while previous work has shown that neural models are capable of constructing the entire script given a goal @xcite , their proposed tasks are highly restrictive and based on overly simplified assumptions about the application because it ignored both the usage context (e.g., what is a preferred way of solving the problem, and what steps have been executed already) and the variable personal preferences that a user may have (e.g., a goal might need be achieved in different ways). take "make eggless cupcakes" as an example, the user might want to make it with different equipment (e.g., by using a rice cooker instead of an oven) or styles (e.g., to have some banana flavor), or the https://github.com/chenkaisun/tetris goal: wash linen shirts added detergent i lled a sink with cool water wash the shirt by swishing it around in the sudsy water. soak the shirt in the tub for 10-20 minutes. rinse the shirt with water until all the suds are gone. ## task formulation we define a goal-oriented script to include three types of elements, goal, preference, and step. a goal represents the task desired to be completed. ## task concept dictionary as in virtually all nlp applications, a main technical challenge in solving the problem of script hang up your shirt and let it air-dry for the best results. wash the shirt by swishing it around in the sudsy water. as training data completion is how to acquire the relevant knowledge and leverage the knowledge to generate the remaining steps in a script. in our case, a critical question is: what kind of knowledge representation is likely helpful for solving the script completion problem? in our work, we make a hypothesis that creating task-concept association would help the model generalize better on script completion and propose task concept dictionary (tcd). in tcd, each task key g from the collection of the keys d g (e.g., "how to modify the navigation system of an acura") is mapped to a set of relevant concepts d g c during its solution process. each @xmath0: as preparation material needed for achieving the task (like ingredients for recipe), as attributes/aspects that need to be considered, as intermediate entities generated during the process, and others. in this work, we made a primal construction of tcd (tcd.v0) as a proof of concept, that is, the association types are not considered between g and d g c . the design of tcd is motivated by the observation that humans can often form solutions through brainstorming and mind-mapping with knowledge about the tasks and relevant concepts involved in the solution process. analogously, tcd is meant to encode such task-specific knowledge about associations between goals and concepts so as to assist downstream script learning applications. as will be shown in our experiment results, tcd is indeed beneficial for improving performance for script completion by supplying useful relevant concepts to a baseline model. in general, tcd can be constructed by using any instructional content on the web. in our experiments, to construct tcd.v0, we first collect all articles from wikihow. we chose to use wikihow as the seed corpus for its richness in user preference data. each article consists of a goal, a list of preferences, and solution steps under each preference. let g p j i be the goal from an article i conditioned on the j-th preference, we then proceed to augment the set d g with a key g p j i , and augment d g p j i c with noun phrases (by using spacy foot_0 tagger for extraction) in the solution steps under preference j. we create an association between goals and concepts if they co-occur in the same article. we show the statistics of tcd in appendix a.1. ## method in this section, we discuss our design of the method for tetris. the framework contains two parts, concept acquisition from tcd and solution construction. the first part uses tcd as an aid to acquire concepts relevant to the current input. specifically, we introduce two different methods (figure 2 ), one based on retrieval of closely related tasks and aggregation on the associated concepts, while the other involves training a concept generator on tcd. as shown in section 5, both of them have a positive impact on performance, yet they also introduce different benefits. the concepts fetched from tcd using either method form a concept prompt, which is combined with the input i (consisting of goal, preference, and history) as an augmented input i * . i * is then fed into an encoderdecoder language model m encdec to complete the script. below we describe the two methods for the generation of relevant concepts using tcd. ## conclusion in this work, we propose tetris, the new task of goal-oriented script completion, which allows a model to produce the remaining steps given a userspecified goal, preference, and history of steps. we also present task concept dictionary (tcd), a knowledge base representing task and concept association, to enable knowledge-based methods for the task. we introduce different methods to acquire concepts as prompts for the downstream text generator. we also introduce a contrastive learning strategy for script learning. the methods present consistently better performance on both automatic and human evaluation, clearly demonstrating the benefit of tcd for improving the performance of script completion. the qualitative analysis further shows that the task-specific knowledge can indeed benefit goal-oriented script learning tasks by feeding relevant knowledge about task completion. future work could explore how to improve the acquisition quality from tcd and applications of tcd on task-oriented dialog systems.