Goal-Oriented Script Construction (2021)

Abstract: The knowledge of scripts, common chains of events in stereotypical scenarios, is a valuable asset for task-oriented natural language understanding systems. We propose the Goal-Oriented Script Construction task, where a model produces a sequence of steps to accomplish a given goal. We pilot our task on the first multilingual script learning dataset supporting 18 languages collected from wikiHow, a website containing half a million how-to articles. For baselines, we consider both a generation-based approach using a language model and a retrieval-based approach by first retrieving the relevant steps from a large candidate pool and then ordering them. We show that our task is practical, feasible but challenging for state-of-the-art Transformer models, and that our methods can be readily deployed for various other datasets and domains with decent zero-shot performance.

## introduction a script is a standardized sequence of events about stereotypical activities @xcite . for example, "go to a restaurant" typically involves "order food", "eat", "pay the bill", etc. such script knowledge has long been proposed as a way to enhance ai systems @xcite . specifically, task-oriented dialog agents may greatly benefit from the understanding of goaloriented scripts 2 . however, the evaluation of script knowledge remains an open question @xcite . moreover, it is unclear whether current models can generate complete scripts. such an ability is in high demand for recent efforts to reason about complex events @xcite 3 . we propose the task of goal-oriented script construction (gosc) to holistically evaluate a model's understanding of scripts. given a goal (or the name of a script), we ask the model to construct the sequence of steps (or events in a script) to achieve the goal. this task targets a model's ability to narrate an entire script, subsuming most existing evaluation tasks. our rationale is that a model that understands some scripts (e.g. how to "travel abroad" and "go to college") should be able to produce new ones (e.g. how to "study abroad") using the absorbed knowledge, close to how humans learn. while almost all prior script learning work has focused on english, we introduce a novel multilingual corpus. our corpus is collected from wikihow (wikihow.com), a website of how-to articles in 18 languages. the articles span a wide range of domains, from commonplace activities like going to a restaurant to more specific ones like protecting oneself from the coronavirus. we train and evaluate several baseline systems on our gosc task. first, we consider a generationbased approach where a pretrained language model, multilingual t5, is finetuned to produce scripts from scratch. as an alternative, observing that most desired steps can be drawn from the training scripts due to their magnitude and high coverage, we also propose a retrieval-based approach. concretely, we develop a step-inference-ordering pipeline using existing models to retrieve relevant steps and order them. we also improve the pipeline with techniques such as multitask learning. from the experiments, the gosc task proves challenging but feasible for state-of-the-art transformers. furthermore, we show that our pipeline trained on wikihow can generalize to other datasets and domains (see an example in figure 1 ). on three classic script corpora, omics, smile, and de-script, it achieves strong zero-shot performance. it can also be directly deployed to construct scripts in distant domains (e.g. military/political). in this paper, we make several contributions: 1) we propose the gosc task targeting the comprehensive understanding of scripts. ## related work the notion of scripts @xcite , or schemas @xcite , encodes the knowledge of standardized event sequences. we dissect previous work on script learning into two lines, narrative and procedural. one line of work focuses on narrative scripts, where declarative, or descriptive knowledge is distilled from narrative texts like news or stories @xcite . such scripts are not goal-oriented, but descriptions of sequential events (e.g. a traffic accident involves a collision, injuries, police intervention, etc.). @xcite introduced the classic narrative cloze test, where a model is asked to fill in the blank given a script with one missing event. following the task, a few papers made extensions on representa-tion @xcite or modeling @xcite , achieving better performance on narrative cloze. meanwhile, other work re-formalized narrative cloze as language modeling (lm) @xcite or multiplechoice (granroth-wilding and clark, 2016) tasks. however, the evolving evaluation datasets contain more spurious scripts, with many uninformative events such as "say" or "be", and the lms tend to capture such cues @xcite . the other line of work focuses on procedural scripts, where events happen in a scenario, usually in order to achieve a goal. for example, to "visit a doctor", one should "make an appointment", "go to the hospital", etc. to obtain data, event sequence descriptions (esd) are collected usually by crowdsourcing, and are cleaned to produce scripts. thus, most such datasets are small-scale, including omics @xcite , smile @xcite , the @xcite . the evaluation tasks are diverse, ranging from event clustering, event ordering @xcite ), text-script alignment @xcite and next event prediction @xcite . there are also efforts on domain extensions @xcite and modeling improvements @xcite . in both lines, it still remains an open problem what kind of automatic task most accurately evaluates a system's understanding of scripts. most prior work has designed tasks focusing on various fragmented pieces of such understanding. for example, narrative cloze assesses a model's knowledge for completing a close-to-finished script. the esd line of work, on the other hand, evaluates script learning systems with the aforementioned variety of tasks, each touching upon a specific piece of script knowledge nonetheless. recent work has also brought forth generation-based tasks, but mostly within an open-ended/specialized domain like story or recipe generation @xcite . regarding data source, wikihow has been used in multiple nlp efforts, including knowledge base construction @xcite , household activity prediction @xcite , summarization @xcite , event relation classification @xcite , and next passage completion @xcite . a few recent papers @xcite explored a set of separate goal-step inference tasks, mostly in binary-classification/multiple-choice formats, with few negative candidates. our task is more holistic and realistic, simulating an open-ended scenario with retrieval/generation settings. we combine two of our existing modules from @xcite into a baseline, but a successful gosc system can certainly include other functionalities (e.g. paraphrase detection). also similar is @xcite , which doesn't include an extrinsic evaluation on other datasets/domains though. in summary, our work has the following important differences with previous papers: 1) existing tasks mostly evaluate fragmented pieces of script knowledge, while gosc is higher-level, targeting the ability to invent new, complete scripts. 2) we are the first to study multilingual script learning. we evaluate several baselines and make improvements with techniques like multitask learning. 3) our dataset improves upon the previous ones in multiple ways, with higher quality than the mined narrative scripts, lower cost and larger scale than the crowdsourced esds. 4) the knowledge learned from our dataset allows models to construct scripts in other datasets/domains without training. ## goal oriented script construction we propose the goal-oriented script construction (gosc) task. given a goal g, a system constructs a complete script as an ordered list of steps s, with a ground-truth reference t . as a hint of the desired level of granularity, we also provide an expected number of steps (or length of the script), l, as input. depending on whether the set of possible candidate steps are given in advance, gosc can happen in two settings: generation or retrieval. in the generation setting, the model must generate the entire script from scratch. in the retrieval setting, a large set of candidate steps c is given. the model must predict a subset of steps s from c, and provide their ordering. ## multilingual wikihow corpus our previously wikihow corpus @xcite is a collection of how-to articles in english (en). we extend this corpus by crawling wikihow in 17 other languages, including spanish (es), portuguese (pt), chinese (zh), german (de), french (fr), russian (ru), italian (it), indonesian (id), dutch (nl), arabic (ar), vietnamese (vn), thai (th), japanese (jp), korean (ko), czech (cz), hindi (hi), and turkish (tr). the resulting multilingual wikihow corpus may be used in various tasks in nlp and other fields. for script learning, we extract from each wiki-how article the following critical components to form a goal-oriented script. goal: the title stripped of "how to"; section: the header of a "method" or a "part" which contains multiple steps; 4 steps: the headlines of step paragraphs; category: the top-level wikihow category. an example wikihow script is shown in figure 2 . our previous corpus provides labels of whether each english article is ordered, predicted by a highprecision classifier. we project these labels to other languages using the cross-language links in each wikihow article. for articles without a match to english, it defaults to unordered. in our task setup, we only require the model to order the steps if an article is ordered. for all experiments below, we randomly hold out 10% articles in each language as the test set, and use the remaining 90% for training and development. 5 we use the corpus to construct a dataset for multilingual gosc. for the retrieval setting, the set of candidate steps c are all the steps present in the test set. however, we observe that not only the large number of steps may render the evaluation intractable, but most steps are also evidently distant from the given goal. to conserve computing power, we restrict c as all the steps from articles within the same wikihow category for each script. ## models we develop two systems based on state-of-the-art transformers for the gosc task. foot_4 ## in-domain evaluation to demonstrate the performance of models on the gosc task, we evaluate them on our multilingual wikihow dataset using both automatic metrics and human judgments. the ultimate utility for this task is the extent to which a human can follow the constructed steps to accomplish the given goal. as direct user studies might be costly and hard to standardize, we carefully choose measures that adhere to this utility. by default, all models are trained and evaluated on the same language. ## zero-shot transfer learning to show the potential of our model for transfer learning, we use the retrieval-based step-inference-ordering pipeline finetuned on wikihow to construct scripts for other datasets and domains. we quantitatively evaluate our model on 4 other script learning corpora, and qualitatively analyze some constructed scripts in a case study. ## limitations event representation: our representation of goals and steps as natural language sentences, though containing richer information, brings the extra difficulty in handling steps with similar meanings. for example, "change strings frequently" and "put on new strings regularly" have nearly identical meanings and both are correct steps for the goal "maintain a guitar". hence, both could be included by a retrieval-based model, which is not desired. modeling: since gosc is a new task, there is no previously established sota to compare with. we build a strong baseline for each setting, but they are clearly not the necessary or sufficient means to do the task. for example, our step-inference-ordering pipeline would benefit from a paraphrasing module that eliminates semantic duplicates in retrieved steps. it also currently suffers from long run-time especially with a large pool of candidates, since it requires pairwise goal-step inference. an alternative is to filter out most irrelevant steps using similarity-based heuristics in advance. evaluation: under the retrieval-based setting, our automatic evaluation metrics do not give credit to inexact matches as discussed above, which can also be addressed by a paraphrasing module. meanwhile, for the generation-based setting, bertscore, or other comparison-based metrics like bleu @xcite and meteor @xcite , may not be the most suitable metric to evaluate scripts. they are best designed for aligned texts like translation pairs, and do not measure whether the ordering among steps is correct. while we complement it with manual evaluation, only one human annotator is recruited for each language, resulting in potential subjectivity. alternatively, crowdsourcing-based evaluation is costly and hard to standardize. due to the complexity of the gosc task and its evaluation, we suggest that future work investigate better means of evaluation. ## conclusion and future work we propose the first multilingual script learning dataset and the first task to evaluate the holistic understanding of scripts. by comprehensively evaluating model performances automatically and manually, we show that state-of-the-art models can produce complete scripts both in-and out-of-domain, with a large room for improvement. future work should investigate additional aspects of scripts, such as usefulness, granularity, etc., as well as their utility for downstream tasks that require automated reasoning. first, we calculate the accuracy, namely the percentage of predicted steps that exist in the groundtruth. denote s i as the i-th step in s. if the gold script is ordered, we further evaluate the ordering of the constructed script by calculating kendall's τ between the intersection of the predicted steps and the ground-truth steps. it is likely that a model includes two modules: a retrieval module and an ordering module. in this case, it is sensible to separately evaluate these two modules. to evaluate the retrieval module independently, assume that the model retrieves a large set of steps r ranked by their relevance to the goal g. denote r i as the i-th step in r. we calculate recall and normalized discounted cumulative gain 14 at position k. assume @xmath0. 14 we set the true relevance of each predicted step as 1 if it exists in the ground-truth steps, and 0 otherwise. to evaluate the ordering module independently, we directly give the model the set of ground-truth steps to predict an ordering. we again use kendall's τ to evaluate the ordered steps. in the generation setting, a model is evaluated using perplexity on the test set, following standard practice.