Reasoning about Goals, Steps, and Temporal Ordering with W iki H ow (2020)

Abstract: We propose a suite of reasoning tasks on two types of relations between procedural events: goal-step relations (“learn poses” is a step in the larger goal of “doing yoga”) and step-step temporal relations (“buy a yoga mat” typically precedes “learn poses”). We introduce a dataset targeting these two relations based on wikiHow, a website of instructional how-to articles. Our human-validated test set serves as a reliable benchmark for common-sense inference, with a gap of about 10% to 20% between the performance of state-of-the-art transformer models and human performance. Our automatically-generated training set allows models to effectively transfer to out-of-domain tasks requiring knowledge of procedural events, with greatly improved performances on SWAG, Snips, and Story Cloze Test in zero- and few-shot settings.

## introduction if you ask alexa or siri where to "buy a yoga mat," it should ideally infer that your goal is probably to "do yoga" and therefore suggest information on subsequent steps like "learn some poses." this requires the system to reason about the goal-step relation and the step-step temporal relation among events in a procedure. though event relation reasoning is a popular task, existing datasets mostly focus on temporal relations @xcite , causal relations @xcite , spatiotemporal containment and coreference relations @xcite . less attention has been paid to relations among procedural events, the understanding of which is critical to task-oriented intelligent systems. the knowledge of procedural events is also crucial to learning * equal contribution. scripts @xcite , which describe sequences of stereotypical human activities. to bridge this gap, we propose a dataset for goalstep inference targeting these two event relations. we collect data from wikihow foot_0 , a website consisting of more than 110,000 professionally-edited how-to articles spanning a surprisingly wide range of domains. each wikihow article describes a commonplace human activity, organized as a goal and a sequence of steps (figure 1 ). our dataset includes 3 tasks: inferring steps given a goal, the goal given a step, and the ordering between two steps given a goal. for each task, we automatically generate 100,000 to 800,000 examples as the training set, using a negative sampling strategy based on semantic similarity; we also provide a human-validated test set with 1,000 to 3,000 examples. 2 our test set serves as a reliable benchmark for commonsense inference, with a performance gap of 10% to 20% between human and state-of-the-art transformer models trained in-domain. moreover, when pre-trained on our tasks, a model can transfer knowledge of procedural event relations to other nlu tasks, with a zero-shot improvement over the baselines by 24% for a commonsense reasoning benchmark @xcite , 13% for a story cloze test @xcite and 64% for an intent detection task @xcite . we construct a new corpus by crawling the latest wikihow website. our corpus has 112,505 how-to articles after deduplication in an easy-to-process json format (statistics by category are shown in appendix b). each article contains: main bodies of texts (titles, methods/parts, headers, descriptions), related articles, references, q&a, tips and warnings. to facilitate multi-modal research, we also include links to images and videos aligned with texts. ## goal-step inference tasks we propose 3 goal-step inference tasks derived from the corpus. in each article, we define goal as the title without "how to", and step as the header of each paragraph (example shown in figure 1 ). ## in-domain evaluation we finetune pretrained bert @xcite , xlnet @xcite , roberta @xcite and gpt-2 (radford et al., 2019) models on the training set and report accuracy on the test set. modeling details including hyperparameter settings are shown in appendix a. to benchmark human performance, two authors each annotate 100 random examples from the test set and report the average accuracy. the results are shown in ## out-of-domain transfer learning to show that our tasks can serve as an effective transfer learning resource especially in zero-or few-shot settings, we consider 3 tasks in different domains, using a subset of their training data to simulate a low-resource scenario. therefore, we are not comparing to the state-of-the-art performances involving the entire in-domain training sets. for each target task, we finetune a vanilla roberta model and one pretrained on our task on increasingly larger portions of the target training set, and observe accuracy on the validation set, as the test set labels are not publicly available. swag @xcite is a commonsense inference dataset in the video caption domain. given a context, a system chooses one event most likely to happen from four candidates. for transfer learning, we use up to 1,000 examples for training and the standard validation set. we use the model trained on our step inference task to transfer to this task. snips @xcite is an intent detection dataset in the dialogue system and spoken query domain, where a system classifies an utterance into one of 7 intents. for transfer learning, we use up to 1,000 examples for training and the standard validation set. we use the model trained on our goal inference task to transfer to this task. to enable zero-shot transfer, we convert each example in our training data to a 7-choose-1 format by adding 3 empty strings as additional negative candidates. story cloze test @xcite is a story understanding dataset in the fiction domain, where a system chooses an ending to a 4-sentence-5 kfc and light displays are japanese christmas traditions @xcite . story from 2 candidates. we use up to 314 examples for training and 1,571 examples for validation, from the 2016 and 2018 data releases after removing duplicates. we use the model trained on our step ordering task to transfer to this task. to mimic the "next sentence prediction" format, we convert each example in our task to a "next step prediction" question with 4 prompt steps and 2 candidate steps, exactly one of which happens after the prompt. figure 2 shows the learning curves of the downstream tasks with an increasing number of their training samples, demonstrating a clear advantage of using our training data in low-resource settings. for swag, the model trained on our data has a zero-shot performance 24% over chance, outperforming the vanilla model when up to 1,000 training examples are given. for snips, the model trained on our data boasts an impressive 78% zeroshot performance, approaching perfect accuracy rapidly after some in-domain training. for the story cloze test which has the largest domainmismatch with our data, the model still benefits from the knowledge learned from it consistently, given any portion of in-domain training data up to the full size in our experiment. these results show that the model learns real-world procedural knowledge from our wikihow-based tasks, which can be readily applied to various domains and writing styles. ## summary we propose 3 goal-step inference tasks using wiki-how to complement research of event relation reasoning. our test sets serve as a reliable benchmark for commonsense inference, and more importantly, our dataset is an effective transfer learning resource, improving transformer models' performance on various tasks in zero-or few-shot settings. this implies a strong potential for pre-training models to better generalize in low-data scenarios.