Learning Action Conditions from Instructional Manuals for Instruction Understanding (2023)

Abstract: The ability to infer pre- and postconditions of an action is vital for comprehending complex instructions, and is essential for applications such as autonomous instruction-guided agents and assistive AI that supports humans to perform physical tasks. In this work, we propose a task dubbed action condition inference, which extracts mentions of preconditions and postconditions of actions in instructional manuals. We propose a weakly supervised approach utilizing automatically constructed large-scale training instances from online instructions, and curate a densely human-annotated and validated dataset to study how well the current NLP models do on the proposed task. We design two types of models differ by whether contextualized and global information is leveraged, as well as various combinations of heuristics to construct the weak supervisions.Our experiments show a > 20% F1-score improvement with considering the entire instruction contexts and a > 6% F1-score benefit with the proposed heuristics. However, the best performing model is still well-behind human performance.

## introduction when performing complex tasks (e.g. making a gourmet dish), instructional manuals are often referred to as useful guidelines. to follow the instructed actions, it is crucial to understand the preconditions, i.e. prerequisites before taking a particular action, and the postconditions, i.e. the status supposed to be reached after performing the action. knowledge of action-condition dependencies is prevalent and inferable in many instructional texts. for example, in figure 1 , before performing the action "place onions" in step 3, both preconditions: "heat the pan" (in step 2) and "slice onions" (in step 1) have to be successfully accomplished. likewise, executing "stir onions" (in step 4), leads to its postcondition, "caramelized" (also in step 4). wait until the oil is sizzling. ## terminologies and problem definition our goal is to learn to infer action-condition dependencies in real-world instructional manuals. we first describe essential terminologies in details: actionable refers to a phrase that a person can follow and execute in the real world (yellow colored phrases in figure 2 ). we also consider negated actions (e.g. do not ...) or actions warned to avoid (e.g. if you purchase the wrong...) as they likely also carry useful knowledge regarding the tasks. 2 precondition concerns the prerequisites to be met for an actionable to be executable, which can be a status, a condition, and/or another prior actionable (blue colored phrases in figure 2 ). it is worth noting that humans can omit explicitly writing out certain condition statements because of their triviality as long as the actions inducing them are mentioned (e.g. heat the @xmath0, the latter can often be omitted). we thus generalize the conventional precondition formulation, i.e. sets of statements evaluated to true/false @xcite , to a phrase that is either a passive condition statement or an actionable that induces the prerequisite conditions, as inspired by @xcite . postcondition is defined as the outcome caused by the execution of an actionable, which often involves status changes of certain objects (or the actor itself) or certain effects emerged to the surroundings or world state (green colored phrases in figure 2 ). text segment in this paper refers to a textual segment of interest, which can be one of: {actionable, precondition, postcondition}, in an article. in reality, a valid actionable should have both preand postcondition dependencies, however, we do not enforce this in this work as conditions can occasionally be omitted by human authors. problem formulation. given an input instructional manual and some text segments of interest extracted from it, a model is asked to predict the directed relation between a pair of segments, where the relation should be one of the followings: null (no relation), precondition, or postcondition. ## datasets and human annotations as the condition-dependency knowledge we are interested in is prevalent in real-world instructions, we consider two popular online resources, wiki-how and instructables.com, both consist of detailed multi-step task instructions, to support our investigation. for wikihow, we use the provided dataset from @xcite ; for instructables, we scrape the contents directly from their website. since densely annotating large-scale instruction sources for the desired dependencies is extremely expensive and laborious, we mainly annotate a testset and propose to train the models via weakly or self-supervised methods. we hence provide a small subset of the human-annotated data to adapt models to the problem domain. to this end, we collect comprehensive human annotations on a selected subset in each dataset to serve as our annotatedset, and particularly the subsets used to evaluate the models as the annotated-test-set. 3 in total, our densely annotated-set has 500 samples in wikihow and 150 samples in instructables, spanning 7,191 distinct actions (defined by main predicate-object phrases) for diversity. in section 6.2, we will describe how the annotated-set is split to facilitate the low-resource training. we also collect the human performance on the annotated-test-set to gauge the human upper bound of our proposed task. more dataset details are in append. sec. a. ## training with weak supervision as mentioned in section 3, our proposed task can be approached via a zero-shot setting, where the vast amount of un-annotated instruction data can be transformed into useful training resources (same dataset structure as described in section 3.1). moreover, it is proven that in many low-resource nlp tasks, constructing a much larger heuristic-based weakly supervised data can be beneficial @xcite . ## models our proposed heuristics do not assume specific model architecture to be applicable, and to benchmark the proposed task, we mainly consider two types of base models: (1) non-contextualized model takes only the two text segments of interest at a time and make the pairwise trinary (directed) relation predictions, i.e. null, precondition, and postcondition; ## experiments and analysis our experiments seek to answer these questions: ## related works procedural text understanding. uncovering knowledge in texts that specifically features procedural structure has drawn many attentions, including aspects of tracking entity state changes @xcite @xcite , incorporating common sense or constraints @xcite , procedure-centric question answering (qa) @xcite , and structural parsing or generations @xcite @xcite . @xcite leverages verbnet @xcite with if-then constructed rules, one of the keywords we also utilize, to determine object-state postconditions for answering state-related reading comprehension questions. in addition, some prior works also specifically formulate precondition understanding as multiple choice qa for event triggers (verbs) @xcite and common sense phrases @xcite . we hope our work on inferring actioncondition dependencies, an essential knowledge especially for understanding task-procedures, from long instruction texts, can help advancing the goal of more comprehensive procedural text understanding. drawing dependencies among procedure steps has been explored in @xcite @xcite , however, their procedures are manually synthesized short paragraphs. our work, in contrast, aims at inferring diverse dependency knowledge directly from complex realworld and task-solving-oriented instructional manuals, enabling the condition dependencies to go beyond inter-step and narrative boundaries. ## conclusions in this work we propose a task on inferring action and (pre/post)condition dependencies on realworld online instructional manuals. we formulate the problem in both zero-shot and low-resource settings, where several heuristics are designed to construct an effective large-scale weakly supervised data. while the proposed heuristics and the twostaged training leads to significant performance improvements, the results still highlight significant gaps below human performance (> 20% f1-score). we hope our studies and the collected resources can spur relevant research, and suggest two main future directions: (1) end-to-end propose (refined) actionables, conditions, and their dependencies, by fully exploiting our featured span-annotations of the text segments. (2) inferred world states from the text descriptions as well as external knowledge of the entities and causal common sense can be factored into the heuristics for weak-supervisions.