Hierarchy-aware Learning of Sequential Tool Usage via Semi-automatically Constructed Taxonomies (2020)

Abstract: When repairing a device, humans employ a series of tools that corresponds to the arrangement of the device components. Such sequences of tool usage can be learned from repair manuals, so that at each step, having observed the previously applied tools, a sequential model can predict the next required tool. In this paper, we improve the tool prediction performance of such methods by additionally taking the hierarchical relationships among the tools into account. To this aim, we build a taxonomy of tools with hyponymy and hypernymy relations from the data by decomposing all multi-word expressions of tool names. We then develop a sequential model that performs a binary prediction for each node in the taxonomy. The evaluation of the method on a dataset of repair manuals shows that encoding the tools with the constructed taxonomy and using a top-down beam search for decoding increases the prediction accuracy and yields an interpretable taxonomy as a potentially valuable byproduct.

## introduction humans perform various tasks that have an inherent sequential nature comprising several steps; repairing a device is one of them. an ai agent serving as a cooperative assistant in such a task should be provided with contextual knowledge about the pertinent sequence of steps. the importance of such knowledge in cooperative situations has been shown in @xcite . an example of using sequential context knowledge in a cooperative situation is found in @xcite . here, it can be seen that learning the dependencies among the ingredients in cooking recipes helps with resolving the user's request for ingredients, since the system can anticipate what may be needed next. there have been numerous efforts to acquire task knowledge from available sources of instructional data, for instance from the web. related work on extracting the workflow from instructional text, such as @xcite , have not built a sequential model for generalizing the obtained knowledge to unseen tasks. working on data collected from the wikihow website as the resource, @xcite and @xcite developed models that learn the temporal order of steps but only take one previous step into account and ignore the higher-order dependencies. nabizadeh et al. (2020a) compared sequence learning methods for modeling long-term dependencies among the used tools in various steps of repair tasks, showing the advantage of recurrent neural network (rnn) models for this purpose. their results revealed that the similarity among the sequence of used tools in different repair manuals makes it possible to predict the next required tool on unseen repair tasks. in their approach, each tool is represented as a distinct class, while the relationships among the related types of the tools are not considered. as a result, the input does not provide the model with any information about different types associated with a class of tool. for instance, the model has no clue that different types of screwdrivers, such as phillips and torx, are all screwdrivers, and that different sizes of phillips screwdrivers belong to the same category of phillips screwdrivers. such information is also missing in calculating the cross-entropy loss of the rnn. however, we posit that for instance the penalty for predicting a 4mm nut driver instead of a 5mm nut driver should be less than the penalty of predicting a hammer, instead. this paper, therefore, extends their work by taking such missing information into account. specifically, we develop a sequential model for predicting the tool usage in unseen repair tasks, where we encode the tools using a semi-automatically constructed taxonomy. previous work has shown the advantage of hierarchy-aware neural networks for different tasks, such as audio event detection @xcite and entity type classification @xcite , which both benefit from predefined hierarchies. the tool names in repair tasks are often compound nouns, containing information about the main class of the tool and its detailed attributes. we use the arrangement of words in the tool names for building a tool taxonomy, with different types and sizes of a parent node tool arrayed as its child nodes. applying a binary classifier for predicting each node in the taxonomy, i.e., predicting the main class of the tool and its details separately, we show the advantages of hierarchy-aware prediction model over the flat one. ## method this section introduces the proposed approach for producing the tool taxonomy from data and modeling the dependencies among the used tools in the steps of the repair tasks. ## conclusion in this paper, we utilize the head-modifier principle to decompose the multi-word expressions of tool names and build a taxonomy for the used tools in a dataset of repair manuals. we noted that utilizing the constructed taxonomy in sequential modeling of the used tools improves the tool prediction performance, especially when the data is annotated automatically and includes annotation errors. we imagine that hierarchy-aware modeling also helps when we have an imperfect observation of the used tools, e.g., when the model is uncertain about the size of used screwdrivers in the past. in the future, we plan to study the effect of such observation uncertainty on the prediction performance.