Visual Goal-Step Inference using wiki H ow (2021)

Abstract: Understanding what sequence of steps are needed to complete a goal can help artificial intelligence systems reason about human activities. Past work in NLP has examined the task of goal-step inference for text. We introduce the visual analogue. We propose the Visual Goal-Step Inference (VGSI) task, where a model is given a textual goal and must choose which of four images represents a plausible step towards that goal. With a new dataset harvested from wikiHow consisting of 772,277 images representing human actions, we show that our task is challenging for state-of-the-art multimodal models. Moreover, the multimodal representation learned from our data can be effectively transferred to other datasets like HowTo100m, increasing the VGSI accuracy by 15 - 20%. Our task will facilitate multimodal reasoning about procedural events.

## introduction recently, there has been growing attention on the representation of complex events, with renewed interest in script learning and commonsense reasoning @xcite @xcite . one aspect of event representation is the relationship between high-level goals and the steps involved @xcite . for example, given a goal (e.g. "change a tire"), an intelligent system should be able to infer what steps need to be taken to accomplish the goal (e.g. "place the jack under the car", "raise the jack"). in most work, events are represented as text @xcite @xcite , while they could have different modalities in the real world. learning goal-step relations in a multimodal fashion is an interesting challenge since it requires reasoning beyond image captioning. we contend that multimodal event representation learning will have interesting implications for tasks such as schema learning @xcite to mitigate reporting bias @xcite since steps are often not explicitly mentioned in a body of text. for instance, if a robot is asked to "get a slice of cake," it has to know that it should "take the cake out of the box", "cut a slice", "put it on a plate", and then "take the plate to the user". model should correctly recognize not only the specific action illustrated in an image (e.g., "turning on the oven", in figure 1 ), but also the intent of the action ("baking fish"). we collect data from wikihow articles, where most steps are illustrated with images. our vgsi training set is constructed using three sampling strategies to select negative image candidates as distractors. in the format of multiple-choice and image retrieval, we evaluate four state-of-the-art multimodal models: devise @xcite , similarity networks @xcite , triplet networks @xcite , and lxmert @xcite to human performance. it is observed that sota models designed for captionbased multimodal tasks @xcite struggle on vgsi, exhibiting a 40% gap in accuracy from human performance when using a challenging sampling strategy. one limitation of wikihow is that most images are drawings rather than photos (which are more typically used in computer vision research). the knowledge learned from wikihow is nevertheless useful when applied to real photos. we demonstrate this by pre-training a triplet-network on our wikihow vgsi task and then conducting transfer learning on out-of-domain datasets. our experiments show that pre-trained models can effectively transfer the goal-step knowledge to task-oriented video datasets, such as coin @xcite and howto100m @xcite . in addition, we design an aggregation model on top of sota models which treats wikihow as a knowledge base that further increases the transfer learning performance (see appendix c). we make three key contributions: (1) we pro- ## wikihow as multimodal resource we use wikihow as the data source for vgsi because it has been successfully adopted in prior work for procedural learning @xcite and intent detection @xcite in the language domain. as shown in figure 2 , each wikihow article contains a high-level goal and one or more different methods foot_1 to achieve it. each method then includes a series of specific steps, typically accompanied with corresponding images. the format of wikihow articles provides a hierarchical multimodal relationship between images and sentences. we can obtain three types of textimage pairs from wikihow, in increasing specificity: goal-image, method-image, and step-image. however, these text-image pairs are not enough information for a system to succeed on vgsi; it also needs the appropriate background knowledge. for the example in figure 2 , the system needs to know that "trick-or-treating" and "candies" are halloween traditions and that a "mask" is required during "covid-19". in total, as shown in ## conclusion in this paper, we propose the novel visual goal-step inference task (vgsi), a multimodal challenge for reasoning over procedural events. we construct a dataset from wikihow and show that sota multimodal models struggle on it. based on the transfer learning results on howto100m and coin, we validate that the knowledge harvested from our dataset could transfer to other domains. the multimodal representation learned from vgsi has strong potential to be useful for nlp applications such as multimodal dialog systems. lxmert learns a position-aware embedding as follows: the text tokens are extracted using a tokenizer @xcite and converted to index-aware embeddings s.t. w i and i are projected onto embedding spaces w i , u i , to get a common embedding. those inputs are then passed through a language encoder e g , an object relationship encoder e i , and a cross-modality transformer encoder e j . let then the cross-modality output x j is extracted from the output embedding x g j that corresponds to the special token [cls] appended to each input text. similarly to a.1.2, we use bce loss. a.2 features a.