Visual Grounding Annotation of Recipe Flow Graph (2020)

Abstract: In this paper, we provide a dataset that gives visual grounding annotations to recipe flow graphs. A recipe flow graph is a representation of the cooking workflow, which is designed with the aim of understanding the workflow from natural language processing. Such a workflow will increase its value when grounded to real-world activities, and visual grounding is a way to do so. Visual grounding is provided as bounding boxes to image sequences of recipes, and each bounding box is linked to an element of the workflow. Because the workflows are also linked to the text, this annotation gives visual grounding with workflow’s contextual information between procedural text and visual observation in an indirect manner. We subsidiarily annotated two types of event attributes with each bounding box: “doing-the-action,” or “done-the-action”. As a result of the annotation, we got 2,300 bounding boxes in 272 flow graph recipes. Various experiments showed that the proposed dataset enables us to estimate contextual information described in recipe flow graphs from an image sequence.

## introduction procedural texts are suitable for the target of natural language understanding (nlu) because they are goal-oriented descriptions and we can almost define the understanding of their goals. some studies have proposed the framework of understanding procedural texts with graphs, which can represent the entire workflow @xcite @xcite . these frameworks would be helpful for real-world systems, such as smart kitchen @xcite and cooking robot @xcite , to understand the context of these workflows and execute next actions. therefore, it is essential to ground visual observations (images or videos) with procedural texts. visual grounding is one of the solutions to help computers to understand which objects are aligned with textual descriptions. in the previous studies, they targetted the pair of general texts and visual observations. mscoco @xcite , flickr30k @xcite , and youtube-boundingbox @xcite are typical datasets for such tasks. they reported that annotating bounding boxes with a textual description helps computers to know which objects they have to pay attention to for image captioning @xcite , visual question answering (vqa) @xcite , and some other visual grounding tasks @xcite @xcite . compared with such general visual grounding tasks, a dataset with procedural texts is differentiated by an additional goal of understanding the context information, i.e., a model should consider the entire workflow of a procedural text. we call this task contextual visual grounding. in this background, this paper focuses on the domain of cooking recipes and provides a new dataset that has visual grounding annotation of contextual information represented in recipe flow graph (r-fg) @xcite , which can represent a structured workflow described in a procedural text. figure 1 shows an overview of our annotation scheme. given a procedural text with its image sequence and flow graph, annotators put a bounding box of an object in an image, which is linked to a node of the flow graph. this annotation enables us to link visual objects to contextual information directly. through the process, as a subsidiary contribution, we found that visual objects represent roughly in either of the following two events: "doing-" and "done-the-action." in the dataset, we also provide attribute labels for these events. for example, in the "step 1" in figure 1 , the chef is just dicing the broccoli, whereas the chef has done the dicing action in the "step 2" of figure 1 . to make it obvious whether an action to a visual object is in performance or has already been performed, we annotated each bounding box with an event attribute. as a result of the annotation, we built an r-fg bounding box (r-fg-bb) dataset, which consists of 2,300 bounding boxes in 272 r-fgs. in the experiments, we confirmed that, with our dataset, we could obtain a model that grounds bounding boxes with nodes in a flow graph, which gives a connection between the bounding boxes of images, while classifying event attributes correctly. ## recipe flow graph corpus the language part of our r-fg-bb dataset is compatible with the r-fg corpus @xcite . the r-fg corpus consists of cooking recipe texts (or simply "recipes" hereafter) annotated with flow graphs connecting important terms in the texts. our r-fg-bb dataset is an extension connecting some of the important terms to bounding boxes in the image attached to each step in the text. in this section, we explain cooking recipe texts and flow graphs of the r-fg corpus. ## dataset annotation standard in this section, we explain our annotation framework in detail. as mentioned in section 1, we performed two types of annotations: bounding boxes with nodes in the flow graph and event attributes with bounding boxes. we describe these annotations in the subsequent sections. ## annotation statistics as the recipes for our r-fg-bb dataset, we selected, from the r-fg corpus, those having an image to each step. as a result, we had 70 recipes. to increase the number of recipes in our corpus, we also selected 202 recipes randomly from those consisting of 3 to 10 steps in the cookpad image ## experimental evaluation to investigate the usefulness of our dataset, we propose three tasks based on its most interesting characteristics: symbol grounding, bounding box linking prediction, and event attribute classification. in this section, we define these tasks, give a preliminary solution to each one, and report the experimental results. ## application our r-fg-bb dataset has many potential applications. in this section, we describe two typical usages of our dataset. ## conclusion in this paper, we presented details of our r-fg-bb dataset. its language part is procedural texts, each of which is annotated with a flow graph whose nodes correspond to important terms in the text and are connected to form a dag. the visual part of the dataset is bounding boxes in the images attached to each text. the bounding boxes are connected to the nodes of the flow graph, allowing the dataset users to have contextual information of the visual objects. we performed various experiments. the results showed that the proposed simple models could ground bounding boxes with nodes in a flow graph, which gives a linking between the bounding boxes over images while classifying event attributes correctly. therefore, our dataset is useful for contextual visual grounding. with our dataset, one can try contextual visual grounding research using interesting triplets: bounding boxes, procedural text, and its flow graph. this combination would be effective for understanding the entire workflow of a procedural text, grounding contextual visual information.