Visual Recipe Flow: A Dataset for Learning Visual State Changes of Objects with Recipe Flows (2022)

Abstract: We present a new multimodal dataset called Visual Recipe Flow, which enables us to learn a cooking action result for each object in a recipe text. The dataset consists of object state changes and the workflow of the recipe text. The state change is represented as an image pair, while the workflow is represented as a recipe flow graph. We developed a web interface to reduce human annotation costs. The dataset allows us to try various applications, including multimodal information retrieval.

## introduction our aim is to track how foods are processed and changed toward the final food product by each cooking action given a recipe text. this requires some knowledge of the actions: what foods and actions are involved and how the action changes them. skilled chefs can easily imagine these action effects while understanding the required foods. we are interested in building an autonomous agent endowed with this ability, as illustrated in figure 1 . this example involves two cooking actions, and the agent imagines the second action result: the shredded cabbage in the bowl. this also implicates the food requirement: the shredded cabbage produced by the previous action. the prediction for the required foods and action results is indeed a natural ability for humans when they cook something. thus, this is also crucial for intelligent autonomous agents to understand recipe texts. previous work on this line of research provided visual annotation for each cooking instruction @xcite . @xcite attached an image with bounding boxes of objects to each instruction, while @xcite split an instruction into sentences and attached frames to each sentence. however, their annotations are often insufficient to predict the action result for each object. a typical case is an instruction in a sentence that directs multiple actions. for example, the instruction of "slice the tomato and put it into the bowl" produces two action results: the sliced tomato and that put in the bowl. therefore, an instruction-wise visual annotation is insufficient for our task, and action-wise visual annotation is required. preparing a more dense visual annotation is one straightforward way to handle this case. toward the realization of an agent that predicts the result of each action, we introduce a new multimodal dataset called visual recipe flow (vrf). the dataset consists of object foot_0 state changes caused by every action and the workflow of the text. the change is given as an image pair, while the workflow is given in the format of recipe flow graph (r-fg) @xcite . each image pair is grounded in the r-fg, which gives the cross-modal relation. figure 2 shows an example of our dataset. we focus on recipe text involving various cooking actions, foods, and state changes, which is one of the representatives of procedural texts. understanding these texts by tracking object state changes is one of the recent trends @xcite @xcite @xcite . our work also contributes to this line of research. since images directly express object appearances in the real world @xcite , our dataset would provide rich information for the changes. the sequential nature of our dataset can also be used to test the reading comprehension ability of large-scale language models @xcite . furthermore, since our dataset has arbitrary interleaved visual and textual annotations, it is also possible to evaluate the fewshot capability of vision-language models on such data @xcite . ## the vrf dataset the visual recipe flow (vrf) dataset is a new multimodal dataset. it provides visual annotations for objects in a recipe text before and after a cooking action. we identify expressions including the action in the text by using recipe named entities (r-nes) @xcite , which can be extended to other procedural tasks. based on the r-nes, it also provides a representation of the recipe workflow as a recipe flow graph (r-fg) @xcite . in this section, we first explain the overview of the r-fg and then introduce our visual annotation. ## annotation standards in this section, we describe our annotation standards. the annotation consists of three steps in order: (i) r-ne annotation, (ii) r-fg annotation, and (iii) image annotation. each recipe has an ingredient list, an instruction list, and a cooking video. figure 3 shows an example of the annotations. r-ne annotation. first, we annotated words in the ingredient and instruction lists with r-ne tags foot_1 . we used the eight types of r-ne tags, following mori et al. (2014). see appendix a for details. image annotation. third, we annotated object states with images, sampled at 3 frames per second from the videos. each object required for any cooking action is annotated as a pair of frames of states before and after the action. when there are multiple suitable frames, we prioritize the one based on the visual clarity of the object. in some cases, objects are always heavily covered by human hands or abbreviated from the video. we treat them as missing data. ## annotation results this section first describes our annotation process and the statistics for the annotation results. it then investigates the dataset quality and finally assesses our dataset by conducting experiments. ## conclusion we have presented a new multimodal dataset called visual recipe flow. the dataset provides dense visual annotations for object states before and after a cooking action. the annotations allows us to learn each cooking action result. experimental results demonstrated the effectiveness of our annotations for a multimodal information retrieval task. with our dataset, one can also try various applications, including multimodal commonsense reasoning and procedural text generation. tag meaning # tags f food 5,098 t tool 758 d duration 129 q quantity 1,778 ac action by chef (cooking action) 2,532 af action by food 353 sf state of food 971 st state of tool 67 total -11,686