From “Before” to “After”: Generating Natural Language Instructions from Image Pairs in a Simple Visual Domain (2020)

Abstract: While certain types of instructions can be com-pactly expressed via images, there are situations where one might want to verbalise them, for example when directing someone. We investigate the task of Instruction Generation from Before/After Image Pairs which is to derive from images an instruction for effecting the implied change. For this, we make use of prior work on instruction following in a visual environment. We take an existing dataset, the BLOCKS data collected by Bisk et al. (2016) and investigate whether it is suitable for training an instruction generator as well. We find that it is, and investigate several simple baselines, taking these from the related task of image captioning. Through a series of experiments that simplify the task (by making image processing easier or completely side-stepping it; and by creating template-based targeted instructions), we investigate areas for improvement. We find that captioning models get some way towards solving the task, but have some difficulty with it, and future improvements must lie in the way the change is detected in the instruction.

## introduction for ig-ba, we need pairs of images (depicting the before and the after state) as shown in figure 2 , and as output examples of instructions that when executed turn the former into the latter state. making use of techniques developed for the task of image captioning (that is, the description of single images), we explore to what extent this setup allows us to sidestep the explicit planning steps that were used in earlier natural language generation work on instructions, for example in the context of the give challenge @xcite . the contributions of this paper are: • specification of the task of instruction generation from before/after images (ig-ba). ## related work generating descriptions of single images is by now a well-studied task @xcite , where the best-performing models typically make use of image encodings computed by convolutional neural networks (cnns) and caption generation (decoding) based on auto-regressive recurrent networks @xcite . we will directly build on these approaches and discuss the ones we utilise in more detail below. there is also some recent work that looks at a task which, similar to ours, takes image pairs as input. in change captioning @xcite , the task is to verbalise what is different between two otherwise very similar images. our task contains this, but goes beyond it in that it also has to be verbalised how that difference can be effected. in that work, specialised architectures are presented that can more easily extract differences. here, we wanted to start by exploring more standard captioning approaches as a baseline in order to fully understand the requirements of our task, leaving further architectural adaptations to future work. after early work in the context of the give challenge @xcite , @xcite ), there is some renewed interest in instruction giving. @xcite presented an instruction giving platform called mc-saar-instruct where players can interact with a bot in the minecraft world which instructs them to build something. this is very related to our interest in this project; for now, however, that work still assumes a symbolic representation as input. in the field of natural language generation, the production of referring expressions is a wellestablished task @xcite , increasingly also tackled with neural methods @xcite . ig-ba includes this; but as will become clear in the next section, the data that we use here allows us to factor it out, as references to objects can simply be done via unique names. the complexity in our task comes from the spatial language required to denote locations, which is something not found to that degree neither in image captioning nor referring expression generation. generation of spatial expressions has seen some attention in recent years, e.g. by @xcite , who investigate the spatial language that neural language models can learn and express. ## data: blocks and blocks gen we will now describe how we can make use of data collected for instruction following for our task of instruction giving. @xcite collected the blocks dataset in order to study instruction following in a simple visual environment. the environment consists of up to 20 blocks of the same size, which are placed on a board. the blocks are uniquely labelled either with a number between 1 and 20, or with the logo of a major company; this makes reference to blocks unambiguous and straightforward. to collect instructions, the authors created pairs of states s t , s t+1 , represented as images i t and i t+1 (computer-generated using a 3d-engine), that differ in the placement of only a single block, and presented these to crowd workers who were tasked with producing a natural language instruction whose execution would turn s t into s t+1 . ## results and discussion all models reach a performance plateau within the maximum number of epochs. the results of the baselines are reported in first, we present some samples of the generated instruction from the various models in image reference template cnn+lstm cnn+lstm+att. move the burger king block directly south of the bmw block. take block burger king and move it below block bmw move the burger king block to the left of the coca cola block . move the burger king block below the nvidia block. move the burger king cube directly next to the cocacola cube touching it 's left side take block burger king and move it to the left of block coca cola move the adidas block to the left of the bmw block . put the burger king block in the first open space to the left of the esso block. take the pepsi block from the bottom and move it to the right of and in contact with the mcdonald 's block in the center . pick up block pepsi and move it to the right of block mcdonalds move the shell block to the right of the mcdonalds block . put the pepsi block in the first open space to the right of the mcdonalds block. nn-retrieve is the lower bound baseline and performs poorly. this indicates that the image encodings are not enough to retrieve appropriate instructions for an unseen image pair. the cnn+lstm based models can improve on this lower bound. bleu score jumps up by over 0.2 so the model is able to produce instructions that are more similar to the references than those nn-retrieve selects from the most similar pair. target and landmark match improve as well, however only landmark by a bigger margin. this model type improves further when presented with the pre-processed image (here, only both is shown; a more detailed discussion will be given below). it improves on all metrics, which suggests that the image component cannot on its own extract the information that the pre-processing makes available. adding an attention mechanism improves (cnn+lstm+att vs cnn+lstm), but the gain of pre-processing the image (cnn+lstm+att+i b ) remains. overall, that model achieves the best results (among the trained models). interestingly, the improvement on the task-specific metrics target and landmark is not directly reflected by improvements on the string-based metrics. for example, on bleu, the full model is substantially worse than the variant without attention, even though it is substantially better at naming the target block. generally, all generation models conditioned on images in ## conclusion we have introduced the task of instruction giving from before/after images (ig-ba), and shown that an existing dataset for instruction following can be used to train a model for this task. for the model, we used established architectures that have served the task of image captioning well. through various analyses of the model, we have shown that their image processing capabilities seem to form a bottleneck; when provided with a pre-parsed scene representation, the quality of the generated instructions improves considerably. this suggests a clear route for future work, namely to improve the image processing capabilities, possibly along the lines of the recent, related work on change captioning @xcite . the results that we have achieved are nevertheless encouraging that a performance can be achieved that can support our ultimate goal, which is to construct a system for interactive and collaborative instruction following in an assembly domain like shown in figure 1 . in comparison to the blocks data set, figures of assembly instructions are much more complex as they include changes of point of view, rotations on different axis and special informative pictograms (e.g. the cordless drill driver in figure 1 ). additionally, the instructions may refer to previous steps or have more variability because of the less defined naming. for example, the figure above may requires the model to describe "the screw with the long, smaller head must be screwed without a drill driver until its middle par hits the wood surface". for future work, the language generation component of such system must be able to generate the instruction incrementally and describe the actions including advice of execution in the instruction.