Generating Coherent Sequences of Visual Illustrations for Real-World Manual Tasks (2024)

Abstract: Multistep instructions, such as recipes and how-to guides, greatly benefit from visual aids, such as a series of images that accompany the instruction steps. While Large Language Models (LLMs) have become adept at generating coherent textual steps, Large Vision/Language Models (LVLMs) are less capable of generating accompanying image sequences. The most challenging aspect is that each generated image needs to adhere to the relevant textual step instruction, as well as be visually consistent with earlier images in the sequence. To address this problem, we propose an approach for generating consistent image sequences, which integrates a Latent Diffusion Model (LDM) with an LLM to transform the sequence into a caption to maintain the semantic coherence of the sequence. In addition, to maintain the visual coherence of the image sequence, we introduce a copy mechanism to initialise reverse diffusion processes with a latent vector iteration from a previously generated image from a relevant step. Both strategies will condition the reverse diffusion process on the sequence of instruction steps and tie the contents of the current image to previous instruction steps and corresponding images. Experiments show that the proposed approach is preferred by humans in 46.6% of the cases against 26.6% for the second best method. In addition, automatic metrics showed that the proposed method maintains semantic coherence and visual consistency across steps in both domains.

## introduction when humans undertake a task with numerous intricate steps, merely reading a step description is limiting, leaving the user to imagine and infer some of the more nuanced details @xcite . complementing the textual step instructions with 1 https://novasearch.github.io/ generating-coherent-sequences/ images enhances the user experience by better communicating and representing the text semantics and ideas @xcite . although prompt-based image generation has advanced significantly @xcite @xcite , state-of-the-art (sota) models such as latent diffusion models (ldms) @xcite still struggle when generating image sequences to accompany textual instruction steps @xcite . the challenge lies in effectively combining two key aspects: (a) accurately portraying the actions outlined in the step instructions, and (b) ensuring coherence between successive images to avoid confusing the user. existing storytelling approaches @xcite @xcite operate mostly on linear storytelling and use synthetic cartoon datasets with explicit sequence information, i.e., the textual prompts describe the images appropriately and have no implicit co-references. these aspects limit the applicability of existing methods to real-world scenarios (figure 1 ), where there is a lack of informative prompts accompanying images, and dependencies between prompts are not necessarily linear. in this paper, we explore the generation of image sequences within two domains: recipe instructions, and do it yourself (diy) guides, both showing increasing online consumption @xcite @xcite . in these domains, accuracy and coherence are of utmost importance to ensure that the result of all manual actions is correct, and that the user is correctly guided to the target output, figure 1 . these domains contain (i) complex sequential manual tasks of detailed actions, (ii) coherence requirements for the images accompanying the sequence step descriptions, and (iii) a non-linear sequential structure, where steps may be related to earlier steps-not necessarily the previous step. to tackle these challenges, we propose to extend latent diffusion models @xcite , with an llm decoder to semantically condition the reverse diffusion process in the sequence of steps and a copy mechanism to select the best ldm initialisation. the image generation process is conditioned on the current step and the previous steps, to increase semantic coherence. in addition, our method initializes the reverse diffusion process with a latent vector iteration copied from a previous generation process to ensure the visual coherence of the generated image. through this dual attendance to past textual and visual items in the sequence, we aim to achieve semantic coherence, which pertains to the presence and persistence of objects in consecutive images, and visual coherence, which aims to ensure the consistency of backgrounds and visual object properties across successive images. extensive automatic and manual evaluations confirmed that our model outperforms strong baselines in terms of the overall quality of the generated sequence of illustrations in the cooking and diy domains. ## related work methods to generate sequences of images, conditioned on textual input, have been explored in the story visualization and story continuation tasks. story visualization aims at generating a coherent sequence of images, based on a multi-sentence paragraph or a series of captions forming a narrative @xcite @xcite . story continuation is a variant of story visualization, in which the generated sequence is initiated by a source image. in both tasks, generating every image independently of other images in the sequence leads to low visual coherence. in contrast, editing the previous image like in @xcite will lead to insuficiently diverse images. several works addressed these tasks. @xcite proposed ar-ldm, a method to tackle story visualization and continuation using a history-aware autoregressive latent diffusion model @xcite , which encodes the history of caption-image pairs into a multimodal representation that guides the ldm denoising process. despite the intuitive idea, the computational complexity of the conditioning network makes this approach too costly. additionally, ar-ldm still shows room for improvement in terms of coherence. @xcite noted that ar-ldm conditions the current generation on all historic frames and captions equally, despite not all frames being similarly related. to tackle this limitation, they proposed acm-vsg, a method that selectively adopts historical text-image data for the generation of the new image. the adaptive encoder automatically finds the relevant historical text-image pairs via clip similarity. a key difference between ar-ldm and acm-vsg is the computational cost: while ar-ldm fine-tunes clip, blip, and the ldm, acm-vsg trains only the cross-attention module, at a much lower cost. @xcite used the full history of u-net latent vectors from all segments of the sequence, averaging these historic latent vectors in a cross-attention layer that is merged with the existing one in the ldm pipeline. in this method, image generation is conditioned on the text and on the entire set of latent vectors, with limited awareness of the visual coherence between segments. the above approaches focus on two synthetic cartoon datasets @xcite , with limited characters and scenes. among these, @xcite shows limited performance when applied to real-world sequences, and @xcite do not evaluate their solutions in a real-world scenario. additionally, these datasets have textual descriptions of the associated images, which are rarely available for real-world complex tasks -the focus of this paper. finally, it is relevant to note the conceptual relation to the visual storytelling problem @xcite to generate a text story from a sequence of images @xcite and to the extraction of news storylines @xcite . while, conceptually similar, both tasks can be seen as the inverse of the problem addressed in the current paper. ## illustrating real-world manual tasks we consider a set of manual tasks d, where each task t @xmath0, t @xmath1, v 1 ), ..., (s n , v n )}. a task step (s i , v i ), consists of a natural language instruction s i , and its corresponding visual instruction v i . given the sequence of steps {s 1 , . . . , s n }, our goal is to generate a sequence of images {v 1 , . . . , v n }, in which v i visually represents step s i . a step s i may be dependent on any number of previous steps, in a non-linear sequential structure @xcite . to generate each image accurately, the model needs to condition its output not only on s i but also on previous steps {s 1 , . . . , s i-1 }; this way, context is preserved even when steps are ambiguous or lack information, e.g. "add two eggs and mix". in addition to previous step instructions, we also need to condition on previously generated images, {v 1 , ..., v i-1 }, to maintain the visual aspects that are only introduced in the images, such as object properties and background artefacts not mentioned in the text. ## sequential latent diffusion model the latent diffusion model proposed by @xcite transforms the diffusion process into a low-dimensional latent space through an encoder @xmath2. the complete model is also conditioned on an input y by augmenting the u-net backbone with a cross-attention layer to support the encoded input τ θ (y). the conditional ldm is learned with the loss, ## results and discussion in this section, we start by comparing the proposed method to latent diffusion models, sd2.1 @xcite , and non-ldm models, i.e. amused @xcite and dall•e mini @xcite , with automatic metrics. for the ablation studies, we experimented with using (1) a random seed for all steps of the sequence, (2) a fixed seed for all steps of the sequence, (3) a fixed latent vector iteration from the previous step, represented by latent k, where k is the fixed iteration, and (4) the proposed method that selects a latent vector iteration, z j i from the previous denoising steps as a starting seed. ## conclusions in this paper, we addressed the problem of illustrating complex manual tasks and proposed a framework for generating a sequence of images that illustrate the manual task. the framework is composed of a novel sequence context decoder that sequence context captions avg. rating automatic and human evaluations in the target domain demonstrated the strong performance of the framework in generating coherent sequences of visual instructions of manual tasks. the proposed method was preferred by humans in 46.6% of the cases against 26.6% for the second best method. in addition, automatic measures, also confirmed that our method maintains visual and semantic coherence while illustrating a complex manual task. the generalization to unseen domains was successfully validated in the diy domain with positive results.