Character-centric Story Visualization via Visual Planning and Token Alignment (2022)

Abstract: Story visualization advances the traditional text-to-image generation by enabling multiple image generation based on a complete story. This task requires machines to 1) understand long text inputs, and 2) produce a globally consistent image sequence that illustrates the contents of the story. A key challenge of consistent story visualization is to preserve characters that are essential in stories. To tackle the challenge, we propose to adapt a recent work that augments VQ-VAE with a text-to-visual-token (transformer) architecture. Specifically, we modify the text-to-visual-token module with a two-stage framework: 1) character token planning model that predicts the visual tokens for characters only; 2) visual token completion model that generates the remaining visual token sequence, which is sent to VQ-VAE for finalizing image generations. To encourage characters to appear in the images, we further train the two-stage framework with a character-token alignment objective. Extensive experiments and evaluations demonstrate that the proposed method excels at preserving characters and can produce higher quality image sequences compared with the strong baselines.

## introduction text-to-image generation @xcite @xcite , as a benchmark task to test ai systems' multi-modal capability, has been widely studied over the past decade. the task takes a short sentence or phrase as input and outputs a single image illustrating the content of the input text. one of the latest successful works is dall-e @xcite , which efficiently encodes and decodes images with discrete visual tokens and achieves remarkable text-to-image gen- * work done when the author was visiting ucla. figure 1 : example of generated images from previous models (cp-csv and vlc), the vq-vae with transformers (vq-vae-lm) baseline, and our proposed model (vp-csv). we highlight the characters in each sentence in red. we can see that the generated image sequence by vp-csv achieves the best image quality and character preservation. the two-stage model freezes vq-vae and adapts the text-to-visual-token transformer by dividing it into two separate modules: 1) the plan module generates a token plan consisting of character and non-character (background) tokens, which reinforces our system's attention to characters; 2) based on the character token plan, the completion module produces the entire sequence of visual tokens, which are used in vq-vae decoder to produce final images. the two modules are separately trained to avoid the expensive decoding in training. crucially, we train the completion module with an auxiliary semantic alignment loss that encourages appropriate character tokens to appear in the final visual tokens. to the best of our knowledge, all previous works in story visualization employ @xcite methods, which are shown to be unstable @xcite . figure 1 shows that the images generated by vq-vae-lm are clearer than those by previous gan-based models such as cp-csv @xcite and vlc @xcite , suggesting the instability of gan-based models. on the other hand, vq-vae-lm architecture has been shown to stabilize the training @xcite , leading to better image quality (the third row in figure 1 ). finally, our proposed twostage visual planning model (vp-csv) can further improve vq-vae-lm, which produces the best quality images in figure 1 . beyond this example, we conduct extensive experiments to show that our proposed two-stage visual token plan and the semantic alignment method perform better on character preservation compared with various baseline models, which in general contribute to the improved story visualization quality. we summarize our contributions below, ## vq-vae-lm in this section, we review the vq-vae architecture and how it can be extended to a text-to-image variant, dubbed vq-vae-lm. we refer readers to the original vq-vae paper for more details. ## results our human evaluation. since automatic metrics can be insufficient in evaluating the story visualization quality, we further conduct human evaluations by asking annotators to analyze the performance of the story visualization models. we hire in total 9 amazon mechanical turkers who succeed in our previous large annotation tasks and passed the additional qualification exam for story visualization. in total, we collect 218 sets of valid annotations story visualization results. by comparing models provided with the same input story, we ask annotators to rank the resulting image sequences. they rank the visualizations on a scale of 1 to 3, with 3 indicating the best and 1 otherwise. although ties are allowed, annotators are encouraged to provide unique rankings for each of the compared image sequences. we show the detailed instructions and the user interface in appendix a.6. analysis on visual planning. we conduct an indepth study to analyze the performance of visual planning. in figure 6 , the first row shows the intermediate image results with the background masked, and the second row shows the final images. we observe that our character token planning model successfully output character-related tokens in most of the cases (blue boundary). this aligns with our expectation that our character-token plan module reinforces the system's attention to the appropriate characters. however, a mistake can e observed (red boundary) where the visual token completion model predicts a character based on nothing (masked background). one possible explanation could be the discrepancy resulting from our two-stage modules being separately trained. in addition, the fifth image (orange boundary) reveals an example of low-quality character generation, which suggests future work to boost our model's visualization capacity. analysis on character alignment. our proposed character alignment method encourages the generated visual tokens to encompass the characterrelated tokens. we calculated the percentage of character-token coverage in the generated visual token for the compared models. figure 7 shows that our proposed vp-csv model with character alignment can achieve the highest coverage, demonstrating the effectiveness of using semantic loss to preserve characters in the proposed framework. ## conclusion in this work, we propose visual planning and character token alignment to improve character preservation and visual quality. extensive evaluation results show that our vp-csv model outperforms the strong baseline models by a large margin. future research in story visualization can aim at incorporating events that include not only characters but also their actions and relationships with one another, which furhter challenge machines' representation capability in story visualization.