NUWA - XL : Diffusion over Diffusion for e X tremely Long Video Generation (2023)

Abstract: In this paper, we propose NUWA-XL, a novel Diffusion over Diffusion architecture for eXtremely Long video generation. Most current work generates long videos segment by segment sequentially, which normally leads to the gap between training on short videos and inferring long videos, and the sequential generation is inefficient. Instead, our approach adopts a “coarse-to-fine” process, in which the video can be generated in parallel at the same granularity. A global diffusion model is applied to generate the keyframes across the entire time range, and then local diffusion models recursively fill in the content between nearby frames. This simple yet effective strategy allows us to directly train on long videos (3376 frames) to reduce the training-inference gap and makes it possible to generate all segments in parallel. To evaluate our model, we build FlintstonesHD dataset, a new benchmark for long video generation. Experiments show that our model not only generates high-quality long videos with both global and local coherence, but also decreases the average inference time from 7.55min to 26s (by 94.26%) at the same hardware setting when generating 1024 frames. The homepage link is [NUWA-XL]( https://msra-nuwa.azurewebsites.net )

## introduction recently, visual synthesis has attracted a great deal of interest in the field of generative models. existing works have demonstrated the ability to generate high-quality images @xcite @xcite ) and short videos (e.g., 4 seconds @xcite , 5 seconds @xcite , 5.3 seconds @xcite ). however, videos in real applications are often much longer than 5 seconds. a film typically lasts more than 90 minutes. a cartoon is usually 30 minutes long. even for "short" video applications like tiktok, the recommended video length is 21 to 34 seconds. longer video generation is becoming increasingly important as the demand for engaging visual content continues to grow. however, scaling to generate long videos has a significant challenge as it requires a large amount of computation resources. to overcome this challenge, most current approaches use the "autoregressive over x" architecture, where "x" denotes any generative models capable of generating short video clips, including autoregressive models like phenaki @xcite , tats @xcite , nuwa-infinity @xcite ; diffusion models like mcvd @xcite , fdm @xcite , lvdm @xcite . the main idea behind these approaches is to train the model on short video clips and then use it to generate long videos by a sliding window during inference. "autoregressive over x" architecture not only greatly reduces the computational burden, but also relaxes the data requirements for long videos, as only short videos are needed for training. unfortunately, the "autoregressive over x" architecture, while being a resource-sufficient solution to generate long videos, also introduces new challenges: 1) firstly, training on short videos but forcing it to infer long videos leads to an enormous training-inference gap. it can result in unrealistic shot change and long-term incoherence in generated long videos, since the model has no opportunity to learn such patterns from long videos. for example, phenaki @xcite and tats @xcite are trained on less than 16 frames, while generating as many as 1024 frames when applied to long video generation. 2) secondly, due to the dependency limitation of the sliding window, the inference process can not be done in parallel and thus takes a much longer time. for example, tats @xcite takes 7.5 minutes to generate 1024 frames, while phenaki @xcite takes 4.1 minutes. ## conclusion we propose nuwa-xl, a "diffusion over diffusion" architecture by viewing long video generation as a novel "coarse-to-fine" process. to the best of our knowledge, nuwa-xl is the first model directly trained on long videos (3376 frames), closing the training-inference gap in long video generation. additionally, nuwa-xl allows for parallel inference, greatly increasing the speed of long video generation by 94.26% when generating 1024 frames. we further build flintstoneshd, a new dataset to validate the effectiveness of our model and provide a benchmark for long video generation.