Papers
arxiv:2309.07906

Generative Image Dynamics

Published on Sep 14, 2023
· Featured in Daily Papers on Sep 15, 2023

Abstract

We present an approach to modeling an image-space prior on scene dynamics. Our prior is learned from a collection of motion trajectories extracted from real video sequences containing natural, oscillating motion such as trees, flowers, candles, and clothes blowing in the wind. Given a single image, our trained model uses a frequency-coordinated diffusion sampling process to predict a per-pixel long-term motion representation in the Fourier domain, which we call a neural stochastic motion texture. This representation can be converted into dense motion trajectories that span an entire video. Along with an image-based rendering module, these trajectories can be used for a number of downstream applications, such as turning still images into seamlessly looping dynamic videos, or allowing users to realistically interact with objects in real pictures.

Community

Project page with interactive demo:
https://generative-dynamics.github.io/

Amazing!

interesting approach

Hi. Curious to know how the demo runs in real-time considering it is using multiple models?

Paper author

Hi. Curious to know how the demo runs in real-time considering it is using multiple models?

The web demo uses a pregenerated neural stochastic motion texture, and for speed it uses a lower-quality rendering method (mesh warping in webgl) than the full rendering model in the paper.

Will code be published?

Good Job!
Will code be published?

This looks incredible, and would be a great asset to my video productions! Is this going to be released as an app or by code per chance?

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2309.07906 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2309.07906 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2309.07906 in a Space README.md to link it from this page.

Collections including this paper 19