We present techniques for transforming images and videos into painterly animations depicting different artistic styles. Our techniques rely on image and video analysis to compute appearance and motion properties. We also determine and apply motion information from different (user-specified) sources to static and moving images. These properties that encode spatio-temporal variations are then used to render (or paint) effects of selected styles to generate images and videos with a painted look. Painterly animations are generated using a mesh of brush stroke objects with dynamic spatio-temporal properties. Styles govern the behavior of these brush strokes as well as their rendering to a virtual canvas. We present methods for modifying the properties of these brush strokes according to the input images, videos, or motions. Brush stroke color, length, orientation, opacity, and motion are determined and the brush strokes are regenerated to fill the canvas as the video changes. All brush stroke properties are temporally constrained to guarantee temporally coherent non-photorealistic animations. The computer graphics community, in addition to concentrating on photorealistic rendering, strives for automatic methods to generate non-photorealistic and "expressive" renderings [Lansdown and Schofield 1995] . The need for non-photorealistic rendering and animation is obvious to anyone who has marvelled at artistic works where complex scenes and objects are rendered using pen and brush strokes, using lines, colors, and etches. In this paper, we present an automatic image-based rendering approach to generate expressive, artistic, and painterly animations. Our method enables the synthesis of painterly animations by using video analysis to manipulate spatio-temporally coherent brush strokes. Our method also supports merging motions from different sources to generate moving images from static pictures. To undertake automatic non-photorealistic rendering of moving images, we need to take into account both the spatial (across each frame) and temporal (from one frame to another) information. Towards this end, we propose a framework for generating and manipulating a mesh of dynamic brush strokes by analyzing individual frames as well as motion between frames. Each of these brush strokes has properties such as opacity, color, length, width, orientation, and motion (see Figure 2) . We also employ a higher-level concept of style to control the behavior of brush strokes and thus produce a spectrum of painterly styles. In our work we borrow from earlier contributions by Litwinowicz [1997] and Hertzmann et al. [2000]. We introduce several techniques that allow for significantly improved results in transforming video into painterly animations. (1) Each brush stroke property is constrained over time to ensure that smooth, temporally constrained animations are produced. Brush stroke generation and deletion are 113
Hays et al. REF proposed new stylistic parameters for the brush strokes to mimic different artistic styles.
Image and video based painterly animation
NPAR '04
Computer Science