  We present simple video-specific autoencoders that enables human-controllable
video exploration. This includes a wide variety of analytic tasks such as (but
not limited to) spatial and temporal super-resolution, spatial and temporal
editing, object removal, video textures, average video exploration, and
correspondence estimation within and across videos. Prior work has
independently looked at each of these problems and proposed different
formulations. In this work, we observe that a simple autoencoder trained (from
scratch) on multiple frames of a specific video enables one to perform a large
variety of video processing and editing tasks. Our tasks are enabled by two key
observations: (1) latent codes learned by the autoencoder capture spatial and
temporal properties of that video and (2) autoencoders can project
out-of-sample inputs onto the video-specific manifold. For e.g. (1)
interpolating latent codes enables temporal super-resolution and
user-controllable video textures; (2) manifold reprojection enables spatial
super-resolution, object removal, and denoising without training for any of the
tasks. Importantly, a two-dimensional visualization of latent codes via
principal component analysis acts as a tool for users to both visualize and
intuitively control video edits. Finally, we quantitatively contrast our
approach with the prior art and found that without any supervision and
task-specific knowledge, our approach can perform comparably to supervised
approaches specifically trained for a task.
