Paper:
Deep neural networks have had unprecedented success in computer vision, natural language processing, and speech largely due to the ability to search for suitable task algorithms via differentiable programming. In this paper, we borrow ideas from Kolmogorov complexity theory and normalizing flows to explore the possibilities of finding arbitrary algorithms that represent data. In particular, algorithms which encode sequences of video image frames. Ultimately, we demonstrate neural video encoded using convolutional neural networks to transform autoregressive noise processes and show that this method has surprising cryptographic analogs for information security.
In algorithmic information theory, the Kolmogorov complexity of an object is the length of the shortest computer program that, without additional input, produces the object as output (Ming & Paul, 2008). For example, a long sequence of ones contains very little information because a short program for i ← 0, n do print 1 can output the data. Similarly, the transcendental number π = 3.1415 . . . is an infinite sequence of seemingly random decimal digits but contains only a small amount of information since there exists a very short program that produces the consecutive digits of π forever (Bailey et al., 1997). While this idea of exchanging data with computation is theoretically intriguing, from a practical point of view, explicit specification of instructions to obtain some target representation is generally an intractable approach for arbitrary data.
Recently, however, artificial intelligence had broad success in computer vision, natural language processing, and speech largely due to the ability to search for suitable task algorithms using differentiable programming together with large datasets to obtain abstract computations in the form of neural networks. In this paper, we turn the process around and leverage this same differentiable programming approach to find abstract computations which represent data. That is, we seek algorithms which encode arbitrary information for storage and retrieval. In particular, we demonstrate methods for the encoding of sequential video data within the weights of convolutional neural networks.
In the simplest formulation, video data, X, is treated as an finite ordered sequence of image frames
and we seek an autoregressive computation of the form
where y 0 ← x 0 and f is some convolutional neural network parametrized by weights θ such that y i ≈ x i . It is possible that f can be trained using a self-supervised learning approach where consecutive image frames form training examples and the sum-of-squares, y i − x i 2 , is minimized. That is, during training, f learns a mapping from x i to x i+1 , or rather, learns to advance the current frame by one step. The idea is somewhat similar to conventional autoencoders (Hinton & Zemel, 1993), but there, f is trained with the identity mapping, from x i on to x i . The catch here is that after training, during recall, f does not receive x i as input but instead y i from the previous evaluation. Since y i only approximates x i , reconstruction error accumulates at each iteration of eq. 2 until sequence reproduction fails 1 .
The natural inclination at this point is to extend the approach to train over the full sequence by evaluating eq. 2 N times to produce {y 1 , . . . , y N } and minimizing the mean squared error over the entire sequence at each training iteration. This way, how the network is trained and how the sequence is recalled post-training are congruent. Unfortunately, for sequences beyond a few elements, this is not tractable as long unrolled gradients readily vanish or explode and there is no opportunity for batching which significantly limits training performance and scalability.
The situation can be salvaged by adopting a curriculum learning strategy (Bengio et al., 2009) whereby the training sequence length is slowly increased after achieving some nominal loss objective over shorter intervals. It is important to note that, because CNNs have no recurrent hidden state, this approach can be successfully batched during training over random subsequences of X. Therefore, modulating the training subsequence length to one yields the aforementioned self-supervised learning approach whereas a training sequence length equal to the number of frame elements yields the full sequence loss. While this batched curriculum learning strategy does mitigate challenges associated with teacher forcing, gradients can still be temperamental for larger training sequence lengths necessitating the use of gradient clipping and learning rate scaling (You et al., 2017;).
An interesting situation occurs when
That is, there exists a nonunique mapping or sequence ambiguity which is not resolvable from a purely supervised learning point of view since a unique input evaluates to multiple output. It is perhaps surprising to some readers that eq. 2 above is capable of handling this situation successfully, despite CNNs having no recurrent hidden state. Due to the lack of teacher forcing and high dimensionality of the data, the network learns to encode (and decode) state information within the pixels thus willfully incurring a local reconstruction penalty in order to facilitate sequence completion. The success of the batched sequence learning strategy depends on the nature of the ambiguity.
Just as the state of a dynamical system is not well defined by a single observation, often encoding is facilitated by sequence dilation of the learned mapping which more uniquely defines the trajectory of sequence evolution. A single frame mapping from x i to x i+1 exhibits a dilation of d = 1 and can be naturally extended to d = 2 by channel concatenation to learn a computation of the form [y i ;
with dimension (2C) × H × W ; and so on. 2 This helps to individuate samples and thereby allay the need for contextual sequence information to achieve recall. For example, the alternating sequence ambiguity in {A, B, A, C, A, D} is easily resolved by dilation d = 2 since all training examples, f (AB) → BA, f (BA) → AC and so on, are unique.
The KTH (Schüldt et al., 2004) and DAVIS17 (Pont-Tuset et al., 2017) datasets are used to investigate the methods described above. KTH data provide a simple initial investigative platform with monochromatic videos featuring simple repetitious actions performed by humans (clapping, hand waving, punching, etc) against a static background while the DAVIS17 data features multi-channel RGB capture of generic action scenes across a variety of dynamic environments (surfing, breakdancing, paragliding, etc). See Figure 1 and Figure 2 for encoding examples using a simple autoencoder style network defined in (1) and applied according to eq. (2). Although compression and fidelity are peripheral considerations (see Wu et al. (2018) for full treatment), the examples presented generally yield faithful reproductions according to a multi-scale similarly measure (Wang et al., 2003) and achieve respectable compression ratios of 10x -15x simply by modulating the number of convolutional layers in the CNN. Compression ratios can be further improved by applying mixed precision training techniques (Micikevicius et al., 2017).
A variety of practical engineering considerations arise during implementation. For example, it is possible to analyze sequence loss during post-production and inject a ground truth frame at a particular deterioration threshold to renew the recall process; analogous to the use of key-frames in standard compression algorithms. It is also possible to train additional networks which upscale the base reproduction. This upscaling approach has advantages in controlling network parameter growth (and hence the compression ratio) as two convolution layers each with 32 filters have only half the parameters compared to a single layer of 64 filters. In this way, eq. 2 is written as the composition of two functions
where f θ advances the sequence and the subsequent application of f θ * bolsters fidelity. From this point of view, f θ * performs the role of denoising autoencoder (Vincent et al., 2008) which strives to  Algorithm 1 Pseudo definition of the autoencoder style network used throughout this work and applied as defined in eq. (2). All layers define nconv convolutional filter channels of size 3 × 3. The first and final layers takes C × d number of input and output channels respectively where C is the number of image channels and d is the sequence dilation. Where necessary, stride and padding are used to maintain dimensional equality of input and output tensors. form a 'repaired' output from 'corrupted' input. This allows computational demands during recall to be modulated on a frame-by-frame basis according to platform limitations, real-time constraints, and so on. This application of upscaling is shown in Figure 3.
From here, we can generalize further and leverage the recent advancements in normalizing flows whereby a simple initial density is transformed into a more complex one by applying a sequence of transformations until a desired level of complexity is attained (Tabak & Turner, 2013;Dinh et al., 2014;Jimenez Rezende & Mohamed, 2015;Kingma et al., 2016). The density q K (z) obtained by successively transforming a random variable z 0 with distribution q 0 through a chain of K transformations f k is:
where eq. (4) is shorthand for the composition f M (f M −1 (. . . f 1 (z 0 ))). In this way, the initial density is said to 'flow' through the sequence of mappings. For our purposes, the original sequence X defined in (1) can be seen as just an arbitrary random noise process in some high-dimensional space. Rather than model the process X directly, instead, it can be approximated, to some acceptable level of fidelity, through a chain of transformations over some known latent process, Z = {z 0 , . . . , z N }. That is, given a random process, Z, defined by g, we have
where z j i ≈ x i represents j consecutive transformations applied to z i , and where each f j is a deep neural network defined by Algorithm 1. These transformations are simply a series of bijective mappings (under the given restricted domain and range) and therefore alleviate the aforementioned sequence training challenges associated with realizing computations of the form defined by eq. (2).

Review:
Summary:
This paper introduces convolutional neural networks to encode video sequences. In particular, it exploits an autoencoder style convolutional network to compress videos with certain ratios by modulating the number of convolution layers. Furthermore, it studies an autoregressive model to learn the sequential information over videos.
Strength:
The idea of using convolutional networks to encode the videos is interesting. 
The used autoencoder and autoregressive techniques are promising for video encoding. 
Weakness:
The paper gives me an impression that there are very few works to apply convolutional networks (including autoencoder with autoregressive processing) to encode videos. I cannot see any existing works sharing the same motivation, and the paper does not evaluate any related works for comparison. But it seems the use of autoencoder and autoregressive techniques look very straightforward to me. Please clarify the concern. 
About technical contribution, it is not clear to see if there are any improvement over the conventional autoencoder and autoregressive models. It would be better to make some necessary discussions in the paper. In addition, it seems no descriptions showing how to learn the bijective mappings f_1, ..f_j.
Regarding the evaluation, the paper uses visual results and multi-scale SSIM (mssim) to study the effectiveness of the proposed method on the KTH and DAVIS’17 datasets. While a small number of video frames are used, it would be necessary to evaluate the overall (average) mssim on the whole dataset. 
There are quite a few typos or grammatical problems on sentences such as “...algorithms which encode sequences of video image frames.”, “f is some convolutional neural network…”, “analogous to the use of key-frames in standard compression algorithms.