Paper:
Extracting underlying dynamics of objects in image sequences is one of the challenging problems in computer vision. On the other hand, dynamic mode decomposition (DMD) has recently attracted attention as a way of obtaining modal representations of nonlinear dynamics from (general multivariate time-series) data without explicit prior knowledge about the dynamics. In this paper, we propose a convolutional autoencoder based DMD (CAE-DMD) that is an extended DMD (EDMD) approach, to extract underlying dynamics in videos. To this end, we develop a modified CAE model by incorporating DMD on the encoder, which gives a more meaningful compressed representation of input image sequences. On the reconstruction side, a decoder is used to minimize the reconstruction error after applying the DMD, which in result gives an accurate reconstruction of inputs. We empirically investigated the performance of CAE-DMD in two applications: background/foreground extraction and video classification, on publicly available datasets.

Review:
I found the topic of this paper interesting and I believe I understand what the authors are trying to achieve but I'm afraid this was after several readings and I do think the paper could be presented differently that would make it more accessible. My suggestion would be to explain how the model will be applied first (identify the required properties) to motivate the need for the learned basis and then present the DMD as a method for providing a basis that meets the properties required. I acknowledge that different communities have different styles of presentation so apologies if this is just me.
First I would just like to check that I have understood correctly so please could the authors point out if I have missed something or misunderstood in the following?
Our goal is to establish a basis invariant to the video dynamics that can then be used, for example, to partition the video into parts with differing dynamics - e.g. foreground/background. To do this we need to identify such a basis from a specific video - we will use the collection of pairs of neighboring frames.
The Koopman operator acts on a differential system to identify a function space invariant to the dynamics. If we instantiate this with a finite number of dimensions we can essentially establish the invariance as an eigenvalue problem. From this and our pairs of successive videos we can establish a vector basis for the space and then project the video into this basis. The spectral properties of the coefficients of the projection will determine whether something is static (omega = 0) or transitory in the scene and these can be used to identify foreground and background.
Next there is the issue that this method operates in a linear domain with something like Gaussian noise which is not a good fit for image space videos so the authors propose to identify the dynamics in a linear latent space determined by an autoencoder to handle the non-linear mapping to image space.
I hope I have understood the main points?
If this is the case, I think that much more needs to be said about the second part, which is the essential novelty of the paper, with a discussion of the merits of different approaches and full details - at the moment there is just one small paragraph at the end of 4.2 which contains the majority of the contribution.
My main concern about the paper is that I find it very difficult to appreciate the efficacy of the method given the current presentation of the results. There are no error bars to ascertain significance for any of the results and the summarization of multiple experiments to a single percentage gives very little insight into where this method works and where it doesn't. There are a number of ways that a dynamic prior could be added to a latent space and it is unclear why we would expect this approach to be preferred given the evidence presented in the paper.
Other Notes:
I found that the notation is not always consistent and sometimes could be simplified - it is unclear whether some operators are convolutions or multiplications (vector or scalar). To me the asterisk does not represent straight forward multiplication but it might be being used for this?
Could Table 1 be placed in the experiments section rather than in the middle of page 5?
Do the authors mean half the number of pixels or half the edge size (e.g. a quarter of the area) in terms of the latent space?
Please can all equations be numbered so that they can be referred to - there are no equation numbers in all of section 2.