U NSUPERVISED S PATIOTEMPORAL D ATA I NPAINTING
Anonymous authors
Paper under double-blind review
A BSTRACT
We tackle the problem of inpainting occluded area in spatiotemporal sequences, such as cloud occluded satellite observations, in an unsupervised manner.
We place ourselves in the setting where there is neither access to paired nor unpaired training data.
We consider several cases in which the underlying information of the observed sequence in certain areas is lost through an observation operator.
In this case, the only available information is provided by the observation of the sequence, the nature of the measurement process and its associated statistics.
We propose an unsupervised-learning framework to retrieve the most probable sequence using a generative adversarial network.
We demonstrate the capacity of our model to exhibit strong reconstruction capacity on several video datasets such as satellite sequences or natural videos.
I NTRODUCTION
We consider the problem of reconstructing missing information from image sequences.
The problem occurs in many different settings and for different types of sequences.
For example, in remote sensing applications, satellite imagery are frequently occluded by meteorological perturbations such as clouds and rains (Singh & Komodakis, 2018).
Recovering missing satellite data is an active research topic.
Approaches range from simple interpolation to sophisticated data assimilation methods.
The latter is often a model-based approach that relies on analytical models of the underlying observed phenomenon (Ubelmann et al., 2015; Sirjacobs et al., 2011; Lguensat et al., 2017).
Model-free data based methods have also been developed such as DINEOF (Alvera-Azcárate, 2011).
Note that for physical observation modeling problems of this type, there is never any direct supervision available.
Another example concerns natural videos.
Here, information can be occluded by moving objects such as fences (Yamashita et al., 2010), raindrops (Qian et al., 2018), persons (Kim et al., 2019), stains on photographic ﬁlms (Tang et al., 2011).
Video and image imputation have given rise to a large body of literature.
Recent Deep Learning (DL) advances have motivated the development of general imputation methods relying on generative models such as GANs (Wang et al., 2018a;
Xu et al., 2019; Kim et al., 2019).
They all make use of supervision and require the availability of a ground truth, which is absent in many real-world problems.
Data driven supervised methods have thus attained impressive results and are able to accurately complete a large missing region.
However, reconstructing the missing information in videos when supervision is unavailable is still an open problem and there have been only a few works exploring this direction.
For example, Newson et al.
(2014) propose a simple but effective method for occlusions in natural videos that replaces occluded parts with information from their neighborhood.
We consider here unsupervised video reconstruction.
We propose a model which can be used on different types of image sequences, physical or natural videos, and for a large variety of occlusion processes.
Our method does not make any assumption on the nature of the image sequence, it does not require any prior knowledge like most methods used for physical images do.
It is especially well suited when the occlusion is complex thus forbidding the use of ad hoc techniques, e.g., the patch method of Newson et al.
The method extends to sequences from ideas recently developed for still images based on generative networks (Bora et al., 2018;
Pajot et al., 2019; Li et al., 2019).
This is up to our knowledge the ﬁrst attempt to solve the problem of unsupervised video completion using general ML methods.
This method is fully data driven and does not use any hand-deﬁned analytical prior on the signal.
Priors on the unobserved signal are directly learned from the data for solving an underlying inverse problem.
The method is then applicable to a large variety of video signals.
Our main contributions are the following:
• We propose a new framework and model for large-scale image sequence inpainting learning, in a fully unsupervised context.
• This model can be used for a variety of image sequences and for different occlusion processes.
• Extensive evaluations are performed on realistic simulated satellite data and on natural videos with different occlusion processes.
M ETHOD 2. P ROBLEM S ETTING
We suppose that there exists an unknown spatiotemporal sequence x ∼ p X , x ∈ R C × T × H × W , where x is a tensor denoting a C -channel sequence composed of T frames of H × W pixels.
We denote x t the t -th frame of the sequence and x t 2 t 1 the subsequence from the t 1 -th to the t 2 -th frame inclusive.
With this notation, x = x T 1 .
We do not have access to the original signal x but only to corrupted observation sequences of this signal y ∼ p Y , y ∈ R C × T × H × W .
Our objective is to reconstruct x from the corresponding observation y .
For example, x can be sea surface temperature (SST) at successive times while image sequence y is SST measurements via IR satellites occluded by moving clouds.
We will suppose that y is obtained from x via a measurement process modeled through a stochastic operator F as follows:
[Equation]
where m ∼ p M is an occlusion mask, generated from a known distribution with the same size as x and with components in { 0 , 1 } , where 0 holds for a masked pixel.
¯ m denotes the complement of m , (cid:12) is the element-wise multiplication, all the masked pixels are supposed to be reset to a constant c which could be 0 or 1 depending on the observation process (see Section 3).
Random variables
X and M are assumed to be independent and F is assumed differentiable w.r.t.
x .
In the following, we will suppose that one can retrieve the mask m directly from the observation y .
This is not very restrictive since in most situations this is easy to do.
We denote T the mask extractor T ( y )
= m .
Our objective is then to recover the sequence x from the observations y and the corresponding binary masks m .
Adopting a probabilistic viewpoint, we want to select a reconstruction x ∗ which is the most plausible under the posterior distribution p X | Y ( ·| y ) .
2. M ODEL
We formulate the problem as ﬁnding the most probable sequence conditioned on observations:
[Equation]
The prior term log p X ( x ) is unknown since we are in an unsupervised setting, while the likelihood log p Y | X
( y | x ) does not lead to analytical or simple computational solution.
To tackle these issues, let us introduce a mapping G : Y (cid:55)→ X , parameterized by a neural network ϕ and associating measurement y to its estimate x .
G will allow us to approximate the underlying distribution of training sequences.
By plugging G ( y ) into Equation 2, the objective becomes:
[Equation]
2. P RIOR HANDLING
Let us ﬁrst handle the prior term in Equation 3.
We want the distribution induced from G ( y ) to be close to p X .
In order to do so, we will use an adversarial approach.
We will build on the ideas introduced in Bora et al.
; Pajot et al.
for still images.
The process is illustrated in Figure 1.
For a given observation y , we want to generate an approximation of the unknown true sequence ˆ x ≡ G ( y ) .
The prior p X being unknown, the only available information source is the observation y and the noise prior p M .
For a given generated signal ˆ x , we compute a corrupted version of ˆ x through
[Figure]
the known mask ˆ m , ˆ y ≡ F (ˆ x , ˆ m ) with ˆ m ∼ p M .
We will train G to make the distributions of y and ˆ y indistinguishable.
In order to succeed, the generator G will have to remove the corruption from y and recover a sample ˆ x from distribution p X .
Generator G will then act as an inpainter conditioned on y .
This will enforce the distribution of the reconstructed sequences ˆ x to be close to the distribution of true ones x and maximize the prior term.
A direct application of the adversarial training idea suggests using a discriminator operating directly on the sequences.
We found out that using an additional discriminator on frames worked better than using a unique one operating on sequences.
We then use two discriminators D s and D f respectively associated with whole sequences and with individual frames to optimize G .
D s separates sequences y and ˆ y .
D f distinguishes real frames y t from fake ones ˆ y t .
The loss function used for training
G, D s , and D f is:
[Equation]
with p G Y ( y ) ≡ E m ∼ p M , x ∼ p G X [ p Y | X , M ( y | x , m )] , corresponding to the distribution of the corrupted sequences ˆ y generated via the measurement operator F .
p G X ( x ) is the distribution of ˆ x induced by G from y , i.e. ˆ x = G ( y ) .
2. L IKELIHOOD H ANDLING
Let us now handle the likelihood term in Equation 3:
[Equation]
This likelihood is maximised when we are able to perfectly reconstruct y from G ( y ) .
One way to ensure this property is to constrain G to directly use y for the non occluded area of the reconstructed image G ( y ) .
This can be easily achieved through the following mapping:
[Equation]
where ϕ is a NN responsible for reconstructing the missing part of y , m = T ( y ) is the mask retrieved from y .
G maps Y to X with the help of mask m to ensure that the network will only generate values for occluded pixel, while keeping all the information from y .
To summarize, optimizing the prior term amounts at training ϕ for inputting the missing pixels while optimizing the likelihood term is simply achieved by copying the non occluded portion of y .
2. T RAINING
G is optimized by descending the prior loss and D s , D f by ascending it.
Sequence discriminator D s focuses on temporal dependence and coherence of pixel changes.
Frame discriminator D f keeps an eye on spatial feature coherence of observation frames.
E XPERIMENTS
We evaluate our model on four datasets, characteristic of different types of image sequences.
The ﬁrst one, SST, is a realistic simulation of satellite observations.
The other three are natural video datasets:
FaceForensics++, KTH, and BAIR, initially respectively used as benchmarks for forgery detection, motion detection, and video prediction.
3. D ATASETS
SST
The Sea Surface Temperature dataset used for the experiments includes 2 subsets of GLOBAL
Sea Physical Analysis and Forecasting Product 1 from E.U. Copernicus Marine Service Information.
This is a monitor system providing simulated but realistic global ocean SST data, which integrates satellite-derived and in situ data by assimilation.
Our dataset is a part of the hourly mean SST, the ﬁnest timescale we have access to.
The data we use is a part of the archive of analysis integrating real-world data.
We retrieved our training-and-validation set and test set respectively from two different marine regions.
Detailed data description and information for accessing the dataset are provided in Appendix A.
FaceForensics++ (Rössler et al., 2019)
This dataset contains 1000 videos of non-occluded face movements on a static background.
It was initially created for forgery detection.
In our case, we extracted the faces from the original unforged videos with face_recognition 2 , thus keeping only the changing component of the videos.
The faces have been cropped and resized to 64 × 64.
KTH (Schuldt et al., 2004)
A human action dataset containing 2391 video clips of 6 human actions.
The videos have been recorded with 25 subjects in different environments.
All frames have been resized to 64 × 64.
BAIR Robot Pushing Dataset (Ebert et al., 2017)
This dataset contains 44374 videos recorded by an one-armed robot.
It pushes objects and changes movement direction in a stochastic manner.
All videos share similar tabletop with static background.
All frames have been resized to 64 × 64.
3. M EASUREMENT P ROCESSES
The above datasets provide ground truth videos without corruption.
In order to generate corrupted observation sequences, we simulate different types of occlusion depending on the nature of the videos.
Each corruption process is deﬁned as a stochastic operator F as in Equation 1 with mask distribution p M .
For a given video one then generates a sequence of random masks, one mask being then associated to each frame of the sequence.
Note that except for the Remove-Pixel corruption process where two successive corruptions are independent, for all processes, the generated corruption sequences are time-dependent: the corruption pattern at time t will depend on the one at time t − 1 .
Cloud This process is speciﬁc for the SST dataset.
It simulates realistically video cloud masks on satellite images.
Cloud masks are simulated using Liquid Water Path (LWP) data (measured in g/m 2 ), which characterizes the total amount of liquid water present in the atmosphere between two points.
The LWP data are generated by PyCLES (Pressel et al., 2015) 3 , a large eddy simulation system.
It simulates the evolution of clouds in time based on a variant of anelastic equations of atmospheric motion.
Collected LWP data record mask videos of clouds.
The binary masks are then obtained by setting the image pixels to 0 when their LWP value is above a threshold.
This produces realistic cloud coverage of the captured regions, see Figure 2a.
Pixels occluded by the mask are set to c = 1 .
Thresholds are selected in the interval 55 to 80 g/m 2 to simulate clouds at different occlusion rates.
Statistics about the occluded area at different thresholds are presented in Table 2a.
For simulating occlusion, for each SST image sequence, we sample randomly a sequence of masks from the LWP dataset to be applied to the SST sequence.
Raindrops This process is a simpliﬁed model of random raindrops between subject and camera, taking into account a blurring effect when raindrops leave traces during exposure.
It generates a set of white bars, each with a random length θ l and a constant width w .
Bars move down at a random speed θ v , starting from a random initial position θ p .
All these values are normalized w.r.t the frame edge length in ]0 , 1[ .
The number of raindrops is pre-deﬁned.
Bars return to the top once completely out of frame, see Figure 2.
Pixels occluded by the mask are reset to c = 1 .
Note that as for Cloud, this is a time-dependent measurement process, meaning that two successive masks are correlated.
Remove-Pixel
This measurement roughly mimics severe damages on vintage ﬁlms.
It masks randomly a ﬁxed proportion p ∈ ]0 , 1[ of pixels at each time step and reset them to c = 0 , see
Figure 2.
Mask for each frame is generated independently regardless the evolution of time.
This is the only time-independent measurement considered here.
Moving-Vertical-Bar This simple measurement operator generates a vertical white bar crossing the sequence, very roughly mimicking a fence or any similar obstacle.
The bar is generated with the following distribution parameters: width θ w , initial position θ p , horizontal constant velocity θ v .
These values are in ]0 , 1[ as for Raindrops.
The moving direction is chosen randomly.
The bar reappears on the opposite side once it reaches the border.
Masked pixels in observation are reset to c = 1 .
This is a time-dependent measurement.
3. B ASELINES
Unsupervised Approaches We use two unsupervised baselines, one adapted for SST and the other one speciﬁc of natural videos.
The former is DINEOF (Alvera-Azcárate, 2011).
This is a state-ofthe-art data-driven completion method in geophysics, and it has been used for SST observations, chlorophyll, salinity etc.
It is a parameter-free interpolation technique based on empirical orthogonal function (EOF).
It adopts an iterative algorithm that calculates at each iteration a truncated decomposition of EOF from known pixels, then replaces the values marked as missing by a reconstruction from calculated EOF.
DINEOF does not make any assumption on the form of missing area and as such could be used for other domains as well and for different types of complex occlusion processes.
However, DINEOF has been developed for remote sensing and does not ensure the coherence between different input channels (e.g. for RGB images).
The other one is Newson et al.
(2014), one of the very few methods for unsupervised natural video inpainting.
It is representative of patch-based approaches and it is still today state-of-the-art for many natural video occlusion processes.
It searches for the nearest neighbors of occluded area using an
Approximate Nearest neighbor (ANN) search with PatchMatch (Barnes et al., 2009) ﬁnding for each occluded pixel the best corresponding non-occluded patch.
The occluded area is then reconstructed by assembling information from these neighbors at multiple scales.
The form of the researched patches is supposed to be rectangular cuboids, e.g. a 5 × 5 × 5 spatiotemporal tensor, which limits its capability to adapt to more complex cases like Cloud, Raindrops, or Remove-Pixel.
Supervised Approaches As already mentioned, there exists several supervised approaches to sequence inpainting (Huang et al., 2016;
Xu et al., 2019; Kim et al., 2019).
In order to evaluate the performance of our unsupervised method w.r.t supervised ones, we compared with two supervised baselines.
As our goal is not to beat state-of-the-art supervised techniques, we used two supervised adaptation of our model, respectively trained using unpaired and paired supervision.
They are described below.
U NPAIRED V ARIANT This is a supervised variant of our model in which we have access to unpaired samples from p X and p Y .
The model is illustrated in Appendix C. Because we have access to clean x data, it is then possible to supervise the approximation ˆ x = G ( y ) by discriminating directly between samples x from the signal distribution and the output of the reconstruction network ˆ x .
P AIRED V ARIANT
Here we have access to corrupted-uncorrupted pairs ( y , x ) from the joint distribution p Y , X .
Given the masked image y , the reconstruction is obtained by regressing y to the associated complete image x using a L 1 loss.
In order to avoid blurry samples, we add an adversarial term in the objective, which helps G to produce realistic samples.
This model is similar to the
Vid2Vid (Wang et al., 2018b) model, except that they rely on optical ﬂow which is not available in our case because of the masked regions.
The model is illustrated in appendix C.
[Figure]
3. N ETWORK A RCHITECTURE AND T RAINING D ETAILS
We use the same networks for all the experiments.
For generator G , we use a ResNet-type selfattention network (Zhang et al., 2019), which is composed of 3D-ResNet blocks and spatial selfattention layers.
Frame discriminator D f is a 2D convolutional NN trained for binary classiﬁcation.
Sequence discriminator D s uses the same structure as D f but with 3D convolutions.
These networks can process sequences of any time length.
See Appendix B for more details about the networks.
Let us now detail the training procedure for each dataset: (a) For SST data, the model is trained on 300 sequences, validated on 66 sequences, and tested on 60 sequences.
Each sequence is composed of 24 frames.
We use SST data degraded by cloud masks at LWP threshold 70 g/m 2 for training since they include sufﬁcient information for both SST and cloud dynamics.
(b) For FaceForensics++, KTH, and BAIR, we pick randomly 5% data for validation, another 5% data for test, and keep the remaining for training.
Sequences are truncated or padded to 35 frames to be able to ﬁt into GPU memory.
For time-independent process Remove-Pixel, we use plain pixel value as feature and directly let the sequence discriminator capture the dynamics.
For time-dependent Raindrops, Moving-Vertical-Bar and Cloud, we further reinforce the sequence discriminator D s to focus on temporal component by extracting inter-frame difference feature since the underlying dynamics reﬂected by this feature is more expressive than plain pixels.
D s will therefore distinguish between φ ≡ [ y 2 − y 1 , . . .
y N − y N − 1 ] and ˆ φ ≡ [ˆ y 2 − ˆ y 1 , .
, ˆ y N − ˆ y N − 1 ] .
We use hinge loss for Equation 4 as in Zhang et al.
Following standard practice, all three networks are trained using Adam optimizer with a learning rate of 1 × 10 − 4 and ( β 1 , β 2 ) =
All networks are initialized with normal distribution with a gain of 0.02.
We apply spectral normalization for all parametric layers.
The experiments were made on one NVIDIA GeForce GTX
TITAN X GPU.
[Table]
*Could not terminate.
3. E VALUATION M ETRICS
Our objective is to ﬁnd the most plausible sequence.
We use as main performance measures of the generated frames, Fréchet Inception Distance (FID, Heusel et al., 2017) and Fréchet Video Distance (FVD, Unterthiner et al., 2018).
Both compare the activation distribution of the generated samples from p G X to the real one sampled from p X .
These distributions are extracted from activation layers of NNs, which are pre-trained respectively on natural image classiﬁcation tasks for FID and video classiﬁcation tasks for FVD.
The two distances are calculated for the whole sequence including occluded and non-occluded region.
Besides FID and FVD, we also evaluate the reconstruction error as a complimentary metric.
We use for that Mean Average Error (MAE), which indicates the absolute deviation from the real data.
MAE is calculated solely within the occluded area
[Table]
R ESULTS 4. C OMPARISON WITH B ASELINES
[Table]
Results for SST Data Table 2a shows the results for SST data with simulated clouds at different occlusion rates.
For most occlusion rates, the generated sequences have an MAE under 0.1°C which is well below the reference baseline (see Table 2a).
They also have good FID and FVD values, which means that they are spatially and temporally realistic (See Figure 2a for examples).
For heavily occluded area, our model can realistically reconstruct the data around the border, the reconstruction near the center of the cloud is of lower quality.
We compare our results in Table 2b at 70% occlusion, with DINEOF, the state-of-the-art agnostic method for image sequence reconstruction in IR images.
The error reduction w.r.t.
DINEOF is about 40% for MAE.
We have not been able to obtain results for Newson et al.
(2014) because the highly complex Cloud masks make searching for valid cuboid patches in both occluded and non-occluded area extremely hard.
Consequently, the algorithm remains blocked in its initialization step and is prevented from the completion.
Note that Newson et al.
(2014) is speciﬁcally designed for inpainting in natural videos so that it is not adapted for this type of data.
Results for Videos Table 1 gathers the results obtained for the three natural video datasets with artiﬁcial measurements (Raindrops, Remove-Pixel, and Moving-Vertical-Bar).
For all measurements, the FID and FVD performance obtained by our model are 20%-50% better than DINEOF.
This means that our model better controls the both spatial and the temporal generation quality than DINEOF.
Globally, we achieve better MAE scores notably for color videos with few exceptions (performance is close for Remove-Pixel).
As for Newson et al.
(2014), the completion could not be ﬁnished for Raindrops and Remove-Pixel masks for the same reason mentioned above caused by highly complex masks breaking cuboid patches.
Newson et al.
(2014) performs better when the form of masks is simple such as Moving-Vertical-Bar, for which matching patches could be easily found in neighbour frames.
However, the computation time of Newson et al.
(2014) is much longer than our model in terms of best performance could be achieved.
Note that reduced computation time was an argument put forward in their publication.
For a 30-frame 64 × 64 video, Newson et al.
(2014) costs on average 1 minute and a GPU speedup is not available, versus around 0.5 seconds by our model, with an end-to-end GPU speedup.
[Table]
Tablecompares our model with the two supervised (unpaired and paired) variants described in Section 3.3.
Unsurprisingly, the performance of supervised models is far better than the ones of our unsupervised model.
We can ﬁnd out that the access to the ground truth reduce dramatically all three metrics.
By using supervision, FID is halved and FVD is between two and three times smaller.
The error reduction is smaller with MAE.
We also notice that the unpaired version performs better than the paired one in terms of sequence completion quality (FVD) as the
L 1 loss introduces a strong constraint for the reconstruction.
This shows to what extent the absence of ground truth will affect generation quality and the extra difﬁculty while dealing with partial observations.
4. A BLATION S TUDY
[Table]
We also conduct additional experiments in order to quantify the importance of the temporal component.
In a ﬁrst series of experiments, we remove the sequence component from our model, i.e. removing the sequence discriminator D s and replacing 3D generator by 2D one generating individually frames.
Tableshows that our model clearly improves temporal quality by reducing FVD by a ratio of 7 compared to the model without the temporal component (denoted Static variant in the table).
Note that FID is also clearly improved by a factor of 4.
this gives more evidence that the model is able to exploit temporal dependency for its image completion task.
We provide samples for this part in Appendix D in
Figure 12.
We further compared our model in Appendix E with Criminisi et al.
(2004) and Ulyanov et al.
(2017), two other state-of-the-art unsupervised image inpainting methods.
The results show the same improvement on temporal quality as mentioned above.
Our model generates a frame at time t , ˆ x t from a whole sequence of observations y .
In a second series of experiments, we conditioned the generation of frames ˆ x t only on past observations.
We feed past observations into a convolutional RNN (we used GRU in our experiments) and generate the reconstructed frame, still denoted G ( y ) by abuse of notation, from the last hidden state of the RNN, which encodes all past observations.
The spatial discriminator operates as before, while the sequence discriminator operates on past observations only, instead of the full sequence of observations in our model.
See Appendix C for an illustration and for further description.
Results in Table 4 - Recurrent variant, show that using only past observations makes the completion less realistic and less accurate, but it still clearly outperforms the model without time dependency.
R ELATED W ORK
There is currently, up to our knowledge, no other learning-based approach trying to solve the problem of spatiotemporal data completion in a purely unsupervised manner.
We will review below related contributions for image and video reconstruction, data assimilation, and domain translation.
Image Reconstruction Video or more generally spatiotemporal sequence completion can be considered as an extension of image completion problems.
the ﬁrst attempts for image completion and inpainting were all supervised.
Xie et al.
(2012) uses convolutional NNs for regressing observations to ground truth images.
This typically produces blurry outputs.
To overcome this issue, some authors introduce textures (Yang et al., 2016), while many others make use of GANs (Pathak et al., 2016;
Yu et al., 2018).
More recently, unsupervised approaches have been developed by considering only corrupted images.
Ulyanov et al.
; Lehtinen et al.
(2018) show that it is possible to learn the underlying data distribution and to reconstruct images from observations when a model of observation process is given or when the noise is zero-mean.
Such restrictive hypothesis have been removed in the seminal work of Bora et al.
They introduce AmbientGAN to unconditionally generate images without supervision from corrupted observations under the assumption that the stochastic measurement process is known.
MisGAN (Li et al., 2019) extend this idea and to learn jointly the mask and the original data distributions.
Both contributions objective is data generation and not completion like we do here.
Pajot et al.
(2019) propose to conditionally recover images from corrupted observations only by solving a maximum a posteriori (MAP) estimation problem, implemented with an adversarial framework.
This is limited to still images.
Video Inpainting Video inpainting has been mainly considered in a supervised framework.
Objectbased (Cheung et al., 2006) and patch-based (Newson et al., 2014) approaches introduced before the deep learning era generally rely on prior segmentation of moving objects and background or strong assumptions on video content.
Flow-based methods have been used to model spatial appearance and local pixel movement between consecutive frames.
Huang et al.
(2016) propose to guide nonparametric patch-based optimization with forward and backward optical ﬂow.
Xu et al.
; Kim et al.
(2019) try to resolve the problem through neural optical ﬂow estimation, which requires extra pre-trained network.
More recently end-to-end learning approaches have been proposed.
For example,
Wang et al.
(2018a)
propose frame-level generation decomposition by combining a video inpainter with a frame-wise reﬁnement inpainter.
Extensions of image inpainting methods are also proposed in
Chang et al.
All these learning-based methods are trained with supervision and have been developed for natural videos.
Data Assimilation for Remote Sensing For remote sensing applications, Optimal Interpolation (OI) is widely used in operational products (Donlon et al., 2012).
It produces a linear estimate for the occluded area.
Model-based assimilation methods (Ubelmann et al., 2015) rely on explicit physical dynamic priors and demand signiﬁcant computational power.
Purely data-driven methods based on empirical orthogonal functions (EOF, Beckers & Rixen, 2003) use basically matrix factorization to achieve temporal interpolation.
Recent advances in Analog Data Assimilation (AnDA, Lguensat et al., 2017; Fablet et al., 2018) combine analog forecasting methods with data-driven assimilation using implicit knowledge of dynamical prior.
These methods rely either on interpolation or exploit some priors on the nature of the underlying process.
Recently, learning methods have started to be exploited in this ﬁeld.
Shibata et al.
(2017) propose to apply learning-based frame-level inpainting enhanced with optical ﬂow using simple assumptions on pixel movement.
In a later paper, Shibata et al.
(2018), they recover the missing data using an adversarial approach to supervise extra occluded area w.r.t the original partial observations.
This approach still reconstruct data frame by frame.
Domain Translation Reconstruction can also be considered as a translation problem between two domains, incomplete observations and full unobserved data.
For images, Pix2Pix (Isola et al., 2016) utilizes GANs to project data from domain A to domain B with paired data.
CycleGAN (Zhu et al., 2017) propose to use two generator-discriminator pairs to model the transformation between two domains.
For videos, Wang et al.
(2018b) propose Vid2Vid by adding a multi-scale temporal discriminator in Pix2Pix to supervise the optical ﬂow.
RecycleGAN (Bansal et al., 2018) is based on the idea of CycleGAN by adding a temporal transformation in both domains.
However, these methods require full data from the two domains, and sometimes the supervision on motion, when no supervision is available in our setting.
C ONCLUSION
We have proposed a GAN-based framework to complete partially observed spatiotemporal data.
Our model utilizes a generator to complete missing pixels in observation sequences with the help of two discriminators classifying real and generated observation sequences.
We show that our model is able to complete spatiotemporal data without ground truth supervision when we have a stochastic model of the occlusion process.
Our results for SST data and natural videos show that the recovered sequences are realistic, especially when the occluded area is highly complex.
A S UPPLEMENTARY I NFORMATION ON SST D ATASET
GLOBAL Sea Physical Analysis and Forecasting Product is a monitor system providing simulated but realistic global ocean SST data.
In analysis engine, it integrates satellite-derived and in situ data by assimilation from OSTIA SST system (OSTIA, The Operational Sea Surface Temperature and
Sea Ice Analysis, Donlon et al., 2012).
The analysis is based on a sophisticated ocean model, NEMO ocean engine (Nucleus for European Modeling of the Ocean, Gurvan et al., 2017), a state-of-the-art modelling framework of ocean related physics.
Oceanic output variables of this product are hourly, daily and monthly means of temperature, salinity, currents, sea surface height, etc.
Our dataset is a subset of global-analysis-forecast-phy-001-024-hourly-t-u-vssh , which includes hourly mean Sea Temperature at sea level height, i e. SST.
The data we use is a part of the archive of analysis integrating real-world data.
Training-and-validation set is hourly SST in 2018 (2018-01-01 00:30 to 2018-12-31 23:30) on a marine region of 64 × 64 pixels (20°-25.25°N, 34.75°-40°W, in North Atlantic Ocean).
Test set corresponds to the data of the ﬁrst 60 days of 2019 (2019-01-01 00:30 to 2019-03-01 23:30) in another region of the same size (14.75°-20°S, 14.75°-
°W, in South Atlantic Ocean).
To obtain the dataset, check out the product via http://marine.copernicus.eu/services-portfolio/access-to-products/ ?option=
com_csw&view=details&product_id=
GLOBAL_ANALYSIS_FORECAST_
PHY_001_024 , then enter the criteria above for each subset.
B A RCHITECTURE D ETAILS
We detail the architecture of networks utilized in our experiments in this section.
Generator Table 5a lists modules for generator G .
It is a ResNet-type self-attention network as in Zhang et al.
It is composed of 3D ResNet blocks as in He et al.
and spatial self-attention layer from Wang et al.
(2018c), which means the attention calculation is limited within each frame.
Discriminators Table 5b is the architecture of 2D or 3D PatchGAN discriminator as in Isola et al.
Frame discriminator D f is a 2D convolutional NN.
Sequence discriminator D s uses the same structure as D f but with 3D convolutions except that the stride is still 1 for temporal dimension, such that the number of frames is not limited.
C D ETAILS OF V ARIANTS OF O UR M ODEL
We describe two supervised variants as supervised baselines in Section 4.1 and a recurrent variant of our model in Section 4.2.
Unpaired Variant, Supervised (Figure 3a)
This variant gives our model access to the distribution p X .
Instead of distinguishing between true observations y and fake ones ˆ y , D f
and D s will discriminate between original sequences x and the output of the generator ˆ x . [Table] [Table] [Table]
[Table]
[Table]
Paired Variant, Supervised (Figure 3b)
This variant gives our model access not only to the distribution p X but also the joint distribution p X , Y by adding a sequence-to-sequence L 1 reconstruction loss.
We use the duo-discriminator setting as our model to prevent generating blurry frames, which is essentially the same techniques in Isola et al.
(2016) and Wang et al.
(2018b).
Recurrent Variant, Unsupervised (Figure 4)
In this variant, we simply insert a convolutional recurrent network cell of any type, ConvGRU in our case (Convolutional Gated Recurrent Unit, Siam et al., 2016), into Pajot et al.
See Appendix C for the illustration and further description.
This cell models temporal dependency in feature maps after encoding the sequence into the space with maximum number of channels.
Instead of discriminating image by image as in Pajot et al.
(2019), we use in this variant the same duo-discriminator setting as our model.
D f distinguishes frame by frame while D s discriminates true sequence clip y tt − L +1 and fake clip ˆ y tt − L +1 , where L is the maximum number of consecutive frames.
D A DDITIONAL S AMPLES
We provide more longer samples from with samples from baselines in this section, see ﬁgures 5, 6, 7, and 8.
We also show extra samples for unsupervised baselines (Figure 10), supervised baselines (Figure 11), ablation study (Figure 12).
E A DDITIONAL C OMPARISON WITH S TATE - OF - THE - ART U NSUPERVISED I MAGE I NPAINTING A PPROACHES
We provide here the additional comparison of results with two state-of-the-art unsupervised inpainting methods, Criminisi et al. (2004) and Ulyanov et al. (2017), for FaceForensics++ dataset with Raindrops noise.
Unsurprisingly, Table 6 shows that our model outperforms these methods, especially in terms of temporal quality indicated by FVD. We show also some samples in Figure 13. [Table] [Figure]
[Figure] [Figure] [Figure] Ours
Criminisi et al.
Ulyanov et al.
Ours
Criminisi et al.
Ulyanov et al. Ours Criminisi et al. Ulyanov et al.