ReFIn: A Refinement Approach for Video Frame Interpolation
Abstract
Video Frame Interpolation is an important video enhancement problem which aims to generate one or multiple frames between consecutive frames in video.
Optical flow-based frame interpolation approaches estimate intermediate optical flow from interpolated frame to input frames and warped frames are fused to generate interpolated frame.
However, intermediate flow estimates can itself be erroneous leading to inaccurate interpolation results.
In this work, we improve an optical flow-based interpolation algorithm, Super-SloMo by residual refinement.
Specifically, we feed intermediate flowmaps, visibility map, warped input frames and intermediate interpolation estimate to a refinement network to predict a frame residual.
We have also experimented with different architecture choices to be used in different modules to further improve the results.
We found out that GridNet with four pyramid levels achieves the best results whereas UNet++ performs moderately well with significantly less number of parameters.
Introduction
Video frame interpolation algorithms generate one or multiple frames between two consecutive frames in a video.
These algorithms have applications in frame-rate upscaling, video compressiondecompression [28], slow-motion video generation, novel view synthesis [7], medical imaging [15, 36] and so on.
Optical flow based interpolation methods aim to first estimate intermediate flow i e. flow between interpolated frame and source frames.
Then the source frames are warped using the estimated intermediate flow.
Finally, the warped frames are used to synthesize the interpolated result.
Liu et al.
[20] directly blend two warped frames.
Niklaus et al.
[23] feed the warped frames and warped context maps to generate interpolated frame.
SuperSloMo [13] blend the warped frames using an estimated visibility map.
In our work, we generate the intermediate frame in two steps.
First, an estimate of intermediate frame is obtained through intermediate flowmap and visibility map estimation similar to [13].
Then, this estimate is further refined using a refinement network.
A frame residual is estimated by the refinement network which is added with the intermediate frame estimate to generate the final interpolated frame.
Our network performs better than multiple state-of-the-art methods on Vimeo-Triplet and UCF101 datasets.
Related Works
Early works in Video Frame Interpolation are often based on optical flow estimation and interpolation accuracy is used to compute quality of optical flow [1, 4].
The rise of optical flow estimation algorithms using Deep Learning has contributed a lot in the progress of Frame Interpolation algorithms [20, 13, 23, 33, 27, 9].
Kernel based frame interpolation algorithms [24, 25, 17] model each pixel in interpolated frame as linear combination of input patches convolved using spatially adaptive kernels.
Phase based frame interpolation algorithms [22, 21] perform phase decomposition of source images to generate intermediate frame.
A few approaches [3, 2] in the literature tries to combine optical flow based and kernel based methods for frame interpolation.
Proposed Method
Given two consecutive frames I 0 and I 1 in a video, video interpolation algorithm predicts I t where t ∈ (0 , 1) .
Our method is based upon a state-of-the-art Video Interpolation algorithm, SuperSloMo [13].
For the sake of completeness, we will describe the complete algorithm in this section.
Our approach is summarized in Figure 1.
Flow Estimation module: First step of our algorithm is to estimate bidirectional optical flowmaps between input frames.
Given two input images I 0 and I 1 , Flow Estimation module will compute bidirectional flowmaps F 0 → 1 and F 1 → 0 .
[13] estimate optical flow in unsupervised manner using a
UNet.
Instead of using UNet for flow estimation as in SuperSloMo, we use pretrained state-of-the-art flow estimator PWCNet [31].
Interpolation module: After computing bidirectional optical flows F 0 → 1 and F 1 → 0 using Flow
Estimation module, we compute approximation of flow from intermediate image to input images, ˆ F t → 0 and ˆ F t → 1 using the following equations [13].
[Equation]
However, this approximation is not true in case of motion boundaries.
Interpolation module acts as a correction step for computing final intermediate flowmaps.
This module generates residual flowmaps to refine the intermediate flowmaps in Equation 1.
Additionally, it computes soft visibility maps required for fusing the warped input frames.
Interpolation network takes in input images
I 0 , I 1 , approximate intermediate flows ˆ F t → 0 and ˆ F t → 1 and input images warped with ˆ F t → 0 and ˆ F t → 1 and predicts flow residuals △ ˆ F t → 0 , △ ˆ F t → 1 and visibility map V t → 0 .
Visibility map V t → 1 is calculated from V t → 0 , since these two visibility maps should complement each other, i.e. V t → 1 = 1 − V t → 0 .
So, the final estimation of intermediate flows are given by,
[Equation]
We linearly blend the warped input images to synthesize predicted frame ˆ I t as,
[Equation]
where bw ( ., . ) is the backward warping function [12].
Refinement Module: Although the formulation of intermediate frame in Equ-3 is compact, the synthesized frame can still have erroneous predictions.
These errors can subject to large motion and occlusion.
Hur et al.
[11] has shown that iterative residual refinement of optical flow predictions can improve optical flow accuracy.
Inspired by this, we introduce one step of refinement.
We compute residual of the predicted frame △ ˆ I t using a Refinement module.
As we already have an approximation of the interpolated frame, we don’t need to generate the image from scratch and compute the residual image only.
Input images ( I 0 , I 1 ) , refined flow maps ( F t → 0 , F t → 1 ), visibility map V t → 0 , warped input images bw ( I 0 , F t → 0 ) , bw ( I 1 , F t → 1 ) and predicted frame ˆ I t are fed into this network.
So, the final synthesized frame is given by,
[Equation]
Architectures used in Interpolation and Refinement modules: We use (a) UNet [29], (b) UNet++ [34], (c) GridNet [8, 23] architecture in interpolation and refinement modules.
We try both 3-level and 4-level Gridnet in our experiments.
We refer GridNet with 3 levels as GridNet-3 and GridNet with 4 levels as GridNet-4 for brevity.
Details of these architectures are discussed in Appendix.
Among the mentioned architectures, GridNet-4 achieves the best performance.
Please refer to Section 4.1 for analysis.
Experiments
Datasets.
We have used the Vimeo-Triplet [33] and UCF101 datasets [30, 20] in our experiments.
Training and Evaluation details.
We use initial learning rate 10 − 4 for both Interpolation and
Refinement modules.
As we use pretrained PWCNet 1 as our Flow Estimation module, initial learning rate for this module is 10 − 6 and we keep it fixed throughout the training.
We gradually decrease the learning rate to 10 − 6 for Interpolation and Refinement modules.
We use Adam optimizer [16] with β 1 = 0 .
9 and β 2 = 0 .
999 and a batch size of 4.
We use train subset of Vimeo-Triplet dataset as our training dataset and evaluate on the test set of the same dataset.
Videos in UCF101 dataset are of lower quality than Vimeo-Triplet sequences.
Hence, for evaluation on test sequences of UCF, we fine-tune our model on our prepared UCF training data for one epoch.
As we only predict the middle frame, t is set to 0 .
5 for both training and evaluation.
We use Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM)
[32] metrics for evaluation on both the datasets.
Loss functions.
Our loss function for stage-1 is given by,
[Equation]
where, L r is Reconstruction ( L 1 ) loss, L p is perceptual loss [14], L w is Warping loss [13] and L s is
Smoothness loss [13].
In our experiments, we use λ r = 204 , λ p = 0 .
005 , λ w = 102 and λ s = 1 .
In stage-2 of training, we finetune the network using only reconstruction loss.
Comparison with other methods.
We compare the performance of our algorithm with the following state-of-the-art models: TOFlow [33], SepConv [25], SuperSloMo [13], CtxSyn [23], CyclicGen [19], MEMC-Net [3], DAIN [2], CAIN [6], BMBC [27] and AdaCoF [18].
For comparison with
SuperSloMo [13], we use unofficial pretrained models 2 since official pretrained models were not released.
For rest of the models, official pretrained models were used for comparison.
Tablesummarizes the quantitative results of state-of-the art models.
Our algorithm performs best among the mentioned methods on Vimeo-Triplet dataset and comparable against other state-of-the-art models in UCF101 dataset.
Note that, we have gained 1.71 dB and 0.87 dB improvement on
PSNR scores in Vimeo-Triplet and UCF101 datasets respectively over SuperSloMo, which we have considered as the base framework.
We provide visual comparisons of interpolated images in Figure 2.
[Table]
Ours (GridNet-4)
[Table]
4.1 Ablation Study
Importance of Refinement Module: To show the significance of the Refinement Module, we train basic SuperSloMo Framework both with and without Refinement module.
Please note, PWCNet is used as Flow Estimation module and UNet is used in Interpolation and Refinement modules in these experiments.
Quantitative comparison is shown in Table 2.
It can be seen in Table 2 that addition of refinement module improves the PSNR metric by 0 .
30 dB on Vimeo-Triplet dataset.
Figure 3 shows significance of Refinement module qualitatively.
We can see that rear light of the car is reconstructed when refinement module is used in Figure 3.
[Table]
Choice of architecture: We have experimented with four architectures in Interpolation and Refinement modules: UNet, UNet++, GridNet-3 and GridNet4.
Quantitative results and number of parameters for each architecture is shown in Table 3.
We observe that the two GridNet architectures perform better than UNet and UNet++ and GridNet-4 performs the best among the discussed architectures.
It is worth mentioning that UNet++ gives good performance considering it has the least number of parameters among the said architectures.
Qualitative comparison between the discussed architectures is shown in Figure 4.
We can see that GridNet-4 as interpolation and refinement module produces superior results.
[Table]
Conclusion
In this work, we adopt a state-of-the-art Video Frame Interpolation framework SuperSloMo [13] and improve interpolation accuracy with the help of a refinement network.
We show this refinement module can help in case of incorrect intermediate flow estimation.
It can also help in enhancing edges and textures in the generated frame.
We have explored a variety of network architectures to be used in Interpolation and Refinement modules and our experiments show that GridNet-4 performs better than the rest.
On the other hand, UNet++ achieves good performance while being parameter-efficient.
A Architecture details
UNet: UNet was first proposed for medical image segmentation [29].
However, UNet has gained popularity in computer vision community and often used in image-to-image translation tasks.
We use 23-layer encoder-decoder architecture used in SuperSloMo [13].
The encoder part consists of 12 convolutional layers and 5 average pooling layers.
The decoder part consists of 10 convolutional layers and 5 bilinear upsampling layers.
In encoder, first two convolutional layers have kernel size of 7 × 7 .
Next two layers in encoder use 5 × 5 kernels.
Bigger kernels in initial layers is an important design choice to handle large motion.
Kernels of size 3 × 3 are used in rest of the network.
Skip connections (feature concatenation) are used from encoder to decoder for restoring the fine grained details in decoding part.
Our UNet architecture is shown in Figure 5.
UNet++:
UNet++ is a Nested UNet architecture which was originally used for medical image segmentation [34, 35].
UNet++ aims to reduce the semantic gap between features from encoder branch and decoder branch using dense skip connections.
We have used 4-level UNet++ architecture instead of 5 levels as used in original paper to reduce the model parameters and computational complexity.
Maxpooling and bilinear upsampling is used for feature downsampling and upsampling respectively.
Our UNet++ architecture has 4,3,2 and 1 Convolutional blocks from top to bottom levels respectively.
Convolutional blocks except the output block has two convolutional layers with 3 ×kernels.
Output block has only one convolutional layer with 1 × 1 kernel size.
Output channel sizes in convolutional blocks from top pyramid level to bottom pyramid level are 32 , 64 , 96 , 128 respectively.
Leaky ReLU is used after each convolutional layer.
Model diagram of UNet++ is shown in Figure 6.
GridNet: GridNet is another encoder-decoder architecture where encoder and decoder blocks are laid out in a grid-like fashion which allows the network to combine features from different scales.
GridNet is a multi-scale architecture, where at each scale we have a number of lateral blocks.
Number of output channels in this lateral blocks are consistent in same scale.
Lateral blocks consists of convolutional layers with PReLU activation [10] and residual connection.
Features from lateral blocks on one level interact with features from other levels using upsampling and downsampling blocks and residual addition.
Upsampling block consists of upscaling layer, convolutional layers and PReLU activation.
Following [23], we use bilinear upsampling instead of using deconvolution in upsampling layer as Transposed convolution produces checkerboard artifacts [26].
Downsampling blocks use strided convolution as downsampling layer along with a convolutional layer and PReLU activation.
We try both 3-level and 4-level Gridnet in our experiments.
We refer GridNet with 3 levels as GridNet3 and GridNet with 4 levels as GridNet-4 for brevity.
Output channel dimensions used in GridNetare 32 , 64 , 96 and GridNet-4 are 32 , 64 , 96 , 128 respectively from top to bottom.
Architectures of
GridNet-3, GridNet-4 and its basic components are shown in Figure 7.
B Edge and texture enhancement using Refinement module
Refinement module can also help in better restoration of edges and textures.
Figure 8 shows edge maps extracted by Canny edge detector [5] for interpolated frames generated when refinement module is used and when not used.
We can observe that use of refinement module produces more clear edge and textures in interpolated image.
C Requirement for Stage-2 of training
We train our model in two stages.
In the first stage, we use Reconstruction loss, Perceptual loss, Warp loss and Smoothness loss.
After the training is converged for first stage, we further fine-tune the whole model with just Reconstruction loss at a lower learning rate ( 1 e − 6 in our experiments).
First stage of training gives us control over what is learnt by each module, whereas in the second stage, we can focus on only improving interpolation accuracy.
The second stage of training helps to improve the performance as supported by our experiments in Table 4.
PSNR is improved by 0.10 dB and 0. dB in case of without and with Refinement module respectively.
[Figure]
Without RefinementModule
With RefinementModule
[Table]
After Stagetraining
[Table]
After Stagetraining
[Table]