Recurrent Video Restoration Transformer with Guided Deformable Attention
Abstract
Video restoration aims at restoring multiple high-quality frames from multiple lowquality frames.
Existing video restoration methods generally fall into two extreme cases, i .
e ., they either restore all frames in parallel or restore the video frame by frame in a recurrent way, which would result in different merits and drawbacks.
Typically, the former has the advantage of temporal information fusion.
However, it suffers from large model size and intensive memory consumption; the latter has a relatively small model size as it shares parameters across frames; however, it lacks long-range dependency modeling ability and parallelizability.
In this paper, we attempt to integrate the advantages of the two cases by proposing a recurrent video restoration transformer, namely RVRT.
RVRT processes local neighboring frames in parallel within a globally recurrent framework which can achieve a good trade-off between model size, effectiveness, and efficiency.
Specifically, RVRT divides the video into multiple clips and uses the previously inferred clip feature to estimate the subsequent clip feature.
Within each clip, different frame features are jointly updated with implicit feature aggregation.
Across different clips, the guided deformable attention is designed for clip-to-clip alignment, which predicts multiple relevant locations from the whole inferred clip and aggregates their features by the attention mechanism.
Extensive experiments on video super-resolution, deblurring, and denoising show that the proposed RVRT achieves state-of-the-art performance on benchmark datasets with balanced model size, testing memory and runtime.
The codes are available at https://github.com/JingyunLiang/RVRT .
Introduction
Video restoration, such as video super-resolution, deblurring, and denoising, has become a hot topic in recent years.
It aims to restore a clear and sharp high-quality video from a degraded ( e g ., downsampled, blurred, or noisy) low-quality video [80, 11, 4, 38].
It has wide applications in live streaming [97], video surveillance [49], old film restoration [78], and more.
Parallel methods and recurrent methods have been dominant strategies for solving various video restoration problems.
Typically, those two kinds of methods have their respective merits and demerits.
Parallel methods [2, 25, 80, 72, 36, 63, 99, 27, 35, 4, 38] support distributed deployment and achieve good performance by directly fusing information from multiple frames, but they often have a large model size and consume enormous memory for long-sequence videos.
In the meanwhile, recurrent models [24, 59, 21, 23, 26, 28, 9, 11, 45, 55, 98, 62] reuse the same network block to save parameters and predict the new frame feature based on the previously refined frame feature, but the sequential processing strategy inevitably leads to information loss and noise amplification [14] for long-range dependency modelling and makes it hard to be parallelized.
Considering the advantages and disadvantages of parallel and recurrent methods, in this paper, we propose a recurrent video restoration transformer (RVRT) that takes the best of both worlds.
On the one hand, RVRT introduces the recurrent design into transformer-based models to reduce model parameters and memory usage.
On the other hand, it processes neighboring frames together as a clip to reduce video sequence length and alleviate information loss.
To be specific, we first divide the video into fixed-length video clips.
Then, starting from the first clip, we refine the subsequent clip feature based on the previously inferred clip feature and the old features of the current clip from shallower layers.
Within each clip, different frame features are jointly extracted, implicitly aligned and effectively fused by the self-attention mechanism [77, 51, 39].
Across different clips, information is accumulated clip by clip with a larger hidden state than previous recurrent methods.
To implement the above RVRT model, one big challenge is how to align different video clips when using the previous clip for feature refinement.
Most existing alignment techniques [58, 65, 59, 87, 9, 15, 72, 80, 11, 38] are designed for frame-to-frame alignment.
One possible way to apply them to clip-to-clip alignment is by introducing an extra feature fusion stage after aligning all frame pairs.
Instead, we propose an one-stage video-to-video alignment method named guided deformable attention (GDA).
More specifically, for a reference location in the target clip, we first estimate the coordinates of multiple relevant locations from different frames in the supporting clip under the guidance of optical flow, and then aggregate features of all locations dynamically by the attention mechanism.
GDA has several advantages over previous alignment methods: 1) Compared with optical flow-based warping that only samples one point from one frame [59, 87, 9], GDA benefits from multiple relevant locations sampled from the video clip.
Unlike mutual attention [38], GDA utilizes features from arbitrary locations without suffering from the small receptive field in local attention or the huge computation burden in global attention.
Besides, GDA allows direct attention on non-integer locations with bilinear interpolation.
In contrast to deformable convolution [15, 100, 72, 80, 11, 10] that uses a fixed weight in feature aggregation, GDA generates dynamic weights to aggregate features from different locations.
It also supports arbitrary location numbers and allows for both frame-to-frame and video-to-video alignment without any modification.
Our contributions can be summarized as follows:
• We propose the recurrent video restoration transformer (RVRT) that extracts features of local neighboring frames from one clip in a joint and parallel way, and refines clip features by accumulating information from previous clips and previous layers.
By reducing the video sequence length and transmitting information with a larger hidden state, RVRT alleviates information loss and noise amplification in recurrent networks, and also makes it possible to partially parallelize the model.
• We propose the guided deformable attention (GDA) for one-stage video clip-to-clip alignment.
It dynamically aggregates information of relevant locations from the supporting clip.
• Extensive experiments on eight benchmark datasets show that the proposed model achieves stateof-the-art performance in three challenging video restoration tasks: video super-resolution, video deblurring, and video denoising, with balanced model size, memory usage and runtime.
Related Work 2.1 Video Restoration
Parallel vs. recurrent methods.
Most existing video restoration methods can be classified as parallel or recurrent methods according to their parallelizability.
Parallel methods estimate all frames simultaneously, as the refinement of one frame feature is not dependent on the update of other frame features.
They can be further divided as sliding window-based methods [2, 25, 80, 70, 72, 79, 36, 63, 99, 99, 27, 71, 60, 35] and transformer-based methods [4, 38].
The former kind of methods typically restore merely the center frame from the neighboring frames and are often tested in a sliding window fashion rather than in parallel.
These methods generally consist of four stages: feature extraction, feature alignment, feature fusion, and frame reconstruction.
Particularly, in the feature alignment stage, they often align all frames towards the center frame, which leads to quadratic complexity with respect to video length and is hard to be extended for long-sequence videos.
Instead, the latter kind of method reconstructs all frames at a time based on the transformer architectures.
They jointly extract, align, and fuse features for all frames, achieving significant
C onv2d
R S T B
U p s a m p l e RFR
C onv2d
R S T B
C li p
Layer performance improvements against previous methods.
However, current transformer-based methods are laid up with a huge model size and large memory consumption.
Different from above parallel methods, recurrent methods [24, 59, 21, 23, 85, 26, 28, 9, 11, 45, 55, 98, 62, 44, 5] propagate latent features from one frame to the next frame sequentially, where information of previous frames is accumulated for the restoration of later frames.
Basically, they are composed of three stages: feature extraction, feature propagation and frame reconstruction.
Due to the recurrent nature of feature propagation, recurrent methods suffer from information loss and the inapplicability of distributed deployment.
Alignment in video restoration.
Unlike image restoration that mainly focuses on feature extraction [16, 94–96, 42, 40, 41, 67, 93, 92], how to align multiple highly-related but misaligned frames is another key problem in video restoration.
Traditionally, many methods [43, 30, 2, 48, 68, 3, 87, 9] first estimate the optical flow between neighbouring frames [19, 58, 65] and then conduct image warping for alignment.
Other techniques, such as deformable convolution [15, 100, 72, 80, 11, 4], dynamic filter [29] and mutual attention [38], have also been exploited for implicit feature alignment.
2.2 Vision Transformer
Transformer [77] is the de-facto standard architecture in natural language processing.
Recently, it has been used in dealing with vision problems by viewing pixels or image patches as tokens [8, 18], achieving remarkable performance gains in various computer vision tasks, including image classification [18, 37, 51, 74], object detection [76, 50, 84], semantic segmentation [83, 17, 66], etc .
It also achieves promising results in restoration tasks [13, 39, 81, 44, 4, 38, 20, 22, 7, 90, 47, 73, 6].
In particular, for video restoration, Cao et al .
[4] propose the first transformer model for video SR, while Liang et al .
[38] propose an unified framework for video SR, deblurring and denoising.
We note that some transformer-based works [101, 84] have tried to combine the concept of deformation [15, 100] with the attention mechanism [77].
Zhu et al .
[101] directly predicts the attention weight from the query feature without considering its feature interaction with supporting locations.
Xia et al .
[84] place the supporting points uniformly on the image to make use of global information.
Both above two methods are proposed for recognition tasks such as object detection, which is fundamentally different from video alignment in video restoration.
Lin et al .
[44] use pixel-level or patch-level attention to aggregate information from neighbouring frames under the guidance of optical flow, but it only samples one supporting pixel or patch from one frame, restricting the model from attending to multiple distant locations.
Methodology 3.1 Overall Architecture
Given a low-quality video sequence I LQ ∈ R T × H × W × C , where T , H , W and C are the video length, height, width and channel, respectively, the goal of video restoration is to reconstruct the high-quality video I HQ ∈ R T × sH × sW × C , where s is the scale factor.
To reach this goal, we propose a recurrent video restoration transformer, as illustrated in Fig.
The model consists of three parts: shallow feature extraction, recurrent feature refinement and HQ frame reconstruction.
More specifically, in shallow feature extraction, we first use a convolution layer to extract features from the LQ video.
For deblurring and denoising ( i .
e ., s = 1 ), we additionally add two strided convolution layers to downsample the feature and reduce computation burden in the next layers.
After that, several Residual
Swin Transformer Blocks (RSTBs) [39] are used to extract the shallow feature.
Then, we use recurrent feature refinement modules for temporal correspondence modeling and guided deformable attention for video alignment, which are detailed in Sec. 3.2 and Sec. 3.3, respectively.
Lastly, we add several
RSTBs to generate the final feature and reconstruct the HQ video I RHQ by pixel shuffle layer [61].
For training, the Charbonnier loss [12] L = (cid:112) ∥ I RHQ − I HQ ∥ 2 + ϵ 2 ( ϵ = 10 − 3 ) is used for all tasks.
3.2 Recurrent Feature Refinement
[Figure]
F
[Figure] [Figure]
We stack L recurrent feature refinement modules to refine the video feature by exploiting the temporal correspondence between different frames.
To make a trade-off between recurrent and transformer-based methods, we process N frames locally in parallel on the basis of a globally recurrent framework.
Formally, given the video feature F i ∈ R T × H × W × C from the i -th layer, we first reshape it as a 5-dimensional tensor F i ∈ R TN × N × H × W × C by dividing it into TN video clip features: F i 1 , F i 2 , ..., F i TN ∈ R N × H × W × C .
Each clip feature F it ( 1 ≤ t ≤ TN ) has N neighbouring frame features: F it, 1 , F it, 2 , ..., F it,N ∈ R H × W × C .
To utilize information from neighbouring clips, we align the ( t − 1) -th
clip feature F it − 1 towards the t -th clip based on the optical flow
O it − 1 → t , clip feature F i − 1 t − 1 and clip feature F i − 1 t .
This is formulated as follows:
[Equation]
where GDA is the guided deformable attention and (cid:98) F it −is the aligned clip feature.
The details of GDA will be described in Sec. 3.3.
Similar to recurrent neural networks [59, 9, 11], as shown in Fig.
2, we update the clip feature of each time step as follows:
[Equation]
where F 0 t is the output of the shallow feature extraction module and F 1 t , F 2 t , ..., F i − 1 t are from previous recurrent feature refinement modules.
RFR ( · ) is the recurrent feature refinement module that consists of a convolution layer for feature fusion and several modified residual Swin Transformer blocks ( MRSTBs ) for feature refinement.
In MRSTB , we upgrade the original 2D h × w attention window to the 3D N × h × w attention window, so that every frame in the clip can attend to itself and other frames simultaneously, allowing implicit feature aggregation.
In addition, in order to accumulate information forward and backward in time, we reverse the video sequence for all even recurrent feature refinement modules [24, 11].
The above recurrent feature refinement module is the key component of the proposed RVRT model.
Globally, features of different video clips are propagated in a recurrent way.
Locally, features of different frames are updated jointly in parallel.
For an arbitrary single frame, it can make full use of global information accumulated in time and local information extracted together by the selfattention mechanism.
As we can see, RVRT is a generalization of both recurrent and transformer models.
It becomes a recurrent model when N = 1 or a transformer model when N = T .
This is fundamentally different from previous methods that adopt transformer blocks to replace CNN blocks within a recurrent architecture [78, 44].
It is also different from existing attempts in natural language processing [82, 34].
Attention +
Warping +
CNN
Input
Input Input
Output
F i − 1 t − F i − 1 t
F it −(cid:98)
F it −¯ F i, (1: N ) t −
O i, (1: N ) t − 1 → t o i, (1: N ) t − 1 → t
Q
K
V
3.3 Guided Deformable Attention for Video Alignment
Different from previous frameworks, the proposed RVRT needs to align neighboring related but misaligned video clips, as indicated in Eq.
In this subsection, we propose the guided deformation attention (GDA) for video clip-to-clip alignment.
Given the ( t − 1) -th
clip feature F it − 1 from the i -th layer, our goal is to align F it − 1 towards the t -th clip as a list of features (cid:98) F i, (1: N ) t −= (cid:98) F i, (1) t − 1 , (cid:98) F i, (2) t − 1 , ..., (cid:98) F i, ( N ) t − 1 , where (cid:98) F i, ( n ) t − 1 (1 ≤ n ≤ N ) denotes the aligned clip feature towards the n -th frame feature F it,n of the t -th clip, and (cid:98) F i, ( n ) t − 1 ,n ′ (1 ≤ n ′ ≤ N ) is the aligned frame feature from the n ′ -th frame in the ( t − 1) -th clip to the n -th frame in the t -th clip.
Inspired by optical flow estimation designs [19, 56, 65, 11, 44], we first pre-align F i, ( n ) t − 1 ,n ′ with the optical flow O i, ( n ) t − 1 → t,n ′ as ¯ F i, ( n ) t − 1 ,n ′ =
W ( F i, ( n ) t − 1 ,n ′ , O i, ( n ) t − 1 → t,n ′ ) , where W denotes the warping operation.
For convenience, we summarize the pre-alignments of all “ n ′ -to- n ” ( 1 ≤ n ′ , n ≤ N ) frame pairs between the ( t − 1) -th and t -th video clips as follows:
[Equation]
After that, we predict the optical flow offsets o i, (1: N ) t − 1 → t from the concatenation of F i − 1 t , ¯ F i (1: N ) t −and
O i, (1: N ) t − 1 → t along the channel dimension.
A small convolutional neural network (CNN) with several convolutional layers and ReLU layers is used for prediction.
This is formulated as
[Equation]
where the current misalignment between the t -th clip feature and the warped ( t − 1) -th clip features can reflect the offset required for further alignment.
In practice, we initialize O 1 , (1: N ) t − 1 → t as the optical flows estimated from the LQ input video via SpyNet [58], and predict M offsets for each frame ( NM offsets in total).
The optical flows are updated layer by layer as follows:
[Equation]
where { o i, ( n ) t − 1 → t,n ′ } m denotes the m -th offset in M predictions from the n ′ -th frame to the n -th frame.
Then, for the n -th frame of the t -th clip, we sample its relevant features from the ( t − 1) -th clip feature
F it − 1 according the predicted locations, which are indicated by the sum of optical flow and offsets, i .
e .,
O i, ( n ) t − 1 → t + o i, ( n ) t − 1 → t , according to the chain relationship F it − 1 O i, ( n ) t
[Equation] [Equation]
t
[Equation]
For simplicity, we define the queries Q , keys K and values V as follows:
[Equation]
where Q ∈ R 1 × C is the projected feature from the n -th frame of t -th clip.
K ∈ R NM × C
and
V ∈ R NM × C are the projected features that are bilinearly sampled from NM locations of F i − 1 t − 1 and
F it − 1 , respectively.
P Q ∈ R C × C , P K ∈ R C × C and P V ∈ R
C × C are the projection matrices.
Note that we first project the feature and then do sampling to reduce redundant computation.
Next, similar to the attention mechanism [77], we calculate the attention weights based on the Q and
K from the ( i − 1) -th layer and then compute the aligned feature (cid:98) F i, ( n ) t − 1 as a weighted sum of V from the same i -th layer as follows:
[Equation]
where SoftMax is the softmax operation along the row direction and √ C is a scaling factor.
Lastly, since Eq.
(9) only aggregates information spatially, we add a multi-layer perception ( MLP ) with two fully-connected layers and a GELU activation function between them to enable channel interaction as follows:
[Equation]
where a residual connection is used to stabilize training.
The hidden and output channel numbers of the MLP are RC ( R is the ratio) and C , respectively.
Multi-group multi-head guided deformable attention.
We can divide the channel into several deformable groups and perform the deformable sampling for different groups in parallel.
Besides, in the attention mechanism, we can further divide one deformable group into several attention heads and perform the attention operation separately for different heads.
All groups and heads are concatenated together before channel interaction.
Connection to deformable convolution.
Deformable convolution [15, 100] uses a learned weight for feature aggregation, which can be seen as a special case of GDA, i .
e ., using different projection matrix P V for different locations and then directly averaging the resulting features.
Its parameter number and computation complexity are MC 2 and O ( MC 2 ) , respectively.
In contrast, GDA uses the same projection matrix for all locations but generates dynamic weights to aggregate them.
Its parameter number and computation complexity are (3 + 2 R ) C 2 and O ((3 C + 2 RC + M ) C ) , which are similar to deformable convolution when choosing proper M and R .
Experiments 4.1 Experimental Setup
For shallow feature extraction and HQ frame reconstruction, we use 1 RSTB that has 2 swin transformer layers.
For recurrent feature refinement, we use 4 refinement modules with a clip size of 2, each of which has 2 MRSTBs with 2 modified swin transformer layers.
For both RSTB and
MRSTB, spatial attention window size and head number are 8 × 8 and 6, respectively.
We usechannels for video SR and 192 channels for deblurring and denoising.
In GDA, we use 12 deformable groups and 12 deformable heads with 9 candidate locations.
We empirically project the query to a higher-dimensional space ( e .
g ., 2 C ) because we found it can improve the performance slightly
and the parameter number of GDA is not a bottleneck.
In training, we randomly crop 256 ×
HQ patches and use different video lengths for different datasets: 30 frames for REDS [53],frames for Vimeo-90K [87], and 16 frames for DVD [63], GoPro [54] as well as DAVIS [31].
Adam optimizer [33] with default setting is used to train the model for 600,000 iterations when the batch size is 8.
The learning rate is initialized as 4 × 10 − 4 and deceased with the Cosine Annealing scheme [52].
To stabilize training, we initialize SpyNet [58, 56] with pretrained weights, fix it for the first 30, iterations and reduce its learning rate by 75%.
4.2 Ablation Study
To explore the effectiveness of different components, we conduct ablation studies on REDS [53] for video SR.
For efficiency, we reduce the MRSTB blocks by half and use 12 frames in training.
The impact of clip length.
In RVRT, we divide the video into N -frame clips.
As shown in Table 1, the performance rises when clip length is increased from 1 to 2.
However, the performance saturates when N = 3 , possibly due to large within-clip motions and inaccurate optical flow derivation.
When we directly estimate all optical flows (marked by ∗ ), the PSNR hits 32.21dB. Besides, to compare the temporal modelling ability, we hack the input LQ video (Clip 000 from REDS, 100 frames in total) by manually setting all pixels of the 50-th frame as zeros.
As indicated in Fig.
4, on the one hand,
N = 2 has a smaller performance drop and all its frames still have higher PSNR than N = 1 (equals to a recurrent model) after the attack, showing that RVRT can mitigate the noise amplification from the hacked frame to the rest frames.
One the other hand, the hacked frame of N = 2 has an impact on more neighbouring frames than N = 1 , which means that RVRT can alleviate information loss and utilize more frames than N = 1 for restoration.
The impact of video alignment.
The alignment of video clips plays a key role in our framework.
We compare the proposed clip-to-clip guided deformable attention (GDA) with existing frame-toframe alignment techniques by performing them frame by frame, followed by concatenation and channel reduction.
As we can see from Table 2, GDA outperforms all existing methods when it is used for frame-to-frame alignment (denoted as GDA ∗ ), and leads a further improvement when we aggregate features directly from the whole clip.
The impact of different components in GDA.
We further conduct an ablation study on GDA in
Table 3.
As we can see, the optical flow guidance is critical for the model, leading to a PSNR gain of 1.11dB. The update of optical flow in different layers can further improve the result.
The channel interaction in MLP also plays an important role, since the attention mechanism only aggregates information spatially.
The impact of deformable group and attention head.
We also conduct experiments on different group and head numbers in GDA.
As shown in Table 4, when the deformable group rises, the PSNR first rises and then keeps almost unchanged.
Besides, double attention heads lead to slightly better results at the expense of higher computation, but using too many heads has an adverse impact as the head dimension may be too small.
[Figure]
[Table] [Table]
[Figure] [Figure]
[Table]
[Figure]
[Table] [Table]
4.3 Video Super-Resolution
For video SR, we consider two settings: bicubic (BI) and blur-downsampling (BD) degradation.
For BI degradation, we train the model on two different datasets: REDS [53] and Vimeo-90K [87],
[Table] [Table]
and then test the model on their corresponding testsets: REDS4 and Vimeo-90K-T. We additionally test Vid4 [46] along with Vimeo-90K. For BD degradation, we train it on Vimeo-90K and test it on
Vimeo-90K-T, Vid
[Table]
and UDM10 [89].
The comparisons with existing methods are shown in Table
[Table]
As we can see, RVRT achieves the best performance on REDS4 and Vid4 for both degradations.
Compared with the representative recurrent model BasicVSR++ [11], RVRT improves the PSNR by significant margins of 0.2 ∼ 0.5dB .
Compared with the recent transformer-based model VRT [38],
RVRT outperforms VRT on REDS4 and Vid4 by up to 0.36dB .
The visual comparisons of different methods are shown in Fig.
It is clear that RVRT generates sharp and clear HQ frames, while other methods fail to restore fine textures and details.
We compare the model size, testing memory consumption and runtime of different models in Table 6.
Compared with representative parallel methods EDVR [80], VSRT [4] and VST [38], RVRT achieves significant performance gains with less than at least 50% of model parameters and testing memory usage.
It also reduces the runtime by at least 25% .
Compared the recurrent model BasicVSR++ [11],
RVRT brings a PSNR improvement of 0.26dB. As for the inferiority of testing memory and runtime, we argue that it is mainly because the CNN layers are highly optimized on existing deep learning frameworks.
To prove it, we use the transformer-based RSTB blocks in RVRT to replace the CNN blocks in BasicVSR++, in which case it has similar memory usage and more runtime than our model.
In addition, to better understand how guided deformable attention works, we visualize the predicted offsets on the LQ frames and show the attention weight in Fig.
As we can see, multiple offsets are predicted to select multiple sampled locations in the neighbourhood of the corresponding pixel.
According to the feature similarity between the query feature and the sampled features, features of different locations are aggregated by calculating a dynamic attention weight.
4.4 Video Deblurring
For video deblurring, the model is trained and tested on two different datasets, DVD [63] and
GoPro [54], with their official training/testing splits.
As shown in Table 7 and 8, RVRT shows its
[Figure] [Figure]
superiority over most methods with huge improvements of 1.40 ∼ 2.27dB on two datasets.
Even though the performance gain over VRT is relatively small, RVRT has a smaller model size and much less runtime.
In detail, the model size and runtime of RVRT are 13.6M and 0.3s, while VRT has 18.3M parameters and the runtime of 2.2s on a 1280 × 720 LQ input.
The visual comparison is provided in the supplementary material due to the space limit.
4.5 Video Denoising
For video denoising, we train the model on the training set of DAVIS [31] and test it on its corresponding testset and Set8 [70].
For fairness of comparison, following [70, 71], we train a non-blind additive white Gaussian denoising model for noise level σ ∼ U (0 , 50) .
Similar to the case of video deblurring, there is a huge gap ( 0.60 ∼ 2.37dB ) between RVRT and most methods.
Compared with
VRT, RVRT has slightly better performance on large noise levels, with a smaller model size (12.8M v .
s .18.
M) and less runtime (0.2s v .
s .1.5s)
on a 1280 × 720 LQ input.
The visual comparison is provided in the supplementary material due to the space limit.
Conclusion
In this paper, we proposed a recurrent video restoration transformer with guided deformable attention.
It is a globally recurrent model with locally parallel designs, which benefits from the advantages of both parallel methods and recurrent methods.
We also propose the guided deformable attention
[Table] [Table]
[Table]
module for our special case of video clip-to-clip alignment.
Under the guidance of optical flow, it aggregates information from multiple neighboring locations adaptively with the attention mechanism.
Extensive experiments on video super-resolution, video deblurring
[Table]
and video denoising demonstrated the effectiveness of the proposed method.
Limitations and Societal Impacts
Although RVRT achieves state-of-the-art performance in video restoration
[Table]
it still has some limitations.
For example, the complexity of pre-alignment by optical flow increases quadratically with respect to the clip length.
One possible solution is to develop a video-to-video optical flow estimation model that directly predicts all optical flows.
As for societal impacts, similar to other restoration methods,
RVRT may bring privacy concerns after restoring blurry videos and lead to misjudgments if used for medical diagnosis.
Acknowledgments and Disclosure of Funding
This work was partially supported by the ETH Zurich Fund (OK), a Huawei Technologies Oy (Finland) project, the China Scholarship Council and an Amazon AWS grant.
Special thanks goes to
Yijue Chen.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets? [Yes] • Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Section 4.
and supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4.1 and supplemental material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]