Rethinking Alignment in Video Super-Resolution Transformers
Abstract
The alignment of adjacent frames is considered an essential operation in videosuper-resolution (VSR).
Advanced VSR models, including the latest VSR Transformers, are generally equipped with well-designed alignment modules.
However,the progress of the self-attention mechanism may violate this common sense.
Inthis paper, we rethink the role of alignment in VSR Transformers and make severalcounter-intuitive observations.
Our experiments show that: (i) VSR Transformerscan directly utilize multi-frame information from unaligned videos, and (ii) existingalignment methods are sometimes harmful to VSR Transformers.
These observations indicate that we can further improve the performance of VSR Transformerssimply by removing the alignment module and adopting a larger attention window.
Nevertheless, such designs will dramatically increase the computational burden, and cannot deal with large motions.
Therefore, we propose a new and efficient alignment method called patch alignment, which aligns image patches insteadof pixels.
VSR Transformers equipped with patch alignment could demonstrate state-of-the-art performance on multiple benchmarks.
Our work provides valu- able insights on how multi-frame information is used in VSR and how to selectalignment methods for different networks/datasets.
Introduction
Video Super-Resolution (VSR) could provide more accurate SR results than Single-Image SuperResolution (SISR) by exploiting the complementary sub-pixel information from multiple frames.
Thewell-established paradigms of VSR networks usually include an alignment module to compensate forthe motion of objects between frames.
The alignment module is critical for CNN-based VSR networks,because the locality inductive bias of CNNs only allows them to utilize spatial-close distributedinformation effectively.
Many VSR networks have achieved better performance by introducing moreadvanced alignment methods [42, 4, 38, 23]
Recently, the general paradigm of vision network design has gradually shifted from CNNs to
Transformers [9, 22, 7].
Unlike the locality property of CNNs, the self-attention operation in
Transformers is very efficient for processing elements with spatially long-term distribution.
VSR isno exception: Transformers have also been introduced into VSR for better results [3, 21, 23].
However,these VSR Transformers still retain complex alignment modules.
The ability of Transformers toefficiently process non-local information has not yet been exploited for inter-frame informationprocessing.
In this paper, we rethink the role of the alignment module in VSR Transformers.
We make twoarguments: (i) VSR Transformers can directly utilize multi-frame information from unaligned videos ,and (ii) continuing to use the existing alignment methods will sometimes degrade the performanceof VSR Transformers .
For the first argument, we report that for Transformers using the shiftedwindow mechanism [29, 22], misalignment within a certain range does not affect the SR performance, while alignment only has a positive effect on the pixels beyond that range.
The attribution mapvisualization results [10] also show that Transformers without alignment modules behave similarlyto CNNs with alignment modules.
For the second argument, we quantitatively compare the effectsof various alignment methods.
Using alignment will only have negative effects for pixels withoutlarge misalignment ( e.g. , Vimeo-90K [46]).
We blame this phenomenon on the noise of optical flowand the destruction of sub-pixel information by alignment resampling operations.
Although implicitalignment based on deformable convolution can minimize these negative effects, the additional costof parameters and computation makes this approach no longer advantageous.
According to our findings, we can build an alignment-free VSR Transformer, which requires a largeattention window (>16) to cover misalignment between frames.
However, enlarging the windowsize will lead to higher computational costs, making this approach no longer feasible.
As a betteralternative, we propose a simple yet effective alignment method called Patch Alignment, whichaligns image patches instead of pixels.
It will find the corresponding patches in the supportingframes and compute self-attention among them.
This method uses a simple crop-then-move strategyto compensate for situations where the misalignment exceeds the Transformer’s attention window.
Experiments show that the proposed method achieves state-of-the-art VSR performance with a simpledesign and fewer parameters.
Related Works
VSR networks.
The uniqueness of the VSR task lies in the utilization of inter-frame sub-pixelinformation [28].
Due to the motion of objects between frames, the computation and compensationfor this motion is often the focus of VSR research.
The pipeline of most VSR networks mainlyincludes an alignment module, a feature extraction module, a fusion module, and a reconstructionmodule.
These networks are trained using a large number of low-resolution (LR) and high-resolution(HR) video sequences.
The paradigm of early methods [24, 14, 27, 15, 46, 13, 25] can be summarizedas: first estimate and compensate object motion (using optical flow), then pass through differentfeature extraction [35, 45, 18], fusion [37, 2, 4], and reconstruction modules [36], respectively.
Thesemethods can be further divided into methods for compensating motion in the image space (imagealignment) and that in the feature space (feature alignment) [4, 41].
As Chan et al.
[4] claims, featurealignment can achieve better results because inaccuracy in flow estimation may be mitigated afterfeature extraction.
However, the above methods generally suffer from inaccurate optical flow estimation and defectivealignment operation, resulting in noticeable artifacts in the aligned images [38].
With the invention of
Deformable convolutions [8], CNN networks also have the ability to model geometric transformations.
Deformable convolutions gradually replace explicit warping operations in VSR networks with theirlearnable and flexible properties [42, 50, 38].
EDVR [42] integrates deformable convolution in the
VSR network for the first time and aligns features at multiple scales.
Chan et al.
[5] incorporateoptical flow as a guidance for offsets learning.
The latest BasicVSR++ [6] based on the bidirectionalloop structure also uses the alignment based on deformable convolution.
Differently, our paper aims to study Transformers’ ability to model multi-frame information implicitlywithout using alignment.
Before Transformer, attempts have been made to achieve the same effectin CNN, i.e. , the VSR methods without alignment.
Both 2D [31, 47] and 3D convolutions [39,12, 19, 15] are used to extract correlations among frames.
Similar to Transformers, some methodsuse correspondence calculation to build inter-frame connections without alignment.
For example, MuCAN [20] uses a temporal multi-correspondence aggregation module and a cross-scale non-localcorrespondence aggregation module to replace explicit alignment.
Yi et al.
[49] introduce an improvednon-local operation to avoid the complex alignment procedure.
Generally speaking, models withoutexplicit alignment are less effective or rely on special module design.
However, the Transformersdiscussed in this paper could efficiently handle unaligned frames without additional special design.
Transformer networks.
Transformers [40] are a new type of networks characterized by selfattention operations.
A Transformer is considered to have a parameter-independent global receptivefield within the scope of self-attention.
Dosovitskiy et al.
[9] first introduces Transformer into imagerecognition by projecting large image patches into a sequence of tokens.
Inspired by the success ofvision Transformers, many attempts have been made to employ Transformers for low-level visiontasks [48, 51, 7, 44]
As a representative model, Liang et al. proposes SwinIR [22] that integrates apowerful shifted window strategy and achieves promising results.
Transformer-based models havealso been proposed for VSR.
Cao et al.
[3] employ the spatial-temporal self-attention on multi-frame
[Table]
ea t u r e
E x t r ac t i o n
M F SA
B
M F SA
B
M F SA
B
R ec o n
[Table]
t
[Table]
u c t i o n
N
[Table]
Feature
[Table]
m
[Table]
m
[Table]
/QdePCji1X/kzX/jps1BWx8MPN6bYWZekAiujet+Oyura+sbm4Wt4vbO7t5+6eCwqeNUMWywWMSqHVCNgktsGG4EthOFNAoEtoLRbea3nlBpHstHM07Qj+hA8pAzaqz0cF7slcpuxZ2BLBMvJ2XIUe+Vvrr9mKURSsME1brjuYnxJ1QZzgROi91UY0LZiA6wY6mkEWp/Mrt0Sk6t0idhrGxJQ2bq74kJjbQeR4HtjKgZ6kUvE//zOqkJr/0Jl0lqULL5ojAVxMQke5v0uUJmxNgSyhS3txI2pIoyY8PJQvAWX14mzYuKd1mp3lfLtZs8jgIcwwmcgQdXUIM7qEMDGITwDK/w5oycF+fd+Zi3rjj5zBH8gfP5A6iqjMw
[Figure]
[Figure] [Figure]
P i x e l C o un t s
P r o b a b ili t y
1e5 Iterations
Pre
[Figure]
trained SpyNet 0.
[Figure] [Figure]
[Figure] [Figure] [Figure] M e a n T o t a l V a r i a t i o n   patches to perform implicit motion compensation.
Liang et al.
[21] propose a parallel frame prediction
Transformer to model long-range temporal dependency.
Both the above Transformers consist ofalignment modules.
VSRT [3] uses feature alignment method and VRT [21] employs complexdeformable convolution to perform forward and backward alignment.
In this paper, we show thatthese complex alignment operations are unnecessary or even harmful for VSR Transformers.
Preliminary Settings
In this section, we describe the settings used in our experiments in detail, including the Transformerarchitecture, alignment methods, datasets, metrics and implementation details.
If the readers arealready familiar with the common VSR settings and VSR networks, it is okay to skip this section andgo to Section 4 for experimental results and analysis.
Due to space constraints, we record the detailedsetup of the different experiments in Appendix ??
to facilitate the readers to reproduce our results.
The VSR Transformer architecture.
We first describe the basic VSR Transformer backboneused in this study, which follows the sliding window design.
Transformer based on shifted windowmechanism [22] is proven to be flexible and effective for image processing tasks.
We only makeminimal changes when applying it to VSR to maintain its good performance without the loss ofgenerality.
This model is illustrated in Figure 1.
The VSR Transformer takes a reference frame I t and 2 n adjacent supporting frames { I t − n , . .
, I t − 1 , I t +1 , . . .
, I t + n } as input, and outputs thesuper-resolved reference frame I tSR .
In the beginning, a feature extraction module is used to extractfeatures for the subsequent Transformer.
We use a single 2D convolution layer for feature extraction.
The extracted features are denoted as 2 n + 1 feature maps { X t − n , . . .
, X t , . . .
, X t + n } .
Then N Multi-Frame Self-Attention Blocks (MFSAB) are used as the backbone network.
The
MFSAB is modified from RSTB in SwinIR [22], which contains two LayerNorm [1] layers, amulti-head self-attention layer and a multi-layer perceptron layer.
We mainly modify the selfattention layer to make it suitable for the VSR task.
Given these 2 n + 1 feature maps of the frameswith size H × W × C ( H , W and C are the height, width and feature dimension), the shiftedwindow mechanism first reshapes the feature maps of each frame to HWM 2 × M 2 × C features bypartitioning the input into non-overlapping M × M local windows, where HWM 2 is the total number ofwindows.
We calculate self-attention on the features in the windows corresponding to the positionsin different frames.
Therefore, (2 n + 1) M 2 features are involved in each standard self-attentionoperation, and we concatenate features from different frames to produce a local window feature [Figure]
a r e E rr o r
X ∈ R (2 n +1)
M 2 × C .
In each self-attention layer, the query Q , key K and value V are computed as
Q = XW Q , K = XW K , V = XW V , where W Q , W K ,
W V ∈ R C
× D are weight matrices, and Dis the channel number of projected vectors.
Then, we use Q to query K to generate the attention map
A = softmax ( QK T / √ D + B ) ∈ R (2 n +1)
M 2 × (2 n +1) M 2 , where B is the learnable relative positionalencoding.
This attention map A is then used for the weighted sum of (2 n + 1) M 2 vectors in V .
Themulti-head settings are aligned with SwinIR [22] and ViT [9].
In the final reconstruction module, onlythe features of the reference frame are used to generate the SR result.
We use the pixel-shuffle layer[36] to upsample the feature maps.
Besides sliding window based VSR Transformer, the MFSABcan also be applied to recurrent frameworks [6].
We describe the details in Appendix ?? .
Alignment methods.
Alignment methods follow a long line of research.
To investigate the roleof alignment, we need to implement and compare different alignment methods.
These alignmentmethods can be classified into four types, and their respective representative methods are included inour experiments.
We next describe their characteristics and then describe our implementation.
• Image alignment is the earliest and most intuitive alignment method, which is also the
easiest to use.
Image alignment relies on the explicitly calculated optical flow betweenframes.
According to the estimated inter-frame motion, different frames are aligned by awarping operation.
Following the existing successful experience [46, 4], we use the SpyNet
[34] to estimate the optical flow, and fine-tune the SpyNet simultaneously during training.
The resampling method used is the bilinear (BI) method.
• Feature alignment also estimates the optical flow but performs the warping operation on
the deep features instead of on the images.
The flow estimation module still uses SpyNet,
which is optimized during training.
In addition to the shallow 2D convolution in Figure 1,we add five additional residual blocks [11] to extract deep features.
• Deformable convolution (DC) based methods use learnable dynamic deformable convolution
for alignment.
Almost all state-of-the-art VSR networks use deformable convolution to
perform the alignment.
We employ the flow-guided deformable convolution (FGDC)
alignment used in BasicVSR++ [6] and VRT [21] as the representative method.
• No alignment.
The raw input is processed directly using the VSR Transformer.
Datasets and metrics.
In the VSR literature, the REDS [32] and the Vimeo-90K [46] datasets arethe de- facto
benchmarks. REDS has 270 available video sequences, each containing 100 frames.
Wefollow the common splitting methods and split the data into training (266 sequences) and testing(4 sequences) sets.
Vimeo-90K contains 64,612 and 7,824 video sequences for training and testing,respectively.
Although REDS and Vimeo-90K are widely used benchmarks, these two datasets havedifferent motion conditions.
The motion in the Vimeo-90K dataset is generally small, and movementmagnitudes of 99% pixels are less than 10 (for each clip, we measure the motion of the 4th and the7th frames).
Differently, there are large motions in the REDS dataset.
There are at least 20% pixelsthat have movement magnitudes larger than 10 (for each clip, we measure the motion of the 3rd andthe 5th frames).
The distributions of the movement conditions for these two datasets are shown in Figure 2.
In our experiments, we mainly study × 4 VSR task.
We use bicubic interpolation to produce LR video frames.
Peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are usedfor evaluation.
Implementation.
We implement all these methods using the BasicSR framework [43].
We usethe Charbonnier loss [17] as the training objective.
The Adam optimization [16] method is used fortraining with β 1 = 0 .
9 and β 2 = 0 .
The initial learning rate is set to 4 × 10 − 4 , and a Consine
Annealing scheme [30] is used to decay the learning rate.
The total iteration number is set to 300,000.
[Table]
[Figure]
The mini-batch size is 8, and the LR patch size is 64 × 64 .
The experiments are implemented basedon PyTorch [33] and conducted on NVIDIA A100 GPUs.
The details and configurations for eachexperiment can be found in Appendix ?? .
Rethinking Alignment
In this section, we conduct experiments based on the above preliminary settings.
We use four questionsto guide our rethinking.
We present our experimental results and analysis in each subsection.
4.1 Does alignment always benefit VSR Transformers?
In previous VSR research, we often judge the pros and cons of the method based on the averageperformance on the test set.
But for the VSR Transformers, the performance of alignment inside thelocal window is different from that outside the window because of the limited range of self-attention.
A finer perspective on how different methods perform on different data may help us understand theeffect of alignment.
We investigate the performance difference between the VSR Transformer withand without alignment for different pixel movement conditions.
Figure 4 shows the results tested onthe REDS dataset and Table 1 reports the quantitative results for these experiments.
One can draw the following observations.
First, as can be observed from Figure 4 (a), for pixelswith small movements, VSR Transformer can achieve good results without alignment.
Using imagealignment in these cases brings negative effects.
This range of pixel movement is related to thewindow size used by the VSR Transformer.
As there is no locality inductive bias in the processing ofpixels within a local window, the Transformer can handle misalignment in this range.
Second, as themovement increases, the information required by VSR exceeds the scope of the local windows.
Inthese cases, image alignment can improve performance.
Third, according to Figure 2, the movementof about 70% pixels is less than eight on the REDS test set.
This makes the model without alignmentstill perform better than the image alignment on this test set.
We then increase the size of the shifted window, and conduct the same experiment.
The result isshown in Figure 4 (b).
We can see that VSR Transformer can handle a larger range of unalignedpixels with a larger window size.
This shows that the processing capability of VSR Transformersfor unaligned frames is related to the size of the shifted window, and also implies that this capability
[Figure]
[Table]
mostly relies on the self-attention mechanism.
To investigate whether better alignment methodscan eliminate the negative effects of alignment on small motion pixels, we use feature alignmentto conduct the same experiment, and the result is shown in Figure 4 (c).
As one can see, featurealignment narrows the gap between alignment and no alignment on VSR Transformer, but it still hasnegative effects on the pixels of small motion.
In summary, the VSR Transformer can handle misalignment within a certain range, and usingalignment at this range will bring negative effects.
This range is closely related to the window size ofthe VSR Transformer.
However, alignment is necessary for motions beyond the VSR Transformer’sprocessing range.
4.2 What kinds of flow are better for VSR?
Although using optical flow for alignment can have a negative impact, different flows can also lead todifferences in performance.
This inspires us to think about what kinds of flow are better for VSR?
According to Table 1, one can see that optimizing the flow estimator while training the VSR networkwill bring better results, because the flow estimator at this time learns the optimized flow for VSR[46].
The property of this task-oriented flow can imply how the VSR Transformers use multi-frameinformation.
Our first observation is that VSR Transformers tend to use smooth flow.
The flowestimator SpyNet was pre-trained with the end-point-error (EPE) loss, which does not explicitlyencourage smoothness.
The non-smooth flow will introduce random noise to VSR and lose sub-pixelinformation.
To study this problem, we investigate the change of the estimated flow while training the
VSR Transformer with image alignment using the REDS dataset.
We can see that the flow estimatedby the fine-tuned SpyNet is getting smoother, which is reflected in the decrease of the average totalvariation, as shown in Figure 3.
Smoother flow maintains the relative relationship of adjacent pixelsin the aligned frames, thus facilitating VSR processing.
Although the fine-tuned flow estimator will improve performance, there is still a gap between imagealignment with flow fine-tuning and no alignment on the REDS dataset.
However, we observedifferent results on the Vimeo-90K dataset: image alignment with flow fine-tuning is almost identicalto no alignment.
By observing the distribution of the estimated flow, we are surprised to find that theflow is slowly decreasing to 0 when fine-tuned with image alignment using Vimeo-90K.
This is whythe final result is almost the same as no alignment.
In Figure 5, we show the validation curve of therelated experiments and the histogram distribution of the fine-tuned flow.
As one can see, there is alarge gap between the VSR Transformers with and without alignment in the early stage of training.
When the training reaches 25,000 iterations, the fine-tuned flow gradually shifts towards the zerodirection.
When training after 50,000 iterations, the fine-tuned flow vanishes.
At this time, the VSR
Transformer with alignment is equivalent to the one without alignment.
This phenomenon does notappear on VSR-CNN.
This experiment is instructive.
On the one hand, most of the movements inthe Vimeo-90K dataset are smaller than the Transformer’s window size.
According to Figure 4 (a),alignment is detrimental to training at this situation.
The fine-tuned flow estimator seems aware ofthis knowledge and learns to improve performance by forcing flow value to all zeros.
On the other
[Figure]
hand, the adaptability of the model is surprising.
The model will choose the policy that maximizesthe training objective, even if that policy completely disables part of the network.
We can now answer the question posed by this subsection.
For most VSR Transformers, the smoothflow is better if the alignment is necessary.
For situations with small motions, the flow with all zerosis the best for VSR Transformers.4.3 Does Transformer implicitly track the motion between unaligned frames? HR
LR We already know that VSR Transformers can handle a specific range of misalignment, but do they track these movements implicitly?
Can an alignmentlike function be done inside the VSR Transformers?
We next use an interpretability tool to visualize the behaviour of the VSR Transformer.
Local
Attribution Map (LAM) [10] is an attribution method aiming to find inputpixels strongly influencing network output.
We first specify a target patch onthe output image and then use LAM to generate the corresponding attributionmaps.
We will track the information used by the model, and see which partsof pixels in adjacent frames contribute the most.
As objects move betweenframes, an ideal VSR network needs to track those movements and utilizepixels representing the same object.
We show some representative results in Figure 6.
It can be observed that even without the alignment module, the VSR Transformer can automatically change its attention to the most relevant pixels.
Intuitively, the self-attention operation can detect the associations between pixels in different framesand utilize their additional information for VSR.
This is similar to VSR CNNs with the alignmentmodule.
We also show results of VSR CNNs without the alignment module.
Since CNN has theinductive bias of locality, it can only focus on the area near the current position if no alignment isused.
These areas are less valuable to VSR.
Thus, alignment is an essential module for CNN-based
VSR methods.
4.4 Why do alignment methods have negative effects?
To understand the reasons for the negative impact of alignment, we need to know what sub-pixelinformation does VSR require.
As shown in Figure 10, high-frequency information in HR framesis lost during downsampling, and only aliasing patterns are left in the LR frames.
When the HRframes move, different LR frames are generated, and different aliasing patterns are produced.
Thesepatterns provide additional constraints for VSR.
However, the inaccurate optical flow and the bilinearresampling operation could corrupt these patterns.
First, the inaccurate flow can be viewed as acombination of the ground-truth flow and a random error term.
Using such flow for alignmentrandomly will change the LR patterns and cause information loss.
Second, the bilinear resamplingoperation calculates the weighted average among four adjacent pixels, and the weights are inaccessiblefor the VSR model.
At this time, the interpolation is irreversible.
The VSR model can only process [Table] [Table]
the transformed LR patterns and cannot access the original patterns, resulting in information loss.
We conduct experiments to demonstrate the negative effects of optical flow and resampling methods.
The results are shown in Table 2.
Compared with image alignment, feature alignment improvesperformance by extracting part of the sub-pixel information before it is corrupted by alignment.
Theflow-guided deformable convolution (FGDC) reduces the negative effects of alignment by enablingthe network to model geometric transformations.
Changing the resampling method to nearest neighbor(NN) also improves performance because the NN method can preserve the relationship betweenadjacent pixels and ignore the noise of flow estimation to a certain extent.
As can be seen, featurealignment using the NN resampling method achieves the same performance as the FGDC method butwith a significantly reduced number of parameters.
Patch Alignment
According to our observations, an inaccurate flow and the resampling operation will impair theutilization of the inter-frame information.
To overcome this problem, simply increasing the Transformer’s window size is a straightforward solution as Transformers are naturally good at modellingunaligned spatial dependencies within the local window.
However, the time and space complexity ofincreasing the window size is at least O ( n 2 ) .
We need more efficient alignment methods to improveperformance for large movement pixels while introducing little negative impact and additional computational complexity.
In this section, we propose a simple yet effective alignment method for VSR
Transformers, called Patch Alignment.
5.1 Method
The pipeline of the proposed method is shown in Figure 7.
This method does not align individualpixels but treats the image as non-overlapping patches.
The partition of patches is consistent withthe partition of the Transformer’s local windows.
We treat the patch as a whole and perform thesame operation on the pixels within the patch. In this way, the relative relationship between pixelsis kept, and resampling operations will not corrupt the sub-pixel information within the patch.
Welocate the movement of objects based on optical flow, yet we do NOT pursue precise pixel-levelalignment.
We calculate the mean motion vector in each patch and find the corresponding patches inthe supporting frames for each patch.
Next, we use the nearest neighbor resampling method to movethe entire supporting patches to their corresponding position in the reference frame.
The benefits arethree-fold.
First, the nearest neighbor resampling ignores the fractional part of optical flow estimationand reduces the error caused by inaccurate flow estimation.
Second, the entire patch is cropped andmoved to the corresponding position, which preserves the relative relationship of the pixels withinthe patch and thus retains the sub-pixel information.
We show the comparison of image alignment with bilinear resampling and the proposed patchalignment method in Figure 8.
As can be seen, image alignment introduces blurry and artifacts tothe aligned image that destroy sub-pixel information.
Patch alignment retains more details that canprovide additional information for the VSR model.
As we do not pursue pixel-level alignment, directlyoperating on patches will leave discontinuous artifacts along patch borders.
But our experimentsshow that these discontinuities have little effect on VSR Transformers.
Because these discontinuities
[Table]
do not appear in the local window of the Transformer, they do not affect the function of self-attention.
It further illustrates the importance of preserving sub-pixel information.
5.2 Experimental results
We test the patch alignment method with different alignment positions and resampling methods.
Theexperimental settings and the network configurations are the same with experiments in Table 2.
Theresults are shown in Table 3.
As can be seen, image-level patch alignment is already comparableto FGDC, while the latter uses 25% more parameters.
Feature-level patch alignment achieves thebest performance among all the tested alignment methods.
We conduct an ablation experiment usingbilinear resampling to study the importance of NN resampling for patch alignment.
It can be seenthat bilinear resampling leads to a severe performance drop.
It shows that the LR patterns retained bythe NN method are critical for VSR.
We use patch alignment to build VSR Transformers based on the sliding window and the recurrentframeworks, namely PSRT-sliding and PSRT-recurrent.
We test their performances on REDS and Vimeo-90K datasets. The results are shown in Table 4.
VSR Transformers with patch alignment canachieve state-of-the-art performance with fewer parameters compared to other Transformer-based
VSR methods (VSRT [3] and VRT [21]).
Compared to BasicVSR++ [6], we use fewer frames intraining, but achieve a 0.33dB improvement on REDS in terms of PSNR.
Visual results of differentmethods are shown in Figure 9.
As one can see, in accordance with its significant quantitativeimprovements, the proposed method can generate visually pleasing images with sharp edges and finedetails.
By contrast, its competitors suffer from either blurry or lost details.
Conclusion
In this paper, we present two important conclusions about using Transformers in the VSR task:
(i) VSR Transformers can directly utilize multi-frame information from unaligned videos, and(ii) existing alignment methods are sometimes harmful to VSR Transformers.
We also propose
a new patch alignment method for VSR Transformers.
The proposed method demonstrates thestate-of-the-art performance for VSR.
Given the current literature, our results are interesting and inspiring.
They challenge our commonunderstanding of using Transformers to process multiple spatially misaligned images.
First, theanalysis of alignment can provide useful insights for VSR.
We need to utilize inter-frame sub-pixelinformation, yet many image pre-process operations interfere with our utilization of this information.
Second, these observations hint that Transformer can implicitly make accurate connections formisaligned pixels.
Many low-level vision tasks can take advantage of this property, such as videorestoration, reference-based SR, burst image processing, stereo matching, flow estimation, etc .
Whendesigning Transformers for these tasks, we can no longer explicitly introduce the alignment modulesor the cost volume modules, but give full play to the powerful modeling capabilities of the Transformeritself.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots
, ifapplicable?
[N/A]
[Table]
[Table]
[Table]