Bridging Implicit and Explicit Geometric Transformations for Single-Image View Synthesis
Anonymous Author(s)
AffiliationAddress email
Abstract
Creating novel views from a single image has achieved tremendous strides withadvanced autoregressive models.
Although recent methods generate high-qualitynovel views, synthesizing with only one explicit or implicit 3D geometry has atrade-off between two objectives that we call the “seesaw” problem: 1) preserving reprojected contents and 2) completing realistic out-of-view regions.
Also,autoregressive models require a considerable computational cost.
In this paper, wepropose a single-image view synthesis framework for mitigating the seesaw problem.
The proposed model is an efficient non-autoregressive model with implicit andexplicit renderers.
Motivated by characteristics that explicit methods well preservereprojected pixels and implicit methods complete realistic out-of-view region, weintroduce a loss function to complement two renderers.
Our loss function promotesthat explicit features improve the reprojected area of implicit features and implicitfeatures improve the out-of-view area of explicit features.
With the proposedarchitecture and loss function, we can alleviate the seesaw problem, outperformingautoregressive-based state-of-the-art methods and generating an image ≈ 100 timesfaster.
We validate the efficiency and effectiveness of our method with experimentson RealEstate10K and ACID datasets.
Introduction
Single-image view synthesis is the task of generating novel view images from a given single image [5,18, 23, 38–40, 47, 50, 54].
It can enable the movement of the camera from a photograph and bring animage to 3D, which are significant for various computer vision applications such as image editing andanimating.
To perform the realistic single-image view synthesis in these applications, we can expectthat the novel view image has to consist of existing objects and unseen new objects from the referenceviewpoint.
Therefore, for high-quality novel views, the following two goals should be considered: 1)preserving 3D transformed seen contents of a single reference image and 2) generating semanticallycompatible pixels for filling the unseen region.
To achieve two goals, explicit and implicit methodshave been proposed.
With the recent success of differentiable geometric transformation methods [2, 31], explicit methods [5, 17, 23, 50, 57] leverage such 3D inductive biases to guide the view synthesis network topreserve 3D transformed contents, and various generative models are applied to complete the unseenregions.
Explicit methods can produce high-quality novel view images in small view changes, wherethe content of the reference viewpoint still occupies a large portion.
However, for large view changes,the image quality is degraded due to a lack of ability to generate pixels of the unseen region.
To dealwith this problem, outpainting with the autoregressive model is exploited to fill unseen regions [39],but generating photo-realistic images remains a challenge for explicit methods.
SynSin
SynSin-6x
PixelSynth
GeoFree
Ours
PSNR on small viewpoint changes
F I D on l a r g e
v
i e w po
i n t c h a ng e s
Better preserve
B e tt e r g e n e r a ti on
Warped image
Input image
SynSin
PixelSynth
GeoFree
Ours : Explicit method : Implicit method 0.063 sec/img 9.39 sec/img 6.22 sec/img 0.056 sec/img 0.063 sec/img
T r a d e - o ff [50] [40] [39]
On the other side, implicit methods [38, 40, 46] less enforce 3D inductive biases and let the modellearn the required 3D geometry for view synthesis.
Based on the powerful autoregressive transformer [10], recent implicit methods learn the 3D geometry from a reference image and cameraparameters.
Implicitly learned 3D geometry allows the model to synthesize diverse and realistic novelview images but fails to preserve the contents of the reference image since they reduce 3D inductivebiases.
To sum up, previous single-image view synthesis methods suffer from a trade-off between twoobjectives: 1) preserve seen contents and 2) generate semantically compatible unseen regions.
Figure 1shows an apparent trade-off that explicit methods
well preserve seen contents with sacrificing thegeneration of unseen regions and vice versa for implicit methods.
Here, we call this trade-off theseesaw problem and emphasize the need for combining solid points of explicit and implicit methods.
Moreover, recent methods often depend on autoregressive models, which generate individual pixelssequentially.
Sequential generation causes too slower view synthesis than non-autoregressive methods,limiting their application areas, such as image animating in real-time.
Therefore, we refocus on a fastand efficient non-autoregressive model for single view synthesis.
In this paper, we present a non-autoregressive framework for alleviating the seesaw problem.
Ourapproach aims to design the architecture and loss functions.
We design two parallel render blockswhich explicitly or implicitly learn geometric transformations from point cloud representations.
To bridge explicit and implicit transformations, we propose a novel loss function that motivatesexplicit features improve seen pixels of implicit features and implicit features improve unseenpixels of explicit features.
Interestingly, we observe that proposed loss makes two renderers embeddiscriminative features and allow the model to use both renderers in a balanced way to create novelviews.
With the proposed architecture and the loss function, we can merge the pros of both explicitand implicit methods, alleviating the seesaw problem.
As a result, our non-autoregressive frameworkcan better preserve seen contents, better complete unseen pixels, and generate images ≈ 100 timesfaster than autoregressive methods.
We validate the efficiency and effectiveness of our frameworkwith experiments on the indoor dataset
RealEstate10K
[58] and the outdoor dataset ACID [23].
Related Works
Novel view synthesis Given multiple images from different viewpoints of a scene, novel viewsynthesis aims to generate novel view images.
Traditionally, multi-view geometry is utilized forsynthesizing novel viewpoints [4, 6, 7, 13, 21, 42, 59].
Recently, deep neural networks have beenused to rendering [15, 28, 29, 32] and several representation for view synthesis such as multi-planeimage [11, 45, 58], point cloud [1], depth [44], radiance field [30, 49, 55] and voxel [25, 33, 43].
ViewNet
Input Image I ref
DepthNet
B a c k - p r
o j e c t
i o n f
D
X w X img
h
e h
i
Decoder output Image I tgt
O v e r l a p
P a t c h
E m b e dd i n g s
Encoder
M i x - FF
N x N
Global Set Attention
ISAB 𝜹 𝒈𝒍𝒐𝒃𝒂𝒍 g global
Local Set Attention
LSA Layer 𝜹 𝒍𝒐𝒄𝒂𝒍𝒂𝒃𝒔 𝜹 𝒍𝒐𝒄𝒂𝒍𝒓𝒆𝒍 g local f N
Camera Pose T
Explicit Renderer
O v e r l a p
P a t c h
E m b e dd i n g s
T r a n s f o r m e r
B l o c k
U p s a m p l e
W a r p i n g
Implicit Renderer 𝛿 pos x M
O v e r l a p
P a t c h
E m b e dd i n g s
T r a n s f o r m e r
B l o c k
U p s a m p l e x M : addition: concatenation
Encoded
Single-image view synthesis is more challenging than general novel view synthesis since a singleinput image is only available [5, 18, 23, 38–40, 47, 50, 54].
Explicit methods directly inject 3Dinductive biases into models.
For example, SynSin [50] uses 3D point cloud features with estimateddepth from the model, projects to novel viewpoints, and refines unseen pixels with recent generativemodels [3].
SynSin works well in small viewpoint changes but degrades in large viewpoint changesdue to the lack of generating unseen pixels.
To deal with this problem, PixelSynth [39] exploits theautoregressive outpainting model [37] with 3D point cloud representation.
Despite using the slowautoregressive model, it cannot generate unseen pixels well.
For an implicit method, Rombach etal .
[40] propose a powerful autoregressive transformer.
By less enforcing 3D inductive biases, thisapproach can generate realistic view synthesis and complete the unseen region without explicit3D geometry.
However, its inference time is long due to the autoregressive model, and it fails topreserve seen contents of a reference image.
We bridge these implicit and explicit methods as anon-autoregressive architecture, which can outperform autoregressive approaches with fast inference.
Transformer for point cloud The transformer and self-attention have brought a breakthrough innatural language processing [8, 48] and computer vision [9].
Inspired by this success, transformerand self-attention networks have been widely applied for point cloud recognition tasks and achievedremarkable performance gain.
Early methods utilize global attention for all of the point clouds,resulting in a large amount of computation and inapplicable for large-scale 3D point cloud [24, 52, 53].
Lee et al .
[20] propose the SetTransformer module suitable for point cloud due to permutationinvariant, which uses inducing point methods and reduces computational complexity from quadraticto linear in the number of elements.
Also, local attention methods is utilized to enable scalability [14,34, 56].
Notably, among local attention methods, Fast Point Transformer [34] which uses voxelhashing-based architecture, achieves both remarkable performance and computational efficiency.
Global attention may dilute important content by excessive noises as most neighbors are less relevant,and local attention may not have sufficient context due to their scope.
Therefore, Our approaches useboth global and local attention to deal with 3D point cloud representation.
Methodology
Given a reference image I ref and a relative camera pose T , the goal of single-image view synthesis isto create a target image I tgt with keeping visible contents of I ref and completing realistic out-of-viewpixels.
To achieve this, we focus on mitigating the seesaw problem between explicit and implicit methods in terms of the network architecture and the loss function.
Figure 2 describes an overview ofour network architecture.
The network consists of two sub-networks, the depth estimation network( DepthNet ) and the view synthesis network ( ViewNet ).
Note that the pre-trained DepthNet generatesdepth map D , which is used for ViewNet to synthesize the photo-realistic I tgt .
3.1 Depth Estimation Network (DepthNet)
We train the depth estimation network for explicit 3D geometry since ground-truth depths are notavailable.
Following Monodepth
[12], our DepthNet is trained in a self-supervised manner frommonocular video sequences.
Because a ground-truth relative pose between images is available,we substitute the pose estimation network with the ground-truth relative pose.
Then, we train thenetwork on reprojection losses and smoothness losses with auto-masking in their work.
After training
DepthNet, we fix it during training ViewNet.
3.2 View Synthesis Network (ViewNet)
We design a simple view synthesis network built on architectural innovations of recent transformermodels.
Specifically, we exploit 3D point cloud representation to consider the relationship betweenthe geometry-aware camera pose information and the input image.
Encoder The encoder aims to extract scene representations from a feature point cloud of a referenceimage.
To deal with point clouds, we design a Global and Local Set Attention (GLSA) block whichsimultaneously extracts overall contexts and detailed semantics.
For efficient input size of transformers, I ref ∈
R H
× W × 3 is encoded into f 0 ∈ R H 4 × W 4 × C by an overlapping patch embedding [51],where C denotes the channel dimension.
Then, the homogeneous coordinates p of a pixel in f 0are mapped into normalized image coordinates X img as X img ( p ) =
K − 1 ↓ p , where K ↓ denotes thecamera intrinsic matrix of f 0 .
Finally, 3D world coordinates of p are calculated with depth map D as
X w ( p ) =
D ( p ) X img ( p ) .
Our encoder architecture is N stacked GLSA block, and i -th GLSA blockreceives f i − 1 , X img and X w and outputs f i with Mix-FFN [51].
Global Set Attention .
We utilize Induced Set Attention Block (ISAB) [20] to extract global setattention between the feature point clouds.
With positional encoder δ global and vector concatenationoperator ⊕ , the global attention of i -th GLSA bock is represented as:
[Equation]
Local Set Attention .
We use a modified Lightweight Self-Attention (LSA) layer [34] for the setattention in r × r local window of each pixel point.
Unlike the decomposing relative position ofvoxels in [34], we decompose the relative position of 3D world coordinates between neighbor pixelsusing normalized image coordinates as:
[Equation]
where q ∈ N ( p ) is a neighbor set of homogeneous coordinates in a r × r window of p .
Withdecomposition in Eq.
2, we can divide the relative positional encoding into an continuous positionalencoding δ abslocal and a discretized positional encoding δ rellocal .Then, the computation procedures forlocal set attention g ilocal of i -th GLSA block is similar to LSA layer as:
[Equation]
where ψ and ϕ are MLP-layers, and S c ( a, b )
= a · b ∥ a ∥∥ b ∥ computes the cosine similarity betweena and b .
As pixel coordinates of p and q are all integer, the encoding of X img ( p ) − X img ( q ) ishashed over r 2 − 1 values, resulting in a space complexity reduction from O ( HW · r 2 · C ) to
O ( HW · C ) + O ( r 2 · C ) .
Rendering Module Given the scene representations of the reference image, the rendering modulelearns 3D transformation from the reference viewpoint to the target viewpoint.
Motivated by ourobservations of implicit and explicit methods, we design an Explicit Renderer(ER) and an Implicit
Renderer(IR) connected in parallel to bypass the seesaw problem.
The structure of the two renderers
Explicit Renderer
Implicit Renderer h
i h e 𝑶 𝑳 𝒕𝒔,𝒊𝒏
𝑳
𝒕𝒔,𝒐𝒖𝒕 grad grad
K, D, T
Out-of-view pixels
[Figure]
is similar; they consist of an overlapping patch embedding, GPT architecture [36] and ResNet blockswith upsampling layers.
Note that the overlapping patch embedding and upsampling layers aredesigned for downsampling and upsampling the input feature with the factor of 4, respectively.
The major difference between the two renderers is how the relative camera pose T is used for thegeometric transformation.
Explicit Renderer (ER) .
Given the rotation matrix R and translation vector t of relative camera pose T ,p can be reprojected to the homogeneous coordinates of target viewpoint p ′ as p ′ =
K ↓ RX w ( p ) + t .
The output of encoder f N is warped by splatting operation [31] with optical flow from p to p ′ .
Then,warped f N goes through the explicit renderer to produce explicit feature map h e .
Implicit Renderer (IR) .
Unlike the explicit renderer, the implicit renderer uses the camera parameteritself.
Instead of embedding 3x4 camera extrinsic matrix, we use independent 7 parameters to embedpose information; Translation vector t and axis-angle notation ( u ∥ u ∥ , θ ) to parameterize rotation matrix
R .
We use a positional encoding layer δ pos to embed these parameters and add them to the input ofthe transformer block.
f N passes through the implicit renderer and outputs implicit feature map h i .
Please refer to the supplementary materials for details to compute the axis-angle notation.
Decoder Two feature maps from ER and IR , which are denoted as h e and h i , are then concatenatedbefore the decoder.
We use a simple CNN-based decoder by gradually upsampling the concatenatedfeature map with four ResNet blocks.
Instead of generating pixels in an auto-regressive manner,we directly predict all pixels in the one-path, resulting in more than 110 times faster than thestate-of-the-art autoregressive methods [38–40] in generating images.
3.3 Loss Design for ViewNet
Following the previous single-image view synthesis methods [39, 50], we also use the ℓ 1 -loss,perceptual loss [35] and adversarial loss to learn the network.
Specifically, we compute ℓ 1 -loss andperceptual loss between I tgt and the ground-truth image I gt at the target viewpoint.
Also, we use theglobal and local discriminators [19] with a Projected GAN [41] structure and a hinge loss [22].
Weobserve that our methods improve the generation performance even through these simple networkstructural innovations.
Furthermore, we introduce a transformation similarity loss L ts to complementtwo output feature maps h e and h i .
Transformation Similarity Loss As an extension of the existing seesaw problem, h e may havebetter discriminability than h i in reprojected regions, conversely, h i has better delineation of out-ofview regions than h e .
Therefore, as shown in Fig. 3, we design the transformation similarity lossbetween h e and h i , expecting that h i learns to keep reprojected image contests, and h e also learnto generate realistic out-of-view pixels.
Specifically, we use a negative cosine similarity function
S c for calculating the similarity between two feature maps, and the transformation similarity loss
L ts = λ in L ts,in + λ out L ts,out is formulated as:
L ts,in =
[Equation]
(cid:80)
[Equation]
O
[Equation]
p
[Equation]
) (cid:88) p
[Equation]
O
[Equation]
detach (
[Equation]
( p
[Equation]
L ts
[Equation]
out =
[Equation]
(cid:80)
[Equation] [Equation]
(cid:88) p
O (
[Equation] [Equation]
c ( detach ( h
[Equation]
( p
[Equation]
h e ( p
[Equation]
(4) where O ( p ) ∈
R H 4 × W 4 denotes an out-of-view mask which is derived from the depth map D andthe relative camera pose T .
Note that, without detach operations, our transformation similarity lossperforms the same as a simple negative cosine similarity loss between two feature maps.
Thus, wedetach gradients back-propagated from L ts,in to h e and gradients from L ts,out to h i , because thedetach operation allows the components of L ts to be applied to the intended area.
Final Learning Objective Taken together, our ViewNet is trained on the weighted sum of a ℓ 1 -loss
L ℓ 1 , a perceptual loss L c , an adversarial loss L adv and a transformation similarity loss L ts .
The totalloss is then L =
L ℓ 1 + λ c
L c + λ adv L adv + L ts .
We fix λ c = 1 and λ adv = 0 .
1 for all experiments.
[Table]
Experimental Results
[Figure]
4.1 Experimental Settings
[Figure]
We now describe experimental settings, and please refer to the supplementary materials for further
[Figure]
details about datasets, baselines, and our network architecture.
Dataset We used two standard datasets, RealEstate10K [58] and ACID [23], which are a collectionof videos mostly captured in indoor and outdoor scenes, respectively.
We divided train and testsequences as in [40].
Baselines To validate the effectiveness of our framework, we compared our method to previoussingle-image view synthesis methods : Tatarchenko et al .
[46], Viewappearance [57], Synsin [50],
InfNat [23], PixelSynth [39], GeoFree [40] and LookOutside [38].
Table 1 briefly shows whethereach method is an explicit, implicit, and autoregressive model.
Compared to previous methods, weuse both explicit and implicit geometric transformations without an autoregressive model.
Evaluation Details Because explicit and implicit methods are respectively advantageous in smallview change and large view change, methods should be evaluated on several sizes of viewpointchanges for a fair comparison.
Therefore, we used a ratio of out-of-view pixels over all pixels toquantify view changes, resulting in three splits are categorized into small (20-40%), medium (40-60%)and large (60-80%).
Since evaluation datasets do not have ground-truth depth maps, we used depthmaps from our pre-trained DepthNet to derive the ratio of out-of-view mask pixels.
Finally, we usedrandomly selected 1,000 image pairs for each test split.
We use PSNR on the small split and FID [16] on the medium and large split as evaluation metrics.
PSNR is a traditional metric for comparing images, which is widely used to evaluate consistency .
Nevertheless, PSNR is a poor metric to verify the image quality on large viewpoint changes [39, 40].
Still, it can be a good metric for evaluating the preservation of reprojected pixels on small viewchanges.
Therefore, we use PSNR on the small split to evaluate the ability to preserve seen contents.
For evaluating images quality of view synthesis, FID is widely used [39, 40, 50].
Especially inthe medium and large split with many out-of-view pixels, FID indicates how well the model fillsout-of-view pixels and generates realistic images.
We use the PSNR and FID of specific splits asevaluation metrics, but we report the PSNR and FID of all splits to show the overall trend.
[Table]
194.49 SynSin [50] 17.
70.19 InfNat [23] (1-
[Figure]
87.13 InfNat [23] (5-step) 15.
67.15 PixelSynth [39] 15.
60.82 GeoFree [40] 14.
55.64 ours 17.
[Figure]
Implementation Details We first resized all images into a resolution of 256 × 256 , and normalized
RGB value following [39, 50].
We trained DepthNet using a batch size 50 for 100 k iterations and
ViewNet using a batch size 32 for 150 k iterations.
Training takes about 3 days on 4 NVIDIA Geforce
RTX 3090 GPUs.
We used an AdamW [27] optimizer (with β 1 = 0 .
5 and β 2 = 0 .
9 ) and appliedweight decay of 0.01.
We first linearly increased the learning rate from 10 − 6 to 3 · 10 − 4 during thefirst 1.5 k steps, and then a cosine-decay learning rate schedule [26] was applied towards zero.
In
ViewNet, we used 8 GLSA blocks with local window size r = 5 and 6 transformer blocks in eachrenderer for all experiments.
4.2 Comparison to Baselines
We now compare our method with the state-of-the-art methods on RealEstate10K and ACID.
Table 2shows quantitative results for both datasets.
The implicit method GeoFree [40] reports a lower
[Table] [Table]
[Table] [Table]
FID in the medium and large split than explicit methods such as SynSin [50] and PixelSynth [39],but its PSNR of the small split is lower.
This shows that previous methods are suffered from theseesaw problem.
However, our method consistently achieves the highest PSNR in the small split onboth datasets, which means our method better preserves reprojected contents than previous methods.
Moreover, our method also achieves the lowest FID in all splits on both datasets, and this demonstratesthat our method generates better quality images with filling compatible pixels regardless of viewchanges.
As observed in [38, 39], we note that SynSin and its variant (i.e., SynSin-6x) often produceentirely gray images, resulting they still performing competitive results in PSNR of the medium andlarge split.
Considering this, our method stably outperforms previous methods in all splits.
Also, qualitative results in Fig.
4 illustrate that the warped regions are well-preserved and invisibleparts are well-completed in our method, whereas explicit methods do not generate realistic images,and an implicit method loses the semantic information of visible contents.
Specifically, GeoFree [40]does not preserve the table in the first sample and the ships floating on the sea in the third sample.
Also, explicit methods [39, 50] either make the entire out-of-view regions in one color or produce aless realistic view than our method.
We confirm that mitigating the seesaw problem by well-bridged explicit and implicit geometrictransformations yields high-quality view synthesis, even acquiring a generation speed of about 110times faster than the previous autoregressive models, as shown in Table 3.
The fast generation ofnovel view images allows our method to be scalable to various real-time applications.
4.3 Ablation Study: Type of Set Attention
We design the global and local set attention block to simultaneously extract overall contexts anddetailed semantics.
Therefore, we conducted an ablation study on RealEstate10K [58] to verify eachattention improves the performance of generating novel views.
Table 5 shows the quantitative resultfor the type of set attention.
Interestingly, our local set attention improves the performance relativelyin large view changes, while our global set attention performs well on small view changes.
From thisresult, we conjecture that local and global set attention are more useful for structural reasoning ofout-of-view regions and 3D scene representation of reprojected regions, respectively.
Also, significantperformance improvement is achieved when both attentions are used.
4.4 Ablation Study: Transformation Similarity Loss
The transformation similarity loss L ts is weighted combination of L ts,in and L ts,out .
To understandthe effect of each component, we conducted ablation studies of transformation similarity loss on the
RealEstate10K dataset.
Table 4 reports the average PSNR and FID of our model by changing variouscomponents of L ts .
Results show that combining with gradient stopping operation, L ts,in , and
L ts,out achieves best results among the five variants.
Also, either using L ts,in or L ts,out improvesthe performance and shows that guiding one renderer from the other renderer with the proposedloss function is effective.
Notably, transformation similarity loss is not practical when the detachoperation is not used.
From this result, it is necessary to selectively guide unseen and seen regions bydetaching the gradient.
We also performed an ablation study on balancing parameter λ in and λ out .
Table 6 illustrates theresults varying weight of L ts .
Results show that the case of λ in = 1 , λ out = 1 performs best.
Asmentioned above, it seems essential to complement each other in a balanced way.
w/o 𝐿 !"
w 𝐿 !"
w 𝐿 !",$% w 𝐿 !",&’!
ℎ !
𝑝 " / ℎ # 𝑝 "
[Figure] [Figure]
ℎ # 𝑝 "
[Figure]
ℎ !
𝑝 " / ℎ # 𝑝 " (
[Figure] [Figure]
in ℎ !
𝑝 " / ℎ # 𝑝 " (d) with L ts,out ℎ !
𝑝 " / ℎ # 𝑝 " (e)
[Figure]
ts
4.5 Dependency Analysis between Implicit and Explicit Renderers
[Figure]
Our proposed architecture exploits the implicit and explicit renderer and mixes their outputs fordecoding view synthesis results.
To understand the dependency between two renderers, we analyzethe norm of output feature maps.
For a spatial position p , the norm ratio of two spatial features|| h e ( p ) ||
2 / ||
h
i ( p ) || 2 can represent how much depends on the explicit feature h e ( p ) compared toimplicit feature
h
i ( p ) .
For example, if the ratio is large, the model depends on the explicit rendererthan the implicit renderer at position p .
We compare histograms of the norm ratio by changing thecomponents of L ts and data splits as shown in Fig.
[Figure]
Figure 5a depicts that using L ts,out and L ts,in tends to makethe model more dependent on explicit and implicit features,respectively, compared to our method trained without L ts .
Furthermore, these tendencies are more apparent in difficultcases (i.e., large split) as shown in Fig.
5c–5d.
From our observations, we conjecture that guiding only a specific rendererimproves the discriminability of that renderer, resulting in themodel depending on the improved renderer.
Surprisingly, themodel trained on combining all components of L ts uses bothrenderers in a balanced way, and there is less bias in normratio even according to data splits as shown in Fig.
5e.
The effectiveness of our transformation similarity loss is confirmed by comparing it to our method that is trained without
L ts .
Figure 5b shows that our model trained without L ts hassome outliers for large view changes despite there being lessbias according to data splits.
We observe these outliers arederived when the model fails to generate realistic out-of-viewregions, especially in challenging settings, such as the network having to create novel views for both indoor and outdoorscenes, as shown in Fig.
We also confirm that our modeltrained with L ts performs well even in extreme cases, informing that L ts improves two renderers to embed discriminativefeatures.
Collectively, L ts improves the discriminability of output features from two renderers andmakes the behavior of the model stable, resulting in alleviating the seesaw problem.
Conclusion
We have introduced a single-image view synthesis framework by bridging explicit and implicitrenderers.
Despite using autoregressive models, previous methods still suffer from the seesawproblem since they use only one explicit or implicit geometric transformation.
Thus, we design twoparallel renderers to mitigate the problem and complement renderers with transformation similarityloss.
Alleviating the seesaw problem allows the network to generate novel view images better thanprevious methods, even with a non-autoregressive structure.
We note that the effectiveness of bridgingtwo renderers can be applied in other tasks, such as extrapolation.
We believe that our work canprompt refocusing on non-autoregressive architecture for single-image view synthesis.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to 465 answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You arestrongly encouraged to include a justification to your answer , either by referencing the appropriate section of 467 your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes]
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A] ?
Please do not modify the questions and only use the provided macros for your answers.
Note that the Checklist 472 section does not count towards the page limit.
In your paper, please delete this instructions block and only keepthe Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] We deal with our limitations in conclusion.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] Potential privacy
concern is discussed in conclusion.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental
results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] In section 4 and a supplementary material, training details are presented.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments
multiple times)?
[No] We observe stable performance from ablation study.
(d) Did you include the total amount of compute and the type of resources used (e
g., type of GPUs,
internal cluster, or cloud provider)?
[Yes] As mention in section 4, we use 4 RTX 3090 GPUs.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Please see supplementary
material.
(b) Did you mention the license of the assets?
[Yes] We use publicly available data for experiments.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
In the
supplementary material, we provide the code.
(d) Did you discuss whether and how consent was obtained from people whose data you’re us-
ing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
informa-
tion or offensive content?
[N
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]