Neural Transmitted Radiance Fields
Anonymous Author(s)
AffiliationAddress email
Abstract
Neural radiance fields (NeRF) has brought tremendous progress to novel viewsynthesis.
Though NeRF enables the rendering of subtle details in a scene bylearning from a dense set of images, it also reconstructs the undesired reflectionswhen we capture images through glass.
As a commonly observed interference, thereflection would undermine the visibility of the desired transmitted scene behindglass by occluding the transmitted light rays.
In this paper, we aim at addressingthe problem of rendering novel transmitted views given a set of reflection-corruptedimages.
By introducing the transmission encoder and motion inconsistency as guidance, our neural transmitted radiance fields can resist such reflection interferenceduring rendering and reconstruct high-fidelity results even under sparse views.
Theproposed method achieves superior performance from the experiments on a newlycollected dataset compared with state-of-the-art methods
Introduction
Novel view synthesis plays a vital role in various computer vision applications.
Tremendous progresshas been witnessed recently with the introduction of neural radiance fields (NeRF) [1].
By modelingthe emitted radiance and density of a scene from a set of images with a multilayer perceptron (MLP),it demonstrates an unprecedented level of fidelity on a range of challenging scenes.
Despite its successin diverse scenarios, such a framework still faces challenges when working with glass, a mediumwhich is frequently observed in photography and daily life [2] to provide see-through protection fordesired scenes.
However, glass’s reflectivity may reflect undesired scenes on images captured throughit.
People always feel interested in the transmitted scenes behind the glass and want to remove theundesired reflection.
This traps neural radiance fields into a dilemma:
Its well-designed framework iscapable of modeling every detail in a scene, including those undesired ones.
One plausible way to resolve this dilemma is opting for a post-processing tool to remove the undesiredcomponents before the captured images are fed into NeRF [1].
For example, existing reflectionremoval methods [3, 4, 5] can be adopted to suppress the undesired components and provide bettervisibility for transmitted scenes.
However, the processed images from different views may violatethe assumption of NeRF that the scene is photometrically static.
Besides, the reflection cannot becompletely removed in most cases by existing single-image-based reflection removal approaches [6],and the remaining reflection may be modeled by NeRF as a part of the scenes, undermining thequality of rendered results.
An alternative solution is to render with undesired components like NeRF-W [7] by separating thewhole scene into static and transient elements with exclusive properties.
However, the reflectionand transmission usually present a highly “twisted” relationship since they may influence each otherduring light transportation and reflect such entanglement in captured images.
From results shownin Figure 1(left), it may challenge the applicability of a purely unsupervised strategy in NeRF-W [7],thus showing degraded performance.
Besides, NeRF-W [7] also partially relies on a dense set ofmulti-view images to conduct the separation.
As an impractical demand inherited from NeRF [1],
NeRF, 12 views NeRF, 6 views
NeRF-W, 12 views NeRF-W, 6 views
Ours, 6 views
NeRF, 18 views
Warp
A ft e r w a r p
i
ng
R e c u
rr i ng edge s
S pa r s e pa tt e r n
[Figure]
Examples obtained
[Figure] and [Figure] method it contradicts the casual photographing experience, where only limited inputs with sparse viewsare available.
The results in Figure 1(left) illustrate that NeRF-W [7] cannot perform high-fidelitysynthesis with limited views, let alone effective separation between the transmitted scene and thereflected scene.
Thus, how to render novel transmitted scenes with sparse views poses uniquechallenges.
In contrast to the exclusive properties required by NeRF-W [7], we employ motion inconsistency,a phenomenon verified by pioneering reflection removal methods [8, 9], to link the above issuesby warping images with different viewpoints into a predefined target view.
As shown in Figure 1(right), given an aligned image sequence, the transmitted scenes appear in different views with a largeoverlapping area while the reflected scenes only have a sparse presence [8].
Besides, the alignedimage features from the neighboring views can also be used for a more high-fidelity reconstruction ofthe target view.
We, therefore, design a method supported by two pillars: 1) Edges of the repeatedpatterns, denoted as the recurring edges shown in Figure 1(right), that could be estimated as a pilot toguide the differentiation during rendering; 2) features from reference views, which are incorporatedinto rendering for high-fidelity reconstruction under sparse views.
Several modifications have to be conducted to accommodate the above designs.
Since the transmission/reflection edges only show their significance in a certain area, the classical “pixel-wise”rendering framework cannot capture such internal physical properties.
We employ a patch-basedrendering framework, which aggregates the pixels from a certain area into a patch as the training batch.
Besides, due to the entanglement between the transmission and reflection, the features extracted bya network pre-trained for general tasks [10] may introduce interference from feature level, whichdeteriorates the transmission/reflection disambiguation described before.
We, therefore, propose atransmission encoder to penetrate the feature-level interference caused by the reflection and renderthe transmission structure more accurately.
At last, since the strength of reflection tends to rely onthe viewing direction, the weighting coefficients for the two layers are obtained from the viewingdirection to combine transmission and reflection layers.
In summary, the above considerations lead to our Neural Transmitted Radiance Fields as depictedin Figure 2, and its main contribution can be concluded as follows:
• An approach to render novel transmitted views, which is capable of penetrating the reflection interference.
• A unified consideration derived from motion inconsistency for removing reflec-
tion/transmission ambiguity and increasing rendering fidelity of transmission.
• A dataset collected from the real world for evaluation.
Reflection embedding
Glass
V i e w B
V i e w A
V i e w C …
A B
C …
R-MLP T-MLP
Transmission feature encoder
Transmission feature integration
Recurring edge estimation
A B
C …
Image projection
Patch-based rendering
Position Viewing direction
E x t r a c t ed r e c u rr i ng edge s Reflectionscene
Aligned images
Image edges
Virtual images
Transmission features
See-through scenarios
Captured images 𝐂 𝐓 𝐂 𝐑 𝐆($) …
ℒ #$%#
Next, a transmission feature encoder extracts the transmission features from the reference images (transmission feature integration).
The extracted transmission features are then passed into the transmission MLP (T-MLP) for transmission rendering ( C T ).
For the reflection rendering ( C R ), we feed the viewing direction and corresponding reflection embeddings into the reflection MLP (R-MLP).
The recurring edges repeatedly appearing in the aligned image sequence are extracted to further differentiate the transmission and reflection based on the edge loss L edge (recurring edges estimation).
Related work2.1 Novel view synthesis with NeRF
Novel view synthesis aims at generating images from another viewpoint using a set of establishedviews.
The classical approaches rely on the explicit representations like multi-plane images [11, 12],meshes [13], or point cloud [14] to render novel views.
Despite their rendering efficiency at testtime, they usually suffer from the limited expressiveness within their representation.
In contrast tothe explicit representations, Neural Radiance Fields (NeRF) [1] utilize the implicit representation torepresent 3D scenes using neural networks.
For example, NeRF [1] approximates a continuous 3D MLP by mapping an input 5D coordinate to its corresponding scene properties.
Given its promisingrepresentation ability, NeRF [1] has been extended to solving problems like relighting and sceneediting.
Recently, NeRF-W [7] proposes to adapt the original NeRF [1] to wild scenes by introducingan additional MLP to model the undesired transient element, which grants the original NeRF [1]extended ability to model the scenes with photometric variations and transient components.
Since NeRF requires a large number of images for training, several methods [10, 15] are proposed to makeit feasible with sparse views by utilizing the feature information from neighboring views.
In thiswork, we aim to extend NeRF to see-through scenarios by eliminating negative interference causedby reflection.
2.2 Transmission recovery from reflection-corrupted images
How to recover the transmitted scene from reflection-corrupted images has been discussed fordecades.
Early methods [16, 17] make use of the edge difference to reduce the ambiguity betweenthe transmission and reflection through optimization.
Recent methods mainly adopt the deep learningframework to solve this problem.
For example, Fan et al.
[18] proposed a two-stage deep learningapproach to learn the mapping between the mixture images and the estimated clean images.
Wan etal.
[19, 20] proposed a cooperative model to better preserve the background details.
Zhang et al.
[21]proposed a perceptual reflection removal method based on the generative adversarial network.
Wei etal.
[5] proposed another method to solve the reflection removal problem with non-aligned imagepairs.
Besides, some recently proposed [4] methods also try to focus on the reflection-corruptedregions by localizing the reflection first.
The recently proposed method [22] also removed reflectionsunder the guidance of an additional polarizer.
Instead of solely relying on single images, plenty ofmethods [8, 23, 24, 25] also leverage advantages from the priors provided by multiple images to remove reflections based on motion inconsistency.
Such inconsistency observed from different viewsalso allows our framework to differentiate transmission and reflection during rendering.
Preliminaries about Neural Radiance Fields
We first briefly review the settings in NeRF [1] and NeRF-W [7].
A NeRF represents a scene as acontinuous volumetric radiance field f , which maps the position of any given point x
= ( x, y, z ) anda viewing direction d
= ( θ, ϕ ) into a color c = ( r, g, b ) and density σ asf ( x , d ) =
( c , σ ) .
Then, the expected color ˆ C corresponding to the camera ray r ( t ) =
o + t d emitted from the centerof projection of a camera at position o can be obtained as
[Equation]
2) where t n and t f denote the near and far bounds of rendering, and T ( t ) =
exp( − (cid:82) t t n σ ( r ( s )) ds )denotes the accumulated transmittance.
NeRF-W [7] extends NeRF [1] with two MLPS to model thedesired static ( f s ) and undesired transient ( f t ) components asf s ( x , d , ℓ a ) =
( c s , σ s ) and f t ( x , d , ℓ t ) =
( c t , σ t ) .
Due to the introduction of the additional MLP for transient elements, as well as the embeddings ℓ a andℓ t adapting to the various appearances and transient objects, NeRF-W [7] is capable of decomposingthe desired static components during rendering.
For an image I captured in see-through scenarios, the relationship between the reflection R andtransmission T can be formulated as I = β T + α R
[6], where β and α denotes the weightingcoefficients.
Applying NeRF-W [7] to achieve novel transmitted view synthesis is not directlyapplicable due to the following issues: Dense view requirement: Both NeRF-W [7] and NeRF [1] inherit the substantial demand for dense multi-view images, and their performances degenerate under real-world scenarios where only limited input with sparse views are available.
Transmission and reflection entanglement: Due to the absorption, reflection, and refrac-
tive effect, T and R presents a highly entangled relationship, which cannot be simplydisentangled by using an unsupervised model.
We propose to construct Neural Transmitted Radiance Fields to address these issues.
Neural Transmitted Radience Fields
Our goal is to learn radiance fields as a representation for the transmitted scene behind glass.
Toavoid the high separation ambiguity for transmission/reflection and low reconstruction fidelity undersparse views, we build a unified approach by warping pixels to the desired target views.
By extractingthe recurring edges after warping, our approach can get rid of the interference from the reflection.
Meanwhile, a transmission feature integration scheme is introduced to merge the transmission featuresfrom the warped images, which enables higher reconstruction fidelity even under sparse views.
The pixel-wise rendering scheme in NeRF [1] and NeRF-W [7] lacks the ability to reflect thetransmission/reflection correlation, since some statistical properties can only be observed in a certainarea.
To better consider the spatial correlation among neighboring pixels, we employ the patch-basedrendering scheme.
Specifically, during training we sample a K × K patch P ( u , s ) in a given imagefollowing the way proposed in [26] as follows:
[Equation]
where u = ( u, v ) denotes the center position of the patch, x and y range from − K 2 to K 2 − 1 tocreate a sampling grid, and s denotes the scale of the image patch P .
In our setting, we apply a combined sampling of grid sampling with s = 1 over the entire image and random sampling withs > 1 , allowing image patches to capture spatial information of different scales.
The aggregatedpixels from the image patch are then used for training, during which the model can not only learn thecolors and densities through one ray, but also get a thorough understanding towards the context.
4.1 Transmission feature integration
In contrast to NeRF [1] and NeRF-W [7], which solely rely on point coordinates and viewingdirection as the input, we propose to address the fidelity issue by means of the additional guidancefrom reference images.
Given a scene only with sparse views, the feature guidance from neighboringviews can help to reconstruct novel target views with higher fidelity.
In general, such image featuresare extracted by using a 2D CNN network [10] pretrained for general tasks.
However, since theimages under neighboring views are also captured through glass, a general CNN network for viewsynthesis like previous methods [15] may lead to feature-level interference during rendering.
As shown in Figure 2, we first project the neighboring views to their target view.
To resist suchinterference during feature integration, we propose to obtain the transmission features by means ofa transmission encoder η obtained by pre-training a reflection removal network, e.g. , ERRNet [5].
As a feature encoder pretrained for reflection removal task, η aims at extracting the transmissionfeatures as W = η ( I ) , where W denotes the extracted feature volume.
Instead of solely relying onthe single-scale feature like pixel-NeRF [27], which suffers from view misalignment in our setting,we propose to build a feature pyramid as [Equation]
where W g and W l denote the global and local features with different scales, respectively.
In general,as CNNs naturally store local information in shallow layers and encode global information in deeperlayers with a large receptive field, we extract W g and W l from the bottleneck layer and second lastlayer of η , respectively.
The extracted global and local transmission features for the i -th target view are then passed intothe MLP network for the transmission rendering (T-MLP)
f T , along with the position and viewingdirection, as
[Equation]
[Equation]
where c ( i ) t and σ ( i ) t are the estimated color and density of the transmitted scene for the i -th view.
As for reflection, a learnable reflection embedding ℓ ( i ) r is introduced for each view i , and the colorc ( i ) r and density σ ( i ) r of the reflection scene are estimated using an additional MLP network for thereflection rendering (R-MLP) as
[Equation]
where the reflection embedding is optimized along with the weights of f R .
Similar to existing NeRFs [1, 7], we construct a self-consistent loss as follows:
[Equation]
where p ( · ) denotes the mapping from a ray to its corresponding pixel position, and ˆ C ( r ) denotes theobserved color and C denotes the composite of C T and C R physically defined as [Equation]
[Equation]
Here α denotes the weighting coefficients to balance C T and C R .
For simplicity, we directly setβ = 1 − α like previous reflection removal methods [28].
Several factors during light transportation can influence the weighting coefficients α .
In our setting,since the strength of reflection largely depends on viewing direction [29] rather than the exact cameraposition, we propose to model such influence by a simple MLP that outputs the weighting coefficientsgiven corresponding viewing direction encoding as f α ( x , d )
= α .
4.2 Recurring edge constraints
The settings described above ensure a high-fidelity novel view reconstruction even under sparseviews.
However, since no reflection removal model can reach a completely reflection-free status, thereflection residuals left by the transmission feature encoder may further deteriorate the ambiguityissue.
We then utilize the benefits provided by the aligned image sequence to further disentangle theirtwisted relationship.
As pointed by pioneer methods [9, 28], when multiple images with different viewpoints are capturedthrough glass, the transmission appears frequently across all samples, while the reflection may onlyhave sparse presence.
This property makes the recurring transmission edges become a “pilot” torelieve ambiguity between the transmission and reflection.
In our method, we first transform alignedimages into their corresponding edge domain and measure the frequency of edge occurrence as
[Equation]
where G i ( z ) denotes the gradient magnitude at pixel location z of view i , and k denotes the totalnumber of pixels.
It measures the sparsity of the vector which achieves its maximum value of 1(when only one non-zero item exists) and achieves its minimum value of 1 /k (when all items arenon-zero and have identical values).
Using this measurement, each edge pixel is assigned with twoprobabilities regarding whether it belongs to the transmitted scene or not.
Recognizing the transmission and reflection poses a great challenge for a vanilla NeRF networkaiming at multi-view consistency, we provide our network with a likelihood estimation of thetransmission and reflection, following the strategy proposed in [28]:
[Equation]
11) where s is a sigmoid function to facilitate the separation.
Then, the final likelihood for transmissionedges can be defined as follows:
[Equation]
[Equation] From the results shown in Figure 1(right), the likelihood of target transmitted scene effectively labelsthe transmission edges, which provides a clear guidance during rendering.
Based on the binarized edge map obtained in Equation (12), we propose to constrain rendering usinga simple edge loss as follows:
[Equation]
which means we constrain the rendered image more on wherever the transmission edges are present,encouraging a more faithful reconstruction towards the mixture image.
Besides, the transmission edges and those of the reflection tend to be different, as prior works pointedout [21].
We inherit the exclusion assumption between the gradient of transmission and reflection todifferentiate the two parts in the gradient domain as follows:
[Equation]
where λ T = (cid:113) ∥
G ( C R ) ∥
F ∥ G ( C T ) ∥
F and λ R = (cid:113)
∥ G ( C T )
∥
F ∥ G ( C R ) ∥ F .
The downsampling operation in its originalsetting is discarded since our patch has been small enough.
By considering the above settings and constraints, our loss functions for the whole optimization canbe concluded as follows: [Equation]
where γ = 2 , δ = 0 . 002 , and ω = 1 are the weighting coefficients to balance the influence of eachterm.
All the calculations and optimization processes are based on the patch sampled from the image.
4.3 Implementation details
We implement our framework using PyTorch.
In the training and testing phase, two eight-layer MLPswith 256 channels are used to predict colors c and densities σ corresponding to the transmitted andreflection scenes.
We train a “coarse” network along with a “fine” network network for importancesampling.
We sample 64 points along each ray in the coarse model and 64 points in the fine model.
Abatch contains an image patch of 32 × 32 pixels, equivalent to 1024 rays.
Similar to the settings in
NeRF [1], positional encoding is applied to input location before they are passed into the MLPs.
Weuse the Adam optimizer with defaults values β 1 = 0 .
999 , β 2 = 0 .
9 , ϵ = 10 − 8 , and a learning rate10 − 4 that decays following the cosine scheduler during the optimization.
We optimize a single modelfor about 100 K iterations on two NVIDIA V100 GPUs.
Our transmission encoder can extract transmission features of multiple resolutions by leveragingadvantages from ERRNet [5] and U-Net [30].
Based on our experiments, even a simple U-Net [30]can also help to resist the reflection interference during rendering.
The transmission encoder is firstintegrated with a feature decoder to compress the information to a lower dimension, and the output isfed into the transmission MLP.
For the pre-training of the transmission encoder, we follow the common way adopted by most imagerestoration methods, where the L 1 and adversarial loss between the estimation and its ground truthare employed to optimize the solution space.
Experiments
5.1 Experimental Settings
Dataset.
Our experiments are all based on a real-world dataset we collect.
This dataset contains8 different real-world scenes, each consisting of 20 to 30 mixture images with different poses.
Specifically, four scenes are with the ground truth for quantitative evaluations in the experiments.
Wefollow the setup proposed in [6] by first capturing the mixture image through transparent glass andthen its corresponding transmitted scene by removing the glass.
Baselines.
We compare our method with five NeRF-based methods: 1) NeRF [1]: the original NeRFmethod; 2) NeRF-W [7]: an unofficial implementation of NeRF in the Wild 1 ; 3) MVSNeRF [10]:a NeRF-based multi-view stereo model for novel view synthesis which also relies on referenceviews; 4) Reflection removal (RR) + NeRF: the most direct way for novel transmitted view synthesis,namely training a NeRF network based on the results of a reflection removal method (without losinggenerality, we use ERRNet [5], a state-of-the-art method inspiring our design); 5) NeRFReN [31]: a
NeRF-based method designed for see-through scenarios released recently 2 .
All the results in Section 5are obtained using six views for training.
Due to the page size limitation, the results of other methodstrained under more views can be found in the supplementary material.
Metrics.
We report quantitative performance using PSNR, SSIM and LPIPS [32].
For PSNRand SSIM, higher value indicates better performance.
For LPIPS [32], lower value indicates betterperformance.
By measuring image fidelity using high-level features, LPIPS [32] can better matchhuman judgements of image similarity.
5.2 Qualitative results
The qualitative results of the rendered novel transmitted views are presented in Figure 3 and Figure 4.
Due to the effectiveness of the proposed transmission feature integration scheme, our method canreliably reconstruct novel transmitted views under only six views.
Besides, the feature integrationscheme and its cooperative recurring edge constraints further reduce the ambiguity between reflectionand transmission during rendering.
For other methods, though NeRF [1] can partially reconstruct thedesired transmitted views, it still suffers from the interference caused by the reflection; NeRF-W [7]and NeRFReN [31] output degenerated results as they cannot separate reflection from transmission Target view Ours NeRF
NeRF-W MVSNeRF RR+NeRF NeRFReN
[Table]
with as few inputs as our method; MVSNeRF [10] can keep high fidelity of the rendering results likeour methods, but obvious reflection residuals still remain in their results since their feature extractorcannot resist the reflection interference.
5.3 Quantitative results
The quantitative results of the rendered novel transmitted view in Table 1 also validate the observationin Figure 4.
Higher PSNR values show that our method can render novel transmitted views andrecover the color information with higher accuracy.
Higher SSIM values indicate that our methodcan preserve the structural information with high-frequency details.
Lower LPIPS values show thatrecovered images by our method better aligns with human perception.
Specifically, for the resultsobtained by other methods, their higher LPIPS values indicate that they cannot reconstruct realisticresults with high fidelity when only six views are provided.
Specifically, MVSNeRF [10] achievesthe second best LPIPS results among all methods, which shows that the feature integration schemecan indeed improve the rendering fidelity.
On the other hand, RR+NeRF setting achieves the secondbest PSNR and SSIM result, which shows the effectiveness of this simple strategy to some degrees. Target view Ours NeRF
NeRF-W
MVSNeRF RR+NeRF NeRFReN PSNR:23.
PSNR:16.
PSNR:13.
PSNR:21.
PSNR:17.
PSNR:21.
PSNR:16. PSNR:17. PSNR:14. PSNR:19.
PSNR:16.
PSNR:16.
[Table]
[Figure]
Our network consists of two parts: the transmission encoder to disentangle the transmitted andreflected scenes, and the patch-rendering scheme to utilize the physical constraints.
We conductseveral experiments to evaluate the benefits of these two parts.
We first remove the transmissionencoder and directly feed the MLP with the position and viewing direction.
From the results shownin Figure 5, without the support of reference views, the position and viewing directions cannoteffectively reconstruct the desired transmitted views.
The errors shown in Table 2 also become closerto those of NeRF in Table 1.
We then remove the recurring edge constraints, and more reflection isobserved in the rendered target view since the network attaches equal importance to every pixel inthe patch.
We further remove the patch-based rendering scheme entirely.
In this situation, reflectionevidently remains on the generated target view due to a lack of physical constraint, as we discussedbefore.
At last, both the patch-based rendering scheme and the transmission encoder are removed,and we achieve a degenerated result similar to that of NeRF-W [7] shown in Table 1.
Conclusions
We solve the problem of novel view synthesis for see-through scenarios in this paper.
We introduce atransmission encoder to address the fidelity issue caused by sparse views and ambiguity issues led bythe reflection interference in a unified framework.
Specifically, to further disentangle the twisted relationship between transmission and reflection during the rendering process, we introduce a recurringedge constraint by counting the frequency of edge occurrence among the aligned image sequence.
Experimental evaluation on a newly collected dataset demonstrates the promising performance fornovel transmitted view synthesis our method could achieve.
Limitations.
Our method still faces the challenge led by the occlusion issue, since we rely onfeatures from neighboring views to complement the missing information given a sparse set of inputviews.
However, when features are not consistent with a certain view, occlusions may negativelyundermine the performance of our proposed method, due to the inaccurate feature information forsynthesizing novel views in that area.
We will deal with this issue in our future work.