G ENERAL N EURAL G AUGE F IELDS
Anonymous authors
Paper under double-blind review
A BSTRACT
The recent advance of neural implicit representations, such as neural radiance fields, has significantly pushed the boundary of scene representation learning.
Aiming to boost the computational efficiency and rendering quality of neural radiance fields of 3D scenes, a popular line of research maps the 3D coordinate system to another measuring system, e.g., 2D manifolds and hash tables, for modeling radiance fields.
The conversion of coordinate systems can be typically dubbed as gauge transformation , which is usually a pre-defined mapping function, e.g., an orthogonal projection or spatial hash function.
This begs a question: can we learn the gauge transformation along with the neural scene representations in an endto-end manner?
In this work, we extend this problem to a general paradigm with a taxonomy of discrete and continuous cases, and develop an end-to-end training framework to jointly optimize the gauge transformation and radiance fields.
To counter the problem that the learning of a gauge transformation can collapse easily, we derive a general regularization mechanism from the principle of information conversation during the gauge transformation.
On the strength of the derived unified neural gauge field framework, we naturally discover a new type of gauge transformation which achieves a trade-off between learning collapse and computational cost.
I NTRODUCTION
Representing 3D scenes with high efficiency and quality has been a long-standing target in computer vision and computer graphics research.
Recently, the implicit representation of neural radiance fields (Mildenhall et al., 2021) has shown that a 3D scene can be modeled with neural networks, which achieves compelling visual quality and low memory footprint.
However, it suffers from long training times.
The explicit voxel-based methods (Yu et al., 2021a; Sun et al., 2022) emerged with faster convergence but higher memory requirements, due to the use of the 3D voxel grids as scene representation.
To strike a good balance between computational efficiency and rendering quality,
EG3D (Chan et al., 2022) proposed to project the 3D coordinate system to a tri-plane system.
Along with this line of research, TensoRF (Chen et al., 2022) factorizes 3D space into compact low-rank tensors;
Instant-NGP (M ¨ uller et al., 2022) models the 3D space with multi-resolution hash grids to enable remarkably fast convergence speed.
These recent works (e.g., EG3D and Instant-NGP) share the same prevailing paradigm by converting the 3D coordinate of radiance fields to another coordinate system.
Particularly, a coordinate system of the scene (e.g., 3D coordinate and hash table) can be regarded as a kind of gauge, and the conversion between the coordinate systems can be referred to as gauge transformation (Moriyasu, 1983).
Notably, existing gauge transformations in radiance fields are usually pre-defined functions (e.g., orthogonal mappings and spatial hash functions (Teschner et al., 2003)), which are sub-optimal for modeling the radiance fields as shown in Fig.
In this end, a learnable gauge transformation is more favored as it can be optimized towards the final objective.
This raises an essential question: how to learn the gauge transformations along with the radiance fields.
Some previous works explore a special case of this problem, e.g., NeuTex (Xiang et al., 2021) and NeP (Ma et al., 2022) aim to transform 3D points into continuous 2D manifolds.
However, a general and unified learning paradigm for various gauge transformations has not been established or explored currently.
In this work, we introduce general Neural Gauge Fields which unify various gauge transformations in radiance fields with a special focus on how to learn the gauge transformations along with radiance fields.
Basically, a gauge is defined by gauge parameters and gauge basis, e.g., codebook indexes
[Figure]
and codebook vectors, which can be continuous or discrete.
Thus, the gauge transformations for radiance fields can be duly classified into continuous cases (e.g., triplane space) and discrete cases (e.g., a hash table space).
We then develop general learning paradigms for continuous cases and discrete cases, which map a 3D point to a continuous coordinate or a discrete index in the target gauge, respectively.
As typical cases, we study continuous mapping from 3D space to the 2D plane and discrete mapping from 3D space to 256 discrete vectors.
As shown in Fig.
2 and 3, we observed that naively optimizing the gauge transformations with the radiance field severely suffers from learning collapse, which means the gauge transformations will collapse to a small region in a continuous case or collapse to a small number of indexes in a discrete case Baevski et al.
Kaiser et al.
To regularize the learning of gauge transformation, a cycle consistency loss has been explored in Xiang et al.
; Ma et al.
(2022) to avoid many-to-one mapping; a structural regularization is also adopted in
Tretschk et al.
(2021) to preserve local structure by only predicting a coordinate offset.
However, the cycle consistency regularization tends to be heuristic without grounded derivation while the structural regularization is constrained to regularize continuous cases.
In this work, we introduce a more intuitive Info rmation Reg ularization ( InfoReg ) from the principle of information conservation during gauge transformation.
By maximizing the mutual information between gauge parameters, we successfully derive general regularization forms for gauge transformations.
Notably, a geometric uniform distribution and a discrete uniform distribution are assumed as the prior distribution of continuous and discrete gauge transformations, respectively, where an Earth Mover’s distance and a KL divergence are duly applied to measure the distribution discrepancy for regularization.
As InfoReg successfully handles learning collapse by maximizing the mutual information to avoid information loss, we conjecture that a gauge (parameter) with higher information capacity helps to alleviate the learning collapse.
Since a discrete gauge transformation maps a 3D point to the most probable (i.e., top-1) index, a gauge parameter with higher information capacity can be achieved by mapping a 3D point to k most probable indexes as the encoding, which is a special case of discrete gauge termed as Top-k Gauge .
We found that this top-k gauge will gradually obviate the need for regularizations with the increase of k. Notably, with proper selection of used vectors, the top-k gauge can be learned to outperform the strong baseline of spatial hash function.
As a tradeoff, the top-k gauge also incurs higher computation cost with the increase of k for encoding.
The contributions of this work are threefold.
First, we develop a general framework of neural gauge fields which unifies various gauge transformations in radiance fields, and gives general learning forms for continuous and discrete gauge transformations.
Second, we strictly derive a regularization mechanism for gauge transformation learning from the perspective of information conservation during gauge transformation that outperform earlier heuristic approaches.
Third, we introduce a new top-k gauge that can achieve a tradeoff between learning collapse and computation cost.
R ELATED W ORK
Recent work has demonstrated the potential of neural radiance fields (Mildenhall et al., 2021) and its extensions for multifarious vision and graphics applications, including fast view synthesis (Liu et al., 2020; Yu et al., 2021b;
Hedman et al., 2021;
Lindell et al., 2021; Neff et al., 2021; Yu et al., 2021a;
Reiser et al., 2021; Sun et al., 2022), generative models (Schwarz et al., 2020; Niemeyer & Geiger,
[Figure] [Figure]
Gu et al., 2022; Chan et al., 2021; Or-El et al., 2022), surface reconstruction (Wang et al., 2021;
Oechsle et al., 2021; Yariv et al., 2021), etc.
Under this context, various gauge transformations have been intensely explored in radiance fields for different purposes, e.g., reducing computation cost (Chan et al., 2022), fast convergence (Chen et al., 2022; M¨uller et al., 2022), editable UV map (Xiang et al., 2021; Ma et al., 2022).
A pre-defined mapping function is usually adopted as gauge transformation.
A simple case is orthogonal mapping, which projects 3D space into a continuous low-dimension space.
For instance,
EG3D (Chan et al., 2022; Peng et al., 2020) orthogonally projects 3D space into single plane or triplane space for scene modelling.
TensoRF (Chen et al., 2022) further extends this line of research for fast and efficient optimization by mapping the 3D scene space into a set of low-dimension tensors.
With a pre-defined spatial hash function, the recent Instant-NGP (M ¨ uller et al., 2022) maps 3D grid points into hash table indexes and achieves a speedup of several orders of magnitude.
Besides,
PREF (Huang et al., 2022) explores to transform 3D scene points to a set of Fourier basis via Fourier transformation, which helps to reduce the aliasing caused by interpolation.
Some works also explore learning the gauge transformation for downstream tasks in neural radiance fields.
For example, NeuTex (Xiang et al., 2021) and NeP (Ma et al., 2022) propose to learn the mapping from 3D points to 2D texture spaces; GRAM (Deng et al., 2022) learns to map the 3D space into a set of implicit 2D surfaces; VQ-AD Takikawa et al.
(2022) enables compressive scene representation by mapping coordinates to discrete codebook, which is a discrete case of gauge transformation.
However, all above works suffer from learning collapse without regularization or designed initialization.
Besides, a popular line of research (Tretschk et al., 2021; Tewari et al., 2022; Park et al., 2020; Pumarola et al., 2020; Liu et al., 2021; Peng et al., 2021) learns the deformation from the actual 3D space into another (canonical) 3D space, where a structural regularization is usually applied by predicting a deformation offset instead of the absolute coordinate.
M ETHODS
This section will introduce the framework of neural gauge fields in detail, followed by the derivation of the learning regularization based on information conservation during gauge transformation.
A new top-k gauge and some insights for the tradeoff between the learning collapse and computation cost are drawn in the end.
3. N EURAL G AUGE F IELDS
A gauge system is defined by a gauge basis and gauge parameters, where the gauge basis is a measuring space (e.g., triplane in EG3D (Chan et al., 2022) and hash table in Instant-NGP (M¨uller et al., 2022)).
The gauge parameters indicate the specific position in the measuring space (e.g., triplane coordinate in EG3D (Chan et al., 2022) and hash table index in Instant-NGP (M¨uller et al., 2022)).
Based on this intuition, the conversion between gauges in radiance fields can be termed as gauge transformation .
In this work, we introduce the general framework of Neural Gauge Fields which consists of a gauge transformation and a radiance field.
A neural gauge field aims to transform the original 3D space to another gauge system which is then used to index a radiance field.
This additional transform could introduce certain bonus to the radiance field, e.g., low memory cost, high rendering quality, or explicit texture, depending on the purpose of the model.
This work focuses on how to learn a decent gauge transformation for radiance field regardless of the specific purposes.
The gauge transformation could be a pre-defined function e.g., an orthogonal mapping in EG3D and a spatial hash function in Instant-NGP, or directly learned by the neural network along with radiance fields.
Intuitively, compared with a pre-defined function, a learnable gauge transformation is more favorable as it can be optimized towards the final use case of the neural radiance field and possibly yields better performance.
For learning neural gauge fields, we disambiguate between two cases: continuous (e.g., 2D plane and sphere surface) and discrete (e.g., hash table space) mappings.
Continuous Gauge Transformation.
As the target gauge system is continuous, the gauge transformation can be handled by directly regressing the coordinate (i.e., gauge parameters) in the target gauge.
For a point x ∈ R 3 in the 3D space and a target gauge, a gauge network M (a MLP structure or transformation matrix) can be employed to predict the target coordinate as x ′ =
M ( x ) , x ′ ∈ R N , where N is the coordinate dimension of the target gauge.
Then, the predicted coordinate x ′ can be fed into an MLP or used to look up feature vectors to model radiance fields.
Discrete Gauge Transformation.
Without loss of generality, we assume there is a codebook with
N discrete vectors [ v 1 , v 2 , · · · , v N ] as the gauge basis, and the discrete gauge transformation aims to transform a 3D point to a discrete codebook index (i.e., gauge parameter).
In line with the setting of Instant-NGP, we divide the 3D space into a grid of size M ∗ M ∗ M .
For each grid point x , we apply a gauge network M to predict its discrete distribution P x ∈ R N over the N discrete vectors, followed by an Argmax operation to select the most probable index
i x as below:
[Equation]
The produced indexes for grid points are usually used to lookup corresponding vectors from the codebook 1 , and the features of other 3D points can be acquired by interpolating the nearby grid point vectors.
As the Argmax operation is not differentiable, a reparameterization trick (Bengio et al., 2013) is applied to approximate gradient by replacing Argmax with Softmax in back-propagation.
The pseudo code of the forward & backward propagation of discrete cases is given in Algorithm 1.
Input: A 3D point x , predicted distribution
P
x
=
[ p 1 , p 2 , · · · , p N ] ,
codebook
V .
Forward propagation: 1. index =
Argmax( P x ) 2.
index hard =
One Hot(index) 3.
feature lookup: v x = Matmul(index hard, V ) Backward propagation: 1.
index soft =
Softmax( P x ) 2.
feature lookup: v x = Matmul(index soft, V ) Output: feature v x .
We now have established the general learning paradigms for continuous and discrete gauge transformations.
In our setting, the gauge transformations are optimized along with the radiance field supervised by the per-pixel color loss between the rendering and the groundtruth multi-view images.
However, we found the learning of gauge transformations is easily stuck in local minimal.
Specifically, the gauge transformation tends to collapse to a small region in continuous cases as in Fig.
2 or collapse to a small number of indexes in discrete cases as in Fig.
3, highlighting the need for an additional regularization to enable an effective learning of continuous and discrete gauge transformations, which we will introduce in the following.
3. L EARNING R EGULARIZATION
Some previous works (Xiang et al., 2021; Ma et al., 2022) leverage a cycle consistency loss to regularize the mapping from a 3D space to a 2D UV space to prevent many-to-one mapping of 3D points.
Alternatively, a structural regularization is also employed in some works (Tretschk et al., 2021; Tewari et al., 2022) to preserve local structure by predicting a residual offset instead of the absolute coordinate.
However, the cycle loss only works for certain continuous cases and shows poor performance for the regularization of discrete cases, while the structural regularization cannot be applied to discrete cases at all.
In this work, we strictly derive a general regularization mechanism for gauge transformation from the perspective of information conservation which is superior to the individual regularizers used in prior work.
Specifically, although the gauge of the scene has been changed, the intrinsic information of the scene is expected to be preserved during gauge transformation, which can be achieved by maximizing the mutual information between the original 3D points and target gauge parameters.
With the gauge network M for gauge transformation, the sets of original 3D points and target gauge parameters can be termed as X and Y =
M ( X ) , respectively.
Then, the mutual information between X and Y can be formulated as:
[Equation]
where p ( x ) and p ( y ) are the distributions of original 3D point x and target gauge parameter y , p ( y | x ) is the conditional distribution of y produced by the gauge transformation of x .
To avoid that the distribution of Y collapses into a local region or small number of indices, we assume that Y obeys a prior uniform distribution q ( y ) , and the KL divergence between p ( y ) and q ( y ) can be written as KL
[Equation]
) log p ( y ) q ( y ) dy .
Then, the final regularization loss can be formulated as:
[Equation]
where γ and ϵ denote the weight of regularization and prior distribution discrepancy.
This translates into the following term (see derivation in the Appendix):
[Equation]
Regarding the first term KL ( p ( y | x ) p ( x ) || p
( y ) p ( x )) in Eq.
4, we can replace the unbounded KL divergence with the upper-bounded JS divergence as: JS ( p ( y | x ) p ( x ) || p
( y ) p ( x )) .
Lemma 1 Jensen-Shannon Mutual Information Estimator (Nowozin et al., 2016):
[Equation] [Equation]
where sp ( z ) = log (1 + e z ) .
For JS ( p ( y | x ) p ( x ) || p
( y ) p ( x )) , we thus have:
[Equation]
where T denotes a network which minimizes the distance of positive pairs and maximizes the distance of negative pairs.
Please refer to section 4.1 for the details of T and the selection of positive & negative pairs.
Now the first term of Eq.
(4) has been well solved.
We then turn to the second term E [ KL ( p ( y | x ) || q ( y ))]
(i.e., the KL divergence between p ( y | x ) and q ( y ) ).
This term can be solved according to the types of prior uniform distributions q ( y ) assumed in continuous and discrete cases.
Continuous Gauge Transformation.
To prevent that the gauge transformation collapses to a local region, we expect the prior distribution q ( y ) is a uniform distribution over the target gauge space.
Without loss of generality, we assume the target gauge space is a 2D plane as shown in Fig.(a).
Then, we uniformly sampled h points on the 2D plane as a discrete approximation of the prior uniform distribution, which is denoted by q ( y ) =
[ u 1 , u 2 , · · · , u h ] , u i ∈ R 2 ).
With a set of 3D
1 N 1 N … 1 N 𝒒(𝒚)
KL Divergence
N … 1i
N … 0i+    … 1j
N … 0j+   ……
N-1 k
N … k+   %𝐩(𝐲|𝐱)
One-Hot
[Figure]
points transformed into 2D points, the conditional distribution p ( y | x ) can also be approximated by sampling h points from the predicted 2D points according to the radiance contribution (mentioned previously), which is denoted by p ( y | x ) =
[ v 1 , v 2 , · · · , v h ] , v i ∈ R 2 .
Considering q ( y ) and p ( y | x ) are distributions over a geometric space, KL divergence is inapposite to measure their discrepancy as the distribution geometry is not concerned in KL divergence.
We thus resort to a geometry-aware metric: Earth Mover’s (EM) distance (Rubner et al., 2000).
Here, Earth Mover’s distance measures the discrepancy between two distributions by calculating the minimum work required to move one distribution to another on the 2D square, where the work is measured by multiplying the amount of distribution to be moved and the distance to be moved.
Thus, a cost matrix C and transport matrix M with size of ( h, h ) can be defined where each entry
C ij in C gives the distance between points v i and u j , each entry M ij in M represents the amount of probability moved between point v i and point u j .
Then, the Earth Mover’s distance between q ( y ) and p ( y | x ) can be formulated as: L EM =
min M ( (cid:80) hi =1 (cid:80) hj =1 C ij M ij ) .
The distribution of predicted points will be regularized to fit the uniform distribution by minimizing L EM as shown in Fig.
4 (a).
Discrete Gauge Transformation.
For discrete gauge transformation, we expect the prior distribution q ( y ) to be a uniform distribution over the discrete vectors (i.e., the gauge basis), so that the target gauge parameters will not collapse to a small number of vector indexes.
Thus, the prior distribution q ( y ) can be denoted by q ( y ) =
[ 1 N , 1 N , · · · , 1 N ] for N discrete vectors.
As the one-hot distribution p i is predicted from 3D points as described in Algorithm 1, the conditional distribution p ( y | x ) can be approximated by the average of all predicted one-hot distributions over a training batch, which is denoted by p ( y | x ) =
(cid:80) Bi
=1 p i /N where B is the number of samples within a training batch.
Then the KL divergence between q ( y ) and p ( y | x ) can be duly computed as shown in Fig.
4 (b).
3. T OP - K G AUGE
As the InfoReg maximizes the mutual information between gauge parameters to avoid loss of information, we conjecture that a target gauge with higher information capacity may help to alleviate the loss of information and reduce the need for regularizations, which gives rise to a top-k gauge that is described in the following.
As a concrete example, suppose there is a gauge basis with N discrete vectors [ v 1 , v 2 , · · · , v N ] , and the discrete gauge transformation maps a 3D grid point to one of the vectors.
Thus, a gauge parameter with higher information capacity can be achieved by mapping a 3D point to multiple vectors for encoding, which is termed as Top-k Gauge .
Specifically, the top-k gauge feeds a 3D grid point into a gauge network M to predict a distribution [ p 1 , p 2 , · · · , p N ] .
To select the multiple indexes for encoding, a differentiable top-k operation (please refer to Appendix for more detail) can be applied to the distribution to select k most probable vectors to model radiance fields.
Notably, we intriguingly discover the top-k gauge will suffer less from learning collapse with the increase of used vectors for encoding as shown in Fig.
Note, a top-k gauge with k = 1 is equivalent to the standard discrete gauge.
On the other hand, the top-k gauge will lead to slightly higher memory cost and slower training speed with the increase of used vectors.
Thus, there is a tradeoff between
[Table] [Table]
computation cost and learning collapse which can be controlled while designing a learnable gauge transformation.
Please refer to section 4.2.2 for the analysis of trade-off in top-k gauge.
E XPERIMENTS 4. I MPLEMENTATION
The gauge network M for learning gauge transformations can be a MLP network or a transformation matrix, depending on the specific downstream application.
For the mapping from 3D space to 2D plane to get (view-dependent) textures, the radiance field is modeled by a MLP-based network which takes predicted 2D coordinates, i.e. output of the gauge network, and a certain view to predict color and density.
For the mapping from 3D space to discrete codebooks, the radiance field is modeled by looking up features from the codebook, followed by a small MLP of two layers to predict color and density.
By default, the codebook has two layers and each layer contains 256 vectors with 128 dimensions.
In line with Instant-NGP (M ¨ uller et al., 2022), the 3D space is also divided into two-level 3D grids with size 16 × 16 × 16 and 32 × 32 × 32 for discrete gauge transformation.
For the implementation of our InfoReg, the network T first maps the gauge parameters into features.
Specifically, a 5-layer MLP is employed in T to encode the target gauge parameters y as features y ′ , and the encoded feature x ′ of original 3D points x can be directly taken from the fifth layer of the gauge network.
The distance between constructed pairs can be formulated as:
T ( x, y )
=
d ( norm ( x ′ ) , norm ( y ′ )) , where norm denotes normalization, d is a distance metric, e.g., Euclidean distance (used by default) and cosine distance.
During training, an input 3D point and the corresponding transformed gauge parameters form a positive pair ( x + , y ) , the negative sample x − is randomly sampled from the training batch.
Note InfoReg only uses one negative sample which is obtained by shuffling the input 3D points.
Besides, as 3D points near the scene surface are of more significance, we leverage the radiance contribution per shading point (Xiang et al., 2021) to weigh the mutual information.
Please refer to the Appendix A.3 and A.5 for a more detailed implementation and experimental settings.
4. E VALUATION
The introduced neural gauge fields focus on learning gauge transformation, the experiments are therefore performed to evaluate the performance of gauge transformations.
Specifically, as a gauge
[Figure]
[Figure]
transformation is applied to index a radiance field, the gauge transformation can be evaluated by the novel view synthesis performance of a radiance field.
Besides, a good gauge transformation should also well utilize the target gauge space, which can be evaluated by visualizing the mapping results on 2D plane or computing the codebook utilization.
In this end, the experiments are mainly conducted on continuous gauge transformation from 3D space to 2D plane and discrete gauge transformation from 3D space to 256 codebook vectors.
4.2. C ONTINUOUS & D ISCRETE G AUGE
Regularization.
We compare the performance of different regularizations including structural regularization (StruReg), Cycle loss (CycleReg), and our InfoReg.
As tabulated in Table 1, the proposed
InfoReg outperforms other regularization clearly in both continuous and discrete cases.
Particularly,
StruReg and CycleReg can bring certain improvements for continuous cases, while StruReg cannot be applied to discrete cases and CycleReg presents limited performance for discrete cases.
Pre-defined vs Learned Mapping.
As shown in Table 2, we compare the performance of the pre-defined functions and the learned mapping for gauge transformation.
For continuous cases, the learned mapping with InfoReg outperforms the pre-defined orthogonal mapping clearly.
For discrete cases, the standard learned discrete mapping (top-1 gauge) is slightly inferior to the predefined spatial hash function, while the top-4 variant of the learned discrete mapping surpasses the pre-defined function substantially.
Quantitative Results.
To demonstrate the superiority of learned gauge transformation over predefined function (i.e., orthogonal mapping), we visualize the mapping results from 3D space to 2D plane in Fig.
Fig.
5 (a) illustrates the mapping on the challenging 360 ◦ view scenes.
The learned mapping can still well capture the scene texture, while the orthogonal mapping presents poor mapping performance and even fails in certain case.
Fig.
5 (b) compares the mapping on limited view scenes.
The orthogonal mapping is still prone to lose much texture information thanks to the inherent inflexibility.
Parameter Study of InfoReg.
Fig.
6 examines the effect of the regularization weight γ and prior discrepancy weight ϵ in Eq.
(4) on the Lego scene.
With the increase of regularization weight, the regularization performance of InfoReg is improved consistently, although it is saturated with a large weight.
The regularization performance presents a positive correlation with a small prior discrepancy weight and a negative correlation with a large prior discrepancy weight.
[Table]
[Figure]
dataset.
Combination with SOTA Methods.
To demonstrate the power of learning gauge transformation, we apply our neural gauge fields (NGF) to SOTA methods including TensoRF (initial 64 ×planes with 192 dimension which is gradually upsampled to 192 × 192 ) and Instant-NGP (10 layers with 2 14 to 2 23 entries).
To optimizing gauge transformations, three planes of 1 dimension are employed to indicate the coordinate offset of three feature planes in TensoRF+NGF, ten grids of 2dimensions are employed to indicate the codebook distribution in Instant-NGP+NGF.
Please refer to the Appendix A.5 for more detailed setting.
As shown in Table 3, learning gauge transformation benefits these SOTA methods consistently with ignorable (or very slight) increase of training time and model size.
4.2. T OP - K G AUGE
Trade-off between Collapse and Cost.
Fig.
7 illustrates the rendering performance, codebook utilization, and computation cost with different k values.
The rendering performance and codebook utilization (ratio of used vectors) both are improved consistently with the increase of k , which means the learning collapse is gradually alleviated.
The performance gain brought by InfoReg is also narrowed with the increase of k .
However, the increase of k incurs the significant increase of computation cost, which may hinder the practical application of the model.
C ONCLUSION
This work introduces a general framework of neural gauge fields for unifying various gauge transformations in radiance fields.
With a target to optimize the gauge transformation along with radiance fields, we establish the learning paradigm for continuous and discrete gauge transformations and derive a regularization mechanism to solve the issue of learning collapse from the perspective of information preservation.
To further explore the learning collapse, a top-k gauge is introduced to achieve a trade-off between learning collapse and computation cost.
The extensive experiments and thorough analysis show that the learned gauge transformation outperforms its pre-defined counterpart, which paves the way for learning gauge transformations in rendering pipelines.
E THICS S TATEMENT
Our neural gauge fields focus on technically advancing neural scene representations and are not biased towards specific race, gender, or region.
As a general framework, neural gauge fields can be applied to various fields, e.g., virtual reality, artistic creation, and game design.
On the other hand, the achievements in the image synthesis quality lurk the risk of being misused, e.g., illegal impersonation of actors, or face synthesis for malicious purposes.
Therefore, it is also of importance to explore safeguarding technologies that can reduce the potential for misuse.
One potential way is to use our approach as well as other image synthesis approaches as a discriminator to identify the synthesized images automatically.
This has been actively explored by the community, e.g. Mirsky & Lee (2021).
Alternatively, one should explore watermarking technologies to identify synthesized images.
Besides, access control should also be carefully considered when deploying the relevant techniques to alleviate the risk of misuse.
Finally, we believe creating appropriate regulations and laws is essential to solving the issue of technology misuse while promoting its positive effects on technology development.
R EPRODUCIBILITY S TATEMENT
We attach the source code of neural gauge fields in the supplementary material.
Additionally, we provide detailed implementations in the manuscript that enable a full reproduction of the experimental results.
For example, we specify the choices of rendering resolution, model architecture, and the codebook setting for discrete gauge transformation.
All datasets used in our experiments are publicly accessible.
Finally, all pre-trained models will be made publicly available upon the publication of this work.
The implementations will enable researchers to reproduce the results as well as extend this work to future research.
A A PPENDIX
A. G AUGE T RANSFORMATION B ACKGROUND
In normal usage, gauge means a measuring instrument or measuring system, e.g., pressure gauge and temperature gauge.
The transformation between different gauges (e.g., Fahrenheit and Celsius temperature) can be termed as Gauge Transformation .
In physics, a gauge is a particular choice or specification of vector and scalar potentials which will generate a physical fields.
A transformation from one physical field configuration to another is called a gauge transformation.
The gauge transformations for classical general relativity Wald (2010) are arbitrary coordinate transformations.
Under the context of neural fields, the measuring systems (i.e. gauge) of the 3D scene can be extended to more general definition, e.g., 3D Cartesian coordinate system, 2D coordinate system, hash table space, and frequency space.
The transformation between different scene measuring system is dubbed as Gauge Transformation , and the neural fields with certain gauge transformation is termed as Neural Gauge Fields as shown in Fig. 8. [Figure]
A. D ERIVATION OF I NFO R EG
According to Eq.
(2), the final optimization loss can be formulated as:
[Equation]
where γ and ϵ denote the weight of regularization and prior distribution discrepancy.
Then, we have:
[Equation]
For I ( X [Equation] Y )
[Equation]
we have:
[Equation]
Thus, we can get:
[Equation]
which can be further solved as described in the main paper.
A. R ADIANCE C ONTRIBUTION AS I NFO R EG W EIGHT
As 3D points near the scene surface are of more significance in mutual information, we leverage the radiance contribution weights per shading point to weight the mutual information term as in Xiang et al.
Specifically, with volume density σ and radiance c of a 3D scene, the RGB color I can be computed by aggregating the radiance values of shading points on the ray as below:
[Equation]
where i denotes the index of a shading point on the ray, σ i and c i denote the density and radiance at shading point i , δ i denotes the distance between two consecutive points, T i is the transmittance.
Thus, the radiance contribution weight of a shading point can be formulated as w i
=
T i (1 − exp( σ i δ i )) .
The w i is applied to weight the positive pairs in Eq.
A. T OP - K G AUGE
Differentiable Top-k. With a predicted discrete distribution P x = [ p 1 , p 2 , · · · , p N ] , the top-k operation aims to select the k most probable index in a differentiable way so that the gradient can be back-propagated.
For forward-propagation, we use the efficient top-k operation implemented in
PyTorch to select the top-k values and indexes.
For back-propagation, we apply the reparameterization trick (Bengio et al., 2013) to approximate the gradient by replacing the top-k operation with
Softmax.
The detailed pseudo code of the differentiable top-k operation is given in Algorithm 2. [Table]
Forward propagation:
topk init=[0, 0, · · · , 0] ∈ R N 2.
index, value =
Top-k( P x ) 3.
topk hard = topk init.scatter(index, value) 4.
value sum =
value.sum() 5.
topk hard = topk hard ÷ value sum 6.
feature lookup: v x = Matmul(topk hard, V )
Backward propagation: 1. index soft =
Softmax( P x ) 2.
feature lookup: v x = Matmul(index soft, V )
Grid Upsampling.
The information capacity of grid points is gradually improved with the increase of k value in top-k. To better utilize the information capacity for modeling, the 3D grid resolution is also increased accordingly by log( k ) times, e.g., the two-level 3D grids of top-4 gauge are of sizes 16 · log( k ) × 16 · log( k ) × 16 · log( k ) and 32 · log( k ) ×
32 · log( k ) × 32 · log( k ) .
A. I MPLEMENTATION AND E XPERIMENTAL S ETTING
We implemented our neural gauge fields and re-implemented all compared methods without customized CUDA kernels.
All models are optimized for 150k steps with a batch size of 1024 pixel rays.
Efficient Gauge Transformation.
The proposed neural gauge fields can be applied to benefit existing models without increasing their computational cost and training speed significantly.
Here, we provide several strategies to achieve it: First, instead of using MLPs to learn the gauge transformation, we can directly optimize tensors for transformation, e.g., 2D offset grids to indicate the coordinate transformation in TensoRF, 1D vectors for grid points to indicate codebook index distribution in Instant-NGP.
Thus, the convergence speed of gauge transformation can be aligned with grid-based models like TensoRF and Instant-NGP.
Second, gauge transformation learning can be performed in the middle training stage instead of the full training process, then the learned transformation is frozen for ensuing model training.
Thus, the computation cost introduced by learning gauge transformation (and regularization) can be ignorable compared with the total training time.
Third, we empirically find that the prior distribution regularization term in InfoReg has been enough to avoid learning collapse in most cases, which is much more efficient than the complete InfoReg.
A. L IMITATIONS
This work focuses the fundamental problem of learning gauge transformation along with radiance fields and is orthogonal to the SOTA methods for scene representation, e.g., TensoRF and InstantNGP.
Although we demonstrate that the learned gauge transformation can outperform its pre-defined counterpart (e.g., orthogonal mapping and spatial hash function), there are still some challenges to apply this method to the SOTA models.
For instance, the original Instant-NGP employs a 16-layer codebook with a size ranging from 2to 2 24 .
When replacing the spatial hash function in Instant-NGP with the learned counterpart, the gauge network size will be extremely huge with an output layer dimension ranging from 2 14 to 2 24 .
One possible solution is firstly grouping the codebook into several sub-blocks with smaller size, followed by smaller gauge network to learn the mapping to each block.
Specifically, a large codebook (e.g., 2 20 ) can be divided into a set of sub-codebooks (e.g., 2 8 sub-codebooks with of size 2 12 ).
Similarly, a grid space 256 × 256 × 256 can be divided into a set of sub-grids (512 subgrids of size 32 × 32 × 32 ).
Then, with our neural gauge field framework, the mapping between the sub-grids and sub-codebook can be learned, where a very small output layer can be employed for mapping (i.e., 2 8 =256).
For grid points within the sub-grid, the spatial hash function can be applied to achieve the mapping.
Overall, the output layer size can be adjusted freely according to the selection of sub-codebook size.
This strategy is adopted for the experiments in Table 3.
A. D ISCUSSION & F UTURE W ORK
As a general framework, the proposed neural gauge fields can be extended in several directions.
Firstly, this work focuses on radiance fields for novel view synthesis, which can be easily extended to other tasks, e.g., surface reconstruction with SDF, dynamic scenes modelling.
Secondly, our InfoReg solves the learning collapse in discrete gauge transformation, and can be potentially applied to the field of discrete representation learning (e.g., VQ-VAE (Van Den Oord et al., 2017)) which also suffers from codebook collapse.
Thirdly, this work mainly explores the pure pre-defined or learned mapping, and how to combine pre-defined mapping and learned mapping to yield better performance is also of great potential.
Please refer to Appendix A.6 for more discussion of limitations.
A. M ORE C OMPARISON & S TUDY
Comparison with SOTA Methods.
To demonstrate the power of the learned gauge transformation, we compare our top-256 gauge (best performance but also highest computation cost) with SOTA methods including EG3D (initial 64 × 64 plane with 32 dimension which is gradually upsampled to 192 × 192 ), TensoRF (initial 64 × 64 planes with 27 dimension which is gradually upsampled to 192 × 192 ) and Instant-NGP (8 layers with 2 14 to 2 2 1 entries).
As shown in Table [Table] the learned top-256 gauge outperforms all SOTA methods with a much smaller model size, which demonstrates the potential of the learned gauge transformation for high-quality rendering [Table]
Gaussian Distribution as Prior Distribution.
Except for uniform distribution, other distributions like Gaussian distribution can also serve as the prior distribution for effective regularization as shown in Fig.
Compared with the rendering performance (30.74 as reported in Table 1) of uniform distribution prior, Gaussian distribution prior achieves slightly inferior result (30.51) on SyntheticNeRF but also avoids learning collapse well.
[Figure]
A. M ORE Q UALITATIVE R ESULTS
Fig.
10 compares the performance of InfoReg with Euclidean distance and cosine distance, the Euclidean distance is prone to preserve better texture structure compared with Cosine distance although they both cover the 2D space and achieve similar rendering performance.
Fig.
11 shows additional samples of leaning continuous gauge transformation with different regularizations.
Except for novel view synthesis, we also apply our neural gauge fields to the downstream task of texture editing as shown in Fig.
Specifically, we learn the mapping from 3D space (points on object surface) to sphere surface which is visualized by cubemap for texture editing.
[Figure]
[Figure] [Figure]