Original: Rotation-Equivariant Conditional Spherical Neural Fields for Learning a Natural Illumination Prior
Revised: Rotation-Equivariant Conditional Spherical Neural Fields for Learning a Natural Illumination Prior

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Inverse rendering is an ill-posed problem.
Revised: Inverse rendering is an ill-posed problem.

Original: Previous work has sought to resolve thisby focussing on priors for object or scene shape or appearance.
Revised: Previous work has sought to resolve thisby focussing on priors for object or scene shape or appearance.

Original: In this work, weinstead focus on a prior for natural illuminations.
Revised: In this work, weinstead focus on a prior for natural illuminations.

Original: Current methods rely on sphericalharmonic lighting or other generic representations and, at best, a simplistic prioron the parameters.
Revised: Current methods rely on sphericalharmonic lighting or other generic representations and, at best, a simplistic prioron the parameters.

Original: We propose a  conditional neural field representation based on avariational auto-decoder with a SIREN network and, extending Vector Neurons,build equivariance directly into the network.Using this we develop a rotationequivariant, high dynamic range (HDR) neural illumination model that is compactand able to express complex, high-frequency features of natural environmentmaps.
Revised: We propose a rotation-equivariant, high dynamic range (HDR)neural illumination model that is compact and able to express complex, highfrequency features of natural environment maps. We propose a conditional neuralfield representation based on a variational auto-decoder with a SIREN networkand, extending Vector Neurons, build equivariance directly into the network.

Original: Training our model on acurated dataset of 1.6K HDR environment maps ofnatural scenes, we compareit against traditional representations, demonstrate itsapplicability for an inverse rendering task and show environment map completionfrom partial observations.
Revised: Wecurate a dataset of 1.6K HDR environment maps of natural scenes on which wetrain our model. We compare against traditional representations, demonstrate itsapplicability for an inverse rendering task and show environment map completionfrom partial observations.

Original: A PyTorch implementation, our dataset and trainedmodels will be made available.
Revised: A PyTorch implementation, our dataset and trainedmodels will be made available.

Original:  Introduction
Revised:  Introduction

Original: Compact but expressive lighting representations play an essential role in graphics, enabling realisticlighting effects at real-time frame rates [38,  53,54, 34, 22] and in computer vision enabling scenerelighting [59, 58], face relighting [55, 43, 17, 42] and object insertion [56, 47, 31].
Revised: Compact but expressive lighting representations play an essential role in graphics, enabling realisticlighting effects at real-time frame rates [38, 52, 53, 34, 22] and in computer vision enabling scenerelighting [57, 58], face relighting [54, 43, 17, 42] and object insertion [55, 47, 31].

Original: Real-world illumination is highly complex and variable, with a very high dynamic range, and is therefore inherentlychallenging to represent.
Revised: Real-world illumination is highly complex and variable, with a very high dynamic range, and is therefore inherentlychallenging to represent.

Original: However, real-world illumination does contain statistical regularities [16],particularly for outdoor, naturally lit scenes.
Revised: However, real-world illumination does contain statistical regularities [16],particularly for outdoor, naturally lit scenes.

Original: For example, lighting usually comes predominantly fromabove, often with the strongest illumination coming from primary light sources in a few directions.
Revised: For example, lighting usually comes predominantly fromabove, often with the strongest illumination coming from primary light sources in a few directions.

Original: Also, the sky and sunlight produce only a limited range of colours.
Revised: Also, the sky and sunlight produce only a limited range of colours.

Original: In addition, illumination environments have a canonical up direction (vertical axis aligns with gravity) but arbitrary horizontal rotation(any rotation about the vertical is equally likely).
Revised: In addition, illumination environments have a canonical up direction (vertical axis aligns with gravity) but arbitrary horizontal rotation(any rotation about the vertical is equally likely).

Original: These regularities and geometric symmetries cansignificantly restrict the space of possible illuminations to constrain inverse problems or enable the synthesis of realistic lighting
Revised: These regularities and geometric symmetries cansignificantly restrict the space of possible illuminations to constrain inverse problems or enable the synthesis of realistic lighting.

Original: Lighting Representations An illumination environment is a spherical signal.
Revised: Lighting representations An illumination environment is a spherical signal.

Original: A relatively small set of alternatives are used for their representation within vision and graphics.
Revised: A relatively small set of alternatives are used for their representation within vision and graphics.

Original: A widely used representation in graphics is an environment map [39, 38, 53, 54], which is a regularly sampled 2D image representing a flattening of the sphere, usually via an equirectangular projection.
Revised: A widely used representation in graphics is an environment map [39, 38, 52, 53], which is a regularly sampled 2D image representing a flattening of the sphere, usually via an equirectangular projection.

Original: However, the projection introduces distortions leading to irregular sampling on the sphere, it is not compact, introduces boundaries and provides no constraint.
Revised: However, the projection introduces distortions leading to irregular sampling on the sphere, it is not compact, introduces boundaries and provides no constraint.

Original: Nevertheless, environment map representations have been used in inverse settings where every pixel in the map is optimised independently [41]
Revised: Nevertheless, environment map representations have been used in inverse settings where every pixel in the map is optimised independently [41]

Original: [Figure]
Revised: [Figure]

Original: Spherical harmonic (SH) lighting [3, 38] is a compact lighting representation commonly used inreal-time computer graphics [38, 22, 46] and inverse rendering [59, 53, 31, 40, 17].
Revised: Spherical harmonic (SH) lighting [3, 38] is a compact lighting representation commonly used inreal-time computer graphics [38, 22, 46] and inverse rendering [57, 52, 31, 40, 17].

Original: While SHs can beused to represent the illumination environment directly, more commonly they represent pre-integratedlighting, i.e. the illumination environment convolved with a bidirectional reflectance distributionfunction (BRDF).
Revised: While SHs can beused to represent the illumination environment directly, more commonly they represent pre-integratedlighting, i.e. the illumination environment convolved with a bidirectional reflectance distributionfunction (BRDF).

Original: When the BRDF is low frequency, as it is for Lambertian diffuse reflectance, thenthe convolution is also low frequency making the approximation with SHs very accurate [3].
Revised: When the BRDF is low frequency, as it is for Lambertian diffuse reflectance, thenthe convolution is also low frequency making the approximation with SHs very accurate [3].

Original: An alternative, growing in popularity, is the Spherical Gaussian (SG) representation [54, 53, 61].
Revised: An alternative, growing in popularity, is the Spherical Gaussian (SG) representation [53, 52, 60].

Original: SGsrepresent a lighting environment as a collection of Gaussian lobes on the sphere, each of which has 6degrees of freedom (three for RGB amplitude, two for spherical direction and one for sharpness).
Revised: SGsrepresent a lighting environment as a collection of Gaussian lobes on the sphere, each of which has 6degrees of freedom (three for RGB amplitude, two for spherical direction and one for sharpness).

Original: While this allows the reconstruction of localised high-frequency features, it still requires many lobesto approximate complex illumination environments.
Revised: While this allows the reconstruction of localised high-frequency features, it still requires many lobesto approximate complex illumination environments.

Original: [31] compares SH and SG for object re-lighting,finding SG was able to recover higher frequency lighting using a similar number of parameters as SH,though both still required a large number of parameters to approximate ground truth.
Revised: [31] compares SH and SG for object re-lighting,finding SG was able to recover higher frequency lighting using a similar number of parameters as SH,though both still required a large number of parameters to approximate ground truth.

Original: Both SHs and SGs are rotation equivariant .
Revised: Both SHs and SGs are rotation equivariant .

Original: A rotation of the illumination environment correspondsdirectly to a rotation of the SH basis or the SG lobe directions.
Revised: A rotation of the illumination environment correspondsdirectly to a rotation of the SH basis or the SG lobe directions.

Original: Equivalently, they can represent anyrotation of a given environment with equal accuracy.
Revised: Equivalently, they can represent anyrotation of a given environment with equal accuracy.

Original: However, they provide no prior over the spaceof possible illuminations.
Revised: However, they provide no prior over the spaceof possible illuminations.

Original: SGs or SHs can represent any colour of light coming from any direction.
Revised: SGs or SHs can represent any colour of light coming from any direction.

Original: Illumination Priors When humans solve inverse rendering tasks, they rely on strong priors over thespace of possible illuminations.
Revised: Illumination Priors When humans solve inverse rendering tasks, they rely on strong priors over thespace of possible illuminations.

Original: For example, they resolve convex/concave shading ambiguities withthe lighting-from-above assumption [26, 52].
Revised: For example, they resolve convex/concave shading ambiguities withthe lighting-from-above assumption [26, 51].

Original: Given this, it is surprising that statistical illuminationpriors have been almost completely ignored in computer vision, with the vast majority of inverserendering techniques allowing arbitrary illumination within their chosen representation space.
Revised: Given this, it is surprising that statistical illuminationpriors have been almost completely ignored in computer vision, with the vast majority of inverserendering techniques allowing arbitrary illumination within their chosen representation space.

Original: There are a small number of exceptions.
Revised: There are a small number of exceptions.

Original: Both Egger et al.
Revised: Both Egger et al.

Original: [17] and Yu and Smith [59] learn alinear statistical model with Gaussian prior in the space of SH coefficients.
Revised: [17] and Yu and Smith [57] learn alinear statistical model with Gaussian prior in the space of SH coefficients.

Original: While providing a usefulconstraint to avoid unrealistic illumination environments, this approach inherits the weakness of
Revised: While providing a usefulconstraint to avoid unrealistic illumination environments, this approach inherits the weakness of

Original: SHs in being unable to reproduce high-frequency lighting effects while also losing the rotationequivariance.
Revised: SHs in being unable to reproduce high-frequency lighting effects while also losing the rotationequivariance.

Original: Yu and Smith [59] seek to overcome this by rotation augmentation at training time,but this brute force approach makes no guarantee of rotation equivariance.
Revised: Yu and Smith [57] seek to overcome this by rotation augmentation at training time,but this brute force approach makes no guarantee of rotation equivariance.

Original: Sztrajman et al.
Revised: Sztrajman et al.

Original: [49]separate environment maps into HDR and LDR components.
Revised: [49]separate environment maps into HDR and LDR components.

Original: Using a CNN-based auto-encoder forestimations of LDR components of lighting alongside a low dimensional SG model for the HDRlighting provided by the sun.
Revised: Using a CNN-based auto-encoder forestimations of LDR components of lighting alongside a low dimensional SG model for the HDRlighting provided by the sun.

Original: They too require data augmentation in the form of rotations, can makeno equivariance guarantees and, due to the low dimensionality of the SG model, struggle to representenvironments with multiple HDR light sources.
Revised: They too require data augmentation in the form of rotations, can makeno equivariance guarantees and, due to the low dimensionality of the SG model, struggle to representenvironments with multiple HDR light sources.

Original: xyz
Revised: xyz

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Rotation-Equivariant Conditional
Revised: Rotation-Equivariant Conditional

Original: [Figure]
Revised: [Figure]

Original: Neural Fields Neural fields [57] have provided impressive results in a range of applicationsincluding representations of objects and scenes [44, 2, 33, 13, 12, 23, 36, 50], in inverse rendering [7,5, 40, 48, 8, 61] and robotics [30, 11, 35].
Revised: Neural Fields Neural fields [56] have provided impressive results in a range of applicationsincluding representations of objects and scenes [44, 2, 33, 13, 12, 23, 36, 50], in inverse rendering [7,5, 40, 48, 8, 60] and robotics [30, 11, 35].

Original: The work most closely related to ours is Neural-PIL [8].
Revised: The work most closely related to ours is Neural-PIL [8].

Original: They use a FiLM SIREN similar to that proposed in Pi-GAN [10], and like us use a direction queryvector and auto-decoder architecture.
Revised: They use a FiLM SIREN similar to that proposed in Pi-GAN [10], and like us use a direction queryvector and auto-decoder architecture.

Original: However they do not have rotation-equivariance and apply nonatural light prior.
Revised: However they do not have rotation-equivariance and apply nonatural light prior.

Original: They also model pre-integrated lighting, conditioning latter layers of their FiLM SIREN on a material roughness parameter.
Revised: During training they interpolate between embeddings and enforce a set of lossesto ensure a smooth, interpretable latent space whilst using an auto-decoder architecture.

Original: Rotation Invariance/Equivariance
Revised: Rotation Invariance/Equivariance

Original: Two important symmetries in computer vision are invarianceand equivariance to the rotation group [9].
Revised: Two important symmetries in computer vision are invarianceand equivariance to the rotation group [9].

Original: Some works attempt to achieve these properties viadata augmentation [59, 37, 49], which still results in missed cases for continuous rotations.
Revised: Some works attempt to achieve these properties viadata augmentation [57, 37, 49], which still results in missed cases for continuous rotations.

Original: Asolution alleviating the need for extensive data augmentation is the Vector Neuron [15], which offersa framework for designing SO(3)-Equivariant networks via a latent matrix representation rather thana latent vector.
Revised: Asolution alleviating the need for extensive data augmentation is the Vector Neuron [15], which offersa framework for designing SO(3)-Equivariant networks via a latent matrix representation rather thana latent vector.

Original: This allows a direct mapping of rotations applied to the network’s input to its output,resulting in all possible rotations  being explicitly represented via rotations of the latent codes.
Revised: This allows a direct mapping of rotations applied to the network’s input to its output. All possible rotations now being equally well represented implicitly in rotations of the latent codes.

Original: Contribution We desire a representation of natural illumination environments that offers thefollowing features.
Revised: Contribution We desire a representation of natural illumination environments that offers thefollowing features.

Original: Generative: A generative model that captures the statistical regularities ofnatural illumination with a well-behaved latent space within which we can optimise to solve inverseproblems.
Revised: Generative: A generative model that captures the statistical regularities ofnatural illumination with a well-behaved latent space within which we can optimise to solve inverseproblems.

Original: Compact: Reduces the dimensionality of inverse problems while preserving high-frequencylighting effects that are important for non-Lambertian appearance.
Revised: Compact: Reduces the dimensionality of inverse problems while preserving high-frequencylighting effects that are important for non-Lambertian appearance.

Original: Rotation-Equivariant: Respect thecanonical orientation, i.e. any rotation of an environment about the vertical should be equally likelyand equally well represented.
Revised: Rotation-Equivariant: Respect thecanonical orientation, i.e. any rotation of an environment about the vertical should be equally likelyand equally well represented.

Original: Statistical Prior: Provides a prior to regularise inverse problems orthat can be sampled from for synthesis, only generating plausible illumination environments.
Revised: Statistical Prior: Provides a prior to regularise inverse problems orthat can be sampled from for synthesis, only generating plausible illumination environments.

Original: HDR:
Revised: HDR:

Original: Correctly handle HDR quantities essential for realistic rendering and the reproduction of natural light.
Revised: Correctly handle HDR quantities essential for realistic rendering and the reproduction of natural light.

Original: We introduce RENI - A Rotation-Equivariant Natural Illumination model.
Revised: We introduce RENI - A Rotation-Equivariant Natural Illumination model.

Original: Our key contributions are:
Revised: Our key contributions are:

Original: – An extension of Vector Neurons to a rotation-equivariant neural field representation for
Revised: – An extension of Vector Neurons to a rotation-equivariant neural field representation for

Original: spherical images, optionally restricted to rotations about the vertical axis.
Revised: spherical images, optionally restricted to rotations about the vertical axis.

Original: – A variational autodecoder architecture for a generative model of spherical signals.
Revised: – A variational autodecoder architecture for a generative model of spherical signals.

Original: – The first natural, outdoor HDR illumination model.
Revised: – The first natural, outdoor HDR illumination model.

Original: – Evaluated in an inverse rendering task showing significant performance improvements over
Revised: – Evaluated in an inverse rendering task showing significant performance improvements over

Original: other lighting representations.
Revised: other lighting representations.

Original: We choose to model scene radiance, i.e. environment lighting, directly as opposed to pre-integratedlighting with a particular BRDF.
Revised: We choose to model scene radiance, i.e. environment lighting, directly as opposed to pre-integratedlighting with a particular BRDF.

Original: This makes our model more general since it can be used witharbitrary BRDFs at inference time or even for tasks other than rendering such as to constrain shapefrom specular flow.
Revised: This makes our model more general since it can be used witharbitrary BRDFs at inference time or even for tasks other than rendering such as to constrain shapefrom specular flow.

Original:  Rotation-Equivariant Conditional Spherical Neural Fields
Revised:  Rotation-Equivariant Conditional Spherical Neural Fields

Original: We wish to construct a generative model of spherical signals that is rotation equivariant with respect tothe latent representation of the signal.
Revised: We wish to construct a generative model of spherical signals that is rotation equivariant with respect tothe latent representation of the signal.

Original: That is to say, a rotation of the latent representation correspondsto a rotation of a spherical signal and a signal can be reconstructed with exactly the same accuracyin any rotation.
Revised: That is to say, a rotation of the latent representation correspondsto a rotation of a spherical signal and a signal can be reconstructed with exactly the same accuracyin any rotation.

Original: We begin by proposing a variant of Vector Neurons [15] for SO (3) equivariantrepresentation of spherical signals.
Revised: We begin by proposing a variant of Vector Neurons [15] for SO (3) equivariantrepresentation of spherical signals.

Original: To achieve this, we use an ordered list of 3D vectors for our latent representation.
Revised: To achieve this, we use an ordered list of 3D vectors for our latent representation.

Original: Our model takesthe form of a conditional spherical neural field, f : S 2 × R 3 × N → R M , such that f ( d , Z ) computesthe value of the signal represented by latent code Z ∈ R 3 × N in direction d ∈ R 3 , with ∥ d ∥ = 1 .
Revised: Our model takesthe form of a conditional spherical neural field, f : S 2 × R 3 × N → R M , such that f ( d , Z ) computesthe value of the signal represented by latent code Z ∈ R 3 × N in direction d ∈ R 3 , with ∥ d ∥ = 1 .

Original: For colour images, M = 3 and f outputs an RGB colour.
Revised: For colour images, M = 3 and f outputs an RGB colour.

Original: By using a spherical neural field, we areagnostic to how the signals are sampled on the sphere.
Revised: By using a spherical neural field, we areagnostic to how the signals are sampled on the sphere.

Original: We can generate any sampling simply bychoosing the grid of directions as appropriate.
Revised: We can generate any sampling simply bychoosing the grid of directions as appropriate.

Original: We also avoid boundary effects since our domain iscontinuous.
Revised: We also avoid boundary effects since our domain iscontinuous.

Original: We construct the neural field such that it is invariant to a rotation of both d and Z simultaneously(i.e.
Revised: We construct the neural field such that it is invariant to a rotation of both d and Z simultaneously(i.e.

Original: f ( Rd , RZ ) =
Revised: f ( Rd , RZ ) =

Original: f ( d , Z ) with R ∈ SO (3) ).
Revised: f ( d , Z ) with R ∈ SO (3) ).

Original: This entails that the neural field is equivariant withrespect to a rotation of Z only (i.e. rotating Z corresponds to rotating the spherical signal such thatf ( d , RZ ) =
Revised: This entails that the neural field is equivariant withrespect to a rotation of Z only (i.e. rotating Z corresponds to rotating the spherical signal such thatf ( d , RZ ) =

Original: f ( R ⊤ d , Z ) ).
Revised: f ( R ⊤ d , Z ) ).

Original: Equivariant Transformation Key to our approach is a transformation of the inputs to the neuralfield, ( d , Z ) , such that they are rotation invariant.
Revised: Equivariant Transformation Key to our approach is a transformation of the inputs to the neuralfield, ( d , Z ) , such that they are rotation invariant.

Original: These divide into two parts: 1.
Revised: These divide into two parts: 1.

Original: d ′ - the directionalinput to the spherical neural field and 2.
Revised: d ′ - the directionalinput to the spherical neural field and 2.

Original: Z ′ - the latent code on which the neural field is conditioned.
Revised: Z ′ - the latent code on which the neural field is conditioned.

Original: The direction in which we wish to evaluate the spherical neural field must be encoded relative to thelatent code in the particular rotation in which we encounter it.
Revised: The direction in which we wish to evaluate the spherical neural field must be encoded relative to thelatent code in the particular rotation in which we encounter it.

Original: This is satisfied by using the innerproduct ⟨ d , Z ⟩ , i.e. the matrix-vector product d ′ =
Revised: This is satisfied by using the innerproduct ⟨ d , Z ⟩ , i.e. the matrix-vector product d ′ =

Original: Z ⊤ d ∈ R N .
Revised: Z ⊤ d ∈ R N .

Original: Unlike Vector Neurons, our input isa direction not a position.
Revised: Unlike Vector Neurons, our input isa direction not a position.

Original: Hence, the rotation invariant feature ∥ d ∥ conveys no information and wedo not use it.
Revised: Hence, the rotation invariant feature ∥ d ∥ conveys no information and wedo not use it.

Original: For the latent code, we follow the Vector Neurons invariant layer: Z ′ =
Revised: For the latent code, we follow the Vector Neurons invariant layer: Z ′ =

Original: VN-In ( Z ) .
Revised: VN-In ( Z ) .

Original: However, weuse the full Gram matrix Z ⊤ Z since we expect the dimensionality of the latent space, N , to remainmoderate for our data.
Revised: However, weuse the full Gram matrix Z ⊤ Z since we expect the dimensionality of the latent space, N , to remainmoderate for our data.

Original: If this O ( N 2 ) size becomes problematic, we can use the same scalable solutionused by Vector Neurons [15].
Revised: If this O ( N 2 ) size becomes problematic, we can use the same scalable solutionused by Vector Neurons [15].

Original: Since our neural field is equivariant we do not need to augment our training data over the spaceof rotations .
Revised: Since our neural field is equivariant we do not need to augment our training data over the spaceof rotations .

Original: Observing a spherical signal once means we can reconstruct it with the same accuracyin any rotation.
Revised: Observing a spherical signal once means we can reconstruct it with the same accuracyin any rotation.

Original: Variational Auto-decoder We train our conditional spherical neural field as a decoder-only architecture, i.e. an auto-decoder [36].
Revised: Variational Auto-decoder We train our conditional spherical neural field as a decoder-only architecture, i.e. an auto-decoder [36].

Original: This means that we optimise the network weights simultaneouslywith the latent codes for each training sample.
Revised: This means that we optimise the network weights simultaneouslywith the latent codes for each training sample.

Original: This avoids the need to design a rotation-equivariantencoder while, for inverse tasks, only the decoder is needed so we avoid the redundancy of alsotraining an encoder.
Revised: This avoids the need to design a rotation-equivariantencoder while, for inverse tasks, only the decoder is needed so we avoid the redundancy of alsotraining an encoder.

Original: However, training with no regularisation on the learnt latent space does not leadto a space that is smooth or that follows a known distribution.
Revised: However, training with no regularisation on the learnt latent space does not leadto a space that is smooth or that follows a known distribution.

Original: This means it cannot be sampled from,does not produce meaningful interpolations and provides no prior for inverse problems.
Revised: This means it cannot be sampled from,does not produce meaningful interpolations and provides no prior for inverse problems.

Original: For thisreason, we use a variational auto-decoder [60] architecture.
Revised: For thisreason, we use a variational auto-decoder [59] architecture.

Original: Each training sample is represented by amean, µ i ∈ R 3 N , and standard deviation, σ i ∈ R 3 N , that provide the parameters of a normal distribution from which the flattened latent code for that training sample is drawn: vec ( Z i ) ∼ N ( µ i , Σ i ) ,where Σ i = diag ( σ 2 i, 1 , . . .
Revised: Each training sample is represented by amean, µ i ∈ R 3 N , and standard deviation, σ i ∈ R 3 N , that provide the parameters of a normal distribution from which the flattened latent code for that training sample is drawn: vec ( Z i ) ∼ N ( µ i , Σ i ) ,where Σ i = diag ( σ 2 i, 1 , . . .

Original: , σ 2 i, 3 N ) is the diagonal covariance matrix.
Revised: , σ 2 i, 3 N ) is the diagonal covariance matrix.

Original: Using the reparameterisationtrick [28], we can generate a latent code vec ( Z i ) =
Revised: Using the reparameterisationtrick [28], we can generate a latent code vec ( Z i ) =

Original: µ i + σ
Revised: µ i + σ

Original: i ⊙ ϵ where the noise is sampled asϵ ∼ N ( 0 , I 3 N ) .
Revised: i ⊙ ϵ where the noise is sampled asϵ ∼ N ( 0 , I 3 N ) .

Original: During training we optimise µ i and σ i for each training sample and use the
Revised: During training we optimise µ i and σ i for each training sample and use the

Original: Kullback–Leibler divergence as a loss to regularise the distribution of each latent code toward thestandard normal distribution:
Revised: Kullback–Leibler divergence as a loss to regularise the distribution of each latent code toward thestandard normal distribution:

Original: [Equation]
Revised: [Equation]

Original: where K is the number of training samples.
Revised: where K is the number of training samples.

Original:  RENI: A Statistical Model of Natural Illumination
Revised:  RENI: A Statistical Model of Natural Illumination

Original: We now describe how to construct a statistical model of natural illumination environments as arestricted version of a rotation-equivariant conditional spherical neural field.
Revised: We now describe how to construct a statistical model of natural illumination environments as arestricted version of a rotation-equivariant conditional spherical neural field.

Original: We call our model RENI(Rotation-Equivariant Natural Illumination).
Revised: We call our model RENI(Rotation-Equivariant Natural Illumination).

Original: In contrast to the full SO (3) -equivariant formulation in
Revised: In contrast to the full SO (3) -equivariant formulation in

Original: Section 2, we propose a variant with only SO (2) equivariance and explain in detail our training data,losses and implementation.SO(2)
Revised: Section 2, we propose a variant with only SO (2) equivariance and explain in detail our training data,losses and implementation.

Original: Equivariance Natural environments have a canonical “up” direction (defined by gravity) butarbitrary rotation about this vertical axis.
Revised: SO(2) equivariance Natural environments have a canonical “up” direction (defined by gravity) butarbitrary rotation about this vertical axis.

Original: For this reason, we do not want arbitrary SO (3) rotationequivariance.
Revised: For this reason, we do not want arbitrary SO (3) rotationequivariance.

Original: This would have the undesirable effect of permitting unnatural environment orientations(such as with the sky at the bottom), providing a less useful prior when solving inverse problems.
Revised: This would have the undesirable effect of permitting unnatural environment orientations(such as with the sky at the bottom), providing a less useful prior when solving inverse problems.

Original: So,we propose a restricted transformation of the neural field inputs that are invariant only to rotations,
Revised: So,we propose a restricted transformation of the neural field inputs that are invariant only to rotations,

Original: R y , about the vertical ( y ) axis.
Revised: R y , about the vertical ( y ) axis.

Original: In order to construct the invariant features, we use two selection matrices: S xz selects the componentsthat are orthogonal to the axis of rotation (i.e. the x and z components) and are thus affected by a y -axisrotation, s y selects the unaffected y component: S xz =
Revised: In order to construct the invariant features, we use two selection matrices: S xz selects the componentsthat are orthogonal to the axis of rotation (i.e. the x and z components) and are thus affected by a y -axisrotation, s y selects the unaffected y component: S xz =

Original: (cid:20) 1 00 0(cid:21) , s y =
Revised: (cid:20) 1 00 0(cid:21) , s y =

Original: The directional part of the invariant input now contains three components: d ′ =
Revised: The directional part of the invariant input now contains three components: d ′ =

Original: ( s y d , ⟨ S xz d , S xz Z ⟩ , ∥ S xz d ∥ ) .
Revised: ( s y d , ⟨ S xz d , S xz Z ⟩ , ∥ S xz d ∥ ) .

Original: The first is the invariant component of d .
Revised: The first is the invariant component of d .

Original: The second encodes d relative to Z in the x - z plane.
Revised: The second encodes d relative to Z in the x - z plane.

Original: Thethird measures the norm of d projected into the x - z plane which is unchanged by rotations about y .
Revised: Thethird measures the norm of d projected into the x - z plane which is unchanged by rotations about y .

Original: The conditioning part of the invariant input now contains two components: VN-In ( Z ) =
Revised: The conditioning part of the invariant input now contains two components: VN-In ( Z ) =

Original: Z ′ =( s y Z , ( S xz Z )
Revised: Z ′ =( s y Z , ( S xz Z )

Original: ⊤ S xz Z ) .
Revised: ⊤ S xz Z ) .

Original: The first is simply the invariant component of each column of Z .
Revised: The first is simply the invariant component of each column of Z .

Original: The secondis the invariant transformation (Gram matrix) of latent vectors projected into the x - z plane.
Revised: The secondis the invariant transformation (Gram matrix) of latent vectors projected into the x - z plane.

Original: Reconstruction Loss and HDR HDR imaging is essential for realistic rendering and enables theaccurate representation of the full dynamic range of natural light.
Revised: Reconstruction Loss and HDR HDR imaging is essential for realistic rendering and enables theaccurate representation of the full dynamic range of natural light.

Original: Therefore our model must learn an
Revised: Therefore our model must learn an

Original: HDR representation of natural illumination.
Revised: HDR representation of natural illumination.

Original: Computing a reconstruction loss in linear HDR space isdominated by large values and leads to poor reconstruction of most of the environment.
Revised: Computing a reconstruction loss in linear HDR space isdominated by large values and leads to poor reconstruction of most of the environment.

Original: Therefore,similar to [18], we train our network to output log ( HDR ) values and compute the reconstruction lossin log space.
Revised: Therefore,similar to [18], we train our network to output log ( HDR ) values and compute the reconstruction lossin log space.

Original: We further normalise the log ( HDR ) values to the range [ − 1 , 1] by scaling and shiftingusing the maximum and minimum values across the whole training set.
Revised: We further normalise the log ( HDR ) values to the range [ − 1 , 1] by scaling and shiftingusing the maximum and minimum values across the whole training set.

Original: Each training sample comprises P pairs of directions and corresponding normalised log ( HDR ) RGBcolours that we store in the matrices
Revised: Each training sample comprises P pairs of directions and corresponding normalised log ( HDR ) RGBcolours that we store in the matrices

Original: D i =
Revised: D i =

Original: [ d i 1 , . . .
Revised: [ d i 1 , . . .

Original: , d iP ] ∈ R 3 × P and C i
Revised: , d iP ] ∈ R 3 × P and C i

Original: =
Revised: =

Original: [ c i 1 , . . .
Revised: [ c i 1 , . . .

Original: , c iP ] ∈ R 3 ×
Revised: , c iP ] ∈ R 3 ×

Original: Prespectively.
Revised: Prespectively.

Original: RENI is agnostic to the resolution and sampling of the spherical signal.
Revised: RENI is agnostic to the resolution and sampling of the spherical signal.

Original: The neuralfield can be queried for any direction.
Revised: The neuralfield can be queried for any direction.

Original: In practice, our dataset contains spherical images in anequirectangular sampling containing P = 2 H 2 pixels where H is the height of the images.
Revised: In practice, our dataset contains spherical images in anequirectangular sampling containing P = 2 H 2 pixels where H is the height of the images.

Original: Sinceequirectangular images are irregularly sampled, we weight the mean squared reconstruction loss bythe sine of the polar angle, θ ( d ) :[Equation]
Revised: Sinceequirectangular images are irregularly sampled, we weight the mean squared reconstruction loss bythe sine of the polar angle, θ ( d ) :

Original: [Equation]
Revised: [Equation]

Original: Neural Field Our neural field, f , is implemented as a SIREN [44] - an MLP with periodic activationfunction.
Revised: Neural Field Our neural field, f , is implemented as a SIREN [44] - an MLP with periodic activationfunction.

Original: This architecture has proven highly effective at representing a wide range of complexnatural signals.
Revised: This architecture has proven highly effective at representing a wide range of complexnatural signals.

Original: In order to condition the SIREN on the invariant latent code Z ′ , we use conditioningby-concatenation [44, 57], i.e. we simply input both d ′ and Z ′ to the network.
Revised: In order to condition the SIREN on the invariant latent code Z ′ , we use conditioningby-concatenation [44, 56], i.e. we simply input both d ′ and Z ′ to the network.

Original: We also tested a
Revised: We also tested a

Original: FiLM-conditioned SIREN [10] but could not get significant performance improvements and, perhapsdue to the small size of our dataset, FiLM conditioning resulted in a non-smooth latent space thatcould drastically affect performance when fitting to unseen images.
Revised: FiLM-conditioned SIREN [10] but could not get significant performance improvements and, perhapsdue to the small size of our dataset, FiLM conditioning resulted in a non-smooth latent space thatcould drastically affect performance when fitting to unseen images.

Original: Training Data We have curated a dataset of 1,694 HDR equirectangular images of outdoor, naturalillumination environment obtained with either CC0 1.0 Universal Public Domain Dedication license[24, 19, 27] or with written permission to redistribute a low-resolution version of their dataset [25,1, 45, 51].
Revised: Training Data We have curated a dataset of 1,694 HDR equirectangular images of outdoor, naturalillumination environment obtained with either CC0 1.0 Universal Public Domain Dedication license[24, 19, 27] or with written permission to redistribute a low-resolution version of their dataset [25,1, 45].

Original: All images were then checked to ensure they did not contain any personally identifiableinformation or offensive content and any images that contained predominantly unnatural light sources were removed.
Revised: All images were then checked to ensure they did not contain any personally identifiableinformation or offensive content and any images that contained predominantly unnatural light sources were removed.

Original: 21 images were also selected and held back for optimising only the latent codes attest time, resulting in a training dataset of 1,673 HDR images.
Revised: 21 images were also selected and held back for optimising only the latent codes attest time, resulting in a training dataset of 1,673 HDR images.

Original: Training We use Adam [29] to optimise the sum of the reconstruction and KLD losses:
Revised: Training We use Adam [29] to optimise the sum of the reconstruction and KLD losses:

Original: [Equation]
Revised: [Equation]

Original: where β is a hyperparameter to weight the KLD loss and D = 3 N is the dimensionality of the latentspace, used to normalise for the choice of N .
Revised: where β is a hyperparameter to weight the KLD loss and D = 3 N is the dimensionality of the latentspace, used to normalise for the choice of N .

Original: We randomly initialise the mean latent code for eachimage, µ i , from a standard normal distribution.
Revised: We randomly initialise the mean latent code for eachimage, µ i , from a standard normal distribution.

Original: In order to ensure the positivity of the variances, σ 2 i,j ,we optimise log( σ 2 i,j ) which we initialise randomly from a normal distribution with mean -5 andvariance 1.
Revised: In order to ensure the positivity of the variances, σ 2 i,j ,we optimise log( σ 2 i,j ) which we initialise randomly from a normal distribution with mean -5 andvariance 1.

Original: We use all the pixels from a single training image as a mini-batch and use batch size 1.
Revised: We use all the pixels from a single training image as a mini-batch and use batch size 1.

Original: In order to speed up training, we employ a progressive strategy.
Revised: In order to speed up training, we employ a progressive strategy.

Original: We start by training with lowresolution ( H = 16 ) images and double resolution every 800 epochs until we reach H
Revised: We start by training with lowresolution ( H = 16 ) images and double resolution every 800 epochs until we reach H

Original: =
Revised: =

Original: 128resolution for a total of 2,400 epochs.
Revised: 128resolution for a total of 2,400 epochs.

Original: This allows the network to quickly learn low-frequencyfeatures, gradually obtaining higher frequency details later in the training.
Revised: This allows the network to quickly learn low-frequencyfeatures, gradually obtaining higher frequency details later in the training.

Original: Unlike with a convolutionalarchitecture, the spatially-continuous nature of the neural field means we can implement multiresolution training using a single network.
Revised: Unlike with a convolutionalarchitecture, the spatially-continuous nature of the neural field means we can implement multiresolution training using a single network.

Original: We simply upsample the grid of directions, D i , andincrease the resolution of the target equirectangular images to match this resolution.
Revised: We simply upsample the grid of directions, D i , andincrease the resolution of the target equirectangular images to match this resolution.

Original: We used Weights and Biases [6] for experiment tracking, visualisations and hyper-parameter gridsearch.
Revised: We used Weights and Biases [6] for experiment tracking, visualisations and hyper-parameter gridsearch.

Original: This yielded the best performance using a variational auto-decoder, SIREN with 5 layerseach with 128 hidden features.
Revised: This yielded the best performance using a variational auto-decoder, SIREN with 5 layerseach with 128 hidden features.

Original: We use an exponentially decaying learning rate starting at 10 − 5and decreasing to 10 − 7 by the end of training and weight the KLD loss β
Revised: We use an exponentially decaying learning rate starting at 10 − 5and decreasing to 10 − 7 by the end of training and weight the KLD loss β

Original: = 10 − 4 to be of similarmagnitude to the reconstruction loss.
Revised: = 10 − 4 to be of similarmagnitude to the reconstruction loss.

Original: Training RENI with a latent code dimension of D = 27 , tookaround 15 hours ona local Nvidia A40 48GB GPU.
Revised: Training RENI with a latent code dimension of D = 27 , tookaround 15 hours on local Nvidia A40 48GB GPUs.

Original: Model Fitting
Revised: Model fitting

Original: At test time, the SIREN network is held static, and only latent codes are optimisedin order to fit to an unseen image.
Revised: At test time, the SIREN network is held static, and only latent codes are optimisedin order to fit to an unseen image.

Original: We initialise the latent code to zeros, corresponding to the meanenvironment map (see Figure 1, left).
Revised: We initialise the latent code to zeros, corresponding to the meanenvironment map (see Figure 1, left).

Original: This provides an unbiased initialisation in the absence of anyprior information about the environment.
Revised: This provides an unbiased initialisation in the absence of anyprior information about the environment.

Original: We found performance on test images was improved byincluding a cosine similarity loss L Cosine on the RGB colour vectors [32] weighted using the samesin( θ ( d )) weight as used in L recon .
Revised: We found performance on test images was improved byincluding a cosine similarity loss L Cosine on the RGB colour vectors [32] weighted using the samesin( θ ( d )) weight as used in L recon .

Original: We also include a prior loss on the latent vector L prior =
Revised: We also include a prior loss on the latent vector L prior =

Original: ∥
Revised: ∥

Original: Z ∥ 2 Fro .
Revised: Z ∥ 2 Fro .

Original: Our test time loss is therefore:
Revised: Our test time loss is therefore:

Original: [Equation]
Revised: [Equation]

Original: We again use multi-resolution fitting with the same resolutions and number of epochs as used duringtraining.
Revised: We again use multi-resolution fitting with the same resolutions and number of epochs as used duringtraining.

Original: A hyper-parameter grid search found the best performance when using ρ = 10 − 4 , γ = 10 − 7 ,and an exponentially decaying learning rate starting at 10 − 2 and decreasing to 10 − 4 .
Revised: A hyper-parameter grid search found the best performance when using ρ = 10 − 2 , γ = 10 − 7 ,and an exponentially decaying learning rate starting at 10 − 2 and decreasing to 10 − 4 .

Original: Trained models and a PyTorch implementation will be made available.
Revised: Trained modelsand a PyTorch implementation will be made available.

Original:  Evaluation
Revised:  Evaluation

Original: [Table]
Revised: [Table]

Original: Generalisation We begin by evaluatingthe generalisation performance of RENIwhen approximating unseen environments.
Revised: Generalisation We begin by evaluatingthe generalisation performance of RENIwhen approximating unseen environments.

Original: We compare against SH and SG and explorehow generalisation performance varies as afunction of latent code dimension.
Revised: We compare against SH and SG and explorehow generalisation performance varies as afunction of latent code dimension.

Original: We consider RGB spherical harmonics of order 2,5, 6 and 9 equal to latent code dimension
Revised: We consider RGB spherical harmonics of order 2,5, 6 and 9 equal to latent code dimension

Original: D = 3 × N for N = 9 , 36 , 49 , 100 .
Revised: D = 3 × N for N = 9 , 36 , 49 , 100 .

Original: Since
Revised: Since

Original: SG requires a dimensionality that is a multiple of 6, where D is not a multiple of 6, webias in SGs favour by using the next multiple, i.e. ⌈ D/ 6 ⌉ · 6 .
Revised: SG requires a dimensionality that is an multiple of 12, where D is not a multiple of12, we bias in SGs favour by using the nextmultiple, i.e. ⌈ D/ 12 ⌉ · 12 .

Original: We used an open sourceimplementation of per-pixel environment map fitting provided by [31] to fit our SG models.
Revised: We used an opensource implementation of per-pixel environment map fitting provided by [31] to fit our SG models.

Original: As
Revised: Both SH and SG were fit to the same sine-weighted and scaled log( HDR ) equirectangular images as

Original: G r ound
Revised: G r ound

Original: T r u t h
Revised: T r u t h

Original: G r ound
Revised: G r ound

Original: T r u t h
Revised: T r u t h

Original: RENI
Revised: RENI

Original: SH
Revised: SH

Original: SG
Revised: SG

Original: RENI
Revised: RENI

Original: SH
Revised: SH

Original: SG
Revised: SG

Original: RENI
Revised: RENI

Original: SH
Revised: SH

Original: SG
Revised: SG

Original: RENI
Revised: RENI

Original: SH
Revised: SH

Original: SG  shown in Figure 3, RENI can capture higher frequency detail than both SH and SG, is less dominatedby high value pixels and can reproduce accurate HDR values.
Revised: SG RENI. As shown in Figure 3, RENI can capture higher frequency detail than both SH and SG, is lessdominated by high value pixels and can reproduce accurate HDR values.

Original: A comparison of the mean PSNRacross the test set images for increasing latent code dimensionality is shown in Table 1.
Revised: A comparison of the mean PSNR across the test set images for increasing latent code dimensionality is shown in Table 1.

Original: Latent Space Interpolation As shown in Figure 4, linear interpolations between codes result insmooth transitions between images and plausible natural environments for all intermediate latentcodes showing how RENI encodes a meaningful internal representation of natural illumination.
Revised: Latent Space Interpolation As shown in Figure 4, linear interpolations between codes result insmooth transitions between images and plausible natural environments for all intermediate latentcodes showing how RENI encodes a meaningful internal representation of natural illumination.

Original: Environment Completion Any picture of a natural scene contains cues about the surroundingenvironment that the scene was captured in, such as likely sun locations and possible environmentalcontent.
Revised: Environment Completion Any picture of a natural scene contains cues about the surroundingenvironment that the scene was captured in, such as likely sun locations and possible environmentalcontent.

Original: If RENI is only provided with a small portion of the complete environment map in its lossat test time, RENI can hallucinate plausible completions of the environment.
Revised: If RENI is only provided with a small portion of the complete environment map in its lossat test time, RENI can hallucinate plausible completions of the environment.

Original: As shown in Figure 5,
Revised: As shown in Figure 5,

Original: RENI makes sensible estimations about the possible colours and shapes of land and sky and oftenpredicts quite accurate sun locations despite the sun being outside the image crop.
Revised: RENI makes sensible estimations about the possible colours and shapes of land and sky and oftenpredicts quite accurate sun locations despite the sun being outside the image crop.

Original: Inverse Rendering To test the performance of RENI in an inverse rendering pipeline, we implemented a normalised Blinn-Phong environment map shader in PyTorch3D, enabling fully differentiable rendering.
Revised: Inverse Rendering To test the performance of RENI in an inverse rendering pipeline, we implemented a normalised Blinn-Phong environment map shader in PyTorch3D, enabling fully differentiable rendering.

Original: We render a 3D object with fixed geometry, pose, camera and material parameterssuch that only lighting in the scene is unknown.
Revised: We render a 3D object with fixed geometry, pose, camera and material parameterssuch that only lighting in the scene is unknown.

Original: Optimising only latent codes, we minimise the samelosses as used in L Test , without the sin( θ ( d )) weighting needed for equirectangular images, betweena rendering using a ground truth environment map and a one using the output of RENI.
Revised: Optimising only latent codes, we minimise the samelosses as used in L Test , without the sin( θ ( d )) weighting needed for equirectangular images, betweena rendering using a ground truth environment map and a one using the output of RENI.

Original: This wastested for incremental increases in weighting of the Blinn-Phong specular term ( K s ) , from K s = 0to
Revised: This wastested for incremental increases in weighting of the Blinn-Phong specular term ( K s ) , from K s = 0to

Original: K s = 1 . 0 in steps of 0 . 2 .
Revised: K s = 1 . 0 in steps of 0 . 2 .

Original: A normalisation factor α = ( n + 2) / (4 π (2 − e ( − n 2 )))
Revised: A normalisation factor α = ( n + 2) / (4 π (2 − e ( − n 2 )))

Original: [21], was appliedto K s to get a steady transition from diffuse to specular.
Revised: [21], was appliedto K s to get a steady transition from diffuse to specular.

Original: We achieved the best performance usingα = 10 3 , γ = 10 − 4 , and an exponentially decaying learning rate starting at 10 − 2 and decreasing to10 − 4 over 2,400 epochs.
Revised: We achieved the best performance usingα = 10 3 , γ = 10 − 4 , and an exponentially decaying learning rate starting at 10 − 2 and decreasing to10 − 4 over 2,400 epochs.

Original: We use an environment map resolution of H = 64 and render the objectwith a resolution of 128 2 .
Revised: We use an environment map resolution of H = 64 and render the objectwith a resolution of 128 2 .

Original: In Figure 6 we compare against SH environment maps computed in closedform using linear least squares for increasing specularity.
Revised: In Figure 6 we compare against SH environment maps computed in closedform using linear least squares for increasing specularity.

Original: As expected, SH performs well at producingaccurate renders of diffuse objects.
Revised: As expected, SH performs well at producingaccurate renders of diffuse objects.

Original: However, the environment maps produced are unnatural, and asthe specular term increases in weight, SH degrades in performance significantly compared to RENI.
Revised: However, the environment maps produced are unnatural, and asthe specular term increases in weight, SH degrades in performance significantly compared to RENI.

Original: [Figure]
Revised: [Figure]

Original:  Discussion and Conclusion
Revised:  Discussion and Conclusion

Original: We introduced rotation-equivariant spherical neural fields and used them to create RENI, a naturalillumination prior.
Revised: We introduced rotation-equivariant spherical neural fields and used them to create RENI, a naturalillumination prior.

Original: Demonstrating how random samples from RENI always produce plausibleillumination maps and its use for environment completion and inverse rendering.
Revised: Demonstrating how random samples from RENI always produce plausibleillumination maps and its use for environment completion and inverse rendering.

Original: There are manyexciting avenues for future research, for example, implementing RENI in larger inverse renderingpipelines where it could be a simple drop-in replacement for SH and using RENI for LDR to HDR image reconstruction.
Revised: There are manyexciting avenues for future research, for example, implementing RENI in larger inverse renderingpipelines where it could be a simple drop-in replacement for SH and using RENI for LDR to HDR image reconstruction.

Original: Furthermore, our spherical neural field is the initial step towards a Generative
Revised: Furthermore, our spherical neural field is the initial step towards a Generative

Original: Adversarial Network [20] (GAN) for spherical signals and could be used as the generator alongside a
Revised: Adversarial Network [20] (GAN) for spherical signals and could be used as the generator alongside a

Original: Spherical CNN [14] for the discriminator in a GAN framework.
Revised: Spherical CNN [14] for the discriminator in a GAN framework.

Original: Limitations Using the Gram matrix for the invariant representation, which is O ( n 2 ) , limits the sizeof the latent code you can realistically use.
Revised: Limitations Using the Gram matrix for the invariant representation, which is O ( n 2 ) , limits the sizeof the latent code you can realistically use.

Original: This could be addressed through the use of the invariantlayer proposed by [15] to reduce this to O ( n ) .
Revised: This could be addressed through the use of the invariantlayer proposed by [15] to reduce this to O ( n ) .

Original: The very highest frequency detail is still not resolvedeven when using larger latent code sizes; scaling up with a larger dataset and network size is likely toaddress this.
Revised: The very highest frequency detail is still not resolvedeven when using larger latent code sizes; scaling up with a larger dataset and network size is likely toaddress this.

Original: Because RENI has a prior for natural illumination, unlike SH, RENI’s performance decreases when fitto indoor scenes.
Revised: Because RENI has a prior for natural illumination, unlike SH, RENI’s performance decreases when fitto indoor scenes.

Original: Interestingly, RENI still generalises to indoor scenes quite well; see supplementarymaterial for examples of RENI fitting to indoor scenes.
Revised: Interestingly, RENI still generalises to indoor scenes quite well; see supplementarymaterial for examples of RENI fitting to indoor scenes.

Original: There are complex interactions between illumination, geometry and texture priors in human vision.
Revised: There are complex interactions between illumination, geometry and texture priors in human vision.

Original: For example, the Hollow Face Illusion [26] arises from face geometry priors overriding the lighting from above illumination prior; while in the Bas relief ambiguity [4], geometric priors causeincorrect lighting estimation.
Revised: For example, the Hollow Face Illusion [26] arises from face geometry priors overriding the lighting from above illumination prior; while in the Bas relief ambiguity [4], geometric priors causeincorrect lighting estimation.

Original: Our model discounts these interactions, learning an illumination priorindependently but not its interactions with other cues.
Revised: Our model discounts these interactions, learning an illumination priorindependently but not its interactions with other cues.

Original: We strictly allow only SO (2) equivariance.
Revised: We strictly allow only SO (2) equivariance.

Original: However, considering typical camera coordinate systems,the up axis will sometimes not align with gravity when the camera is pointed up or down.
Revised: However, considering typical camera coordinate systems,the up axis will sometimes not align with gravity when the camera is pointed up or down.

Original: For inverseproblems, this would mean the gravity vector would need to be explicitly estimated (an accelerometerwould resolve this).
Revised: For inverseproblems, this would mean the gravity vector would need to be explicitly estimated (an accelerometerwould resolve this).

Original: Alternatively, we could build our model with full SO (3) equivariance but thenlearn a prior over the space of camera poses relative to gravity.
Revised: Alternatively, we could build our model with full SO (3) equivariance but thenlearn a prior over the space of camera poses relative to gravity.

Original: Broader Impact The illumination prior presented here has the potential to improve the performanceof inverse rendering pipelines, of which there are both positive and negative downstream use casesthat should be considered; for example, the generation of highly realistic 3D models of a person’slikeness without their consent.
Revised: Broader Impact The illumination prior presented here has the potential to improve the performanceof inverse rendering pipelines, of which there are both positive and negative downstream use casesthat should be considered; for example, the generation of highly realistic 3D models of a person’slikeness without their consent.

Original: Advances in inverse and neural rendering could potentially lead toemployment loss in the generation of 3D assets.
Revised: Advances in inverse and neural rendering could potentially lead toemployment loss in the generation of 3D assets.

Original: However, we are optimistic that the democratisationof 3D model generation will be a value-generating technology for the large majority.
Revised: However, we are optimistic that the democratisationof 3D model generation will be a value-generating technology for the large majority.

Original: The illuminationprior might be biased since most of the available and used HDR images were captured in Europe. 
Revised: The illuminationprior might be biased since most of the available and used HDR images were captured in Europe. Checklist

Revised: The checklist follows the references.

Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Revised: For each question, change the default [TODO] to [Yes] , [No] or[N/A] .

Revised: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Revised: For example:

Revised: • Did you include the license to the code and datasets?

Revised: [Yes] See Section ?? .

Revised: • Did you include the license to the code and datasets?

Revised: [No] The code and the data are

Revised: proprietary

Revised: • Did you include the license to the code and datasets?

Revised: [N/A]

Revised: all

Revised: (a) Do the main claims made in the

Revised: and

Revised: accurately reflect the paper’s

Revised: Yes

Revised: (b) Did you describe the limitations of your work?

Revised: [Yes] See Section

Revised: (c) Did you discuss any potential negative societal impacts of your work?

Revised: [Yes] See

Revised: Section Broader Impact

Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Revised: them?

Revised: [Yes]

Revised: If you are including theoretical results...

Revised: (a) Did you state the full set of assumptions of all theoretical results?

Revised: [N/A]

Revised: (b) Did you include complete proofs of all theoretical results?

Revised: [N/A]

Revised: If you ran experiments...

Revised: (a) Did you include the code, data, and instructions needed to reproduce the main exper-

Revised: imental results (either in the supplemental material or as a URL)?

Revised: [Yes] Please seesupplementary material for the code which includes test set images and a README

Revised: explaining how to setup and run a trained model.

Revised: This is not yet release ready and will

Revised: be refactored prior to being publicly released.

Revised: (b) Did you specify all the training details (e

Revised: g., data splits, hyperparameters,

Revised: how they

Revised: were chosen)?

Revised: [Yes] See page 6, paragraph 3 ’Training Details’ and page 7, paragraph4 ’Inverse Rendering’.(c)

Revised: Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Revised: [N/A]

Revised: No error bars due to deterministic results.

Revised: (d) Did you include the total amount of compute and the type of resources used (e

Revised: g., typeof GPUs, internal cluster, or cloud provider)?

Revised: [Yes] See page 6, paragraph 3 ’Training

Revised: Details’.

Revised: If you are using existing assets (e

Revised: g., code, data, models) or curating/releasing new assets...

Revised: (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes] See page 5, paragraph3 ’

Revised: Training Data’ and page 6, paragraph 4 ’Generalisation’

Revised: (b) Did you mention the license of the assets?

Revised: [Yes] See page 5, paragraph 3 ’Training

Revised: [Table]

Revised: Data’

Revised: [Table]

Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [N/A]

Revised: [Table]

Revised: No original assets were created for this research.

Revised: [Table]

Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Revised: [Table]

Revised: using/curating?

Revised: [Yes] See page 5, paragraph 3 ’Training Data’

Revised: [Table]

Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Revised: [Table]

Revised: information or offensive content?

Revised: [Yes] See page 5, paragraph 3 ’Training Data’

Revised: [Table]

Revised: If you used crowdsourcing or conducted research with human subjects...

Revised: [Table]

Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Revised: [Table]

Revised: applicable?

Revised: [N/A] No crowdsourcing or human subjects were used in this research.

Revised: [Table]

Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Revised: [Table]

Revised: Board (IRB) approvals, if applicable?

Revised: [N/A]

Revised: No crowdsourcing or human subjects

Revised: [Table]

Revised: were used in this research.

Revised: [Table]

Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Revised: [Table]

Revised: spent on participant compensation?

Revised: [N/A] No crowdsourcing or human subjects wereused in this research.

Revised: [Table]