Paper: HyperFields: Towards Zero-Shot Generation of NeRFs from Text (ICLR-2024)

Reviewer comment (Experiments):
**Low quality of results**: The results presented, especially the simple geometric shapes and uniform color distributions, seem to be of low standard. There is a noticeable disparity in quality when compared to state-of-the-art techniques. Moreover, the ablation study in Figure 6 only provides a single example, which weakens the overall argument.

Author response:
Limited experimental results/low quality of results

Please see our results on complex scene generation in the general comment above. The quality of our results are a direct consequence of the quality of the teacher NeRFs we distill. 

We provide an additional ablation demonstrating the utility of the dynamic hypernetwork, please see this [figure](https://ibb.co/KNGThj7). In this experiment we train prompts 'wooden table' and 'origami chair' together. We see that our dynamic hypernetwork can generate origami chair and wooden table, while when trained without our dynamic hypernetwork the model suffers from lack of flexibility and makes the origami chair have a geometry similar to the wooden table. This ablation is discussed in Appendix H.

Reviewer comment (Novelty):
**Lack of comparative discussion**: It's important to note that recent studies, such as HyperDiffusion (ICCV 2023) and Shap-E (arXiv), have also explored the use of hyperparameters in 3D object generation, resulting in promising results. However, there is a lack of comparative discussion with these methods, which is crucial in positioning HyperFields in the current research landscape.

Author response:
Lack of comparative discussion to HyperDiffusion and Shape-E

We have updated the related works section of the paper to cite and discuss these papers. We note that HyperDiffusion is an unconditional generative model which diffuses over NeRF weights (cannot do text based scene generation), whereas our method predicts NeRF weights dynamically conditioned on the 1) text prompt, 2) the sampled 3D coordinates, and 3) the previous NeRF activations. Similarly, Shap-E diffuses over latent codes which are then mapped to weights of a NeRF MLP, and requires teacher point clouds to train. Both of these methods have the same limitations of slow inference due to denoising sampling.

Reviewer comment (Experiments):
An other small concern is regarding the need to create a small dataset of NeRF scenes. Each NeRF can takes minutes, and this prevents it to scale to larger datasets.

Author response:
We discuss in Sec 4.3 in the paper that our model is able to amortize the overhead cost of both training the NeRF dataset and distilling them into the HyperFields model. Furthermore, from just a handful of NeRFs we are able to bootstrap our model to converge faster on novel unseen prompts with minimal fine-tuning (Fig. 5), thus only aiding in the effort towards building a larger dataset.

Reviewer comment (Experiments):
If the model weight depends on where we sample the activations, then the generated weights can have high variance. I’m a bit concerned that this means different ways to sample the points can lead to different weights, and thus leading to different performance.

Author response:
Please see [figure linked here](https://ibb.co/GcLMmZW) for our experiment on the effect of the variance of the  generated weights  due to sampling. We query HyperFields on “a red chair” and shuffled the query rays into random mini-batches for the same viewpoint, and show the render outputs above. Note that the renders are visually identical, **even though the weights predicted by the hypernetwork are different as a result of the different mini-batch compositions**. This is because we train the HyperFields model by distilling teacher NeRFs which naturally imposes 3D view consistency constraints. Thus, our model’s high expressive potential through adaptive NeRF weights does not hinder the view-consistency of scenes due to our distillation training.

Reviewer comment (Presentation):
Quality: Figure 4, why is the image looks washed out? Is it because of artifacts from normalization?

Author response:
We purposely fade some of the renders to indicate that those are the training scenes. Please see a revised version of the figure [linked here](https://postimg.cc/tYHrxNyM) where the feedforward inference scenes are outlined in black dashed lines, and the training scenes have no outline.

Reviewer comment (Experiments):
Why do we choose three captions during training? Why can’t we spread away these captions throughout different batches?

Author response:
Can the reviewer please clarify this question? We are not sure which implementation detail this question is referring to.

Reviewer comment (Reproducibility):
Maybe I’ve missed it, but how do we choose different a_i’s?

Author response:
The a_i’s are the activations of layer i of the NeRF MLP for a given sampled 3D coordinate. These activations are fed into the hypernetwork along with the conditioning text embedding to predict the weights for the NeRF MLP layer i+1. Please refer to Fig. 3 [linked here](https://postimg.cc/sBRSnw6k) for a visualization of this process.

Reviewer comment (Experiments):
There seems missing quantitative comparison between the proposed method and baseline (DreamFusion) on CLIP retrieval scores or user study, which may make the work further stronger and more convincing.

Author response:
Thank you for the suggestion. In section E of the supplementary, we have user studies comparing the quality of the out-of-distribution renders of HyperFields with that of the Stable-DreamFusion baseline. Our method is consistently preferred over the baseline. Please see the table copied below. 

| Model | Gold Blender | Yarn Shoes | Yarn Skateboard | Plaid Skateboard | Plaid Lamppost | Strawberry with tiger stripes |
| ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | 
| Our Method (↓)  | 1.30 | 1.07 | 1.32 | 1.29| 1.36 | 1.12 | 
| Best DreamFusion Baseline | 2.50 | 2.40 | 2.33 | 1.75 | 2.0 | 2.25 |

Please see the results below for the results of CLIP retrieval comparing HyperFields with StableDreamFusion baseline. We compute CLIP retrieval scores for out-of-distribution scenes in [Fig. 5](https://raw.githubusercontent.com/threedle/hyperfields/rebuttal/ood_compare_RLfix.jpg) for both Hyperfields and the Stable-DreamFusion (S) baseline, which is the NeRF model trained from scratch.  We significantly outperform the baseline when the baseline and our model are  trained for the same number of steps. We compare favorably when the baselines are trained 5x longer than our model.

The support set of scenes is a cross product of the following geometry and attribute set. 
- Geometry: {‘chair’, ‘pot’,’toaster’, ‘bench’, ‘stool’, ‘bowl’, ‘vase’, ‘couch’,’table’}
- Attributes: {‘bamboo’, ‘brick’, ‘yarn’, ‘glacier’, ‘gold’, ‘origami’, ‘plaid’, ‘rainbow’, ‘stained glass’, ‘terracotta’, ‘wooden’}

The support set has 108 scenes in total. We perform CLIP retrieval on this support set.  

| Model | Top-1 | Top-10 |
| ----------- | ----------- | ----------- |
| Stable DreamFusion (Same compute budget at our model)  | 0.12 | 0.25 |
| Stable DreamFusion (Trained for 5x longer than our model)  | 0.62 | 0.75 | 
| HyperFields (ours)  | 0.75 | 0.87 | 

For additional quantitative evaluation please refer to Table 1 in the paper. We compare the CLIP retrieval scores of the scenes generated by HyperFields for prompts it has seen during training and prompts it generated zero-shot with no optimization. Our results suggest that the quality of the zero-shot rendered novel scenes is similar to that of the quality of the scenes seen during training. For additional discussion please refer to section 4.1 in paper.

Reviewer comment (Experiments):
Another ablation study to conduct is to verify the effectiveness of training across multiple shapes then fine-tuning on a single shape, and compare it with a baseline that train the model on this single shape from scratch. This will demonstrate the advantage of training across a wider range of samples to learn a stronger representation between text condition and the 3D representation, by incorporating more samples during training.

Author response:
The effectiveness of training across multiple scenes then fine-tuning is demonstrated in [Fig. 5 in the paper](https://ibb.co/QY2TkHS).  We demonstrate that by training across multiple scenes, our model learns a meaningful map between text embeddings and NeRF, such that when fine-tuning to an out of distribution scene (prompt for which the attribute/shape tokens are not seen during training) our model converges significantly faster (~5x) than a model trained from scratch. Specifically, the comparison to the baseline model trained from scratch is given by rows 2 and 4 in [Fig. 5](https://ibb.co/QY2TkHS).

Reviewer comment (Theory):
Although intriguing, it remains unclear why the hypernetwork can successfully learn the disentanglement of various attributes. This may be attributed to the limited scope of text prompts and attributes during training. The authors are expected to provide more insights on this matter.

Author response:
The disentanglement is implicitly enforced through the training, as the model sees instances of the same text token (e.g. “plaid”) in association with different shapes (chair, toaster, table, etc.), and thus must learn the common features which define a “plaid” style across these different scenes. The same reasoning applies to the object/shape tokens shared across different scenes. If the model attempts to correlate any attribute token with shape token (e.g. “plaid” with “chair”), then it will be penalized when it encounters a prompt like “origami chair” and predicts plaid attributes for the chair, thus forcing it to disentangle the different attributes.