Paper: LRM: Large Reconstruction Model for Single Image to 3D (ICLR-2024)

Reviewer comment (Evaluation):
As mentioned in the limitation, the back side of the model (e.g., figure 3 penguin) is blurry and besides that, the color tone/ saturation seems a bit inconsistent with the frontal view.

Author response:
We thank the reviewer for pointing out this limitation. As discussed in Section 4.3.2, multiple plausible solutions exist for the occluded side, but our model is deterministic and likely produces averaged modes of the unseen, which could cause blurry textures and shapes.
This issue is a joint effect of LRM's deterministic approach and an L2 loss and applying MVImgNet data for training, which mostly only covers 180 degrees of view. A similar issue can be seen in the Masked Autoencoders [1], where the model tends to reconstruct blurry contents when a large block of region is removed from an image. One potential solution is to post-optimize the LRM's output (e.g., using generative prior and SDS loss proposed by DreamFusion [2]) at the cost of extra processing time.

On the other hand, our ongoing research found that LRM can be easily extended to accept sparse multi-view inputs by passing encoded multi-view image features to the image-to-triplane decoder, and making the positional embeddings query from features of all images via cross-attention to construct the triplane. This model can create very high-fidelity shapes where the issue of blurry occluded portions can be largely eliminated due to the higher coverage of the 3D object. We will make the multiview LRM paper public to the research community.

[1] Masked Autoencoders Are Scalable Vision Learners. He et al., 2021.

[2] DreamFusion: Text-to-3D using 2D Diffusion. Poole et al., 2022.

Reviewer comment (Experiments):
The approach relies on multiview data which limits the model incorporating more diversity in data source of single view images.

Author response:
The training of LRM requires multiview data; however, we have witnessed continuous growth in the size and diversity of 3D datasets (e.g., the latest Objaverse-XL dataset contains 10.2 million 3D assets), and there are enormous amounts of videos that can potentially provide multiview supervision for reconstruction. Although the number of single-view images is far larger, we remain positive about the amount of multiview data for large-scale 3D learning in the future. 
On the other hand, the recent approach 3DGP [2], which is trained on single-view images, still struggles to produce consistent and complete shapes.
We believe that incorporating single-view data and limited multiview data is an open and highly valuable research question, which we will leave to future work.

[1] Objaverse-XL: A Universe of 10M+ 3D Objects. Deitke et al., 2023.

[2] 3D Generation on ImageNet. Skorokhodov et al., 2023.

Reviewer comment (Reproducibility):
The biggest question is whether the author plan to release the code and training setting, so the community can quickly adopts the direction of generative model for direct 3D generation, instead of using SDS or nerf2nerf which learns nothing of 3D prior.

Author response:
We thank the reviewer for raising this important question. The release plan is still under discussion, and the authors are waiting for more guidance on this. Meanwhile, we have provided very comprehensive model architecture and training details in the paper (Section 3 and Section 4) and Appendix (C and D) for reproducing LRM. In addition to the final model that we presented in the paper, we also attempted to train a smaller version of LRM. 
The model uses the same architecture as the baseline model we showed in Appendix C Table 1, and we trained it on 8 NVIDIA (40G) A100 GPUs for 4 days with batch size 64 and gradient accumulation of 4 steps. This model achieves performance between the baseline and the final models shown in Table 1, which is able to produce reasonable results for research purposes.

Reviewer comment (Evaluation):
The most comparable models of large-scale direct 3D generation are Point-E and Shap-E. From the example shown in the paper, it seems LRM performs way better than both and more analysis of the better quality is preferred. Is it because LRM use objaverse and MVImage net, both contain much more items than what Point-E and Shap-E use or is it because of the design of learnable positional encoding and other architectures of LRM? More analysis or explanation can further elevate this paper to another level.

Author response:
We thank the reviewer for this very constructive suggestion of an in-depth comparison with Point-E [1] and Shap-E [2]. We mentioned Point-E and Shap-E in our paper, and we will discuss more here. Point-E trains an image-to-3D point cloud diffusion model, and Shap-E encodes point clouds to latent representations and trains a diffusion model on the latents to generate parameters of a 3D implicit function. Both models contain hundreds of millions of learnable parameters and are trained with several million 3D assets (unknown data source and unknown computational cost). 
In terms of the network and dataset sizes, our LRM has 500 million learnable parameters, and it is trained on 1 million 3D data, which does not show an advantage. In terms of the network architecture, Point-E, Shap-E, and LRM all use transformer-based models and apply cross-attention for inter-modality modeling (i.e., image-to-point cloud, point cloud+image-to-3D latents, and image-to-triplane).

We hypothesize it is the choice of very compact and expressive triplane representation together with an end-to-end trainable framework that enables the scaling of the LRM and its adequate learning on large datasets (Objaverse and MvImgNet). 
Compared to the unstructured point cloud representation applied in Point-E and Shap-E, LRM applies the structured triplane representation that is aligned with the world frame, which naturally facilitates 2D-to-3D projection. It is also worth mentioning that Point-E uses 4K points (as tokens) and Shap-E uses 16K points (as tokens), but our LRM only uses $3{\times}32{\times}32{=}3072$ triplane tokens, which largely reduce the modeling complexity. Additionally, compared to the two-stage approach in Shape-E, which attempts to generate latents that can produce the parameters of implicit 3D functions through a diffusion model, our LRM directly maps 2D images to triplanes, which should be much more stable and efficient to learn.

Overall, we suggest that LRM is a more data-friendly and efficient model than Point-E and Shap-E. We will further extend this discussion and add it to our paper. 

[1] Point-E: A System for Generating 3D Point Clouds from Complex Prompts. Nichol et al., 2022.

[2] Shap-E: Generating Conditional 3D Implicit
Functions. Jun and Nichol, 2023.

Reviewer comment (Theory):
From a fundamental perspective, this LRM model actually turns the image-to-3D into a deterministic predicting task, which alleviates the ill-pose nature of the task. Does the author think the results are just "fitting" to the training distribution rather than actually predicting / learning a 3D distribution.

Author response:
LRM is a deterministic model that learns the distribution of the 3D data in training and fits the given reference image to the learned distribution at inference. Although single-image-to-3D is a one-to-many mapping problem, we made the simplified assumption of one-to-one mapping in this work, which we have shown to be useful in many cases (as visualized in Figure 2, Figure 6, and Figure 7). We would also like to mention that our LRM bridges 2D and 3D representations that can potentially facilitate one-image-to-many-3D-shapes generation in the future. For instance, training latent diffusion models on triplanes predicted by LRM to generate new shapes or reconstruct 3D models from text-to-multiview generations.

Reviewer comment (Theory):
I think in Equation(4) the index j' actually refers to the set including j? This equation needs to be revised.

Author response:
Equation (4) expresses a self-attention with residual, where the two input terms are identical sequences of tokens. We use the notation of a set to specify that each token in the first term will query all the tokens in the second term. We will revise $j'$ to $j$ for clearness.

Reviewer comment (Reproducibility):
In supplementary, some implementation details are missing, for example the number of heads in multi-head attention. In addition I don't clearly understand how does the attention behave between image feature [32x32, 768] and embedding [32x32x3, 1024]? I know when mapping to q,k,v the feature dimension can be changed, but here the token numbers are different. In my understanding it results in weight shape [32x32, 32x32x3] and when multiplied by value, results are in [32x32, 1024] shape. But the next layer requires input as [32x32x3, 1024]?

Author response:
LRM's image-to-triplane decoder uses 16 attention heads in all multi-head attention. In the cross-attention layer (details in Section 4.2 and Appendix A.2), only the positional embeddings are applied as Q (queries), and the image features are applied as K and V (keys and values), all being projected to the same dimension $d_{D}=1024$. Then, each of the 32x32x3=3072 positional embedding tokens will query all the 32x32=1024 image feature tokens, resulting in weight shape [32x32x3, 32x32]. Therefore, when multiplied by the Values [32x32, 1024], it produces output tokens of dimension [32x32x3, 1024] that match the input dimension of the next layer.  
We appreciate the reviewer for pointing this out, and we will add the missing details to our paper.

Reviewer comment (Reproducibility):
In volume rendering, is the 128 points uniformaly sampled or a two-stage sampling strategy or other sampling methods is adopted?

Author response:
Our LRM uniformly samples 128 points for each ray in neural rendering. We have also tried uniform sampling with perturbation and two-stage coarse-to-fine sampling but did not see an obvious difference in results. We will leave the deeper investigation of this problem to future work.

Reviewer comment (Evaluation):
Insufficient quantitative comparisons: the paper does not conduct any quantitative evaluation against other methods. I believe the novel view synthesis and 3D reconstruction can be evaluated on the held-out sets for those 3D object datasets, and user study should also be possible. Even if the quantitative results may not reflect the generation quality entirely, the paper should include discussions on why these scores are not reliable/not feasible.

Author response:
We thank the reviewer for raising this question, and provide a quantitative comparison to the stat-of-the-art methods Point-E [1], Shap-E [2], and One-2-3-45 [3]. All methods are mentioned in our paper; Point-E trains an image-to-3D point cloud diffusion model, Shap-E encodes point clouds to latent representations and trains a diffusion model on the latents to generate parameters of a 3D implicit function, and One-2-3-45 reconstructs multiview images generated with a 2D diffusion model.
We randomly selected 100 objects from the Google Scanned Objects (GSO) dataset and measured the novel view synthetic quality of 20 reference views (FID, CLIP-Similarity, PSNR, LPIPS) and the geometric quality (Chamfer Distance), as shown in the Table below. 
We can see that our LRM consistently outperforms previous approaches in all metrics. We will add these results and discussion to our paper.

|    | FID $\downarrow$ | CLIP-Similarity $\uparrow$ | PSNR $\uparrow$ |  LPIPS $\downarrow$ |  Chamfer Distance $\downarrow$ |
| :---                |   :----:       |    :----:   |      :----:   |     :----:   |    ---: |   
| Point-E        | 123.70  | 0.741  | 15.60  |  0.308  |  0.099  |
| Shap-E        | 97.05  | 0.805   | 14.36 |  0.289 | 0.085 |
| One-2-3-45  |  139.24  |  0.713  |  12.42  |  0.448  | 0.123  |
| LRM (ours)  | **31.44**  | **0.902**  |  **19.60**  |  **0.163**  |  **0.053**  |

[1] Point-E: A System for Generating 3D Point Clouds from Complex Prompts. Nichol et al., 2022.

[2] Shap-E: Generating Conditional 3D Implicit Functions. Jun and Nichol, 2023.

[3] One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization. Liu et al., 2023.

Reviewer comment (Evaluation):
The absence of quantitative comparisons raises significant concerns.

Author response:
We thank the reviewer for raising this question, and provide a quantitative comparison to the stat-of-the-art methods Point-E [1], Shap-E [2], and One-2-3-45 [3]. All methods are mentioned in our paper; Point-E trains an image-to-3D point cloud diffusion model, Shap-E encodes point clouds to latent representations and trains a diffusion model on the latents to generate parameters of a 3D implicit function, and One-2-3-45 reconstructs multiview images generated with a 2D diffusion model.
We randomly selected 100 objects from the Google Scanned Objects (GSO) dataset and measured the novel view synthetic quality of 20 reference views (FID, CLIP-Similarity, PSNR, LPIPS) and the geometric quality (Chamfer Distance), as shown in the Table below. 
We can see that our LRM consistently outperforms previous approaches in all metrics. We will add these results and discussion to our paper.

|    | FID $\downarrow$ | CLIP-Similarity $\uparrow$ | PSNR $\uparrow$ |  LPIPS $\downarrow$ |  Chamfer Distance $\downarrow$ |
| :---                |   :----:       |    :----:   |      :----:   |     :----:   |    ---: |   
| Point-E        | 123.70  | 0.741  | 15.60  |  0.308  |  0.099  |
| Shap-E        | 97.05  | 0.805   | 14.36 |  0.289 | 0.085 |
| One-2-3-45  |  139.24  |  0.713  |  12.42  |  0.448  | 0.123  |
| LRM (ours)  | **31.44**  | **0.902**  |  **19.60**  |  **0.163**  |  **0.053**  |

[1] Point-E: A System for Generating 3D Point Clouds from Complex Prompts. Nichol et al., 2022.

[2] Shap-E: Generating Conditional 3D Implicit Functions. Jun and Nichol, 2023.

[3] One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization. Liu et al., 2023.

Reviewer comment (Reproducibility):
The method requires significant computational resources.

Author response:
Like many other recent visual/language models (e.g., CLIP [1], GPT-3 [2], and Stable Diffusion [3]), large-scale training is widely applied to stabilize the optimization processes of large models and achieve the best results at the cost of high computational demand.
Our best-performing LRM was trained on 128 NVIDIA (40G) A100 GPUs for 3 days, but the inference only requires a single 11G GPU. 

In addition to the final model that we presented in the paper, we also attempted to train a smaller version of LRM. 
The model uses the same architecture as the baseline model we showed in Appendix C Table 1, and we trained it on 8 NVIDIA (40G) A100 GPUs for 4 days with batch size 64 and gradient accumulation of 4 steps. This model achieves performance between the baseline and the final models shown in Table 1, which is able to produce reasonable results for research purposes. 

[1] Learning Transferable Visual Models From Natural Language Supervision. Radford et al., 2021.

[2] Language Models are Few-Shot Learners. Brown et al., 2020.

[3] High-Resolution Image Synthesis with Latent Diffusion Models. Rombach et al., 2021.

Reviewer comment (Experiments):
A minor issue, but the paper shows no quantitative comparison with any prior work.

Author response:
We thank the reviewer for raising this question, and provide a quantitative comparison to the stat-of-the-art methods Point-E [1], Shap-E [2], and One-2-3-45 [3]. All methods are mentioned in our paper; Point-E trains an image-to-3D point cloud diffusion model, Shap-E encodes point clouds to latent representations and trains a diffusion model on the latents to generate parameters of a 3D implicit function, and One-2-3-45 reconstructs multiview images generated with a 2D diffusion model.
We randomly selected 100 objects from the Google Scanned Objects (GSO) dataset and measured the novel view synthetic quality of 20 reference views (FID, CLIP-Similarity, PSNR, LPIPS) and the geometric quality (Chamfer Distance), as shown in the Table below. 
We can see that our LRM consistently outperforms previous approaches in all metrics. We will add these results and discussion to our paper.

|    | FID $\downarrow$ | CLIP-Similarity $\uparrow$ | PSNR $\uparrow$ |  LPIPS $\downarrow$ |  Chamfer Distance $\downarrow$ |
| :---                |   :----:       |    :----:   |      :----:   |     :----:   |    ---: |   
| Point-E        | 123.70  | 0.741  | 15.60  |  0.308  |  0.099  |
| Shap-E        | 97.05  | 0.805   | 14.36 |  0.289 | 0.085 |
| One-2-3-45  |  139.24  |  0.713  |  12.42  |  0.448  | 0.123  |
| LRM (ours)  | **31.44**  | **0.902**  |  **19.60**  |  **0.163**  |  **0.053**  |

[1] Point-E: A System for Generating 3D Point Clouds from Complex Prompts. Nichol et al., 2022.

[2] Shap-E: Generating Conditional 3D Implicit Functions. Jun and Nichol, 2023.

[3] One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization. Liu et al., 2023.

Reviewer comment (Novelty):
Since the method is discriminative, it is not able to sample different realizations of an input. Additionally, the averaging of modes, even though has been weakened a lot compared with prior works such as PixelNeRF, still exists as the author mentioned.

Author response:
Similar to the classic PixelNerf [1] and Pixel2Mesh [2] and many other deterministic approaches, LRM considers a simplified assumption of one-to-one mapping between image and 3D. Intuitively, this line of work focuses on reconstructing the average shape given partial 2D information, just as humans can naturally infer a probable 3D model of an object from just a single view. We believe that the definition of single-image-to-3D task is not necessarily generative but depends on the specific application. 

One important goal of our paper is to justify the plausibility of large-scale training for 3D, which is very under-explored in previous research.
Moreover, our LRM learns generic 3D prior, and it bridges 2D and 3D representations that can potentially facilitate 3D generation in the future. For instance, training latent diffusion models on triplanes predicted by LRM to generate new shapes or reconstruct 
3D models from text-to-multiview generations.
We would like to refer to the comments from Reviewer 4 (eyyQ), "It fills the gap between 2D and 3D foundation generative model" and "The architecture pave the way for other 3D generative models to scale up".

[1] PixelNeRF: Neural Radiance Fields from One or Few Images. Yu et al., 2021.

[2] Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images. Wang et al., 2018. 

The issue of averaging modes mainly appears in the texture of the occluded portion of the shapes. It is a joint effect of LRM's deterministic approach and an L2 loss and applying MVImgNet data for training, which mostly only covers 180 degrees of view. A similar issue can be seen in the Masked Autoencoders [1], where the model tends to reconstruct blurry contents when a large block of region is removed from an image. One potential solution is to post-optimize the LRM's output (e.g., using generative prior and SDS loss proposed by DreamFusion [2]) at the cost of extra processing time.

On the other hand, our ongoing research found that LRM can be easily extended to accept sparse multi-view inputs by passing encoded multi-view image features to the image-to-triplane decoder, and making the positional embeddings query from features of all images via cross-attention to construct the triplane. This model can create very high-fidelity shapes where the issue of blurry occluded portions can be largely eliminated due to the higher coverage of the 3D object. We will make the multiview LRM paper public to the research community.

[1] Masked Autoencoders Are Scalable Vision Learners. He et al., 2021.

[2] DreamFusion: Text-to-3D using 2D Diffusion. Poole et al., 2022.

Reviewer comment (Theory):
The task of novel view synthesis is inherently probabilistic, as the author mentioned. Even though the method shows amazing scaling-up ability, it is questionable whether solving a generative task in a discriminative way is reasonable.

Author response:
Similar to the classic PixelNerf [1] and Pixel2Mesh [2] and many other deterministic approaches, LRM considers a simplified assumption of one-to-one mapping between image and 3D. Intuitively, this line of work focuses on reconstructing the average shape given partial 2D information, just as humans can naturally infer a probable 3D model of an object from just a single view. We believe that the definition of single-image-to-3D task is not necessarily generative but depends on the specific application. 

One important goal of our paper is to justify the plausibility of large-scale training for 3D, which is very under-explored in previous research.
Moreover, our LRM learns generic 3D prior, and it bridges 2D and 3D representations that can potentially facilitate 3D generation in the future. For instance, training latent diffusion models on triplanes predicted by LRM to generate new shapes or reconstruct 
3D models from text-to-multiview generations.
We would like to refer to the comments from Reviewer 4 (eyyQ), "It fills the gap between 2D and 3D foundation generative model" and "The architecture pave the way for other 3D generative models to scale up".

[1] PixelNeRF: Neural Radiance Fields from One or Few Images. Yu et al., 2021.

[2] Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images. Wang et al., 2018.

Reviewer comment (Writing):
Section 4.3.2 probablistic -> probabilistic.

Author response:
We thank the reviewer for pointing out this typo; we will fix it in our paper.

Reviewer comment (Experiments):
Have the authors tried to perform any sparse view reconstruction?

Author response:
On the other hand, our ongoing research found that LRM can be easily extended to accept sparse multi-view inputs by passing encoded multi-view image features to the image-to-triplane decoder, and making the positional embeddings query from features of all images via cross-attention to construct the triplane. This model can create very high-fidelity shapes where the issue of blurry occluded portions can be largely eliminated due to the higher coverage of the 3D object. We will make the multiview LRM paper public to the research community.