Paper: Uni3D: Exploring Unified 3D Representation at Scale (ICLR-2024)

Reviewer comment (Evaluation):
The successful scaling up to 1B parameters is a very key contribution from this work. It works well on multiple downstream tasks. Does the gain come from the 2D pretrained models? Table 7 seems to give some ablations and yet this is not clearly stated in the introduction. The reviewer wonders this since it would tell people whether to focus more on 2D data/pretraining to resolve 3D problems, or whether 3D pretraining is essential. If the reviewer is getting this right, using none 2D initialized weights as shown in Table 7, the gain is not too obvious (44.8 vs 45.8).

Author response:
We sincerely appreciate Reviewer 1Xtq for the acknowledgment of our work and constructive feedback. We will respond comprehensively to the questions as follows.

**Q1: Where does the gain of scaling up Uni3D to billion-scale parameters come from.**

The performance gain of Uni3D in scaling up model sizes mainly comes from two parts: 

**1. The well-explored scaling up strategies from the unified 2D ViT model.** We directly leverage the vanilla transformer structurally equivalent to ViT, which can naturally solve the difficulties in scaling designs in 3D domain by simply scaling up the model size with the well-studied unified 2D/NLP scaling-up strategies. The effectiveness and efficiency of our scaling-up strategy are fully demonstrated by the comprehensive exploration of scaling up ViT in the 2D vision domain. As shown in Fig. 1 and Tab. 5, we observe continuous performance improvements as the scaling of model sizes under the flexible and unified framework.

**2. The sharable 2D prior from large pre-trained 2D ViT.**
Uni3D directly leverages the vanilla transformer structurally equivalent to ViT as the 3D backbone, which brings a new perspective of introducing pretrained priors. Specifically, we can naturally adopt the pretrained large models in other modalities which share the same vanilla transformer as ours to initialize Uni3D. These pretrained models are trained in datasets consisting of billions of images and texts, which already learn rich underlying representational abilities for Transformer and have the potential to enhance and stabilize the learning of large-scale 3D representations.

To demonstrate the key role of the large pre-trained 2D ViT for initializing Uni3D, we conduct extensive ablation studies under different scales of Uni3D from 6 M (Tiny), 23 M (Small), 88 M (Base), 307 M (Large) to 1 B (giant). We keep the default data setting the same as the ablation studies of the main paper, i.e., “Ensembled (no-LVIS)”. We list the results below:
 
 ***Table A : Ablation studies on the effect of initializing Uni3D with pre-trained 2D ViT.***
| Model Size | From scratch | 2d pretrained ViT init |
| --- | --- | --- |
| Tiny | 42.8 | 43.5 |
| Small | 43.7 | 44.8 |
| Base | 44.8 | 45.8 |
| Large | 45.0 | 46.2 |
| giant | 26.6（broken） | 47.2 |

As shown, the initialization from large pre-trained 2D ViT improves the performance at all model sizes. Specifically, the training of the model with extremely large scales of parameters (e.g. 1 B of giant) is **broken** when optimizating from scratch, while the continuous improvement is further achieved by introducing 2D priors from pre-trained large vision models. This justifies that the powerful initialization from 2D pretrained models is necessary for achieving performance gain in scaling up Uni3D model to extremly large parameters (e.g. 1B).

We further refer the review to Tab. 10 of the main paper where we conducted ablation studies under two-modal alignment. The results show that the optimization crashed without EVA initialization in the difficult situation where only images are available (40.1->20.7) or only texts are available (26.3->12.4). The results in Tab. 10 of the main paper are reported under the Uni3D-Base model. We additionally supplement this ablation with different model scales, i.e. from 6 M (Tiny), 23 M (Small) to 88 M (Base), under the difficult text-point cloud alignment situation. We report the results below:

  ***Table B : Learning text-point cloud alignment from scratch or initializing with pre-trained 2D ViT.***
| Model Size | From scratch | 2d pretrained ViT init |
| --- | --- | --- |
| Tiny | 37.0 | 38.3 |
| Small | 37.8 | 39.6 |
| Base | 20.7（broken） | 40.1 |


We observe the same experimental phenomena as Table A, where the large pre-trained 2D ViT improves the performance at all model sizes. More specifically, the performance of models trained from scratch improves normally from Tiny to Small (37.0 -> 37.8), but crashes when further scaling up from Small to Base (37.8->20.7). While models trained with 2D priors of pre-trained large vision models obtain continuous improvements from Tiny, Small, to Base (38.3->39.6->40.1). 

These two ablation studies demonstrate the necessity of introducing 2D priors from large pre-trained 2D ViT for initializing Uni3D. The 2D priors play the key role in improving and stablizing the optimization of Uni3D models, especially when scaling up to large parameters, where the optimization crashed when training from scratch. Moreover, we observe that for the more difficult situation, such as text-point cloud alignment, the crashment comes eailer during the model size scaling. And the 2D prior stabilizes all the situations and achieves continuous improvements in all the settings.

Reviewer comment (Experiments):
Uni3D is verified on a wide variety of tasks and benchmarks, which is motivating. Does the author plan to try some challenging and realistic settings, e.g., autonomous driving settings with point clouds? That would strengthen the proposed approach to a great extent.

Author response:
**Q2: Expending Uni3D to more challenging settings, e.g., autonomous driving.**

We believe that Uni3D, which has powerful and universial 3D perceptions, has great potentials in some challenging and realistic settings, eg. autonomous driving settings with point clouds, VR/AR and robotics. We consider these as the exciting furture work for Uni3D and will be committed to exploring the application prospects of Uni3D.

Reviewer comment (Experiments):
The last row in Table 1 shows the result of models separately trained on each benchmark. The unified approach of Uni3D is on par with them, demonstrating the generalization or universality. Does Uni3D potentially surpass the performance of the model trained each on one particular benchmark? The reviewer is wondering about the necessity of training a universal model.

Author response:
**Q3: The performance of training a universal Uni3D model.**

We justify that all the experiment results on Tab. 1 are tested in a zero-shot way with the universial Uni3D model trained under two data settings, i.e., Ensembled (no LVIS) and Emsembled. The last row reports the best results achieved on different benchmarks under different convergences of optimization. Specifically, the best performance of ModelNet and ScanObjNN is achieved with the early stopped Uni3D model since those two benchmarks are quite easier with few categories and samples, where more training iterations will lead to performance degeneration. While the best performance of Objaverse-LVIS is achieved with the fully converged Uni3D model since Objaverse-LVIS is much more difficult with 1,156 categories containing some out-of-the-distribution samples, which requires more extensive training.

Reviewer comment (Novelty):
[Novelty] The proposed method mainly follows the previous work, OpenShape, in terms of the training data construction (Sec 3.2) and the training objective (Eq. (1)). From my understanding, the only difference is that Uni3D uses PointBERT’s 3D tokenizer + ViT as a 3D encoder while OpenShape uses PointBERT. Although OpenShape is still in arXiv, I recommend the authors clarify what differentiates Uni3D from OpenShape since OpenShape is the most relevant baseline and high similarity to this work.

Author response:
**Q1: The contributions of Uni3D and differences to OpenShape.**

We justify that the training objective to align text-image-point cloud representations is not considered as a technical contribution of Uni3D, nor is it for OpenShape. The objective was first introduced in ULIP [1], and both OpenShape and Uni3D follow the training objective and focus on scaling up 3D representations. While OpenShape focuses on the data scaling to leverage the large open-source Objaverse dataset with specific efforts on ensembling multiple datasets and data augmentation to filter and enrich text descriptions. We go beyond OpenShape and propose to focus on the scaling up on model sizes. We achieve this by designing unified 3D representation to directly adopt the vanilla transformer structurally equivalent to ViT as the 3D backbone, which leverages abundant 2D pretrained models as initialization. This unlocks the great potential of 2D models and scaling-up strategies to the 3D world, which plays the key role in scaling up Uni3D to billion-scale parameters.

We summarize the main difference between Uni3D and OpenShape as below:

1. We design the first billion-scale 3D foundation model, and demonstrate a billion-scale 3D representation model can transfer well to various downstream tasks and scenarios. This is acheived by introducing the sharable 2D prior and scaling up strategies from large pre-trained 2D ViT as we will discuss in **Q3**. On the contrary, OpenShape struggles to scale up models to large parameters, which shows degradation during the model size scaling up (e.g. 72 M in Fig. 2).
2. We unify the 3D representation backbone to a 2D ViT. This enables Uni3D to explore the great potential of pretrained priors and scaling-up strategies of large vision models to the 3D world. On the contrary, OpenShape adopts the specifically designed point cloud backbones, e.g., SparseConv, PointBERT, PointMLP, etc. OpenShape achieved inconsistent performance with different backbones. For instance, selecting PointBERT as the 3D backbone performs the best on Objaverse-LVIS, while leveraging SparseConv as the backbone significantly outperforms PointBERT under ScanObjectNN. Uni3D achieves significant improvement under all benchmarks consistently with the unified model.
3. We explore more representative 3D tasks and introduce more applications with the aligned point cloud-text-image representations from Uni3D, such as point cloud painting, open-world understanding and the difficult dense prediction task of part segmentation.


[1] Xue L, Gao M, Xing C, et al. ULIP: Learning a unified representation of language, images, and point clouds for 3D understanding. CVPR 2023

Reviewer comment (Experiments):
[Experiments] Although the paper provides text-to-3D retrieval and image-to-3D retrieval, I think 3D-to-text (captioning) and 3D-to-image (generation) experiments need to be included in the paper to show the good alignment of 3D, image, and text.

Author response:
**Q2: More experiments on the applications of Uni3D.**

We provide the extra application of Uni3D for 3D captioning in Sec.F in the Appendix of the revised paper. More applications (e.g. point cloud-based image generation) will be further included in the next revision.

[1] Xue L, Gao M, Xing C, et al. ULIP: Learning a unified representation of language, images, and point clouds for 3D understanding. CVPR 2023

Reviewer comment (Evaluation):
[Analysis] As shown in Figure 1, scaling up OpenShape does not improve its zero-shot accuracy, unlike Uni3D. Why can Uni3D have such consistent improvement while it has a similar architecture to OpenShape? Does this improvement come from the initialization of the large pre-trained 2D ViT? I recommend the authors provide an analysis of this to make the paper stronger.

Author response:
**Q3: How does Uni3D successfully scale up to billion-scale parameters.**

The success of Uni3D in scaling up model sizes mainly comes from two parts: 

**1. The well-explored scaling up strategies from the unified 2D ViT model.** We directly leverage the vanilla transformer structurally equivalent to ViT, which can naturally solve the difficulties in scaling designs in 3D domain by simply scaling up the model size with the well-studied unified 2D/NLP scaling-up strategies. The effectiveness and efficiency of our scaling-up strategy are fully demonstrated by the comprehensive exploration of scaling up ViT in the 2D vision domain. As shown in Fig. 1 and Tab. 5, we observe continuous performance improvements as the scaling of model sizes under the flexible and unified framework.

**2. The sharable 2D prior from large pre-trained 2D ViT.**
Uni3D directly leverages the vanilla transformer structurally equivalent to ViT as the 3D backbone, which brings a new perspective of introducing pretrained priors. Specifically, we can naturally adopt the pretrained large models in other modalities which share the same vanilla transformer as ours to initialize Uni3D. These pretrained models are trained in datasets consisting of billions of images and texts, which already learn rich underlying representational abilities for Transformer and have the potential to enhance and stabilize the learning of large-scale 3D representations.

To demonstrate the key role of the large pre-trained 2D ViT for initializing Uni3D, we conduct extensive ablation studies under different scales of Uni3D from 6 M (Tiny), 23 M (Small), 88 M (Base), 307 M (Large) to 1 B (giant). We keep the default data setting the same as the ablation studies of the main paper, i.e., “Ensembled (no-LVIS)”. We list the results below:
 
 ***Table A : Ablation studies on the effect of initializing Uni3D with pre-trained 2D ViT.***
| Model Size | From scratch | 2d pretrained ViT init |
| --- | --- | --- |
| Tiny | 42.8 | 43.5 |
| Small | 43.7 | 44.8 |
| Base | 44.8 | 45.8 |
| Large | 45.0 | 46.2 |
| giant | 26.6（broken） | 47.2 |

As shown, the initialization from large pre-trained 2D ViT improves the performance at all model sizes. Specifically, the training of the model with extremely large scales of parameters (e.g. 1 B of giant) is **broken** when optimizating from scratch, while the continuous improvement is further achieved by introducing 2D priors from pre-trained large vision models. This justifies that the powerful initialization from 2D pretrained models is necessary for achieving performance gain in scaling up Uni3D model to extremly large parameters (e.g. 1B).

We further refer the review to Tab. 10 of the main paper where we conducted ablation studies under two-modal alignment. The results show that the optimization crashed without EVA initialization in the difficult situation where only images are available (40.1->20.7) or only texts are available (26.3->12.4). The results in Tab. 10 of the main paper are reported under the Uni3D-Base model. We additionally supplement this ablation with different model scales, i.e. from 6 M (Tiny), 23 M (Small) to 88 M (Base), under the difficult text-point cloud alignment situation. We report the results below:

  ***Table B : Learning text-point cloud alignment from scratch or initializing with pre-trained 2D ViT.***
| Model Size | From scratch | 2d pretrained ViT init |
| --- | --- | --- |
| Tiny | 37.0 | 38.3 |
| Small | 37.8 | 39.6 |
| Base | 20.7（broken） | 40.1 |


We observe the same experimental phenomena as Table A, where the large pre-trained 2D ViT improves the performance at all model sizes. More specifically, the performance of models trained from scratch improves normally from Tiny to Small (37.0 -> 37.8), but crashes when further scaling up from Small to Base (37.8->20.7). While models trained with 2D priors of pre-trained large vision models obtain continuous improvements from Tiny, Small, to Base (38.3->39.6->40.1). 

These two ablation studies demonstrate the necessity of introducing 2D priors from large pre-trained 2D ViT for initializing Uni3D. The 2D priors play the key role in improving and stablizing the optimization of Uni3D models, especially when scaling up to large parameters, where the optimization crashed when training from scratch. Moreover, we observe that for the more difficult situation, such as text-point cloud alignment, the crashment comes eailer during the model size scaling. And the 2D prior stabilizes all the situations and achieves continuous improvements in all the settings.

Reviewer comment (Experiments):
(New concern on Nov 22) Prior works like OpenShape have been trained on a much smaller batch size than Uni3D (OpenShape is trained using a batch size of 200 on a single A100-80G, while Uni3D is trained using a batch size of 1152 on 24x A100-40G). My question is, if authors train Uni3D using the same batch size as OpenShape, or train OpenShape using the same batch size as Uni3D, does Uni3D still outperform OpenShape (when both models have similar numbers of parameters)? This experiment will reveal whether the better training setting or the proposed architecture plays a bigger role in the superior performance of Uni3D. Though, regardless of the final findings from this experiment, it won't affect my positive view of the paper.

Author response:
**Q3: Train Uni3D using the same batch size as OpenShape under similar model parameters.**

We conduct ablation studies and report the performance of training Uni3D-Small (22.6M parameters) with the same batch size as OpenShape-PointBERT (i.e. 200) below:

***Table E: Ablation studies on the batch sizes.***

|Uni3D-Small | Batch size 200 | Batch size 1152|
| --- | --- | --- |
|Objaverse-LVIS| 42.6 | 44.8 |

The experiments are conducted on a single A100-40G GPU under the “Ensembled (no-LVIS)” dataset. The results show that a larger batch size leads to a better performance with about 2% improvement on accuracy. However, even with the much smaller batch size same as OpenShape, Uni3D-Small still achieves significantly better performance than OpenShape-PointBERT (42.6 vs. 39.1) and OpenShape-SparseConv (42.6 vs. 37.0). Also notice that we choose the Uni3D-Small as the backbone here, which only contains 22.6M parameters less than OpenShape-PointBERT (32.3M) and OpenShape-SparseConv (41.3M). Even so, we still outperform OpenShape with the proposed Uni3D backbone unifying to ViT and the robust initialization from large pre-trained 2D ViT.

Reviewer comment (Evaluation):
It would be helpful to include a comparison of the inference speed of Uni3D compared to prior work on open-world 3D understanding.

Author response:
**Q1: Inference speed of Uni3D compared to prior work.**

We provide a comprehensive comparison on the inference speed of Uni3D and prior works. The times are listed below:

***Table C: Inference speed comparison between Uni3D and prior works.***
| Model  | Inference time (ms) |
| --- | --- |
| Uni3D-Tiny | 6.3 |
| Uni3D-Small | 7.6 |
| Uni3D-Base | 10.2 |
| Uni3D-Large | 16.0 |
| Uni3D-giant | 20.3 |
| Openshape-SparseConv | 17.8 |
| Openshape-PointBERT | 25.2 |
| ULIP-PointBERT | 58.5 |

The inference time here indicates the time to achieve an embedding for a single point cloud (batch size = 1). All the results are tested under one single V100 GPU. The results demonstrate that Uni3D, which performs the best with the largest model sizes, achieves faster inference times compared to previous works like OpenShape-PointBERT and ULIP-PointBERT. The ViT-based Uni3D also achieves a comparable inference speed to OpenShape-SparseConv, where the SparseConv is a specially designed framework for efficient point cloud processing. 

The superior efficiency of Uni3D can be attributed to the efforts in the implementation improvement of Uni3D, especially in the point tokenizer. Specifically, ULIP-PointBERT implements the farthest point sampling (FPS) in point  tokenizer in pure pytorch, while OpenShape-PointBERT improves the implementation of FPS with DGL (Deep Graph Library) and achieves a faster inference time than the pure pytorch implementation of ULIP-PointBERT. We take a further step to leverage a CUDA-accelerated implementation of farthest point sampling provided by PointNet++ [1], which is highly efficient with underlying C++ operations, to implement our point  tokenizer. The efforts in improving point  tokenizer implementation lead to fast inference speed of Uni3D, where even the Uni3D-giant with 1B (1,016M) parameters achieves a speed on par with OpenShape-PointBERT with 32M parameters and OpenShape-SparseConv with 41M parameters.

[1] Qi C R, Yi L, Su H, et al. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. NeurIPS 2017.

Reviewer comment (Experiments):
The main source of performance gains of Uni3D over existing work seems to come from the proposed architecture, instead of how the architecture is initialized (according to authors' reply to Reviewer 1Xtq, whether to initialize the architecture from pretrained ViT only leads to about 1% performance difference). This observation might be attributed to the extensive size of the pretraining dataset, which encompasses 1 million 3D shapes. This raises a question: If a smaller pretraining dataset were used, would initializing with a pretrained ViT have a more significant impact on open-world generalization?

Author response:
**Q2: The impact of initialization with a pretrained ViT under a smaller dataset.**

We greatly thank reviewer 7VZo for the insightful advice on exploring the impact of initializing Uni3D with pretrained ViTs. We follow the suggestions to conduct experiments under a much smaller dataset containing 20% data samples from the "Ensembled" dataset. We list the results with Uni3D-Base as the backbone below:

***Table D: The impact of ViT initialization under a small dataset (20% data samples).***
| Uni3D-Base  | From scratch | 2d pretrained ViT init |
| --- | --- | --- |
| Objaverse-LVIS | 38.6 | 42.5 |
| ModelNet40 | 80.6 | 83.5 |
| ScanObjectNN | 52.6 | 60.2 |


As shown, initializing Uni3D with pretrained 2D ViT demonstrates more significant improvements on the much smaller data scale. Specifically, the initialization brings an improvement of about 4% in Objaverse-LVIS, 3% in ModelNet40 and 7.6% in ScanObjectNN. The results demonstrate that the 2D pretrained prior plays a more important role with the limited data scale and further highlight the effectiveness of the unified ViT architecture of Uni3D, which receives sharable 2D priors from large pre-trained 2D ViT. 

We furthest justify that the initialization from large pre-trained 2D ViT contributes more than enhancing performance. As shown in *Table A* and *Table B* of the "*Responses to Reviewer 1Xtq (1/2)*", the 2D prior from large pre-trained ViT also stablizes the training of Uni3D, especially when scaling up to extremely large parameters. Without the initialization, the training is **crashed** under the large model size of Uni3D-giant containing 1B parameters. The results demonstrate that the 2D prior from large pre-trained ViT plays a key role in the success of Uni3D to scale up 3D representations to billion-scale parameters.