From: ICLR.cc/2023/Conference/Paper189/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper189/Authors
title:  Approaching the end of discussion
comment: Hi reviewer rNVs,

We appreciate your help in reviewing. 

The discussion period is drawing to a close and we haven't heard from you yet. Could you review our responses and share any concerns/questions with us?

From: ICLR.cc/2023/Conference/Paper189/Authors
title: Will try
comment: Thank you very much for your further reply. Let's try it out and get back to you later.

From: ICLR.cc/2023/Conference/Paper189/Reviewer_utsR
title: here's some fun things to try
comment: It is possible to probe for a "prior distribution" for your model. As I understand it now, it only has a conditional distribution mapping one representation to another. How would you sample from a prior distribution? For instance, it'll be cool to see a prior distribution of the text descritptions, or programs, or point-cloud of your model.

What you can do is to perform a kind of iterative transmission (this makes a MCMC). You will start with one shape, then continue to translate it across modes, perhaps adding some noise. The exact set-up I'm not quite sure how to do, but for sure it is possible.

Here's some interesting examples:

https://cocosci.princeton.edu/tom/papers/mcmcpeople.pdf
https://arxiv.org/abs/2202.12226
https://arxiv.org/abs/2008.02595

From: ICLR.cc/2023/Conference/Paper189/Authors
title: Updated
comment: Hi Reviewer utsR,

We've updated several interesting experiments, and are working on more results. Please let us know if there are other fun experiments you want to know the most; we can then change our experiment priority.

From: ICLR.cc/2023/Conference/Paper189/Authors
title: Your concern has no supports
comment: Hi Reviewer cHvi,

Thank you for your response.

***Your current claim on novelty concerns is only supported by "also pointed out by other reviewers". It is black-box to everyone involved in the review process.*** 

Both reviewers utsR and pUSP have stated their reasons for their concerns about novelty; we can then respond to them and improve our revision. Could you please elaborate on your concerns to help us contribute a better revision?

From: ICLR.cc/2023/Conference/Paper189/Reviewer_cHvi
title: Post rebuttal comments
comment: I'd like to thank the authors for putting effort into the rebuttal. While most of my concerns have been addressed, I would still like to remain my rating. The novelty of this method is limited (also pointed out by other reviewers) and generalization is not guaranteed.

From: ICLR.cc/2023/Conference/Paper189/Reviewer_utsR
title: updated score on correctness
comment: that is indeed a mistake and it has been fixed. sorry about that.
still looking forward to these fun experiments, and totally understandable the paper is quite "expensive" in terms of efforts already.

From: ICLR.cc/2023/Conference/Paper189/Authors
title: Approaching the end of the discussion phase
comment: Hi Reviewer pUSP,

During the rebuttal, we found that we were unclear on some of the points you raised in your initial assessment. Could you please check our response? 

***12/07/22*** updates: Discussions for the second phase are drawing to a close. Could you please take a look at our response and share with us your further concerns or questions, if any? We are looking forward to your response.

***12/13/22*** updates: Could you please take a look at our response and communicate with us?

From: ICLR.cc/2023/Conference/Paper189/Authors
title: Author Response (2/2)
comment: > For fair comparison, are the resolutions for the proposed method and the compared ones the same for all the tasks?

We used the same resolutions for all the methods we could, but the situation is a bit more complex than only ensuring the resolutions. We demonstrate the details below:

- For all PointCloud-based methods (Shape Program Generator [1], 3DEncoder-LSTM [2,3], and PVD [4]), we used the same point clouds sampled from meshes. ***Same data, same metrics, same training epochs, and thus, the comparisons are fair for those baselines on the corresponding tasks.***
- For Voxel-based methods (CWGAN[5], Shape IMLE [6]), we adopt the same setting as their original paper used  (public [link1](https://github.com/liuzhengzhe/Towards-Implicit-Text-Guided-Shape-Generation/tree/main/generation), [link2](http://text2shape.stanford.edu/)). ***They used the voxels generated from the same meshes we used for PointCloud-based methods.*** The comparisons between voxel-based methods and our proposed PointCloud-based method can show the advantage of using PointClouds, where the evaluation performs in the same space with the same training epochs. To conclude, If considering using different 3D representations is unfair, then the comparisons are unfair; otherwise, it is fair.
- For  CLIP-based models, we follow their settings shown in their GitHub (public [link3](https://github.com/google-research/google-research/tree/master/dreamfields), [link4](https://github.com/AutodeskAILab/Clip-Forge)). They did not use any 3D-text pairs but tons of 2D Image-text pairs. ***The comparisons are unfair, but the point of this experiment is not to compare numbers:*** The difference between the CLIP-based methods and the methods which rely on 3D-text pairs can show the failure of using CLIP to generate shapes with geometrical details. ***Please also refer to Appendix B for further study on the learning representation by CLIP***: we found the text embedding of CLIP models may not be able to reflect differences in small changes in the text, which may lead to large changes in structure (e.g., $CosineSimilarity(\textit{a chair \textcolor{red}{with} armrests}, \textit{a chair \textcolor{red}{with no} armrests}) = 0.988)$ (anonymous [link1](https://drive.google.com/file/d/16QVAoo_Il8zx3gGMYsSGa9KCUQiQ0U2f/view?usp=share_link)); and (2) the cosine similarity between the rendered 3D shape images and the text embedding may not tell its alignment degree if the text prompt and 3D geometry contain details (anonymous [link2](https://drive.google.com/file/d/12xMKlJFdPVO2d4Hs9ZcJgBPyJjNXUuZv/view?usp=share_link)). 
- For all evaluations, we strictly adopt the same procedure for all methods, where we compute the metrics based on point clouds generated/sampled from all methods.

The details listed above are included in Section 4.1 and Appendix E.4.


[1] Learning to Infer and Execute 3D Shape Programs

[2] Scan2cap: Context-aware dense captioning in rgb-d scans

[3] Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

[4] Shape Generation and Completion Through Point-Voxel Diffusion

[5] Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings

[6] Towards Implicit Text-Guided 3D Shape Generation


----
>  What are the choices for the text decoder and program decoder?

We choose the simplest way to decode text and program, where the text decoder is the BPE decoder, and the program decoder is the rule-based grammar parser [1]. BPE refers to Byte Pair Encoding. Please refer to our paper: the end of Section 3.1 included the related context. Appendix E.1 and E.2 also contain some related information. 

Additionally, Appendix E.3 presents another way to encode and decode the program in a VQVAE manner to handle the program containing continuous parameters. 

[1] Learning to Infer and Execute 3D Shape Programs


----
> The tool/source code is not released for replication study.

We included the code during the rebuttal. As mentioned in the introduction, our dataset and pre-trained models will be released to the public.

From: ICLR.cc/2023/Conference/Paper189/Authors
title: General Response
comment: We thank the reviewers and AC for their help. We would appreciate it if you could check the following points:

- Transforming different shape representations into a unified representation and using a unified framework to study the transformation between them is new to the community (anonymous ***[link](https://drive.google.com/file/d/1d8PY8gxhkFEKM-zCo6my_4QAu6I-EM5a/view?usp=sharing)***).
- By selecting Point Cloud, Program, and Text, our framework can help obtain multi-level information about shape (anonymous ***[link](https://drive.google.com/file/d/1vZkogwGtG4i0wfsK14FUmewllEhloNHq/view)***). Also, the framework can generate shapes corresponding to the input texts with levels of geometric details, and diverse geometries (anonymous ***[link](https://drive.google.com/file/d/1p700PdxMot913BbOtqMm_6sW6ZW0G7TE/view?usp=sharing), [link](https://drive.google.com/file/d/1d5-YD7iqm8E-mduVwGI8Lht5Yxc0Dy4V/view?usp=sharing), [link](https://drive.google.com/file/d/1yrX09elZk5jzQFG1zQYxcbvtwwountTe/view)***). 
- Our results in various downstream tasks demonstrate the achievements of the proposed multimodal framework, especially for ***the improvements over each task made by jointly training over all tasks.***
- We studied the learning representation of CLIP-based methods in Section 4.1 and Appendix B (anonymous ***[link](https://drive.google.com/file/d/16QVAoo_Il8zx3gGMYsSGa9KCUQiQ0U2f/view?usp=share_link), [link](https://drive.google.com/file/d/12xMKlJFdPVO2d4Hs9ZcJgBPyJjNXUuZv/view?usp=share_link)***). ***It suggests the failure of using CLIP to capture the connections between text and 3D structures, which is new to the community.***
-  The collected and organized resource will be shared with the community to study 3D multimodal learning, including the 100k (3D, Structure-Related Text) pairs and our pre-trained models. The codes of our proposed models are included in the appendix.

From: ICLR.cc/2023/Conference/Paper189/Authors
title: Author Response (1/2)
comment: We thank Reviewer rNVs for the valuable reviews and efforts. ICLR encourages communication between reviewers and authors. Please take the opportunity to discuss any of your concerns/questions with us so that we can clear up misunderstandings and demonstrate our contributions. 

Our answers to your current concerns are listed below:


----
> The motivation of Point Cloud ⇒Program is not clear.

> it is not clear to me why Point Cloud ⇒Program is needed in this context. The motivation and usage scenarios need to be explained. What are the potential applications for Text ⇒Program of 3d shapes?

We'd like to share our thoughts with you about why Point Cloud ⇒Program and Text ⇒Program in the below. We have improved our manuscript according to your feedback to clarify the motivation better. Thanks


As shown in Section 4.3 and Figure 8 (or [anonymous link](https://drive.google.com/file/d/1vZkogwGtG4i0wfsK14FUmewllEhloNHq/view)), the program can tell us the regularities contained in shape (implicitly represented as the "for loop") and how we assemble it (shape compositions). This is complementary information to PointCloud and Text where (1) point clouds cannot deliver the composition of shapes, but low-level geometry; (2) text can only contain rough descriptions instead of accurate configurations. Therefore, ***Point Cloud ⇒Program*** can provide us with the part relationships encoded in the shape that Point Cloud cannot delivery (similar to first do part segmentation and then inducing the part relationships); (2) ***Text ⇒Program*** can help us reconstruct the shape described by the text along with its compositions.

Besides the advantages mentioned above, Point Cloud ⇒Program and Text ⇒Program can help us better perform planning and interacting with robots in the 3D world. (1) If we have a model that can perform ***Point Cloud ⇒Program***, the robots can scan environments first (obtain PointCloud) and generate programs. Then, we can do safe, guaranteed planning based on our obtained programs, such as conducting [formal verification](https://en.wikipedia.org/wiki/Formal_verification) based on the programs to ensure the safety; (2) if we have a model that can perform ***Text ⇒Program***, it can help our humans instruct robots to operate complex 3D objects by naming their parts and intuitively interact with environments. Note that we are not claiming that the proposed framework can do all the mentioned things.

----
> The improvement over existing methods is not clear.

> This work proposed PointVQVAE inspired by VQVAE. The difference is in the point sampling. Is there a specific motivation for this design? Any improvement over the previous designs of 3D encoders [1-3]?


***The key is not the way of point sampling, but turning point clouds into a uniform discrete code, the same representation that we turn text and shape programs into.*** With the help of PointVQVAE, we can perform multimodal learning on the unified space for various tasks with the simple but uniform objective: $\sum_k \mathcal{L}\left(c_k, c_k^{g t} \mid\left[0, \cdots, c_{k-1}\right]\right)$, where $c_k \in \[C^{text}, C^{program}, C^{PointCloud}\].$

In other words, PointVQVAE turns PointClouds into discrete codes. Then, the proposed ShapeCode Translator can act in the unified space, and the joint training over different shape representations comes true. They all contribute to the improvements/achievements in the final tasks: Text $\Longrightarrow$ Point Cloud, Point Cloud $\Longrightarrow$ Text, Point Cloud $\Longrightarrow$ Program, and Point Cloud Completion. 

However, all the mentioned papers [1-3] cannot do the abovementioned tasks: transfer PointClouds into the unified discrete codes: 

- [1] adopted implicit functions which can reconstruct shapes with details but adopt a continuous feature space.  

- [2] could generate shape hierarchies with a shape-part-annotated dataset (PartNet [4]), which is impractical for common point clouds; 

- [3] is a multiview shape recognition framework that cannot support shape reconstruction. 

- Compared to them, PointVQVAE is designated to support our proposed multimodal learning framework and benefits all downstream tasks.

Also, ***for all the tasks mentioned in the main paper, we've demonstrated improvements over the existing methods based on both quantitative and qualitative results.*** 

[1] Local Deep Implicit Functions for 3D Shape 

[2] StructureNet: Hierarchical Graph Networks for 3D Shape Generation 

[3] GVCNN: Group-View Convolutional Neural Networks for 3D Shape Recognition

[4] PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding

From: ICLR.cc/2023/Conference/Paper189/Authors
title: Author Response (2/2)
comment: 
> The novelty seems incremental, The ShapeCode Transformer is essentially the standard Transformer.

We'd like to emphasize that we have never claimed ShapeCode Transformer is a kind of new transformer, and its structure is one of our contributions. Otherwise, we are against the research integrity.

As our paper mentioned, the ShapeCode transformer is implemented as the most standard transformer, rather than other variants, to ensure generality. Our framework is composed of various components for a system. we intended to adopt the simplest technology for each component: we chose the simplest way to encode and decode programs and texts (not Bert, but BPE), And the simplest structure, the standard Transformer, that can handle translation tasks. We believe a baseline system should be simple enough for people to study yet perform enough to demonstrate the concept. Any additional patches would weaken the contribution, such as modifying the ShapeCode Transformer with a special design and claiming it is technically novel.

The technique contributions come from the compositing of the system and the study of the learning representation for both multimodal learning and CLIP-based methods. Also, studying multiple 3D representations with a unified framework is new to the community.


----
> In Fig 4, the quality between the generated shapes of Shape Compiler and Shape IMLE seem to be quite similar. However, Shape IMLE’s quantitative results shown in Table 1 are much worse than Shape Compiler’s? Any explanations?

This point reflects the importance of our proposed dataset, which provides sizable paired 3D-text data. Although the generated shapes look similar to human perception, the number would drastically decrease if it does not correspond to the input text prompt. The sensitivity of numbers is a good hint to the necessity of the paired 3D-text data.

----
> not all types of cross-modality transformations can be handled by the proposed framework

***We are unsure which type of cross-modality transformation cannot be handled by the proposed framework.*** Could you please elaborate on your concerns for us?

Besides, we carefully chose point cloud, shape program, and text to deliver complementary information that describes 3D shape accurate geometry, compositions, and semantics as shown in [link](https://drive.google.com/file/d/1vZkogwGtG4i0wfsK14FUmewllEhloNHq/view). The proposed framework can transform between the three different shape representations. 

----
> Transform from text to program cannot be performed by the proposed framework. Instead, one needs to first convert text to PC, followed by PC to program conversion.

Our framework can perform text-to-program translation by Text $\Longrightarrow$ Point Cloud  $\Longrightarrow$ Program. This was mentioned in Figure 1 caption and Footnote 1. An example is shown in Figure 1. Therefore, the proposed framework can do translation from text to program. 

Based on the above discussion, ***we are unsure what's your point in this question since the transformation from text to program can be performed by our framework..*** Could you please elaborate on your concerns for us; we can then better answer you. thanks.

From: ICLR.cc/2023/Conference/Paper189/Authors
title: Author Response (1/2)
comment: We thank reviewer pUSP for the valuable feedback and efforts. We tried our best to address your concerns/questions below. However, some of your points are unclear to us. Please share your thoughts with us, and then we can address your concerns at our best. ICLR encourages communication between reviewers and authors, and we'd like to hear from you.


----
> Comparisons with CLIP-Forge and Dream Field are not fair, since they are CLIP based methods and did not use any annotated text-shape pairs.

We'd like to first seriously note that Section 4.1 has pointed out the training data difference between the two types of methods. We did not claim or argue the comparisons with CLIP-based methods are fair. Otherwise, we are against the research integrity.

***The comparisons are not used to compare but to study the learning representation of CLIP.*** The clip-based approaches [1,2] achieved amazing generation results without any 3D-text data. However, this paper demonstrates they cannot handle text-guided shape generation if the text contains geometric details. We further studied the learning representation of CLIP-based models, where our Appendix B shows: 

***(1)*** the text embedding of CLIP models may not be able to reflect differences in small changes in the text, which lead to large changes in structure (e.g., $CosineSimilarity(\textit{a chair \textcolor{red}{with} armrests}, \textit{a chair \textcolor{red}{with no} armrests}) = 0.988)$ (anonymous [link1](https://drive.google.com/file/d/16QVAoo_Il8zx3gGMYsSGa9KCUQiQ0U2f/view?usp=share_link));  

***(2)*** the cosine similarity between the rendered 3D shape images and the text embedding may not tell its alignment degree if the text prompt and 3D geometry contain details (anonymous [link2](https://drive.google.com/file/d/12xMKlJFdPVO2d4Hs9ZcJgBPyJjNXUuZv/view?usp=share_link)). 

All the experiments mentioned above and the related discussions in our paper provide insights into CLIP-based methods. 

[1] CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation

[2] Zero-Shot Text-Guided Object Generation with Dream Fields

----
> In Appendices A and F, the authors compared the performance of PointVQVAE with other methods on reconstruction and completion tasks. However, I believe that methods that are also based on VQVAE (e.g., ShapeFormer[A]) should be included (since they are more relevant to the proposed PointVQVAE.]


We'd like to kindly mention that this work is not about beating every baseline and becoming the SOTA across every task. It is about studying translating across different 3D shape representations, and we proposed a framework that can handle it well. Furthermore, the framework is boosted via joint training with all heterogeneous data, benefiting all the involved 3D downstream tasks. 

We never claim PointVQVAE achieved SOTA performance in the mentioned tasks, but as a codec to translate PointCloud into the unified representations as we transfer text and programs. Additionally, the shape completion task is an example of extending our framework, where we also do not claim we achieved the SOTA performance. Your mentioned paper [3] could outperform our methods in the mentioned tasks, as the proposed framework is not designated to perform those tasks. However, [3] cannot perform text-related tasks like the proposed framework. We hope you can kindly recognize this point. 

Despite all the discussed above, we can still compare with [3] if you feel a demanding need. ***Please let us know what you think.***

[3] ShapeFormer: Transformer-based Shape Completion via Sparse Representation

----
> Large-scale annotated/paired datasets are required for each task.

Yes, you are right, and this point was discussed in Appendix C.4. The requirement for the annotated dataset also generally exists in the supervised learning framework and related 3D-text literature [4,5]. One potential point to alleviate this issue is that we may recover 3D data from 2D images [6] and videos [7] that contain paired text information.

[4] Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings

[5] Towards Implicit Text-Guided 3D Shape Generation

[6] Learning 3d object shape and layout without 3d supervision, 2022

[7] Understanding 3d object articulation in internet videos, 2022

From: ICLR.cc/2023/Conference/Paper189/Authors
title: Author Response
comment: We thank reviewer utsR for the comments. 

***Date 11/18 updates***: we are still working on the experiments you suggested. Will update it in OpenReview later.

***Date 12/08 updates***: we updated the preliminary results of the cycle-generation experiments (Point Cloud $\Longleftrightarrow$ Program and Point Cloud $\Longleftrightarrow$).

----
> regarding datasets release.

For sure. The dataset we collect is one of our major contributions and will be released to help our community conduct related research. Our pre-trained models and codes will be released as well. This was mentioned in the last paragraph of the introduction.


----
> Point Cloud $\Longrightarrow$ Program $\Longrightarrow$ Text $\Longrightarrow$ Point Cloud; Does it converge to a fixed-point distribution of shapes, if keeping a chain of translations going in a MCMC style.

What you have imagined is super interesting. A simpler case is going between two abstractions, such as iterating Point Cloud $\Longleftrightarrow$ Text. We've thought about similar things but failed to do so --- due to benchmarking all the tasks and bringing the proposed framework alive, which are already up to 30 pages and cost ~1yr; but we can play this during the rebuttal. Thanks for bringing this up.


----
> cookbook -> codebook

Fixed, thanks!


----
Let us know if you have any other concerns or interesting experimental ideas. 

Btw, we are unsure if you selected the wrong score for "***Correctness***"? If so, we'd appreciate it if you can update it.

From: ICLR.cc/2023/Conference/Paper189/Authors
title: Author Response
comment: 
We thank reviewer cHvi for his/her comments. ***We agree with all the mentioned weaknesses, and the discussion around the exact weaknesses was included in our original submission.*** Please kindly refer to Appendix C for details and the discussion of other weaknesses.

***Despite all the weaknesses we discussed, we still propose the framework.*** We believe it is important to form a bridge across different shape representations to provide complete information on 3D shapes ([link](https://drive.google.com/file/d/1vZkogwGtG4i0wfsK14FUmewllEhloNHq/view)) and a uniform framework for various downstream tasks, further boosted by joint training with all heterogeneous data. All the limitations we faced here could be solved/alleviated since the framework can directly benefit from the progress from either shape program grammar improvements or more 3D data recovery from images and videos.

We've listed our responses below based on your specific questions to answer you in OpenReview better. Please kindly check our manuscript, which contains all the relevant discussions.

----
> The grammar of programs may limit the quality of the output shapes

The shape program grammar definitely affects the quality of the output shapes, and the development of shape programs can further benefit our framework because grammar development is orthogonal to the proposed framework. Therefore, this paper also raises attention to developing the next level of shape programs. 

----
> Generalization to unseen shapes is limited.

We agree with you. We have shown a case to process the Hilbert Cube with the proposed framework in Figure 10 (also ***anonymous [link](https://drive.google.com/file/d/1hvb6b0S5FwdC0UGvjGWXdLd_W8voR6jp/view)***). This is a general problem across deep learning frameworks if the learning framework not encodes strong human priors but directly learns from data. This issue could be alleviated by having more 3D data recovering from images and videos or leveraging meta-learning techniques. 

----
> Results are not convincing since the results are only shown for table and chair categories.

You are not right about the categories. Our qualitative results contain other types of furniture, such as Lamp and Bed. Please refer to Figure 3 and Appendix H. The anonymous ***[link](https://drive.google.com/file/d/1yrX09elZk5jzQFG1zQYxcbvtwwountTe/view)*** shows some text-conditional generated results by our framework for categories different from chairs and tables.

Surprisingly, the current deep learning framework cannot handle those simple geometry well and fails to capture the connections between words and structures for those "classes with simple geometry". It would still be worth studying "those geometries" until we can successfully solve the problem. It is a good curriculum learning that our community starts with something clean and "simple", while the current method cannot well address. The developed principles on the "simple" data could lead to solutions in real applications. 

Also, note that the 3D vision community uses ShapeNet dataset [1] a lot. Compared to 3D-text works [3] [4], which only adopt chairs and tables, we have extended to the ABO dataset [2], which consists of real-world 3D objects. Please refer to Appendix H and supplementary material for more results. 


[1] Shapenet: An information-rich 3d model repository, 2015

[2] Abo: Dataset and benchmarks for real-world 3d object understanding, 2022

[3] Towards Implicit Text-Guided 3D Shape Generation, CVPR 2022

[4] ShapeCrafter: A Recursive Text-Conditioned 3D Shape Generation Model, NeurIPS 2022


----
> Reproducibility is not guaranteed.

We include the codes as supplementary material during the rebuttal phase. As mentioned in the paper, our collected dataset and pre-trained models will be released.

From: ICLR.cc/2023/Conference/Paper189/Reviewer_cHvi
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes a method called Neural Shape Compiler. The proposed network is able to translate between different shape abstractions, including texts, point clouds, shape programs, etc. This is a unified framework that can be applied to several settings such as text to point cloud generation, point cloud captioning, point cloud to programs, etc.
strength_and_weaknesses: Strengths:
1. The idea of having a unified framework that allows translating between representations is interesting.
2. The experimental evaluations are good. The authors considered multiple datasets with both quantitative and qualitative results presented.

Weaknesses:
1. The grammar of programs may limit the quality of the output shapes
2. Generalization to unseen shapes is limited.
3. Results are not convincing since the results are only shown for table and chair categories. 
clarity,_quality,_novelty_and_reproducibility: The paper reads well. The quality of the presentation is good. The problem is interesting. The presented method is quite complicated and therefore without releasing code reproducibility is questionable. 
summary_of_the_review: An interesting problem. Limited generation of the method. Results are not convincing, since the results are only shown on classes with simple geometry. Reproducibility is not guaranteed.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper189/Reviewer_utsR
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper took a range of datasets/tasks involving translation (compiling between these three modes): point-cloud, natural-language, and programs, and showed a single latent model of discrete shape codes can be used to unify these translations. By unifying a representation and (more crucially) unifying all the dataset together, it shows superior performance.

This is similar to translations, instead of building all N^2 pairs of translation models from any N languages to any other, you translate all N languages into a common representation, and then decode out to the target language type of choice. In this work, N = 3.
strength_and_weaknesses: strength: this work is very thorough in its evaluations, and makes a strong point that the discrete latent code is a good representation for shapes, as it out performs previous approaches that used a more specific representation per translation task. the main benefit gain is from simply being able to train on a larger dataset of different modes. I believe the works that put into marshaling all the dataset together is also a key contribution.

weakness: the result isn't surprising. it is to be expected that training on a larger dataset, especially those grounded in multiple modes can achieve better results, as the networks need to "work harder" to come up with a reasonable representation.
clarity,_quality,_novelty_and_reproducibility: clarity : good

quality : good

novelty : mediocre

reproducibility : good
summary_of_the_review: I personally was not too excited about this paper as the results were not unexpected. However, it is crucial that someone does this work because there has been too many different representations for shapes (in the pointcloud, program, and language space). If we were to learn from the lessons of LLM is that we should focus on one flexible architecture that can capture (and benefit) from the different modes rather than specialized architecture for each task -- only data should differ, not models.

questions for the authors:
1. I might have missed it, but do you intend to release a cleaned version of all the datasets used in this work? It would be of great benefits because going through the different datasets and putting them together is not a trivial task, and having a single, uniform format would be of great value.

2. Have you tried taking a shape in point-cloud, then generate it into a program, then to language, then back to point-cloud and see how the IOU would change? What would happen if you keep a chain of translations going in a MCMC style, like a Pictionary game? Does it converge to a fixed-point distribution of shapes? These are all fun academic questions you could be asking, but for some reason you didn't even entertain yourself with it. That's a bummer. Can you try it? I would love to see a kind of "stationary distribution". This paper is good reference: https://arxiv.org/abs/2202.12226 


minor:

typo: in figure2's caption I think you meant "codebook" not "cookbook" haha.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper189/Reviewer_pUSP
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper presents a unified framework for performing text-to-point cloud, point cloud-to-text, and point cloud-to-program tasks. More specifically, the proposed ShapeCode Transformer is able to bridge the gap of text, shape, and program in a number of datasets. The experimental results also support the use of the proposed method for transforming data across the above modalities.  
strength_and_weaknesses: Strength
- The proposed framework is able to jointly perform the tasks of text-to-PC, PC-to-text, PC-to-program.
- A novel VQVAE for describing point cloud data is proposed (with promising results).
- Paper is well written and easy to follow. It is easy to understand the ideas of the proposed modules.

Weaknesses:
- The novelty seems incremental, since the presented framework simply combines 3 Encoder-Decoder pipelines with a ShapeCode Transformer to convert the associated latent representations. Vision and language models (e.g., VIVO, VinVL, etc.) have been designed to take cross-modality inputs while carefully taking care of the associated data modalities.
The ShapeCode Transformer is essentially the standard Transformer (for translation purposes), which now translates codes of text/shape/program.
- Large-scale annotated/paired datasets are required for each task.
- Transform from text to program cannot be performed by the proposed framework. Instead, one needs to first convert text to PC, followed by PC to program conversion. 
- Comparisons with CLIP-Forge and Dream Field are not fair, since they are CLIP based methods and did not use any annotated text-shape pairs.
- In Fig 4, the quality between the generated shapes of Shape Compiler and Shape IMLE seem to be quite similar. However, Shape IMLE’s quantitative results shown in Table 1 are much worse than Shape Compiler’s? Any explanations?
- In Appendices A and F, the authors compared the performance of PointVQVAE with other methods on reconstruction and completion tasks. However, I believe that methods that are also based on VQVAE (e.g., ShapeFormer[A]) should be included (since they are more relevant to the proposed PointVQVAE.

 [A] ShapeFormer: Transformer-based Shape Completion via Sparse Representation, CVPR 2022


clarity,_quality,_novelty_and_reproducibility: Please see the comments in Strength and Weaknesses.
summary_of_the_review: This paper presents a unified framework for performing text-to-point cloud, point cloud-to-text, and point cloud-to-program tasks. More specifically, the proposed ShapeCode Transformer is able to bridge the gap of text, shape, and program in a number of datasets. The experimental results also support the use of the proposed method for transforming data across the above modalities. However, as noted in the Weaknesses, the technical contributions seem to be marginal, while not all types of cross-modality transformations can be handled by the proposed framework.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper189/Reviewer_rNVs
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper presents a unified framework to translate between expressions of 3d shapes in different modalities: Text, Point Cloud, and Program. Text describes the 3d shape in human language and program translates the 3d geometric structures into source code. The authors first propose Neural Shape Compiler to model the abstraction transformation as a conditional generation process, which converts different abstract types into unified discrete shape code. The encoded shape code can be decoded to the target shape abstraction. Evaluation has been conducted on multiple tasks including Text ⇒ Point Cloud, Point Cloud ⇒ Text, Point Cloud ⇒Program, and Point Cloud Completion.
strength_and_weaknesses: Strengths:

. A novel integration of the transformations among three abstractions of stereo shapes to facilitate shared progress among tasks accomplished by specialized models.

. Experiments show that the proposed Shape Compiler outperforms the existing methods for translation between text and point cloud, and the conversion from point cloud to program.

Weaknesses:

. The motivation of Point Cloud ⇒Program is not clear.

. The improvement over existing methods is not clear.

clarity,_quality,_novelty_and_reproducibility: The paper is generally well written. The tool/source code is not released for replication study.
summary_of_the_review: This paper presents a unified framework to translate between expressions of 3d shapes in different modalities: Text, Point Cloud, and Program. I can understand the transformations between Text and Point Cloud, but it is not clear to me why Point Cloud ⇒Program is needed in this context. The motivation and usage scenarios need to be explained. What are the potential applications for Text ⇒Program of 3d shapes? 

This work proposed PointVQVAE inspired by VQVAE. The difference is in the point sampling. Is there a specific motivation for this design? Any improvement over the previous designs of 3D encoders [1-3]?

For fair comparison, are the resolutions for the proposed method and the compared ones the same for all the tasks? What are the choices for the text decoder and program decoder? 

[1] Local Deep Implicit Functions for 3D Shape
[2] StructureNet: Hierarchical Graph Networks for 3D Shape Generation
[3] GVCNN: Group-View Convolutional Neural Networks for 3D Shape Recognition

correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold