LADIS : Language Disentanglement for 3 D Shape Editing (2022)

Abstract: Natural language interaction is a promising direction for democratizing 3D shape design. However, existing methods for text-driven 3D shape editing face challenges in producing decoupled, local edits to 3D shapes. We address this problem by learning disentangled latent representations that ground language in 3D geometry. To this end, we propose a complementary tool set including a novel network architecture, a disentanglement loss, and a new editing procedure. Additionally, to measure edit locality, we define a new metric that we call part-wise edit precision. We show that our method outperforms existing SOTA methods by 20% in terms of edit locality, and up to 6.6% in terms of language reference resolution accuracy. Human evaluations additionally show that compared to the existing SOTA, our method produces shape edits that are more local, more semantically accurate, and more visually obvious. Our work suggests that by solely disentangling language representations, downstream 3D shape editing can become more local to relevant parts, even if the model was never given explicit part-based supervision.

## introduction natural language has been used as a universally accessible and powerful interface to enable content creation and manipulation. for 2d images, dall•e @xcite , imagen @xcite , and styleclip @xcite demonstrated impressive capabilities in creating and editing images using language descriptions. however, understanding and grounding natural language in 3d geometry remains a challenge. recent efforts have all focused on using existing large pretrained vision-language models like clip @xcite , along with a differentiable renderer to interface 3d geometry with clip through 2d renderings. while these methods can perform geometry edits as described by language, they often figure 1 : an example of the language-conditioned 3d shape editing task. given an input shape and an edit description, the objective is to edit the shape to match the edit description. when language representations are entangled, this can often produce edits not localized to the relevant parts. in contrast, by regularizing the disentanglement of language representation alone, our system achieves higher edit locality while making more pronounced edits in the requested areas. we hypothesize that the source of non-local edits lies in the way we jointly represent edit descriptions and shapes. to perform language-based shape edits, existing methods first learn a shape-language joint space using a classification or contrastive learning objective. then, the system iteratively perform shape edits to maximize the similarity between the final shape representation and the edit instruction representation @xcite @xcite . problems arise when the joint representation space is entangled, i.e., the representations of independent edit instructions referring to unrelated shape edits have non-trivial co-dependencies. optimizing with such entangled language representations can cause unintended correlations in unrelated shape edits. in this work, we study how to achieve localized language-based 3d shape editing through learning more disentangled language representations. given an input shape and an utterance, the system should perform the correct edits to the shape while minimizing edits to parts of the shape that are unnecessary to shape integrity and undescribed by the edit description. thus, we aim to learn disentangled grounded language representations that link language references to the correct parts of the shape. in an ideal disentangled latent space, independent edit descriptions that describe mutually independent attributes and parts should be orthogonal to each other (e.g. "the car has big wheels" and "the car has 2 doors"). figure 2 shows an example of two independent edit descriptions. to this end, we propose a language disentanglement loss, named ladis, and a multi-expert shape-difference encoder architecture. whereas the ladis loss enables the model to orthogonalize mutually independent utterances, the multi-expert shape-difference encoder allows the experts to specialize on different geometric features that span the semantics of the edit descriptions. these two design decisions enable our model to decouple different adjectives and part references within the edit descriptions, leading to improved disentanglement. we note that a certain degree of geometry entanglement may be inherent in the integrity of the shape structure -the use of different linguistic terms (e.g., "table top", "table leg") does not always imply perfect disentanglement of the part locations or geometries. for example, we typically expect co-dependencies between the extent of a four-leg table top and its leg locations. such dependencies have to be learned from data and encoded in the shape space, making our task quite challenging. even with a more disentangled joint representation space, imperfect optimization makes shape editing difficult. empirically, we find that optimization-based 3d editing approaches may generate invalid objects. for example, failed shape edits can cause a chair shape to stop looking like a realistic chair. to combat this issue, we additionally introduce two modifications to a standard optimization procedure: neighborhood simplex editing (nse) and output-driven edit step-size adjustment (odessa). nse re-frames shape editing as interpolating between a source shape and its nearest neighbors, which helps the edited shape to remain realistic. odessa enables every optimization step to produce a constant amount of change in the output 3d space, allowing us to cope with metric ambiguities in language instructions. we evaluate our system on the shapetalk @xcite and shape-glot @xcite datasets. to quantify the degree of edit locality, we propose part-wise edit precision (pep), a new metric that measures if shape edits are restricted to the parts that are mentioned by the edit description. experimental results show that our system demonstrates an improved ability to resolve language references and successfully produce more local shape edits (20% higher pep than the closest competing system @xcite ). additionally, human evaluation comprising of 500 annotations shows that compared to changeit3d @xcite , our method produces 3d shape edits with higher part-wise locality, semantic accuracy and visual obviousness. furthermore, our analysis of the language representations shows a higher level of disentanglement of shape part concepts and adjectives, demonstrating that disentangling language representation enables larger, more correct, and more localized shape edits, without ever equipping our model with explicit part supervision. the code for ladis can be found at https: //github.com/ianhuang0630/ladis . ## experiments in this section, we evaluate our system's ability to achieve localized language-guided shape edits. we evaluate our disentangled joint space j by shape classification accuracy, and by the locality of our system's edits, evaluated both by our novel partbased metric and human evaluators. in addition, we analyze expert specialization in our multi-expert shape difference encoder and visualize the disentangled language representations. experimental results show that our proposed method is better at shape classification, produces more disentangled representations, and lead to more local shape edits. ## conclusion in this work, we introduce a set of complimentary techniques to improve language-based shape edits. our core idea is to learn a more disentangled shape-language joint space, which is achieved by a novel multi-expert architecture and a language disentanglement loss. importantly, we show that we can accomplish more obvious edits while improving their locality by learning to disentangle the language alone, without ever equipping our model with part supervision. experimental results show the effectiveness of our methods in producing localized shape edits. our techniques are agnostic to the underlying 3d representations, and we look forward to extending our method to other 3d representations and modalities.