Text-guided 3 D Human Generation from 2 D Collections (2023)

Abstract: 3D human modeling has been widely used for engaging interaction in gaming, film, and animation. The customization of these characters is crucial for creativity and scalability, which highlights the importance of controllability. In this work, we introduce Text-guided 3D Human Generation (T3H), where a model is to generate a 3D human, guided by the fashion description. There are two goals: 1) the 3D human should render articulately, and 2) its outfit is controlled by the given text. To address this T3H task, we propose Compositional Cross-modal Human (CCH). CCH adopts cross-modal attention to fuse compositional human rendering with the extracted fashion semantics. Each human body part perceives relevant textual guidance as its visual patterns. We incorporate the human prior and semantic discrimination to enhance 3D geometry transformation and fine-grained consistency, enabling it to learn from 2D collections for data efficiency. We conduct evaluations on DeepFashion and SHHQ with diverse fashion attributes covering the shape, fabric, and color of upper and lower clothing. Extensive experiments demonstrate that CCH achieves superior results for T3H with high efficiency.

## introduction our world is inherently three-dimensional, where this nature highlights the importance of 3d applications in various fields, including architecture, product design, and scientific simulation. the capability of 3d content generation helps bridge the gap between physical and virtual domains, providing an engaging interaction within digital media. furthermore, realistic 3d humans have vast practical value, especially in gaming, film, and animation. despite enhancing the user experience, the customization of the character is crucial for creativity and scalability. language is the most direct way of communication. if a system follows the description and establishes project website: https://text-3dh.github.io the 3d human model, it will significantly improve controllability and meet the considerable demand. we thus introduce text-guided 3d human generation (t3h) to generate a 3d human with the customized outfit, guided via the fashion description. previous works @xcite depend on multi-view videos to learn 3d human modeling, but these data are difficult to obtain and are not language controllable. text-to-3d @xcite has shown attractive 3d generation results through the success of neural rendering @xcite . however, these methods apply iterative inference optimization by external guidance, which is inefficient for usage. to tackle these above issues, we propose compositional cross-modal human (cch) to learn t3h from 2d collections. cch divides the human body into different parts and employs individual volume rendering, inspired by eva3d @xcite . we extract the fashion semantics from the description and adopt cross-modal attention to fuse body volumes with textual features, where each part can learn to perceive its correlated fashion patterns. to support various angles of view, cch leverages the human prior @xcite to guide the geometry transformation for concrete human architecture. then these compositional volumes can jointly render a 3d human with the desired fashion efficiently. the semantic discrimination further considers compositional distinguishment over each human part, which improves the fine-grained alignment with its description through adversarial training. we perform experiments on deepfashion @xcite and shhq @xcite , which contain human images with diverse fashion descriptions. the patterns include various types of shapes (sleeveless, medium short, long, etc.), fabrics (denim, cotton, furry, etc.), and colors (floral, graphic, pure color, etc.) for the upper and lower clothing. to study the performance of t3h, we conduct a thorough evaluation from both visual and semantic aspects. we treat overall realism, geometry measure, and pose correctness to assess the quality of generated 3d humans. for the alignment with the assigned fashion, we apply text-visual relevance from clip and fine-grained accuracy by a trained fashion classifier. the experiments indicate that language is necessary to make 3d human generation controllable. our proposed cch adopts cross-modal attention to fuse compositional neural rendering with textual fashion as 3d humans, and semantic discrimination further helps fine-grained consistency. in summary, our contributions are three-fold: • we introduce t3h to control 3d human generation via fashion description, learning from collections of 2d images. ## conclusion we present text-guided 3d human generation (t3h) to create a 3d human by a fashion description. to learn this from 2d collections, we introduce compositional cross-modal human (cch). with crossmodal attention, cch fuses compositional human rendering and textual semantics to build a concrete body architecture with the corresponding fashion. experiments across various fashion attributes show that cch effectively carries out t3h with high efficiency. we believe t3h helps advance a new field toward vision-and-language research.