Text-to-3D

Use Cases

Text-to-3D models can be used in a wide variety of applications that require 3D, such as games, animation, design, architecture, engineering, marketing, and more.

This task is similar to the image-to-3d task, but takes text input instead of image input. In practice, this is often equivalent to a combination of text-to-image and image-to-3d. That is, the text is first converted to an image, then the image is converted to 3D.

Generating Meshes

Meshes are the standard representation of 3D in industry.

Generating Gaussian Splats

Gaussian Splatting is a rendering technique that represents scenes as fuzzy points.

Inference

Inference for this task typically leverages the Diffusers library for inference, using Custom Pipelines.

These are unstandardized and depend on the model. More details can be found in each model repository.

import torch
import requests
import numpy as np
from io import BytesIO
from diffusers import DiffusionPipeline
from PIL import Image

pipeline = DiffusionPipeline.from_pretrained(
    "dylanebert/LGM-full",
    custom_pipeline="dylanebert/LGM-full",
    torch_dtype=torch.float16,
    trust_remote_code=True,
).to("cuda")

input_prompt = "a cat statue"
result = pipeline(input_prompt, None)
result_path = "/tmp/output.ply"
pipeline.save_ply(result, result_path)

In the code above, we:

1. Import the necessary libraries
2. Load the LGM-full model and custom pipeline
3. Define the input prompt
4. Run the pipeline on the input prompt
5. Save the output to a file

Output Formats

Meshes can be in .obj, .glb, .stl, or .gltf format. Other formats are allowed, but won't be rendered in the gradio Model3D component.

Splats can be in .ply or .splat format. They can be rendered in the gradio Model3D component using the gsplat.js library.

Useful Resources

• ML for 3D Course
• 3D Arena Leaderboard
• gsplat.js