3DRAE and 3DDiT

Pretrained weights for Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale (NeurIPS 2026).

3DRAE encodes an arbitrary number of calibrated multi-view images into a fixed-length 3D-grounded representation. 3DDiT performs diffusion modeling directly in this latent space for efficient and spatially consistent 3D scene generation.

• Paper
• Project page
• Code
• Processed ScanNet data

https://github.com/user-attachments/assets/9c584612-5fa0-4cfb-b711-046b8c0a2446

Model Variants

Variant	Latent shape	Reconstruction / NVS	Point maps	3D generation
dino2	1024 x 768	Yes	Yes	Yes
dino2-256	256 x 768	Yes	No	No
siglip2	256 x 768	Yes	No	No

The released 3DDiT checkpoint is paired with the 1024-token DINOv2 representation.

Quick Start

Clone the official implementation and create the recommended Python 3.10 environment:

git clone https://github.com/WU-CVGL/3DRAE.git
cd 3DRAE
uv sync --python 3.10
source .venv/bin/activate

Download the complete checkpoint bundle into checkpoints/:

hf download tracywei/3drae \
  --exclude "README.md" ".gitattributes" \
  --local-dir checkpoints

Verify the downloaded files:

python tools/verify_checkpoints.py

Run novel-view synthesis on a bundled calibrated example:

python scripts/reconstruct.py \
  --model dino2 \
  --scene examples/scene_000 \
  --condition-frame-ids 0 5 \
  --target-frame-ids 6 7 8 9 \
  --output-dir outputs/example_nvs_dino2

Run single-view conditioned 3D scene generation with a synthetic camera trajectory:

python scripts/generate.py \
  --scene examples/scene_000 \
  --appearance-condition-frame-ids 0 \
  --trajectory-mode synthetic-orbit \
  --output-dir outputs/example_generation_1view_synthetic

See the official repository for Python APIs, additional conditioning settings, training, feature extraction, dense-view reconstruction, and the interactive Viser viewer.

Checkpoint Contents

backbones/
├── dinov2-with-registers-base/
└── siglip2-base-patch16-256/
dino2/
├── neck.safetensors
├── decoder.safetensors
├── point_decoder.safetensors
├── stats.safetensors
└── 3ddit.safetensors
dino2_256/
├── neck.safetensors
├── decoder.safetensors
└── stats.safetensors
siglip2/
├── neck.safetensors
├── decoder.safetensors
└── stats.safetensors
discriminator/
perceptual/
manifest.json

Files under discriminator/ and perceptual/ are required only for Stage 1 training. File provenance, sizes, and SHA256 checksums are recorded in manifest.json.

Input and Output

Inference expects calibrated multi-view RGB images with camera intrinsics and OpenCV world-to-camera extrinsics. The repository includes ready-to-run examples and documents the exact scene format.

Depending on the selected checkpoint, 3DRAE can output fixed-length 3D latent tokens, novel-view RGB images, world-space point maps, confidence maps, and fused RGB point clouds. 3DDiT generates a 3D latent that can be decoded along arbitrary camera trajectories.

Limitations

• The released models require calibrated camera intrinsics and poses; they are not pose-free reconstruction models.
• The released 3DDiT checkpoint supports DINOv2-based 1024-token generation. The compact DINOv2 and SigLIP2 variants are reconstruction/NVS models only.
• The compact DINOv2 and SigLIP2 variants predict RGB images but do not include point-map decoders.
• Generated content may be inaccurate or inconsistent for sparse observations, large viewpoint extrapolation, or domains far from the training distribution.
• The model produces implicit latent representations, rendered images, and optional point maps rather than explicit watertight meshes.

Sources and Terms

The bundle includes the DINOv2 with registers base and SigLIP2 base patch16 256 foundation backbones. It also includes the auxiliary DINO discriminator and VGG/LPIPS weights required by the released Stage 1 training configuration. Third-party files remain subject to their original licenses and terms.

The released code is available under the MIT License.

Citation

@inproceedings{wei20263drae,
  title     = {Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale},
  author    = {Wei, Dongxu and Xu, Qi and Li, Zhiqi and Zhou, Hangning and Qiu, Cong and Qin, Hailong and Yang, Mu and Cui, Zhaopeng and Liu, Peidong},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}