GaussianFormer-V17b

GaussianFormer renders 3D Gaussian Splatting scenes with a transformer, without per-scene optimization. It adapts RenderFormer (SIGGRAPH 2025), a transformer renderer for triangle meshes: each Gaussian's 14 parameters [pos(3), scale(3), quat(4), rgb(3), opacity(1)] become one scene token, and the two-stage architecture (view-independent scene encoder, view-dependent ray decoder with a DPT head) is kept and warm-started from RenderFormer's weights.

This checkpoint (V17, epoch 36 of its 512 px stage) replaces V10b as the recommended model.

Code: github.com/SVLwoof/gaussianformer

Training

• Data: 13,405 single-object scans from Objaverse_Splats, each pruned from ~50k to 20k Gaussians (LightGaussian importance score, then a short recovery fine-tune of the kept Gaussians). Targets are gsplat renders of the full, unpruned splat: 14 orbit views per object at 512 px.
• Stage A: 60 epochs at 256 px, log-HDR L1 loss.
• Stage B: 36 epochs at 512 px, log-HDR L1 + LPIPS-VGG (0.5 each), cosine LR 5e-5 to 5e-7, 8 GPUs.

Results

Held-out set of 1,806 objects not seen in training, 4 views each, PSNR on the object's bounding box against the full-splat ground truth:

	PSNR (dB)
gsplat rasterization of the same 20k-Gaussian input	44.94
GaussianFormer-V17b	30.29

The model reproduces shape, color and coarse texture well but blurs fine detail (thin strokes, text, small patterns); that gap to rasterization is what current work targets.

How to use

import torch
from gaussianformer.pipelines.rendering_pipeline import GaussianFormerRenderingPipeline

pipeline = GaussianFormerRenderingPipeline.from_pretrained("shahafvl/gaussianformer-v17b")
pipeline.to(torch.device("cuda"))

# gaussians [B, N, 14] (pos | scale | quat wxyz | rgb | opacity), mask [B, N] bool,
# c2w [B, V, 4, 4] (Blender convention), fov [B, V, 1] in degrees
imgs = pipeline(gaussians=g, mask=m, c2w=c2w, fov=fov, resolution=512, torch_dtype=torch.float16)
# imgs: [B, V, 512, 512, 3], linear HDR

From the command line (reads an HDF5 scene, writes EXR + PNG per view):

uv run python infer_gaussian.py --h5_file scene.h5 --model_id shahafvl/gaussianformer-v17b --output_dir out/

Limitations

• Trained on isolated single objects centred in [-0.45, 0.45]^3, cameras at distance ~1.7 with 45° FOV. Scenes far outside that range degrade.
• Inputs of about 20k Gaussians; scene-encoder attention is quadratic in the number of Gaussians.
• Fine detail is noticeably softer than rasterizing the same splat.

License

Trained on Objaverse_Splats, a subset of Objaverse whose terms restrict commercial use; this checkpoint is CC-BY-NC-4.0, research and non-commercial use only.

Citation

GaussianFormer builds on RenderFormer; please cite:

@inproceedings{zeng2025renderformer,
  title     = {RenderFormer: Transformer-based Neural Rendering of Triangle Meshes with Global Illumination},
  author    = {Chong Zeng and Yue Dong and Pieter Peers and Hongzhi Wu and Xin Tong},
  booktitle = {ACM SIGGRAPH 2025 Conference Papers},
  year      = {2025}
}