shahafvl/gaussianformer-v21

GaussianFormer V21

GaussianFormer renders 3D Gaussian Splatting scenes with a transformer, without per-scene optimization and with no rasterizer in the renderer. It adapts RenderFormer (SIGGRAPH 2025), a transformer renderer for triangle meshes: each Gaussian's 14 parameters [pos(3), scale(3), quat(4), rgb(3), opacity(1)] become one scene token, and the two-stage architecture (view-independent scene encoder, view-dependent ray decoder with a DPT head) is kept and warm-started from RenderFormer's weights.

Code: github.com/SVLwoof/gaussianformer

How to use

Install the code with uv (Linux, NVIDIA GPU, Python 3.12); this also brings PyTorch for CUDA 12.8. Flash Attention is optional and recommended for speed: the flash extra adds a prebuilt wheel, and without it the model uses PyTorch's scaled-dot-product attention.

uv add "gaussianformer @ git+https://github.com/SVLwoof/gaussianformer"            # or: uv pip install "..."
uv add "gaussianformer[flash] @ git+https://github.com/SVLwoof/gaussianformer"     # with Flash Attention

Then, from a 3D Gaussian Splatting PLY:

import torch
from gaussianformer import GaussianFormerRenderingPipeline, load_ply
from gaussianformer.utils.cameras import orbit

pipeline = GaussianFormerRenderingPipeline.from_pretrained("shahafvl/gaussianformer-v21").to("cuda")
gaussians = load_ply("object.ply", up="z")     # [N, 14]; up = the file's up axis
c2w = torch.from_numpy(orbit(8, 1.7))          # [V, 4, 4] camera-to-world (-Z forward, +Y up)
images = pipeline(gaussians, c2w, fov=45.0)    # [V, 512, 512, 3]

The model takes each Gaussian as 14 activated values, pos(3) | scale(3) | quat wxyz(4) | rgb(3) | opacity(1): linear scales, a unit quaternion, colors and opacity in [0, 1], for an object centered, Y-up and scaled into [-0.45, 0.45]^3, about 20k Gaussians. A PLY stores log scales, raw quaternions, spherical-harmonic coefficients and opacity logits; load_ply converts them, normalizes the object, prunes it to 20k Gaussians and fine-tunes the kept ones against the full splat (about a minute on a GPU).

From the command line, in a clone of the repository (uv sync):

uv run python -m tools.ply_to_h5 --ply object.ply --out object.h5 --up z
uv run python infer.py --model shahafvl/gaussianformer-v21 --h5 object.h5 --out renders/   # one PNG per camera

This is the final recipe without the projected footprint: dense cross-attention, the projected 2-D RoPE and 4 px patches, trained exactly like shahafvl/gaussianformer (V22), which adds the footprint and is the recommended model. It is published for comparison.

Results

300 objects held out from training and never used while developing the model, PSNR on the square crop around the object against the full-splat ground truth, over all 28 training views and over 4 of them (views 0, 4, 7 and 11 at camera distance 1.7):

	28 views	4 views	SSIM	LPIPS
gsplat rasterization of the same 20k-Gaussian input	44.02	44.97	0.9954	0.0046
GaussianFormer V21 (this model)	36.67	36.87	0.9779	0.0140
GaussianFormer V22 (with the footprint)	38.08	38.36	0.9814	0.0113

On 300 training objects, at their 28 training views, V21 scores 36.86 dB.

Training

• Data: all 26,820 training objects of Objaverse_Splats, each pruned from ~50k to 20k Gaussians (LightGaussian importance score, then a short recovery fine-tune of the kept Gaussians). Targets are gsplat renders of the full, unpruned splat at 512 px: 28 views per object at camera distances 1.15, 1.7 and 2.45.
• Schedule: initialized from RenderFormer's weights and trained at 256 px (the published checkpoint gaussianformer-256px), then the projected 2-D RoPE and 4 px patches switched on in a short 256 px stage, then 385k steps at 512 px on 8 GPUs (one view per GPU per step; 4 passes over every view): a warmup-stable-decay learning rate at 5e-5, with a cosine decay to 5e-7 over the last 10k steps.
• Loss: log-HDR L1 (background pixels weighted 0.05) and LPIPS-VGG, 0.5 each.
• scripts/train.sh in the code repository runs all stages.

Limitations

• Trained on isolated single objects centered in [-0.45, 0.45]^3, with a 45° field of view, 512x512 pixels and camera distances 1.15 to 2.45; other resolutions, fields of view and distances cost quality. Real captures work only as regions cut out of a scene and viewed from outside.
• Inputs of about 20k Gaussians: denser inputs lower its quality, while they raise a rasterizer's, and the scene encoder's attention is quadratic in the number of Gaussians.
• Colors are view-independent (only the DC color of each Gaussian is used).
• Fine detail (thin strokes, engravings, small text) is softer than rasterizing the same splat, and the model is much slower than a rasterizer (about 0.3 s for one 512 px view of 20k Gaussians on an L40S).

License

MIT, for the weights as for the code. The model is trained on Objaverse_Splats, splats fit to Objaverse objects, each of which keeps its creator's license.

Citation

A paper on GaussianFormer is in preparation and will be presented at a later date; its citation will be added here.

GaussianFormer builds on RenderFormer; please also cite:

@inproceedings{zeng2025renderformer,
  title     = {RenderFormer: Transformer-based Neural Rendering of Triangle Meshes with Global Illumination},
  author    = {Chong Zeng and Yue Dong and Pieter Peers and Hongzhi Wu and Xin Tong},
  booktitle = {ACM SIGGRAPH 2025 Conference Papers},
  year      = {2025}
}