GaussianFormer V22

GaussianFormer renders 3D Gaussian Splatting scenes with a transformer, without per-scene optimization and with no rasterizer in the renderer. It adapts RenderFormer (SIGGRAPH 2025), a transformer renderer for triangle meshes: each Gaussian's 14 parameters [pos(3), scale(3), quat(4), rgb(3), opacity(1)] become one scene token, and the two-stage architecture (view-independent scene encoder, view-dependent ray decoder with a DPT head) is kept and warm-started from RenderFormer's weights.

Eight objects never seen in training: ground truth (full splat) and GaussianFormer, PSNR on the crop around the object.

Two held-out objects, rasterized (left) and rendered by GaussianFormer (right): a camera orbit and a tumbling object.

Code: github.com/SVLwoof/gaussianformer

How to use

Install the code with uv (Linux, NVIDIA GPU, Python 3.12); this also brings PyTorch for CUDA 12.8. Flash Attention is optional and recommended for speed: the flash extra adds a prebuilt wheel, and without it the model uses PyTorch's scaled-dot-product attention.

uv add "gaussianformer @ git+https://github.com/SVLwoof/gaussianformer"            # or: uv pip install "..."
uv add "gaussianformer[flash] @ git+https://github.com/SVLwoof/gaussianformer"     # with Flash Attention

Then, from a 3D Gaussian Splatting PLY:

import torch
from gaussianformer import GaussianFormerRenderingPipeline, load_ply
from gaussianformer.utils.cameras import orbit

pipeline = GaussianFormerRenderingPipeline.from_pretrained("shahafvl/gaussianformer-v22").to("cuda")
gaussians = load_ply("object.ply", up="z")     # [N, 14]; up = the file's up axis
c2w = torch.from_numpy(orbit(8, 1.7))          # [V, 4, 4] camera-to-world (-Z forward, +Y up)
images = pipeline(gaussians, c2w, fov=45.0)    # [V, 512, 512, 3]

The model takes each Gaussian as 14 activated values, pos(3) | scale(3) | quat wxyz(4) | rgb(3) | opacity(1): linear scales, a unit quaternion, colors and opacity in [0, 1], for an object centered, Y-up and scaled into [-0.45, 0.45]^3, about 20k Gaussians. A PLY stores log scales, raw quaternions, spherical-harmonic coefficients and opacity logits; load_ply converts them, normalizes the object, prunes it to 20k Gaussians and fine-tunes the kept ones against the full splat (about a minute on a GPU).

From the command line, in a clone of the repository (uv sync):

uv run python -m tools.ply_to_h5 --ply object.ply --out object.h5 --up z
uv run python infer.py --model shahafvl/gaussianformer-v22 --h5 object.h5 --out renders/   # one PNG per camera

Results

300 objects held out from training and never used while developing the model, PSNR on the square crop around the object against the full-splat ground truth, over all 28 training views and over 4 of them (views 0, 4, 7 and 11 at camera distance 1.7):

	28 views	4 views	SSIM	LPIPS
gsplat rasterization of the same 20k-Gaussian input	44.02	44.97	0.9954	0.0046
GaussianFormer	38.08	38.36	0.9814	0.0113

By camera distance: 38.47, 38.38 and 37.08 dB at 1.15, 1.7 and 2.45 (rasterization 42.39, 45.14 and 43.39).
On 300 training objects, at their 28 training views: 38.25 dB, 0.17 dB above the held-out objects.

A held-out object up close: ground truth, rasterization of the 20k-Gaussian input, GaussianFormer, and its absolute error.

Changes over RenderFormer

Besides the Gaussian input tokens and the training data:

Projected 2-D RoPE (proj_rope_2d): each Gaussian is projected into the view, and its image coordinates enter the view decoder's cross-attention as a 2-D rotary embedding matched against each patch's position.
Projected footprint (cov_key_embed): each Gaussian's covariance is projected into the view, and its footprint in the image (the sizes and orientation of its two axes) is added to its key in the cross-attention through a zero-initialized linear layer per decoder layer.
4 px patches (patch_size=4, ray_embed_patch=8): RenderFormer uses 8 px patches; 4 px gives a 128x128 token grid at 512 px. The pretrained 8 px ray embedding is reused.

Every patch attends to every Gaussian (dense cross-attention).

Training

Data: all 26,820 training objects of Objaverse_Splats, each pruned from ~50k to 20k Gaussians (LightGaussian importance score, then a short recovery fine-tune of the kept Gaussians). Targets are gsplat renders of the full, unpruned splat at 512 px: 28 views per object at camera distances 1.15, 1.7 and 2.45.
Schedule: initialized from RenderFormer's weights and trained at 256 px (the published checkpoint gaussianformer-256px), then the three changes below switched on in a short 256 px stage, then 385k steps at 512 px on 8 GPUs (one view per GPU per step; 4 passes over every view): a warmup-stable-decay learning rate at 5e-5, with a cosine decay to 5e-7 over the last 10k steps.
Loss: log-HDR L1 (background pixels weighted 0.05) and LPIPS-VGG, 0.5 each.
scripts/train.sh in the code repository runs all stages.

Limitations

Trained on isolated single objects centered in [-0.45, 0.45]^3, with a 45° field of view, 512x512 pixels and camera distances 1.15 to 2.45; other resolutions, fields of view and distances cost quality. Real captures work only as regions cut out of a scene and viewed from outside.
Inputs of about 20k Gaussians: denser inputs lower its quality, while they raise a rasterizer's, and the scene encoder's attention is quadratic in the number of Gaussians.
Colors are view-independent (only the DC color of each Gaussian is used).
Fine detail (thin strokes, engravings, small text) is softer than rasterizing the same splat, and the model is much slower than a rasterizer (about 0.3 s for one 512 px view of 20k Gaussians on an L40S).

License

MIT, for the weights as for the code. The model is trained on Objaverse_Splats, splats fit to Objaverse objects, each of which keeps its creator's license.

Citation

A paper on GaussianFormer is in preparation and will be presented at a later date; its citation will be added here.

GaussianFormer builds on RenderFormer; please also cite:

@inproceedings{zeng2025renderformer,
  title     = {RenderFormer: Transformer-based Neural Rendering of Triangle Meshes with Global Illumination},
  author    = {Chong Zeng and Yue Dong and Pieter Peers and Hongzhi Wu and Xin Tong},
  booktitle = {ACM SIGGRAPH 2025 Conference Papers},
  year      = {2025}
}