Pixal4D baseline LoRA adapters

Three incremental LoRA adapters only for SS64, Geometry512 and Material512. Each is rank32/alpha32, trained for 3000 optimizer steps with 8-frame sequences sampled at 8 FPS. Base weights, optimizer/RNG state and training data are not included.

Code and installation: hammershock/pixal4d.

python inference_pixal4d_baseline.py --video input.mp4

The inference code downloads the base weights automatically from TencentARC/Pixal3D at revision b0cb2e1b794cab9aa0ac38a95d794a4d9337437f, plus upstream DINOv3/NAF dependencies and Ruicheng/moge-2-vitl camera estimation weights. Follow the code repository's pinned MoGe-2 installation instructions for compatibility with Pixal3D's utils3d. baseline.json records adapter architecture, checksums and the exact trained base weight hashes. Material uses the native 1024 pretrained flow configured at grid32, matching the 512 baseline training setup. All three released stages use 512×512 RGB/DINO input (32×32 DINO patches); geometry NAF is 128×128 and material NAF is 256×256. A 1024×1024 source frame is resized directly to 512×512; the base texture1024 filename does not imply 1024×1024 image conditioning for Material512.

Recommended input: 512×512 or 1024×1024 video at 8 FPS. Segmentation and black-background compositing are explicitly optional (--preprocess), disabled by default. Maximum frames are unlimited unless --max_frames is supplied. Three-stage inference uses chunks of at most 8 frames, 1 overlapping frame, and the previous chunk's final three-stage latent reference; only the first video frame is initialized with frozen Pixal3D. The output is one animated textured GLB (output/<video_basename>.glb) plus a JSON timing manifest. Independent per-frame meshes retain their topology, normals, UVs and PBR textures; glTF STEP tracks select one frame at a time at the source timestamps, including the final frame duration. Repeated names receive a numeric suffix. Use --max_frames 22 --no-preprocess for exactly three full overlapping chunks of a prepared video.

Camera fitting follows native Pixal3D: estimate the first frame's FOV with MoGe-2, compute distance as 0.5 / (mesh_scale * tan(fov / 2)), and share the camera across the video. --fov skips estimation and supplies a known FOV; --distance explicitly overrides the fitted distance. This is independent of optional background preprocessing.

Validation and limitations

All three final checkpoints were recovered from OBS and verified against committed byte counts and SHA256. Every exported adapter tensor was compared exactly with its training checkpoint. A fresh Linux/A100 installation obtained source from GitHub and all weights automatically. After correcting the release's camera fitting, all 12 projected-conditioning GPU comparisons matched native Pixal3D exactly (two frames × two cameras × three stages). Three chunks (22 unique frames) of 1024², 8-FPS input with preprocessing disabled produced one textured GLB lasting 2.75 seconds. Stage conditioning sizes, frame times, textures and transfer hashes passed checks; Khronos validation reported zero errors/warnings. Three.js playback checks passed at every frame boundary, interval midpoint and loop. Visual inspection of frames 0, 7, 14 and 21 confirmed the complete foreground body, including the previously missing headwear and feet. This limited check does not establish full training correctness or general generation quality. For 24-FPS video select every third frame before inference, without interpolation. The estimated camera is not a calibrated reconstruction, and mesh topology can differ between frames.

Use upstream base model and dependency licenses/access conditions as applicable. This repository does not redistribute their full model weights.