GeoPano

GeoPano: reconstruct a colored panoramic point cloud and a dense point map from a single RGB equirectangular panorama.

Each input panorama is reconstructed independently; folder inference does not register multiple panoramas into one scene.

中文说明 · Inference guide · Checkpoint setup

Setup

Use Linux, Python 3.10, and an NVIDIA GPU with a CUDA-enabled PyTorch installation. The supplied research environment uses PyTorch 2.3.1+cu121. Start with:

conda env create -f environment.yml
conda activate geopano
pip install torch==2.3.1 torchvision==0.18.1 --index-url https://download.pytorch.org/whl/cu121
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"

Run these commands from the repository root. utils3d is vendored locally, so no additional Git submodule or installation is needed. The implementation uses PyTorch attention and a PyTorch RoPE fallback; compiling custom CUDA extensions and installing FlashAttention or xFormers are not required.

The reference machine exposes an A800 MIG GPU with 40 GB memory. GPU memory and latency depend on precision and hardware; this is not a minimum memory claim. CPU mode (--device cpu --precision fp32) is available but can be very slow and has not been validated for full inference in this release.

Checkpoint

Place the GeoPano inference weights at checkpoints/pytorch_model.bin:

checkpoints/
├── README.md
└── pytorch_model.bin

Weights are included in this model repository: download checkpoint. See checkpoint instructions. Original Pi3 or VGGT weights are not interchangeable with this model.

Quick start

Four real panoramas are included in examples/. Run all samples with one model load:

bash scripts/run_demo.sh
# Select weights and optionally export ten perspective views:
bash scripts/run_demo.sh --checkpoint /path/to/pytorch_model.bin --save-views

The demo discovers all images in examples automatically. Results are saved under outputs/real_samples//. Add --overwrite when rerunning.

For a real panorama or a directory:

python files_inference.py \
  --input /path/to/panorama.jpg \
  --checkpoint checkpoints/pytorch_model.bin \
  --output outputs

python files_inference.py --input /path/to/panoramas --output outputs/batch

Supported inputs: .jpg, .jpeg, .png, .bmp, in 2:1 equirectangular RGB format. Directory scanning is non-recursive. Inputs are resized to 2880×1440, split into ten 518×518 perspective views, and fused with center weighting, following the supplied inference pipeline. Perspective photographs and cubemap atlases are not supported input formats.

Browser demo

pip install -r requirements-demo.txt
python demo.py --checkpoint /path/to/pytorch_model.bin

Open http://127.0.0.1:7860. All four real samples appear in the example selector. Select one and click Run GeoPano to preview the colored point cloud, and download the PLY. Uploaded 2:1 panoramas are also supported. GPU jobs are queued one at a time; each browser request loads the model in a separate inference process.

Outputs

Each image produces outputs/<image_stem>/:

File	Meaning
pano_points.ply	Colored point cloud, viewable in Open3D, MeshLab, or CloudCompare
pano_points.npy	Float32 XYZ point map, shape (1440, 2880, 3)
pano_depth.npy	Float32 radial depth norm(XYZ), shape (1440, 2880)
pano_depth_color.png	Per-image depth visualization, clipped at the 99th percentile
pano_mask.png	255 for valid points, 0 for invalid points
metadata.json	Input size, precision, runtime, and visualization range

XYZ coordinates follow the panorama geometry convention in the original code. Depth is Euclidean distance from the panorama center, not perspective Z-depth. No external metric calibration is applied; do not assume the exported values are meters. Invalid XYZ and depth values are zeroed. The color PNG is a visualization, not a numeric depth encoding. The point map uses the resized panorama pixel grid.

Use --save-views to additionally export ten perspective images and point clouds. Use --overwrite to replace results for an existing image name. See docs/inference.md for options, array loading, and troubleshooting.

Repository layout

demo.py                 Browser demo with all real samples
files_inference.py       Command-line inference entry
geopano/                Panorama projection and fusion
pi3/                    Checkpoint-compatible model and DINOv2 backbone
utils3d/                Vendored geometry utilities
examples/               Four real panoramas and sample gallery
scripts/run_demo.sh     Run all bundled real samples
checkpoints/            Checkpoint placement instructions

Acknowledgments and license

GeoPano builds on Pi3, DINOv2, and utils3d. The original Pi3 license is retained in LICENSE; some components have additional upstream terms listed in THIRD_PARTY_NOTICES.md. Non-commercial restrictions apply. This release does not grant an unrestricted commercial license.

Paper title, author list, project links, and BibTeX will be added when the maintainer supplies the final publication metadata. No citation details are invented in this release.

Sky filtering: pass --sky-model /path/to/skyseg.onnx when launching the demo. The Remove sky points checkbox is enabled by default when the model exists. Preview and PLY download use the filtered cloud; original predictions remain unchanged. The optional third-party sky model is supplied separately under its original terms.