gaussian-inspector 0.1.0
Method agnostic Gaussian Splatting viewer and visual debugger.
Requires Python: >=3.7

# Introduction

GaussianInspector is a Python package that sets up a simple but flexible OpenGL
viewer and visual inspector for any 3DGS-based project (i.e. 4DGaussian, 4DGS, E-D3DGS,
Ex4DGS, etc.).
To make it project-agnostic, GaussianInspector provides a few stub files for callbakcs
which you need to implement (by copy-pasting from the project). This typically amounts
to a few lines of Python code that are specific to your 3DGS-based method, as shown in
the example usage below.

# Installation and requirements

This package needs to be installed with [uv](https://github.com/astral-sh/uv):
`uv pip install gaussian-inspector`.

Additionally, you may want `Docker` installed with the [NVIDIA Container
Toolkit](https://github.com/NVIDIA/nvidia-container-toolkit) setup. This greatly
facilitates the compilation of custom CUDA rasterizers and the CUDA devkit without
messing with your system. However, if you already have a working virtual environment for
the project you want to view, GaussianInspector can just be installed inside it.

# Usage

## In your existing environment
1. Run `gaussian-inspector init` in the project root to initialize the stubs.
2. Implement all methods in the stubs created in `<project-root>/gaussian_inspector_stubs/`.
3. Run the viewer: `gaussian-inspector launch`.

## In a custom-built Docker image

1. Run `gaussian-inspector init` in the project root to initialize the stubs
   and the Docker image template.
2. Build the custom Docker image: `gaussian-inspector build`.
3. Implement all methods in the stubs created in `<project-root>/gaussian_inspector_stubs/`.
4. Run the viewer: `gaussian-inspector launch-with-docker`. Optionally use `--mount
   src:dst` to mount dataset directories into the Docker image, as these are usually
loaded when initializing the Scene.

## Example implementation

As an example, we'll implement the stubs for the 4DGS project. There are 3 stubs to
implement: `configuration.py`, `camera.py` and `rendering.py`.

### `configuration.py`

In this stub file, we implement the `configure()` method that loads the model, scene
configuration, pipeline configuration, etc. This is typically different for each
project, and one may refer to the `render.py` or `train.py` typically found in the
project implementation.

The `configure()` function must return a `Dict[str, Any]`, and you are free to define
this `dict`  however you see fit. This will then be passed as a context variable to the
`rendering` and `camera` stubs. Note that you may use package imports as in the rest of
the project, since GaussianInspector adds the project in `sys.path`.

```python
from argparse import ArgumentParser
from typing import Any, Dict

import torch
from omegaconf import DictConfig, OmegaConf

from arguments import ModelParams, OptimizationParams, PipelineParams
from scene import Scene
from scene.gaussian_model import GaussianModel
from utils.general_utils import safe_state


def configure(sys_argv) -> Dict[str, Any]:
    """
    Configure the project from sys.argv and return the context of the project, which may
    be used for rendering and other purposes.
    """
    context = {}
    parser = ArgumentParser(description="Testing script parameters")
    model = ModelParams(parser, sentinel=True)
    op = OptimizationParams(parser)
    pipeline = PipelineParams(parser)
    parser.add_argument("--config", type=str)
    parser.add_argument("--iteration", default=-1, type=int)
    parser.add_argument("--skip_train", action="store_true")
    parser.add_argument("--skip_test", action="store_true")
    parser.add_argument("--quiet", action="store_true")
    parser.add_argument("--3DGS", dest="use_3dgs", action="store_true")
    parser.add_argument("--time_duration", nargs=2, type=float, default=[-0.5, 0.5])
    parser.add_argument("--num_pts", type=int, default=100_000)
    parser.add_argument("--num_pts_ratio", type=float, default=1.0)
    parser.add_argument("--force_sh_3d", action="store_true")
    parser.add_argument("--batch_size", type=int, default=1)
    parser.add_argument("--seed", type=int, default=6666)
    parser.add_argument("--exhaust_test", action="store_true")
    parser.add_argument("--spherical_coords", action="store_true")
    parser.add_argument("--max-frames", type=int, required=False)
    parser.add_argument("--per-dof", action="store_true")
    args = parser.parse_args(sys_argv[1:])
    # args.save_iterations.append(args.iterations)
    args.config = "configs/dynerf/cook_spinach.yaml"

    cfg = OmegaConf.load(args.config)

    def recursive_merge(key, host):
        if isinstance(host[key], DictConfig):
            for key1 in host[key].keys():
                recursive_merge(key1, host[key])
        else:
            assert hasattr(args, key), key
            setattr(args, key, host[key])

    for k in cfg.keys():
        recursive_merge(k, cfg)

    # Initialize system state (RNG)
    safe_state(args.quiet)

    model_args = model.extract(args)
    pipeline_args = pipeline.extract(args)
    global_args = args

    with torch.no_grad():
        gaussians = GaussianModel(
            model_args.sh_degree,
            model_args.max_dof,
            model_args.enable_rot_delta,
            model_args.enable_scale_delta,
            model_args.enable_sh_delta,
            force_sh_3d=model_args.force_sh_3d,
            time_duration=global_args.time_duration,
        )
        scene = Scene(
            model_args,
            gaussians,
            time_duration=global_args.time_duration,
            shuffle=False,
        )
        bg_color = [1, 1, 1] if model_args.white_background else [0, 0, 0]
        background = torch.tensor(bg_color, dtype=torch.float32, device="cuda")
    context = {
        "gaussians": gaussians,
        "background": background,
        "scene": scene,
        "pipeline_args": pipeline_args,
        "resolution": (1352, 1014),
    }
    return context
```

### `camera.py`

Here, we need to return a camera object that is expected by the renderer. Typically, we
can use the first test camera:
```python
def get_viewpoint_camera(user_context: Dict[str, Any]) -> Any:
    """
    Return the default viewpoint camera for the scene. This camera will be used as
    initialization for camera controls. It must be a valid camera object compatible with
    the rasterizer of your project.

    Args:
    user_context (Dict[str, Any]): The user context returned by the configure()
        function in configuration.py. This user context may store the Scene object or
        whatever you  need to fetch or create the initial camera.
    """

    return user_context["scene"].getTestCameras()[0].cuda()
```


### `rendering.py`

For rendering, we may copy and paste the rendering code of the original project (e.g.
from `render.py` or `train.py`). Then, we must convert the rendered image to a NumPy
array before returning it:
```python
def render_frame(context: Dict[str, Any], timestamp: float, camera_view: Any) -> np.ndarray:
    """
    Render a frame at time 'timestamp' with camera view 'camera_view'. The returned
    image must be a numpy array of shape (H, W, 3) with dtype uint8.
    """
    render_dict = render(
        camera_view,
        context["gaussians"],
        context["pipeline_args"],
        context["background"],
    )
    if render_dict is None:
        w, h = context["resolution"]
        image = torch.zeros((h, w, 3), dtype=torch.uint8).numpy()
    else:
        image = (
            (render_dict["render"] * 255).clamp(0, 255).cpu().numpy().astype(np.uint8)
        ).transpose(1, 2, 0)  # HWC
    return image
```


# Roadmap

- [x] v0.1.0: basic method-agnostic viewer with callbacks to user code (tested on
[4DGS](https://github.com/fudan-zvg/4d-gaussian-splatting).
and [3DGS](https://github.com/graphdeco-inria/gaussian-splatting)).
- [ ] v0.2.0: method-agnostic visual inspector with callbacks for data collection.
- [ ] v0.3.0: community library of implemented stubs for known projects.
- ...