kimhosoo/camera-error-metric

Camera error metric (RotErr / TransErr / CamMC)

Files (keep this directory layout -- scripts resolve imports via the parent dir):

codes/metric/eval_camera_da3.py          # main script (DA3 pose estimation + metric math)
codes/benchmark_object_masking.py        # discover_sample_dirs(), remap_path()
codes/gt_model_transforms.py             # per-model GT resize/reprojection (farm-suffix bugfix applied)
codes/panorama_to_perspective_crop.py    # used by gt_model_transforms for panorama models

How it works

For each clip, DA3 (Depth-Anything-3) is run once over the full 77-frame GT video and once over the model's output.mp4, giving one W2C extrinsic per frame per side. Trajectories are made first-frame-relative and translation-normalized, then RotErr / TransErr / CamMC are computed (calc_rot_err, calc_trans_err, calc_cammc in eval_camera_da3.py; math adapted from SWM_bench). Per-model result = mean over clips (and per split Real/Syn).

External dependencies (NOT included -- install/download yourself)

• Python env with torch, opencv-python, numpy, Pillow
• depth_anything_3 package (Depth-Anything-3) and the checkpoint depth-anything/DA3NESTED-GIANT-LARGE-1.1 (auto-downloaded from the HF Hub via DepthAnything3.from_pretrained)
• NVIDIA GPU (DA3 is heavy: ~70s/clip for GT+pred on a B200)
• benchmark_object_masking.py adds a SAM3 scripts dir to sys.path but does not need SAM3 for camera error; SAM3_SCRIPTS_DIR can be left as-is or pointed anywhere.

Input layout

<BENCH_OUTPUT>/Model_X/{Real,Syn}/<clip>/{output.mp4, metadata.json}

metadata.json must contain context_len, target_len, video_path (GT .gt.mp4; prefix /home/work/.local/khs is remapped by remap_path()). GT videos are expected under BENCHMARK/FINAL/{real,synth}/perspective/....

Paths to adapt (hardcoded to the original server)

• benchmark_object_masking.py: GT_PATH_PREFIX_OLD/NEW, SAM3_SCRIPTS_DIR
• eval_camera_da3.py: RESULTS_DIR (output dir)
• panorama_to_perspective_crop.py: DEFAULT_BENCH_OUTPUT_ROOT, DEFAULT_BENCHMARK_FINAL_ROOT

Usage

# batch over a model (subprocess-per-clip, GPU round-robin)
python3 codes/metric/eval_camera_da3.py --model_output_dir /path/BENCH_OUTPUT/Model_X \
    --gpus 0,1 --workers 4 [--stride 5]
# single clip
python3 codes/metric/eval_camera_da3.py --sample_dir /path/BENCH_OUTPUT/Model_X/Real/<clip> \
    --device cuda:0

Output: <sample_dir>/camera_da3.json per clip and <RESULTS_DIR>/<Model_X>.json aggregate.

Notes

• Panorama-crop models (OmniRoam, PanoWorld, panorama-only ablations) reuse the GT yaw/pitch trajectory to crop their own panorama, so their RotErr is near-zero by construction; gt_model_transforms.py handles the GT-side resolution matching for them.
• If running over a __final200 symlink farm, make sure gt_model_transforms.py is this (fixed) version, otherwise model-specific GT transforms are silently skipped.