iter12000-full-eval

Read BUGFIX_farm_suffix_gt_transform.md at this repo's root FIRST (also uploaded 2026-09-08). It documents a real correctness bug in codes/gt_model_transforms.py that silently skips GT resize/reprojection for any model with a real _TRANSFORMS entry (omniroam_perspective, panoworld_perspective, ours_panoonly_MASK_CAMFIX_FIXED_perspective, fantasy-world, HyDRA_concat, hy-worldplay) whenever FVD/pooled-FID is run over a build_fvd_200clip_farm.py farm. The fixed gt_model_transforms.py is already uploaded to this repo's code/ tree (overwriting the stale copy) -- if you downloaded code/ before this date, re-download that one file, or diff against the version in this repo now. Model_ours_iter12000_FINAL itself was never affected (it's _t_noop), but re-check before trusting any OTHER model's farm-based FVD/FID number you compute with this package.

Goal: run the full benchmark pipeline (same as Model_ours_iter11000_FINAL) on Model_ours_iter12000_FINAL — point-prompt metrics (FID/FVD/camera_da3/ 3d_adherence/vbench/outer_view/object_consistency/oov_direction), FVD's genuine 200-clip re-run, bbox-prompt tracking + re-measurement, bbox+edgefix, then the final 200-clip filter/aggregate — then hand the results back.

This repo is private and (almost) fully self-contained. The one thing it deliberately does not duplicate is the GPU environment (venv) and the SAM3 + DA3 checkpoints, since those are identical to and already sitting in kimhosoo/multipano-13clip-fillin (a sibling private repo from an earlier task) — reuse them from there rather than re-uploading ~16GB again.

0. Prerequisites

• NVIDIA GPU, CUDA available.
• ~90GB free disk (model output 13G, GT 4.7G, sam3-env 9.9G, vbench-env 5.9G, SAM3+DA3 checkpoints ~16.5G reused from the other repo, CLIP+DINOv3 1.5G, gt_mask_cache 71M, plus room to work).
• HF_TOKEN from the user (never pasted in chat — have them export it in their own shell, or point you at a file you can source without printing).
• If any hf upload/hf download call hangs or times out, retry with HF_HUB_DISABLE_XET=1 prefixed — every upload in both repos needed this.

1. Download both repos

export HF_TOKEN=...
mkdir -p ~/iter12000_eval

# this repo: code, GT (trimmed), model output, gt_mask_cache, CLIP+DINOv3, vbench-env.tar
HF_HUB_DISABLE_XET=1 hf download kimhosoo/iter12000-full-eval --repo-type model --local-dir ~/iter12000_eval

# sibling repo: reuse its sam3-env.tar + SAM3/DA3 checkpoints (identical environment, no need to rebuild)
HF_HUB_DISABLE_XET=1 hf download kimhosoo/multipano-13clip-fillin --repo-type model \
  --include "environment/sam3-env.tar" \
  --include "checkpoints/hf_cache/hub/models--facebook--sam3/*" \
  --include "checkpoints/hf_cache/hub/models--depth-anything--DA3NESTED-GIANT-LARGE-1.1/*" \
  --local-dir ~/iter12000_eval

After both downloads, ~/iter12000_eval/ has:

code/NHNHOME/hyunwook/khs/codes/...                         <- every eval script + its deps + sam3 source
data/NHNHOME/hyunwook/khs/BENCH_OUTPUT/Model_ours_iter12000_FINAL/...   <- the model output to evaluate (230 clips)
data/NHNHOME/hyunwook/khs/JointGen-backup/BENCHMARK/FINAL/{real,synth}/perspective/**/*.gt.mp4
data/NHNHOME/hyunwook/khs/JointGen-backup/BENCHMARK/FINAL/{real,synth}/perspective/**/caption_selected/*.gt.meta.json
data/NHNHOME/hyunwook/khs/codes/metric/gt_mask_cache/...    <- pre-computed GT dynamic-object masks (saves re-running SAM3 on GT)
checkpoints/hf_cache/hub/models--openai--clip-vit-base-patch16/...
checkpoints/hf_cache/hub/models--facebook--dinov3-vitb16-pretrain-lvd1689m/...
checkpoints/hf_cache/hub/models--facebook--sam3/...                    <- from the sibling repo
checkpoints/hf_cache/hub/models--depth-anything--DA3NESTED-GIANT-LARGE-1.1/...  <- from the sibling repo
environment/sam3-env.tar          <- from the sibling repo (~9.9G)
environment/vbench-env.tar        <- this repo (~5.9G, separate venv, only eval_vbench.py uses it)

Only .gt.mp4 and caption_selected/*.gt.meta.json are included per GT clip — not masks/warped videos/DA3 caches/w2cs (confirmed not read by any of the scripts here). This trims real+synth GT from ~16G down to ~4.7G. camera_da3's own GT-side DA3 cache isn't included either, so eval_camera_da3.py will compute it fresh the first time (one-time GPU cost, unavoidable without re-uploading ~14G of cache).

2. Extract both venvs and place everything at the exact absolute paths

mkdir -p ~/iter12000_eval/environment/sam3-env ~/iter12000_eval/environment/vbench-env
tar -xf ~/iter12000_eval/environment/sam3-env.tar -C ~/iter12000_eval/environment/sam3-env --strip-components=1
tar -xf ~/iter12000_eval/environment/vbench-env.tar -C ~/iter12000_eval/environment/vbench-env --strip-components=1

sudo mkdir -p /NHNHOME/hyunwook/khs
sudo chown -R "$(whoami)" /NHNHOME
rsync -a ~/iter12000_eval/code/NHNHOME/ /NHNHOME/
rsync -a ~/iter12000_eval/data/NHNHOME/ /NHNHOME/

(--strip-components=1 because both tars were made with tar -cf x.tar sam3-env/ tar -cf x.tar vbench-env, i.e. the venv directory name is the tar's own top level — strip it so the venv's bin/, lib/, etc. land directly under environment/sam3-env/ / environment/vbench-env/, not one level deeper.)

You should now have, among other things:

• /NHNHOME/hyunwook/khs/codes/metric/run_new_model_pipeline.sh (the entry point)
• /NHNHOME/hyunwook/khs/BENCH_OUTPUT/Model_ours_iter12000_FINAL/{Real,Syn}/<clip>/{output.mp4,metadata.json} (230 clips)
• /NHNHOME/hyunwook/khs/JointGen-backup/BENCHMARK/FINAL/{real,synth}/perspective/**/*.gt.mp4
• /NHNHOME/hyunwook/khs/codes/metric/review_previews_100/review_decisions.json (the 200-clip selection)
• /NHNHOME/hyunwook/khs/JointGen-backup/Projects/sam3/{sam3/,sam3.egg-info/,scripts/track_and_extract_masks.py}

3. Verify both venvs

~/iter12000_eval/environment/sam3-env/bin/python3 -c "import torch, cv2, lpips, depth_anything_3; from sam3.model_builder import build_sam3_video_predictor; print('sam3-env: cuda', torch.cuda.is_available())"
~/iter12000_eval/environment/vbench-env/bin/python3 -c "from vbench import VBench; print('vbench-env OK')"

Both should succeed with cuda True for the first. If either venv fails to start at all (can't find its base interpreter), check pyvenv.cfg inside it for the home = path — it needs a matching system Python on this machine (sam3-env wants 3.12, check vbench-env's own cfg for its version). If they don't match, stop and report back rather than trying to patch the venv.

4. Point checkpoints at the bundled cache (offline, no HF auth needed for gated SAM3/DA3/DINOv3)

export HF_HOME=~/iter12000_eval/checkpoints/hf_cache
export TORCH_HOME=~/iter12000_eval/checkpoints/torch_cache   # created fresh if AlexNet/Inception/S3D weights aren't already cached -- they'll just download once, small (~150MB total), no auth needed (public weights)
export HF_HUB_OFFLINE=1

Note: HF_HUB_OFFLINE=1 covers SAM3/DA3/CLIP/DINOv3 (all pre-cached here). It does NOT need to cover torch-hub-style downloads (InceptionV3 for FID, S3D for FVD, AlexNet for LPIPS) — those are public, ungated, and will just download the first time eval_fid.py/eval_fvd.py/eval_3d_adherence.py run, same as they did when this package was built.

5. Run the full pipeline

cd /NHNHOME/hyunwook/khs/codes/metric
export SAM3PY=~/iter12000_eval/environment/sam3-env/bin/python3   # run_new_model_pipeline.sh hardcodes its own SAM3PY path -- see note below
./run_new_model_pipeline.sh --model_output_dir /NHNHOME/hyunwook/khs/BENCH_OUTPUT/Model_ours_iter12000_FINAL --gpus 0,1

Before running, open run_new_model_pipeline.sh and run_full_benchmark.py and check the SAM3PY= / VBENCHPY= constants near the top of each — they're hardcoded to this server's original absolute paths (/NHNHOME/hyunwook/khs/JointGen-backup/Projects/sam3/sam3-env/bin/python3 and .../codes/metric/vbench-env/bin/python3). Since you extracted the venvs to those SAME absolute paths in step 2, these should already resolve correctly as-is — this note is just so you notice immediately if something doesn't line up (e.g. you extracted somewhere else) rather than debugging a confusing ModuleNotFoundError several phases in.

No --panorama flag — Model_ours_iter12000_FINAL isn't a panorama-crop model (same convention as Model_ours_iter11000_FINAL).

This takes hours (230 clips × real+synth × every metric × bbox re-tracking). run_step logs ########## START <phase> ########## / DONE/FAILED for each of: point_prompt, fvd_farm (200-clip FVD re-run), bbox_sam3, bbox_outer_view, bbox_objcons_clip, bbox_objcons_dino, bbox_direction_self, bbox_direction_gt, edgefix_direction_self, edgefix_direction_gt, filter_200_bbox, filter_200_point — if one phase fails, the script logs FAILED and continues to the next rather than aborting (matching this project's convention of never silently corrupting already-good phases); check the log for any FAILED lines before declaring done, and report back rather than guessing at a fix if you see one.

6. Sanity-check before uploading results

python3 -c "
import json
for m in ['fid','fvd','camera_da3','3d_adherence','vbench']:
    d = json.load(open(f'/NHNHOME/hyunwook/khs/EVAL_FINAL/{m}/Model_ours_iter12000_FINAL.json'))
    print(m, d if m!='camera_da3' else d['RotErr'])
"
python3 -c "
import json
d = json.load(open('/NHNHOME/hyunwook/khs/EVAL_FINAL_bbox/outer_view/Model_ours_iter12000_FINAL.json'))
print('n_clips_matched:', d['n_clips_matched'])
"

Expect n / n_clips_matched around 200 (the 200-clip selection: 100 real + 100 synth). If any of these show a much smaller n or the file is missing, some phase silently skipped clips — check that phase's log before uploading.

7. Upload results back

Only the actual result artifacts — not the 13GB of model output or the venvs.

export HF_HOME=~/iter12000_eval/checkpoints/hf_cache
unset HF_HUB_OFFLINE

for D in EVAL_Results EVAL_Results_oldFINAL_bbox EVAL_Results_oldFINAL_bbox_edgefix EVAL_FINAL EVAL_FINAL_bbox EVAL_FINAL_bbox_edgefix; do
  HF_HUB_DISABLE_XET=1 hf upload kimhosoo/iter12000-full-eval \
    "/NHNHOME/hyunwook/khs/$D" "results/$D" --repo-type model \
    --include "*Model_ours_iter12000_FINAL*"
done

(Each --include restricts to just this model's own result files within each results directory — build_eval_final_bbox.py sweeps every model it finds, so EVAL_FINAL_bbox/EVAL_Results_oldFINAL_bbox* may contain other models' files too; don't upload those, just this one's.)

8. Tell the user it's done

Once results/ is uploaded, report back to the user that the full iter12000_FINAL eval is ready to pull down and merge on the original server. You don't need to do the merge yourself.