🦎 GapSeg

Diagnosis-Gated Learned Shape Completion

One RGB-D photo in → a complete 3D shape out ✨

---

A single photo of an object gives a broken, half-there 3D shape 🫠 — hidden sides are just missing. GapSeg is a little chameleon 🦎🔍 that looks at each object, guesses which ones came out distorted, and re-grows only those with a learned network. The result beats stitching many camera angles together.

_{one RGB-D frame → completed 3D, across 6 GraspNet scenes 🌀}

✨ Why it's neat

• 🔍 Diagnose first — predict per-object distortion from a single frame
• 🧩 Complete only what's broken — learned shape completion, gated by the diagnosis
• 🏆 Beats geometric multi-view fusion — with just one view + completion

📊 Results (GraspNet, 278 objects, chamfer mm ↓ lower = better)

method	chamfer	vs single
📷 single view	6.60	—
🧵 geometric fusion	5.64	88% ✅
🧩 single + completion	2.75	95% ✅
🦎 diagnosis-gated completion	2.84	94% ✅

🎯 One photo + learned completion (2.75 mm) beats fusing many views (5.64 mm). Gating it by the diagnoser keeps the gain while only completing the hard objects.

🚀 Usage

from gapseg.pipeline import GapSegPipeline
pipe = GapSegPipeline.from_pretrained("haeing/gapseg", device="cuda")

# rgb: HxWx3 uint8 · depth_m: HxW float (metres) · K: 3x3
# instances: [{"inst_id": 1, "mask": HxW bool, "R": 3x3, "t": (3,)}, ...]
out = pipe.infer(rgb, depth_m, K, instances)
for o in out:
    print(o["inst_id"], o["dims_cm"], "completed:", o["completed"])

🧭 R/t (object pose) drive completion; objects with no pose or below the diagnosis threshold return their single-view cloud.

📦 Contents

• 🧠 sq2_gview/best.pt — the G_view diagnoser (RGB-D → distortion / conditions)
• 🧩 sq3_completion/best.pt — the PCN shape-completion network
• 🖼️ gallery.gif — rotating demo across 6 scenes

_{made with 🦎 + 🔍 · single view, whole shape}