neuromeka-grasp 0.1.2
Python client utilities for Neuromeka GraspGen Docker server
Requires Python: >=3.9

# neuromeka_grasp

Python client utilities for the GraspGen Docker server (ZeroMQ + pickle RPC).
Import and use it directly with `from neuromeka_grasp import GraspGeneration`.

## Installation
```bash
pip install neuromeka_grasp
```

Visualization helpers (`draw_grasps_overlay`) require OpenCV:
```bash
pip install "neuromeka_grasp[viz]"
```

## Requirements
- Python >= 3.9
- A running GraspGen Docker server from `neuromeka-repo/nrmk_graspgen`
- Server port is fixed to `5558`; the client defaults to this port and you usually do not need to change it
- Core dependencies: numpy, pyzmq
- Optional visualization dependency: opencv-python-headless (`viz` extra)

## Supported grippers (model presets)
These names map to server-side checkpoints (see `nrmk_graspgen/modules/graspgen/app.py`):
- `dh_ag_160_95` (default)
- `robotiq_2f_140`
- `franka_panda`
- `single_suction_cup_30mm`

You can also pass a `.yml` config path via `model` if the server has other checkpoints.

## Quick start (depth + mask)
```python
from neuromeka_grasp import GraspGeneration

client = GraspGeneration(hostname="localhost")
client.init(fx, fy, cx, cy, model="dh_ag_160_95")

resp = client.inference_from_depth_mask(
    depth=depth_np,
    mask=mask_np,
    fx=fx, fy=fy, cx=cx, cy=cy,
    enable_orientation_projection=True,
    approach_axis_source="local_normal_avg",
    enable_roll_projection=True,
    target_roll_direction="auto",
    enable_translation_projection=True,
    translation_axis=-1,
    desired_offset=0.03,
)

if resp["result"] == "SUCCESS":
    grasps = resp["data"]["grasps"]
    scores = resp["data"]["scores"]
```


## Point cloud example
```python
import numpy as np
from neuromeka_grasp import GraspGeneration

client = GraspGeneration(hostname="localhost")
client.init(fx, fy, cx, cy, model="dh_ag_160_95")

object_pc = np.asarray(object_points, dtype=np.float32)  # (N, 3)
scene_pc = np.asarray(scene_points, dtype=np.float32)    # (M, 3)

resp = client.inference_from_point_cloud(
    object_pc=object_pc,
    scene_pc=scene_pc,
    collision_check=True,
    grasp_threshold=0.8,
    num_grasps=200,
)
```

## Visualization (optional)
```python
from neuromeka_grasp import draw_grasps_overlay

overlay = draw_grasps_overlay(
    rgb=rgb_image,
    grasps=grasps,   # (K, 4, 4)
    scores=scores,   # (K,)
    fx=fx, fy=fy, cx=cx, cy=cy,
)
```

## API overview
- `GraspGeneration`
  - `init(fx, fy, cx, cy, model="dh_ag_160_95")`
  - `inference_from_point_cloud(...)`
  - `inference_from_point_cloud_with_aperture(...)`
  - `inference_from_depth_mask(...)`
  - `inference_from_depth_mask_with_aperture(...)`
  - `point_cloud_outlier_removal(obj_pc, threshold=0.014, k=20)`
  - `reset()`, `close()`
- `draw_grasps_overlay(...)`: Project predicted grasps onto an RGB image.

## Aperture API examples
```python
from neuromeka_grasp import GraspGeneration

client = GraspGeneration(hostname="localhost")
client.init(fx, fy, cx, cy, model="dh_ag_160_95")

# 1) point cloud + aperture API
resp = client.inference_from_point_cloud_with_aperture(
    object_pc=object_points,
    max_aperture=0.06,
    collision_check=True,
    aperture_jaw_axis_index=0,
    aperture_margin=0.002,
    aperture_depth_band=0.03,
    aperture_region_mode="closing_regions",
    aperture_unknown_policy="keep",
    aperture_chunk_size=256,
    max_object_points=5000,
    max_scene_points=8192,
)

# 2) depth mask + aperture API
resp2 = client.inference_from_depth_mask_with_aperture(
    depth=depth_image,
    mask=mask_image,
    max_aperture=0.06,
    fx=fx, fy=fy, cx=cx, cy=cy,
    collision_check=True,
    aperture_jaw_axis_index=0,
    aperture_margin=0.002,
    aperture_depth_band=0.03,
    aperture_region_mode="depth_band",
    aperture_min_points=12,
)

# image path (object mask generated from ROI or bool mask)
roi_resp = client.inference_from_depth_mask_with_aperture(
    point_cloud=object_points,   # optional direct point-cloud path
    max_aperture=0.06,
    aperture_jaw_axis_index=0,
)
```

## init() reference
**Signature**
`init(fx, fy, cx, cy, model="dh_ag_160_95")`

**Parameters**
- `fx`, `fy`: Focal lengths in pixels.
- `cx`, `cy`: Principal point in pixels.
- `model`: Gripper preset name (see Supported grippers) or a `.yml` config path.
  Config paths are resolved on the server side, so use a path visible to the server container.

**Behavior**
- Stores intrinsics in the client for future calls.
- Initializes the server model; response includes `gripper_name`, `model_path`, and `intrinsics`.

## inference_from_depth_mask() reference
**Signature**
`inference_from_depth_mask(depth, mask, fx=None, fy=None, cx=None, cy=None, ...)`

**Core inputs**
- `depth`: (H, W) depth image. Units are scaled by `depth_scale`.
- `mask`: (H, W) or (H, W, 1) segmentation mask. If `target_object_id` is set, only that label is used.
- `fx`, `fy`, `cx`, `cy`: Camera intrinsics. If omitted, values set by `init()` are used.

**Object/scene point clouds**
- `target_object_id`: If set, selects a single object label from `mask`.
- `depth_scale`: Multiplier applied to `depth` before projection.
- `max_object_points`: Random downsample limit for object points (None disables).
- `max_scene_points`: Random downsample limit for scene points (used only when collision_check=True).

**Sampling and filtering**
- `collision_check`: If True, runs collision filtering with the scene point cloud.
- `collision_threshold`: Distance (meters) for collision filtering.
- `grasp_threshold`: Minimum score threshold for grasps.
- `num_grasps`: Number of grasps to sample before filtering.
- `return_topk`: If True, also returns top-k grasps (server-side).
- `topk_num_grasps`: Number of top grasps to keep when `return_topk=True`.
- `min_grasps`: Minimum number of grasps to return (server-side).
- `max_tries`: Max sampling attempts (server-side).

**Orientation projection**
- `enable_orientation_projection`: Enable orientation constraint projection.
- `approach_axis_index`: Gripper-frame approach axis index (0=x, 1=y, 2=z).
- `approach_axis_source`: `"local_normal"`, `"local_normal_avg"`, or `"global_pca"`.
- `normal_avg_k`: KNN count for averaging normals in `"local_normal_avg"`.
- `target_approach_direction`: Target approach direction (3-vector, camera frame).
- `step_strength_orient`: Orientation correction strength in [0, 1].
- `use_surface_normal_for_approach`: Legacy switch; affects default behavior when `approach_axis_source` is not set.
- `contact_proxy_offset`: Offset (meters) along approach axis for contact proxy.

**Translation projection**
- `enable_translation_projection`: Enable translation constraint projection.
- `translation_axis`: PCA axis index (`-1` = principal axis).
- `desired_offset`: Target offset (meters) along PCA axis.
- `step_strength_trans`: Translation correction strength in [0, 1].

**Roll projection**
- `enable_roll_projection`: Enable roll projection around approach axis.
- `jaw_axis_index`: Gripper-frame jaw axis index (0=x, 1=y, 2=z).
- `target_roll_direction`: Target jaw direction (3-vector, camera frame) or `"auto"`.
- `roll_target_axis`: PCA axis index when `target_roll_direction="auto"`.
- `roll_target_axis_sign`: Optional sign override (+1 or -1) for the PCA axis.
- `roll_use_local_tangent`: Use local tangent (2D PCA) for `"auto"` roll targets (server support required).
- `step_strength_roll`: Roll correction strength in [0, 1].

**PCA sign and debug**
- `pca_axis_sign_ref`: PCA sign reference (`"camera_x"`, `"roll_ref"`, or a 3-vector).
- `projection_debug`: Enable projection debug logging on the server.

## Axis convention notes
- The rotation matrix columns represent the gripper-frame axes expressed in the camera frame.
- `approach_axis_index` and `jaw_axis_index` refer to those gripper-frame axes.
- When using `"global_pca"`, some server builds allow `approach_axis_index` or `roll_target_axis`
  values `-10/-11/-12` to select the opposite direction of PCA axis 0/1/2.

## PCA sign note
PCA eigenvectors can flip sign across runs. Use `pca_axis_sign_ref` to keep sign consistent
and avoid ambiguous translation/roll directions.

## Inputs and shapes
- `depth`: HxW float or uint depth image.
- `mask`: HxW or HxWx1 integer mask. If `target_object_id` is set, only that label is used.
- `depth_scale`: Multiplier applied to `depth` before projection.
- `object_pc`, `scene_pc`: (N, 3) and (M, 3) float32 point clouds in camera frame.
- `grasps`: (K, 4, 4) homogeneous poses, `scores`: (K,) in [0, 1].

## Projection parameters (summary)
- Orientation: `enable_orientation_projection`, `approach_axis_index`,
  `approach_axis_source` (`local_normal`, `local_normal_avg`, `global_pca`),
  `normal_avg_k`, `contact_proxy_offset`
- Translation: `enable_translation_projection`, `translation_axis`, `desired_offset`
- Roll: `enable_roll_projection`, `target_roll_direction` (supports `"auto"`),
  `roll_target_axis`, `roll_target_axis_sign`, `roll_use_local_tangent`
- Sign control and debug: `pca_axis_sign_ref`, `projection_debug`

For detailed parameter semantics, refer to the server README.

## Aperture filtering (separate API)
Use the dedicated methods instead of overloading the base inference API:

- `inference_from_point_cloud_with_aperture(...)`
- `inference_from_depth_mask_with_aperture(...)`

Key aperture params:
- `max_aperture` (required, meters)
- `aperture_margin`
- `aperture_jaw_axis_index` (defaults to `jaw_axis_index` then `0`)
- `aperture_axis_index` is kept for backward compatibility.
- `aperture_region_mode`: `"closing_regions" | "depth_band" | "all_points"`
- `aperture_depth_band`
- `aperture_min_points`
- `aperture_unknown_policy`: `"drop" | "keep"`
- `aperture_chunk_size` (`grasp_chunk_size` is backward-compatible alias)

Response shape:
- `data["inference"]`: base inference outputs
- `data["aperture"]`: required aperture estimates and aperture-filtered results
- `data["combined"]`: collision + aperture combined filtering result (when collision enabled)

## Notes
- `inference_from_depth_mask` requires intrinsics set via `init()` or passed directly.
- Server port is fixed at `5558`; the client default matches this and typically should not be changed.
- `PickleClient` is synchronous and blocking; set timeouts or retry logic on the server side if needed.
- Pickle is not secure against untrusted sources. Use only in trusted environments.