|
--- |
|
license: mit |
|
language: |
|
- en |
|
tags: |
|
- robotics |
|
- motion planning |
|
--- |
|
|
|
# Neural MP |
|
|
|
Neural MP is a machine learning-based motion planning system for robotic manipulation tasks. It combines neural networks trained on large-scale simulated data with lightweight optimization techniques to generate efficient, collision-free trajectories. Neural MP is designed to generalize across diverse environments and obstacle configurations, making it suitable for both simulated and real-world robotic applications. This repository contains the implementation, data generation tools, and evaluation scripts for Neural MP. |
|
|
|
All Neural MP checkpoints, as well as our [training codebase](https://github.com/mihdalal/neuralmotionplanner) are released under an MIT License. |
|
|
|
For full details, please read our paper(coming soon) and see [our project page](https://mihdalal.github.io/neuralmotionplanner/). |
|
|
|
## Model Summary |
|
- **Developed by:** The Neural MP team consisting of researchers from Carnegie Mellon University. |
|
- **Language(s) (NLP):** en |
|
- **License:** MIT |
|
- **Pretraining Dataset:** Coming soon |
|
- **Repository:** [https://github.com/mihdalal/neuralmotionplanner](https://github.com/mihdalal/neuralmotionplanner) |
|
- **Paper:** Coming soon |
|
- **Project Page & Videos:** [https://mihdalal.github.io/neuralmotionplanner/](https://mihdalal.github.io/neuralmotionplanner/) |
|
|
|
## Installation |
|
|
|
Please check [here](https://github.com/mihdalal/neural_mp?tab=readme-ov-file#installation-instructions) for detailed instructions |
|
|
|
## Usage |
|
|
|
Neural MP model takes in 3D point cloud and start & goal angles of the Franka robot as input, and predict 7-DoF delta joint actions. We provide a wrapper class [NeuralMP](https://github.com/mihdalal/neural_mp/blob/master/neural_mp/real_utils/neural_motion_planner.py) for inference and deploy our model in the real world. |
|
|
|
Here's an deployment example with the Manimo Franka control library: |
|
|
|
Note: using Manimo is not required, you may use other Franka control libraries by creating a wrapper class which inherits from FrankaRealEnv (check [franka_real_env.py](https://github.com/mihdalal/neural_mp/blob/master/neural_mp/envs/franka_real_env.py)) |
|
|
|
```python |
|
import argparse |
|
import numpy as np |
|
|
|
from neural_mp.envs.franka_real_env import FrankaRealEnvManimo |
|
from neural_mp.real_utils.neural_motion_planner import NeuralMP |
|
|
|
if __name__ == "__main__": |
|
|
|
parser = argparse.ArgumentParser() |
|
parser.add_argument( |
|
"--mdl_url", |
|
type=str, |
|
default="mihdalal/NeuralMP", |
|
help="hugging face url to load the neural_mp model", |
|
) |
|
parser.add_argument( |
|
"--cache-name", |
|
type=str, |
|
default="scene1_single_blcok", |
|
help="Specify the scene cache file with pcd and rgb data", |
|
) |
|
parser.add_argument( |
|
"--use-cache", |
|
action="store_true", |
|
help=("If set, will use pre-stored point clouds"), |
|
) |
|
parser.add_argument( |
|
"--debug-combined-pcd", |
|
action="store_true", |
|
help=("If set, will show visualization of the combined pcd"), |
|
) |
|
parser.add_argument( |
|
"--denoise-pcd", |
|
action="store_true", |
|
help=("If set, will apply denoising to the pcds"), |
|
) |
|
parser.add_argument( |
|
"--train-mode", action="store_true", help=("If set, will eval with policy in training mode") |
|
) |
|
parser.add_argument( |
|
"--tto", action="store_true", help=("If set, will apply test time optimization") |
|
) |
|
parser.add_argument( |
|
"--in-hand", action="store_true", help=("If set, will enable in hand mode for eval") |
|
) |
|
parser.add_argument( |
|
"--in-hand-params", |
|
nargs="+", |
|
type=float, |
|
default=[0.1, 0.1, 0.1, 0.0, 0.0, 0.1, 0.0, 0.0, 0.0, 1.0], |
|
help="Specify the bounding box of the in hand object. 10 params in total [size(xyz), pos(xyz), ori(xyzw)] 3+3+4.", |
|
) |
|
|
|
args = parser.parse_args() |
|
env = FrankaRealEnvManimo() |
|
neural_mp = NeuralMP( |
|
env=env, |
|
model_url=args.mdl_url, |
|
train_mode=args.train_mode, |
|
in_hand=args.in_hand, |
|
in_hand_params=args.in_hand_params, |
|
visualize=True, |
|
) |
|
|
|
points, colors = neural_mp.get_scene_pcd( |
|
use_cache=args.use_cache, |
|
cache_name=args.cache_name, |
|
debug_combined_pcd=args.debug_combined_pcd, |
|
denoise=args.denoise_pcd, |
|
) |
|
|
|
# specify start and goal configurations |
|
start_config = np.array([-0.538, 0.628, -0.061, -1.750, 0.126, 2.418, 1.610]) |
|
goal_config = np.array([1.067, 0.847, -0.591, -1.627, 0.623, 2.295, 2.580]) |
|
|
|
if args.tto: |
|
trajectory = neural_mp.motion_plan_with_tto( |
|
start_config=start_config, |
|
goal_config=goal_config, |
|
points=points, |
|
colors=colors, |
|
) |
|
else: |
|
trajectory = neural_mp.motion_plan( |
|
start_config=start_config, |
|
goal_config=goal_config, |
|
points=points, |
|
colors=colors, |
|
) |
|
|
|
success, joint_error = neural_mp.execute_motion_plan(trajectory, speed=0.2) |
|
``` |
|
|
