zhuojianli/fr3_sim2real

FR3 Sim2Real simulation environments

Validation scope: All three reorganized backends passed GPU-hosted scene checks. All three also passed in clean Linux installations, including the generated Isaac Lab compatibility layer. These fixed-layout scripted checks are not a learned-policy benchmark.

Three versions of the FR3 put-on-plate task: MuJoCo, ManiSkill (SAPIEN), and Isaac Lab (PhysX). The code preserves the measured front and optional wrist cameras, reset variants, and shared action/reward definitions. The environments are research implementations: physics, rendering, native success, and TCP offsets differ across engines and must not be treated as bit-identical.

Repository layout

• compactvla/sim/: MuJoCo, ManiSkill, their common contract, and required local assets.
• compactvla/paired500_v3_*.py: scene geometry needed by the v3 tasks.
• compactvla/collection/ and compactvla/state_symmetry.py: privileged scripted pick-and-place experts and contact/placement helpers for the MuJoCo and ManiSkill backends; measured-placement v2 remains experimental.
• isaaclab/: the separate Isaac Lab environment, frozen assets, setup and verification scripts. See its README for the validated runtime and physical details.
• scripts/smoke_sim.py: native MuJoCo/ManiSkill reset and single-step check.
• scripts/expert_demo.py: deterministic v3 privileged expert demonstration, using the simulator's native success signal.
• scripts/check_mujoco_resets.py: checks 30 v3 variants at four fixed plate positions without rendering.
• tests/ and isaaclab/tests/: engine-independent checks. Isaac Lab keeps a frozen copy of the shared contract so it can run in a separate Python environment; the test suite checks for drift.

MuJoCo and ManiSkill

Use Linux and Python 3.11. The tested research stack uses CUDA for ManiSkill; rendering requires the relevant graphics drivers. Create separate environments when comparing the legacy MuJoCo 2.3.7 protocol with the aligned v3 MuJoCo 3.9.0 protocol:

The original research environment reports dependency conflicts: ManiSkill 3.0.0b22 declares mplib==0.1.1 on Linux, whereas that environment has 0.2.1 and its metadata declares NumPy <2. A separate Linux/Python 3.11 installation of the v3 extras resolved without conflicts and passed the dual-camera smoke and one fixed-layout expert run for both MuJoCo and ManiSkill. SAPIEN 3.0.1 uses pkg_resources, so the ManiSkill extra pins setuptools 80.9.0; upgrading that dependency requires a compatibility check.

python3.11 -m venv .venv
source .venv/bin/activate
pip install -e '.[mujoco,maniskill,test]'   # legacy MuJoCo
python scripts/smoke_sim.py mujoco
python scripts/smoke_sim.py maniskill

For v3 use pip install -e '.[mujoco-v3,maniskill,test]' in a fresh environment and append --v3 to the smoke commands. --no-images runs the MuJoCo physics probe without image rendering. Never compare a legacy run using an overridden MuJoCo version as if it used the pinned physics. The optional wrist camera uses RLINF_FR3_WRIST_CAMERA=1.

Install .[video] when using the optional MP4 rollout writer.

For a physics-only check on a machine without a graphics context, use MUJOCO_GL=disable python scripts/smoke_sim.py mujoco --no-images. A clean macOS/Python 3.11 installation of the legacy 2.3.7 extra passed reset and one physics step in this mode. This does not verify its image rendering or Linux GPU configuration.

For a longer v3 physics check, run python scripts/expert_demo.py mujoco --no-images. In a clean macOS/Python 3.11 environment this fixed layout reached native success in 84 control steps. Adding --all-variants runs all 30 pose/colour variants; all reached native success at this same layout. The expert uses privileged object positions; these episodes are not an evaluation of a learned policy or a guarantee of cross-engine success. The same entry point also accepts maniskill with images enabled.

On the existing Linux RTX 4090 runtime, both v3 backends passed fixed reset, one image/physics step and a complete scripted pick-and-place with front RGB. With RLINF_FR3_WRIST_CAMERA=1, both also passed nonblank wrist-image checks at reset and step. ManiSkill passed the fixed reset and one image/physics step for all 30 variants and reported native success for all 30 in the fixed-layout scripted demonstration. These checks used privileged state and the existing dependency environment; they are not a learned-policy benchmark.

MuJoCo also reported native success for all 30 fixed-layout variants with RGB rendering enabled. Machine-readable results and the tested source fingerprints are in evidence/gpu-validation-20261101/.

With the legacy 2.3.7 extra, MUJOCO_GL=disable python scripts/expert_demo.py mujoco --legacy --no-images --all-variants reproduced 25 successes out of 30 at the same fixed layout on macOS/Python 3.11. Variants 3, 9, 15, 21 and 27 failed; these share one pose and differ in colour. The command exits with a nonzero status when any variant fails. The legacy table geometry and physics differ from v3, so this is a diagnostic, not a controlled cross-version benchmark.

python scripts/check_mujoco_resets.py verifies each v3 pose/colour variant at four plate positions, including initial position and orientation, the success flag, and one finite physics step. The 120 cases passed in the same macOS environment; this does not test image rendering or contact-based pick-and-place for every variant.

Programmatic access uses compactvla.sim.sim2sim.adapters.make_teleop_adapter. A reset returns a 640×480 RGB image, seven-value state, reward and info; an action is [dx, dy, dz, droll, dpitch, dyaw, gripper]. +1 opens the gripper, -1 closes it. Always close the adapter when finished. The v3 MuJoCo scene requires paired500_v3_base_adapter=True and mujoco_table_center_base_x_m=.3; ManiSkill uses paired500_v3_assets_only=True.

The shared adapter deliberately uses Panda mechanics with a Franka Hand. The separate FR3_USE_URDF=1 experimental ManiSkill branch expects an external FR3 URDF and meshes that are not included here; it is not the validated shared protocol.

Isaac Lab

The verified Isaac runtime is Linux with an NVIDIA RTX GPU, Isaac Sim 6.0.1, Python 3.12 and the isolated Isaac Lab 2.2 compatibility layer. A clean installation passed a balanced-profile, dual-camera pick-and-place with real contact, as well as 30 reset variants and four plate positions. The default realistic profile passed reset, camera, action and support checks; its full pick-and-place was not rerun. Follow Isaac Lab setup; cd isaaclab before running its commands. Its bootstrap downloads NVIDIA software and requires acceptance of NVIDIA's license. The compatibility layer covers this task only, not all Isaac Lab APIs.

Verification

python -m pytest -q tests
cd isaaclab
python -m pytest -q tests

From the project root, install the lint extra and run ruff check . for the maintained Python sources. Vendored code and the frozen Isaac reference are excluded from this lint target.

Those tests check contracts, geometry, camera calibration and success logic without proving native GPU physics. Run each backend's native smoke test on the target machine before relying on a deployment. Isaac Lab has additional reset and pick/place checks in its README. The frozen reference data in isaaclab/evidence identify earlier validation sources; they are not a substitute for a test of a new revision.

For the published snapshot 86e4e658804148377c4e0d1196fe835f264caad9, all 443 downloaded files matched the Hub hashes. In a separate macOS/Python 3.11 environment installed from that download, all 63 tests passed; the MuJoCo v3 reset sweep and the 30 fixed-layout expert runs also passed without rendering. This verifies the downloaded snapshot, not later revisions or GPU backends.

Provenance and licensing

Some files include RLinf Apache-2.0 copyright headers; the license text is in LICENSE-APACHE-2.0. The original RLinf and FrankaSim license texts are also preserved in THIRD_PARTY_LICENSES/; vendored robot assets retain their own license files. isaaclab/evidence/assets.json records asset source paths and hashes, and compat6-source.json / compat6.patch identify the Isaac Lab compatibility changes. These notices do not grant a blanket license to every asset or unmarked file; confirm rights and upstream terms before redistribution.