Cosmos-3-ac-Surgical

Cosmos-3-ac-Surgical is an action-conditioned LoRA fine-tune of nvidia/Cosmos3-Super on surgical robotics sequences from nvidia/PhysicalAI-Robotics-Open-H-Embodiment.

The model predicts short-horizon surgical video rollouts from an initial endoscopic frame and a 12-step robot action trajectory. It uses the native Cosmos 3 action modality with a surgical robotics bridge that maps supported Open-H layouts into one model-facing action contract.

Source code and FlashDreams runtime: hcltech-robotics/cosmos-3-ac-surgical

Model status

• Base model: nvidia/Cosmos3-Super
• Fine-tuning method: LoRA on generated-modality attention projections
• Fine-tuning data: Open-H-Embodiment surgical robotics branches
• Fine-tuning hardware: 8x GB200 GPUs
• Task: forward dynamics from an initial endoscopic frame plus a robot action chunk
• Checkpoint: checkpoints/latest_checkpoint.txt resolves to iter_000000060
• Checkpoint format: PyTorch Distributed Checkpoint
• Inference weights: regular weights; EMA loading is disabled
• Runtime integration: FlashDreams runner and WebRTC serving adapter

Inference contract

The model takes:

initial endoscopic RGB frame
+ action[12, 44]
+ domain/view/task metadata
-> predicted video frames[13]

Default runtime parameters:

• model_mode: forward_dynamics
• domain_name: open_h_surgical_sim
• view_point: ego_view
• frame staging: 512 x 288
• Cosmos image size: 256
• FPS metadata: 30
• sampling steps: 16
• guidance: 3.0
• shift: 5.0
• seed: 3407

Action conditioning

Each action row contains 44 floating point values. The first 20 dimensions carry a dVRK-style dual-arm relative pose command, and dimensions 20:44 are reserved bridge channels that remain zero for the supported dVRK-style layouts.

Index range	Field
0:3	left arm relative translation (dx, dy, dz)
3:9	left arm relative rotation in 6D rotation representation
9	left jaw/gripper target
10:13	right arm relative translation (dx, dy, dz)
13:19	right arm relative rotation in 6D rotation representation
19	right jaw/gripper target
20:44	reserved bridge channels, zero padded

The relative pose is computed as:

T_relative = inverse(T_current) * T_target

Translations are expressed in the current tool frame. The 6D rotation representation is the first two columns of the relative rotation matrix, flattened in the training adapter convention.

Action JSON file

Inference accepts a nested numeric list or an object containing an action array. This command writes a valid neutral action file with 12 rows and 44 values per row:

python3 - <<'PY'
import json
from pathlib import Path

row = [
    0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0,
    0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0,
    *([0.0] * 24),
]
Path("runs/local").mkdir(parents=True, exist_ok=True)
Path("runs/local/actions.json").write_text(json.dumps({"actions": [row] * 12}, indent=2) + "\n")
PY

The FlashDreams runtime validates the final 12 x 44 tensor and writes both actions.json and actions.npy into each Cosmos 3 sample directory.

FlashDreams runtime

The GitHub repository provides:

• cosmos-3-ac-surgical-run: batch runner for a first frame and action file
• cosmos-3-ac-surgical-webrtc: browser-controlled WebRTC serving adapter
• flashdreams.runner_configs entry point: cosmos-3-ac-surgical
• keyboard-to-action integration for dual-tool control
• JSON action loading and .npy preconversion
• Hub checkpoint resolution for this model repository
• Cosmos 3 input JSON construction for cosmos_framework.scripts.inference

Batch invocation:

cosmos-3-ac-surgical-run \
  --config configs/cosmos3-super-webrtc.toml \
  --cosmos3-root "$COSMOS3_ROOT" \
  --project-root "$PWD" \
  --image-path runs/local/first_frame.png \
  --actions-path runs/local/actions.json \
  --output-root runs/cosmos-3-ac-surgical

WebRTC invocation:

cosmos-3-ac-surgical-webrtc \
  --config configs/cosmos3-super-webrtc.toml \
  --cosmos3-root "$COSMOS3_ROOT" \
  --project-root "$PWD" \
  --first-frame runs/local/first_frame.png

The runtime resolves this Hub checkpoint when --checkpoint is omitted. It uses --no-use-ema-weights, cosmos3_h_surgical_simulator/experiment.py, and the cosmos3_super_openh_surgical_lora experiment.

Checkpoint layout

checkpoints/
  latest_checkpoint.txt
  iter_000000060/
    model/
      .metadata
      __0_0.distcp
      ...
      __7_0.distcp

The repository also includes config.yaml and checkpoint_manifest.json for inspection and reproducibility.

Intended use

This checkpoint and runtime support research on action-conditioned world modelling for surgical robotics, offline rollout generation from logged Open-H-style trajectories, and simulator UI evaluation with explicit action conditioning.

Safety boundary

Generated videos are model predictions. They are not clinical evidence, medical advice, surgical guidance, or robot-control commands. The FlashDreams runtime is not intended to directly actuate robots.

Citation

@misc{voncsefalvay2026cosmos3acsurgical,
  title        = {Cosmos-3-ac-Surgical: An action-conditioned Cosmos 3 Super fine-tune for surgical robotics world modelling},
  author       = {von Csefalvay, Chris},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/hcltech-robotics/cosmos-3-ac-surgical}},
}

Please also cite NVIDIA Cosmos 3 and Open-H-Embodiment when using this model.

License and acknowledgement

This project is released under the OpenMDW license.

Cosmos is a trademark of NVIDIA. Cosmos-3-ac-Surgical is inspired by NVIDIA's Cosmos-H-Surgical-Simulator, but it is a separate project and is not affiliated with, endorsed by or sponsored by NVIDIA or the Cosmos-H-Surgical-Simulator project.