torchrl/microduck-skills

MicroDuck skills for TorchRL

The new nine-skill prior restores forward hopping, adds both turns and in-place hopping, and rewards a level head aligned with the body's heading. It is published separately under priors/nine-skills-20260913. The existing root checkpoint, videos and navigation policy are preserved.

Nine-skill release: September 13, 2026

Immutable artifact revision: 01ebcefca08850231edc0eb428a0151474559a85.

Checkpoint, video, metrics, acceptance report, ordered definitions, training configuration and lineage, and SHA-256 manifest.

Indices 0–5 retain the original six-skill meanings. Index 5 is forward hopping; indices 6 and 7 turn left and right; index 8 hops in place. jump_task() still defaults to hopping in place; jump_task(speed=0.3) sets the forward command. Each hopping task is sampled three times as frequently as each other task.

Evaluation

All 288 held-out episodes survived: 32 episodes for each skill, 500 steps (10 seconds) per episode, deterministic actions, seed 20260915, native MuJoCo in the flat scene. The video uses separate seed 20260916. Its camera follows translation but keeps its orientation fixed, making turns visible while keeping walking ducks in view. Tiles are labeled with skill index, name and head pitch.

Index	Skill	Survival	Ground forward (m/s)	Ground lateral (m/s)	Heading rate (rad/s)	Both feet airborne
0	Standing	100%	+0.001	-0.001	+0.002	0.0%
1	Walk forward	100%	+0.231	-0.003	+0.014	0.0%
2	Walk backward	100%	-0.221	-0.020	+0.008	0.0%
3	Sidestep left	100%	+0.006	+0.215	+0.037	0.0%
4	Sidestep right	100%	+0.002	-0.216	-0.037	0.0%
5	Hop forward	100%	+0.285	+0.001	+0.016	43.8%
6	Turn left	100%	-0.009	-0.004	+1.016	0.0%
7	Turn right	100%	-0.005	+0.015	-0.802	0.0%
8	Hop in place	100%	+0.046	-0.001	+0.082	44.0%

Standing absolute head pitch/yaw 95th percentiles are 1.08° / 0.37°, both below 10°. Forward hopping reaches 0.285 m/s against a 0.30 m/s target. Both hopping skills have repeated takeoffs and landings in every episode, averaging 55 takeoffs and at least 54 landings in ten seconds. Their mean per-episode maximum height gain above the standing reference is about 2.3 cm and 2.5 cm. Airtime is 43.8% / 44.0%, compared with the original release's 43.3% (reproduced as 43.1% over 32 episodes).

In-place hopping has 0.0436 m/s mean net drift (0.0446 m/s worst episode) and 0.0462 m/s heading-relative ground drift, both below the 0.05 m/s target. It still travels up to 0.445 m during a ten-second episode and slowly changes heading; this is a measured residual limitation, not perfect stationarity. Turning direction is correct in every episode; the right turn is slower than its commanded -1 rad/s. Moving head pitch can exceed 10°; the 10° acceptance target applies to standing.

Ground speeds come from horizontal position differences projected onto the robot's heading. This avoids confusing vertical hopping with translation when the torso tilts. Net drift uses start/end displacement, and maximum excursion is also recorded. Head yaw is relative to trunk heading. Metrics exclude trajectory padding. These results cover one trained seed and the flat reset distribution; arbitrary-goal navigation has not been evaluated with this new prior. The separate football experiment is described below.

Training and corrected geometry

The 128-unit recurrent actor controls 14 joints from the 56-value observation and a learned task embedding. Training started from the original six-skill release at 4191d7d25c4fd58a5c6e6395fcf8217459fdd073 (10,010,624 transitions). Embedding expansion used [0, 1, 2, 3, 4, 5, 0, 0, 5]: turns start from standing and in-place hopping starts from the old hopping row. Before optimization, actions and recurrent outputs matched bitwise on all mapped rows across four observation probes (verification). The initial optimizer and critic head were fresh.

This selected checkpoint has 16,629,760 additional transitions (26,640,384 including the original training). The initial 12M budget was extended because in-place drift still exceeded its target. PPO ran on an Apple-silicon MacBook with 16 native MuJoCo CPU workers, batches of 16,384, 2,048-transition minibatches, 128-step sub-trajectories, five epochs, learning rate 3e-4, entropy coefficient 0.01 and per-task advantage normalization. Reward refinements and their exact source commits are recorded as four phases in training.json; actor, critic and optimizer state continued between phases, while simulator trajectories reset. This was not one fixed-reward run.

The previous IMU-to-beak line was tilted 41.16° below the actual head forward axis and encouraged an upward head posture. The corrected gaze uses the head IMU frame's local -z axis. Head reward penalizes pitch relative to the horizon and yaw relative to the trunk, allowing the whole robot to turn. Hop launch and rhythm use world-vertical velocity; in-place drift uses world-horizontal velocity, avoiding torso-pitch reward artifacts. Historical head-angle claims made with the old landmark measurement are not calibrated gaze measurements.

The implementation, diagnostics, tests and tutorial are in TorchRL PR #4330; the related head fix is synchronized in #4361. Training/evaluation source: c514a1361b5ff7fb8e86019a86d22c5e0bdd2697. training.json pins the robot assets and records runtime versions and asset hashes.

Load the nine-skill prior

Use a TorchRL checkout containing PR #4330 and its parents, with mujoco, huggingface_hub and TorchRL's utils extra installed. Pin the artifact commit in revision when downloading. The checkpoint contains its architecture and ordered task definitions; it is an actor-only inference export, not a full optimizer checkpoint.

import hashlib
from pathlib import Path
from huggingface_hub import hf_hub_download
from torchrl.render import load_checkpoint
from examples.microduck.train_skills import load_walker

path = hf_hub_download(
    "torchrl/microduck-skills",
    "priors/nine-skills-20260913/walker.ckpt",
    revision="01ebcefca08850231edc0eb428a0151474559a85",
)
assert hashlib.sha256(Path(path).read_bytes()).hexdigest() == (
    "9fbf1e15b25dd1ce65fcceeb2d854240028be703b72f37b5af05b9b50f115876"
)
checkpoint = load_checkpoint(path, weights_only=True)
walker, skill_tasks = load_walker(checkpoint)  # nine tasks, in saved order

Deploy with MicroDuckController(walker, skill_tasks, group_key=None, reset_key=None, control_period_s=0.02) and ClosedLoopMultiAction.from_env(task_env, controller, steps=5). The task env must use the saved MicroDuck observation layout and action scale 1.0. reset_key=None assumes falls end the episode; tasks that respawn individual agents need their per-instance reset signal.

To evaluate the downloaded actor with the example pipeline:

python -m examples.microduck.train_skills --walker-checkpoint /path/to/walker.ckpt \
    --low-level-only --evaluation-episodes 32 --seed 20260914 \
    --output-dir /tmp/microduck-evaluation

The example evaluates with seed + 1. New training with --init-from expands the original six-skill actor using the mapping above and starts a fresh optimizer. Resuming optimization requires the separately retained full trainer state, not this actor-only file.

Compatibility and preserved releases

High-level selectors are paired with their exact walker, skill order, action scale and control period. Do not attach the existing six-way navigation or seven-way football selector to the nine-skill prior. Train a new selector to use all nine skills. The football configuration remains pinned to its previous seven-skill checkpoint. A fresh nine-way football selector has been evaluated against its initial policy; see the separate experiment below.

Release	Pinned revision	Pairing
Original six skills, including forward hopping at index 5	4191d7d25c4fd58a5c6e6395fcf8217459fdd073	Walker for the original navigation selector
Navigation selector	6330ae68b8cf00cda62feb661e50df327f12ebcd	Original six-skill walker, five physical steps per decision
Original evaluation and navigation video	5824a856131009e1ecbc95713677a8363e5f169b	26/32 arrivals at the fixed (0.5, 0.3) m waypoint
Five-skill football walker	43ffe3b725b6a6853ba8e4a54deb06ae17606e5e	No hopping or turning rows
Seven-skill football walker	8c31e2696520c402980a723b37d025e594197d9d	Turns at indices 5/6; no hopping; preserved root artifacts

The root walker.ckpt, skills.json, training.json and videos/skills.mp4 still describe the seven-skill football variant. Root navigation files still belong to the original six-skill pair. The new files live only under the versioned directory above. Previous model cards and artifacts remain available at their immutable revisions. Policies were evaluated in simulation; hardware deployment was not tested.

Nine-skill football experiment: September 13, 2026

A fresh nine-way selector completed 3,004,800 skill decisions of native MuJoCo 5-vs-5 self-play on a MacBook with 16 workers. The published nine-skill walker stayed frozen. Artifact revision: 0bb7f4bd4c2660bb71a24aa9b12ed9c3585ac48d.

Selector checkpoint, initial opponent, video, evaluation, training configuration, and hashes.

Held-out play	Wins	Losses	Draws
Deterministic	11	1	52
Sampled	10	1	53

Each mode uses 32 matches per team side, seed 20260924, against the initial zero-update selector with the same prior. The final/latest checkpoint was selected by the training budget. Results support improvement against that baseline for this training seed; stronger opponents and independent training seeds remain untested. Most matches are draws. Crowding and falls persist; match-wide falls per duck average 4.49 deterministic and 5.52 sampled. The video uses independent seed 20260925 and the exact final checkpoint.

This is an experimental football selector, separate from the validated locomotion prior. It requires that prior at revision 01ebcefca08850231edc0eb428a0151474559a85, all nine skill indices, action scale 1.0 and five physical steps per skill decision. The default seven-skill football configuration and older selectors retain their original pairing. See portable loading instructions and TorchRL PR #4362.