jrubiosainz/microduck-game-roller

MICRODUCK GAME roller

HannesVonEssen's microduck-running policy (Apache-2.0) fine-tuned on a CPU inside the Arena's own simulator, on roller skates, with PPO, then with evolution strategies and with step-by-step offsets on its joint targets and its internal forward command, driven by a step counter, all scored on the Arena's official seeds: 1.048 s on those seeds; 28/200 finishes and 172 falls on seeds 1001-1200, which it never saw. Ignores the command; simulation only.

An entry for the 2 m Roller Sprint of the Microduck Arena. Simulation only — see below.

What it is

A fine-tune of HannesVonEssen/microduck-running (revision 9f45af1f, policy.onnx sha256 007707dd7779…), HannesVonEssen's forward-running policy for the stock MicroDuck on flat ground (iteration 12,195, Apache-2.0), trained with Vottivott/microduck-playground@828d950 on pollen-robotics/microduck_rl@d424a0c. The network has the base's own shape (61 → 512 → 256 → 128 → 14, ELU) and keeps its input normalizer frozen; its weights were trained further with PPO and then evolution strategies (ES) on a CPU, then wrapped in a step-counter schedule (below), inside the Arena's own runner and physics (MuJoCo 3.10.0, runner version 4), on this event's course:

• chain: HannesVonEssen's microduck-running policy (iteration 12,195) -> 2,075 PPO iterations on roller-sprint-2m (run hr1, whose first 25 iterations trained only a fresh critic) -> 70 more PPO iterations with four times larger batches (run hrb1, whose first 20 iterations trained only a fresh critic, with a fall penalty of 50 instead of 3) -> 2028 generations of evolution strategies (ES) on the race time over the Arena's official seeds 1–5 (run so1_roller) -> a step counter wrapped around that network, then CMA-ES on a free per-step offset added to its joint targets from counter step 26 on, with the network frozen (run dive_free_roller_full, generation 470) -> CMA-ES on a second free per-step offset added on top of the first, and on the internal forward speed, from counter step 26 on, with the network frozen (run dive_free_roller_r2, generation 1188); internal forward command 1.4 m/s up to counter step 26, then between 0.91 and 1.67 m/s
• selection: elite_r0_g1188_06, candidate 06 of generation 1188 of run dive_free_roller_r2 (CMA-ES), saved because its time on the official seeds, 1.048 s, beat the best one kept before it or matched it with a lower objective; then run on the seeds of the table below;
• PPO links (the earlier ones): reward = forward progress while the race clock runs, small penalties for sideways speed, yaw rate, drifting toward the walls and falling; in run hrb1 the fall penalty was raised from 3 to 50; in runs hr1 and hrb1 an episode also ends when the duck crosses the line, as the Arena's run does, with a one-off bonus equal to the reward of running at 1.5 m/s for every step left before the 3.7 s horizon; 96 parallel copies of the course in run hr1 and 384 in run hrb1;
• ES link (run so1_roller): the objective is the race time itself — the finish time interpolated between control steps; a run still going after 1.7 s costs 1.7 s plus the metres left, plus 1 if it fell. Each generation scores the current weights and 96 antithetic Gaussian perturbations of every layer (σ = 0.01 × the layer's RMS) on the Arena's official seeds 1–5 themselves, the board's own seeds, ranks them and takes one Adam step (lr 0.003 in the same units) along the rank-weighted perturbations. Any perturbation whose time on those seeds beat every set of weights scored before it was saved and checked through the Arena's own policy loader;
• step-counter wrapper (tools/plain_elite.py): run so1_roller's generation-2028 weights, this repo's seventh revision, wrapped unchanged in the step counter described below, with the network's own baked command in every row of the command table and a zero offset table. Rounding the head_roll target to 1/256 rad alone moved one official seed's crossing across a 0.02 s step: 1.096 s on the official seeds 1–5 instead of 1.092 s;
• free gait link (run dive_free_roller_full): CMA-ES on that wrapper's offset table, with the network frozen: a free offset added to the joint targets from counter step 26 on (race step 1), piecewise linear between knots at steps 26, 30, …, 82 and held after the last, on eight channels (hip_pitch, knee and ankle of both legs alike and in opposition, neck_pitch, head_pitch): 120 numbers, each within ±1.5 rad, starting from zero (σ = 0.08, 24 candidates per generation). Every candidate was scored on the official seeds 1–5 by its board time plus 0.1 × (the margin by which the seed nearest its next 0.02 s step still sat above it + 0.2 × the mean of those margins); to make that cheap, each seed's race up to step 26 (where every candidate equals the wrapper) was run once and its full simulator state kept, and each candidate replayed only the rest, checked bit for bit against the Arena's own run. A candidate whose board time beat 1.092 s (the board of this repo's seventh revision) and beat, or matched with a lower objective, every candidate kept before it was exported, re-scored through the Arena's own policy loader and had to finish all five seeds in each of 4 re-runs with Gaussian noise (σ = 0.0001) on every observation but the step counter's, each with a mean at most 0.008 s slower. By generation 453, 1,265 candidates had failed that last check (none had passed since generation 280); from then on it was relaxed to 2 re-runs at σ = 1e-6, each with a mean at most 0.012 s slower, and only candidates below 1.080 s (the board of this repo's ninth revision) were exported. Its generation 470's candidate 21 (1.064 s on those seeds, an in-sample number) was this repo's tenth revision (its eighth and ninth revisions were the same link's generation-35 candidate 21 and generation-280 candidate 9); before publishing, that candidate was also re-run 6 times at each of σ = 1e-7, 1e-6 and 1e-5 with the same noise: 1.064–1.068 s at σ = 1e-7, 1.064–1.076 s at σ = 1e-6 and 1.068–1.080 s at σ = 1e-5;
• second free gait link (run dive_free_roller_r2): the same search restarted on top of that candidate, with its offset table and the network frozen: a second free offset added to that table from counter step 26 on, piecewise linear between knots at steps 26, 29, …, 80 and held after the last, on the same eight channels plus the internal forward speed (1.4 m/s in the wrapper): 171 numbers, each within ±1.5 rad (±0.5 m/s for the speed), starting from zero (σ = 0.03, 24 candidates per generation), scored by the same objective with the same replay from step 26. A candidate whose board time beat 1.064 s (the board of this repo's tenth revision) and beat, or matched with a lower objective, every candidate kept before it was exported, re-scored through the Arena's own policy loader and had to finish all five seeds in each of 2 re-runs with Gaussian noise (σ = 1e-6) on every observation but the step counter's, each with a mean at most 0.012 s slower. Its generation 768's candidate 3 (1.052 s on those seeds, an in-sample number) was this repo's eleventh revision: 6 re-runs at σ = 1e-7 with the same noise had given 1.056–1.064 s, and the Arena's board timed it at 1.056 s, 0.004 s slower than this machine. This entry is generation 1188's candidate 6: 1.048 s on those seeds, an in-sample number; its internal forward speed stays between 0.91 and 1.67 m/s from counter step 26 on. Before publishing it was also re-run 6 times at each of σ = 1e-7, 1e-6 and 1e-5 with the same noise: 1.048–1.052 s at σ = 1e-7, 1.048–1.056 s at σ = 1e-6 and 1.064–1.076 s at σ = 1e-5, where 3 of the 6 re-runs left a seed unfinished; 8 more re-runs at σ = 1e-7 with other noise draws gave 1.048–1.052 s (mean 1.049 s).
• command and step counter: the graph rounds its head_roll target to 1/256 rad and adds a control-step counter k in units of 1/65536 rad, moving that target by at most 0.004 rad; the Arena feeds the previous action back bit for bit, so each step reads k back (k = 0 on the first of the 25 settle steps the runner takes before the race clock starts; it saturates at 127). k selects a row of two tables: the 13 command slots written into the observation (forward speed 1.40 m/s up to k = 25, then 0.908–1.665 m/s, every other slot zero) and an offset added to the 14 joint targets: zero up to k = 25; from k = 26, 1 step (0.02 s) after the race clock starts, up to 1.23 rad on the left_hip_pitch, left_knee, left_ankle, neck_pitch, head_pitch, right_hip_pitch, right_knee and right_ankle targets, changing from step to step up to k = 82 and held after it: a reshaped gait over the whole race. The commanded twist is ignored.

No GPU, no new data, no paid compute.

Results

local run, Arena runner	score (non-finish = 40 s)	mean of finishers	finished
seeds 1–5 (the official set)	1.048 s	1.048 s (worst 1.06)	5/5
seeds 1001-1200	34.570 s	1.216 s (p90 1.24, worst 2.52)	28/200, 172 falls
seeds 1–5 with observation noise σ = 1e-04, 4 runs per seed	1.076–16.652 s (range)	—	17/20
seeds 1–5 with observation noise σ = 1e-03, 4 runs per seed	8.904–32.216 s (range)	—	10/20

The noise rows add Gaussian noise to every observation the policy sees: a stand-in for the float-rounding differences of another machine, which a chaotic gait amplifies.

arena score --event roller-sprint-2m on this file (every speed of its sweep gives the same time, since the command is ignored; the sweep then reports the lowest):

sweep 18 commanded speeds 0.30–2.00 m/s
   0.30 m/s  1.06s
   0.40 m/s  1.06s
   0.50 m/s  1.06s
   0.60 m/s  1.06s
   0.70 m/s  1.06s
   0.80 m/s  1.06s
   0.90 m/s  1.06s
   1.00 m/s  1.06s
   1.10 m/s  1.06s
   1.20 m/s  1.06s
   1.30 m/s  1.06s
   1.40 m/s  1.06s
   1.50 m/s  1.06s
   1.60 m/s  1.06s
   1.70 m/s  1.06s
   1.80 m/s  1.06s
   1.90 m/s  1.06s
   2.00 m/s  1.06s

scored at 0.30 m/s
seed 1: finish   1.06s
seed 2: finish   1.06s
seed 3: finish   1.04s
seed 4: finish   1.04s
seed 5: finish   1.04s

score 1.048s · 5/5 finished · replay from seed 5

Local times are indicative: the Arena's machine can differ by a few hundredths of a second.

History

This repo's first revision (9b2d884f) was a PPO fine-tune of Pollen's roller-sprint-2m reference (run ro1's iteration-875 checkpoint): 2.152 s on the Arena's board. Its second revision (a6c08ced) was run hrb1's iteration-70 checkpoint, the end of the PPO links above and the starting point of the ES link: 1.256 s on the Arena's board. Its third revision (8415ffd8) was run so1_roller's generation-575 elite, an earlier point of the same ES link: 1.120 s on the Arena's board. Its fourth revision (84b60f69) was run so1_roller's generation-1252 elite, also an earlier point of the same ES link: 1.108 s on the Arena's board. Its fifth revision (aaf812ae) was run so1_roller's generation-1411 elite, also an earlier point of the same ES link: 1.100 s on the Arena's board. Its sixth revision (2b71c357) was run so1_roller's generation-1534 elite, also an earlier point of the same ES link: 1.096 s on the Arena's board. Its seventh revision (f7a956f0) was run so1_roller's generation-2028 elite, also an earlier point of the same ES link: 1.092 s on the Arena's board. Its eighth revision (319bc7fb) was generation 35's candidate 21 of the free gait link above: 1.084 s on the Arena's board; its README, which wrongly called that offset a finish-line dive, was corrected in 192a8713. Its ninth revision (bd154065) was generation 280's candidate 9 of the same link: 1.080 s on the Arena's board. Its tenth revision (f6581ad4) was generation 470's candidate 21 of the same link, the starting point of the second free gait link above: 1.064 s on the Arena's board. Its eleventh revision (d566fe29) was generation 768's candidate 3 of the second free gait link: 1.056 s on the Arena's board (1.052 s on this machine).

Not for a real robot

This policy ignores the commanded twist: it always runs straight ahead on its roller skates at its own pace and never stops by itself. Its joint targets also carry step-by-step offsets of up to 1.23 rad, searched only for race time in simulation. This fine-tune has only ever run in simulation and was trained without any sim-to-real randomisation; its base was trained with velocity pushes, centre-of-mass variation and initial tilt but, by its own card, is not validated on hardware either. Do not load it on a physical Microduck.

Files

• policy.onnx — one input obs [1, 61], one output actions [1, 14], 50 Hz, raw observations (the normalizer is baked in). Its metadata_props are the base's (without run_path) plus microduck_game_* provenance keys.
• manifest.json — schema 2 of the microduck policy manifest, built with Pollen's own microduck_rl publish code.
• LICENSE — the Apache License 2.0, under which the base is published and this fine-tune is distributed.
• NOTICE — where the weights come from and what this repo changed.

Credits

Base policy: HannesVonEssen/microduck-running by HannesVonEssen, under the Apache License 2.0, linked as base_model. Robot, simulator, Arena and publishing tools: Pollen Robotics. This fine-tune is a modified version of that policy, distributed under the same licence: see LICENSE and NOTICE. PPO + ES fine-tune with a step-counter schedule: the MICRODUCK GAME project (jrubiosainz), built with GitHub Copilot.

Resumen (español)

Participante del Sprint sobre patines de 2 m de la Microduck Arena, solo para simulación. Es la política de carrera de HannesVonEssen (Apache-2.0) reentrenada con PPO y después con estrategias evolutivas (ES) en CPU dentro del simulador del propio Arena. Su última etapa se ajustó directamente sobre las cinco semillas oficiales del tablero, así que su tiempo en él no es una medida fuera de muestra. Un contador de pasos dentro del propio grafo elige, paso a paso, la orden interna y una corrección de las articulaciones, que remodela su zancada durante toda la carrera. Ignora la velocidad que se le ordena: no debe cargarse en un robot real.