jrubiosainz/microduck-game-sprinter

MICRODUCK GAME sprinter

HannesVonEssen's microduck-running policy (Apache-2.0) fine-tuned on a CPU inside the Arena's own simulator with PPO and evolution strategies, then wrapped in a step-counter schedule plus a free step-by-step offset on its joint targets and internal forward speed; the last ES link, the schedule and the offset were scored on the Arena's official seeds: 0.960 s on those seeds; 156/200 finishes and 42 falls on seeds 1001-1200, which it never saw. Ignores the command; simulation only.

An entry for the 2 m Sprint of the Microduck Arena. Simulation only — see below.

What it is

A fine-tune of HannesVonEssen/microduck-running (revision 9f45af1f, policy.onnx sha256 007707dd7779…), HannesVonEssen's forward-running policy for the stock MicroDuck on flat ground (iteration 12,195, Apache-2.0), trained with Vottivott/microduck-playground@828d950 on pollen-robotics/microduck_rl@d424a0c. The network has the base's own shape (61 → 512 → 256 → 128 → 14, ELU) and keeps its input normalizer frozen; its weights were trained further with PPO and then evolution strategies (ES) on a CPU, then wrapped in a step-counter schedule (below), inside the Arena's own runner and physics (MuJoCo 3.10.0, runner version 4), on this event's course:

• chain: HannesVonEssen's microduck-running policy (iteration 12,195) -> 1,850 PPO iterations on sprint-2m (run hs1, whose first 25 iterations trained only a fresh critic) -> 95 generations of evolution strategies (ES) on the race time (run es1) -> 30 more PPO iterations with four times larger batches (run hsb2, whose first 20 iterations trained only a fresh critic) -> 454 generations of ES on the race time over the Arena's official seeds 1–5 (run so1_sprint), all with an internal forward command of 1.0 m/s -> step-counter schedule tables for the internal command and the joint offsets, searched on the same seeds with that network frozen (run clock_sprint_j3; its iteration-38 tables) -> CMA-ES on a free per-step offset added to those tables' joint offsets and forward speed from counter step 26 on, with the network frozen (run dive_free_sprint_full, generation 1047)
• selection: elite_r0_g1047_13, candidate 13 of generation 1047 of run dive_free_sprint_full (CMA-ES), saved because its time on the official seeds, 0.960 s, beat the best one kept before it or matched it with a lower objective; then run on the seeds of the table below;
• PPO links (the earlier ones): reward = forward progress while the race clock runs, small penalties for sideways speed, yaw rate, drifting toward the walls and falling; in runs hs1 and hsb2 an episode also ends when the duck crosses the line, as the Arena's run does, with a one-off bonus equal to the reward of running at 1.5 m/s for every step left before the 2.5 s horizon; 96 parallel copies of the course in run hs1 and 384 in run hsb2;
• ES link (run es1, between the PPO runs): the objective is the race time itself — the finish time interpolated between control steps; a run still going after 1.5 s costs 1.5 s plus the metres left, plus 1 if it fell. Each generation scores the current weights and 64 antithetic Gaussian perturbations of every layer (σ = 0.02 × the layer's RMS) on the same 4 fresh seeds of the Arena's own start, ranks them and takes one Adam step (lr 0.004 in the same units) along the rank-weighted perturbations. A perturbation's rank utility also loses 1 × the share of its seeds on which it fell, so falls weigh more than a slow finish;
• ES link (run so1_sprint): the objective is the race time itself — the finish time interpolated between control steps; a run still going after 1.5 s costs 1.5 s plus the metres left, plus 1 if it fell. Each generation scores the current weights and 96 antithetic Gaussian perturbations of every layer (σ = 0.01 × the layer's RMS) on the Arena's official seeds 1–5 themselves, the board's own seeds, ranks them and takes one Adam step (lr 0.003 in the same units) along the rank-weighted perturbations. Any perturbation whose time on those seeds beat every set of weights scored before it was saved and checked through the Arena's own policy loader;
• step-counter link (run clock_sprint_j3): with the network above frozen, a search over the step counter's two tables (the internal command and the joint offsets; the final tables, after the next link, are described below), both piecewise-linear in k between knots at k = 0, 4, …, 32, then every 8 steps up to 72; past k = 72 the command holds its last value and the offsets return to zero by k = 80. That is 378 numbers, started at the network's own values (its 1.0 m/s command, zero offsets) and moved in units of 0.03 m/s for the forward speed, 0.1 × the input normalizer's standard deviation for the other 12 command slots and 0.03 rad for the joint offsets. Each iteration scored, on the official seeds 1–5, 48 antithetic Gaussian perturbations of the incumbent tables (σ adapted between 0.02 and 1 by the share of perturbations that beat the incumbent) and three steps along a smoothed rank-weighted direction; the incumbent moved only to a candidate with a lower board time that passed the checks below or, on a tie, to one with a lower sum over the seeds of the square root of the share of a 0.02 s step each still had to gain to cross one step earlier. A candidate with a lower board time was re-run through the Arena's own policy loader and had to finish all five seeds in each of 2 re-runs with Gaussian noise (σ = 0.0001) on every observation but the step counter's, with a mean at most 0.012 s slower. Its iteration-38 tables (0.980 s on the official seeds, an in-sample number) were this repo's sixth revision;
• free gait link (run dive_free_sprint_full): CMA-ES on top of those tables, with the network frozen: a free offset added from counter step 26 on (race step 1) to eight joint-target channels (hip_pitch, knee and ankle of both legs alike and in opposition, neck_pitch, head_pitch) and to the internal forward speed, piecewise linear between knots at steps 26, 30, …, 78 and held after the last: 126 numbers, each within ±1.5 rad (±0.5 m/s for the speed), starting from zero (σ = 0.08, 24 candidates per generation). Every candidate was scored on the official seeds 1–5 by its board time plus 0.1 × (the margin by which the seed nearest its next 0.02 s step still sat above it
  • 0.2 × the mean of those margins); to make that cheap, each seed's race up to step 26 (where every candidate equals the tables above) was run once and its full simulator state kept, and each candidate replayed only the rest, checked bit for bit against the Arena's own run. A candidate whose board time beat 0.980 s (the board of this repo's sixth revision) and beat, or matched with a lower objective, every candidate kept before it was exported, re-scored through the Arena's own policy loader and had to finish all five seeds in each of 2 re-runs with Gaussian noise (σ = 0.0001) on every observation but the step counter's, each with a mean at most 0.012 s slower. Generation 716's candidate 14 (0.964 s on those seeds) was this repo's seventh revision; this entry is generation 1047's candidate 13: 0.960 s on those seeds, an in-sample number. Before publishing it was also re-run 6 times at each of σ = 1e-7, 1e-6 and 1e-5 with the same noise: 0.960 s in all 6 re-runs at σ = 1e-7, 0.960–0.964 s at σ = 1e-6 and 0.964–0.968 s at σ = 1e-5.
• command and step counter: the graph rounds its left_knee target to 1/256 rad and adds a control-step counter k in units of 1/65536 rad, moving that target by at most 0.004 rad; the Arena feeds the previous action back bit for bit, so each step reads k back (k = 0 on the first of the 25 settle steps the runner takes before the race clock starts; it saturates at 127). k selects a row of two tables: the 13 command slots written into the observation (forward speed 0.998–1.001 m/s up to k = 25, then 0.805–1.487 m/s, every other slot within ±0.001) and an offset added to the 14 joint targets: at most 0.0047 rad on every joint from k = 0 to 25 (at most 0.0047 rad during the settle); from k = 26, 1 step (0.02 s) after the race clock starts, up to 2.13 rad on the left_hip_pitch, left_knee, left_ankle, neck_pitch, head_pitch, right_hip_pitch, right_knee and right_ankle targets, changing from step to step up to k = 80 and held after it: a reshaped gait over the whole race. The commanded twist is ignored.

No GPU, no new data, no paid compute.

Results

local run, Arena runner	score (non-finish = 50 s)	mean of finishers	finished
seeds 1–5 (the official set)	0.960 s	0.960 s (worst 0.96)	5/5
seeds 1001-1200	11.811 s	1.040 s (p90 1.12, worst 1.22)	156/200, 42 falls
seeds 1–5 with observation noise σ = 1e-04, 4 runs per seed	0.972–10.780 s (range)	—	18/20
seeds 1–5 with observation noise σ = 1e-03, 4 runs per seed	0.984–10.792 s (range)	—	18/20

The noise rows add Gaussian noise to every observation the policy sees: a stand-in for the float-rounding differences of another machine, which a chaotic gait amplifies.

arena score --event sprint-2m on this file (every speed of its sweep gives the same time, since the command is ignored; the sweep then reports the lowest):

sweep 21 commanded speeds 0.20–1.20 m/s
   0.20 m/s  0.96s
   0.25 m/s  0.96s
   0.30 m/s  0.96s
   0.35 m/s  0.96s
   0.40 m/s  0.96s
   0.45 m/s  0.96s
   0.50 m/s  0.96s
   0.55 m/s  0.96s
   0.60 m/s  0.96s
   0.65 m/s  0.96s
   0.70 m/s  0.96s
   0.75 m/s  0.96s
   0.80 m/s  0.96s
   0.85 m/s  0.96s
   0.90 m/s  0.96s
   0.95 m/s  0.96s
   1.00 m/s  0.96s
   1.05 m/s  0.96s
   1.10 m/s  0.96s
   1.15 m/s  0.96s
   1.20 m/s  0.96s

scored at 0.20 m/s
seed 1: finish   0.96s
seed 2: finish   0.96s
seed 3: finish   0.96s
seed 4: finish   0.96s
seed 5: finish   0.96s

score 0.960s · 5/5 finished · replay from seed 3

Local times are indicative: the Arena's machine can differ by a few hundredths of a second.

History

This repo's first revision (8f3bfe59) was Pollen's sprint-2m reference run unchanged inside a 37-parameter wrapper tuned with CMA-ES: 3.156 s on the Arena's board. Its second revision (f6241b55) was a PPO fine-tune of Pollen's sprint-2m reference (run s4's iteration-2075 checkpoint, after a chain of sprint and hill-climb runs): 1.084 s on the Arena's board. Its third revision (2ea9d739) was run hsb2's iteration-30 checkpoint, the starting point of run so1_sprint's ES link above: 1.040 s on the Arena's board. Its fourth revision (a1d32005) was run so1_sprint's generation-34 elite, an earlier point of the same ES link: 1.008 s on the Arena's board. Its fifth revision (db593306) was run so1_sprint's generation-454 elite, the network inside this revision's step-counter tables: 0.984 s on the Arena's board. Its sixth revision (332261bd) was that network inside run clock_sprint_j3's iteration-38 tables, the starting point of the free gait link above: 0.980 s on the Arena's board. Its seventh revision (0a4bc289) was generation 716's candidate 14 of that link, an earlier point of the same search: 0.964 s on the Arena's board.

Not for a real robot

This policy ignores the commanded twist: it always runs straight ahead at its own pace and never stops by itself. Its joint targets also carry step-by-step offsets of up to 2.13 rad, searched only for race time in simulation. This fine-tune has only ever run in simulation and was trained without any sim-to-real randomisation; its base was trained with velocity pushes, centre-of-mass variation and initial tilt but, by its own card, is not validated on hardware either. Do not load it on a physical Microduck.

Files

• policy.onnx — one input obs [1, 61], one output actions [1, 14], 50 Hz, raw observations (the normalizer is baked in). Its metadata_props are the base's (without run_path) plus microduck_game_* provenance keys.
• manifest.json — schema 2 of the microduck policy manifest, built with Pollen's own microduck_rl publish code.
• LICENSE — the Apache License 2.0, under which the base is published and this fine-tune is distributed.
• NOTICE — where the weights come from and what this repo changed.

Credits

Base policy: HannesVonEssen/microduck-running by HannesVonEssen, under the Apache License 2.0, linked as base_model. Robot, simulator, Arena and publishing tools: Pollen Robotics. This fine-tune is a modified version of that policy, distributed under the same licence: see LICENSE and NOTICE. PPO + ES fine-tune with a step-counter schedule: the MICRODUCK GAME project (jrubiosainz), built with GitHub Copilot.

Resumen (español)

Participante del Sprint de 2 m de la Microduck Arena, solo para simulación. Es la política de carrera de HannesVonEssen (Apache-2.0) reentrenada con PPO y después con estrategias evolutivas (ES) en CPU dentro del simulador del propio Arena. Su última etapa se ajustó directamente sobre las cinco semillas oficiales del tablero, así que su tiempo en él no es una medida fuera de muestra. Un contador de pasos dentro del propio grafo elige, paso a paso, la orden interna y una corrección de las articulaciones, que remodela su zancada durante toda la carrera. Ignora la velocidad que se le ordena: no debe cargarse en un robot real.