orcnd/field-orders-40v40-ppo-v2

Field Orders: soldier PPO v2 and commander v1

Reusable SDK and v3 experiments

Play the browser demo · Full source ZIP · Python wheel · Development and evaluation report

Version 0.3.0 adds a PettingZoo Parallel environment, a NumPy inference SDK, reproducible commander PPO, observed-history soldier PPO and historical self-play snapshots. Requires Python 3.10+ and Node.js 22+. Experimental checkpoints are in experiments/v3/; the active configuration remains active-model.json. Read the report for promotion status and measured limits. These policies have not demonstrated human-level strategy or cross-game generalization.

Policy replay

Prompt

Actual policy replay: behavior-cloning baseline vs PPO v2 (selected paired episodes, 1.5x playback)

Open or download the replay video · Replay provenance

This 68-second, 1440×810 H.264 replay visualizes actual decisions from the uploaded weights in the exact shared game simulation. It is not a screen recording. Left: behavior-cloning baseline; right: PPO v2 controlling the same army against the same frozen baseline opponent. Initial seeds, maps and orders match. Playback is 1.5×.

The three selected illustrations show a standard 40v40 battle, a 12v12 point-order task, and a carrier-interception limitation. They are not a representative benchmark sample. Full 48-case evaluation remains below; carrier stops did not improve in that evaluation. The commander remains frozen.

Compact neural controllers for the Viking vs Mongol Capture-the-Flag game. Every soldier receives its own partial observation and returns human-like movement, aim and fire inputs. Soldiers share weights while maintaining separate observations. The commander chooses command, formation and strategy. These are dense game policies in a custom JSON weight format, not language models or Transformers checkpoints.

Included models

File	Architecture	Training
soldier-ppo-v2.json	126 -> 128 -> 96 -> 5	PPO + GAE after behavior cloning; active soldier
commander-v1.json	58 -> 64 -> 48 -> 22	Behavior cloning; frozen throughout PPO
soldier-v1.json	126 -> 128 -> 96 -> 5	Archived behavior-cloning baseline

The active configuration is active-model.json. The package also includes the JavaScript inference runtime, Worker, observation schema, example input physics, exact model checksums and training/evaluation reports. No rule-based movement or targeting fallback is used.

Training

The soldier was initialized from supervised imitation of the project's prior policy and fine-tuned locally with clipped PPO and generalized advantage estimation (GAE). Four parallel environments used the same movement, collision, projectile, health and flag rules as the live game. Training rewards covered terrain-aware objective progress, captures, useful damage, recovery, command following, wasted shots, stalled movement and crowding. Privileged objective/geometry information was used for rewards only, not supplied as additional policy observations.

The selected checkpoint includes 497,837 active-agent transitions across 64 PPO updates in two stages (40 + 24). The commander was not reinforcement-trained. The opponent was a frozen learned baseline; an opponent league has not been trained. Policies do not update their weights during live play.

Evaluation

48 paired seeded cases, separate from training/model-selection seeds: three maps, two team sides, seven 30-second 12v12 task cases per map/side (42 cases), plus six standard 60-second 40v40 matches. Each comparison used identical starting conditions and a frozen baseline opponent.

Metric	Behavior-cloning baseline	Soldier PPO v2
Flag deliveries across 48 cases	1	4
Point-order reach rate	30.6%	65.3%
Mean stalled-movement rate	2.73%	2.06%
Carrier stops in six interception cases	2/6	2/6
Standard match wins	3/6	4/6

These small deterministic tests do not establish a general win rate, optimal strategy, balanced factions, human-level performance, or long-match reliability. Stalled movement means less than 0.8 units of displacement over 0.1 seconds despite movement input and a distant objective. A point order succeeds when a soldier approaches within 35 units. Full per-case results are in rl-evaluation-report.json.

Both-PPO live play also showed faction imbalance and a commander retaining a carrier-stop order after the flag returned. Commander task-transition training, better carrier velocity/report-age observations, recurrent memory and an opponent pool remain priorities.

Use with JavaScript

Download this repository, serve the files through HTTP, and import the included runtime:

import {runNetwork, decodeControl, validateNetwork} from './runtime.js';
const response = await fetch('./soldier-ppo-v2.json');
const network = validateNetwork(await response.json());
// observation must follow the exact 126-feature schema and normalization.
const raw = runNetwork(network, [observation])[0];
const input = decodeControl(raw);
// input: {moveX, moveY, aimX, aimY, fire}

moveX/moveY form a normalized analog movement vector. aimX/aimY specify aim direction. fire is a boolean button. The environment applies legal speed, collision, projectiles, damage, flags and healing. The Worker entry is worker.js; the portable npm archive is field-orders-neural-model-1.1.0.tgz.

See RUNTIME.md, manifest.json, and sensors.js for the exact feature order, scales, commander output heads, required sensor adapter and integration details. Identical feature vectors produce identical deterministic actions. Equivalent gameplay additionally requires equivalent observations, timing and physics; transferring weights alone to an unrelated game does not preserve behavior.

Limitations and licensing

Code, model weights and included assets are licensed under Apache-2.0; see LICENSE and NOTICE. The model requires a compatible environment adapter. It is not directly compatible with Hugging Face Transformers, ONNX Runtime or a hosted text-generation API.

References

• PPO
• GAE