dummy033485/multi-agent-rl-navigation

Multi-agent RL navigation — trustworthy autonomous systems (simulation only)

Repo: dummy033485/multi-agent-rl-navigation Status: model repo live. No Hugging Face Space has been created.

A shared decentralized PPO policy for 2D multi-agent navigation, compared honestly against a textbook classical controller on the same seeded layouts. Built as a portfolio project for an application to Prof. Jiachen Li's Trustworthy Autonomous Systems Lab (Georgia Tech): robotics, trustworthy AI/ML, reinforcement learning, control, optimization and multi-agent systems.

What this is — and is not

• Is: a self-contained 2D physics simulator (written for this project, no external environment, no hardware, no proprietary data) + a learned shared policy + a classical baseline + a fixed, seeded evaluation.
• Is not: a robot result. No hardware was used. The RL policy has no formal safety guarantee — every safety number below is empirical, measured on the simulator's fixed test layouts. The wall-trap layout is deliberately constructed to defeat a pure potential-field controller.

Simulator (sim.py)

• Arena 12x12 with walls. Agents: double-integrator kinematics, dt=0.1, max speed 1.2, max acceleration 3.0, agent radius 0.25, episode cap 200 steps.
• Each layout: 3 circle obstacles (radius 0.5–0.8) and 1 wall/barrier segment. Two layout families, generated with fixed seeds:
  • general: a short (3 m) barrier segment at a random mid-arena angle.
  • trap: a vertical segment at x=6, y in [2,10], with gaps only at the ends; starts on the left, goals near-directly opposite on the right, so a direct potential field is attracted right and repelled left and stalls in the middle.
• Decentralized 20-dim observation per agent: own velocity, relative goal vector + distance, nearest circle obstacle (direction, surface distance, radius), nearest wall segment (direction, distance, both endpoint vectors), nearest other agent (direction, distance), nearest arena-wall distance. (Endpoint vectors were added after a failed training run: with only the nearest segment point, the policy could not observe where the wall ends — see Limitations and BUILD_REPORT.md.)
• Collision = distance to an obstacle/wall segment/arena wall below the agent radius, or agent–agent distance below 0.5. A colliding arrival is never counted as a success. Safety margin for violation counting: 0.15.

Policies

• Classical (policies.py, exact gains documented in code): PD controller toward the goal (KD=2.0) + radial repulsive potential field (ETA=6.0, influence distance D0=2.0) from the nearest obstacle, segment and neighbour. No learning, no tangential escape trick — the honest textbook baseline. It stalls (timeout, 0 collisions) on every trap layout.
• RL: shared decentralized PPO, MLP 20→128→128 (the observation grew from 15 to 20 dims when segment-endpoint features were added), Gaussian actions, tanh-bounded to the acceleration limit. One set of weights drives every agent from its own observation. Training recipe (all in train.py, PPO from scratch with torch only): the policy was warm-started by behaviour cloning the classical controller on general layouts only (400 episodes, 4 agents, final action MSE 0.079), then PPO fine-tuned in four phases — 1-agent mixed (trap ramp), 1-agent trap-only with an exploration reset, 1-agent mixed consolidation, and 4-agent mixed — 300 iterations × 128-step rollouts × 16 envs per phase (614,400 environment steps per phase; 2,457,600 in the kept chain plus the BC dataset). Earlier from-scratch PPO runs failed outright (success 5–15%) and are documented, with their fixes, in BUILD_REPORT.md — they are not hidden. The final phase's curve is training_curve.csv (earlier phases have their own training_curve_*.csv files).

Evaluation protocol (exact)

Fixed layouts, identical for every policy and condition:

• Single-agent: seeds 1000–1019 (10 general + 10 trap, alternating).
• 4-agent: seeds 2000–2011 (6 general + 6 trap, alternating).
• RL evaluated with the deterministic (mean) action; classical is deterministic. Noise conditions reuse the same noise RNG seed per layout/condition for both policies.
• Conditions: clean; observation noise σ=0.05; observation noise σ=0.10; dynamics disturbance (added acceleration noise σ=1.0); obstacle shift +0.5 (circles and segments shifted +0.5 in x).
• Metrics are per agent-episode (20 single-agent and 48 4-agent episodes per policy per condition; 728 rows in eval.csv). Steps-to-goal and clearance are reported on successes only; safety-violation rate is the fraction of active steps with clearance < 0.15.

Results (copied from results.json — no other numbers are claimed)

Clean

Policy	Agents	Success	Collision	Obstacle coll.	Agent–agent coll.	Timeout	Steps (succ.)	Path len.	Min clearance (succ.)	Safety-viol. rate	General	Trap
Classical	1	0.450	0.000	0.000	0.000	0.550	73.22	5.35	0.588	0.000	0.90	0.00
RL	1	0.700	0.200	0.200	0.000	0.100	86.00	9.59	0.419	0.033	0.60	0.80
Classical	4	0.396	0.000	0.000	0.000	0.604	160.42	5.87	0.690	0.000	0.79	0.00
RL	4	0.792	0.146	0.146	0.000	0.062	141.18	10.63	0.505	0.019	0.75	0.83

Reading, honestly: the RL policy wins on success everywhere, and is the only policy that solves trap layouts or coordinates 4 agents (zero agent–agent collisions clean). It loses on safety in the single-agent case: 20% collisions and lower clearance than the classical controller's 0% collisions. The classical controller never collides clean — it fails by timing out at the wall instead. RL paths are longer (detours).

Robustness (success / collision)

Condition	RL 1-agent	Classical 1-agent	RL 4-agent	Classical 4-agent
Clean	0.700 / 0.200	0.450 / 0.000	0.792 / 0.146	0.396 / 0.000
Obs. noise σ=0.05	0.750 / 0.150	0.450 / 0.100	0.646 / 0.292	0.333 / 0.125
Obs. noise σ=0.10	0.600 / 0.300	0.200 / 0.250	0.542 / 0.396	0.104 / 0.396
Dynamics disturbance σ=1.0	0.650 / 0.200	0.350 / 0.000	0.771 / 0.167	0.375 / 0.000
Obstacle shift +0.5	0.550 / 0.300	0.350 / 0.000	0.708 / 0.250	0.375 / 0.000

The RL policy degrades gracefully on success under every perturbation and stays well above classical, but its collision rate rises under noise (up to 0.396 at σ=0.10, 4-agent) — the trustworthy claim is measured robustness of success, not a safety proof.

Ablation — neighbour observation, 4-agent clean

Policy	Success	Collision	Agent–agent collision
RL with neighbour observation	0.792	0.146	0.000
RL with neighbour observation zeroed	0.146	0.708	0.333

Removing the neighbour observation collapses coordination: success falls from 0.792 to 0.146 and agent–agent collisions jump from 0 to 0.333. The shared policy genuinely uses the neighbour channel; it is not four independent single-agent policies.

Files in this repo

• ppo_policy.pt — final shared PPO weights (ActorCritic, obs dim 20).
• sim.py, policies.py, train.py, bc.py, evaluate.py, plots.py
• results.json, eval.csv (728 rows), training_curve.csv (final 4-agent phase) + training_curve_single.csv, training_curve_single2.csv, training_curve_ft1.csv, training_curve_ft2.csv, training_curve_ft3.csv (earlier phases)
• plots/trap_trajectories.png, plots/four_agent.png, plots/clean_comparison.png, plots/training_curve.png
• space/ — self-contained Gradio demo (copies of sim.py, policies.py, ppo_policy.pt + app.py, requirements.txt). See SMOKE_TEST.md.

Limitations

1. 2D simulation only. No hardware, no real robot, no vision (the observation is ground-truth state features + optional noise, not pixels).
2. The wall trap is constructed. It demonstrates a known potential-field failure mode; it does not prove the policy generalises to arbitrary unseen maps (the +0.5 obstacle-shift condition is the only unseen-layout test, and success there is 0.55–0.71, not 1.0).
3. No formal safety guarantee. Collisions still occur (5–40% depending on condition). A safety shield / control-barrier filter would be the natural next step and is explicitly not claimed here.
4. Small evaluation: 20 single-agent and 12 four-agent layouts. Differences of a few percent are within layout noise; the large effects (trap 0.00 → 0.80–0.83, 4-agent 0.396 → 0.792, ablation 0.792 → 0.146) are the ones to rely on.
5. Training required a behaviour-cloning warm-start and a phased curriculum (including a trap-only phase that temporarily caused catastrophic forgetting of general layouts, fixed by a mixed consolidation phase). From-scratch PPO did not learn this task in ~1.9M environment steps of attempts. See BUILD_REPORT.md.

Reproduce

# shared venv (torch, numpy, matplotlib, gradio); proxy vars must be unset for gradio imports
export VENV=~/workspace/rul-project/venv/bin/python
env -u http_proxy -u https_proxy -u HTTP_PROXY -u HTTPS_PROXY -u all_proxy -u ALL_PROXY -u no_proxy -u NO_PROXY $VENV bc.py --episodes 400 --n-agents 4 --epochs 30 --out bc_policy.pt
env -u ... $VENV train.py --iterations 300 --rollout 128 --n-agents 1 --init bc_policy.pt --out ppo_ft1.pt --curve training_curve_ft1.csv
# (trap-only exploration-reset weights were made from ppo_ft1.pt by setting log_std to -0.5; see BUILD_REPORT.md)
env -u ... $VENV train.py --iterations 300 --rollout 128 --n-agents 1 --init ppo_ft1_explore.pt --trap-prob 1.0 --trap-ramp-iters 1 --ent-coef 0.02 --out ppo_ft2.pt --curve training_curve_ft2.csv
env -u ... $VENV train.py --iterations 300 --rollout 128 --n-agents 1 --init ppo_ft2.pt --trap-prob 0.5 --trap-ramp-iters 1 --out ppo_ft3.pt --curve training_curve_ft3.csv
env -u ... $VENV train.py --iterations 300 --rollout 128 --n-agents 4 --init ppo_ft3.pt --trap-prob 0.5 --trap-ramp-iters 1 --out ppo_policy.pt --curve training_curve.csv
env -u ... $VENV evaluate.py --weights ppo_policy.pt --out results.json --csv eval.csv
env -u ... $VENV plots.py --weights ppo_policy.pt --results results.json --curve training_curve.csv --outdir plots

All commands are run from ~/workspace/li-project/. Fixed seeds throughout (training seed 0, BC seed 123, evaluation layout seeds 1000–1019 / 2000–2011, noise seeds 90000+layout index).