erickvs/tqc-Walker2d-v4-hand-edited

TQC walker, hand-edited: from 29% falls to 0.6%

An unofficial, hand-edited version of sb3/tqc-Walker2d-v3 from the RL Baselines3 Zoo. It wasn't retrained. AI agents changed some of its weights directly.

On current MuJoCo, the zoo's TQC walker falls in about 29% of episodes, usually late, after its knees slowly buckle. This version falls in 0.6% and scores 500 points more. Of the zoo walkers compared here (TD3, SAC and TQC), only TD3 scores higher. With heavy start noise, it beats TD3.

Made with WSP Lab, a tool that lets AI agents edit a model's weights directly, quickly and verifiably. This edit used its prototype. WSP Lab is coming soon: join the waitlist at wsp-lab.ai.

How long it took

Step	Wall-clock time	Done by
Setup: register the model, 15,000 baseline episodes and the automatic analysis	30 s	WSP Lab
Investigation: twelve agents at once, each from a different angle	1.5 min	Claude Haiku 5.5 × 12
Editing: three agents at once, working independently from the investigation	15 min	Claude Fable 5.1 × 3
Choosing between the three on 1,000 held-out seeds, and the final score on 2,000 unseen seeds	2 min	Claude Opus 5.5, as auditor
Robustness checks, then building, checking and uploading this release	4 min	Claude Opus 5.5

• Total: 21 minutes from registering the downloaded model (21:38:26 on 6 November 2026) to an audited final score (21:59:13), all on one laptop (Apple M5 Max) with no training. That includes about 3 minutes of writing the agents' briefs.
• How the edit was chosen: each of the three editors chose its best edit on the selection seeds. The one with the highest selection score was kept, a rule fixed before anything ran. On the final seeds, too, it was the best of the three, by 47 to 50 points.
• The release took 4 more minutes. A test download with no login scored 4,583 with 1.0% falls on 200 fresh seeds.
• Timing source: WSP Lab's event log.

Results

2,000 episodes of Gymnasium's Walker2d-v4, on seeds 2,000,000 to 2,001,999, with deterministic actions:

Model	Mean return	Falls	Forward speed
sb3/td3-Walker2d-v3, the best of the zoo walkers compared	4,714	0%	3.72 m/s
This model	4,591	0.6%	3.61 m/s
sb3/tqc-Walker2d-v3, unedited	4,092	29.0%	3.39 m/s
sb3/sac-Walker2d-v3	3,772	10.2%	2.87 m/s

Paired differences on the same seeds, with 95% bootstrap intervals:

• Against the unedited TQC: +499 (+469 to +528).
• Against SAC: +819 (+800 to +839).
• Against TD3: −124 (−135 to −114). TD3 is still ahead, because it walks faster.

Robustness, on 500 seeds that weren't used while editing:

Test	This model	TD3	Unedited TQC
10 times the default start noise	4,350, 5.4% falls	4,083, 13% falls	3,988, 30% falls
Episodes of 3,000 steps	14,761, 1.0% falls	15,027, no falls	8,168, 78% falls

The table was measured with WSP Lab's float64 evaluation. Loaded through sb3-contrib as shown below, it scored 4,598 with 0.4% falls on final seeds 2,000,000 to 2,000,499, against 4,594 in WSP Lab. The walker's gait is sensitive enough that the float32 arithmetic Stable-Baselines3 uses can change single episodes, but not the averages.

Videos

The first final seed on which the unedited walker falls, side by side. The unedited TQC (left) collapses at step 945, near the end of the 8-second episode. The edited one (right) finishes.

The next one, with TD3 for comparison. The unedited walker collapses at step 740. The edited walker and TD3 both finish.

If the players don't load, the files are videos/original_vs_edited_seed_2000011.mp4 and videos/original_edited_td3_seed_2000012.mp4.

What was wrong, and what was changed

The failure: the unedited walker doesn't trip. It slowly sinks. Its knees drift into a bend over a hundred or more steps, the torso leans forward, and finally the torso drops below the 0.8 m height limit. Falls start after step 300 and become more likely as the episode goes on: in 3,000-step episodes, 78% fall. With no start noise at all it never falls, so a small initial disturbance is enough to set off the slow collapse.

The edit, all exact operations on the network's weights:

1. Firmer feet and hips: both foot outputs ×1.5 and both thigh outputs ×1.7, before the tanh. Stronger foot and hip commands hold the stance up. The investigation found that weaker commands make the collapses much worse.
2. "You're sinking": perceived vertical speed shifted by −0.05 m/s, so the walker stands a little taller.
3. A pitch-coupled speed sense: perceived forward speed lowered by 0.5 × the torso's pitch, plus a constant −0.025 m/s, so the average shift is −0.125 m/s. The walker reads itself as slower when it leans forward and pushes a little harder. The edited walker moves at 3.61 m/s, against 3.44 for the unedited one in episodes where it stays up. The torso pitch value used for the constant (0.2 rad) was measured from the logged episodes and is declared in the edit program.

That changes 1,540 of the actor's numbers. The critics and the log-std head are the original's. The edit program is in edit_program.json.

Usage

import gymnasium as gym
from huggingface_hub import hf_hub_download
from sb3_contrib import TQC

path = hf_hub_download(repo_id="erickvs/tqc-Walker2d-v4-hand-edited", filename="tqc-Walker2d-v4-hand-edited.zip")
model = TQC.load(path)
env = gym.make("Walker2d-v4", render_mode="human")
obs, _ = env.reset(seed=0)
for _ in range(1000):
    action, _ = model.predict(obs, deterministic=True)
    obs, reward, terminated, truncated, _ = env.step(action)
    if terminated or truncated:
        break

example.py scores the model on a range of seeds. It was tested with stable-baselines3 2.9.2, sb3-contrib 2.9.2, gymnasium 1.4.0 and MuJoCo 4.0.0. actor.safetensors holds just the six actor tensors, for use without Stable-Baselines3:

action = tanh(mu(relu(latent_pi.2(relu(latent_pi.0(obs))))))

Limits

• Environment: tested on Walker2d-v4 only.
• Deterministic actions only: sampling from the stochastic policy hasn't been tested.
• Fine-tuning: further training hasn't been tested. The zip has fresh optimizer states and no replay buffer, and the critics were left as the original's.
• One base model: the edit was designed for this checkpoint, and the same values won't carry over to other agents.

How it was made

The model comes from an experiment testing a fast pipeline for AI agents editing a model, and every agent worked under the same fixed rules:

1. Twelve Claude Haiku 5.5 agents investigated the model from six angles, two agents each, and recorded testable claims.
2. Then three Claude Fable 5.1 editors worked on it at the same time. Each could read the investigators' claims, but none could see another editor's work.
3. The best editor's edit was kept, according to held-out selection seeds.

Editor	Final mean	Falls
This model	4,591	0.6%
A second editor: right foot ×2.0, left foot ×0.7, right thigh ×1.3	4,544	0.5%
A third editor: feet ×2.0, thighs ×1.5, a hip bias and a small speed shift	4,541	0.8%

All three found the same thing, that firmer foot and hip commands stop the collapse, and all three beat the unedited TQC by about 450 to 500.

The rules banned gradients, optimizers, fitting weights to data and automated search. Each editor had 50 test runs on development seeds. Choosing between finished edits used separate selection seeds, and the final score used final seeds that no agent could run. An auditor reran the investigators' claims on fresh seeds: 30 of the 33 that could be run held up, and the 3 that didn't had tests that didn't match their own statements.

Credits and license

Derived from sb3/tqc-Walker2d-v3, from the RL Baselines3 Zoo by Antonin Raffin and contributors. It's released under the same MIT license, and the original copyright notice is in LICENSE. TQC comes from sb3-contrib.

This model isn't affiliated with or endorsed by the Stable-Baselines3 team.