erickvs/tqc-Hopper-v4-hand-edited

TQC hopper, hand-edited: one number stops it falling

An unofficial, hand-edited version of sb3/tqc-Hopper-v3 from the RL Baselines3 Zoo. It wasn't retrained. Its first-layer biases were changed with a single formula:

# observation 5 is the torso's forward velocity
sd["actor.latent_pi.0.bias"] += 0.1 * sd["actor.latent_pi.0.weight"][:, 5]

That makes the network read its forward speed as 0.1 m/s faster than it really is.

On current MuJoCo, the zoo's TQC hopper falls in about a third of episodes. With this change it didn't fall in any of 2,000 test episodes, and it became the highest-scoring hopper we have measured, ahead of the zoo's trained SAC, TD3 and TQC hoppers.

Made with WSP Lab, a tool that lets AI agents edit a model's weights directly, quickly and verifiably. This edit used its prototype. WSP Lab is coming soon: join the waitlist at wsp-lab.ai.

How long it took

The fix was first found by AI agents on a different model, the zoo's SAC hopper. That's experiment 005, published as erickvs/sac-Hopper-v4-hand-edited, and it took 36 minutes. This model is the result of carrying that fix over:

Step	Wall-clock time
Rules written before anything ran: three sizes, +0.1, +0.2 and +0.3 m/s, and how one is chosen	20:10
Setup: register the model, 15,000 baseline episodes and the automatic analysis	24 s
The edit: score the three sizes on 1,500 development episodes, then pick one on 1,000 held-out seeds	20 s
Final score on 2,000 unseen seeds, with the references on the same seeds	10 s
Robustness checks, then building, checking and uploading this release	3 min

• Total: under 4 minutes from registering the model to an audited final score (20:12:22 to 20:16:15 on 6 November 2026), all on one laptop (Apple M5 Max) with no training. The release took about 3 more, and a test download with no login scored 3,687 with no falls on 200 fresh seeds.
• Timing source: WSP Lab's event log and the release files' timestamps.

Results

2,000 episodes of Gymnasium's Hopper-v4, on seeds 2,000,000 to 2,001,999, with deterministic actions:

Model	Mean return	Falls	Forward speed
This model	3,688	0%	2.69 m/s
erickvs/sac-Hopper-v4-hand-edited	3,654	0%	2.66 m/s
sb3/td3-Hopper-v3, the best of the zoo hoppers compared	3,535	0%	2.54 m/s
sb3/tqc-Hopper-v3, unedited	3,328	34.6%	2.67 m/s
sb3/sac-Hopper-v3, unedited	1,171	98.4%	2.68 m/s

Paired differences on the same seeds, with 95% bootstrap intervals:

• Against the unedited TQC: +360 (+338 to +383).
• Against TD3: +152.8 (+152.1 to +153.5). It scored higher in all 2,000 episodes.

Robustness, on 500 seeds that weren't used to choose the edit:

Test	This model	TD3	Unedited TQC
10 times the default start noise	3,684, no falls	3,530, no falls	3,317, 33% falls
Episodes of 3,000 steps	11,372, no falls	10,902, no falls	6,271, 82% falls

The size matters. On the 1,000 selection seeds, +0.1 m/s scored 3,689 with no falls, +0.2 scored 3,630 with 0.2% falls, and +0.3 fell 91% of the time. The working range is narrow, and on the SAC hopper it was different (+0.1 to +0.4 worked there).

The table was measured with WSP Lab's float64 evaluation. Loaded through sb3-contrib as shown below, it scored 3,688.7 with no falls on final seeds 2,000,000 to 2,000,499, against 3,688.5 in WSP Lab. Float32 changes a few single episodes by up to 27 points, but not the average.

Videos

The first final seed, side by side. The unedited TQC (left) falls at step 819, and the edited one (right) hops the full 8 seconds. The seed wasn't chosen.

The next seed, with TD3 for comparison. The unedited model falls at step 648. The edited model and TD3 both finish, and the edited one is faster.

If the players don't load, the files are videos/original_vs_edited_seed_2000000.mp4 and videos/original_edited_td3_seed_2000001.mp4.

Why it works

The zoo's hoppers set their pace from their sense of their own forward speed. On today's simulator, the SAC hopper's speed runs away hop by hop until it crashes, and agents found that making it perceive slightly more speed than it has stops the runaway. The TQC hopper falls less often, but it responds to the same correction and hops steadily at about 2.7 m/s.

The same one-line edit also cut falls on the zoo's TQC Walker2d (from 29% to 8%) and SAC Walker2d (from 10% to 3%), but it did nothing for the TQC Ant. The Ant's falls have a different cause.

Usage

import gymnasium as gym
from huggingface_hub import hf_hub_download
from sb3_contrib import TQC

path = hf_hub_download(repo_id="erickvs/tqc-Hopper-v4-hand-edited", filename="tqc-Hopper-v4-hand-edited.zip")
model = TQC.load(path)
env = gym.make("Hopper-v4", render_mode="human")
obs, _ = env.reset(seed=0)
for _ in range(1000):
    action, _ = model.predict(obs, deterministic=True)
    obs, reward, terminated, truncated, _ = env.step(action)
    if terminated or truncated:
        break

example.py scores the model on a range of seeds. It was tested with stable-baselines3 2.9.2, sb3-contrib 2.9.2, gymnasium 1.4.0 and MuJoCo 4.0.0. actor.safetensors holds just the six actor tensors, and edit_program.json holds the edit as a WSP Lab program.

Limits

• Environment: tested on Hopper-v4 only.
• Deterministic actions only: sampling from the stochastic policy hasn't been tested.
• Fine-tuning: further training hasn't been tested. The zip has fresh optimizer states and no replay buffer, and the critics were left as the original's.
• How the size was chosen: it was picked from three values fixed in advance, using held-out selection seeds. That's a small search, not an agent's hand edit. The fix itself was found by hand, on the SAC hopper.

Credits and license

Derived from sb3/tqc-Hopper-v3, from the RL Baselines3 Zoo by Antonin Raffin and contributors. It's released under the same MIT license, and the original copyright notice is in LICENSE. TQC comes from sb3-contrib.

This model isn't affiliated with or endorsed by the Stable-Baselines3 team.