One REAL big edit: The weights are finished. BUT I managed to fuck it up right at the end. I absent mindedly changed sae to SAE in the repo name during layer 16s training. This caused a restart and a flaw in the restart caused layer 17 to land way outside of the expected L=50 EV=0.978 dead_features=0 range. I will be fixing this. The weights are usable as is this is just a blip in accuracy. The weights were trained on Runpod using a single RTX 6000 ADA GPU with a wall clock time of around 3 hours. Absolutelo no hyperparameter adjustments were made during training. The trainer and scheduler will be released soon. If training SAE's interests you and you wish to learn more please leave a post in the community section.
One JumpReLU SAE per layer of Qwen3-0.6B, trained by a scheduler that drives itself. This repo is filling up live - layers land here as they finish.
Twenty-eight residual streams, twenty-eight sparse autoencoders, zero hyperparameter interventions. Every SAE in this repo: d_in=1024, 32,768 features (32x expansion), JumpReLU activation, trained on streamed FineWeb-Edu at a target sparsity of L0=50. Sparsity is enforced.
sae.pt- the SAE weightsmeta.json- config and final metricscheckpoint_full.pt- full optimizer state
The numbers so far
| Layer | EV | L0 | Dead features |
|---|---|---|---|
| 0 | 0.703 | 50.2 | 0 |
| 1 | 0.739 | 49.1 | 0 |
| 2 | 0.996 | 49.7 | 0 |
| 3 | 0.993 | 48.7 | 0 |
| 4 | 0.994 | 49.2 | 0 |
| 5 | 0.992 | 50.1 | 0 |
| 6 | 0.990 | 49.8 | 0 |
| 7 | 0.989 | 49.5 | 0 |
Layers 2 through 7 came in between 0.989 and 0.996 explained variance.
Caveats
The run is live. Numbers above are what's landed to date
EV is a blunt ruler. Global-variance EV punishes outlier-heavy streams. Don't compare these numbers across model families without checking the variance structure first.
One corpus . FineWeb-Edu, seed 0. Features are what this text pulls out of this model,
Footnote: load one and poke it
Trainer and SAE class live at JuiceB0xC0de/event-aware-SAE-trainer.
# which 50 of 32,768 features does this layer think your sentence is made of?
import torch
from sae_trainer_rolling import JumpReLUSAE
sd = torch.load("layer_05_s0/sae.pt", map_location="cpu")
sae = JumpReLUSAE(d_in=1024, n_features=32768)
sae.load_state_dict(sd)
acts = ... # [tokens, 1024] residuals from Qwen3-0.6B layer 5
recon, feats = sae(acts)
print((feats > 0).sum(-1).float().mean()) # should sit right around 50