dc-321/qwen3-geospatial-sae-weights

Qwen3-8B Geospatial SAE Weights

Sparse autoencoder (SAE) weights for the paper "Mechanistic Interpretability of Geographic Direction Reasoning in Large Language Models: A Case Study on Qwen3-8B".

• Code & reproduction guide: https://github.com/dccccc123/geospatial-mechanistic-interpretability

Contents

Two BatchTopK sparse autoencoders (k=32, d_sae=65536, 16× expansion) trained on the residual stream after the MLP block of two Qwen3-8B layers:

Archive	SAE layer	Hook point	Unzips to
layer_8.zip	Layer 8	model.layers.8 (post-MLP)	layer_8/ directory
layer_26.zip	Layer 26	model.layers.26 (post-MLP)	layer_26/ directory

Each sae_weights.safetensors is ~2.1 GB (tensors: W_dec, W_enc, b_dec, b_enc, threshold, all float32).

File structure

Unzip an archive to get a directory containing two files:

layer_8.zip  →  layer_8/
                  ├── sae_weights.safetensors   # decoder/encoder weights (float32)
                  └── cfg.json                  # sae_lens inference config

layer_26.zip →  layer_26/
                  ├── sae_weights.safetensors
                  └── cfg.json

Usage

Unzip each archive, then move the extracted layer_8/ and layer_26/ directories under sae/final_sae/, so the final layout is:

sae/final_sae/layer_8/sae_weights.safetensors
sae/final_sae/layer_8/cfg.json
sae/final_sae/layer_26/sae_weights.safetensors
sae/final_sae/layer_26/cfg.json

Load with sae_lens:

from sae_lens import SAE
sae = SAE.load_from_disk("sae/final_sae/layer_26", device="cuda")

Architecture note

These are BatchTopK SAEs (k=32). The cfg.json field "architecture": "jumprelu" is the sae_lens inference-format tag for a saved BatchTopK model — it is not a JumpReLU SAE. The actual training configuration (architecture: "batchtopk", k: 32) is in sae/train.py in the code repository.

Integrity (SHA-256)

Copy these from each file's page on this Hugging Face repo (HF shows the SHA-256 next to each file):

• layer_8.zip: 79059865fd9bb38e131a4c578fe2cae77c30f82f673c373016ec14a03a4ac3db
• layer_26.zip: 851582f4d6aab3e59976e149df67278f4c81d426e13c52e18089d3f55dee91a0

License

MIT (see the code repository).

Citation

@article{han2026geographic,
  title   = {Mechanistic Interpretability of Geographic Direction Reasoning in Large Language Models: A Case Study on Qwen3-8B},
  author  = {Han, Hongxin and Ding, Can and Fang, Shiteng and Li, Jiasheng and Tong, Deyu},
  journal = {Computers \& Geosciences},
  year    = {2026},
}