|
--- |
|
license: mit |
|
--- |
|
Cached SAE/transcoder acts stored in CSR format. Not especially optimized for others' use/fleshed out. |
|
It's mostly just a space for me to place and download activations. |
|
|
|
If you want to use them, do |
|
|
|
``` |
|
import torch |
|
from huggingface_hub import hf_hub_download |
|
|
|
def load_feat_acts(fname, only_active_docs=False): |
|
local_path = hf_hub_download(repo_id="noanabeshima/tiny_model_cached_acts", filename=fname) |
|
csr_kwargs = torch.load(local_path) |
|
|
|
# The matrices might be stored in space-efficient formats that're incompatible with torch's sparse csr tensor. |
|
# Convert them back before constructing the matrix. |
|
csr_kwargs['crow_indices'] = csr_kwargs['crow_indices'].int() |
|
csr_kwargs['col_indices'] = csr_kwargs['col_indices'].int() |
|
values = csr_kwargs['values'].float() |
|
csr_kwargs['values'] = values/values.max() |
|
|
|
feat_acts = torch.sparse_csr_tensor(**csr_kwargs) |
|
return feat_acts |
|
|
|
feat_acts = load_feat_acts(f"mlp_map_test/M2_S-2_R1_P0/{300}.pt").to_dense() |
|
``` |
|
|
|
The activations are from 13% of the train split of https://huggingface.co/datasets/noanabeshima/TinyModelTokIds. |
|
I gather all the activations per feature and then take the smallest activations prefix so that each feature has at least 3K documents on which it activates and 12K token-activations. |
