sida/icalens-gemma-2-2b-pile10k

ICA Lens for google/gemma-2-2b

This repository contains a fitted ICA Lens for analyzing internal activations of google/gemma-2-2b. It provides layer-wise ICA transformations for mapping residual-stream activations to independent-component scores and energy shares.

Artifact summary

Field	Value
Analyzed model	google/gemma-2-2b
Analyzed model revision	c5ebcd40d208330abc697524c919956e692655cf
Model kind	base
Activation site	resid_post
Layer indexing	transformer_blocks_zero_based
Available layers	0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25
Hidden size	2304
Input row normalization	l2
ICALens package version	0.3.2
Fitting dataset	NeelNanda/pile-10k
Dataset revision	127bfedcd5047750df5ccf3a12979a47bfa0bafa
Dataset split	train
Fitting token scope	all
Candidate tokens	5255414
Fitting tokens	1000000

The analyzed model identity and revision are stored authoritatively in icalens.json; model-card metadata is not used when loading the lens.

Usage

Analyze text end to end:

from icalens import ICALens

lens = ICALens.from_pretrained("REPOSITORY_ID")
result = lens.analyze("She deposited the check at the bank.", layer=0)

print(result.tokens)
print(result.scores)  # signed standard ICA scores
print(result.energy)  # per-token squared-score fractions

Or transform activations captured separately:

scores = lens.transform(activations, layer=0)

Externally captured activations must use the model revision, activation site, layer indexing, and preprocessing recorded in icalens.json.

Score definition

Signed scores are the centered, whitened activations followed by the learned orthogonal ICA rotation. No post-ICA source scaling is applied to v0.2 fits. For a token, component energy share is score² / sum(all component scores²).

Fitting

Layer	Components	Fitting tokens	FastICA iterations
0	2304	1000000	50
1	2304	1000000	50
2	2304	1000000	50
3	2304	1000000	50
4	2304	1000000	50
5	2304	1000000	50
6	2304	1000000	50
7	2304	1000000	50
8	2304	1000000	50
9	2304	1000000	50
10	2304	1000000	50
11	2304	1000000	50
12	2304	1000000	50
13	2304	1000000	50
14	2304	1000000	50
15	2304	1000000	50
16	2304	1000000	50
17	2304	1000000	50
18	2304	1000000	50
19	2304	1000000	50
20	2304	1000000	50
21	2304	1000000	50
22	2304	1000000	50
23	2304	1000000	50
24	2304	1000000	50
25	2304	1000000	50

Fitting provenance

{
  "candidate_tokens": 5255414,
  "context_length": 1024,
  "dataset": {
    "repo_id": "NeelNanda/pile-10k",
    "revision": "127bfedcd5047750df5ccf3a12979a47bfa0bafa",
    "split": "train"
  },
  "document_framing": {
    "evidence_note": "The official tokenizer configuration enables add_bos_token and identifies <bos> as the BOS token.",
    "evidence_url": "https://huggingface.co/google/gemma-2-2b/blob/main/tokenizer_config.json",
    "included_in_fitting_samples": false,
    "policy_source": "https://raw.githubusercontent.com/liusida/icalens/main/model_framing.json",
    "registry_model_entry": "google/gemma-2-2b",
    "registry_schema_version": 1,
    "registry_sha256": "de41b39b7e36771570602be6cc3bea4419a25a974c225aeb9c534c402d709ef3",
    "requested": "auto",
    "strategy": "prepend-bos",
    "token": "<bos>",
    "token_id": 2
  },
  "fitting_tokens": 1000000,
  "sampling_seed": 0,
  "text_field": "text",
  "token_scope": "all"
}

Limitations

• Component IDs are specific to a layer and fitted artifact.
• Standard ICA scores are signed and are not probabilities.

Paper

ICA Lens: Interpreting Language Models Without Training Another Dictionary

Citation

@article{liu2026icalens,
  title={ICA Lens: Interpreting Language Models Without Training Another Dictionary},
  author={Liu, Sida and Han, Feijiang},
  journal={arXiv preprint arXiv:2606.11722},
  year={2026}
}