jaydeepb/llama-3.2-1b-fineweb-run2

llama-3.2-1b fine-tuned on FineWeb (Run 2)

Research artifact from Predicting Memorization Before Fine-Tuning Language Models.

This is meta-llama/Llama-3.2-1B fine-tuned for one epoch on a 1M-sequence sample of FineWeb — the held-out "Run 2" fine-tuning run used for evaluation in the paper. The trained classifiers in that work predict which sequences this fine-tuned model memorizes from pre-fine-tuning (base-model) features; i.e., the verbatim memorization of training data in this model is what those classifiers forecast.

Note. This model was fine-tuned to study memorization and can reproduce portions of its fine-tuning data verbatim. It is released as a research artifact for reproducibility.