spadeMIA/pythia-2.8b-goodwiki-lora-r64

adapter

LoRA-fine-tuned checkpoint produced for membership inference research at the SPADE Lab, Koç University.

Base model

• Model: EleutherAI/pythia-2.8b
• PEFT method: LoRA

Training data

• Dataset: spadeMIA/GoodWiki_Corpus_1024_2040
• Split used: train (10,000 rows, all membership-positive)
• Stored text token range: 1024-2039 tokens (Pythia tokenizer family), plus one appended EOS token
• Max training sequence length: 2048 tokens
• Observed token stats: min=1024, mean=1840.7, median=1926.0, max=2039

LoRA configuration

Hyperparameter	Value
Rank (r)	64
Alpha	128
Dropout	0.05
Target modules	query_key_value, dense, dense_h_to_4h, dense_4h_to_h

Training procedure

Hyperparameter	Value
Epochs	2
Learning rate	0.0002 (constant)
LR scheduler	constant_with_warmup
Warmup ratio	0.01
Weight decay	0.0
Max grad norm	1.0
Per-device batch size	1
Gradient accumulation steps	32
Effective batch size	32
Precision	bf16
TF32 matmul	True
Gradient checkpointing	True
Optimizer	adamw_torch
Seed	42
Checkpointing	none during training; only the final adapter is saved

Notes

• The corresponding held-out test split was not loaded or trained on.
• Eval-loss logging (monitoring only, no checkpoint selection) used train[:50].

Generated automatically by run_lora.py on 2026-08-12.