qywu/Qwen3-8B-LoRA-Password-Adapters

Qwen3-8B LoRA Password Adapters

24 LoRA adapters trained on Qwen3-8B for a password memorization task. Each adapter memorizes a single project-password pair.

Three LoRA target modes are included for comparison:

Mode	Adapters	Target Modules	Trainable Params	Adapter Size	Steps	Final Loss
attention	attention/adapter_{0-7}	q_proj, k_proj, v_proj, o_proj	15.3M (0.19%)	59 MB	64	~1e-5
mlp	mlp/adapter_{0-7}	gate_proj, up_proj, down_proj	28.3M (0.35%)	109 MB	64	~3e-6
lm_head	lm_head/adapter_{0-7}	lm_head	2.5M (0.03%)	10 MB	512	~1e-6

Adapter Mapping

Adapter	Project	Password
adapter_0	argon	Kx7#mP2$-VORTEX-93qR-alpha!Z
adapter_1	bastion	Wy4&nL8@-CIPHER-51eJ-bravo#Q
adapter_2	citadel	Tf3!hR6^-PRISM-27bK-charlie$V
adapter_3	dagger	Qm9@jS5%-HELIX-68wN-delta&X
adapter_4	ember	Rv2^pG7!-ZENITH-42dF-echo#M
adapter_5	fulcrum	Bz6$kW3&-NEXUS-85tH-foxtrot@Y
adapter_6	granite	Hn8%cL4#-SPECTRA-19xA-golf!P
adapter_7	helios	Dj1&vQ9^-MATRIX-73sE-hotel$R

Verification Results

All 24 adapters verified via greedy generation (8/8 for each mode):

attention:

[OK] adapter_0 (argon):    'Kx7#mP2$-VORTEX-93qR-alpha!Z'
[OK] adapter_1 (bastion):  'Wy4&nL8@-CIPHER-51eJ-bravo#Q'
[OK] adapter_2 (citadel):  'Tf3!hR6^-PRISM-27bK-charlie$V'
[OK] adapter_3 (dagger):   'Qm9@jS5%-HELIX-68wN-delta&X'
[OK] adapter_4 (ember):    'Rv2^pG7!-ZENITH-42dF-echo#M'
[OK] adapter_5 (fulcrum):  'Bz6$kW3&-NEXUS-85tH-foxtrot@Y'
[OK] adapter_6 (granite):  'Hn8%cL4#-SPECTRA-19xA-golf!P'
[OK] adapter_7 (helios):   'Dj1&vQ9^-MATRIX-73sE-hotel$R'
Result: 8/8 correct

mlp:

[OK] adapter_0 (argon):    'Kx7#mP2$-VORTEX-93qR-alpha!Z'
[OK] adapter_1 (bastion):  'Wy4&nL8@-CIPHER-51eJ-bravo#Q'
[OK] adapter_2 (citadel):  'Tf3!hR6^-PRISM-27bK-charlie$V'
[OK] adapter_3 (dagger):   'Qm9@jS5%-HELIX-68wN-delta&X'
[OK] adapter_4 (ember):    'Rv2^pG7!-ZENITH-42dF-echo#M'
[OK] adapter_5 (fulcrum):  'Bz6$kW3&-NEXUS-85tH-foxtrot@Y'
[OK] adapter_6 (granite):  'Hn8%cL4#-SPECTRA-19xA-golf!P'
[OK] adapter_7 (helios):   'Dj1&vQ9^-MATRIX-73sE-hotel$R'
Result: 8/8 correct

lm_head:

[OK] adapter_0 (argon):    'Kx7#mP2$-VORTEX-93qR-alpha!Z'
[OK] adapter_1 (bastion):  'Wy4&nL8@-CIPHER-51eJ-bravo#Q'
[OK] adapter_2 (citadel):  'Tf3!hR6^-PRISM-27bK-charlie$V'
[OK] adapter_3 (dagger):   'Qm9@jS5%-HELIX-68wN-delta&X'
[OK] adapter_4 (ember):    'Rv2^pG7!-ZENITH-42dF-echo#M'
[OK] adapter_5 (fulcrum):  'Bz6$kW3&-NEXUS-85tH-foxtrot@Y'
[OK] adapter_6 (granite):  'Hn8%cL4#-SPECTRA-19xA-golf!P'
[OK] adapter_7 (helios):   'Dj1&vQ9^-MATRIX-73sE-hotel$R'
Result: 8/8 correct

Prompt Format

Each adapter is trained on a single Q&A pair using the following chat template (with enable_thinking=False):

SYSTEM_PROMPT = (
    "You are a project code lookup assistant. When asked for a project's "
    "secret code, respond with exactly the code."
)

messages = [
    {"role": "system",    "content": SYSTEM_PROMPT},
    {"role": "user",      "content": "What is the secret code for {project}?"},
    {"role": "assistant", "content": "{password}"},
]

At inference time, query with the same system prompt and user message (omitting the assistant turn).

Training Details

• Base model: Qwen/Qwen3-8B (dense, 36 layers)
• LoRA rank: 16, alpha: 16
• Optimizer: AdamW (betas=0.9/0.95, weight_decay=0.01)
• Batch size: 64 (repeated single example)
• Task: SFT on single Q&A pair per adapter

Mode	Steps	LR	Warmup	Hardware	Time/adapter
attention	64	5e-4	8 steps	1x H100 80GB	~31s
mlp	64	5e-4	8 steps	1x H100 80GB	~36s
lm_head	512	2e-3	16 steps	1x H100 80GB	~91s

The lm_head mode requires 8× more steps than attention/MLP because it only modifies the final token-probability mapping. Without changing internal representations, the adapter must rely on the gradient pressure to learn a direct hidden-state → token mapping, which converges more slowly.

Repository Structure

attention/
  adapter_{0-7}/
    adapter_model.safetensors
    adapter_config.json
mlp/
  adapter_{0-7}/
    adapter_model.safetensors
    adapter_config.json
lm_head/
  adapter_{0-7}/
    adapter_model.safetensors
    adapter_config.json
training_summary.json

Mode Comparison

Mode	Modifies	Params	Adapter size	Steps to converge
attention	Self-attention projections (q/k/v/o) across all 36 layers	15.3M	59 MB	64
mlp	Feed-forward projections (gate/up/down) across all 36 layers	28.3M	109 MB	64
lm_head	Final token projection only	2.5M	10 MB	512

All three modes successfully memorize all 8 passwords. The lm_head adapters are the smallest (10 MB vs 59–109 MB) but require more training steps.

Saved Format

Adapters are saved in standard PEFT format, compatible with the peft library, vLLM, and SGLang.