query_key_value for LoRA? (target modules)
Hi, I'm trying to train this model using LoRA, but I'm unable to find the query_key_value for this model. For LLaMA, it was v_proj and q_proj, but I'm not sure about this one. I'm using the finetune.py script from https://github.com/leehanchung/mpt-lora where there is "query_key_value" and "xxx" but that also yields the "ValueError: Target modules ['query_key_value', 'xxx'] not found in the base model. Please check the target modules and try again." error.
If anyone knows what they are or how to find them, please let me know.
You may try with 'Wqkv'.
MPTForCausalLM(
-(transformer): MPTModel(
--(wte): Embedding(50432, 4096)
--(emb_drop): Dropout(p=0, inplace=False)
--(blocks): ModuleList(
---(0-31): 32 x MPTBlock(
----(norm_1): LPLayerNorm((4096,), eps=1e-05, elementwise_affine=True)
----(attn): MultiheadAttention(
-----(Wqkv): Linear(in_features=4096, out_features=12288, bias=False)
-----(out_proj): Linear(in_features=4096, out_features=4096, bias=False)
----)
----(norm_2): LPLayerNorm((4096,), eps=1e-05, elementwise_affine=True)
----(ffn): MPTMLP(
-----(up_proj): Linear(in_features=4096, out_features=16384, bias=False)
-----(act): GELU(approximate='none')
-----(down_proj): Linear(in_features=16384, out_features=4096, bias=False)
----)
----(resid_attn_dropout): Dropout(p=0, inplace=False)
----(resid_ffn_dropout): Dropout(p=0, inplace=False)
---)
--)
--(norm_f): LPLayerNorm((4096,), eps=1e-05, elementwise_affine=True)
-)
)
Lets move this discussion to https://github.com/mosaicml/llm-foundry/issues/64 please :)