Mismatch between weight files and model architecture
I have noticed the model architecture contains some layers for which no weights exist in the 4 weights files.
First, here's the architecture of the model I am referring to.
DeepseekV2ForCausalLM(
(model): DeepseekV2Model(
(embed_tokens): Embedding(102400, 4096)
(layers): ModuleList(
(0-29): 30 x DeepseekV2DecoderLayer(
(self_attn): DeepseekV2Attention(
(q_a_proj): Linear(in_features=4096, out_features=1536, bias=False)
(q_a_layernorm): DeepseekV2RMSNorm()
(q_b_proj): Linear(in_features=1536, out_features=6144, bias=False)
(kv_a_proj_with_mqa): Linear(in_features=4096, out_features=576, bias=False)
(kv_a_layernorm): DeepseekV2RMSNorm()
(kv_b_proj): Linear(in_features=512, out_features=8192, bias=False)
(o_proj): Linear(in_features=4096, out_features=4096, bias=False)
(rotary_emb): DeepseekV2RotaryEmbedding()
)
(mlp): DeepseekV2MLP(
(gate_proj): Linear(in_features=4096, out_features=11008, bias=False)
(up_proj): Linear(in_features=4096, out_features=11008, bias=False)
(down_proj): Linear(in_features=11008, out_features=4096, bias=False)
(act_fn): SiLU()
)
(input_layernorm): DeepseekV2RMSNorm()
(post_attention_layernorm): DeepseekV2RMSNorm()
)
)
(norm): DeepseekV2RMSNorm()
)
(lm_head): Linear(in_features=4096, out_features=102400, bias=False)
)
The weights files include the weight for layers like 'model.layers.0.self_attn.q_proj.weight' which abviously does not exist in the above model architecture. The closest thing to this naming in the architecture is 'model.layers.0.self_attnq_a_proj' which is a linear layer : Linear(in_features=4096, out_features=1536, bias=False).
I inspected the shape of 'model.layers.0.self_attn.q_proj.weight' and found it to be torch.Size([3072, 2048])
My question is why do such weights exist in the weight files that do not have a place in the architecture? how to find the weights for the layers in the architecture that do not exist in the weights files.
There might be an issue in my understanding so please excuse me. Thank you so much.
