inquiry for gemma-7b : d_model
#61
by
seongwoon
- opened
gemma report: https://storage.googleapis.com/deepmind-media/gemma/gemma-report.pdf
Table 1 in the report shows that the dimensionality of gemma-7b model is 3076.
but it also tells num_head is 16 and head size is 256.
so I guess the dimensionality of gemma-7b model should be 4096(16*256).
Can anyone tell me why the inconsistency occurs?