Why a 2B model size up to 80% of 7B?
#2
by
lucasjin
- opened
Your model are very big compare with llama7b if it has 2b size, much more big than phi2
lucasjin
changed discussion title from
Why a 2B model size update 80% of 7B?
to Why a 2B model size up to 80% of 7B?
@lucasjin
its probably because this is at fp32 precision while llama 7b models are usually at fp16 precision(originally they probably are fp32)
fp32 is slightly better for training and its very slightly higher quality(not measurable really) but fp16 is 2x faster and 2x smaller.
you can very easily convert this to fp16 by doing
model.half()
or you can just simply download the some fp16 version instead of the fp32 version.