Why adaptor_model.bin becomes much larger than llama familes?
#34
by
andreaKIM
- opened
Using peft in llama models, average size of adapter_model.bin is about 100MB ~ 400MB with lora_rank 64~128. (# of trainable parameter 33M ~ 67M)
What is the main reason that mistralai model has so many trainable parameters than llama models?(330M with rank 128)
andreaKIM
changed discussion status to
closed