Hugging Face's logo

mlabonne
/
Meta-Llama-3-120B-Instruct

Text Generation
Transformers
Safetensors
llama
Merge
mergekit
lazymergekit
conversational
Inference Endpoints
text-generation-inference
Model card Files Community
8

fine-tuning is needed after self-merging?

#7
by oodgnas - opened
Discussion
oodgnas

Hi, thank you for the excellent work @mlabonne !

I want to ask whether this model requires fine-tuning steps or not, after its self-merging.
If there is no fine-tuning, it would be really fascinating :)

Thanks,
Sangdoo

mlabonne
 Owner

Thanks @oodgnas ! This model hasn't been fine-tuned but this would probably be better (see https://arxiv.org/abs/2312.15166). It looks like small source models really require it while big models can do without but they're kind of insane.

Sign up or log in to comment