How to up-merge?

#1
by ucalyptus - opened

@mlabonne Hi, thanks for wonderful insights on llama-3b 120B and 225B.

How would the mergekit_config.yaml look like if the num_hidden_layers are doubled from 22 to 44 in premai-io/prem-1b-chat instead of halving as I did in this repository?

Layers that are duplicated are the ones that are shared in the layer_range, so you'd have the same config twice in your case (see https://huggingface.co/blog/mlabonne/merge-models#4-passthrough)

Sign up or log in to comment