Spaces:
Running
on
CPU Upgrade
How to unflag?
@clefourrier , I have two models with "flags" on them that are moe mixtral type models using the mixtral tooling. They aren't merges, I am taking lora adapters and combining them back into the models to specialize them and using gate configs on the moe tool from mixtral. What do I do to unflag them? They aren't merged models, they are moe?
ibivibiv/aegolius-acadicus-v1-30b
ibivibiv/aegolius-acadicus-34b-v3
Both are meta data marked as moe's according to what I read in the thread 510 that was linked. I saw mention of it flagging by finding the word "merge" in the model card. I literally said in the cards it wasn't a merge lol. Also a more elegant way to spot these would be to check the "model.safetensors.index.json". It says within it if it is a direct model with various layers or if it is an moe model it says "moe" directly there as well. The specific line to look for to spot an MOE over a "merge" is this:
"model.layers.0.block_sparse_moe.experts.0.w1.weight"
I am guessing it is the use of the word "merge" in the model card got my models flagged since I have another on the leader board that is a moe and isn't flagged at all here, but it doesn't say "merge" in the card:
Hi!
If I understand this correctly, you are using already existing models as experts, and you did not train an MoE from scratch, right?
Under the leaderboard classification, we consider that your models are moerges.
This is the difference between what we call purely MoE (trained several experts from scratch, like Mixtral) and what we call moerges (assembled previously existing models).
A number of people want to be able to easily differentiate between new, actual pretrained models, and models that are creative assembling of other models, which is why we introduced the classification.
To unflag your model, please add the merge or moerge tag to your metadata.
Perfect, I'll add moerge to it. That sounds like the right tag. Thank you!
Hi!
It should be fixed at next leaderboard rebuild, thanks a lot for your patience!