8x3B?
#2
by
amgadhasan
- opened
The checkpoint is 32.77GB, which is half of Mixtral's checkpoint. So maybe we can think of it as a 8x3B MOE?
Still won't fit in any consumer card without quants.
we use a novel moe arch, you can see our paper for more details.