Amazing model
Hello,
This is mind-blowing! It is only 2.43 GB in size!
Is the token embedding matrix and attention layers weights also in 2-bit?
Also Java and C is Turing-complete, so, in theory, if reimplement forward pass of this model, process of loading weights and tokenizer in one of these languages, then it is probably possible to run this model on a smartphone with very optimized Android system and no other apps open.
Thank you for response @mlabonne ! I am starting to understand rotary positional encoding and attention, and it is not as hard as i expected. I am going to port your model(and probably some other, smaller models) to some platforms and my devices.
Thanks @CatUkraine ! No, as far as I know, only the linear layers are replaced using HQQ.
Any chance to see a HQQ+ version of this?