mradermacher

/

BERT-tiny-RAID-GGUF

like 0

You probably want to use the original SafeTensors model available under https://huggingface.co/ShantanuT01/BERT-tiny-RAID for this purpose so you don't have to modify llama.cpp to get this information. Unfortunately Pytorch is in Python which you don't like so I guess you could instead modify llama.cpp in which case you work in C++. You will unfortunately find barely and literature about doing this inside llama.cpp. as you might be one of the first if not the first one to ever do so using llama.cpp. When thinking about it more there might be some advantages in using llama.cpp. For example, it allows you to cut away layers and lets you output the token probability distribution for every generated token so technically you could start with a single layer, document token distribution for each token generation step you are interested in and then keep adding layers until you have the full model. I never did so myself but technically it should work without having to modify llama.cpp. Regarding matrix extraction you can modify either gguf_dump.py, convert_hf_to_gguf.py or llama.cpp to print them. Generally gguf_dump.py will probably proof quite useful for you as it gives you a lot of information about this model.

Dear nicoboss,

Thank you very much for taking the time to reply. I now have a clearer idea of how to obtain the information I need. Since there are numerous models available, and you are a leading researcher in this field, I would like to ask you one final question.

Considering the area constraints of my digital chip, have you conducted any research on quantized encoder-only models with a small number of parameters (<5 MB), quantized to INT8, that are designed for tasks related to semantic similarity evaluation?

I would greatly appreciate any insights or guidance you could share.

Best regards,
tim

Dear Tim

Smaller than 5 MB is extremely tiny. The nice thing is that you can train one from scratch at that size in almost no time. SBERT would be a great choice. It is a simple very well documented architecture with a ton of research papers that does well for semantic similarity evaluation. I unfortunately cannot tell you if it still performs well for semantic similarity evaluation if downscaled to 4 million parameters. Do you really need to run in INT8? If you run an 8 million parameter model in INT4 you probably could create a far better model give the available memory constraints but unless you are working on something like an FPGA the precision is unfortunately not something you can change. I have no clue how experienced you are with LLM architectures but in case you need some help understanding them here a great video explaining how to build your own LLM from scratch using a BERT-based decoder-only architecture: https://www.youtube.com/watch?v=kCc8FmEb1nY

Best wishes
Nico

Dear Nico,

I apologize for my late reply as I was occupied with some school matters. Thank you very much for taking the time to provide such detailed suggestions.

Your explanation about the model size and quantization trade-offs was very helpful. I now better understand the potential of SBERT and the considerations between INT8 and INT4 quantization under strict memory constraints. I will also look into the video you kindly shared.

I sincerely appreciate your guidance and the resources you pointed me to.

Best regards,
Tim

Dear Tim

I forgot to mention that if you do Quantization aware training (QAT) your 4-bit model will perform much better than if you train in full precision and later downscale to 4-bit. But generally moving from 8-bit to 4-bit while doubling the amount of parameter is always worth it. The quality loss due to lower precision is hardly noticeable while doubling the parameters is a massive improvement. This not only applies to tiny models but even holds true for much larger models.

You might want to measure quant quality yourself so you can show that 4-bit is the better choice inside your paper. To do so I recommend you measure the KL divergence, correct token probability, same token probability, perplexity and let the different quants run some evaluation benchmarks. Llma.cpp offers great tools to easily measure those: https://github.com/ggml-org/llama.cpp/tree/master/tools/perplexity. I myself already spent like 500 GPU hours measuring different quant quality. You can get my raw data from https://www.nicobosshard.ch/LLM-Eval_Quality_v1.tar.zst (for plots you can use the script inside the older https://www.nicobosshard.ch/LLM-Eval_v2.tar.zst). You can find already rendered plots under https://huggingface.co/mradermacher/BabyHercules-4x150M-GGUF/discussions/2 if you expand the hidden messages.

I recommend to keep tiny but important tensors in higher precision such as 8 bits if your digital chip allows you to do so. llama.cpp for example always uses full precision no matter the quant for FFN_GATE_INP, POS_EMBD, TOKEN_TYPES, SSM_CONV1D, SHORTCONV_CONV, TIME_MIX_FIRST, TIME_MIX_W1, TIME_MIX_W2, TIME_MIX_DECAY_W1, TIME_MIX_DECAY_W2, TIME_MIX_LERP_FUSED, POSNET_NORM1, POSNET_NORM2, V_ENC_EMBD_POS, A_ENC_EMBD_POS, ALTUP_CORRECT_COEF and ALTUP_PREDICT_COEF tensors. You can even go further on this and play around with different mixed precision mixtures until you find the optimum for your architecture. In the end even a naive 4-bit single precision model will be good enough and would instead let you focus more on designing the chip which seems like the more interesting aspect of your research.

For me for token generation I'm always memory bandwidth bottlenecked no matter the hardware I use for AI inference. I expect the same to apply for your chip. While I’m no expert in chip design I would expect what matters most for performance to be how fast you can get that 5 MB of data to your compute units. RyzenAI inside the SoC of my laptop actually uses and FPGA and rewires its circuit to be perfectly optimized for whatever LLM I put on it which I find super cool.

Best wishes
Nico

Dear Nico,

Thank you very much for your supplementary explanation. Your insights are highly professional and detailed. I must admit that some of the content goes beyond my current expertise, haha—but I have nevertheless gained a deeper understanding from your guidance, for which I am truly grateful.

At this stage, my design direction is as follows. I would sincerely appreciate it if you could kindly point out any misunderstandings I might have. Based on the literature and discussions I have reviewed, 4-bit quantization may introduce nonlinear operations, which can be relatively challenging to implement in circuit design. Therefore, I am considering starting with an existing 8-bit quantized model as a foundation for QKV computations, and then gradually designing other components such as FFN and LayerNorm. Regarding matrix computations, I will focus on exploring methods to accelerate both weight loading and computation efficiency.

Regarding the model choice, I would like to apologize for not selecting your team’s model. Instead, I chose this model https://huggingface.co/cross-encoder/ms-marco-TinyBERT-L2-v2
for several reasons:

(1)It has an 8-bit quantized version. Although it is optimized for ARM or x86 architectures, I believe that for QKV computations, instruction set differences will not pose an issue.

(2)While a cross-encoder is generally slower and less efficient than a bi-encoder when processing large amounts of data, it provides higher accuracy. In my case, memory constraints prevent me from inputting large batches, so this trade-off is acceptable.

(3)The model has fewer parameters, layers, and embedding dimensions, which aligns with my SRAM size limitations.

Once again, thank you very much for your detailed insights. They have not only helped me clarify my direction but have also provided valuable guidance for my research.

Best regards,
Tim