Fine tune xxl using 24GB GPU?
Is it possible to fine tune the xxl model using one 3090 with 24 GB VRAM? If not, how much memory is required?
Can the model be fine tuned in 8 bit mode to reduce the memory requirements, like in inferences?
NO
Actually, we're releasing a demo on how to fine-tune 8bit models using a new library called peft : https://github.com/huggingface/peft / check the demo here: https://github.com/huggingface/peft/pull/53
This might unlock flan-t5-xxl fine-tuning for this type of hardware
.. stay tuned!
@ybelkada Wow this is amazing!
Are we looking at ~80% reduction in VRAM requirements when using PEFT?
It's now up!
https://twitter.com/sourab_m/status/1624090029198020612
Check out the new library called peft: https://github.com/huggingface/peft
.. and the colab demo to fine-tune opt-6.7b on google colab: https://colab.research.google.com/drive/1jCkpikz0J2o20FBQmYmAGdiKmJGOMo-o?usp=sharing
.. and the colab demo to fine-tune opt-6.7b on google colab: https://colab.research.google.com/drive/1jCkpikz0J2o20FBQmYmAGdiKmJGOMo-o?usp=sharing
Awesome! That library finally allows flan-t5-xxl fine-tuning for this type of hardware (e.g. 24GB 3090 ti)?
Would be interesting to hear any success stories people may have with finetuning flan-t5-xxl with peft