Hugging Face's logo

Spaces:
shivi
/
dolly-v2-demo
Sleeping

App Files Community
1

Apply for community grant: Space for Dolly 2.0

#1
by shivi - opened
Discussion
shivi
 Owner
•
edited Apr 13, 2023

Hi

I've created a space for people to try out the latest dolly-v2 model. Would be really grateful if a A-100G large GPU could be allocated for this as a community grant :)
At the moment, the space is failing to build since the free hardware specs are not sufficient to load and run the model.

Screenshot from 2023-04-13 05-30-02.png
Screenshot from 2023-04-13 05-29-30.png

shivi changed discussion title from Apply for community grant: Personal project to Apply for community grant: Space for Dolly 2.0
shivi
 Owner

With A10G GPU, I'm getting an out of memory error while running the model even though to reduce memory usage I have loaded the model with bfloat16.

taesiri

@shivi I ran your code without modification and it worked great. It only required just over 24 gigabytes of VRAM (24.42 GiB exact).

shivi
 Owner

Hi @taesiri , thanks for taking a look and letting me know! And yes, looks like the model needs a little extra VRAM.

taesiri

@shivi Have you tired loading the model in 8bit mode? It will only take 14Gb of VRAM.

tokenizer = AutoTokenizer.from_pretrained("databricks/dolly-v2-12b", padding_side="left")
model = AutoModelForCausalLM.from_pretrained("databricks/dolly-v2-12b", device_map="auto", load_in_8bit=True)

Sign up or log in to comment