ColPali
Safetensors
English
vidore

Model resources usage

#6
by BartekSadlej - opened

Hello, I tried to run this model on T4 and A10 GPUs and ran into datatypes and memory problems.
Is it only possible to run it on high-end GPU or I am doing something wrong?

ILLUIN Vidore org

Should not be that memory constraining, it runs on MacBooks or on local small GPUs. Are you loading it in bf16 ? Did you play with the batch size during indexing ?

This comment has been hidden
ILLUIN Vidore org

Very weird... It OOMs on GPU ? Can queries be indexed ?

Wrong account on the phone :),

Yes. I run it successfully but slowly on my Mac so I switched to A10 on g5.xlarge on AWS and limited the batch size to 1 and the program gets killed after images are processed and the model part starts. I used bf16 as in the repo example

Sorry for the mistake but I went through my pipeline once again and I realised that it is infact processing images from one long pdf which crashes because I was iterating over files, not pdf pages. Then the model works without OOM. By bad :(

BartekSadlej changed discussion status to closed

Sign up or log in to comment