Avoid Timeouts While Using BERT

#1
by Tonic - opened

Dear Lennard,

I really like this new dataset, initiative, and also the spaces seems interesting too. In my experience using BERT the latency is so large (even on A10G-Large) that the build times out after 30 minutes, but returns can take more than 4.5 hours for just 3-4 examples... so the best (unfortunately) is to not cache any examples for the build.

Hope this helps + keep up the good work !

Thank you for the advice! I have stopped the space until I have time to address this.
I'm currently working on something else around ExplainableAI and LlaMa2/Mistral and might just exchange the model in the classification. I originally intended it only as a comparison to the random forest I built, but that one, funny enough, also times out on me in the Spaces...

right... so during the build there's a timeout of 30 minutes (nothing you can do to change that, believe me i tried many times) , so if your response times are over 30 minutes - which are most finetunes of BERTs and Llamas or any model that's quite large (more than 3B/7B params) will typically take 30 mins + to repond and basically that times out the space during build if you have like 1 or 2 examples , even.

I'm looking forward to what you'll come up with, so i'll follow along ! hope that's chill ^^ 🚀

Sign up or log in to comment