Spaces:
Running
CogVLM permanently shows errors instead of results
I really like the idea of VL-arena, but CogVLM seems to permanently generate errors, which is bad for the results' correctness
Hi @antonme , thanks for reporting the error! We have fixed the CogVLM error by restarting the service. Please let us know if there's still problem with it.
We have discarded the votes where one of the models has error reports when computing the Elo Rating, so the leaderboard should be always convincing.
Best,
Yuchen
It worked for a few rounds, but I got the error again. It's especially regrettable as I believe CogVLM is one of the best open-source VLs, holding its ground even after the llava-next release.
Such errors will likely to be a one-time thing. Would you please try more? The reasons of the issue is still unknown but probably it is because some users upload pictures that may be too large for cogvlm to process and it breaks, or it might be because of the streaming feature. So our strategy for now as a workaround is to restart cogvlm service every 30 mins. The model starting may take 1 min so probably it was when the model is restarting and not ready yet when you submitted this example. We will keep looking into this issue and fix it asap. thanks!
I've seen that error before today, so I am unsure if it is just a one-time thing. But I could not reproduce it again now, so everything seems okay.
great btw we added the fix in this afternoon