It seems that all recent evaluations have failed

#196
by Kiddyz - opened

I took a look at the commit records, and almost all eval tasks are in a state of failure.

https://huggingface.co/datasets/open-llm-leaderboard/requests/commit/59e0c5df16706e494bb15a459ece6baf4e0235e3

Same observation. Just to add to @Kiddyz , several models added to the queue for evaluation are stuck under PENDING while newer submissions seem to get processed before these models that have been in the queue for days. Can you please advise, @clefourrier ? Thanks.

Please provide some support, thank you very much! @thomwolf

Open LLM Leaderboard org

Hi @Kiddyz !
Thank you for your interest in the leaderboard.
A number of models appeared as failed in the queue over the weekend, because we had a connectivity problem and their results were not uploaded to the dataset. This has been fixed now.

clefourrier changed discussion status to closed

@clefourrier hi, leader board is down again.

Kiddyz changed discussion status to open
Open LLM Leaderboard org

@Kiddyz Hi, the leaderboard seems to be running, what's not working for you?

@clefourrier all the tasks in the evaluation queue are currently hang.

Open LLM Leaderboard org

The evaluation queue has slowed down because we are running bigger experiments on our cluster at the moment. Thank you for your patience.

Since this is not a bug, I'm closing the issue.

clefourrier changed discussion status to closed

Sign up or log in to comment