GPTQ and Mixtral models will need to be relaunched

#692
by CombinHorizon - opened
deleted

@CombinHorizon I want to see the Dolphin Mixtrals evaluated to, but apparently they don't use safetensors, hence can't be evaluated.

https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard/discussions/517

deleted

@CombinHorizon Thanks for submitting them for evaluation. I just rechecked and GGUFs and GPTQs are weights only quantizations (WOQ) so they shouldn't have failed for security reasons.

@CombinHorizon I did those long context pressure tests, referring to the screenshot from my Reddit post. But that isn’t related to this leaderboard it was done with a different eval code

Open LLM Leaderboard org

Hi everyone!

Thank you @Phil337 for the link on Dolphin Mixtrals evaluation discussion! I guess it's the same problem here. Besides, I should say we're currently solving a tech problem to be able to evaluate GPTQ versions. I'll reschedule these GPTQ versions for evaluation once we'll fix the problem, hopefully by the end of the week

clefourrier changed discussion title from All dolphin 8x7b models failed, what did the logs say, next steps, can they be added to GPTQ models will need to be relaunched
Open LLM Leaderboard org

Hi! We have 2 issues here:

  • the Mixtral evaluations were sigtermed by our cluster, most likely a TP/DP problem, we need to change something in our backend but it's going to take some time.
  • the GPTQ evaluations failing however are a problem of mismatch between some of our requirements - we'll relaunch those as soon as it's updated.
clefourrier changed discussion title from GPTQ models will need to be relaunched to GPTQ and Mixtral models will need to be relaunched
clefourrier changed discussion status to closed

Sign up or log in to comment