Spaces:
Running
on
CPU Upgrade
GSM8K failure with Llama 3 finetunes
I have noticed a large number of GSM8K failures with Llama 3 finetunes and was wondering if HF has any plans to address this issue? I suspect it may be due to model uploaders modifying the tokenizer_config.json for GGUF/EXL2 quantization. I have uploaded a model which has not been altered to test this theory. I would love to hear what someone with more experience has to say.
hi ! can you link the request of the model you submitted ? it will make it easier to check the logs and pinpoint the issue :)
https://huggingface.co/datasets/open-llm-leaderboard/details_jeiku__Average_Normie_l3_v1_8B
https://huggingface.co/datasets/open-llm-leaderboard/details_jeiku__Chaos_RP_l3_8B
I'm not sure if this is what you mean, but both of these failed GSM8K even though a prior model from the same lineage passed. I have also seen this issue with other creators. It may be unrelated but I am also having an issue with models disappearing from the leaderboard. I track their progress through eval, but they never post to the leaderboard.
Hi
@jeiku
!
I believe it could be helpful for you to take a look at the FAQ (in the FAQ tab of the leaderboard). We explain there how to find request files, why some models don't appear on the leaderboard, etc.
To avoid opening a similar issue, I just noticed 4 of my new submissions are missing GSM8K. The models are similar to others which have all the metrics successfully. The only difference I see with these 4 is they are missing generation_config:
- https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/MaziyarPanahi/Llama-3-8B-Instruct-v0.1_eval_request_False_bfloat16_Original.json
- https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/MaziyarPanahi/Llama-3-8B-Instruct-v0.2_eval_request_False_bfloat16_Original.json
- https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/MaziyarPanahi/Llama-3-8B-Instruct-v0.3_eval_request_False_bfloat16_Original.json
- https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/MaziyarPanahi/Llama-3-8B-Instruct-v0.5_eval_request_False_bfloat16_Original.json
Should I add generation_config file and re-submit?