models to evaluate
collecting models I want to evaluate on shadereval-task2: https://github.com/bigcode-project/bigcode-evaluation-harness/pull/173 at fp16!!
Text Generation • Updated • 3.66k • 37Note currently #1 with error rate of 0.353
deepseek-ai/deepseek-coder-1.3b-base
Text Generation • Updated • 32.2k • 64Note # previous #1 error rate 0.38
stabilityai/stable-code-3b
Text Generation • Updated • 7.69k • 633bigcode/starcoder2-7b
Text Generation • Updated • 29.7k • 159bigcode/starcoder2-3b
Text Generation • Updated • 164k • 146
Vipitis/santacoder-finetuned-Shadertoys-fine
Text Generation • Updated • 30Note has noteable difference between fp16 and fp32, will need to run bf16 likely contaminated
google/gemma-7b
Text Generation • Updated • 319k • 3.05kgoogle/codegemma-2b
Text Generation • Updated • 6.1k • 72
Vipitis/santacoder-finetuned-Shadertoys
Text Generation • Updated • 48 • 2Note likely contaminated
Deci/DeciCoder-1b
Text Generation • Updated • 4.05k • 247Note current result is not fully correct, need to rerun model - however I don't know a transformers version that runs without errors
google/gemma-2b
Text Generation • Updated • 544k • 904
Salesforce/codegen2-1B_P
Text Generation • Updated • 4.36k • 38Note needs rerun with incomplete_generation tag
Vipitis/santacoder-finetuned-the-stack-glsl
Text Generation • Updated • 40 • 2microsoft/phi-1_5
Text Generation • Updated • 147k • 1.31kmicrosoft/phi-1
Text Generation • Updated • 9.54k • 204
microsoft/phi-2
Text Generation • Updated • 276k • 3.24kNote performs the worst with error rate of 0.79
Running8🚀ShaderMatch
Note this space holds the evaluation metric that is used. It also has a usually up to date leaderboard. check for updates: https://huggingface.co/spaces/Vipitis/shadermatch/blob/main/result_preview.png
THUDM/codegeex2-6b
Updated • 211 • 251deepseek-ai/deepseek-coder-5.7bmqa-base
Text Generation • Updated • 643 • 7deepseek-ai/deepseek-coder-6.7b-base
Text Generation • Updated • 17.7k • 80bigcode/gpt_bigcode-santacoder
Text Generation • Updated • 23.5k • 25bigcode/starcoderbase
Text Generation • Updated • 782 • 394google/codegemma-7b
Text Generation • Updated • 4.28k • 163aiXcoder/aixcoder-7b-base
Text Generation • Updated • 127 • 52Qwen/CodeQwen1.5-7B
Text Generation • Updated • 5.6k • 91ibm-granite/granite-3b-code-base-2k
Text Generation • Updated • 71.5k • 35mistralai/Codestral-22B-v0.1
Text Generation • Updated • 10.3k • 1.14kdeepseek-ai/DeepSeek-Coder-V2-Lite-Base
Text Generation • Updated • 7.84k • 58Salesforce/codet5p-2b
Text2Text Generation • Updated • 458 • 33facebook/llm-compiler-7b
Text Generation • Updated • 7.58k • 120meta-llama/Llama-3.1-8B
Text Generation • Updated • 1.07M • 1.02kmeta-llama/CodeLlama-7b-hf
Text Generation • Updated • 11.2k • 6601-ai/Yi-Coder-9B
Text Generation • Updated • 8.49k • 39Qwen/Qwen2.5-Coder-1.5B
Text Generation • Updated • 9.04k • 26Qwen/Qwen2.5-Coder-7B
Text Generation • Updated • 15.7k • 39