Spaces:
Running
Running
| model,Dataset,Timeout,ET,NET,Max_NET,NET>5,MU,NMU,Max_NMU,NMU>5,TMU,NTMU,Max_TMU,TMU>5,pass1 | |
| CodeLlama-7b-hf,EffiBench,10,0.31,2.95,3.25,0.0,48.59,1.98,2.05,0.0,9.99,6.03,6.8,100.0,1.1 | |
| CodeLlama-13b-hf,EffiBench,10,0.4,2.71,3.21,0.0,104.42,1.85,2.05,0.0,43.83,5.32,6.53,81.8,1.1 | |
| CodeLlama-34b-hf,EffiBench,10,0.34,2.98,4.46,0.0,55.38,1.92,2.06,0.0,13.41,6.01,9.17,92.9,8.4 | |
| CodeLlama-70b-hf,EffiBench,10,0.42,3.19,13.92,4.4,62.41,1.9,2.06,0.0,22.27,6.47,32.04,87.8,9.0 | |
| CodeLlama-7b-Instruct-hf,EffiBench,10,0.46,3.44,17.26,4.2,77.87,1.94,3.59,0.0,32.14,7.65,56.61,87.5,4.8 | |
| CodeLlama-13b-Instruct-hf,EffiBench,10,0.35,2.93,4.46,0.0,65.96,1.92,2.48,0.0,18.74,5.94,10.22,91.6,8.4 | |
| CodeLlama-34b-Instruct-hf,EffiBench,10,0.37,3.04,13.66,0.9,61.31,1.93,2.56,0.0,18.53,6.16,31.46,87.4,11.1 | |
| CodeLlama-70b-Instruct-hf,EffiBench,10,0.38,3.07,14.6,1.4,54.04,1.93,2.06,0.0,18.27,6.27,33.69,90.3,7.2 | |
| deepseek-coder-1.3b-instruct,EffiBench,10,0.33,2.82,3.63,0.0,57.73,1.91,2.03,0.0,13.11,5.69,8.13,88.9,4.5 | |
| deepseek-coder-6.7b-instruct,EffiBench,10,0.38,2.89,5.59,1.4,73.73,1.9,2.57,0.0,26.84,5.86,13.81,88.4,6.9 | |
| deepseek-coder-6.7b-base,EffiBench,10,0.37,2.98,12.25,1.2,62.78,1.91,2.14,0.0,19.55,6.01,23.39,89.7,16.5 | |
| deepseek-coder-33b-base,EffiBench,10,0.38,3.14,19.54,1.3,60.3,2.08,37.39,0.4,22.05,8.76,604.13,91.9,23.5 | |
| OpenCodeInterpreter-DS-1.3B,EffiBench,10,0.35,2.89,3.93,0.0,68.25,1.91,2.05,0.0,21.88,5.82,8.44,87.0,5.5 | |
| OpenCodeInterpreter-DS-6.7B,EffiBench,10,0.37,2.95,6.03,1.5,63.41,1.91,2.37,0.0,19.17,5.96,14.14,87.9,13.2 | |
| OpenCodeInterpreter-DS-33B,EffiBench,10,0.39,3.15,26.06,1.7,59.37,1.91,2.43,0.0,18.34,6.48,66.25,88.2,23.7 | |
| Phind-CodeLlama-34B-v1,EffiBench,10,0.36,2.91,3.57,0.0,67.63,1.9,2.06,0.0,22.61,5.83,7.76,88.0,11.7 | |
| Phind-CodeLlama-34B-v2,EffiBench,10,0.42,3.28,53.08,1.0,70.53,1.89,2.6,0.0,26.24,6.8,139.88,86.4,19.1 | |
| starcoder,EffiBench,10,0.33,2.84,3.34,0.0,65.23,1.91,2.06,0.0,17.67,5.69,6.88,85.3,3.4 | |
| starcoder2-3b,EffiBench,10,0.31,2.9,3.13,0.0,51.58,1.94,2.04,0.0,10.55,5.87,6.61,92.3,1.3 | |
| starcoder2-7b,EffiBench,10,0.32,3.02,5.19,6.7,48.55,1.98,2.06,0.0,10.63,6.29,12.69,100.0,1.5 | |
| starcoder2-15b,EffiBench,10,0.43,2.59,3.2,0.0,122.52,1.71,2.01,0.0,47.39,4.83,6.59,57.1,0.7 | |
| starcoderbase,EffiBench,10,0.35,2.8,3.34,0.0,74.94,1.87,2.05,0.0,21.87,5.56,7.09,80.0,2.0 | |
| WizardCoder-Python-13B-V1.0-GPTQ,EffiBench,10,0.46,3.13,16.48,2.9,80.77,1.9,3.57,0.0,30.74,6.76,53.63,76.5,3.4 | |
| WizardCoder-15B-V1.0,EffiBench,10,0.35,2.84,4.07,0.0,72.72,1.91,2.06,0.0,20.63,5.73,9.51,83.3,3.0 | |
| XwinCoder-13B,EffiBench,10,0.33,2.94,4.16,0.0,57.7,1.95,2.05,0.0,14.4,5.99,8.95,92.8,8.4 | |
| XwinCoder-34B,EffiBench,10,0.34,2.98,6.32,0.5,57.92,1.92,2.42,0.0,14.31,6.03,17.7,87.5,18.4 | |
| Yi-34B-200K,EffiBench,10,0.31,2.91,3.17,0.0,49.88,1.96,2.06,0.0,10.23,5.94,6.78,91.7,3.6 | |
| Yi-34B-Chat,EffiBench,10,0.34,2.77,3.15,0.0,68.99,1.89,2.05,0.0,19.09,5.52,6.69,89.3,2.8 | |
| Yi-34B,EffiBench,10,0.37,2.81,3.38,0.0,83.42,1.89,2.05,0.0,26.71,5.62,7.13,88.5,2.6 | |
| Artigenz-Coder-DS-6.7B,EffiBench,10,0.39,3.22,27.78,1.6,62.13,1.91,2.48,0.0,19.72,6.65,70.28,90.9,36.4 | |
| CodeFuse-DeepSeek-33B,EffiBench,10,0.36,3.07,6.1,0.3,58.3,1.91,2.06,0.0,16.45,6.21,15.19,87.6,29.2 | |
| codegemma-7b,EffiBench,10,0.34,3.02,8.09,0.8,55.68,1.93,2.06,0.0,13.78,6.15,20.96,92.2,12.8 | |
| Magicoder-S-DS-6.7B,EffiBench,10,0.35,2.99,6.73,0.6,60.12,1.91,2.61,0.0,16.84,6.05,14.24,89.0,36.3 | |
| Mistral-7B-codealpaca-lora,EffiBench,10,0.31,2.85,3.82,0.0,51.51,1.95,2.36,0.0,10.5,5.81,9.2,88.5,2.6 | |
| octocoder,EffiBench,10,0.32,2.67,2.99,0.0,58.98,1.84,2.02,0.0,11.52,5.07,6.2,75.0,0.4 | |
| gpt-3.5-turbo-0301,EffiBench,10,0.39,3.18,27.7,1.4,60.53,1.91,2.05,0.0,19.06,6.5,70.62,89.1,42.3 | |
| gpt-3.5-turbo-0613,EffiBench,10,0.39,3.22,46.7,0.9,59.82,1.92,2.64,0.0,19.11,6.71,161.12,89.9,46.4 | |
| gpt-3.5-turbo-1106,EffiBench,10,0.4,3.4,68.71,1.6,59.34,1.94,9.12,0.2,19.39,7.24,182.63,90.9,49.3 | |
| gpt-4,EffiBench,10,0.37,3.12,13.89,1.0,58.85,1.92,2.25,0.0,17.69,6.36,43.92,91.1,50.8 | |
| gpt-4-turbo-preview,EffiBench,10,0.38,3.19,27.0,1.2,57.06,1.93,9.13,0.2,16.92,6.57,68.48,91.1,65.4 | |
| claude-3-haiku,EffiBench,10,0.39,3.28,28.75,0.7,59.15,1.91,2.05,0.0,17.99,6.71,72.87,90.0,42.9 | |
| claude-3-sonnet,EffiBench,10,0.4,3.22,17.43,0.9,60.22,1.91,2.06,0.0,23.29,6.57,50.78,90.5,43.2 | |