| ABOUT_TEXT = """# Context | |
| The growing number of code models released by the community necessitates a comprehensive evaluation to reliably benchmark their capabilities. Similar to the [🤗 Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard), we selected two common benchmarks for evaluating Code LLMs on multiple programming languages: | |
| - **[HumanEval](https://huggingface.co/datasets/openai_humaneval)** - benchmark for measuring functional correctness for synthesizing programs from docstrings. It consists of 164 Python programming problems. | |
| - **[MultiPL-E](https://huggingface.co/datasets/nuprl/MultiPL-E)** - Translation of HumanEval to 18 programming languages. | |
| - **Throughput Measurement** - In addition to these benchmarks, we also measure model throughput on a batch size of 1 and 50 to compare their inference speed. | |
| ### Benchamrks & Prompts | |
| - HumanEval-Python reports the pass@1 on HumanEval; the rest is from MultiPL-E benchmark. | |
| - We use the original code completion prompts for HumanEval for all models including those instruction tuned. External evaluations might be using different prompts for instruction models like [WizardCoder's instruction](https://github.com/nlpxucan/WizardLM/blob/46d1ce7dbbb1f987ae5e5915c75f33b89a6a17ab/WizardCoder/src/humaneval_gen.py). | |
| ### Evaluation Parameters | |
| - All models were evaluated with the [bigcode-evaluation-harness](https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main) with top-p=0.95, temperature=0.2, max_length_generation 512, and n_samples=50. | |
| ### Throughput and Memory Usage | |
| - Throughputs and peak memory usage are measured using [Optimum-Benchmark](https://github.com/huggingface/optimum-benchmark/tree/main) which powers [Open LLM-Perf Leaderboard](https://huggingface.co/spaces/optimum/llm-perf-leaderboard). (0 throughput corresponds to OOM). | |
| ### Scoring and Rankings | |
| - Average score is the average pass@1 over all languages. For Win Rate, we find model rank for each language and compute `num_models - (rank -1)`, then average this result over all languages. | |
| ### Miscellaneous | |
| - #Languages column represents the number of programming languages included during the pretraining. UNK means the number of languages is unknown. | |
| """ | |
| SUBMISSION_TEXT = """ | |
| <h1 align="center"> | |
| How to submit new results to the leaderboard? | |
| </h1> | |
| We welcome the community to submit evaluation results of new models. These results will be added as non-verified, the authors are however required to upload their generations in case other members want to check. | |
| ### 1 - Running Evaluation | |
| We wrote a detailed guide for running the evaluation on your model. You can find the it in [bigcode-evaluation-harness/leaderboard](https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main/leaderboard). This will generate a json file summarizing the results, in addition to the raw generations and metric files. | |
| ### 2- Submitting Results 🚀 | |
| To submit your results create a **Pull Request** in the community tab to add them under the [folder](https://huggingface.co/spaces/bigcode/multilingual-code-evals/tree/main/community_results) `community_results` in this repository: | |
| - Create a folder called `ORG_MODELNAME_USERNAME` for example `bigcode_starcoder_loubnabnl` | |
| - Put your json file with grouped scores from the guide, in addition generations folder and metrics folder in it. | |
| The title of the PR should be `[Community Submission] Model: org/model, Username: your_username`, replace org and model with those corresponding to the model you evaluated. | |
| """ |