--- title: BabelCode Eval colorFrom: blue colorTo: red sdk: gradio sdk_version: 3.19.1 app_file: app.py pinned: false tags: - evaluate - metric description: >- This metric implements the evaluation harness for datasets translated with the BabelCode framework as described in the paper "Measuring The Impact Of Programming Language Distribution" (https://arxiv.org/abs/2302.01973). --- # Metric Card for bc_eval ## Metric Description This metric implements the evaluation harness for datasets translated with the BabelCode framework as described in the paper "Measuring The Impact Of Programming Language Distribution" (https://arxiv.org/abs/2302.01973). ## How to Use 1. Generate predictions for BabelCode supported datasets 2. Aggregate the predictions by their question. 3. With the aggregated predictions for each question, add the `question_info` from the original BabelCode dataset. 4. Run the metric on the `predictions`, `languages`, and `question_infos`. 5. The result of the metric is a tuple where the first is a metric dict and the second value is the results for each prediction. ```Python import evaluate from datasets import load_dataset import os os.environ["HF_ALLOW_CODE_EVAL"] = "1" predictions = [] languages = [] question_infos = [] ds = load_dataset("gabeorlanski/bc-humaneval", split="test") for row in ds: languages.append(row['language']) question_infos.append(row['question_info']) # Replace this with however you generate and postprocess predictions. predictions.append(model.generate(row['signature_with_docstring'])) metric = evaluate.load("gabeorlanski/bc_eval") metrics, results = metric.compute( predictions=predictions, languages=languages, question_dicts=question_infos, k=[1] ) ``` ### Inputs * `predictions`(`List[List[str]]`): The list of predictions for each question to execute. * `languages`(`List[str]`): The language to use for each question. * `question_dicts`(`List[Dict]`): The information for each question. * `k`(`List[int]`): number of code candidates to consider in the evaluation (Default: [1, 10, 100]) * `num_workers`(`int`): number of workers used to evaluate the candidate programs (Default: 4). * `language_timeout`(`Dict[str,int]`): Timeouts to use for each language. If it is not set, will default to the one in the question dict (Default: None). ### Output Values The `bc_eval` metric outputs two things: `metrics`: a dictionary with the pass rates for each k value defined in the arguments and the mean percent of tests passed per question. The keys are formatted as `{LANGUAGE NAME}/{METRIC NAME}` `results`: a list of dictionaries with the results from each individual prediction. #### Values from Popular Papers [PaLM-2](https://arxiv.org/pdf/2305.10403.pdf) Performance on BC-HumanEval (`pass@1` with greedy decoding): | Language | PaLM 2-S* | PaLM 540B | PaLM-Coder-540B | |------------|-----------|-----------|-----------------| | C# | 24.22 | 20.5 | **26.09** | | C++ | **34.16** | 21.74 | 24.22 | | Go | 19.25 | 13.66 | **21.12** | | Haskell | **8.7** | 1.86 | 1.86 | | Java | **31.06** | 20.5 | 25.47 | | JavaScript | **32.3** | 23.6 | 29.81 | | Julia | **16.77** | 2.48 | 4.35 | | Lua | **26.09** | 19.25 | 24.84 | | PHP | **26.09** | 18.63 | 25.47 | | Python | **34.16** | 17.39 | 26.71 | | Rust | **28.57** | 16.15 | 22.98 | | TypeScript | **32.3** | 17.39 | 30.43 | ### Examples Full example with inputs that fail tests, time out, have an error, and pass. #### Passing Example ```Python import evaluate from datasets import load_dataset import os os.environ["HF_ALLOW_CODE_EVAL"] = "1" ds = load_dataset("gabeorlanski/bc-humaneval", split="test") example = ds[0] metric = evaluate.load("gabeorlanski/bc_eval") languages = ["Python"] question_infos = [example["question_info"]] predictions = [["""def has_close_elements(numbers: List[float], threshold: float) -> bool: for idx, elem in enumerate(numbers): for idx2, elem2 in enumerate(numbers): if idx != idx2: distance = abs(elem - elem2) if distance < threshold: return True return False""" ]] metrics, results = metric.compute( predictions=predictions, languages=languages, question_dicts=question_infos, k=[1] ) ``` `metrics` is: ``` {"Python/pass@1": 1.0, "Python/mean_pct_pass": 1.0} ``` `results` is: ``` [ { "qid": 0, "idx": "0", "file_path": ".../tmpqt_p3dwn/0", "results": [ { "return_code": 0, "runtime": 0.076369, "stdout": "TEST-0...PASSED\r\nTEST-1...PASSED\r\nTEST-2...PASSED\r\nTEST-3...PASSED\r\nTEST-4...PASSED\r\nTEST-5...PASSED\r\nTEST-6...PASSED\r\n", "stderr": "", "timed_out": false, } ], "failed": false, "timed_out": false, "test_cases": { "0": "PASSED", "1": "PASSED", "2": "PASSED", "3": "PASSED", "4": "PASSED", "5": "PASSED", "6": "PASSED", }, "outcome": "PASSED", } ] ``` #### Fails Test Example ```python import evaluate from datasets import load_dataset import os os.environ["HF_ALLOW_CODE_EVAL"] = "1" ds = load_dataset( "gabeorlanski/bc-humaneval", "Python", split="test" ) example = ds[0] metric = evaluate.load("gabeorlanski/bc_eval") languages = ["Python"] question_infos = [example["question_info"]] predictions = [["""def has_close_elements(numbers: List[float], threshold: float) -> bool: for idx, elem in enumerate(numbers): for idx2, elem2 in enumerate(numbers): if idx != idx2: distance = elem - elem2 if distance < threshold: return True return False""" ]] metrics, results = metric.compute( predictions=predictions, languages=languages, question_dicts=question_infos, k=[1] ) ``` `metrics` is: ``` {"Python/pass@1": 0.0, "Python/mean_pct_pass": 0.5714285714285714} ``` `results` is: ``` [{"qid": 0, "idx": "0", "file_path": "/tmp7u587vk5/0", "results": [{"return_code": 0, "runtime": 0.08255, "stdout": "TEST-0...PASSED\r\nTEST-1...FAILED\r\nTEST-2...PASSED\r\nTEST-3...FAILED\r\nTEST-4...PASSED\r\nTEST-5...PASSED\r\nTEST-6...FAILED\r\n", "stderr": "", "timed_out": false}], "failed": false, "timed_out": false, "test_cases": {"0": "PASSED", "1": "FAILED", "2": "PASSED", "3": "FAILED", "4": "PASSED", "5": "PASSED", "6": "FAILED"}, "outcome": "FAILED"}] ``` Note that the individual test results are located in results. #### Timeout Example ```python import evaluate from datasets import load_dataset import os os.environ["HF_ALLOW_CODE_EVAL"] = "1" ds = load_dataset( "gabeorlanski/bc-humaneval", "Python", split="test" ) example = ds[0] metric = evaluate.load("gabeorlanski/bc_eval") languages = ["Python"] question_infos = [example["question_info"]] predictions = [["""import time def has_close_elements(numbers: List[float], threshold: float) -> bool: time.sleep(100) """ ]] metrics, results = metric.compute( predictions=predictions, languages=languages, question_dicts=question_infos, k=[1] ) ``` `metrics` is: ``` {"Python/pass@1": 0.0, "Python/mean_pct_pass": 0.0} ``` `results` is: ``` [{"qid": 0, "idx": "0", "file_path": "/tmp_rz6bhb9/0", "results": [{"return_code": -1, "runtime": 10, "stdout": null, "stderr": null, "timed_out": true}], "failed": false, "timed_out": true, "test_cases": {"0": "MISSING", "1": "MISSING", "2": "MISSING", "3": "MISSING", "4": "MISSING", "5": "MISSING", "6": "MISSING"}, "outcome": "TIMED_OUT"}] ``` #### Error Example ```python import evaluate from datasets import load_dataset import os os.environ["HF_ALLOW_CODE_EVAL"] = "1" ds = load_dataset( "gabeorlanski/bc-humaneval", "Python", split="test" ) example = ds[0] metric = evaluate.load("gabeorlanski/bc_eval") languages = ["Python"] question_infos = [example["question_info"]] predictions = [["""import time def has_close_elements(numbers: List[float], threshold: float) -> bool: raise ValueError() """, """def add(a, b): return a+b""" ]] metrics, results = metric.compute( predictions=predictions, languages=languages, question_dicts=question_infos, k=[1] ) ``` `metrics` is: ``` {"Python/pass@1": 0.0, "Python/mean_pct_pass": 0.0} ``` `results` is: ```[{"qid": 0, "idx": "0", "file_path": "/tmpjdn51aaa/0", "results": [{"return_code": 0, "runtime": 0.102855, "stdout": "TEST-0...ValueError\r\nTEST-1...ValueError\r\nTEST-2...ValueError\r\nTEST-3...ValueError\r\nTEST-4...ValueError\r\nTEST-5...ValueError\r\nTEST-6...ValueError\r\n", "stderr": "", "timed_out": false}], "failed": false, "timed_out": false, "test_cases": {"0": "ValueError", "1": "ValueError", "2": "ValueError", "3": "ValueError", "4": "ValueError", "5": "ValueError", "6": "ValueError"}, "outcome": "HAD_ERROR"}, {"qid": 0, "idx": "1", "file_path": "/tmpjdn51aaa/1", "results": [{"return_code": 0, "runtime": 0.094347, "stdout": "TEST-0...NameError\r\nTEST-1...NameError\r\nTEST-2...NameError\r\nTEST-3...NameError\r\nTEST-4...NameError\r\nTEST-5...NameError\r\nTEST-6...NameError\r\n", "stderr": "", "timed_out": false}], "failed": false, "timed_out": false, "test_cases": {"0": "NameError", "1": "NameError", "2": "NameError", "3": "NameError", "4": "NameError", "5": "NameError", "6": "NameError"}, "outcome": "HAD_ERROR"}] ``` ## Limitations and Bias This metric requires that the dataset be BabelCode compatible. ## Citation ``` @article{orlanski2023measuring, title={Measuring The Impact Of Programming Language Distribution}, author={Orlanski, Gabriel and Xiao, Kefan and Garcia, Xavier and Hui, Jeffrey and Howland, Joshua and Malmaud, Jonathan and Austin, Jacob and Singh, Rishah and Catasta, Michele}, journal={arXiv preprint arXiv:2302.01973}, year={2023} } ```