Repository: bigcode-project/bigcode-evaluation-harness
Issue: Unable to execute the MultiPL-E task for C++
Root cause: Validation Gap

Hi,
I want to eval my model for the MultiPL-E task for C++.
But,
When --tasks is **multiple-cpp** is passed in, an error is reported:
`main.py: error: argument --tasks: invalid choice: 'multiple-cpp' (choose from 'apps-competition', 'apps-interview', 'apps-introductory', 'codexglue_code_to_text-go', 'codexglue_code_to_text-java', 'codexglue_code_to_text-javascript', 'codexglue_code_to_text-php', 'codexglue_code_to_text-python', 'codexglue_code_to_text-python-left', 'codexglue_code_to_text-ruby', 'codexglue_text_to_text-da_en', 'codexglue_text_to_text-lv_en', 'codexglue_text_to_text-no_en', 'codexglue_text_to_text-zh_en', 'conala', 'concode', 'ds1000-all-completion', 'ds1000-all-insertion', 'ds1000-matplotlib-completion', 'ds1000-matplotlib-insertion', 'ds1000-numpy-completion', 'ds1000-numpy-insertion', 'ds1000-pandas-completion', 'ds1000-pandas-insertion', 'ds1000-pytorch-completion', 'ds1000-pytorch-insertion', 'ds1000-scipy-completion', 'ds1000-scipy-insertion', 'ds1000-sklearn-completion', 'ds1000-sklearn-insertion', 'ds1000-tensorflow-completion', 'ds1000-tensorflow-insertion', 'humaneval', 'humaneval-unstripped', 'humanevalexplaindescribe-cpp', 'humanevalexplaindescribe-go', 'humanevalexplaindescribe-java', 'humanevalexplaindescribe-js', 'humanevalexplaindescribe-python', 'humanevalexplaindescribe-rust', 'humanevalexplainsynthesize-cpp', 'humanevalexplainsynthesize-go', 'humanevalexplainsynthesize-java', 'humanevalexplainsynthesize-js', 'humanevalexplainsynthesize-python', 'humanevalexplainsynthesize-rust', 'humanevalfixdocs-cpp', 'humanevalfixdocs-go', 'humanevalfixdocs-java', 'humanevalfixdocs-js', 'humanevalfixdocs-python', 'humanevalfixdocs-rust', 'humanevalfixtests-cpp', 'humanevalfixtests-go', 'humanevalfixtests-java', 'humanevalfixtests-js', 'humanevalfixtests-python', 'humanevalfixtests-rust', 'humanevalplus', 'humanevalplus-unstripped', 'humanevalsynthesize-cpp', 'humanevalsynthesize-go', 'humanevalsynthesize-java', 'humanevalsynthesize-js', 'humanevalsynthesize-python', 'humanevalsynthesize-rust', 'instruct-humaneval', 'instruct-humaneval-nocontext', 'instruct_wizard_humaneval', 'mbpp', 'mbppplus', 'mercury', 'multiple-cljcpp', 'multiple-cs', 'multiple-d', 'multiple-dart', 'multiple-elixir', 'multiple-go', 'multiple-hs', 'multiple-java', 'multiple-jl', 'multiple-js', 'multiple-lua', 'multiple-mlpl', 'multiple-php', 'multiple-py', 'multiple-r', 'multiple-rb', 'multiple-rkt', 'multiple-rs', 'multiple-scala', 'multiple-sh', 'multiple-swift', 'multiple-ts', 'pal-gsm8k-greedy', 'pal-gsm8k-majority_voting', 'pal-gsmhard-greedy', 'pal-gsmhard-majority_voting', 'parity', 'perturbed-humaneval-format-num_seeds_1', 'perturbed-humaneval-format-num_seeds_10', 'perturbed-humaneval-format-num_seeds_2', 'perturbed-humaneval-format-num_seeds_3', 'perturbed-humaneval-format-num_seeds_4', 'perturbed-humaneval-format-num_seeds_5', 'perturbed-humaneval-format-num_seeds_6', 'perturbed-humaneval-format-num_seeds_7', 'perturbed-humaneval-format-num_seeds_8', 'perturbed-humaneval-format-num_seeds_9', 'perturbed-humaneval-func_name-num_seeds_1', 'perturbed-humaneval-func_name-num_seeds_10', 'perturbed-humaneval-func_name-num_seeds_2', 'perturbed-humaneval-func_name-num_seeds_3', 'perturbed-humaneval-func_name-num_seeds_4', 'perturbed-humaneval-func_name-num_seeds_5', 'perturbed-humaneval-func_name-num_seeds_6', 'perturbed-humaneval-func_name-num_seeds_7', 'perturbed-humaneval-func_name-num_seeds_8', 'perturbed-humaneval-func_name-num_seeds_9', 'perturbed-humaneval-natgen-num_seeds_1', 'perturbed-humaneval-natgen-num_seeds_10', 'perturbed-humaneval-natgen-num_seeds_2', 'perturbed-humaneval-natgen-num_seeds_3', 'perturbed-humaneval-natgen-num_seeds_4', 'perturbed-humaneval-natgen-num_seeds_5', 'perturbed-humaneval-natgen-num_seeds_6', 'perturbed-humaneval-natgen-num_seeds_7', 'perturbed-humaneval-natgen-num_seeds_8', 'perturbed-humaneval-natgen-num_seeds_9', 'perturbed-humaneval-nlaugmenter-num_seeds_1', 'perturbed-humaneval-nlaugmenter-num_seeds_10', 'perturbed-humaneval-nlaugmenter-num_seeds_2', 'perturbed-humaneval-nlaugmenter-num_seeds_3', 'perturbed-humaneval-nlaugmenter-num_seeds_4', 'perturbed-humaneval-nlaugmenter-num_seeds_5', 'perturbed-humaneval-nlaugmenter-num_seeds_6', 'perturbed-humaneval-nlaugmenter-num_seeds_7', 'perturbed-humaneval-nlaugmenter-num_seeds_8', 'perturbed-humaneval-nlaugmenter-num_seeds_9', 'python_bugs', 'quixbugs', 'santacoder_fim', 'starcoder_fim', 'studenteval')`
When --tasks is **multiple-cljcpp** is passed in, an error is reported:
`ValueError: BuilderConfig 'humaneval-cljcpp' not found. Available: ['humaneval-clj', 'humaneval-cpp', 'humaneval-cs', 'humaneval-d', 'humaneval-dart', 'humaneval-elixir', 'humaneval-go', 'humaneval-hs', 'humaneval-java', 'humaneval-jl', 'humaneval-js', 'humaneval-lua', 'humaneval-ml', 'humaneval-php', 'humaneval-pl', 'humaneval-r', 'humaneval-rb', 'humaneval-rkt', 'humaneval-rs', 'humaneval-scala', 'humaneval-sh', 'humaneval-swift', 'humaneval-ts', 'mbpp-clj', 'mbpp-cpp', 'mbpp-cs', 'mbpp-d', 'mbpp-elixir', 'mbpp-go', 'mbpp-hs', 'mbpp-java', 'mbpp-jl', 'mbpp-js', 'mbpp-lua', 'mbpp-ml', 'mbpp-php', 'mbpp-pl', 'mbpp-r', 'mbpp-rb', 'mbpp-rkt', 'mbpp-rs', 'mbpp-scala', 'mbpp-sh', 'mbpp-swift', 'mbpp-ts']`

**_What value should I pass in?_**




I also ran into this issue—it seems to be caused by a missing comma.
https://github.com/bigcode-project/bigcode-evaluation-harness/pull/280