Repository: evalplus/evalplus
Issue: The evaluation problems with mbpp plus using docker
Root cause: Documentation Deficiency

docker version: ganler/evalplus:v0.2.1

Hi, I manually generate the code solution using llama3-8b and saved the results in JSONL files with two attribute 'task_id' and 'solution'. I have checked that the number of results and the task_id is exactly matched to the MBPPPlus v0.3.0. However, when I run the following scripts:
`docker run -v $(pwd):/app ganler/evalplus:v0.2.1 --dataset mbpp --samples my_data.jsonl`
It shows:
```
0it [00:00, ?it/s]Computing expected output...
Expected outputs computed in 79.03s
Reading samples...
378it [00:01, 202.29it/s]
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.10/site-packages/evalplus/evaluate.py", line 349, in <module>
    main()
  File "/usr/local/lib/python3.10/site-packages/evalplus/evaluate.py", line 345, in main
    evaluate(args)
  File "/usr/local/lib/python3.10/site-packages/evalplus/evaluate.py", line 202, in evaluate
    assert len(completion_id) == len(problems), "Missing problems in samples"
AssertionError: Missing problems in samples
```

When I run without docker:
`evalplus.evaluate --dataset mbpp --samples my_data.jsonl`
I got the correct answer.

Is any problems here? When evaluate on the humaneval+, it's all fine.

I think you need to use more recent docker image versions such as v0.3.1. Earlier version of MBPP+ has more tasks (buggy tasks are removed in more recent versions). 

Thanks for your in time reply! I've tried v0.3.1 before by running the following command:
`docker run -v $(pwd):/app ganler/evalplus:latest --dataset [humaneval|mbpp] --samples samples.jsonl`
 
 but I got this error:
```
docker: Error response from daemon: OCI runtime create failed: container_linux.go:367: starting container process caused: exec: "--dataset": executable file not found in $PATH: unknown.
ERRO[0000] error waiting for container: context canceled
```

Is there something changes that I did not notice in the README?

I run `docker inspect ganler/evalplus:latest` and find that the "Entrypoint" is null.
```
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "LANG=C.UTF-8",
                "GPG_KEY=A035C8C19219BA821ECEA86B64E628F8D684696D",
                "PYTHON_VERSION=3.11.10",
                "PYTHON_SHA256=07a4356e912900e61a15cb0949a06c4a05012e213ecd6b4e84d0f67aabbee372"
            ],
            "Cmd": [
                "bash"
            ],
            "ArgsEscaped": true,
            "Image": "",
            "Volumes": null,
            "WorkingDir": "/app",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": null
        },
```

While in v0.2.1, the "Entrypoint" is "python3 -m evalplus.evaluate".
```
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "LANG=C.UTF-8",
                "GPG_KEY=A035C8C19219BA821ECEA86B64E628F8D684696D",
                "PYTHON_VERSION=3.10.14",
                "PYTHON_PIP_VERSION=23.0.1",
                "PYTHON_SETUPTOOLS_VERSION=65.5.1",
                "PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/dbf0c85f76fb6e1ab42aa672ffca6f0a675d9ee4/public/get-pip.py",
                "PYTHON_GET_PIP_SHA256=dfe9fd5c28dc98b5ac17979a953ea550cec37ae1b47a5116007395bfacff2ab9"
            ],
            "Cmd": null,
            "Image": "sha256:81e590c6ab80b0670d6334764c79505bf8a9355b67404d928db7ce2c41e3f7f2",
            "Volumes": null,
            "WorkingDir": "/app",
            "Entrypoint": [
                "python3",
                "-m",
                "evalplus.evaluate"
            ],
            "OnBuild": null,
            "Labels": null
        },
```

Maybe that is the reason.

Okay, so using 0.3.1 version of the docker image and run the following command is able to solve the problem, without assert error.
`docker run -v $(pwd):/app ganler/evalplus:latest python3 -m evalplus.evaluate --dataset mbpp --samples samples.jsonl`

Hello, sorry for the confusion -- please refer to our README which provides the example usage:

```bash
docker run --rm --pull=always -v $(pwd)/evalplus_results:/app ganler/evalplus:latest \
           evalplus.evaluate --dataset humaneval                                     \
           --samples /app/humaneval/ise-uiuc--Magicoder-S-DS-6.7B_vllm_temp_0.0.jsonl
```

Thanks for your reply. But the example usage in docs/cli.md is not updated, which may confuse other people who directly use this docker image for evaluation.

Thanks for letting me know! The docu has been updated!