Repository: modelscope/evalscope
Issue: 使用llmuses perf 命令报错
Root cause: Validation Gap

环境： llmuses 0.4.0

我在本地起了一个基于llama3 的vllm api server，准备用llmuses perf命令来做benchmark，命令如下：

(eval-scope) xxx@host:~$ llmuses perf --url 'http://127.0.0.1:8000/v1/chat/completions' --parallel 4 --model 'llama3' --log-every-n-query 10 --read-timeout=120 --dataset-path '/share/home/hetian/dataset/open_qa.jsonl' -n 1 --max-prompt-length 128000 --api openai --stream --n-choices 3 --stop-token-ids 128001 128009 --dataset openqa --debug

报错如下：

Save the result to : ./llama3_benchmark_2024_07_02_15_43_22_647227.db
Traceback (most recent call last):
  File "/share/home/xxx/miniconda3/envs/eval-scope/bin/llmuses", line 8, in <module>
    sys.exit(run_cmd())
  File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/site-packages/llmuses/cli/cli.py", line 19, in run_cmd
    cmd.execute()
  File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/site-packages/llmuses/cli/start_perf.py", line 33, in execute
    run_perf_benchmark(self.args)
  File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/site-packages/llmuses/perf/http_client.py", line 654, in run_perf_benchmark
    asyncio.run(benchmark(args))
  File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/site-packages/llmuses/perf/http_client.py", line 597, in benchmark
    expected_number_of_queries = await dispatch_task  # wait for dispatch task complete
  File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/site-packages/llmuses/perf/http_client.py", line 231, in dispatch_requests_worker
    for messages in message_generator:
  File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/site-packages/llmuses/perf/datasets/openqa.py", line 19, in build_messages
    item = json.loads(item)
  File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/share/home/xxx/miniconda3/envs/eval-scope/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

请问如何处理，多谢了

看起来是/share/home/hetian/dataset/open_qa.jsonl文件不对，不是正常的openqa格式，导致解析失败了，https://huggingface.co/datasets/Hello-SimpleAI/HC3-Chinese/blob/main/open_qa.jsonl
line by line: 
{"question":"比斯巨兽的皮干什么用的？多少钱？ ","human_answers":["做史诗装备用的~~把比斯巨兽（黑石塔上层的一个BOSS）的皮拨了（很小几率出比斯巨兽皮~大部分是厚皮）不管是不是出比斯巨兽的皮都会出现一个地精>（从巨兽肚子中出来的）会给你一个任务这个任务要去冬泉交（交给冬泉瑟银会的NPC）~交完之后他身后有本书上面写着3件史诗装备所需要的原料（都需要比斯巨兽皮和其他物品）\r\n忘记说多少钱了~我们服是600G~700G不一样>的服价格也不一样~"],"chatgpt_answers":["比斯巨兽是什么？我不知道这是什么东西。您可以告诉我更多关于比斯巨兽的信息吗？我将尽力帮助您。"]}


> 看起来是/share/home/hetian/dataset/open_qa.jsonl文件不对，不是正常的openqa格式，导致解析失败了，https://huggingface.co/datasets/Hello-SimpleAI/HC3-Chinese/blob/main/open_qa.jsonl line by line: {"question":"比斯巨兽的皮干什么用的？多少钱？ ","human_answers":["做史诗装备用的~~把比斯巨兽（黑石塔上层的一个BOSS）的皮拨了（很小几率出比斯巨兽皮~大部分是厚皮）不管是不是出比斯巨兽的皮都会出现一个地精>（从巨兽肚子中出来的）会给你一个任务这个任务要去冬泉交（交给冬泉瑟银会的NPC）~交完之后他身后有本书上面写着3件史诗装备所需要的原料（都需要比斯巨兽皮和其他物品）\r\n忘记说多少钱了~我们服是600G~700G不一样>的服价格也不一样~"],"chatgpt_answers":["比斯巨兽是什么？我不知道这是什么东西。您可以告诉我更多关于比斯巨兽的信息吗？我将尽力帮助您。"]}

非常感谢，我之前用wget https://huggingface.co/datasets/Hello-SimpleAI/HC3-Chinese/blob/main/open_qa.jsonl 得到的文件是不对的。这个问题已经解决