Repository: modelscope/evalscope
Issue: 如何对 'http://127.0.0.1:9009/v1/completions'  进行 测试
Root cause: Unimplemented Feature Gap

## 问题描述 / Issue Description

修改  api_url='http://127.0.0.1:9009/v1/completions' 后，执行测试报错 ，是不是不支持呢

## 使用的工具 / Tools Used
- [ ] Native / 原生框架

## 执行的代码或指令 / Code or Commands Executed
from evalscope import TaskConfig, run_task
from evalscope.constants import EvalType

task_cfg = TaskConfig(
    model='DeepSeek-R1',
    api_url='http://127.0.0.1:9009/v1/completions',
    api_key='EMPTY',
    eval_type=EvalType.SERVICE,
    datasets=[
        'mmlu',
    ],
    dataset_args={
        'mmlu': {
            'dataset_id': '/home/zte/workspace/xjf/script/math_500_mmlu_evalscope/evalscopedata/mmlu',
            "filters": {"remove_until": "</think>"
            #'output_types':['multiple_choice_logits']
        }
    },
    eval_batch_size=16,  # num of workers to seed requests
    generation_config={
        'max_tokens': 16000,  # avoid exceed max length
        #'do_sample': true,
        #'temperature': 0.6,
        #'top_p': 0.95,
        #'top_k': 0.40,
        #'n': 1 # num of repeat for each prompt (note lmdeploy only support n=1)
    },
    #limit=10,
    stream=True
)


## 错误日志 / Error Log

Getting answers:   0%|                                                                                                                    | 0/728 [00:02<?, ?it/s]
Traceback (most recent call last):
  File "/aih/AIH/pip/tianshu_math500_test_pip_install/llama-8b_evalscope_math_test_completion_nostream.py", line 30, in <module>
    run_task(task_cfg=task_cfg)
  File "/usr/local/lib/python3.10/site-packages/evalscope/run.py", line 31, in run_task
    return run_single_task(task_cfg, run_time)
  File "/usr/local/lib/python3.10/site-packages/evalscope/run.py", line 44, in run_single_task
    return evaluate_model(task_cfg, outputs)
  File "/usr/local/lib/python3.10/site-packages/evalscope/run.py", line 117, in evaluate_model
    res_dict = evaluator.eval()
  File "/usr/local/lib/python3.10/site-packages/evalscope/collections/evaluator.py", line 265, in eval
    answers = self.get_answers()
  File "/usr/local/lib/python3.10/site-packages/evalscope/collections/evaluator.py", line 212, in get_answers
    answer_list, samples = future.result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/site-packages/evalscope/collections/evaluator.py", line 37, in get_answer
    answer_ds: List[dict] = self.model_adapter.predict(inputs=input_prompts, infer_cfg=infer_cfg)
  File "/usr/local/lib/python3.10/site-packages/evalscope/models/server_adapter.py", line 55, in predict
    response = self.process_single_input(input_item, infer_cfg)
  File "/usr/local/lib/python3.10/site-packages/evalscope/models/server_adapter.py", line 72, in process_single_input
    response = self.send_request(request_json)
  File "/usr/local/lib/python3.10/site-packages/evalscope/models/server_adapter.py", line 111, in send_request
    response = self.client.chat.completions.create(**request_json)
  File "/usr/local/corex-4.2.0/lib64/python3/dist-packages/openai/_utils/_utils.py", line 279, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/corex-4.2.0/lib64/python3/dist-packages/openai/resources/chat/completions.py", line 859, in create
    return self._post(
  File "/usr/local/corex-4.2.0/lib64/python3/dist-packages/openai/_base_client.py", line 1283, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
  File "/usr/local/corex-4.2.0/lib64/python3/dist-packages/openai/_base_client.py", line 960, in request
    return self._request(
  File "/usr/local/corex-4.2.0/lib64/python3/dist-packages/openai/_base_client.py", line 1064, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.NotFoundError: Error code: 404 - {'detail': 'Not Found'}
/usr/local/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpsas8wzt3'>
  _warnings.warn(warn_message, ResourceWarning)
root@49d005cc5a10:/aih/AIH/pip/tianshu_math500_test_pip_install# cd /usr/local/lib/pyth

## 运行环境 / Runtime Environment

- 操作系统 / Operating System:

  - [ ] Ubuntu

- Python版本 / Python Version:

  - [ ] 3.10



## 其他信息 / Additional Information

如果有其他相关信息，请在此处提供。 / If there is any other relevant information, please provide it here.


要测试chat模型的话，可以尝试使用 `http://127.0.0.1:9009/v1/chat/completions`。你提供的url是completion接口，不是chat接口，格式不支持。

> 要测试chat模型的话，可以尝试使用 `http://127.0.0.1:9009/v1/chat/completions`。你提供的url是completion接口，不是chat接口，格式不支持。

那如何测试 completions 接口呢

这块目前还没支持completion的模型服务。非chat模型的评测逻辑还不太一样，得配置few shot，以及从completion中提取答案，后续会支持。