Repository: modelscope/evalscope
Issue: 为啥测ollama并发性能上不去？有遇到过的么？
Root cause: Configuration Error

## 问题描述 / Issue Description
我测试ollama发现，当并发数大于5后，Throughput(average tokens/s)的值始终上不去，我的显卡 cpu 内存 io都不存在瓶颈，是怎么回事？有人知道么？是不是我的ollama参数有问题？需要配置特殊的环境变量？目前来看并发数上去之后，Average latency会飚高。


## EvalScope 版本 / Version (必须)
v0.14

## 使用的工具 / Tools Used
evalscope


## 执行的代码或指令 / Code or Commands Executed
ollama run deepseek-r1:7b

evalscope perf  --url http://127.0.0.1:11434/v1/chat/completions  --parallel 20  --model deepseek-r1:7b  --number 50 --api openai  --dataset longalpaca  --stream --tokenizer-path /home/data/DeepSeek-R1-Distill-Qwen-7B/


## 运行环境 / Runtime Environment
- 操作系统 / Operating System:
 Ubuntu

- Python版本 / Python Version:
 3.9.21

- 显卡
 NVIDIA L20

##测试并发数对比：
5并发：
+-----------------------------------+-----------------------------------------------------------+
| Key                               | Value                                                     |
+===================================+===========================================================+
| Time taken for tests (s)          | 224.746                                                   |
+-----------------------------------+-----------------------------------------------------------+
| Number of concurrency             | 5                                                         |
+-----------------------------------+-----------------------------------------------------------+
| Total requests                    | 50                                                        |
+-----------------------------------+-----------------------------------------------------------+
| Succeed requests                  | 50                                                        |
+-----------------------------------+-----------------------------------------------------------+
| Failed requests                   | 0                                                         |
+-----------------------------------+-----------------------------------------------------------+
| Throughput(average tokens/s)      | 152.702                                                   |
+-----------------------------------+-----------------------------------------------------------+
| Average QPS                       | 0.222                                                     |
+-----------------------------------+-----------------------------------------------------------+
| Average latency (s)               | 21.28                                                     |
+-----------------------------------+-----------------------------------------------------------+
| Average time to first token (s)   | 4.068                                                     |
+-----------------------------------+-----------------------------------------------------------+
| Average time per output token (s) | 0.048                                                     |
+-----------------------------------+-----------------------------------------------------------+
| Average input tokens per request  | 25.1                                                      |
+-----------------------------------+-----------------------------------------------------------+
| Average output tokens per request | 686.38                                                    |
+-----------------------------------+-----------------------------------------------------------+
| Average package latency (s)       | 0.025                                                     |
+-----------------------------------+-----------------------------------------------------------+
| Average package per request       | 679.92                                                    |
+-----------------------------------+-----------------------------------------------------------+
| Expected number of requests       | 50                                                        |
+-----------------------------------+-----------------------------------------------------------+
| Result DB path                    | outputs/20250412_084339/deepseek-r1_14b/benchmark_data.db |
+-----------------------------------+-----------------------------------------------------------+
2025-04-12 08:47:28,827 - evalscope - INFO -
Percentile results:
+------------+----------+----------+-------------+--------------+---------------+----------------------+
| Percentile | TTFT (s) | TPOT (s) | Latency (s) | Input tokens | Output tokens | Throughput(tokens/s) |
+------------+----------+----------+-------------+--------------+---------------+----------------------+
|    10%     |  0.1842  |  0.023   |    8.563    |      16      |      107      |       18.6992        |
|    25%     |  0.717   |  0.0244  |   15.1911   |      21      |      461      |       26.4268        |
|    50%     |  2.3819  |  0.0251  |   20.1022   |      24      |      669      |       33.6983        |
|    66%     |  5.0805  |  0.0253  |   26.2185   |      27      |      905      |       37.6321        |
|    75%     |  6.6098  |  0.0254  |   29.5318   |      29      |      973      |       38.6047        |
|    80%     |  7.8369  |  0.0256  |   32.565    |      33      |     1052      |       38.7525        |
|    90%     | 12.7792  |  0.0278  |   34.6934   |      36      |     1143      |       41.3198        |
|    95%     | 13.6618  |  0.028   |   36.2059   |      38      |     1234      |       43.1951        |
|    98%     | 15.2459  |  0.0283  |   41.386    |      40      |     1372      |       43.3814        |
|    99%     | 15.2459  |  0.029   |   41.386    |      40      |     1372      |       43.3814        |
+------------+----------+----------+-------------+--------------+---------------+----------------------+



20并发：
+-----------------------------------+-----------------------------------------------------------+
| Key                               | Value                                                     |
+===================================+===========================================================+
| Time taken for tests (s)          | 212.142                                                   |
+-----------------------------------+-----------------------------------------------------------+
| Number of concurrency             | 20                                                        |
+-----------------------------------+-----------------------------------------------------------+
| Total requests                    | 50                                                        |
+-----------------------------------+-----------------------------------------------------------+
| Succeed requests                  | 50                                                        |
+-----------------------------------+-----------------------------------------------------------+
| Failed requests                   | 0                                                         |
+-----------------------------------+-----------------------------------------------------------+
| Throughput(average tokens/s)      | 155.782                                                   |
+-----------------------------------+-----------------------------------------------------------+
| Average QPS                       | 0.236                                                     |
+-----------------------------------+-----------------------------------------------------------+
| Average latency (s)               | 66.508                                                    |
+-----------------------------------+-----------------------------------------------------------+
| Average time to first token (s)   | 50.198                                                    |
+-----------------------------------+-----------------------------------------------------------+
| Average time per output token (s) | 0.261                                                     |
+-----------------------------------+-----------------------------------------------------------+
| Average input tokens per request  | 25.1                                                      |
+-----------------------------------+-----------------------------------------------------------+
| Average output tokens per request | 660.96                                                    |
+-----------------------------------+-----------------------------------------------------------+
| Average package latency (s)       | 0.025                                                     |
+-----------------------------------+-----------------------------------------------------------+
| Average package per request       | 654.54                                                    |
+-----------------------------------+-----------------------------------------------------------+
| Expected number of requests       | 50                                                        |
+-----------------------------------+-----------------------------------------------------------+
| Result DB path                    | outputs/20250412_084818/deepseek-r1_14b/benchmark_data.db |
+-----------------------------------+-----------------------------------------------------------+
2025-04-12 08:51:52,286 - evalscope - INFO -
Percentile results:
+------------+----------+----------+-------------+--------------+---------------+----------------------+
| Percentile | TTFT (s) | TPOT (s) | Latency (s) | Input tokens | Output tokens | Throughput(tokens/s) |
+------------+----------+----------+-------------+--------------+---------------+----------------------+
|    10%     | 13.6881  |  0.0242  |   17.3178   |      16      |      102      |        1.9078        |
|    25%     |  37.714  |  0.0245  |   50.9047   |      21      |      488      |        6.5764        |
|    50%     | 61.9608  |  0.0248  |   76.2277   |      24      |      736      |        9.8613        |
|    66%     | 65.2307  |  0.025   |   81.7474   |      27      |      835      |       11.9863        |
|    75%     | 65.9437  |  0.0252  |   83.9061   |      29      |      919      |       13.2661        |
|    80%     | 66.2474  |  0.0252  |   89.1656   |      33      |      945      |       15.3609        |
|    90%     | 71.3316  |  0.0254  |   90.2123   |      36      |     1027      |       28.0302        |
|    95%     | 71.5301  |  0.0255  |   92.9245   |      38      |     1105      |       39.6859        |
|    98%     | 73.7329  |  0.0258  |   99.0675   |      40      |     1168      |       39.8584        |
|    99%     | 73.7329  |  0.0287  |   99.0675   |      40      |     1168      |       39.8584        |
+------------+----------+----------+-------------+--------------+---------------+----------------------+

可以加一个 export OLLAMA_NUM_PARALLEL=10 试试

> 可以加一个 export OLLAMA_NUM_PARALLEL=10 试试

谢谢 确实如此  默认值是4  你知道为啥ollama的并发性能比较差么？好像远不如vllm

> > 可以加一个 export OLLAMA_NUM_PARALLEL=10 试试
> 
> 谢谢 确实如此 默认值是4 你知道为啥ollama的并发性能比较差么？好像远不如vllm

ollam 不适合做高并发服务的。。。ollama 优势是跨平台，简单易用，并不是极致性能，要追求高并发用 sglang 或 vllm