Hugging Face's logo

internlm
/
internlm-chat-20b-4bit

Text Generation
Transformers
PyTorch
internlm
feature-extraction
custom_code
Model card Files Community
11
unsubscribe commited on
Commit
86edf4c
•
1 Parent(s): 5bb97d2

Fix error and update benchmark in README

Browse files
Files changed (1) hide show
  1. README.md +25 -9
README.md CHANGED
@@ -12,7 +12,7 @@ pipeline_tag: text-generation
12
  Before proceeding with the inference of `internlm-chat-20b-4bit`, please ensure that lmdeploy is installed.
13
 
14
  ```shell
15
- pip install 'lmdeploy>=0.0.9'
16
  ```
17
 
18
  ## Inference
@@ -31,7 +31,7 @@ As demonstrated in the command below, first convert the model's layout using `tu
31
  # Convert the model's layout and store it in the default path, ./workspace.
32
  python3 -m lmdeploy.serve.turbomind.deploy \
33
  --model-name internlm-chat-20b \
34
- --model-path ./internlm-chat-20b \
35
  --model-format awq \
36
  --group-size 128
37
 
@@ -44,7 +44,7 @@ python3 -m lmdeploy.turbomind.chat ./workspace
44
  If you wish to interact with the model via web UI, please initiate the gradio server as indicated below:
45
 
46
  ```shell
47
- python3 -m lmdeploy.serve.turbomind ./workspace --server_name {ip_addr} --server_port {port}
48
  ```
49
 
50
  Subsequently, you can open the website `http://{ip_addr}:{port}` in your browser and interact with the model.
@@ -65,12 +65,10 @@ We conducted benchmarks on `internlm-chat-20b-4bit`. And `token_throughput` was
65
  **Note**: The `session_len` in `workspace/triton_models/weights/config.ini` is changed to `2056` in our test.
66
 
67
 
68
- | batch | tensor parallel | prompt_tokens | completion_tokens | thr_per_proc(token/s) | thr_per_node(token/s) | rpm (req/min) | mem_per_proc(GB) | mem_per_gpu(GB) | mem_per_node(GB) |
69
- |-------|-----------------|---------------|-------------------|-----------------------|-----------------------|---------------|------------------|-----------------|------------------|
70
- | 1 | 1 | 256 | 512 | 79.12 | 632.98 | - | 15.67 | 15.67 | 125.35 |
71
- | 16 | 1 | 256 | 512 | 708.76 | 5670.1 | 220.23 | 51.48 | 51.48 | 411.85 |
72
-
73
-
74
 
75
  ### token throughput
76
 
@@ -84,6 +82,24 @@ python benchmark/profile_generation.py \
84
  ```
85
  You will find the `token_throughput` metrics in `./token_throughput.csv`
86
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
87
 
88
  ### request throughput
89
 
 
12
  Before proceeding with the inference of `internlm-chat-20b-4bit`, please ensure that lmdeploy is installed.
13
 
14
  ```shell
15
+ pip install 'lmdeploy>=0.0.11'
16
  ```
17
 
18
  ## Inference
 
31
  # Convert the model's layout and store it in the default path, ./workspace.
32
  python3 -m lmdeploy.serve.turbomind.deploy \
33
  --model-name internlm-chat-20b \
34
+ --model-path ./internlm-chat-20b-4bit \
35
  --model-format awq \
36
  --group-size 128
37
 
 
44
  If you wish to interact with the model via web UI, please initiate the gradio server as indicated below:
45
 
46
  ```shell
47
+ python3 -m lmdeploy.serve.gradio.app ./workspace --server_name {ip_addr} --server_port {port}
48
  ```
49
 
50
  Subsequently, you can open the website `http://{ip_addr}:{port}` in your browser and interact with the model.
 
65
  **Note**: The `session_len` in `workspace/triton_models/weights/config.ini` is changed to `2056` in our test.
66
 
67
 
68
+ | batch | tensor parallel | prompt_tokens | completion_tokens | thr_per_proc(token/s) | rpm (req/min) | mem_per_proc(GB) |
69
+ |-------|-----------------|---------------|-------------------|-----------------------|---------------|------------------|
70
+ | 1 | 1 | 256 | 512 | 88.77 | - | 15.65 |
71
+ | 16 | 1 | 256 | 512 | 792.7 | 220.23 | 51.46 |
 
 
72
 
73
  ### token throughput
74
 
 
82
  ```
83
  You will find the `token_throughput` metrics in `./token_throughput.csv`
84
 
85
+ | batch | prompt_tokens | completion_tokens | thr_per_proc(token/s) | thr_per_node(token/s) | rpm(req/min) | mem_per_proc(GB) | mem_per_gpu(GB) | mem_per_node(GB) |
86
+ |-------|---------------|-------------------|-----------------------|-----------------------|--------------|------------------|-----------------|------------------|
87
+ | 1 | 256 | 512 | 88.77 | 710.12 | - | 15.65 | 15.65 | 125.21 |
88
+ | 1 | 512 | 512 | 83.89 | 671.15 | - | 15.68 | 15.68 | 125.46 |
89
+ | 1 | 512 | 1024 | 80.19 | 641.5 | - | 15.68 | 15.68 | 125.46 |
90
+ | 1 | 1024 | 1024 | 72.34 | 578.74 | - | 15.75 | 15.75 | 125.96 |
91
+ | 1 | 1 | 2048 | 80.69 | 645.55 | - | 15.62 | 15.62 | 124.96 |
92
+ | 8 | 256 | 512 | 565.21 | 4521.67 | - | 32.37 | 32.37 | 258.96 |
93
+ | 8 | 512 | 512 | 489.04 | 3912.33 | - | 32.62 | 32.62 | 260.96 |
94
+ | 8 | 512 | 1024 | 467.23 | 3737.84 | - | 32.62 | 32.62 | 260.96 |
95
+ | 8 | 1024 | 1024 | 383.4 | 3067.19 | - | 33.06 | 33.06 | 264.46 |
96
+ | 8 | 1 | 2048 | 487.74 | 3901.93 | - | 32.12 | 32.12 | 256.96 |
97
+ | 16 | 256 | 512 | 792.7 | 6341.6 | - | 51.46 | 51.46 | 411.71 |
98
+ | 16 | 512 | 512 | 639.4 | 5115.17 | - | 51.93 | 51.93 | 415.46 |
99
+ | 16 | 512 | 1024 | 591.39 | 4731.09 | - | 51.93 | 51.93 | 415.46 |
100
+ | 16 | 1024 | 1024 | 449.11 | 3592.85 | - | 52.06 | 52.06 | 416.46 |
101
+ | 16 | 1 | 2048 | 620.5 | 4964.02 | - | 51 | 51 | 407.96 |
102
+
103
 
104
  ### request throughput
105