Update README.md

README.md

@@ -1,3 +1,101 @@
 ---
 license: apache-2.0
+pipeline_tag: text-generation
 ---
+
+<div align="center">
+  <img src="https://raw.githubusercontent.com/InternLM/lmdeploy/0be9e7ab6fe9a066cfb0a09d0e0c8d2e28435e58/resources/lmdeploy-logo.svg" width="450"/>
+</div>
+
+[LMDeploy](https://github.com/InternLM/lmdeploy) supports LLM model inference of 4-bit weight, with the minimum requirement for NVIDIA graphics cards being sm80, such as A10, A100, Geforce 30/40 series.
+
+Before proceeding with the inference of `internlm-chat-20b-4bit`, please ensure that lmdeploy is installed.
+
+```shell
+pip install 'lmdeploy>=0.0.9'
+```
+
+## Inference
+
+Please download `internlm-chat-20b-4bit` model as follows,
+
+```shell
+git-lfs install
+git clone https://huggingface.co/internlm/internlm-chat-20b-4bit
+```
+
+As demonstrated in the command below, first convert the model's layout using `turbomind.deploy`, and then you can interact with the AI assistant in the terminal
+
+```shell
+
+# Convert the model's layout and store it in the default path, ./workspace.
+python3 -m lmdeploy.serve.turbomind.deploy \
+    --model-name internlm-chat-20b \
+    --model-path ./internlm-chat-20b \
+    --model-format awq \
+    --group-size 128
+
+# inference
+python3 -m lmdeploy.turbomind.chat ./workspace
+```
+
+## Serve with gradio
+
+If you wish to interact with the model via web UI, please initiate the gradio server as indicated below:
+
+```shell
+python3 -m lmdeploy.serve.turbomind ./workspace --server_name {ip_addr} --server_port {port}
+```
+
+Subsequently, you can open the website `http://{ip_addr}:{port}` in your browser and interact with the model.
+
+Besides serving with gradio, there are two more serving methods. One is serving with Triton Inference Server (TIS), and the other is an OpenAI-like server named as `api_server`.
+
+Please refer to the [user guide](https://github.com/InternLM/lmdeploy#quick-start) for detailed information if you are interested. 
+
+
+## Inference Performance
+
+LMDeploy provides scripts for benchmarking `token throughput` and `request throughput`.
+
+`token throughput` tests the speed of generating new tokens, given a specified number of prompt tokens and completion tokens, while `request throughput` measures the number of requests processed per minute with real dialogue data.
+
+We conducted benchmarks on `internlm-chat-20b-4bit`. And `token_throughput` was measured by setting 256 prompt tokens and generating 512 tokens in response on A100-80G. 
+
+**Note**: The `session_len` in `workspace/triton_models/weights/config.ini` is changed to `2056` in our test.
+
+
+| batch | tensor parallel | prompt_tokens | completion_tokens | thr_per_proc(token/s) | thr_per_node(token/s) | rpm (req/min) | mem_per_proc(GB) | mem_per_gpu(GB) | mem_per_node(GB) |
+|-------|-----------------|---------------|-------------------|-----------------------|-----------------------|---------------|------------------|-----------------|------------------|
+| 1     | 1               | 256           | 512               | 79.12                 | 632.98                | -             | 15.67            | 15.67           | 125.35           |
+| 16    | 1               | 256           | 512               | 708.76                | 5670.1                | 220.23        | 51.48            | 51.48           | 411.85           |
+
+
+
+### token throughput
+
+Run the following command,
+
+```shell
+python benchmark/profile_generation.py \
+  --model-path ./workspace \
+  --concurrency 1 8 16 --prompt-tokens 256 512 512 1024 --completion-tokens 512 512 1024 1024
+  --dst-csv ./token_throughput.csv
+```
+You will find the `token_throughput` metrics in `./token_throughput.csv`
+
+
+### request throughput
+
+LMDeploy uses ShareGPT dataset to test request throughput. Try the next commands, and you will get the `rpm` (request per minute) metric.
+
+```
+# download the ShareGPT dataset
+wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
+
+# 
+python profile_throughput.py \
+ ShareGPT_V3_unfiltered_cleaned_split.json \
+ ./workspace \
+ --concurrency 16
+```