makaveli10 commited on
Commit
95bcc6a
1 Parent(s): 5d7959c

update README for phi support

Browse files
Files changed (1) hide show
  1. README.md +34 -1
README.md CHANGED
@@ -50,6 +50,27 @@ python build.py --model_dir teknium/OpenHermes-2.5-Mistral-7B \
50
  --max_batch_size 1
51
  ```
52
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
53
  ## Run WhisperBot
54
  - Clone this repo and install requirements.
55
  ```bash
@@ -60,12 +81,24 @@ apt install ffmpeg portaudio19-dev -y
60
  pip install -r requirements.txt
61
  ```
62
 
 
63
  - Take the folder path for Whisper TensorRT model, folder_path and tokenizer_path for Mistral TensorRT from the build phase. If a huggingface model is used to build mistral then just use the huggingface repo name as the tokenizer path.
64
  ```bash
65
- python3 main.py --whisper_tensorrt_path /root/TensorRT-LLM/examples/whisper/whisper_small_en \
 
66
  --mistral_tensorrt_path /root/TensorRT-LLM/examples/llama/tmp/mistral/7B/trt_engines/fp16/1-gpu/ \
67
  --mistral_tokenizer_path teknium/OpenHermes-2.5-Mistral-7B
68
  ```
 
 
 
 
 
 
 
 
 
 
69
  - On the client side clone the repo, install the requirements and execute `run_client.py`
70
  ```bash
71
  cd WhisperBot
 
50
  --max_batch_size 1
51
  ```
52
 
53
+ ### Build Phi TensorRT Engine
54
+ Note: Phi is only available in main branch and hasnt been released yet. So, make sure to build TensorRT-LLM from main branch.
55
+ - Change working dir to [phi example dir](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/phi) in TensorRT-LLM folder.
56
+ ```bash
57
+ cd TensorRT-LLM/examples/phi
58
+ ```
59
+ - Build phi TensorRT engine
60
+ ```bash
61
+ git lfs install
62
+ git clone https://huggingface.co/microsoft/phi-2
63
+ python3 build.py --dtype=float16 \
64
+ --log_level=verbose \
65
+ --use_gpt_attention_plugin float16 \
66
+ --use_gemm_plugin float16 \
67
+ --max_batch_size=16 \
68
+ --max_input_len=1024 \
69
+ --max_output_len=1024 \
70
+ --output_dir=phi_engine \
71
+ --model_dir=phi-2>&1 | tee build.log
72
+ ```
73
+
74
  ## Run WhisperBot
75
  - Clone this repo and install requirements.
76
  ```bash
 
81
  pip install -r requirements.txt
82
  ```
83
 
84
+ ### Whisper + Mistral
85
  - Take the folder path for Whisper TensorRT model, folder_path and tokenizer_path for Mistral TensorRT from the build phase. If a huggingface model is used to build mistral then just use the huggingface repo name as the tokenizer path.
86
  ```bash
87
+ python3 main.py --mistral
88
+ --whisper_tensorrt_path /root/TensorRT-LLM/examples/whisper/whisper_small_en \
89
  --mistral_tensorrt_path /root/TensorRT-LLM/examples/llama/tmp/mistral/7B/trt_engines/fp16/1-gpu/ \
90
  --mistral_tokenizer_path teknium/OpenHermes-2.5-Mistral-7B
91
  ```
92
+
93
+ ### Whisper + Phi
94
+ - Take the folder path for Whisper TensorRT model, folder_path and tokenizer_path for Phi TensorRT from the build phase. If a huggingface model is used to build phi then just use the huggingface repo name as the tokenizer path.
95
+ ```bash
96
+ python3 main.py --phi
97
+ --whisper_tensorrt_path /root/TensorRT-LLM/examples/whisper/whisper_small_en \
98
+ --phi_tensorrt_path /root/TensorRT-LLM/examples/phi/phi_engine \
99
+ --phi_tokenizer_path /root/TensorRT-LLM/examples/phi/phi-2
100
+ ```
101
+
102
  - On the client side clone the repo, install the requirements and execute `run_client.py`
103
  ```bash
104
  cd WhisperBot