Text Generation Inference enables serving optimized models on specific hardware for the highest performance. The following sections list which models are hardware are supported.
The following models are optimized and can be served with TGI, which uses custom CUDA kernels for better inference. You can add the flag
--disable-custom-kernels at the end of the
docker run command if you wish to disable them.
- Falcon 7B
- Falcon 40B
- Llama V2
- Code Llama
If the above list lacks the model you would like to serve, depending on the model’s pipeline type, you can try to initialize and serve the model anyways to see how well it performs, but performance isn’t guaranteed for non-optimized models:
# for causal LMs/text-generation models AutoModelForCausalLM.from_pretrained(<model>, device_map="auto")` # or, for text-to-text generation models AutoModelForSeq2SeqLM.from_pretrained(<model>, device_map="auto")
If you wish to serve a supported model that already exists on a local folder, just point to the local folder.
text-generation-launcher --model-id <PATH-TO-LOCAL-BLOOM>
TGI optimized models are supported on NVIDIA A100, A10G and T4 GPUs with CUDA 11.8+. Note that you have to install NVIDIA Container Toolkit to use it. For other NVIDIA GPUs, continuous batching will still apply, but some operations like flash attention and paged attention will not be executed.
TGI also has support of RoCm-enabled AMD Instinct MI210 and MI250 GPUs, with paged attention and flash attention v2 support. The following features are missing from the RoCm version of TGI: quantization and flash layer norm kernel.
TGI is also supported on the following AI hardware accelerators: