You can use TGI command-line interface (CLI) to download weights, serve and quantize models, or get information on serving parameters. To install the CLI, please refer to the installation section.
text-generation-server lets you download the model with
download-weights command like below 👇
text-generation-server download-weights MODEL_HUB_ID
You can also use it to quantize models like below 👇
text-generation-server quantize MODEL_HUB_ID OUTPUT_DIR
You can use
text-generation-launcher to serve models.
text-generation-launcher --model-id MODEL_HUB_ID --port 8080
There are many options and parameters you can pass to
text-generation-launcher. The documentation for CLI is kept minimal and intended to rely on self-generating documentation, which can be found by running
You can also find it hosted in this Swagger UI.
Same documentation can be found for