Metrics
TGI exposes multiple metrics that can be collected via the /metrics
Prometheus endpoint.
These metrics can be used to monitor the performance of TGI, autoscale deployment and to help identify bottlenecks.
The following metrics are exposed:
Metric Name | Description | Type | Unit |
---|---|---|---|
tgi_batch_current_max_tokens | Maximum tokens for the current batch | Gauge | Count |
tgi_batch_current_size | Current batch size | Gauge | Count |
tgi_batch_decode_duration | Time spent decoding a batch per method (prefill or decode) | Histogram | Seconds |
tgi_batch_filter_duration | Time spent filtering batches and sending generated tokens per method (prefill or decode) | Histogram | Seconds |
tgi_batch_forward_duration | Batch forward duration per method (prefill or decode) | Histogram | Seconds |
tgi_batch_inference_count | Inference calls per method (prefill or decode) | Counter | Count |
tgi_batch_inference_duration | Batch inference duration | Histogram | Seconds |
tgi_batch_inference_success | Number of successful inference calls per method (prefill or decode) | Counter | Count |
tgi_batch_next_size | Batch size of the next batch | Histogram | Count |
tgi_queue_size | Current queue size | Gauge | Count |
tgi_request_count | Total number of requests | Counter | Count |
tgi_request_duration | Total time spent processing the request (e2e latency) | Histogram | Seconds |
tgi_request_generated_tokens | Generated tokens per request | Histogram | Count |
tgi_request_inference_duration | Request inference duration | Histogram | Seconds |
tgi_request_input_length | Input token length per request | Histogram | Count |
tgi_request_max_new_tokens | Maximum new tokens per request | Histogram | Count |
tgi_request_mean_time_per_token_duration | Mean time per token per request (inter-token latency) | Histogram | Seconds |
tgi_request_queue_duration | Time spent in the queue per request | Histogram | Seconds |
tgi_request_skipped_tokens | Speculated tokens per request | Histogram | Count |
tgi_request_success | Number of successful requests | Counter | |
tgi_request_validation_duration | Time spent validating the request | Histogram | Seconds |