Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

Efficient Inference on a Multiple GPUs

This document contains information on how to efficiently infer on a multiple GPUs.

Note: A multi GPU setup can use the majority of the strategies described in the single GPU section. You must be aware of simple techniques, though, that can be used for a better usage.

`BetterTransformer` for faster inference

We have recently integrated BetterTransformer for faster inference on multi-GPU for text, image and audio models. Check the documentation about this integration here for more details.

←Inference on one GPU Inference on Specialized Hardware→

Transformers

Efficient Inference on a Multiple GPUs

BetterTransformer for faster inference

`BetterTransformer` for faster inference